Venue: NAACL 2021

# Active\({}^{2}\) Learning: Actively reducing redundancies in Active Learning

methods for Sequence Tagging and Machine Translation

Rishi Hazra, Parag Dutta, Shubham Gupta,

**Mohammed Abdul Qaathir, Ambedkar Dulkkipati {rishihazza, paragdutta, shubhang, mohammedq, ambedkar}@iisc.ac.in Indian Institute of Science, Bangalore**

###### Abstract

While deep learning is a powerful tool for natural language processing (NLP) problems, successful solutions to these problems rely heavily on large amounts of annotated samples. However, manually annotating data is expensive and time-consuming. Active Learning (AL) strategies reduce the need for huge volumes of labeled data by iteratively selecting a small number of examples for manual annotation based on their estimated utility in training the given model. In this paper, we argue that since AL strategies choose examples independently, they may potentially select similar examples, all of which may not contribute significantly to the learning process. Our proposed approach, Active\({}^{2}\) Learning (A\({}^{2}\)L), actively adapts to the deep learning model being trained to eliminate such redundant examples chosen by an AL strategy. We show that A\({}^{2}\)L is widely applicable by using it in conjunction with several different AL strategies and NLP tasks. We empirically demonstrate that the proposed approach is further able to reduce the data requirements of state-of-the-art AL strategies by \(\approx\mathbf{3-25\%}\) on an absolute scale on multiple NLP tasks while achieving the same performance with virtually no additional computation overhead.

+
Footnote †: _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1982–1995 June 6–11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

Active Learning (AL) Freund et al. (1997); McCallum and Nigam (1998) reduces the need for large quantities of labeled data by intelligently selecting unlabeled examples for expert annotation in an iterative process. Many Natural Language Processing (NLP) tasks like sequence tagging (NER, POS) and Neural Machine Translation (NMT) are very data-intensive and require a meticulous, time-consuming, and costly annotation process. On the other hand, unlabeled data is practically unlimited. Due to this, many researchers have explored applications of active learning for NLP Thompson et al. (1999); Figueroa et al. (2012). A general AL method proceeds as follows: **(i)** The partially trained model for a given task is used to (possibly incorrectly) annotate the unlabeled examples. **(ii)** An _active learning strategy_ selects a subset of the newly labeled examples via a criterion that quantifies the perceived utility of examples in training the model. **(iii)** The experts verify/improve the annotations for the selected examples. **(iv)** These examples are added to the training set, and the process repeats. AL strategies differ in the criterion used in step (ii).

We claim that _all_ AL strategies select redundant examples in step **(ii)**. If one example satisfies the selection criterion, then many other _similar_ examples will also satisfy it (see the next paragraph for details). As the examples are selected independently, AL strategies redundantly choose all of these examples even though, in practice, it is enough to label only a few of them (ideally just one) for training the model. This leads to higher annotation costs, wastage of resources, and reduces the effectiveness of AL strategies. This paper addresses this problem by proposing a new approach called A\({}^{2}\)L (read as active-squared learning) that further reduces the redundancies of existing AL strategies.

Any approach for eliminating redundant examples must have the following qualities: **(i)** The redundancy should be evaluated in the context of the trained model. **(ii)** The approach should apply to a wide variety of commonly used models in NLP. **(iii)** It should be compatible with several existing AL strategies. The first point merits more explanation. As a model is trained, depending on the downstream task, it learns to focus on certain properties of the input. Examples that share these properties (for instance, the sentence structure) are similar from the model's perspective. If the model is confused about one such example, it will likely be confused about all of them. We refer to a similarity measure that is computed in the context of a model as a _model-aware similarity_ (Section 3.1).

**Contributions:****(i)** We propose a Siamese twin

(Bromley et al., 1994; Mueller and Thyagarajan, 2016) based method for computing model-aware similarity to eliminate redundant examples chosen by an AL strategy. This Siamese network actively adapts itself to the underlying model as the training progresses. We then use clustering based on similarity scores to eliminate redundant examples. **(ii)** We develop a second, computationally more efficient approach that approximates the first one with a minimal drop in performance by avoiding the clustering step. Both of these approaches have the desirable properties mentioned above. **(iii)** We experiment with several AL strategies and NLP tasks to empirically demonstrate that our approaches are widely applicable and significantly reduce the data requirements of existing AL strategies while achieving the same performance. To the best of our knowledge, we are the first to identify the importance of model-aware similarity and exploit it to address the problem of redundancy in AL.

## 2 Related Work

Active learning has a long and successful history in the field of machine learning (Dasgupta et al., 2009; Awasthi et al., 2017). However, as the learning models have become more complex, especially with the advent of deep learning, the known theoretical results for active learning are no longer applicable (Shen et al., 2018). This has prompted a diverse range of heuristics to adapt the active learning framework to deep learning models (Shen et al., 2018). Many AL strategies have been proposed (Sha and Saul, 2007; Haffari et al., 2009; Bloodgood and Callison-Burch, 2010; Blundell et al., 2015; Gal and Ghahramani, 2016a), however, since they choose the examples independently, the problem of redundancy (Section 1) applies to all.

We experiment with various NLP tasks like named entity recognition (NER) (Nadeau and Sekine, 2007), part-of-speech tagging (POS) (Marcus et al., 1993), neural machine translation (NMT) (Hutchins, 2004; Nepveu et al., 2004; Bahdanau et al., 2014; Cho et al., 2014; Sutskever et al., 2014; Ortiz-Martinez, 2016) and so on (Landes et al., 1998; Tjong Kim Sang and Buchholz, 2000). The tasks chosen by us form the backbone of many practical problems and are known to be computationally expensive during both training and inference. Many deep learning models have recently advanced the state-of-art for these tasks (Bahdanau et al., 2014; Lample et al., 2016; Siddant and Lipton, 2018). Our proposed approach is compatible with any NLP model, provided it supports the usage of an AL strategy.

Existing approaches have used model-independent similarity scores to promote diversity in the chosen examples. For instance, in Chen et al. (2015), the authors use cosine similarity to pre-calculate pairwise similarity between examples. We instead argue in favor of model-aware similarity scores and learn an expressive notion of similarity using neural networks. We compare our approach with a modified version of this baseline using cosine similarity on Infersent embeddings (Conneau et al., 2017).

## 3 Proposed Approaches

We use \({\cal M}\) to denote the model being trained for a given task. \({\cal M}\) has a module called encoder for encoding the input sentences. For instance, the encoder in \({\cal M}\) may be modeled by an LSTM (Hochreiter and Schmidhuber, 1997).

### Model-Aware Similarity Computation

A measure of similarity between examples is required to discover redundancy. The simplest solution is to compute the cosine similarity between input sentences (Chen et al., 2015; Shen et al., 2018) using, for instance, the InferSent encodings (Conneau et al., 2017). However, sentences that have a low cosine similarity may still be similar in the context of the downstream task. Model \({\cal M}\) has no incentive to distinguish among such examples. A good strategy is to label a diverse set of sentences from the perspective of the model. For example, it is unnecessary to label sentences that use different verb forms but are otherwise similar if the task is agnostic to the tense of the sentence. A straightforward extension of cosine similarity to the encodings generated by model \({\cal M}\) achieves this. However, a simplistic approach like this would likely be incapable of discovering complex similarity patterns in the data. Next, we describe two approaches that use more expressive model-aware similarity measures.

### Model-Aware Siamese

In this approach, we use a Siamese twin's network (Bromley et al., 1994) to compute the pairwise similarity between encodings obtained from model \({\cal M}\). A Siamese twin's network consists of an encoder (called the Siamese encoder) that feeds on the output of model \({\cal M}\)'s encoder. The outputs of the 

Siamese encoder are used for computing the similarity between each pair of examples \(a\) and \(b\) as:

\[{\rm sim}(a,b)=\exp{(-||{\bf o^{a}}-{\bf o}^{b}||_{2})}, \tag{1}\]

where \({\bf o}^{a}\) and \({\bf o}^{b}\) are the outputs of the Siamese encoder for sentences \(a\) and \(b\) respectively. Let \(N\) denote the number of examples chosen by an AL strategy. We use the Siamese network to compute the entries of an \(N\times N\) similarity matrix \({\bf S}\) where the entry \(S_{ab}={\rm sim}(a,b)\). We then use the spectral clustering algorithm Ng et al. (2002) on the similarity matrix \({\bf S}\) to group similar examples. A fixed number of examples from each cluster are added to the training dataset after annotation by experts.

We train the Siamese encoder to predict the similarity between sentences from the SICK (Sentences Involving Compositional Knowledge) dataset Marelli et al. (2014) using mean squared error. This dataset contains pairs of sentences with manually annotated similarity scores. The sentences are encoded using the encoder in \(\mathcal{M}\) and then passed on to the Siamese encoder for computing similarities. The encoder in \(\mathcal{M}\) is kept fixed while training the Siamese encoder. The trained Siamese encoder is then used for computing similarity between sentences selected by an AL strategy for the given NLP task as described above. As \(\mathcal{M}\) is trained over time, the distribution of its encoder output changes, and hence we periodically retrain the Siamese network to sustain its model-awareness.

The number of clusters and the number of examples drawn from each cluster are user-specified hyper-parameters. The similarity computation can be done efficiently by computing the output of the Siamese encoder for all \(N\) examples before evaluating equation 1, instead of running the Siamese encoder \(O(N^{2})\) times. The clustering algorithm runs in \(O(N^{3})\) time. For an AL strategy to be useful, it should select a small number of examples to benefit from interactive and intelligent labeling. We expect \(N\) to be small for most practical problems, in which case the computational complexity added by our approach would only be a small fraction of the overall computational complexity of training the model with active learning (see Figure 1).

### Integrated Clustering Model

While the approach described in Section 3.2 works well for small to moderate values of \(N\), it suffers from a computational bottleneck when \(N\) is large. We integrate the clustering step into the similarity computation step to remedy this (see Figure 1) and call the resultant approach as Integrated Clustering Model (_Int Model_). Here, the output of model \(\mathcal{M}\)'s encoder is fed to a clustering neural network \(\mathcal{C}\) that has \(K\) output units with the softmax activation function. These units correspond to the \(K\) clusters, and each example is directly assigned to one of the clusters based on the softmax output.

To train the network \(\mathcal{C}\), we choose a pair of similar examples (say \(a\) and \(b\)) and randomly select a negative example (say \(c\)). We experimented with both SICK and Quora Pairs dataset3. All examples are encoded via the encoder of model \(\mathcal{M}\) and then passed to network \(\mathcal{C}\). The unit with the highest probability value for \(a\) is treated as the ground-truth class for \(b\). Minimizing the objective given below maximizes the probability of \(b\) belonging to its ground truth class while minimizing the probability of \(c\) belonging to the same class:

Footnote 3: [https://github.com/jmlr/datasets/QuoraPairs](https://github.com/jmlr/datasets/QuoraPairs)

\[\begin{split}\mathcal{L}(a,b,c)=&-\lambda_{1}\log p _{i_{a}}^{b}-\lambda_{2}\log(1-p_{i_{a}}^{c})\\ &+\lambda_{3}\sum_{k=1}^{K}p_{k}^{b}\log p_{k}^{b}.\end{split} \tag{2}\]

Here \(\lambda_{1}\), \(\lambda_{2}\), and \(\lambda_{3}\) are user-specified hyperparameters, \(p_{j}^{x}\) is the softmax output of the \(j^{th}\) unit 

for example \(x\), \(j=1,2,\ldots,K\), \(x=a,b,c\), and \(i_{a}=\arg\max_{j\in\{1,2,\dots K\}}p^{a}_{j}\). The third term encourages the utilization of all the K units across examples in the dataset. As before, a trained network \(\mathcal{C}\) is used for clustering examples chosen by an AL strategy, and we select a fixed number of examples from each cluster for manual annotation.

It is important to note that: **(i)** These methods are not AL strategies. Rather, they can be used in conjunction with any existing AL strategy. Moreover, given a suitable Siamese encoder or clustering network \(\mathcal{C}\), they apply to any model \(\mathcal{M}\). **(ii)** Our methods compute model-aware similarity since the input to the Siamese or the clustering network is encoded using the model \(\mathcal{M}\). The proposed networks also adapt to the underlying model as the training progresses. Algorithm 1 describes our general approach called Active2 Learning.

Footnote 2: The AL strategies for NMT are ranking-based techniques, so we select the top 50% of the candidates after sorting them in ascending (6,7) or descending (9) order.

## 4 Experiments

We establish the effectiveness of our approaches by demonstrating that they: **(i)** work well across a variety of NLP tasks and models, **(ii)** are compatible with several popular AL strategies, and further reduce the data requirements of existing AL strategies, while achieving the same performance. In particular, we experiment3 with two broad categories of NLP tasks: (a) Sequence Tagging (b) Neural Machine Translation. Table 1 lists these tasks and information about the corresponding datasets (including the two auxiliary datasets for training the Siamese network (Section 3.2)) used in our experiments. We begin by describing the AL strategies for the two kinds of NLP tasks.

Footnote 3: Codes for the experiments are available at the following github link: [https://github.com/parag1604/A2L](https://github.com/parag1604/A2L).

### Active Learning Strategies for Sequence Tagging

Margin-based strategy:Let \(s(\mathbf{y})=\mathrm{P}_{\boldsymbol{\theta}}(\mathbf{Y}=\mathbf{y}|\mathbf{ X}=\mathbf{x})\) be the score assigned by a model \(\mathcal{M}\) with parameters \(\boldsymbol{\theta}\) to output \(\mathbf{y}\) for a given example \(\mathbf{x}\). Margin is defined as the difference in scores obtained by the best scoring output \(\mathbf{y}\) and the second best scoring output \(\mathbf{y}^{\prime}\), i.e.:

\[\mathrm{M}_{\rm margin}=\max_{\mathbf{y}}s(\mathbf{y})-\max_{\mathbf{y}^{ \prime}\neq\mathbf{y}_{\rm max}}s(\mathbf{y}^{{}^{\prime}}), \tag{3}\]

where, \(\mathbf{y}_{\rm max}=\operatorname*{arg\,max}_{\mathbf{y}}s(\mathbf{y})\). The strategy selects examples for which \(\mathrm{M}_{\rmmargin}\leq\tau_{1}\), where \(\tau_{1}\) is a hyper-parameter. We use Viterbi's algorithm (Ryan and Nudd, 1993) to compute the scores \(s(\mathbf{y})\).

Entropy-based strategy:All the NLP tasks that we consider require the model \(\mathcal{M}\) to produce an output for each token in the sentence. Let \(\mathbf{x}\) be an input sentence that contains \(n(\mathbf{x})\) tokens and define \(\bar{s}_{j}=\max_{\varphi\in\mathcal{O}}\mathrm{P}_{\boldsymbol{\theta}}(y_{ j}=o|\mathbf{X}=\mathbf{x})\) to be the probability of the most likely output for the \(j^{th}\) token in \(\mathbf{x}\). Here \(\mathcal{O}\) is set of all possible outputs and \(y_{j}\) is the output corresponding to the \(j^{th}\) token in \(\mathbf{x}\). We define the _normalized entropy score_ as:

\[\mathrm{M}_{\rm entropy}=-\frac{1}{n(\mathbf{x})}\sum_{j=1}^{n(\mathbf{x})} \bar{s}_{j}(\mathbf{y})\log\bar{s}_{j}(\mathbf{y}). \tag{4}\]

A length normalization \(n(\mathbf{x})\) is added to avoid bias due to the example length as it may be undesirable to annotate longer length examples (Claveau and Kijak, 2017). The strategy selects examples with \(\mathrm{M}_{\rm entropy}\geq\tau_{2}\), where \(\tau_{2}\) is a hyper-parameter.

Bayesian Active Learning by Disagreement (BALD):Due to stochasticity, models that use dropout (Srivastava et al., 2014) produce a different output each time they are executed. BALD (Houlsby et al., 2011) exploits this variability in the predicted output to compute model uncertainty. Let \(\mathbf{y}^{(t)}\) denote the best scoring output for \(\mathbf{x}\) in the \(t^{th}\) forward pass, and let \(N\) be the number of forward passes with a fixed dropout rate, then:

\[\mathrm{M}_{\rm bald}=1-\frac{\mathrm{count}(\mathrm{mode}(\mathbf{y}^{(1)}, \ldots,\mathbf{y}^{(N)}))}{N}. \tag{5}\]

Here the \(\mathrm{mode}(.)\) operation finds the output which is repeated most often among \(\mathbf{y}^{(1)},\ldots,\mathbf{y}^{(N)}\), and the \(\mathrm{count}(.)\) operation counts the number of times this output was encountered. This strategy selects examples with \(\mathrm{M}_{\rm bald}\geq\tau_{3}\) (hyper-parameter).

### Active Learning Strategies for Neural Machine Translation2
Footnote 2: The AL strategies for NMT are ranking-based techniques, so we select the top 50% of the candidates after sorting them in ascending (6,7) or descending (9) order.

Least Confidence (LC)This strategy estimates the uncertainty of a trained model on a source sentence \(\mathbf{x}\) by calculating the conditional probability of the prediction \(\hat{\mathbf{y}}\) conditioned on the source sentence(Lewis and Catlett, 1994).

\[\mathrm{M}_{\rm LC}=\frac{1}{\mathrm{n}(\hat{y})}\log\mathbf{P}(\hat{\mathbf{ y}}|\mathbf{x}) \tag{6}\]

A length normalization of \(\mathrm{n}(\hat{y})\) (length of the predicted translation \(\hat{\mathbf{y}}\)) is added.

 

Coverage Sampling (CS)A translation model is said to _cover_ the source sentence if it translates all of its tokens. Coverage is estimated by mapping a particular source token to its appropriate target token, without which the model may suffer from under-translation or over-translation issues Tu et al. (2016). Peris and Casacuberta (2018) proposed to use translation coverage as a measure of uncertainty by:

\[{\rm M}_{\rm CS}=\frac{\sum^{n(\mathbf{x})}_{\mathrm{j}=1}\log(\min(\sum^{n( \mathcal{Y})}_{\mathrm{i}=1}\alpha_{\mathrm{i}\mathrm{j}},1))}{n(\mathbf{x})} \tag{7}\]

Here \(\alpha_{\mathrm{i}\mathrm{j}}\) denotes the attention probability calculated by the model for the \(j^{th}\) source word in predicting the \(i^{th}\) target word. It can be noted that the coverage score will be \(0\) for samples for which the model almost fully covers the source sentences.

Attention Distraction Sampling (ADS)Peris and Casacuberta (2018) claimed that in translating an uncertain sample, the model's attention mechanism would be _distracted_ (dispersed throughout the sentence). Such samples yield attention probability distribution with light tails (e.g., uniform distribution), which can be obtained by taking the Kurtosis of the attention weights for each target token \(\mathbf{y}_{\mathrm{i}}\).

\[{\rm Kurt}(\mathbf{y}_{\mathrm{i}})=\frac{\frac{1}{n(\mathbf{x})}\sum^{n( \mathbf{x})}_{\mathrm{j}=1}(\alpha_{\mathrm{i}\mathrm{j}}-\frac{1}{n( \mathbf{x})})^{4}}{(\frac{1}{n(\mathbf{x})}(\sum^{n(\mathbf{x})}_{\mathrm{ j}=1}\alpha_{\mathrm{i}\mathrm{j}}-\frac{1}{n(\mathbf{x})}))^{2}} \tag{8}\]

where \(\frac{1}{n(\mathbf{x})}\) is the mean of the distribution of the attention weights (for a target word) over the source words. The kurtosis value will be lower for distributions with light tails, so the average of the negative kurtosis values for all words in the target sentence is used as the distraction score.

\[{\rm M}_{\rm ADS}=\frac{\sum^{n(\mathbf{y})}_{\mathrm{i}=1}-{\rm Kurt}( \mathbf{y}_{\mathrm{i}})}{n(\mathbf{y})} \tag{9}\]

### Details about Training

For sequence tagging, we use two kinds of architectures: CNN-BiLSTM-CRF model (CNN for character-level encoding and BiLSTM for word-level encoding) and a BiLSTM-BiLSTM-CRF model (BiLSTM for both character-level and word-level encoding) (Lample et al. (2016); Siddhant

Figure 1: Comparison of time taken for one data selection step in NMT task by the Model Aware (MA) Siamese and Integrated Clustering (Int) Model across different ALS. It can be observed that A\({}^{2}\)L adds a negligible overhead (\(\approx\frac{1}{12}\) of the time taken for ALS) to the overall process.

 

and Lipton (2018)). For the translation task, we use LSTM based encoder-decoder architecture with Bahdanau attention Bahdanau et al. (2014). These models were chosen for their performance and ease of implementation.

The Siamese network used for model-aware similarity computation (Section 3.2) consists of two bidirectional LSTM (BiLSTM) encoders. We pass each sentence in the pair from the SICK dataset to model \({\cal M}\) and feed the resulting encodings to the Siamese BiLSTM encoder. The output is a concatenation of terminal hidden states of the forward and backward LSTMs, which is used to compute the similarity score using (1). As noted before, we keep model \({\cal M}\) fixed while training the Siamese encoders and use the trained Siamese encoders for computing similarity between examples chosen by an AL strategy. We maintain the model-awareness by retraining the Siamese after every \(10\) iterations.

The architecture of the clustering model \({\cal C}\) (Section 3.3) is similar to that of the Siamese encoder. Additionally, it has a linear layer with a softmax activation function that maps the concatenation of terminal hidden states of the forward and backward LSTMs to \(K\) units, where \(K\) is the number of clusters. To assign an input example to a cluster, we first pass it through the encoder in \({\cal M}\) and feed the resulting encodings to the clustering model \({\cal C}\). The example is assigned to the cluster with the highest softmax output. This network is also retrained after every \(10\) iterations to retain model-awareness.

The initial data splits used for training the model \({\cal M}\) were set at \(2\%\) of randomly sampled data for Sequence Tagging (\(20\%\) for NMT). These are in accordance with the splitting techniques used in the existing literature on AL Siddhant and Lipton (2018); Liu et al. (2018). The model is then used to provide input to train the Siamese/Clustering network using the SICK/Quora Pairs. At each iteration, we gradually add another \(2\%\) of data for sequence tagging (\(5\%\) for NMT) by retrieving low confidence examples using an AL strategy, followed by clustering to extract the most representative examples. We average the results over five independent runs with randomly chosen initial splits. [Hyperparameters details in Appendix A].

### Baselines

We claim that A\({}^{2}\)L mitigates the redundancies in the existing AL strategies by working in conjunction with them. We validate our claims by comparing our approaches with three baselines that highlight the importance of various components.

Cosine:Clustering is done based on cosine similarity between last output encodings (corresponding to sentence length) from encoder in \({\cal M}\). Although this similarity computation is model-aware, it is simplistic and shows the benefit of using a more expressive similarity measure.

None:In this baseline, we use the AL strategy without applying Active2 learning to remove redundant examples. This validates our claim about redundancy in examples chosen by AL strategies.

Footnote 2: We process the dataset to use only those sentences which are present in at least 5 other pairs. We retrieve 16000 sets, each with a source sentence and 5 other samples (comprising both positive and negative labels). An additional 1000 sets were generated for evaluation.

Random:No active learning is used, and random examples are selected at each time.

### Ablation Studies

We perform ablation studies to demonstrate the utility of model-awareness using these baselines:

Infersent:Clustering is done based on cosine similarity between sentence embeddings Chen et al. (2015) obtained from a pre-trained InferSent model Conneau et al. (2017). This similarity computation is not model-aware and shows the utility of model-aware similarity computation.

Iso Siamese:To show that the Siamese network alone is not sufficient and model-awareness is needed, in this baseline, we train the Siamese network by directly using GloVe embeddings of the words as input rather than using output from the

\begin{table}
\begin{tabular}{c c c c} \hline \hline  & & **\% of train** & **\% less data** \\  & & **data used to** & **required to** \\  & & **reach**\(99\%\) & **reach**\(99\%\) \\  & & **of full-data** & **of full-data** \\  & & **F-Score** & **F-score** \\ \hline POS & CoNLL & \multirow{2}{*}{25\%} & \multirow{2}{*}{16\%} \\  & 2003 & & \\ \cline{1-1} NER & CoNLL & \multirow{2}{*}{37\%} & \multirow{2}{*}{3\%} \\  & 2003 & & \\ \hline SEMTR & SEMCOR & \multirow{2}{*}{35\%} & \multirow{2}{*}{25\%} \\ \cline{1-1} CHUNK & CoNLL & \multirow{2}{*}{23\%} & \multirow{2}{*}{11\%} \\  & 2000 & & \\ \end{tabular}
\end{table}
Table 2: Fraction of data used for reaching full dataset performance and the corresponding absolute percentage reduction in the data required over the None baseline that uses active learning strategy without the A\({}^{2}\)L step for the best AL strategy (BALD in all cases). Refer Fig 8 in Appendix for CHUNK plots.

 

Figure 2: [Best viewed in color] Comparison of our approach (A\({}^{\textbf{2}}\)L) with baseline approaches on different tasks using different active learning strategies. \(1^{st}\) row: POS (error bound at convergence: \(\pm 0.05\)), \(2^{nd}\) row: NER (\(\pm 0.08\)), \(3^{rd}\) row: SEMTR (\(\pm 0.09\)), \(4^{th}\) row: NMT (\(\pm 0.04\)). In the first three rows, from left to right, the three columns represent BALD, Entropy, and Margin AL strategies. \(4^{th}\) row represents AL strategies for NMT, from left to right (LC: Least Confidence, CS: Coverage Sampling, ADS: Attention Distraction Sampling) : Legend Description {100% data: full data performance, A\({}^{\textbf{2}}\)L (MA Siamese) : Model Aware Siamese, A\({}^{\textbf{2}}\)L (Int Model) : Integrated Clustering Model, Cosine : Cosine similarity, None : Active learning strategy without clustering step, Random : Random split (no active learning applied) . See Section 4.4 for more details about the baselines. All the results were obtained by averaging over 5 random splits. These plots have been magnified to highlight the regions of interest. For original plots, refer to Fig 8 in the Appendix.

 

model \({\cal M}\)'s encoder. This similarity, which is not model-aware, is then used for clustering.

## 5 Results

Figure 2 compares the performance of our methods with baselines. It shows the test-set metric on the \(y\)-axis against the percentage of training data used on the \(x\)-axis for all tasks. See Figures 7 and 8 in the Appendix for additional results.

1. As shown in Figure 2, our approach consistently outperforms all baselines on all tasks. Note that one should observe how fast the performance increases with the addition of training data (and not just the final performance) as we are trying to evaluate the effect of adding new examples. Our ablation studies in Figure 3 show the utility of using model-aware similarity.
2. In sequence tagging, we match the performance obtained by training on the full dataset using only a smaller fraction of the data (\(\mathbf{3-25\%}\) less data as compared to state-of-art AL strategies) (Table 2). On a large dataset in NMT task (Europarl), A**^2L** takes \(\approx 4300\) sentences fewer than the Least Confidence AL strategy to reach a Bleu score of \(12\).
3. While comparing different AL strategies is not our motive, Figure 2 also demonstrates that one can achieve performance comparable to a complex AL strategy like BALD, using simple AL strategies like margin and entropy, by using the proposed A**^2L** framework.
4. Additionally, from Figure 1, it can be observed that for one step of data selection: (i) The proposed MA Siamese model adds minimal overhead to the overall AL pipeline since it takes less than 5 additional seconds (\(\approx\frac{1}{12}\) of the time taken for ALS); (ii) By approximating the clustering step, Integrated Clustering (Int) Model further reduces the overhead down to 2 seconds. However, owing to this approximation, MA Siamese is observed to perform slightly better than the Int Model (Fig 3). A comparison of training time for various stages of the A**^2L** pipeline is provided in Figure 4.

We wish to state that our approach should be evaluated not in terms of the gain in the F1 score but

Figure 4: Comparison between various components of our approach in terms of the time required by them per training epoch for NMT. LSTM (Base) encoder-decoder translation model (\(\approx 10\) hours), Model Aware (MA) Siamese (\(\approx 3\) minutes) and Integrated Clustering (Int) Model (\(\approx 10\) seconds). It can be observed that A**^2L** adds a negligible overhead to the overall training time (\(\approx 0.45\%\) of the time taken by the base model).

Figure 3: [Best viewed in color] Ablations studies on POS task using different active learning strategies. From left to right, the three columns represent BALD, Entropy and Margin based AL strategies. Legend Description {**100%** data : full data performance, A**^2L** (MA Siamese) : Model Aware Siamese, A**^2L** (Int Model) : Integrated Clustering Model, Iso Siamese : Model isolated Siamese, InferSent : Cosine similarity based on InferSent encodings}. See Figure 7 in Appendix for experiments on other tasks. All the results were obtained by averaging over 5 splits.

 

in terms of the reduction in data required to achieve the same (3-25 % on multiple datasets). More importantly, this improvement comes at a negligible computation overhead cost. The reported improvements are not relative with respect to any baseline but represent an absolute value and are very significant in the context of similar performance improvements reported in the literature. In Figure 5, we provide a qualitative case study that demonstrates the problem of redundancy.

## 6 Conclusion

This paper shows that one can further reduce the data requirements of Active Learning strategies by proposing a new method, A\({}^{2}\)L, which uses a model-aware-similarity computation. We empirically demonstrated that our proposed approaches consistently perform well across many tasks and AL strategies. We compared the performance of our approach with strong baselines to ensure that the role of each component is properly understood.

## Acknowledgement

This work was funded by British Telecom India Research Center project on Advanced Chatbot.

## References

* [1]P. Awasthi, M. Florina Balcan, and P. M. Long (2017) The power of localization for efficiently learning linear separators with noise. J. ACM63 (6), pp. 50:1-50:27. Cited by: SS4.
* D. Bahdanau, K. Cho, and Y. Bengio (2014) Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR 2015 as oral presentation, External Links: Link Cited by: SS2.
* M. Bloodgood and C. Callison-Burch (2010) Bucking the trend: large-scale cost-focused active learning for statistical machine translation. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, ACL '10, New Orleans, Louisiana, USA, pp. 854-864. Note: USACited by: SS2.

Figure 5: [Best viewed in color] Qualitative case study to convey the notion of redundancy and the model aware similarity. We compare the examples grouped in the same cluster using (i) MA similarity scores [left] and (ii) Cosine similarity scores on the InferSent embedding (Infersent baseline described in Section 4.4) [Right]. As expected, when cosine similarity is used, sentences that have roughly similar content have been assigned to the same cluster. However, when model aware similarity is used, in addition to having similar content, the sentences also have a similar tagging structure. It is sensible to eliminate sentences having similar tagging structures, as they are redundant. [Dataset: CoNLL 2003 dataset, AL strategy: BALD, Data usage: \(10\%\) of the total data]

\begin{table}
\begin{tabular}{c c c} \hline No. of & Spectral & Integrated \\ examples & (examples \\ selected by & processed per & processed per \\ AL Strategy & second) & second) \\ \hline
5000 & 491.64 & 2702.70 \\
10000 & 248.08 & 2557.54 \\
15000 & 163.10 & 2508.36 \\
20000 & 122.89 & 2493.76 \\ \hline \end{tabular}
\end{table}
Table 3: Number of samples processed per second by the Spectral Clustering (MA Siamese) compared to the Integrated Clustering (Int Model) methods.

 

Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. 2015. Weight uncertainty in neural networks. In _Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37_. ICML'15, pages 1613-1622. JMLR.org.
* Bromley et al. (1994) Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Sackinger, and Roopak Shah. 1994. Signature verification using a "siamese" time delay neural network. In J. D. Cowan, G. Tesauro, and J. Alspector, editors, _Advances in Neural Information Processing Systems_, pages 737-744. Morgan-Kaufmann.
* Burchfield (1985) Robert Burchfield. 1985. Frequency analysis of english usage: Lexicon and grammar. by w. nelson francis and henry kucera with the assistance of Andrew w. mackie. boston: Houghton mifflin. 1982. x + 561. _Journal of English Linguistics_, 18(1):64-70.
* Chen et al. (2015) Yukun Chen, Thomas A. Lasko, Qiaozhu Mei, Joshua C. Denny, and Hua Xu. 2015. A study of active learning methods for named entity recognition in clinical text. _J. of Biomedical Informatics_, 58(C):11-18.
* Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. In _Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation_, pages 103-111, Doha, Qatar. Association for Computational Linguistics.
* 18th International Conference on Computational Linguistics and Intelligent Text Processing_, pages 1-14.
* Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 670-680.
* Dasgupta et al. (2009) Sanjoy Dasgupta, Adam Tauman Kalai, and Claire Monteleoni. 2009. Analysis of perceptron-based active learning. _J. Mach. Learn. Res._, 10:281-299.
* Figueroa et al. (2012) Rosa Figueroa, Qing Zeng-Treitler, Long Ngo, Sergey Goryachev, and Eduardo Wiechmann. 2012. Active learning for clinical text classification: Is it better than random sampling? _Journal of the American Medical Informatics Association : JAMIA_, 19:809-16.
* Freund et al. (1997) Yoav Freund, H. Sebastian Seung, Eli Shamir, and Naftali Tishby. 1997. Selective sampling using the query by committee algorithm. _Mach. Learn._, 28(2-3):133-168.
* Volume 48_, ICML'16, pages 1050-1059. JMLR.org.
* Gal and Ghahramani (2016b) Yarin Gal and Zoubin Ghahramani. 2016b. A theoretically grounded application of dropout in recurrent neural networks. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, _Advances in Neural Information Processing Systems 29_, pages 1019-1027. Curran Associates, Inc.
* Haffari et al. (2009) Gholamreza Haffari, Maxim Roy, and Anoop Sarkar. 2009. Active learning for statistical phrase-based machine translation. In _Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics_, pages 415-423.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural Comput._, 9(8):1735-1780.
* Houlsby et al. (2011) Neil Houlsby, Ferenc Huszar, Zoubin Ghahramani, and Mate Lengyel. 2011. Bayesian active learning for classification and preference learning. _CoRR_, abs/1112.5745.
* Hutchins (2004) W. John Hutchins. 2004. The georgetown-ibm experiment demonstrated in january 1954. In _Machine Translation: From Real Users to Research_, pages 102-114, Berlin, Heidelberg. Springer Berlin Heidelberg.
* Koehn (2005) Philipp Koehn. 2005. Europarl: A Parallel Corpus for Statistical Machine Translation. In _Conference Proceedings: the tenth Machine Translation Summit_, pages 79-86, Phucket, Thailand. AAMT.
* Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 260-270.
* Landes et al. (1998) Shari Landes, Claudia Leacock, and Christiane Fellbaum. 1998. Building semantic concordances. _WordNet: An Electronic Lexical Database_, pages 199-216.
* Lewis and Catlett (1994) David D. Lewis and Jason Catlett. 1994. Heterogeneous uncertainty sampling for supervised learning. In _Proceedings of the Eleventh International Conference on Machine Learning_, pages 148-156, San Francisco (CA). Morgan Kaufmann.
* Liu et al. (2018) Ming Liu, Wray Buntine, and Gholamreza Haffari. 2018. Learning to actively learn neural machine translation. In _Proceedings of the 22nd Conference on Computational Natural Language Learning_, pages 334-344, Brussels, Belgium. Association for Computational Linguistics.

 

Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. _Comput. Linguist._, 19(2):313-330.
* Marelli et al. (2014) Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A sick cure for the evaluation of compositional distributional semantic models. In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014)_.
* Alonso and Plank (2017) Hector Martinez Alonso and Barbara Plank. 2017. When is multitask learning effective? semantic sequence prediction under varying data conditions. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_, pages 44-53. Association for Computational Linguistics.
* McCallum and Nigam (1998) Andrew McCallum and Kamal Nigam. 1998. Employing em and pool-based active learning for text classification. In _Proceedings of the Fifteenth International Conference on Machine Learning_, ICML '98, pages 350-358.
* Mueller and Thyagarajan (2016) Jonas Mueller and Aditya Thyagarajan. 2016. Siamese recurrent architectures for learning sentence similarity. In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence_, AAAI'16, pages 2786-2792.
* Nadeau and Sekine (2007) David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. _Linguistic Investigationes_, 30(1):3-26.
* Nepveu et al. (2004) Laurent Nepveu, Guy Lapalme, Philippe Langlais, and George Foster. 2004. Adaptive language and translation models for interactive machine translation. In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_, pages 190-197, Barcelona, Spain. Association for Computational Linguistics.
* Ng et al. (2002) Andrew Y. Ng, Michael I. Jordan, and Yair Weiss. 2002. On spectral clustering: Analysis and an algorithm. In T. G. Dietterich, S. Becker, and Z. Ghahramani, editors, _Advances in Neural Information Processing Systems 14_, pages 849-856. Curran Associates, Inc.
* Ortiz-Martinez (2016) Daniel Ortiz-Martinez. 2016. Online learning for statistical machine translation. _Computational Linguistics_, 42(1):121-161.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In _Empirical Methods in Natural Language Processing (EMNLP)_, pages 1532-1543.
* Peris and Casacuberta (2018) Alvaro Peris and Francisco Casacuberta. 2018. Active learning for interactive neural machine translation of data streams. In _Proceedings of the 22nd Conference on Computational Natural Language Learning_, pages 151-160, Brussels, Belgium. Association for Computational Linguistics.
* Ryan and Nudd (1993) Matthew S. Ryan and Graham R. Nudd. 1993. The viterbi algorithm. Technical report, University of Warwick.
* Tsong Kim Sang and De Meulder (2003) Erik F. Tsong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. _Preceeding of the Computational Natural Language Learning (CoNLL)_.
* Sha and Saul (2007) Fei Sha and Lawrence K. Saul. 2007. Large margin hidden markov models for automatic speech recognition. In B. Scholkopf, J. C. Platt, and T. Hoffman, editors, _Advances in Neural Information Processing Systems 19_, pages 1249-1256. Curran Associates, Inc.
* Shen et al. (2018) Yanyao Shen, Hyokun Yun, Zachary C. Lipton, Yakov Kronrod, and Animashree Anandkumar. 2018. Deep active learning for named entity recognition. In _6th International Conference on Learning Representations_.
* Siddhant and Lipton (2018) Aditya Siddhant and Zachary C Lipton. 2018. Deep bayesian active learning for natural language processing: Results of a large-scale empirical study. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2904-2909.
* Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. _J. Mach. Learn. Res._, 15(1):1929-1958.
* Volume 2_, NIPS'14, page 3104-3112, Cambridge, MA, USA. MIT Press.
* Thompson et al. (1999) Cynthia A. Thompson, Mary Elaine Califf, and Raymond J. Mooney. 1999. Active learning for natural language parsing and information extraction. In _Proceedings of the Sixteenth International Conference on Machine Learning_, ICML '99, page 406-414, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
* Sung and Buchholz (2000) Erik F. Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the CoNLL-2000 shared task chunking. In _Fourth Conference on Computational Natural Language Learning and the Second Learning Language in Logic Workshop_.
* Tu et al. (2016)Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 76-85, Berlin, Germany. Association for Computational Linguistics.

 



**Active\({}^{2}\) Learning: Actively reducing redundancies in Active Learning methods for Sequence Tagging and Machine Translation: Appendix**

## Appendix A Implementation Details

We use two different sequence tagging architectures: CNN-BiLSTM-CRF model (CNN for character-level encoding and BiLSTM for word-level encoding) and a BiLSTM-BiLSTM-CRF model Lample et al. (2016) (BiLSTM for both character-level and word-level encoding). The CNN-BiLSTM-CRF architecture is a light-weight variant of the model proposed in Siddhant and Lipton (2018), having one layer in CNN encoder with two filters of sizes 2 and 3, followed by a max pool, as opposed to three layers in the original setup. This modification was found to improve the results. We use glove embeddings Pennington et al. (2014) for all datasets. We apply normal dropout in the character encoder instead of the use of recurrent dropout Gal and Ghahramani (2016b) in the word encoder of the model presented in Siddhant and Lipton (2018) owing to an improvement in performance. For numerical stability, we use log probabilities and, thus, the value for margin-based AL strategy's threshold is outside the interval \([0,1]\). We use the spectral clustering Ng et al. (2002) algorithm to cluster the sentences chosen by the AL strategy. We chose two representative examples from each cluster.

Hyperparameters are given for Sequence Tagging (CoNLL 2003 NER) and NMT.

\begin{tabular}{l l} \hline
**Active Learning strategy** & \\ \hline threshold (Margin) & 15 \\ threshold (Entropy) & 40 \\ threshold (BALD) & 0.2 \\ dropout (BALD) & 0.5 \\ number of forward passes (BALD) & 51 \\ \hline
**Sequence tagging model** & \\ \hline CNN filter sizes & [2,3] \\ training batch size & 12 \\ number of train epochs & 16 \\ dimension of character embedding & 100 \\ learning rate (Adam) & 0.005 \\ learning rate decay & 0.9 \\ \hline
**Siamese encoder** & \\ \hline training batch size & 48 \\ number of train epochs & 41 \\ train/dev split & 0.8 \\ learning rate (Adam) & 1e-5 \\ period (of retrain) & 10 \\ \hline
**Clustering** & \\ \hline Number of clusters & 20 \\ \hline
**Training** & \\ \hline Batch size & 12 \\ \hline \end{tabular}

Figure 6: Modeling similarity using the Siamese encoder (enclosed by dotted lines). A pair of sentences from AUX dataset is fed to the pretrained ST/NMT model. The output of the encoder is then passed to the Siamese encoder. Last hidden state of the Siamese encoder, corresponding to the sequence length of the sentence, is used for assigning a similarity score.





## 6 Conclusion

Figure 7: [Best viewed in color] Ablations studies on different tasks using different active learning strategies. \(1^{st}\) row: NER, \(2^{nd}\) row: SEMTR, \(3^{rd}\) row: CHUNK, \(4^{th}\) row: NMT. In first three rows, from left to right, the three columns represent BALD, Entropy and Margin AL strategies. \(4^{th}\) row represents AL strategies for NMT, from left to right (LC: Least Confidence, CS: Coverage Sampling, ADS: Attention Distraction Sampling). Legend Description {100% data : full data performance, A**2L** (MA Siamese) : Model Aware Siamese, A**2L** (Int Model) : Integrated Clustering Model, Iso Siamese : Model isolated Siamese, InferSett : Cosine similarity based on InferSent encodings}. See Section 4.5 for more details. All results were obtained by averaging over 5 random splits.



Figure 8: [Best viewed in color] Comparison of our approach (A\({}^{2}\)L) with baseline approaches on different tasks using different active learning strategies. 1\({}^{st}\) row: POS, 2\({}^{nd}\) row: NER, 3\({}^{rd}\) row: SEMTR, 4\({}^{th}\) row: CHUNK. In each row, from left to right, the three columns represent BALD, Entropy and Margin based AL strategies. Legend Description {100% data : full data performance, A\({}^{2}\)L (MA Siamese) : Model Aware Siamese, A\({}^{2}\)L (Int Model) : Integrated Clustering Model, Cosine : Cosine similarity, None : Active learning strategy without clustering step, Random : Random split (no active learning applied)}. See Section 4.4 for more details. All the results were obtained by averaging over 5 random splits.