Limitations of Active Learning With Deep Transformer Language Models
Authors: Mike D'Arcy
Doug Downey
Venue: ICLR 2022 Submitted (2022)
Keywords: Active Learning
Machine Learning
Natural Language Processing

Abstract: Active Learning (AL) has the potential to reduce labeling cost when training natural language processing models, but its effectiveness with the large pretrained transformer language models that power today's NLP is uncertain.  We present experiments showing that when applied to modern pretrained models, active learning offers inconsistent and often poor performance.  As in prior work, we find that AL sometimes selects harmful "unlearnable" collective outliers, but we discover that some failures have a different explanation: the examples AL selects are informative but also increase training instability, reducing average performance.  Our findings suggest that for some datasets this instability can be mitigated by training multiple models and selecting the best on a validation set, which we show impacts relative AL performance comparably to the outlier-pruning technique from prior work while also increasing absolute performance. Our experiments span three pretrained models, ten datasets, and four active learning approaches.

Under review as a conference paper at ICLR 2022
LIMITATIONS OF ACTIVE LEARNING WITH DEEP TRANSFORMER LANGUAGE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Active Learning (AL) has the potential to reduce labeling cost when training nat- ural language processing models, but its effectiveness with the large pretrained transformer language models that power today's NLP is uncertain. We present ex- periments showing that when applied to modern pretrained models, active learning offers inconsistent and often poor performance. As in prior work, we find that AL sometimes selects harmful "unlearnable" collective outliers, but we discover that some failures have different explanation: the examples AL selects informative but also increase training instability, reducing average performance. Our findings suggest that for some datasets this instability can be mitigated by training multiple models and selecting the best on a validation set, which we show impacts relative AL performance comparably to the outlier-pruning technique from prior work while also increasing absolute performance. Our experiments span three pretrained models, ten datasets, and four active learning approaches.
INTRODUCTION
Deep learning models typically need large datasets of hand-labeled examples perform well, but acquiring the labels can be expensive, often requiring human annotators to manually provide the label for every example. Active learning (AL) is technique that aims reduce this labeling cost by selecting only the most useful examples to receive labels (Settles, 2009).
AL has been shown to reduce labeling cost on variety of NLP tasks in previous work (Shen et al., 2018; Siddhant & Lipton, 2018). However, these successes predate the rise of large pretrained Transformer language models (LMs) as foundation for NLP. Whether AL's advantages extend to this setting is far less studied. Previous work has shown that AL consistently improves text classifiers that use pretrained Transformers (Lu & MacNamee, 2020; Ein-Dor al., 2020), but those studies use narrow range of model sizes and do not consider the most challenging semantic tasks such as question answering (QA) where pretrained LMs have made great strides in recent years. By contrast, Karancheti al. (2021) recently showed that AL is ineffective for large pretrained transformers in visual QA, and concluded that harmful collective outliers negatively impact AL and that heuristically pruning them improves the benefit of AL. However, whether these findings hold outside of visual QA unknown.
In this paper, we present expanded study of AL applied to fine-tuning large pretrained LMs. We investigate four AL approaches across ten datasets spanning text classification and recent multiple- choice question answering tasks. We find that AL fails improve performance consistently, and often lowers performance significantly below that of random-selection baseline. Further, our analysis suggests that in some cases, the failures have different explanation than that offered by prior work. Specifically, AL sometimes select examples that are informative, but add instability to the training process, hurting average performance. As evidence, we show that a simple approach to reduce -triiiigtt the model multiple times and selecting the best- best-performing on a held-out set, without any special treatment for collective outliers-impren iimrovess the performance of AL by similar amount average the pruning in Karamcheti et al. (2021). Further, we find that the effectiveness of the pruning technique from Karamcheti et al. (2021) is often dataset-specific. improving the relative performance of AL over random sampling some datasets and worsening it others. In most cases, pruning also reduces the absolute performance across all methods, suggesting that it prunes helpful examples in addition to outliers.

---

like our Under review as a conference paper at ICLR 2022
To summarize, contributions
1. We present experiments across four AL methods, ten data sets, and three models show- ing that AL has inconsistent performance with recent pretrained transformer LMs, often underperforming random selection especially as larger models are used.
2. We propose novel explanation for some of AL's limitations: that it sometimes selects examples that are helpful but cause instability in training.
3. We conduct several ablation studies to rule out other factors and find that acquisition batch size and adversarial filtering do not meaningfully affect the performance of AL, while the amount of model pretraining has inconsistent effects between AL methods.
We hope our findings will provide guidance to other researchers considering applying AL their data, and spur further investigation into more effective AL methods for pretrained LMs along with optimization techniques that are more robust to the challenging examples AL selects.
METHODS
2.1
TASK DEFINITION
Our active learning (AL) scenario is formulated as follows. We have large pool of unlabeled data U and wish select subset L of size to label that maximizes model performance on held-out set. Given a model M and an AL scoring function Six, M) (where x is an unlabeled input), we iteratively select a batch of | examples from U according S, add these to L, and retrain the model before selecting again. When re- re-training, we re-initialize all parameters to the original pretrained model state, following previous work (Fin-Dor et al., 2020; Hu al., 2019). The first batch is selected randomly.
2.2 AL METHODS AND MODELS
experiment with the following AL scoring functions:
Random (the baseline);
Entropy' of the model output distribution;
BALD-MC, which is BALD (Houlsby et al., 2011) using Monte Carlo dropout (Gal & Ghahramani, 2016) to compute uncertainty;
BatchBALD-MCC (Kirsch et al., 2019), the batch variant of BALD;
Coreset (Sener & Savarese, 2018), which selects examples that are distant from examples in L measured by distance between model output embeddings. We use the greedy variant of the algorithm and use the pre-trained model to pre-compute fixed output embeddings for the distance calculation."
While many AL methods exist, we believe our choices form representative set. Entropy is commonly used and was found work well the previous work on AL for Transformers (Lu & MacNamee, 2020). BALD has been found to outperform other methods in many cases, including NLP (Gal et al., 2017: Siddhant Lipton, 2018). Coreset is diversity-based (rather than uncertainty-base other methods) and has outperformed other methods (including BALD) ) for deep models on image classification tasks (Sener Savarese, 2018). We note that BALD and Entropy are originally online algorithms (selecting one example time) but used in batch mode here. This is not theoretically sound but is commonplace and often effective (Kasai et al., 2019; Gal et al., 2017), so we include for comparison. BatchBALD adds diversity adjustment to BALD account for this issue.
We test three pretrained Transformers, which are oftte-of-the-ar models with different pretraining methods and sizes: BERT-base (110M parameters) (Devlin et al., 2019), RoBERTa-large (340M) (Liu et al., 2019), and RoBERTa-base (1 10M)).
A similar method is least-confidence (I.C). We tried L.C. on most of datasets and found it be almost identical to entropy, so we omit here.
In preliminary experiments we tried training the model and updating the embeddings during acquisition, but this did not improve results and was more computationally expensive.
2

---

Under review as a conference paper at ICLR 2022
2.3 TASKS AND DATASETS
For datasets, we focus broadly on two tasks. The first is text classification, where AL has been effective with limited class of Transformer models previous work Lu MacNamee (2020); Ein-Dor et al. (2020). This task allows us to verify our implementation against previous work, and to evaluate whether AL's effectiveness on extends to a broader model class. The second is multiple- choice commonsense reasoning, challenging NLP task where AL has not been evaluated, to our knowledge. The text classification datasets are AGN (Zhang et al., 2015), DBpedia (Zhang et al., 2015), Movie review sentence polarity (MRP) (Pang & Lee, 2005), and Movie review subjectivity (MRS) (Pang & Lee, 2004). The multiple-choice datasets are aNLI (Bhagavatula et al., 2020), CODAH (Chen al., 2019), CommonsenseQAA (CSQA) (Talmor et al., 2019), HellaSWAG (Zellers et al., 3019). PIQA (Bisk et al., 2020), and SWAG (Zellers et al., 2018). In the tables, "AGN-SB and "DBpedia-SB" are AGN and DBpedia using only classes (to create binary classification task) and subsampled to have the same number of examples as reported by Lu & MacNamee (2020).
2.4 EXPERIMENTAL SETTINGS
We test on an eval set after each acquisition, which produces a learning curve of accuracy vs. data size. We report the area under the curve (AUC) as measure of how effective the data selection strategy is.
We set Q 500 and A 25 except where otherwise specified. The deep learning models were all trained by doing 1000 steps with the Adam optimizer and early stopping (hyperparameter details Appendix A), and results are averaged over 10 trials.
3 RESULTS
We start by presenting our primary results showing that active learning offers inconsistent and often poor performance on our tasks. For RoBERTa-base, shown Table I, there are few cases where baseline, but in many cases it gives no improvement (and in
AL outperforms the random-samplin fact, for 10 of the results it does significantly worse). the subsequent sections, we investigate several candidate explanations for the poor performance.
Dataset Method
BALD-MC
BatchBALD-MC
Entropy
Coreset
Random
AGN-c
86.8 (0.3)
87.5 (0.2)
87.3 (0.2)
(0.2)
86.6 (0.2)
AGN-SB-C
97.7 (0.0)
97.4 (0.0)
97.8 (0.0)
97.2 (0.0)
96.7 (0.1)
aNLI
58.5 (0.2)
58.9 (0.2)
57.8 (0.1)
59.8 (0.1)
58.8 (0.2)
CODAH
(0.5)
58.5 (0.3)
(0.3)
57.6 (0.4)
59.0 (0.6)
CSQA
42.8 (0.3)
43,9 (0.4)
(0.3)
44.0 (0.2)
43.8 (0.2)
DBpedia-SBB C
98.9 (0.0)
98.9 (0.0)
98.9 (0.0)
98.8 (0.0)
98.8 (0.1)
HellaSWAG
38.1 (0.2)
38.5 (0.2)
38.1 (0.5)
38.9 (0.2)
38.7 (0.2)
MRP-c
83.4 (0.2)
83.3 (0.2)
83.5 (0.2)
81.8
83.0
MRS-c
94.9 (0.0)
95.3 (0.1)
95.2 (0.0)
94.0 (0.1)
94.0
PIQA
(0.3)
(0.2)
(0.4)
57.1 (0.2)
56.8 (0.3)
SWAG
62.5 (0.2)
62.7 (0.2)
60.5 (0.3)
62.7 (0.2)
63.5 (0.2)
Average
70.64 (0.08)
70.96 (0.07)
70.29 (0.08)
70.75 (0.06)
70.88 (0.08)
Table I: RoBERTa-base AUC results. In all tables, bold indicates better-than--ando---ammii
performance; gray background indicates significance at < 0..5 in a two-tailed t-test. Standard errors are in parentheses. "-c" suffix indicates text classification datasets.
3.1 COLLECTIVE OUTLIERS
Recently, Karamcheti et al. (2021) found that "collective outliers" can sometimes explain poor AL performance. Collective outliers are groups of points that together represent an out-of-distribution set of data that the model performs poorly on, and AL is prone to selecting these examples due to their difficulty. To investigate the extent to which this effect explains our results, applied the same pruning strategy described in that work. This consists of creating a dataset map (Swayamdipta et al., 2020) which is made by first training the model for several epochs (10 for our experiments) all
3

---

Under review as a conference paper at ICLR 2022
Dataset Method
BALD-MC
BatchBALD-MC
Entropy
Coreset
Random
Arrnprune500-c
84.8 (0.3)
85.4 (0.3)
85.7 (0.4)
84.7 (0.33
84.6 (0.3)
AGN-SBB-runne550-
97.7 (0.0)
97.5 (0.1)
97.6 (0.0)
97.2 (0.1)
96.9 (0.1)
aNLI-pruned50
57.1 (0.2)
57.4 (0.1)
52.7 (0.6)
58.7 (0.2)
57.5 (0.2)
CODAHl-pruned50
60.1 (0.6)
59.2 (0.4)
58.1 (0.5)
58,9 (0.3)
58,4
USQA-pruned50
46.5 (0.4)
46.0 (0.2)
46.5 (0.3)
45.6 (0.3)
46.1 (0.2)
99.1 (0.0)
99.1 (0.0)
99.0 (0.1)
(0.0)
99.0 (0.0)
HellaSWWA-prruec550
35.0 (0.5)
35.7 (0.5)
35.6 (0.6)
36.9 (0.3)
35.7 (0.3)
MRP prun-ddudd
79.9 (0.4)
80.2 (0.3)
79.9 (0.3)
76.1 (0.4)
79.0
MRS-pruned50-c
94.4 (0.1)
94.9 (0.1)
94.8 (0.1)
94.1 (0.1)
94.1 (0.2)
PIQA-pruned50
56.0 (0.3)
55.7 (0.3)
56.0 (0.3)
57.6 (0.1)
56.1 (0.3)
SWAG-pruned50
59.8 (0.3)
59.4 (0.5)
58.3 (0.3)
59.9 (0.1)
59.8 (0.3)
Average
70.04 (0.10)
70.06 (0.09)
69.48 (0.11)
69.87 (0.07)
69.76 (0.10)
Table 2: RoBERTa-hase AUC results with 50% of the dataset pruned. Compared to the unpruned results, AL does better relative to random on average, but the improvement is not consistent across all datasets, A side-by-side comparison with Table for BatchBALD can be found in Appendix C.
examples in the pool U and calculating the mean ("confidence") and variance ("variability") of the probability the model assigned the correct label for that example during training, We then sort the examples by the product of confidence and variability and prune the lowest 50%*. Of course, this could not be done in a real-world scenario because it requires using the labels for the entire pool, but it is useful way to gain insight into the behavior of AL..
Results are in Table 2, and show mixed results on the relative performance of AL vs random selection. The best AL method in our original experiments (BatchBALD-MC, Table 1) wins over random on the same number of data sets with pruning as without (7), and pruning only improves AL's relative gap with random on 6 of the 11 datasets. Other AL methods show qualitatively similar behavior. We also note that the change in absolute performance across the different datasets, with many datasets getting worse and only four (AGN-SB-c, CODAH, CSQA, and DBpedia-c) showing consistent improvements. This result suggests that collective outliers may not fully explain the poor performance of AL on some datasets, least that pruning based on confidence and variability is not always sufficient to isolate them.
3.2
MODEL SIZE AND INSTABILITY
Dataset \
Method
BatchBALD-MC
Entropy
Coreset
Random
AGN-c
88.7 (0.3)
87.3 (0.5)
87.3 (0.1)
88.0 (0.2)
aNLI
64.0 (0.4)
60.5 (0.4)
64.5 (0.4)
64.8 (0.3)
MRP-c
80.0 (0.9)
(8.8)
84.3 (0.5)
85.1 (0.6)
MRS-c
93.5 (0.5)
90.8 (0.7)
94.9 (0.1)
94,9 (0.1)
SWAG
68.0 (0.3)
66.6 (0.3)
69.3 (0.2)
69.8 (0.4)
Average
78.83 (0.24)
76.65 (0.24)
80.05 (0.13)
80.53 (0.16)
Table RoBERTa-large AUC results. AL does substantially worse than random for most datasets.
Due to the much greater computational cost of this model, BALD is omitted and only a subset of datasets are reported.
Dataset \\ Method
BatchBALD-MC
Entropy
Coresct
Random
AGN-c
89.4 (0.1)
89.2 (0.1)
87.9 (0.1)
88.6 (0.1)
aNLI
68.2 (0.3)
65.8 (0.2)
68.1 (0.2)
68.1 (0.3)
MRP-c
87.2 (0.4)
(0.3)
86.9 (0.3)
87.1 (0.3)
MRS-c
96.4 (0.1)
95.9 (0.2)
95.7 (0.1)
95.6 (0.1)
SWAG
71.0 (0.3)
69.1 (0.2)
70.6 (0.1)
71.8 (0.2)
Average
82.44 (0.12)
81.41 (0.11)
81.84 (0.08)
82.23 (0.10)
Table RoBERTa-large AUC results using the convex hull of each learning curve. AL does much
better here compared to Table 3, suggesting that training instability was large factor in its previous failures.
An ablation with 25% pruning be found in Appendix
4

---

Under review as a conference paper at ICLR 2022
Compared most previous studies where AL has been effective for language tasks, the models have many more trainable parameters. To test whether model size may be a contributor to AL's poor performance, we evaluate the much larger RoBERTa- in Table 3, but limit this experiment to a subset of datasets due to the high computational cost of the model.
(3x more parameters than RoBERTa-bian -aase)
SWAG
MRS-c
0.725
0.95
0.700
0.90
0.650
0.05
0.625
batchbald entrapy
batchbald
0.75
entresy
0.3/2
coresee
0.350
random
0.70
random
2000
200
400
Figure 1: Averaged learning curves for SWAG and MRS with RoBERTa-large. Bars represent standard error. Best viewed in color.
The results show that AL is substantially less effeettive for the larger model. In fact, for BatchBALD and Entropy, RoBERTa-large shows more negative impact from AL than RoBERTa-base across all five datasets we evaluate in this experiment. The average difference between BatchBALD and Random is -1.7 for RoBERTa-large but +0.4 for RoBERTa-baa (for the same datasets). One reason for this is increased frequency of degenerate training runs for RoBERTa-large when using AL, which have unusually low accuracy (often no better than guessing) and manifest in Figure as points that dip sharply below the global trend of the learning curve and have high variance. For example, most results in the second half of the learning curve for SWAG are above 68% accuracy, but with AL several runs randomly drop to anywhere from 25-67%.
Such degenerate cases with large transformers are well-known (Devlin et al., 2019). But the substantial increase failed training runs when using AL leads us to believe that AL methods select sets of examples that amplify the instability of the optimization process, which we define here as the variance in test set accuracy observed when training multiple models (with different random seeds) on the same set of data. The increased instability means that the selected examples are not necessarily harmful as the collective outlier theory suggests (because not training runs fail), but instead are potentially useful examples that are hard to learn from effectively.
Correcting for these failed training runs is hard for two reasons: (1) they are frequent enough that would be computationally prohibitive for us to re-run all of them with different random seeds until getting a success, as in previous work (Devlin et al., 2019; Porada et al., 2019), and (2) the model does not always degrade random-label performance, meaning a threshold would need to be chosen for what consitutes a "failure", and this choice may affect the conclusions.
To roughly approximate how well the model could do if it did not have these optimization failures, we consider the area under the convex hull of the learning curves instead of including all points, and display the results in Table 4. This substantially improves AL relative to random and suggests that the increased frequency of failed training runs is the main culprit for the poor performance of AL. Because the learning curve of accuracy vs dataset size is generally convex, our convex hull approximation is an underestimate of what we would expect if we actually achieved the best optimization that the model could reasonably achieve with the selected data.
To lesser extent, we find evidence of AL-induced instability on RoBERTa-base as well. Figure 2 shows some of the learning curves from the Table results, and while they are much less noisy than the RoBERTa-larg table, it still qualitatively appears that AL has more noise than random on the datasets where AL loses. Quantitatively, we compute the average difference between the convex hull AUC and the normal AUC, which serves to estimate the level of instability in the absence of repeated trials on the same data. On the datasets where AL loses, BatchBALD's 's average difference (2.04) is larger than Random's (1.80), whereas datasets where AL wins the average differences equal (0.77). That is, AL shows greater instability on the losses than on the wins, suggesting that instability may play role in the failures.
5

---

Under review as a conference paper at ICLR 2022
MRS-c
0.96
0.650
0.94
Accuracy
Accuracy
0.600
0.92
0.575
bald
bald
0.550
batchbald
0.90
batchbald
entropy
entropy
0.525
greedy coreset
greedy coreset
random
0.88-
random
0.500
100
200
300
400
500
100
200
300
400
500
CODAH
0.60
0.55
bald
batchbald
0.45
entropy
greedy coreset
0.40
random
100
200
300
400
500
Figure 2: Averaged learning curves for SWAG, MRS-c and CODAH with RoBERTa-base. Bars represent standard error. Best viewed color.
3.3 MULTI-TRAINING ROBERTA-BAR BASE
Multi-training
Original (from Table 1)
Dataset
Method
BatchBALD-MCC
Random
BatchBALD-MC
Random
AGN-c
87.5 (0.3)
(0.1)
87.5 (0.2)
86.6 (0.2)
aNLI
59.5 (0.2)
59.7 (0.1)
58.9 (0.2)
58.8 (0.2)
CODAH
59.0 (0.8)
58.7 (0.4)
58.5 (0.3)
59.0 (0.6)
CSQA
44.6 (0.3)
45.2 45.2 (0.2)
43.9 (0.4)
43.8 (0.2)
DBpedia-SB-e
(0.0)
98.8 (0.0)
(0.0)
98.8 (0.1)
HellaSWAG
40.1 (0.1)
39.7 (0.2)
38.5 (0.2)
38.7 (0.2)
MRP-c
84.5 (0.1)
83.7 (0.2)
83.3 (0.2)
83.0
MRS-c
95.3 (0.1)
(0.1)
95.3 (0.1)
94.0
PIQA
56.7 (0.2)
57.1 (0.3)
55.9 (0.2)
56.8 (0.3)
SWAG
63.8 (0.3)
64.1 (0.2)
62.7 (0.2)
63.5 (0.2)
Average
68.99 (0.10)
68.84 (0.07)
68.32 (0.07)
68.30 (0.09)
Table 5: Multi-trained RoBERTa-base AUC results with 5 trainings per point the learning curve. Original results from Table are reprinted for comparison. Absolute numbers are significantly better with multi-training but relative results are mixed.
Because RoBERTa-base is several times smaller than RoBERTa-large, it is feasible to do a more thorough investigation of the impact of instability by training multiple times on each collected dataset along each learning curve to ensure that get well-optimized model ("muuli-training"" our experiments with BatchBALD and random sampling, training the model 5 times after each batch acquisition and picking the best the dev set use both for evaluation and for selecting the next batch from the pool. Results are reported in Table
We repeat
We find that multi multi-training improves BatchBALD relative to random all of the datasets where BatchBALD underperformed random Table 1. However, the improvements are small enough to attribute to noise in some cases, and in some of the cases where BatchBALD ID previously outperformed

---

Under review as a conference paper at ICLR 2022
random, it became relatively worse after multi-training We conclude from this that the effect of instability on smaller models less pronounced than on larger models.
Finally, note that the effects of multi- multi-training are similar those of pruning in Table 2; for example, both substantially improve AL on CODAH and slightly worsen on CSQA. However, multi-training improves the absolute performance across almost all results (p 10-4 for related- sample t-test), whereas pruning mostly reduces it (p < 0.02). Multi-training should improve accuracy if instability is issue, but there no reason believe helps the model to ignore collective outliers in the training set. Thus, the fact that a similar improvement to pruning can be achieved by multi-training without pruning suggests that instability is distinct (but complementary) phenomenon from collective outliers.
3.4 OTHER ABLATIONS
Although instability appears to largely explain poor AL performance on large models and collective outliers explain it some datasets, there are several datasets for which AL still fails to outperform random on RoBERTa-beed even after attempting to correct for these effects. In this section, we investigate variety of alternative hypotheses for the results we observed.
3.4.1 QUESTION FORMAT
Dataset \ Method
Entropy
Coreset
Random
RoBERTa (base)
aNLI-c
50.9 (0.2)
51.8 (0.1)
53.1 (0.1)
CODAH-C
59.9 (0.5)
60.0 (0.2)
60.3 (0.5)
HellaSWAG-c
51.2 (0.2)
54.0 (0.1)
51.8 (0.2)
PIQA-c
50.3 (0.1)
50.6 (0.1)
50.5 (0.1)
SWAG-c
63.9 (0.4)
65.0 (0.1)
64.3 (0.6)
Average
55.22 (0.13)
56.29 (0.06)
55.98 (0.15)
BERT (base)
aNLI-c
50.3 (0.1)
50.3 (0.1)
50.8 (0.1)
SWAG-c
59.3 (0.4)
60.4 (0.2)
59.5 (0.3)
Average
54.81 (0.20)
55.38 (0.12)
55.15 (0.17)
Table 6: AUC results for classification versions of multiple-choice datasets. Compared to Table 1, AL relatively improves on some datasets and worsens on others vs random, with no clear pattern. BALD and BatchBALD are omitted here due to their high cost.
In Table 1, we see that AL typically performs better on the text classification datasets than the commonsense reasoning ones. One difference between these tasks is the kind of language competency they attempt to evaluate, but another is simply their format: the text classification datasets require choosing one of a fixed set of (typically two) labels, whereas the multiple-choice tasks require selecting which of multiple cnuestion-specifiie passages correct. We investigate this confound by creating text classification versions of the multiple-choice datasets, where each text classification example is the multiple-choice prompt concatenated with one answer choice and the classification task is to determine if the given answer is correct. For multiple-choice tasks with more than two answer choices, we subsample the resulting dataset to balance the number of positive and negative examples.
The results, displayed in Table 6, do not show a consistent effect of question format on AL. SWAG and CODAH in the text classification format appear get bigger boost from AL than in the multiple-choice format, but aNI.I does slightly worse and HellaSWAG and PIQA stay roughly the same. We conclude that the trend observed Table may have more do with the typical types of reasoning used by classification vs. multiple-choic datasets than with the format used to present them to the model.
3.4.2 ADVERSARIAL FILTERING
A possible explanation for the failure of AL commonsense reasoning tasks compared to text classification is that many of the multiple-choice datasets are adversarially filtered Bras et al. (2020). This means questions that baseline model gets right in cross-validation removed during dataset
7

---

Under review as a conference paper at ICLR 2022
construction, which may have similar effect to AL and cancel out the benefit. The adversarially- filtered datasets are aNLI, SWAG, CODAH, PIQA, and HellaSWAG, which together make up the majority of cases where AL fails in experiments.
To probe the effect of filtering, we obtained the unfiltered version of aNLI from its authors. The Entropy, Coreset, and Random methods resulted in AUC values of 78.3 (0.2), 79.8 (0.3), and 80.8 (0.2) respectively (due to the large size, running BALD was not feasible). Perhaps surprisingly, this is no better than the result with the filtered dataset, suggesting that adversarial filtering does not explain the poor performance of AL.
3.4.3 PRETRAINING
Dataset \ Method
BatchBALD-MCC
Entropy
Coreset
Random
AGN-c
87.8 (0.2)
87.3 (0.2)
87.1 (0.2)
87.3 (0.2)
AGN-SB-C
97.8 (0.1)
98.0 (0.0)
97.5 (0.0)
97.1 (0.1)
aNLJ
52.9 (0.1)
52.3 (0.1)
52.3 (0.1)
52.8 (0.1)
CODAH
48.3 (0.4)
44.8 (0.6)
47.7 (0.6)
48.1 (0.7)
CSQA
29.1 (0.4)
28.4 (0.3)
27.6
28.5 (0.2)
DBpedia-SB-e
99.1 (0.0)
99.1 (0.0)
99.1 (0.0)
99.0 (0.0)
HellaSWAG
29.9 (0.3)
(0.3)
30.4
29.8 (0.3)
MRP-c
80.5 (0.2)
(0.3)
77.5
(0.4)
MRS-c
94.5 (0.1)
94.3 (0.1)
(0.1)
93.2
PIQA
52.6 (0.3)
52.6 (0.2)
52.6 (0.2)
52.3 (0.2)
SWAG
60.6 (0.4)
56.0 (0.4)
58.2 (0.4)
60.4 (0.2)
Average
66.63 (0.08)
65.58 (0.09)
65.70 (0.08)
66.26 (0.09)
Table 7: BERT-base AUC results.
Previous studies have found that model regularization (Munjal al., 2020) and semi-supervised training (Chan et al., 2021) can reduce the effectiveness of AL,, so we investigate whether unsupervised pre-training may have similar effect. Using randomly-inittalized model unfortunately produces random-baseline performance in all cases, so we instead use BERT-base for comparison. Because BERT and RoBERTa have the same architecture but RoBERTa uses much more pretraining data (roughly 10x), we expect that if pretraining reduces AL effectiveness then RoBERTa should benefit less from AL than BERT. The results for BERT are reported in Table 7, and by comparing them to the RoBERTa results in Table 1 we see that BERT generally gets better relative performance with BatchBALD, worse with Coreset, and about the same with Entropy. This suggests that the effect of pretraining varies between methods. We suspect that the reason Coreset does better with RoBERTa may be related to better pretraining producing better embeddings with which to construct the core-set. We also note that the benefit from BatchBALD any individual dataset is generally small and not significant, but due to its consistency across datasets we find a significant (p < 0.008) difference in the average performance Random in related-sample t-test.
3.4.4 ANNOTATION BATCH SIZE
As an additional ablation, we consider the possibility that the annotation batch size |AC] is critical to the success of AL, and the necessary batch size is simply too small to be feasible for large models, because computation cost is inversely proportional to the batch size. To test this, we run experiments with batch size 12 instead 25 for the AGN and SWAG datasets in Table and find that using batch size of 12 produces similar result to a batch size of 25.
Dataset \ Method Entropy
Random
AGN-c
87.3 (0.3)
86.1 (0.2)
SWAG
59.7 (0.2)
62.6 (0.2)
Table 8: RoBERTa-base AUC results with | 12. Overall results are slightly lower than in
Table 1, likely due to the extra point on the learning curve where dataset size is 12, but the differences between AL and random are roughly unchanged.
8

---

Under review as a conference paper at ICLR 2022
4 RELATED WORK
Many AL methods have been proposed. Many some estimate of model uncertainty as a heuristic for useful examples; this includes model ensemble agreement (Prabhu et al., 2019), Monte Carlo dropout (Gal& & Ghahramani, 2016), and a variety of simpler methods such as entropy of the model output (Settles, 2009). Others have found that encouraging diversity can be more robust, particularly in the context of deep learning (Sener Savarese, 2018: Sinha et al., 2019). AL has been found to be effective in multiple domains, including computer vision (Gal et al., 2017) and natural language processing (Siddhant& Lipton, 2018).
Despite its successes, recent work has found several cases where the henefit of AL seems to disappear. Munjal et al. (2020) finds that model regularization overlaps with the gain from AL., with better- regularized models having higher task performance but receiving no benefit from AL. Several studies (Mittal et al., 2019; Chan et al., 2021; Siméoni et al., 2019) have found that self-supervised and remi-supervised learning can have similar effect, improving model performance but failing to stack with boosts from AL. Lowell et al. (2019) finds that the benefits of AL on text classification tasks often do not transfer between models.
Unlike the work mentioned above, our focus is on evaluating AL with large pretrained Transformer language models. Previous results have shown that AL be effective for certain pretrained transformers on text classification tasks Lu & MacNamee (2020); Ein-Dor et al. (2020). Our results qualitatively agree with the prior work when we evaluate on the same datasets and models, but we show how AL becomes far less effective when moving to larger and better-performing pretrained models or richer commonsense reasoning tasks.
Similar to our results, Karamcheti et al. (2021) recently found a negative result for AL in the domain of visual question-am -answion-nnswering.... and discovered that the effect was largely explained by "unlearnable" collective outliers that the model often fails on even during training. However, we find that applying their pruning technique to our ten datasets and using recent batch-aware AL methods (Kirsch et al., 2019) yields mixed results, suggesting that the impact of collective outliers may vary by dataset.
Finally, with regard to the instability of large transformers, Liu et al. (2020) find that the position of layemnom relative to residual connections in transformer layers can cause instability by amplifying the effects of small parameter perturbations. This may be related to the effects we observe, as in some sense AL selects examples that maximally change the model, making it more susceptible to this effect. Further investigating this and applying the proposed Admin initialization technique to resolve the instability is interesting avenue for future work.
5 CONCLUSION
We have investigated active learning (AL) with large pretrained Transformers a variety of tasks not covered by previous experiments, finding that AL offers poor and inconsistent performance in many cases. We identify novel explanation for some of these failures in the case of large models: an increase in instability, as evidenced by e.g. a much higher frequency of cases where RoBERTE -a- fails to converge. Attempting to resolve the instability by selecting the best of multiple trainings on each selected dataset behaves similarly to an outlier-pruning technique from prior work in terms of relative performance between AL and random, but with higher absolute performance. This suggests that multiple trainings may be a preferable approach for studying AL compared to pruning, as it allows measurements in more practically relevant thigher--erffrmin of other ablations suggest that factors such as acquisition batch size, adversarial filtering, and input format are not responsible for the suboptimal behavior of AI., and the effects of greater model pretraining are inconsistent between AL methods.
setting. Finally, a variety
Several issues remain to be addressed in future work. Some experiments are limited due to high computational cost, so more exhaustive investigation of whether further improvements in model optimization would enable AL to outperform random in all cases remains a task for future work. In addition, tests with wider range of models and model sizes would be valuable for confirming the generality of these findings. Lastly, we hope our findings will enable the development of improved active learning and optimization algorithms that do not suffer from the drawbacks we discovered.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. Abductive commonsense reasoning. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL Byglv1HKDS
forum? id=
Yonatan Bisk, Rowan Zellers, Ronan LeBras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In The Phirty-Fourdl AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Con- ference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 7432-7439, AAAI Press, 2020. URL ht https:: awwaaaaaioog//o/////..//a//////a ojs/ index. AAAI 62399 6239.
Ronan Le Bras, Swabha Swayamdipta. Chandra Bhagavatula, Rowan Zellers, Matthew Peters, Ashish Sabharwal, and Yejin Choi. Adversarial filters of dataset biases. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1078-1088. PMLR, 13-18 Jul 2020. URL http: http://ppoeeednnnn...e.e wwt.:gssmmr..ppr... / v/vvvvll1// /wt.abra200 aaaras20a.htti.
Yao-Chun Chan, Mingehen Li, and Samet Oymak. On the marginal benefit of active learning: Does self-supervision eat its cake? ICASSP 2021 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3455-3459, 2021.
Michael Chen, Mike D' Arcy, Alisa Liu, Jared Fernandez, and Doug Downey. CODAH: An adversarially-authrre question answering dataset for common sense. In Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP, pp. 63-69, Minneapolis, USA, June 2019. Association for Computational Linguistics. doi: 10.18653/11/119-20000 URL
https: www.wwwwcc.cmmmmmcoooo c web.org/ anthon ogy /W1 9 2008.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT; Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171 14166, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10. 18655//1/119-14222 URL https: //wpwaccleebbrrg. hww.:/b....gg...ooooooooooooo/// wwwaaalthooaooaoaeooooooool /N1 14233.
Liat Ein-Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, and Noam Slonim. Active Learning for BERT: An Em- pirical Study. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7949-7962, Online, November 2020. Association for Computational Linguistics. doi: 10. 18653/v 1/2020.emnlp- main.638. URL https: : /ttp://wwwwwbborrgggggmm anthology matoollgy/222..emmii http://2200.m.. 1tahholoog/2222..mmmmmmmmm..... 13....
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. Maria-Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference Machine Learning, ICML 2016, New York City, NY. USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 1050-1059, JMLR. 2016. URL http: pwoceedoooeeeeeeeeooooo ngs.ml r.press / v /gall 6.html.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In Doina Precup and Yee Whye Tch (eds.), Proceedings of the 34th International Conference Machine Learning. ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 1183-1192. PMLR, 2017. URL http:// www.eeddiing........ss.e. hatpr///ggaaaaahaaaa.
Neil Houlsby, Ferene Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian Active Learning for Classification and Preference Learning. arXiv:1112.574 [cs, stat], December 2011. URL arXiv: 1112.5745.
http:// http:://aradi.orrg/bbs////
10

---

Under review as a conference paper at ICLR 2022
Peiyun Hu, Zachary C. Lipton, Anima Anandkumar, and Deva Ramanan. Active learning with partial feedback. In 7th International Conference on Learning Representations. ICLR 2019, New Orleans, LA. USA, May 6-9, 2019. OpenReview.net, 2019. URL forum? dFJJfSERRKK
Siddharth Karamcheti, Ranjay Krishna, Li Fei-Fei, and Christopher D Manning. Mind your outliers! investigating the negative impact of outliers on active learning for visual question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7265-7281, 2021.
Jungo Kasai, Kun Qian, Sairam Gurajada, Yunyao Li, and Lucian Popa. Low-resource deep entity resolution with transfer and active learning. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pp. 1561,, Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10.18653/v 1/P0991/11111-1588 URL https: / //aalaanchooog..orgg/////
1586.
Andreas Kirsch, Joost R. van Amersfoort, and Y. Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. In NeurlPS, 2019.
Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the difficulty of training transformers. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 5747-5763, 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: Robustly Optimized BERT Pretraining Approach. July 2019. URL https: /ttp::s///axxvvvvr//a
David Lowell, Zachary C. Lipton, and Byron C. Wallace. Practical obstacles to deploying active learn- ing, In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing
pp. 21-30, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 1003.
10.11863//1/DI9-1 1003. URL
Jinghui Lu and Brian MacNamee. Investigating the Effectiveness of Representations Based on Pretrained Transformer-based Language Models in Active Learning for Labelling Text Datasets. arXiv:2004. 13133388001 [cs], April 2020. URL http:://ee http:://axvvvorrreee abs /200..1004.133 arXiv: 2004.13138.
Sudhanshu Mittal, Maxim Tatarchenko, Özgünn Çięek, and Thomas Brox. Parting with Illusions about Deep Active Learning. arXiv: 053661 [es], December 2019. URL htthtt////axxxivorg abs/1912.050 arXiv: 1912.05361.
Prateek Munjal, Nasir Hayat, Munawar Hayat, Jamshid Sourati, and Shadab Khan. Towards Robust and Reproducible Active Learning Using Neural Networks. arXiv:2002.0m 09564 [cs, stat], February 2020. URL
arXiv: 2002.09564.
Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summa- rization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Associa- tion for Computational Linguistics (ACL-04), pp- 271-278, Barcelona, Spain, July 2004. doi: 10.3115/1218855. 1218990. URL https: http://wwwwwwwb...rw.cc// hatttooohhooyyoo//o
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categoriza- tion with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Asso- ciation for Computational Linguistics (ACL'05), pp. 115-124, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. doi: 0..111//111984012199855 URL ht attps:: www.aclweb.orggggpooooo
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Viti,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance
11

---

Under review as a conference paper at ICLR 2022
deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32. pp. 6024-8035. Curran Associates, Inc., 2019. URL http: / http://///pp....eeeeeaec//aaeeea
imperative-se sty eettlleehie wweeiighppppeeeeeeee wwr.rrrrrrrrooe
lan Porada, Kaheer Suleman, and Jackie Chi Kit Cheung. Can gorilla ride a camel? learning semantic plausibility from text. In Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, pp. 123-129, 2019.
Ameya Prabhu, Charles Dognin, and Maneesh Singh. Sampling bias in deep active classification: An empirical study. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 4058-4068, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/w 1417. URL https:: wttp:://www.awww.orrg... http//nnhhhogyyy/ D19-1417.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL
http:///eeeeeeeeeeeeeeeeeeee /
https::
/ contt/foxee forum? (o=Haaluk-Ri
Burr Settles. Active learning literature survey. Technical report, University of Silcoosii-MMadise Department of Computer Sciences, 2009.
Yanyao Shen, Hyokun Yun. Zachary C. Lipton. Yakov Kronrod, and Animashree Anandkumar.
Deep active learning for named entity recognition. In 6th International Conference on Learning Representations, ICI. 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https: / /ttp///ppeeeeeeettt...e// forumiid-
Aditya Siddhant and Zachary C. Lipton. Deep Bayesian active learning for natural language pro- cessing: Results of a large-scale empirical study. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2904-2909, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18653/11/DI---D0 1318. URL httprr//nnnttttttotneeeeeo /D18- 1318.
Oriane Siméoni, Mateusz Budnik, Yannis Avrithis, and Guillaume Gravier. Rethinking deep active [cs], November 2019, URL
learning: Using unlabeled data at model training. arXiv: 1911.08177.
http:// http//xxxvvvrr..ee.o///// aass/////////// 1911.08177
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp-
IEEE, 2019. doi: 10. WWW.CCCV.2019.00660
URL http:: http:/oooooooooo/ 190 ICC..20I 9.00607.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang. Hannanch Hajishirzi, Noah A. Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In EMNLP. 2020.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Q: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4149-4158, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10. 10.18653/1//11-142221 URL https: // //tpw.acwweb.orgg/////// www.:acc.wbb.org///noooooogg// www.taloooooooaoaaoooooooll /N1 9- 1421.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-an natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. Association for
12

---

Under review as a conference paper at ICLR 2022
Computational Linguistics. URL /tt.://ts////.......eee////tt/ta
demos.6. 6.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 93-104, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-110000 1009. // www.aclweb.ory.. http://wwww......ooooooooc////// /01//10118110111
URL https:
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pp. Linguistics. doi: 10.18653/v1/P19- 1472. URL https: / http///wwwwwb.....c.ccmm anthology/ 117-1472.
Florence, Italy, July 2019. Association for Computational
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Corinna Cortes, Neil D. Lawrence, Damiel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.). Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 64996557, 2015. URL https: pttpss/ httpppppccpppppp........ //hah// 355c688811731166666686677888-800
13

---

Under review as a conference paper at ICLR 2022
A HYPERPARAMETERS
Parameter
Value
Learning
2e-5
Adam warmup steps
150
Batch size
16
Adam
le-6
Adam Br
0.9
Adam
0.98
Weight decay
0.01
Table 9: Transformer hyperparameters
Transformer models were fine-tuned for fixed 1,000 steps, evaluating on the dev set every 100 steps and using the checkpoint with best dev accuracy obtain the final test set accuracy. We found from preliminary experiments that the hyperparameters in Table generally worked well on models and datasets. The learning rate followed sloped triangular schedule, warming up over the first several steps and then linearly decreasing until the end of training. The maximum sequence length for models was set per-dataset based on the length of input texts. In general it set that at least 99% of examples fit completely within the sequence length, and those that did not fit were pruned.
For BALD, the number of Monte Carlo dropout samples experiments on CSQA and MRP that increasing the number of samples to 20 did not substantially improve results. We performed the Monte Carlo dropout using the normal dropout probabilities the dropout layers of the models (generally 0.1); increasing the dropout probabilities to 0.5 (just for the BALD step, not model training) added noise due the large number of dropout layers and actually caused performance to slightly decrease in our preliminary tests.
to 5, and we found in preliminary
The experiments were run on servers with Nvidia RTX 2080 Ti and GTX 1080 gpus. We estimate that all reported experiments together represent about 3,000 gpu-hours for an RTX 2080 Ti.
Our experiments are implemented using Pytorch (Paszke al., 2019) and code available in the supplementary material, We used version 2.4.1 of the HuggingFace Transformers library (Wolf et al., 2020), which has small bug in the RoBERTa implementation. Testing two of our datasets with newer version suggests that this doesn't it change our conclusions.
B DATASET SIZES
See Table 10.
C PRUNING LEVEL
For our main experiments we pruned the 50% of examples most likely be collective outliers, as that threshold is what appeared to work best in the work of Karamcheti et al. (2021). However, as we did find pruning be as effective as in that work, we also tried 25% threshold with just one AL method (BatchBALD) to see if the pruning threshold made a difference. The results are displayed in Table 11. We observe that both the relative gain from AL and the absolute performance fall between the corresponding numbers for 0% and 50% on average, and therefore conclude that while the pruning threshold could adjusted trade relative and absolute gains, it would not produce a boost in both at once across datasets.
D LABEL BUDGET
We primarily focus on a low-budget setting in this work, but it is interesting to consider how performance might improve with more labels. In Table 12 show the results of experiments with
14

---

Under review as a conference paper at ICLR 2022
Dataset
Train
Dev
Test
AGN-SB-C
2000
2500
5000
DBpedia-SB-c
2000
2000
2000
CODAH
2376
100
200
CODAH-c
4752
400
CSQA
9141
300
1221
MRP-c
8614
512
1024
MRS-c
9000
500
500
PIQA
14113
1000
1838
PIQA-c
28226
2000
3676
HellaSWAG
39905
1000
8042
HellaSWAG-c
65198
1988
15508
SWAG
65354
4096
20006
AGN
100000
5000
10000
SWAG
130708
8192
40012
aNLI
1500000
1036
1532
aNI.I-c
300000
2072
3064
Table 10: Dataset sizes. "Train" the set we used as the unlabeled pool for active learning. Note also that the numbers add up to slightly less than the official sizes of these datasets, as we held out some additional data that ultimately went unused this work.
0% pruned (irom Table l)
25% pruned
50% pouned (from Table 2)
Methol
BanchBAL.DMMC
BathBALL--MC
AGN-6
$7.5 (0.3)
30.6 (0.2
87.0 (0.2)
30.6 (0.2)
85.4 (0.3)
(4.5
AGI-SB-c
97.4 (0J))
96.7 (0.1)
97.4 (0.1)
96.8 (0.1)
97.5 (0.1)
96.9 (0.1)
58.5 (0.3)
(0.5)
(0.4)
CSQA
43.9 (0.4)
43.8 (0.2)
45.8 (0.3)
45.2 (0.3)
46,0 (0.2)
46.1 (0.2)
DBPedia-
98.9 (0J))
98.8 (0.1)
99.1 (0.0)
99,0 (0.1)
99.1 (0.0)
99.0 (0.0)
HelloWAA
(0.2)
MIS-c
95.3 (0.1)
94.0 (0.2)
(0.1)
94.0 (0.1)
94.9 (0.1)
94.1 (0.2)
PIQA
55.9 (0.2)
5688 (0.3)
55.8 (0.3)
56.2 (0.2)
55.7 (0.3)
56.1 (0.3)
SWAG
Average
(0.08)
78.75 (0.08)
(0.08)
70.06 (0.09)
69.76 (0.10)
Table 11: Roberta-base with different levels of pruning.
15

---

Under review as a conference paper at ICLR 2022
Q=1000, |
Q=500, |=5 (from Table 1)
Dataset Method
BatchBALD-MC
Random
BatchBALD-Me
Random
AGN-c
88.6 (0.1)
88.2 (0.1)
87.5 (0.2)
86.6 (0.2)
AGN-SB-C
97.7 (0.1)
97.3 (0.1)
97.4 (0.0)
96.7 (0.1)
aNLI
60.2(
60.5 (0.2)
58.9 (0.2)
58.8 (0.2)
CODAH
62.6 (0.4)
62.1 (0.4)
(0.3)
59.0 (0.6)
CSQA
47.8 (0.4)
47.7 (0.2)
43,9 (0.4)
43.8 (0.2)
DBpedia-SB-e
99.1 (0.0)
99.1 (0.0)
98.9 (0.0)
98.8 (0.1)
HellaSWAG
41.2 (0.1)
(0.1)
38.5 (0.2)
38.7 (0.2)
MRP-c
85.6 (0.1)
(0.3)
83.3 (0.2)
(0.3)
MRS-c
(0.1)
(0.0)
(0.1)
94.0
PIQA
57.7 (0.1)
57.4 (0.3)
55.9 (0.2)
56.8 (0.3)
SWAG
65.7 (0.3)
65.8 (0.2)
62.7 (0.2)
63.5 (0.2)
Average
72.92 (0.06)
72.63 (0.07)
70.96 (0.07)
70.88 (0.08)
Table 12: RoBERTa-base AUC results with Q = 1000 and |L| = 50. Original results from Table reprinted for comparison. AL has higher relative performance the results with greater label budget and batch size.
budget Q=1000 and batch size* |L|=500. The results show that higher budget generally results in better relative AL performance. This result is encouraging, as it suggests the instability of AL-selected datasets is mitigated simply by collecting more labels. Of course, does not help in where the labels are costly and the budget cannot easily be increased, so an interesting question for future work how to determine the minimum viable label budget needed for AL become effective.
'Unfortunately cannot avoid changing multiple variables here, as either the batch size or the number of batches must increase when larger budget. Due our carlier batch size ablation consider it unlikely that the hatch size plays large role, and generally attribute the results here the hudget increase.
16


Reviews:
[{"review_id": "pZicUZMjsyw", "reviewer": "Reviewer_Seyb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: The experimental scale of this work is undoubtedly large and comprehensive, involving many data sets, different types of active learning algorithms. It is constructive for the researchers who tried to use active learning methods for reducing the labeling cost in NLP tasks. \n\nWeaknesses: \n1) compare with [1], the research questions and the corresponding conclusions of this work are not enough novel and attractive. Actually, current studies have involved: i) marginal benefits of deep active learning methods; ii) employing active learning with pre-trained LMs like Bert on regular NLP tasks; iii) employing active learning on challenging tasks like VQA. In fact, these studies have covered almost all existing research topics that worth studying and revealed the problems existing in deep active learning. For now, it is more meaningful to address these issues than to continue to discuss whether deep active learning has similar problems in some subdivided tasks.\n\n2) In the experiments, the authors only adopt 500 quota for each AL experiment, it's too small. For example, in Figure 2 (CODAH), the performance of AL methods are far from convergence, the author should adopt more quota or at least provide the performance trained on full dataset, like [2].\n--------------------------------------- Minor Issues -------------------------------------------\n\n3) Many writing problems could be found in this paper. For instance, in page 2, in contribution 2, the sentence is incompleted. In section 2.2, should add ';' or '.' at the end of each item when describing the AL algorithms.\n\n4) It is inaesthetic to plot standard error bar, like Figure 1, can use plt.fill_between instead of plt.errorbar.\n5) Should pay more attention to design the layout of tables, e.g., in Table 6, it should be Method | Dataset | AL methods.\n\n\nReferences:\n[1] Karamcheti S, Krishna R, Fei-Fei L, et al. Mind your outliers! investigating the negative impact of outliers on active learning for visual question answering[J]. arXiv preprint arXiv:2107.02331, 2021.\n\n[2] Margatina K, Barrault L, Aletras N. Bayesian Active Learning with Pretrained Language Models[J]. arXiv preprint arXiv:2104.08320, 2021.", "summary_of_the_paper": "This paper test 4 deep active learning mehods across 10 datasets (including text classification and multi-choice commonsense reasoning) based on pre-trained LMs. It conduct many ablation studies to explore whether some inherent factors of like batch size in active learning, the model size of pre-training models would affect the effectiveness of active learning with pre-trained LMs. ", "summary_of_the_review": "This paper conducted a large amount of empirical experiments on AL with pre-trained LMs. However, the research problems they explored are somewhat covered by existing research outputs and their conclusions are less attractive compared with [1]. Additionally, the writing of this article should also be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gaXylHaxXWD", "reviewer": "Reviewer_igJc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n - I could learn various empirical observations of how active learning works in these large pre-trained models on various NLP tasks. I hope to see this paper at the conference and others can learn from it as well. \n\nWeakness:\n - It would be nice to see some macro analysis of data points showing the overall tendency of each setup. \n - I hope to see more insights regarding how these two different AL methods (density-based and entropy-based) would work in a different way.", "summary_of_the_paper": "This paper examines how active learning works with large pre-trained models, with a novel explanation with empirical evidence. ", "summary_of_the_review": "Good empirical findings from extensive experiments, but limited to the individual setup in specific conditions without an overall picture of dataset dynamics.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cZ50f-Pgla-", "reviewer": "Reviewer_e7J6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n \nThe authors connect active learning example selection to more severe training instability than random selection. Further, they show downside-variability (“greater instability on the losses than on the wins”) suggesting specific sets of examples are responsible for systemic failures in active learning, but not necessarily that other selections would greatly outperform the random baseline. Both of these observations are of interest to the NLP active learning community, adding depth to the characterization of the problem.\n \nThe wide range of datasets and active learning techniques they use (including BALD which prior works shows is very competitive) lends credence to the conclusions. It especially corroborates that certain tasks are innately more challenging for active learning than others. \n \nTheir thorough ablation experiments and other analysis yield some interesting findings the authors could emphasize more. For instance, the greater instability of larger Transformers to active learning bodes poorly for practitioners leveraging ever increasing model sizes for low-resource datasets. Additionally, the authors conclude that pre-training is not a significant factor in the efficacy of active learning, but their numerical results suggest active learning methods (Entropy and Coreset) narrow the gap with the random baseline significantly from BERT-Base to RoBERTa-Base! The margin of change seems even larger than some results which are discussed in the paper as significant. More discussion could bolster this finding.\n \nWeaknesses:\n \nThe primary weakness of the paper is the lack of convincing justification that the authors have discovered a phenomenon distinct from “collective outliers” (Karamcheti et al., 2021) -- or if it is distinct, how exactly is it distinct? \n\n1. For context, Karamcheti et al. (2021) introduce an \"oracle\" active learning method that prunes uninformative (outlier) examples from the unlabelled, selectable pool U, but it requires gold labels to filter these. This work tries a single pruning rate and finds it significantly degrades, rather than improves results: for both active learning and random selection. However, they only mention a single pruning rate of 50%, which could easily over-prune (all challenging as well as outlier examples) for the NLP datasets used here. Especially because of the surprising magnitude by which this pruning degrades absolute performance, it is unfortunately necessary to try more pruning rates for a fair comparison.\n\n2. Even if pruning is ineffective (at multiple pruning rates), it’s never really explained why, despite this being a core contribution of the paper. \n\nA qualitative analysis was required in Karamcheti et al., (2021) to explain “collective outliers”. A similar analysis here could greatly demystify why these sets of examples cause instability, and whether they are indeed “informative”.\n \nThere are a couple decisions related to methodology that may need some further examination. \n\n1. The first is the total unlabelled pool size. It appears to vary by orders of magnitude according to Table 10 in the Appendix. While the batch selection L and total selection Q variables are constants, fixed at 25 and 500 respectively, the pool of examples to choose from varies wildly. This is very likely a confounding factor in the efficacy of active learning and pruning techniques. It deserves some discussion or control.\n\n2. The second is the choice of L=25 examples chosen at each acquisition, and also as the seed set. I am concerned this is quite low and could yield exaggerated instability, especially for large Transformer models. If I read them correctly, Ein-Dor et al. (2020) use at least L=50 for simple classification tasks, and Karamcheti et al., (2021) use L>= 400 and larger seed sets. The authors investigate L=12 in the ablation, but in a real setting, it seems unlikely practitioners will label <50 examples before re-training.\n\nParticularly because the reported accuracy margins are so slim, either of these variables could modify the empirical conclusions. A statistical test on the Average performances would also improve confidence in the conclusions.\n\nQuestions and minor suggestions for the authors:\n\n1. Why is Entropy missing from Table 2? And BALD-MC from Tables 3 and 4? And why is BALD missing again from Table 6? Computational expense is used to justify the omission of BALD in one instance, but the others appear inconsistent?\n2. It’s very hard to compare the absolute differences Tables 1 and 2 for ourselves. Perhaps there is some visual representation that could help demonstrate the comparisons you make in the text?\n3. Margins are very small for the Average differences across all datasets as well -- have you considered confidence intervals on those as well?\n4. Minor: Only half the datasets are shown in Tables 3 and 4, but it’s unclear how/why those were chosen?\n\n", "summary_of_the_paper": "This work evaluates active learning methods for pre-trained Transformer models, and diagnoses their inconsistent performance as a consequence of training examples which exacerbate variability in test-time accuracy. They propose re-training the model repeatedly to circumvent bad outcomes due to variability, and show this works particularly well on larger pre-trained models. The authors conduct a wide range of experiments, over many datasets, and also a series of ablation studies to reinforce their claim that these active learning methods are ineffective for pre-trained Transformers.", "summary_of_the_review": "In summary, we would encourage the authors to more rigorously examine the heart of their work -- the source of the instability they uncover: is pruning really ineffective, or are we removing too many challenging examples along with the outliers? Can we qualitatively diagnose the problem and explain what makes an example “informative” but unstable? And as accuracy margins are so small, it may be unfortunately necessary to justify or relax some hyperparameter choices, and run additional experiments to confirm these findings generalize.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PRwSOfCZOTS", "reviewer": "Reviewer_dz3i", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- Experiments on many data sets.\n- The idea of using convex hull of AUC is interesting, but it would it nice to confirm it experimentally. \n\nWeakness:\n- Q seems to be too small, which might amplify the instability issue.\n- Little novelty \n\nWhile their experiments, and findings are interesting, it leaves many questions that could (at least in principle) be answered by more experiments. \nTheir main finding seems to be that larger transformer models lead to more instability for training with the instances selected by AL, Table 4 and Table 5.\nHowever, Table 4 is only an approximation to results that could be achieved with training with different initializations. Though computationally expensive it would be very interesting to see whether the results from the approximation can be confirmed. \n\nSeveral questions:\n- What exactly is meant by the Entropy method?\nMy understanding that is that Entropy is the same as Least Confidence, since high entropy <-> low confidence.\n- Why is Entropy missing in Table 2?\n- Why is Q set to 500? This seems to be too small. It would be interesting to learning curve graphs with larger Q.", "summary_of_the_paper": "This work investigates Active Learning (AL) for fine-tuning large pre-trained transformer models (BERT, RoBERT) with applications to several NLP tasks.\nThey observe that depending on the task/dataset, AL actually does not improve over a random baseline.\nTheir ablation tests suggests that the instability of re-training the large models might be amplified by instances selected by AL. As a remedy, they suggest to train several times with different initializations, and then selecting the best model on the development data.", "summary_of_the_review": "For an experimental paper, it is interesting, and might point to some new directions for AL.\nHowever, more experiments to clarify some points (see above) should be added.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NLDANw4h8d", "reviewer": "Reviewer_DPi3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe topic investigated in this work is interesting and would be useful for further works since pre-trained models have been shown effective at learning with small amounts of annotations and AL is also a method with similar spirit.\n\nThis paper provides a good and solid empirical analysis on this topic with various different models, datasets and AL methods.\n\nThere are several detailed ablation analyses which I find interesting, such as the influence of question format and adversarial filtering on the datasets.\n\nWeakness:\n\nOne of my major concerns is on the point of training instability. I feel that this might not be the problem of AL, but at least some part of the problem is that we need a more robust way to train the model? Surely the random baseline is also treated in similar ways, so the comparisons are fair. Especially, fine-tuning might be sensitive to hyper-parameters, and it seems that there are no specific tunings? Surely I understand the cost of doing this, but I think it would be nice to select some of the datasets to carry out some hyper-parameter tuning.\n\nAs discussed in Section 3.4.4, annotation batch size could be important. I think there should be more experiments on this, for example, with larger sizes rather than smaller.\n\nOne specific question that I have but might not be fully answered in this work is that: why are the examples selected by AL more likely to cause the training instability? Some experiment results show that outliers might not tell the full story. Then why does this happen, is it related with instance diversity or even simpler label diversity (for example, maybe in some iterations, AL only chooses instances with the same label)? I'm quite interested to see more analysis on this.\n\nOther comments:\n\nFor the hyper-parameter issue above, what are the differences of multiple runs in the multi-training (in Section 3.3), only changing seed or changing hyper-parameters? If only seed, I suggest doing similar experiments with some degree of tuning of hyper-parameters.\n\nIt would be nice to have results where all the datasets are available as in standard supervised settings as upper-bounds.\n\nIt would be interesting to explore tasks beyond simple classification.\n", "summary_of_the_paper": "This paper investigates the usefulness of Active Learning (AL) on a variety of NLP tasks with pre-trained models. The authors show that with pre-trained models, AL can sometimes offer inconsistent and even poorer results than random baselines. In addition to outliers, they show that training instability is a major issue. There are also various further ablation studies on this issue.\n", "summary_of_the_review": "Generally I find that this work provides a reasonable study on the usefulness of AL with pre-trained models, there are lots of experiments and there are certain interesting findings. Nevertheless, I still have concerns with the training setup and there are still some analyses that I would like to see.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel explanation for the ineffectiveness of active learning (AL), namely, that AL will select unlearnable collective outliers. 

Reviewers generally find the finding is interesting, but the paper lacks in-depth analyses. There're additional concerns on the experimental setups.
Decision: Reject