Limitations of Active Learning With Deep Transformer Language Models

1 INTRODUCTION . Deep learning models typically need large datasets of hand-labeled examples to perform well , but acquiring the labels can be expensive , often requiring human annotators to manually provide the label for every example . Active learning ( AL ) is a technique that aims to reduce this labeling cost by selecting only the most useful examples to receive labels ( Settles , 2009 ) . AL has been shown to reduce labeling cost on a variety of NLP tasks in previous work ( Shen et al. , 2018 ; Siddhant & Lipton , 2018 ) . However , these successes predate the rise of large pretrained Transformer language models ( LMs ) as a foundation for NLP . Whether AL ’ s advantages extend to this setting is far less studied . Previous work has shown that AL consistently improves text classifiers that use pretrained Transformers ( Lu & MacNamee , 2020 ; Ein-Dor et al. , 2020 ) , but those studies use a narrow range of model sizes and do not consider the most challenging semantic tasks such as question answering ( QA ) where pretrained LMs have made great strides in recent years . By contrast , Karamcheti et al . ( 2021 ) recently showed that AL is ineffective for large pretrained transformers in visual QA , and concluded that harmful collective outliers negatively impact AL and that heuristically pruning them improves the benefit of AL . However , whether these findings hold outside of visual QA is unknown . In this paper , we present an expanded study of AL applied to fine-tuning large pretrained LMs . We investigate four AL approaches across ten datasets spanning text classification and recent multiplechoice question answering tasks . We find that AL fails to improve performance consistently , and often lowers performance significantly below that of a random-selection baseline . Further , our analysis suggests that in some cases , the failures have a different explanation than that offered by prior work . Specifically , AL sometimes select examples that are informative , but add instability to the training process , hurting average performance . As evidence , we show that a simple approach to reduce instability—training the model multiple times and selecting the best-performing on a held-out set , without any special treatment for collective outliers—improves the performance of AL by similar amount on average as the pruning in Karamcheti et al . ( 2021 ) . Further , we find that the effectiveness of the pruning technique from Karamcheti et al . ( 2021 ) is often dataset-specific , improving the relative performance of AL over random sampling on some datasets and worsening it on others . In most cases , pruning also reduces the absolute performance across all methods , suggesting that it prunes helpful examples in addition to outliers . To summarize , our contributions are : 1 . We present experiments across four AL methods , ten data sets , and three models showing that AL has inconsistent performance with recent pretrained transformer LMs , often underperforming random selection especially as larger models are used . 2 . We propose a novel explanation for some of AL ’ s limitations : that it sometimes selects examples that are helpful but cause instability in training . 3 . We conduct several ablation studies to rule out other factors and find that acquisition batch size and adversarial filtering do not meaningfully affect the performance of AL , while the amount of model pretraining has inconsistent effects between AL methods . We hope our findings will provide guidance to other researchers considering applying AL to their data , and spur further investigation into more effective AL methods for pretrained LMs along with optimization techniques that are more robust to the challenging examples AL selects . 2 METHODS . 2.1 TASK DEFINITION . Our active learning ( AL ) scenario is formulated as follows . We have a large pool of unlabeled data U and wish to select a subset L of size Q to label that maximizes model performance on a held-out set . Given a model M and an AL scoring function S ( x , M ) ( where x is an unlabeled input ) , we iteratively select a batch of |∆L| examples from U according to S , add these to L , and retrain the model before selecting again . When re-training , we re-initialize all parameters to the original pretrained model state , following previous work ( Ein-Dor et al. , 2020 ; Hu et al. , 2019 ) . The first batch is selected randomly . 2.2 AL METHODS AND MODELS . We experiment with the following AL scoring functions : • Random ( the baseline ) ; • Entropy1 of the model output distribution ; • BALD-MC , which is BALD ( Houlsby et al. , 2011 ) using Monte Carlo dropout ( Gal & Ghahramani , 2016 ) to compute uncertainty ; • BatchBALD-MC ( Kirsch et al. , 2019 ) , the batch variant of BALD ; • Coreset ( Sener & Savarese , 2018 ) , which selects examples that are distant from examples in L measured by distance between model output embeddings . We use the greedy variant of the algorithm and use the pre-trained model to pre-compute fixed output embeddings for the distance calculation.2 While many AL methods exist , we believe our choices form a representative set . Entropy is commonly used and was found to work well in the previous work on AL for Transformers ( Lu & MacNamee , 2020 ) . BALD has been found to outperform other methods in many cases , including NLP ( Gal et al. , 2017 ; Siddhant & Lipton , 2018 ) . Coreset is diversity-based ( rather than uncertainty-based like our other methods ) and has outperformed other methods ( including BALD ) for deep models on image classification tasks ( Sener & Savarese , 2018 ) . We note that BALD and Entropy are originally online algorithms ( selecting one example at a time ) but are used in batch mode here . This is not theoretically sound but is commonplace and often effective ( Kasai et al. , 2019 ; Gal et al. , 2017 ) , so we include it for comparison . BatchBALD adds a diversity adjustment to BALD to account for this issue . We test three pretrained Transformers , which are state-of-the-art models with different pretraining methods and sizes : BERT-base ( 110M parameters ) ( Devlin et al. , 2019 ) , RoBERTa-large ( 340M ) ( Liu et al. , 2019 ) , and RoBERTa-base ( 110M ) . 1A similar method is least-confidence ( LC ) . We tried LC on most of our datasets and found it to be almost identical to entropy , so we omit it here . 2In preliminary experiments we tried training the model and updating the embeddings during acquisition , but this did not improve results and was more computationally expensive . 2.3 TASKS AND DATASETS . For datasets , we focus broadly on two tasks . The first is text classification , where AL has been effective with a limited class of Transformer models in previous work Lu & MacNamee ( 2020 ) ; Ein-Dor et al . ( 2020 ) . This task allows us to verify our implementation against previous work , and to evaluate whether AL ’ s effectiveness on it extends to a broader model class . The second is multiplechoice commonsense reasoning , a challenging NLP task where AL has not been evaluated , to our knowledge . The text classification datasets are AGN ( Zhang et al. , 2015 ) , DBpedia ( Zhang et al. , 2015 ) , Movie review sentence polarity ( MRP ) ( Pang & Lee , 2005 ) , and Movie review subjectivity ( MRS ) ( Pang & Lee , 2004 ) . The multiple-choice datasets are aNLI ( Bhagavatula et al. , 2020 ) , CODAH ( Chen et al. , 2019 ) , CommonsenseQA ( CSQA ) ( Talmor et al. , 2019 ) , HellaSWAG ( Zellers et al. , 2019 ) , PIQA ( Bisk et al. , 2020 ) , and SWAG ( Zellers et al. , 2018 ) . In the tables , “ AGN-SB ” and “ DBpedia-SB ” are AGN and DBpedia using only two classes ( to create a binary classification task ) and subsampled to have the same number of examples as reported by Lu & MacNamee ( 2020 ) . 2.4 EXPERIMENTAL SETTINGS . We test on an eval set after each acquisition , which produces a learning curve of accuracy vs. data size . We report the area under the curve ( AUC ) as a measure of how effective the data selection strategy is . We set Q = 500 and |∆L| = 25 except where otherwise specified . The deep learning models were all trained by doing 1000 steps with the Adam optimizer and early stopping ( hyperparameter details in Appendix A ) , and results are averaged over 10 trials . 3 RESULTS . We start by presenting our primary results showing that active learning offers inconsistent and often poor performance on our tasks . For RoBERTa-base , shown in Table 1 , there are a few cases where AL outperforms the random-sampling baseline , but in many cases it gives no improvement ( and in fact , for 10 of the 44 results it does significantly worse ) . In the subsequent sections , we investigate several candidate explanations for the poor performance . 3.1 COLLECTIVE OUTLIERS . Recently , Karamcheti et al . ( 2021 ) found that “ collective outliers ” can sometimes explain poor AL performance . Collective outliers are groups of points that together represent an out-of-distribution set of data that the model performs poorly on , and AL is prone to selecting these examples due to their difficulty . To investigate the extent to which this effect explains our results , we applied the same pruning strategy described in that work . This consists of creating a dataset map ( Swayamdipta et al. , 2020 ) which is made by first training the model for several epochs ( 10 for our experiments ) on all examples in the pool U and calculating the mean ( “ confidence ” ) and variance ( “ variability ” ) of the probability the model assigned to the correct label for that example during training . We then sort the examples by the product of confidence and variability and prune the lowest 50 % 3 . Of course , this could not be done in a real-world scenario because it requires using the labels for the entire pool , but it is a useful way to gain insight into the behavior of AL . Results are in Table 2 , and show mixed results on the relative performance of AL vs random selection . The best AL method in our original experiments ( BatchBALD-MC , Table 1 ) wins over random on the same number of data sets with pruning as without ( 7 ) , and pruning only improves AL ’ s relative gap with random on 6 of the 11 datasets . Other AL methods show qualitatively similar behavior . We also note that the change in absolute performance across the different datasets , with many datasets getting worse and only four ( AGN-SB-c , CODAH , CSQA , and DBpedia-c ) showing consistent improvements . This result suggests that collective outliers may not fully explain the poor performance of AL on some datasets , or at least that pruning based on confidence and variability is not always sufficient to isolate them . 3.2 MODEL SIZE AND INSTABILITY . 3An ablation with 25 % pruning can be found in Appendix C Compared to most previous studies where AL has been effective for language tasks , the models we use have many more trainable parameters . To test whether model size may be a contributor to AL ’ s poor performance , we evaluate the much larger RoBERTa-large ( 3x more parameters than RoBERTa-base ) in Table 3 , but limit this experiment to a subset of datasets due to the high computational cost of the model . The results show that AL is substantially less effective for the larger model . In fact , for BatchBALD and Entropy , RoBERTa-large shows more negative impact from AL than RoBERTa-base across all five datasets we evaluate in this experiment . The average difference between BatchBALD and Random is -1.7 for RoBERTa-large but +0.4 for RoBERTa-base ( for the same datasets ) . One reason for this is an increased frequency of degenerate training runs for RoBERTa-large when using AL , which have unusually low accuracy ( often no better than guessing ) and manifest in Figure 1 as points that dip sharply below the global trend of the learning curve and have high variance . For example , most results in the second half of the learning curve for SWAG are above 68 % accuracy , but with AL several runs randomly drop to anywhere from 25-67 % . Such degenerate cases with large transformers are well-known ( Devlin et al. , 2019 ) . But the substantial increase in failed training runs when using AL leads us to believe that AL methods select sets of examples that amplify the instability of the optimization process , which we define here as the variance in test set accuracy observed when training multiple models ( with different random seeds ) on the same set of data . The increased instability means that the selected examples are not necessarily harmful as the collective outlier theory suggests ( because not all training runs fail ) , but instead are potentially useful examples that are hard to learn from effectively . Correcting for these failed training runs is hard for two reasons : ( 1 ) they are frequent enough that it would be computationally prohibitive for us to re-run all of them with different random seeds until getting a success , as in previous work ( Devlin et al. , 2019 ; Porada et al. , 2019 ) , and ( 2 ) the model does not always degrade to random-label performance , meaning a threshold would need to be chosen for what consitutes a “ failure ” , and this choice may affect the conclusions . To roughly approximate how well the model could do if it did not have these optimization failures , we consider the area under the convex hull of the learning curves instead of including all points , and display the results in Table 4 . This substantially improves AL relative to random and suggests that the increased frequency of failed training runs is the main culprit for the poor performance of AL . Because the learning curve of accuracy vs dataset size is generally convex , our convex hull approximation is an underestimate of what we would expect if we actually achieved the best optimization that the model could reasonably achieve with the selected data . To a lesser extent , we find evidence of AL-induced instability on RoBERTa-base as well . Figure 2 shows some of the learning curves from the Table 1 results , and while they are much less noisy than the RoBERTa-large table , it still qualitatively appears that AL has more noise than random on the datasets where AL loses . Quantitatively , we compute the average difference between the convex hull AUC and the normal AUC , which serves to estimate the level of instability in the absence of repeated trials on the same data . On the datasets where AL loses , BatchBALD ’ s average difference ( 2.04 ) is larger than Random ’ s ( 1.80 ) , whereas on datasets where AL wins the average differences are equal ( 0.77 ) . That is , AL shows greater instability on the losses than on the wins , suggesting that instability may play a role in the failures .

Summary: This work evaluates active learning methods for pre-trained Transformer models, and diagnoses their inconsistent performance as a consequence of training examples which exacerbate variability in test-time accuracy. They propose re-training the model repeatedly to circumvent bad outcomes due to variability, and show this works particularly well on larger pre-trained models. The authors conduct a wide range of experiments, over many datasets, and also a series of ablation studies to reinforce their claim that these active learning methods are ineffective for pre-trained Transformers.