Pretrained models are active learners

1 INTRODUCTION . Modern pretrained models can be adapted to new tasks with remarkably little data , enabling downstream applications for tasks with only tens or hundreds of examples ( Brown et al. , 2020 ; Radford et al. , 2021 ) . However , an important but neglected challenge that is especially salient in few-shot settings is task ambiguity , when the desired behavior is not uniquely specified by the provided examples . Task ambiguity can manifest in different ways : For example , the class-relevant features of an input ( e.g. , the shape of an object in Figure 1 ) may be spuriously correlated with other features predictive of the training labels ( e.g. , the color of the object ) , making the desired task unclear . In addition , task ambiguity may arise from an underdiverse training set , causing models to be unsure of the desired behavior for minority groups or during distribution shifts that occur at test-time . We consider whether the task ambiguity problem can be addressed through active learning , where models select informative examples for users to label . In principle , active learning allows models themselves to assist in resolving task ambiguity by identifying examples whose labels would be informative ; for example , in fig . 1 , asking for the label of the blue square helps determine that the desired task is to predict the object shape not the object color . Since it may be challenging in general for users to intuit possible sources of task ambiguity , much less how to address them with example selection , an automated active learning approach may be desirable . In traditional settings with smaller , unpretrained models , several challenges often prevent active learning from seeing success in practice , including label noise , unlearnable examples , and a lack of generalizability across models and tasks ( Lowell et al. , 2019 ; Karamcheti et al. , 2021 ) . However , using pretrained models in few-shot or low-data settings has several key differences from traditional ML pipelines which may make them particularly well-suited for active learning . First , the effect of an individual data point ( and the difference between an informative vs. an uninformative one ) is magnified , compared to the thousands or millions of examples typically acquired through standard active learning pipelines . Second , pretrained models excel at learning high-level representations of inputs , which better surface relevant features and may encourage active learning to select examples that disambiguate between these high-level features ( e.g . shape and color ) . We consider the use of active learning on a range of spuriously correlated , imbalanced , or domain shifted datasets where task ambiguity is salient . We compare an active learning approach with a random-sampling baseline , and compare the difference in performance with and without the use of pretrained models . Our contributions are : 1 . Identifying and motivating task ambiguity as a unified concern across a range of real-world datasets , and an especially salient problem for low-data settings . 2 . Proposing active learning as a potential solution , along with a simple yet practical recipe that does not require adjustment for new datasets or tuning on validation data . 3 . Showing that active learning can enable large gains ( up to 6× reduction in data points , +12 % absolute gain for the same labeling budget ) and presenting scaling trends demonstrating that this is an emergent property of pretraining . 2 METHOD . We study the pool-based active learning setup common in the literature ( Settles , 2009 ) , where we have a ( possibly pretrained ) modelM , a small seed set of training data S = { ( xi , yi ) } , and a larger pool of unlabeled data P = { xi } . The active learning procedure proceeds as follows : first , finetune M on S until convergence ; then , select points xi fromP that are deemed most informative according to an acquisition function a ( x ; M ) , obtaining the corresponding labels yi , until some budget k of data points is exhausted . This newly labeled batch B = { ( xi , yi ) } is then added to the existing data S , and the model is retrained on S for the next acquisition step . This process is repeated for a fixed number of acquisition steps . For our acquisition function , we adopt the classic uncertainty sampling approach to active learning , in particular the least confidence heuristic , where we acquire points for which our model is least confident in its predicted label ( Settles , 2009 ) . Specifically , treating the outputs of the modelM as a probability distribution1 over possible labels p ( y | x ; M ) , we define the acquisition function to be a ( x ; M ) = −max i p ( yi | x ; M ) ( 1 ) Although we could use more complex measures such as the entropy over labels , the least confidence heuristic has shown to be simple and effective in a variety of settings ( Settles , 2009 ; Hendrycks & Gimpel , 2017 ; Mussmann et al. , 2020 ) , and we similarly find good results here ( we refer to least confidence sampling as “ uncertainty sampling ” except in Section 4.3 where we explore other uncertainty-based acquisition functions ) . On top of this standard active learning pipeline , we propose the following change to improve the practical applicability of pretrained models in data-scarce settings : 1While in general there is no guarantee that this probability distribution will be well-calibrated , recent work has found that pretraining improves model calibration across a variety of settings , including on out-of-domain data ( Desai & Durrett , 2020 ; Hendrycks et al. , 2019 ) . Removing the need for a separate validation set The active learning cycle begins by finetuning the pretrained model on the seed set . If the size of the seed set is large enough , the seed set may be partitioned into a training set and a validation set , and early stopping may be performed on the validation set . However , in few-shot settings , labeling costs may be high , and the seed set may be too small to meaningfully partition . This has led to a variety of ad-hoc solutions , e.g . finetuning for an arbitrary constant number of gradient steps depending on the task ( Kolesnikov et al. , 2020 ) . Instead of an arbitrary fixed number of finetuning steps , we propose an alternative method to terminate finetuning in the absence of a validation set . Specifically , we found that a simple but effective heuristic was to stop finetuning when the training loss decreases to 0.1 % of the original training loss at the start of finetuning . In our experiments , this heuristic performed as well as early stopping on an actual validation set ( see Appendix D for more details ) . By using a standardized recipe across tasks and removing the need for a separate validation set , our active learning pipeline is more robust to the real-world difficulties of deploying active learning in data-scarce settings where use of a validation set is impractical ( Lowell et al. , 2019 ; Perez et al. , 2021 ) , although further work is needed to capture the full extent of this recipe ’ s generalizability . 3 DATASETS . We consider a variety of datasets where task ambiguity is especially likely when only a few examples are provided , especially due to spurious correlations , latent minority classes , and domain shifts . These datasets provide an empirical testbed for the ability of pretrained models to choose disambiguating examples using active learning . 3.1 DISTINGUISHING CAUSAL FROM SPURIOUS FEATURES . Spurious correlations arise when multiple features are predictive of the label in a training dataset , yet it is ambiguous which ones are causally linked to the task label ( Geirhos et al. , 2020 ) . We consider two such datasets , and see whether active learning can choose the disambiguating examples where the spurious features are not copresent with the causal features : Waterbirds The Waterbirds dataset ( Sagawa et al. , 2019 ) consists of photographs of landbirds or waterbirds digitally edited onto land or water backgrounds . The task is to classify whether the bird is a landbird or a waterbird . In the train set , 77 % of the pictures feature landbirds and 23 % waterbirds . 95 % of both landbirds and waterbirds appear on land and water backgrounds , respectively . In the validation and test sets , this percentage is decreased to 50 % , instead . Thus , the image background is a spurious feature the model may come to rely on when making the prediction . Treeperson As the Waterbirds dataset was synthetically generated , we also consider a dataset where we perform classification over real , unedited images with spuriously correlated objects . We use the object annotations in Visual Genome ( Krishna et al. , 2016 ) to create a new dataset of 8,638 images called Treeperson , for which the task is to predict whether a person is in a given image . While 50 % of the images contain a person in this dataset , each image also contains either a tree or a building , and the presence of these objects is spuriously correlated with the presence of people . At train time , 90 % of training images with people contain a building , while 90 % of training images without people contain a tree . Thus , a model may be incentivized to form representations that classify according to the presence of trees and buildings , rather than the presence of the actual causal variable of interest ( people ) . These values are changed to 50 % at test time , removing this correlation to evaluate how well the model learned the actual feature of interest . For more details on this dataset , see Appendix C . 3.2 MEASURING ROBUSTNESS TO DISTRIBUTION SHIFT . Distribution shifts occur when algorithms are evaluated on different data distributions than the ones they were trained on . Examples include changing the location or time of day that photos were taken , or changing the topic or author of a particular textual source . These shifts can reduce performance , and we consider whether active learning can help choose diverse , informative examples that clarify how the model should behave over a range of natural distribution shifts . iWildCam2020-WILDS This dataset considers the task of species classification from a database of photos taken from wildlife camera traps ( Beery et al. , 2020 ; Koh et al. , 2021 ) . The dataset is unbalanced , with most images containing no animal , and the distribution of camera locations and species changes between the in domain ( ID ) and out-of-domain ( OOD ) subsets . Amazon-WILDS This dataset considers the task of predicting the number of stars associated with the text of a given Amazon review ( Ni et al. , 2019 ; Koh et al. , 2021 ) . The reviewers are different in the training set versus the test set , and the task is to perform as well as possible on this set of new reviewers . In addition to number of stars , we also consider model performance stratified by different product types , which highlights minority subgroups whose categorization is not visible to the model . 4 EXPERIMENTS . 4.1 MODELS AND TRAINING . Vision For computer vision datasets , we finetune BiT ( Kolesnikov et al. , 2020 ) , a recentlyproposed family of vision models which have achieved state-of-the-art performance on several vision tasks . We primarily consider the BiT-M-R50x1 model , pretrained on ImageNet-21k ( Deng et al. , 2009 ) . To explore the effectiveness of larger architectures and pretraining sources , in Section 5.2 we also consider performance achieved by the same-size BiT-S-R50x1 , trained on ImageNet1k , and the deeper BiT-M-R101x1 model , also trained on ImageNet-21k . These models have been shown to have emergent few-shot learning abilities , where strong classifiers for new tasks can be obtained by simply finetuning on tens or hundreds of examples with typical gradient descent techniques ( rather than meta-learning techniques , for example ) . Text For the text dataset ( Amazon ) , we use RoBERTa-Large ( Liu et al. , 2019 ) , another pretrained model with similar properties as BiT , and a representative of the BERT ( Devlin et al. , 2019 ) family of models which together have obtained state-of-the-art scores on modern NLP benchmarks ( Wang et al. , 2018 ) . Other details , including hyperparameters and seed set/acquisition sizes are deferred to Appendix B . Random acquisition baseline As a running baseline , we compare to the same model finetuned with a random acquisition function ( equivalent to not doing active learning ) . That is , a ( x ; M ) = rand ( 0 , 1 ) , so we simply sample a random batch of data from the pool at each acquisition step . Comparison with unpretrained models To examine whether effective active learning is an emergent property of the pretraining process , we also compare to the performance observed when applying active learning to a randomly initialized , instead of pretrained , BiT-M-R50x1 .

Summary: The authors set out to investigate if active learning is an emergent property of pre-training. That is if running active learning with pre-trained models gives better result than using the same models without pre-training. They run several experiments on different text and image datasets first showing that active learning performs better than random sampling on pre-trained models and secondly that pre-trained models perform better than un-pretrained ones for active learning.