Pretrained models are active learners

1 INTRODUCTION . Modern pretrained models can be adapted to new tasks with remarkably little data , enabling downstream applications for tasks with only tens or hundreds of examples ( Brown et al. , 2020 ; Radford et al. , 2021 ) . However , an important but neglected challenge that is especially salient in few-shot settings is task ambiguity , when the desired behavior is not uniquely specified by the provided examples . Task ambiguity can manifest in different ways : For example , the class-relevant features of an input ( e.g. , the shape of an object in Figure 1 ) may be spuriously correlated with other features predictive of the training labels ( e.g. , the color of the object ) , making the desired task unclear . In addition , task ambiguity may arise from an underdiverse training set , causing models to be unsure of the desired behavior for minority groups or during distribution shifts that occur at test-time . We consider whether the task ambiguity problem can be addressed through active learning , where models select informative examples for users to label . In principle , active learning allows models themselves to assist in resolving task ambiguity by identifying examples whose labels would be informative ; for example , in fig . 1 , asking for the label of the blue square helps determine that the desired task is to predict the object shape not the object color . Since it may be challenging in general for users to intuit possible sources of task ambiguity , much less how to address them with example selection , an automated active learning approach may be desirable . In traditional settings with smaller , unpretrained models , several challenges often prevent active learning from seeing success in practice , including label noise , unlearnable examples , and a lack of generalizability across models and tasks ( Lowell et al. , 2019 ; Karamcheti et al. , 2021 ) . However , using pretrained models in few-shot or low-data settings has several key differences from traditional ML pipelines which may make them particularly well-suited for active learning . First , the effect of an individual data point ( and the difference between an informative vs. an uninformative one ) is magnified , compared to the thousands or millions of examples typically acquired through standard active learning pipelines . Second , pretrained models excel at learning high-level representations of inputs , which better surface relevant features and may encourage active learning to select examples that disambiguate between these high-level features ( e.g . shape and color ) . We consider the use of active learning on a range of spuriously correlated , imbalanced , or domain shifted datasets where task ambiguity is salient . We compare an active learning approach with a random-sampling baseline , and compare the difference in performance with and without the use of pretrained models . Our contributions are : 1 . Identifying and motivating task ambiguity as a unified concern across a range of real-world datasets , and an especially salient problem for low-data settings . 2 . Proposing active learning as a potential solution , along with a simple yet practical recipe that does not require adjustment for new datasets or tuning on validation data . 3 . Showing that active learning can enable large gains ( up to 6× reduction in data points , +12 % absolute gain for the same labeling budget ) and presenting scaling trends demonstrating that this is an emergent property of pretraining . 2 METHOD . We study the pool-based active learning setup common in the literature ( Settles , 2009 ) , where we have a ( possibly pretrained ) modelM , a small seed set of training data S = { ( xi , yi ) } , and a larger pool of unlabeled data P = { xi } . The active learning procedure proceeds as follows : first , finetune M on S until convergence ; then , select points xi fromP that are deemed most informative according to an acquisition function a ( x ; M ) , obtaining the corresponding labels yi , until some budget k of data points is exhausted . This newly labeled batch B = { ( xi , yi ) } is then added to the existing data S , and the model is retrained on S for the next acquisition step . This process is repeated for a fixed number of acquisition steps . For our acquisition function , we adopt the classic uncertainty sampling approach to active learning , in particular the least confidence heuristic , where we acquire points for which our model is least confident in its predicted label ( Settles , 2009 ) . Specifically , treating the outputs of the modelM as a probability distribution1 over possible labels p ( y | x ; M ) , we define the acquisition function to be a ( x ; M ) = −max i p ( yi | x ; M ) ( 1 ) Although we could use more complex measures such as the entropy over labels , the least confidence heuristic has shown to be simple and effective in a variety of settings ( Settles , 2009 ; Hendrycks & Gimpel , 2017 ; Mussmann et al. , 2020 ) , and we similarly find good results here ( we refer to least confidence sampling as “ uncertainty sampling ” except in Section 4.3 where we explore other uncertainty-based acquisition functions ) . On top of this standard active learning pipeline , we propose the following change to improve the practical applicability of pretrained models in data-scarce settings : 1While in general there is no guarantee that this probability distribution will be well-calibrated , recent work has found that pretraining improves model calibration across a variety of settings , including on out-of-domain data ( Desai & Durrett , 2020 ; Hendrycks et al. , 2019 ) . Removing the need for a separate validation set The active learning cycle begins by finetuning the pretrained model on the seed set . If the size of the seed set is large enough , the seed set may be partitioned into a training set and a validation set , and early stopping may be performed on the validation set . However , in few-shot settings , labeling costs may be high , and the seed set may be too small to meaningfully partition . This has led to a variety of ad-hoc solutions , e.g . finetuning for an arbitrary constant number of gradient steps depending on the task ( Kolesnikov et al. , 2020 ) . Instead of an arbitrary fixed number of finetuning steps , we propose an alternative method to terminate finetuning in the absence of a validation set . Specifically , we found that a simple but effective heuristic was to stop finetuning when the training loss decreases to 0.1 % of the original training loss at the start of finetuning . In our experiments , this heuristic performed as well as early stopping on an actual validation set ( see Appendix D for more details ) . By using a standardized recipe across tasks and removing the need for a separate validation set , our active learning pipeline is more robust to the real-world difficulties of deploying active learning in data-scarce settings where use of a validation set is impractical ( Lowell et al. , 2019 ; Perez et al. , 2021 ) , although further work is needed to capture the full extent of this recipe ’ s generalizability . 3 DATASETS . We consider a variety of datasets where task ambiguity is especially likely when only a few examples are provided , especially due to spurious correlations , latent minority classes , and domain shifts . These datasets provide an empirical testbed for the ability of pretrained models to choose disambiguating examples using active learning . 3.1 DISTINGUISHING CAUSAL FROM SPURIOUS FEATURES . Spurious correlations arise when multiple features are predictive of the label in a training dataset , yet it is ambiguous which ones are causally linked to the task label ( Geirhos et al. , 2020 ) . We consider two such datasets , and see whether active learning can choose the disambiguating examples where the spurious features are not copresent with the causal features : Waterbirds The Waterbirds dataset ( Sagawa et al. , 2019 ) consists of photographs of landbirds or waterbirds digitally edited onto land or water backgrounds . The task is to classify whether the bird is a landbird or a waterbird . In the train set , 77 % of the pictures feature landbirds and 23 % waterbirds . 95 % of both landbirds and waterbirds appear on land and water backgrounds , respectively . In the validation and test sets , this percentage is decreased to 50 % , instead . Thus , the image background is a spurious feature the model may come to rely on when making the prediction . Treeperson As the Waterbirds dataset was synthetically generated , we also consider a dataset where we perform classification over real , unedited images with spuriously correlated objects . We use the object annotations in Visual Genome ( Krishna et al. , 2016 ) to create a new dataset of 8,638 images called Treeperson , for which the task is to predict whether a person is in a given image . While 50 % of the images contain a person in this dataset , each image also contains either a tree or a building , and the presence of these objects is spuriously correlated with the presence of people . At train time , 90 % of training images with people contain a building , while 90 % of training images without people contain a tree . Thus , a model may be incentivized to form representations that classify according to the presence of trees and buildings , rather than the presence of the actual causal variable of interest ( people ) . These values are changed to 50 % at test time , removing this correlation to evaluate how well the model learned the actual feature of interest . For more details on this dataset , see Appendix C . 3.2 MEASURING ROBUSTNESS TO DISTRIBUTION SHIFT . Distribution shifts occur when algorithms are evaluated on different data distributions than the ones they were trained on . Examples include changing the location or time of day that photos were taken , or changing the topic or author of a particular textual source . These shifts can reduce performance , and we consider whether active learning can help choose diverse , informative examples that clarify how the model should behave over a range of natural distribution shifts . iWildCam2020-WILDS This dataset considers the task of species classification from a database of photos taken from wildlife camera traps ( Beery et al. , 2020 ; Koh et al. , 2021 ) . The dataset is unbalanced , with most images containing no animal , and the distribution of camera locations and species changes between the in domain ( ID ) and out-of-domain ( OOD ) subsets . Amazon-WILDS This dataset considers the task of predicting the number of stars associated with the text of a given Amazon review ( Ni et al. , 2019 ; Koh et al. , 2021 ) . The reviewers are different in the training set versus the test set , and the task is to perform as well as possible on this set of new reviewers . In addition to number of stars , we also consider model performance stratified by different product types , which highlights minority subgroups whose categorization is not visible to the model . 4 EXPERIMENTS . 4.1 MODELS AND TRAINING . Vision For computer vision datasets , we finetune BiT ( Kolesnikov et al. , 2020 ) , a recentlyproposed family of vision models which have achieved state-of-the-art performance on several vision tasks . We primarily consider the BiT-M-R50x1 model , pretrained on ImageNet-21k ( Deng et al. , 2009 ) . To explore the effectiveness of larger architectures and pretraining sources , in Section 5.2 we also consider performance achieved by the same-size BiT-S-R50x1 , trained on ImageNet1k , and the deeper BiT-M-R101x1 model , also trained on ImageNet-21k . These models have been shown to have emergent few-shot learning abilities , where strong classifiers for new tasks can be obtained by simply finetuning on tens or hundreds of examples with typical gradient descent techniques ( rather than meta-learning techniques , for example ) . Text For the text dataset ( Amazon ) , we use RoBERTa-Large ( Liu et al. , 2019 ) , another pretrained model with similar properties as BiT , and a representative of the BERT ( Devlin et al. , 2019 ) family of models which together have obtained state-of-the-art scores on modern NLP benchmarks ( Wang et al. , 2018 ) . Other details , including hyperparameters and seed set/acquisition sizes are deferred to Appendix B . Random acquisition baseline As a running baseline , we compare to the same model finetuned with a random acquisition function ( equivalent to not doing active learning ) . That is , a ( x ; M ) = rand ( 0 , 1 ) , so we simply sample a random batch of data from the pool at each acquisition step . Comparison with unpretrained models To examine whether effective active learning is an emergent property of the pretraining process , we also compare to the performance observed when applying active learning to a randomly initialized , instead of pretrained , BiT-M-R50x1 .

Summary: The authors describe interesting empirical observations regarding using uncertainty sampling to select examples to fine-tune models that use pretrained embeddings and provide some hypotheses regarding the reasons for these performance improvements. Specifically, (1) from a methodological perspective, they propose using uncertainty sampling (i.e., least confident selection) to select examples for fine-tuning image/NLP pretrained models and (2) from an empirical perspective, they use Waterbirds/Treeperson/iWildCam2020-WILDS for image classification and Amazon-WILDS for review star prediction based on text and compare with random sampling — noting that these are settings where there is known covariate shift between train/test with semantic meaning to induce interpretable spurious associations (e.g., background in images). The proposed method works overall, especially on the image datasets, and they also dig into the types of examples selected — noting that they align with expected ‘difficult’ examples (depending on the setting).