Pretrained models are active learners
Authors: Alex Tamkin
Dat Nguyen
Salil Deshpande
Jesse Mu
Noah Goodman
Venue: ICLR 2022 Submitted (2022)
Keywords: pretraining
active learning
alignment
safety

Abstract: An important barrier to the safe deployment of machine learning systems is the risk of \emph{task ambiguity}, where multiple behaviors are consistent with the provided examples. We investigate whether pretrained models are better active learners, capable of asking for example labels that \textit{disambiguate} between the possible tasks a user may be trying to specify. Across a range of image and text datasets with spurious correlations, latent minority groups, or domain shifts, finetuning pretrained models with data acquired through simple uncertainty sampling achieves the same accuracy with \textbf{up to 6$\times$ fewer labels} compared to random sampling. Moreover, the examples chosen by these models are preferentially minority classes or informative examples where the spurious feature and class label are decorrelated. Notably, gains from active learning are not seen in unpretrained models, which do not select such examples, suggesting that the ability to actively learn is an emergent property of the pretraining process.

Under review as a conference paper at ICLR 2022
PRETRAINED MODELS ARE GOOD ACTIVE LEARNERS
Anonymous authors Paper under double-blind review
ABSTRACT
An important barrier the safe deployment of machine learning systems is the risk of task ambiguity, where multiple behaviors are consistent with the provided examples. We investigate whether pretrained models are better active learners, ca- pable of asking for example labels that disambiguate between the possible tasks a user may be trying to specify. Across range of image and text datasets with spu- rious correlations, latent minority groups, domain shifts, finetuning pretrained models with data acquired through simple uncertainty sampling achieves the same accuracy with up to 6x fewer labels compared random sampling. Moreover, the examples chosen by these models are preferentially minority classes or in- formative examples where the spurious feature and class label are decorrelated. Notably, gains from active learning are not seen in unpretrained models, which do not select such examples, suggesting that the ability to actively learn is emer- gent property of the pretraining process.
AMBIGUOUS TASK (CoLOR SHAPE?)
ACTIVE LEARNING
DISAM BIGUATED TASK (SHAPE!)
(CHOosE EXAMPLE)
50%
90% ACC
ACC
CONFIDENT
CONFIDENT
UNSURE
Figure 1: Active learning can resolve task ambiguity, which is especially salient in few-shot settings. Here, the provided training data leaves the model unsure of the task: is to predict the shape or the color of the object? Pretraining enables models to disentangle and weigh various competing features, making them good active learners that can choose disambiguating examples (e.g. the blue square), resolving this task ambiguity.
INTRODUCTION
Modern pretrained models can be adapted to tasks with remarkably little data, enabling down- stream applications for tasks with only tens or hundreds of examples (Brown et al. 20200] Radford al. 2021). However, an important but neglected challenge that is especially salient in few-shot settings is task ambiguity, when the desired behavior is not uniquely specified by the provided ex- amples. Task ambiguity can manifest in different ways: For example, the class-relevant features of an input (c.g., the shape of an object in Figure 1p may be spuriously correlated with other features predictive the training labels (c.g., the color of the object), making the desired task unclear. In addition, task ambiguity may arise from underdiverse training set, causing models to be unsure of the desired behavior for minority groups or during distribution shifts that occur at test-time.
We consider whether the task ambiguity problem be addressed through active learning. where models select informative examples for users to label. In principle, active learning allows models themselves to assist in resolving task ambiguity by identifying examples whose labels would be

---

Under review as a conference paper at ICLR 2022
informative; for example, in fig. 1 asking for the label of the blue square helps determine that the desired task is to predict the object shape not the object color. Since it may be challenging in general for users to intuit possible sources of task ambiguity, much less how to address them with example selection, an automated active learning approach may be desirable.
In traditional settings with smaller, unpretrained models, several challenges often prevent active learning from seeing success in practice, including label noise, unlearnable examples, and a lack of generalizability across models and tasks (Lowell al. 2019 Karamcheti et al., 2021). However, using pretrained models in few-shot or low-data settings has several key differences from traditional ML pipelines which may make them particularly well-suited for active learning. First, the effect of an individual data point (and the difference between an informative vs. an uninformative one) is magnified, compared to the thousands or millions of examples typically acquired through standard active learning pipelines. Second, pretrained models excel at learning high-level representations of inputs, which better surface relevant features and may encourage active learning to select examples that disambiguate between these high-level features (e.g. shape and color).
We consider the use of active learning on a range of spuriously correlated, imbalanced, or domain shifted datasets where task ambiguity is salient. We compare an active learning approach with aundom-samplee baseline, and compare the difference in performance with and without the use of pretrained models. Our contributions are:
1. Identifying and motivating task ambiguity as a unified concern across a range of real-world datasets, and an especially salient problem for low-data settings.
2. Proposing active learning as a potential solution, along with a simple yet practical recipe that does not require adjustment for new datasets or tuning on validation data.
3. Showing that active learning can enable large gains (up to 6x reduction in data points, + +12% absolute gain for the same labeling budget) and presenting scaling trends demon- strating that this is an emergent property of pretraining.
2 METHOD
We study the pool-hased active learning setup common in the literature (Settles 2009), where we have a (possibly pretrained) model M, a small seed set of training data S = { }:)]. and larger pool of unlabeled data P = {.:}. The active learning procedure proceeds as follows: first, finetune M on S until convergence; then, select points z; from P that are deemed most informative according to an acquisition function a(x; [x; M). obtaining the corresponding labels y/. until some budget & of data points is exhausted. This newly labeled batch B {(I,, ys) yt)} is then added to the existing data S, and the model is retrained on S for the next acquisition step. This process is repeated for fixed number of acquisition steps.
For our acquisition function, we adopt the classic uncertainty sampling approach to active learning, in particular the least confidence heuristic, where we acquire points for which our model is least confident in its predicted label (Settles) 2009). Specifically, treating the outputs of the model M as a probability distribution over possible labels ply x; M), we define the acquisition function to be
a(x; MM) = max p(y M;
(1)
Although we could use more complex measures such as the entropy over labels, the least confidence
heuristic has shown to be simple and effective in variety of settings (Settles) 2009 Hendrycks & Gimpell 2017 Mussmann et al., 2020), and we similarly find good results here (we refer to
least confidence sampling as "uncertainty sampling" except in Section 4.3) where we explore other uncertainty-bact based acquisition functions).
On top of this standard active learning pipeline, we propose the following change to improve the practical applicability of pretrained models in data-scarce settings:
'Whie in general there is no guarantee that this probability distribution will be well-calibrated, recent work has found that pretraining improves model calibration across variety of settings, including out-of-domain data Desai & Durren 2020 Hendrycks ct al. 2019)
2

---

Under review as a conference paper at ICLR 2022
Removing the need for separate validation set The active learning eycle begins by finetuning the pretrained model on the seed set. If the size of the seed set is large enough, the seed set may be partitioned into training set and validation set, and early stopping may be performed on the validation set. However, in few-shot settings, labeling costs may be high, and the seed set may be small meaningfully partition. This has led to variety of ad-hoc solutions, e.g. finetuning for an arbitrary constant number of gradient steps depending on the task (Kolesnikov et al. (020).
Instead of arbitrary fixed number of finetuning steps, we propose an alternative method termi- nate finetuning in the absence of a validation set. Specifically, we found that a simple but effective heuristic was to stop finetuning when the training loss decreases 0.1% of the original training loss at the start of finetuning. In our experiments, this heuristic performed as well as early stopping on an actual validation set (see (see|Appent Appendix D more details). By using a standardized recipe across tasks and removing the need for separate validation set, our active learning pipeline is more robust to the real-world difficulties of deploying active learning in data-scarce settings where use of a validation
set is impractical (Lowell et al. 2019 Perez al. 2021 although further work is needed to capture
the full extent this recipe' generalizability.
3 DATASETS
We consider a variety of datasets where task ambiguity is especially likely when only a few ex- amples are provided, especially due to spurious correlations, latent minority classes, and domain shifts. These datasets provide an empirical testbed for the ability of pretrained models to choose disambiguating examples using active learning.
3.1 DISTINGUISHING CAUSAL FROM SPURIOUS FEATURES
Spurious correlations arise when multiple features are predictive of the label in a training dataset, yet it is ambiguous which ones are causally linked to the task label (Geirhos et al.) (202). We consider
two such datasets, and see whether active learning can choose the disambiguating examples where the spurious features are not copresent with the causal features:
Waterbirds The Waterbirds dataset (Sagawa et al., 2019) consists of photographs of landbirds or waterbirds digitally edited onto land water backgrounds. The task is classify whether the bird is a landbird or waterbird. In the train set, 77% of the pictures feature landbirds and 23% waterbirds. 95% of both landbirds and waterbirds appear on land and water backgrounds, respectively. In the validation and test sets, this percentage is decreased to 50%, instead. Thus, the image background a spurious feature the model may come to rely on when making the prediction.
Treeperson As the Waterbirds dataset was synthetically generated, we also consider dataset where we perform classification over real, unedited images with spuriously correlated objects. We use the object annotations in Visual Genome (Krishna al. 2016) to create a new dataset of 8,638 images called Treeperson, for which the task is to predict whether person is in a given image. While 50% of the images contain person in this dataset, each image also contains either tree or a building, and the presence of these objects is spuriously correlated with the presence of people. At train time, 90% of training images with people contain building, while 90% of training images without people contain a tree. Thus, model may be incentivized to form representations that classify according to the presence of trees and buildings, rather than the presence of the actual causal variable of interest (people). These values changed to 50% at test time, removing this correlation to evaluate how well the model learned the actual feature of interest. For more details on this dataset, Appendix C
3.2 MEASURING ROBUSTNESS TO DISTRIBUTION SHIFT
Distribution shifts occur when algorithms are evaluated on different data distributions than the ones they were trained on. Examples include changing the location or time of day that photos were taken, or changing the topic or author of particular textual source. These shifts can reduce performance, and we consider whether active learning can help choose diverse, informative examples that clarify how the model should behave over range of natural distribution shifts.
3

---

Under review as a conference paper at ICLR 2022
This dataset considers the task of species classification from database of photos taken from wildlife camera traps (Beery et al. 2020; Koh et al. 2021). The dataset is unbalanced, with most images containing animal, and the distribution of camera locations and species changes between the in domain (ID) and out-of-domain (OOD) subsets.
WWiddCam2020-WIIDSS
Amazon-WILDS This dataset considers the task of predicting the number of stars associated with the text of a given Amazon review (Ni et al. 2019: Koh et al., 2021). The reviewers are different in the training set versus the test set, and the task is perform well possible this set of new reviewers. In addition to number of stars, we also consider model performance stratified by different product types, which highlights minority subgroups whose categorization is not visible to the model.
4 EXPERIMENTS
4.1 MODELS AND TRAINING
Vision For computer vision datasets, we finetune BiT (Kolesnikov et al. 2020), recently- proposed family of vision models which have achieved state-of-the-art performance on several vi- sion tasks. We primarily consider the BiT-M-R50x1 model, pretrained on ImageNet-21K (Deng et al. 2009). To explore the effectiveness of larger architectures and pretraining sources, in Section 5.2 we also consider performance achieved by the same-size BiT-S-R50x1, trained on ImageNet- 1k, and the deeper BiT-M-RI01XI model, also trained on ImageNet-21k, These models have been shown to have emergent few-shot learning abilities, where strong classifiers for new tasks can be obtained by simply finetuning tens or hundreds of examples with typical gradient descent tech- niques (rather than meta-learning techniques, for example).
Text For the text dataset (Amazon), we use RoBERTa-Large (Liu et al. 2019). another pretrained
model with similar properties as BiT, and representative of the BERT (Devlin et al.) (219)) family
of models which together have obtained state-of-the-art scores modern NLP benchmarks (Wang jet al. 2018).
Other details, including hyperparameters and seed set/acquisition sizes are deferred to Appendix B
Random acquisition baseline As a running baseline, we compare the same model finetuned with a random acquisition function (equivalent to not doing active learning). That is, a(r; M) r0, 1), so we simply sample random batch of data from the pool each acquisition step.
Comparison with unpretrained models gent property of the pretraining process, we also compare the performance observed when apply- ing active learning to a randomly initialized, instead of pretrained, B-0xM-
To examine whether effective active learning is an emer-
4.2 ACCURACY PER ACQUISITION
For general measure of success, we plot the accuracy active learning versus random sampling on the validation datasets function of the number of samples acquired during training.
Waterbirds Waterbirds evaluated on a balanced dataset where the foreground and background not correlated. In this setting, uncertainty sampling achieves +12% improvement in average validation accuracy over random sampling (Figure 2ap. This comes primarily from a ++31% average increase across the landbird-on- water and waterbird-on-lane images (i.e. those without the spurious correlation; Figuree 26). Uncertainty sampling required 5.8x fewer labels than random sampling to achieve random sampling's final accuracy.
Amazon the Amazon dataset, we also see gains from active learning in the OOD setting, in- cluding +1 % average across reviewers, and +2.5% on the worst 10th percentile (Figure 2g)- This result suggests that our active learning recipe may be of use outside of BiT or computer vi- sion settings more broadly. The confidence not overlapping by the end of training is evidence that
4

---

Under review as a conference paper at ICLR 2022
Waterbirds
Waterbirds
Treeperson
Treeperson
0.8
0.85
0.7
0.50
0.68
0.6
0.75
Uncertainty
0.5
Uncertainty
0.66
0.45
0.4
Random
Random
0.65
0.62
Uncertainty
Uncertainty
100
200 Total Examples
100
200
300
100
200
300
Total Examples
Examples
Total Examples
(a) Overall accuracy
(b) Mismatched accuracy
(c) Overall accuracy
(d) Minority accuracy
0.6
iWildCam
0.35
iWildCam
Amazon
Amazon
Randem
0,40
Uncertainty
0.4
0.25
0.60
0.35
0.20
0.30
0.2
Random
0.15
0.55
Random
Random
Uncertainty
Uncertainty
Uncertainty
500
Examples
1000
1500
500
Total Examples
1000
1500
25
Total Examples
100
50 Total Examples
100
(e) Overall accuracy
(I) Minority accuracy
g Overall accuracy
(h) Worst 10th percentile
Figure 2: Uncertainty sampling outperforms random sampling on all datasets, especially on minority classes. Shaded regions represent 95% CIs (Gaussian approx.).
while the difference between uncertainty and random sampling is not large, it is statistically signif- icant. Uncertainty sampling required 1.3x fewer labels than random sampling achieve random sampling's s final accuracy.
iWildCam With the iWildCam dataset, uncertainty sampling achieved +8% improvement upon random sampling. Uncertainty sampling also required 1.9x fewer labels than random sampling to achieve random sampling's 's final accuracy (Figure 2g).
Treeperson In the Treeperson dataset, uncertainty sampling is +2% improved over random sam- pling by the end of training (Figure 2cp. Uncertainty sampling required 2.5x fewer labels than random sampling to achieve random sampling's final accuracy.
1.00
Randem
0.75
0.50
0.25
0.00
ID Average
ID
OOD Average Accuracies
QOD Class
OOD Other Classes
Class
Other Classes Validiation
Figure 3: All types of uncertainty sampling outperform random sampling on iWildCam.
4.3 ADDITIONAL ACTIVE LEARNING METHODS
We consider two additional active learning methods addition to least confidence sampling: 1) entropy sampling, which chooses the example that maximizes the entropy of the model's predic- tive distribution, and 2) margin sampling, which chooses the example with the smallest difference between the first and second most probable classes (Scheffer al.! 2001 Settles 2009). We run

---

Under review as a conference paper at ICLR 2022
Frequency Of Class
00.6
0.4
Random
Randiom Uncertaintyy
Uicertaiitty
Zrequen
00.4
0.2
0.2
0.0
Landbird
Landbird Water BG
Waterbird
Waterbird + Water BG
0.0
No People
No People
People
People Buildings
Land BG
Land BG
+ Trees
+ Buildings Image Class
+ Trees
Image Class
Waterbirds
(b) Treeperson
Figure 4: Uncertainty sampling identifies and upsamples disambiguating examples. For both Waterbirds and Treeperson, uncertainty sampling selectively acquires examples where the spurious and core features disagree. Y-axis: percentage oversampling of uncertainty over random sampling.
experiments with all methods the 182-class iWildCam datase perform random sampling (Figure 3).
All methods significantly out-
4.4 ADDITIONAL PRETRAINED VISION MODEL
We broaden our coverage of computer vision models include vision transformers Dosovviskkiy et 2021). the other major architecture family currently in use. We train ViT-16/B f on Treeperson, observing smaller gains for AL. than the BiT model (Figure 8). This perhaps reflects the fact that the vision transformer was pretrained on ImageNet-21k Nkk for far fewer epochs then BiT (9 70), and is corroborated by much lower oversampling of minority classes than BiT Figure 9).
ANALYSIS
5.1
ACTIVE LEARNING SELECTS EXAMPLES THAT RESOLVE TASK AMBIGUITY
Overall, we attribute improved performance to pretrained models' ability identify and preferen- tially sample disambiguating examples and latent minority examples that resolve task ambiguity.
Waterbirds Figure 4a depicts the rate at which uncertainty sampling is acquiring examples of each subgroup compared to the expected rate at which random sampling would acquire examples from those same subgroup. Examples where the bird and background mismatched are heavily oversampled. We emphasize that these minority examples are not simply members of the minority class (waterbirds). If active learning were simply upsampling minority classes, then all landbird images would be downsampled and all waterbird images would be upsampled. However, this is not the case-instead, the model identifies and preferentially upsamples informative examples where the spurious feature (background) and the causal feature (bird type) disagree.
the same pattern in Waterbirds: the model identifies and upsamples examples where only one of the spurious or causal features is present (Figure 4b).
Treeperson For Treeperson
Amazon We also see similar behavior in the Amazon dataset, indicating our method's 's applicability multiple modalities and pretrained models. Not only does the model upsample lower star ratings, which are less common, it is also able to upsample rarer product categories an unseen attribute.
5.2 PRETRAINING IS CRUCIAL FOR ACTIVE LEARNING TO WORK
We also consider the role of the pretraining dataset and model size. We conduct active learning experiments with 3 pretrained models: BiT-S-R50x1, BiT-M-R50x1, 100xx, and their cor- responding non-pretrained versions. For all pretrained models (except BiT-M R-101 for Treeperson)
NNte that least confidence, entropy, and margin sampling are identical in the case of binary classification. wttpss//uugginnfacccomm htw.:/ggggnnn.e..oooogoo///////e 66-24

---

Under review as a conference paper at ICLR 2022
0.7
Random
Tran Sct
Uncetaiity
0.6
Uncertaintty
0.4
0.3
0.
0.02
0.1
0.01
5 Stars stars 3 Blars 25lars
1
raang
catlagory
(a) Acquisitions by star rating
(b) Acquisitions by product category
Figure 5: Uncertainty sampling upsamples both visible and latent minority subgroups. Fraction of Amazon examples acquired by random and uncertainty sampling, stratified by star rating and product category. Uncertainty sampling preferentially acquires examples with lower star ratings and rarer product categories, despite the latter attribute not being visible to the model.
uncertainty acquisition outperformed random acquisition (Figure 7a Figure 7b Figure 7c). Impor- tantly, we did not see any gains from AL on the unpretrained models, even when exploring a range of different hyperparameter configurations.
Scaling trend Interestingly, the BiT-S-R50x1 model, which was pretrained on a smaller dataset than the BiT-M models (ImageNet-Ik Ik ImageNet-2Ik) achieves less of a boost from active learn- ing on iWildCam vs the other two models. This suggests that the gains from active learning may continue grow pretained models trained for longer more data. However, a difference between BiT-M-50x1 1 and BiT-M-101x1, which were trained on the same dataset but have different numbers of parameters. We encourage future work that more thoroughly examines these behaviors and scaling laws across a range of models and datasets.
did not
Impact of pretraining on acquisition patters Pretrained models also acquire disambiguating subgroups much more efficiently than their unpretrained counterparts. See Appendix G for addi- tional figures and results illustrating this point.
5.3 PRETRAINING PROVIDES BETTER FEATURE SPACE FOR ACTIVE LEARNING
Why does pretraining improve AL for resolving task ambiguity? We investigate the hypothesis that pretrained representations enable AL models to select objects based on higher-level features (e.g. object foreground and background) through various analyses on Waterbirds.
Linear classifier analysis We investigate the presence of these high-level features by training linear classifiers on the second last layer of BiT models. The classifiers are trained to predict each image's 's bird type and background type (4 classes, each comprising 25% of the data). As shown in Figure 6. these classes are more linearly separable pretrained models both before any finetuning, well as after finetuning class Waterbirds dataset. This demonstrates that pretrained models can better identify both the causal and spurious features in the data, potentially explaining why they can choose them preferentially.
seed set and then acquired examples from the original 2-
tintuniing Iitnttne Sand (40), Fineture (40)
Fiveturi s Fiietune On Seed Set (40)
Fecture Seed Ser 20
Avarage
0.402
9.42
0.424
Aunrage
11111
0:17
DM
Landberd Alandli
bondti id flaodBG
0.306
0.321
Wattrbied /Land50
0:63
0.653
6,233
Whewibidd Bandtic
0:994
0.375
Nenterbo
Lanchind WWttertt
Waterbird /Water 36
0.455
0.438
6.419
Weatlrd (Woletti
EL137
(a) Group accuracies for linear classifier on Water- birds image embeddings attained from a pretrained BiT model after various degrees of finetuning
(b) Group accuracies for linear classifier Wa- terbirds image embeddings attained from an unpre- trained BiT model after various degrees of finetuning
Figure 6: Both causal and spurious features are more linearly separable in pretrained models.
7

---

Under review as a conference paper at ICLR 2022
Uncortainty Random
Uncurtainty Random U
0.70
0.7
0.8
0.6-
0.65
0.5
0.7
0.4
0.6
0.5
S.500
M.850
M.R101
R5 RLO1-NP
0.1
MIR101
5-850
M-R50
M-1101
R50 NP
R101-NP
(a) Waterbirds
(b) iWildCam
(c) Treeperson
Figure 7: Uncertainty sampling only provides gains when using pretrained models. S-R50, M- R50, and M-R101 correspond to the BiT-S-R50x1, BiT-M-R50xl, and BiT-M-R 101x pretrained models, respectively, while R50-NP and RIOINP correspond to ResNet models which are not pre- trained. Shaded regions represent 95% CIs (Gaussian approx.).
t-SNE visualization We visualize the second-to-last layer embeddings of BiT (without any fine- tuning) using t-SNE (van der Maaten & Hinton 2008), shown in Figure 13 We find that examples selected by uncertainty sampling fall closer to the decision boundary for the pretrained model than for the unpretrained model. We also see that the pretrained model exhibits far more separation by class, suggesting better able to identify useful features in the input. These conclusions help further characterize how pretraining provides a feature space more amenable to active learning.
5.4 PINPOINTING THE EFFECT OF TASK AMBIGUITY WITH A DOSE-RESPONSE EXPERIMENT
To examine the impact of task ambiguity from the perspective of the unlabeled dataset, we construct variants of the waterbirds dataset where the percentage of mismatched examples range from 95% to 50%. F We then proceed with AL and report results Figure 10 We find significantly larger gains on versions of the dataset with smaller fraction of mismatched examples (gains average 4% for 50-70% shift, and 10% for 75-95% shift) keeping. This provides stronger evidence that active learning is actually helping the model identify the true task.
5.5 FAILURE CASES
To assess how similar the datasets needed to be to the training distribution of BiT, also perform preliminary experiments on Camelyon 17-WLDS (Bándi et al., 2209 Koh et al. 202 1). which con- siders tumor identification from tissue patches, and FMo W-oo-WWILD (Christie et al. 2018 Koh al. 2021]. which considers land-use classification from satellite images. Active learning performs com- parably or worse than random sampling on these datasets, even when using pretrained BiT model, suggesting that generalization to domains far from the training distribution (ImageNet-21k) may be challenging. However, Camelyon (--lyoonnn-WWLDD WIILD is also known to exhibit high variance across seeds which may also be a contributing factor.
6 RELATED WORK
Task ambiguity and specification Several works address ambiguity or poor specification in ma- chine learning problems. Taylor al. (220)) describe the problem of "inductive ambiguity identifi- cation," and describe active learning as a promising potential solution that has failed to see practical success. D' D'Amour al.] (2020) describe the problem of underspecification, where high variance, instability, and poor model performance result from training overparameterized models on small amounts of data. Geirhos et al. (2020) describes how task ambiguity can arise when both desirable and undesirable features are predictive of the training labels, problem which several works seek to better characterize and address (Nagarajan et al., 2002I; Sagawa et al. 2019 2020). Finally, [Fin et al. (2018) address task ambiguity in few-shot settings via probabilistic meta-lcarning algorithm,
"We construct these datasets using the code taattps://githin com/ wcwp//ootpangwee/rrouu. DRO
'Results are preliminary and will be rerun with more seeds.
https::/ ww dss stanford. wwtpr/dddddggee_trreeeeeee
8

---

Under review as a conference paper at ICLR 2022
and perform an active learning experiment in ID regression setting. We build these works by demonstrating that simple uncertainty sampling with pretrained models can be an effective approach to the task ambiguity problem across wide variety of high -igeedimensional classification settings.
Uncertainty and distribution shift In the face of these challenges, several works have tried to quantify how much pretrained models know about problems or their own uncertainty about them. Rajpurkar et al. (2018) propose a question answering dataset with unanswerable questions, where a model must abstain rather than proceeding with an answer. Pretraining can also improve the calibration of model uncertainty (Hendrycks et 2019) and pretrained features can be used for out- of-distribution detection (Reiss et al., 2021] Wu & Goodman 2020) observations that align with our findings that uncertainty sampling can identify minority subgroups in datasets. Furthermore, our observation that upsampling latent minority groups results better performance aligns well (Sagawa et al., 2020), which found that simply upweighting minority groups performed less well than increasing the relative fraction of minority group examples in the distribution. Importantly, however, our active learning setup does not require these groups to be known in advance.
Active learning and example selection Active learning (AL) (Lewis & Catlemt 1994] Settles & Craven 2008 /Settes] 2009 Houlssy et al. 2011 Aggarwal et al. 2020) is well-studied field that
investigates how machine learning algorithms might automatically select helpful additional data points maximize their performance. Such strategies are especially helpful in imbalanced settings (Ertekin et al.] 2007 MMussmann et al.) 2020) and has been fruitfully applied to deep models (Gal
al.] 2017: Beluch et al. (2018), including pretrained models (Yuan et al.. 2020 Margatina et a
202 1 Shelmanov et al. (021). Past work has also considered AL for few-shot learning (Woodward
Finn 2017). We extend these works by specifically considering AI for resolving task ambiguity, showing that AL can induce something closer to the true task desired by users by selecting examples from unlabeled minority subgroups, as well as examples that disambiguate causal from spurious features. In contrast to prior work, we also demonstrate the causal effect of pretraining on AL by running controlled studies that compare to both random sampling and unpretrained models.
Pretrained models and their emergent properties Our work contributes to broader literature on how pretraining enables new kinds of model capabilities (Bommasani et al.] 2021 Tamkin et al.,
2021. especially when deployed in few-shot settings. For example, (Brown et al.) 2020) identify
the phenomenon of in-context learning, where tasks can be specified for models through a language modeling prompt, while Caron al. (202I) discover that self-supervised vision model implic- itly learns high-quality segmentation maps visible through attention scores. Kaplan et al. (2020); Henighan et al. (2020) conduct scaling laws experiments which chart how capabilities emerge with scale. We identify a new model capability that emerges through pretraining: the capacity to actively learn and resolve task ambiguity in conceptually abstract hypothesis spaces.
DISCUSSION AND LIMITATIONS
We argue that pretrained models are good active learners, capable of identifying informative exam- ples across a diverse range of settings where task ambiguity makes choosing examples challenging. Despite past work largely considering each separate problems. we find that active learning helps in cases where data is spuriously correlated, undergoes domain shift, or contains unlabeled subpopu- lations. These behaviors emerge most clearly as a result of large-scale pretraining, suggesting that active learning may be an underappreciated tool for increasing the reliability of systems in real- world settings. Importantly, pretraining appears necessary for these behaviors to manifest-and in some cases, models pretrained for longer on larger datasets appear to actively learn better.
Our method does suffer from number of limitations. First, it requires a human in the loop for data acquisition, which significantly increases the time required to train model compared to random sampling-e cost which must be weighed against potential benefits. Second, our method requires the labeling method to be relatively free of noise- this may be acceptable if annotators are domain experts or are well-trained, but may also increase the cost per acquired example.
Finally, we note the opportunity for much exciting future work, including deeper investigation of task ambiguity in real- world settings and better understanding how pretraining shapes active learning models continue to scale.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Umang Aggarwal, Adrian Popescu, and Céline Hudelot. Active learning for imbalanced datasets. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1428-1437, 2020.
Péter Bándi, Oscar G. F. Geessink, Quirine F Manson, Marcory Crf van Dijk, Maschenka C. A. Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng. Aox- iao Zhong, Quanzheng Li, Farhad Ghazvinian Zanjani, Svitlana Zinger, Keisuke Fukuta, Daisuke Komura, Vlado Ovtcharov, Shenghua Cheng, Shaoqun Zeng, Jeppe Thagaard, Anders Bjorholm Dahl, Huangjing Lin, Hao Chen, Ludwig Jacobsson, Martin Hedlund, Melih Çetin, Eren Hahecı Hunter Jackson, Richard Chen, Fabian Both, Jörg K.H. Franke, Heidi V. N. Küsters- Vandevelde, W. Vreuls, Peter Bult, Bram van Ginneken, Jeroen A. van der Laak, and Geert J. S. Litjens. From detection of individual metastases classification of lymph node status the patient level: The camelyon17 challenge. IEEE Transactions on Medical Imaging, 38:550-560, 2019.
Sara Beery, Eljjah Cole, and Arvi Gjoka. The iwildcam 2020 competition dataset. ArXiv, abs/2004.10340 2020.
William H Beluch, Tim Genewein, Andreas Nürbberger, and Jan M Köhler. The power of ensembles for active learning in image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9368-9377 2018.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson,, S. Buch, D. Card, Rodrigo Castellon, Niladri S. Chatterji, Annie Chen, Kathleen Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Hen- derson, John Hewitt, Daniel E Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri. Siddharth Karamcheti, Gcoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee. Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir P. Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, J.C. Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Re- ich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jackson K. Ryan, Christopher R'e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srini- vasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. ArXiv, abs/2108.0725 2021.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vore Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Rad- ford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/2005. 14165, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Herv'e J'egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. ArXiv, abs/2104. 142944, 2021.
Gordon A. Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6172-61 61800, 2018.
10

---

Under review as a conference paper at ICLR 2022
Alexander D' Amour, Katherine A. Heller, Dan I. Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D. Hoffman, Farhad Hor- mozdiari, Neil Houlsby, Shaobo Hou, Ghassen Jerfel, Alan Karthikesalingam, Mario Lucic, Yi- An Ma, Cory Y. Mcl.ean, Diana Mincu, Akinori Mitani, Andrea Montanari, Zachary Nado, Vivek Natarajan, Christopher Nielson, Thomas F. Osborne, Rajiv Raman, Kim Ramasamy, Rory Sayres, Jessica Schrouff, Martin G. Seneviratne, Shannon Sequeira, Harini Suresh, Victor Veitch, Max Vladymyrov, Xuezhi Wang, Kellie Webster, Steve Yadlowsky, Taedong Yun, Xiaohua Zhai, and D. Sculley. Underspecification presents challenges for credibility in modern machine learning. ArXiv, 1.33391, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Shrey Desai and Greg Durrett. Calibration pre-trained transformers. In EMNLP, 2020.
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1), 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. ArXiv, 10./20000.11299 2021.
Seyda Ertekin, Jian Huang, Leon Bottou, and Lee Giles. Learning the border: active learning in imbalanced data classification. In Proceedings of the sixteenth ACM conference on Conference information and knowledge management, pp. 127-136, 2007.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In NeurlPS, 2018.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning, pp. 1183-1192, 2017.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard S. Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. Shortcut learning in deep neural networks. ArXiv, abs/2004.07780, 2020.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. ICLR, 2017.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. ICML, 2019.
J. Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Rad- ford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodci, and Sam McCandlish. Scaling laws for autoregressive generative modeling. ArXiv, abs/2010.14701. 2020.
Neil Houlsby, Ferene Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for 2011.
classification and preference learning, arXiv preprint
Jared Kaplan, Sam McCandlish, T. J. Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv, abs/2001.08361. 2020.
Siddharth Karamcheti, Ranjay Krishna, Li Fei-Fei, and Christopher D. Manning. Mind your out- liers! investigating the negative impact of outliers on active learning for visual question answer- ing. ArXiv, abs/2107.00331. 2021.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsub- ramani, Wei hua Hu. Michihiro Yasunaga, Richard L. Phillips, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: benchmark of in-the-wild distribution shifts. In ICML, 2021.

---

Under review as a conference paper at ICLR 2022
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In ECCV, 2020.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123:32-73, 2016.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
David D Lewis and Jason Catlett. Heterogeneous uncertainty sampling for supervised learning. In Machine learning proceedings 1994. pp. 148-156. Elsevier, 1994.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zetrlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
David Lowell, Zachary Chase Lipton, and Byron C. Wallace, Practical obstacles to deploying active learning. In EMNLP, 2019.
Katerina Margatina, Loic Barrault, and Nikolaos Aletras. Bayesian active learning with pretrained language models. ArXiv, abs/2104.08320, 2021.
Stephen Mussmann, Robin Jia, and Perey Liang. On the importance of adaptive data collection for extremely imbalanced pairwise tasks. In FINDINGS, 2020.
Vaishnavh Nagarajan, Anders Johan Andreassen, and Behnam Neyshabur. Understanding the failure modes of out-of-distribution generalization. ArXiv, abs/2010.15775, 2021.
Jianmo Ni. Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly-labeled reviews and fine- tne-ggrineed aspects. In EMNLP, 2019.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. ArXiv, abs/21055111447. 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't 't know: Unanswerable questions for squad. In ACL, 2018.
Tal Reiss, Niv Cohen, Liron Bergman, and Yedid Hoshen. Panda: Adapting pretrained features for anomaly detection and segmentation. In CVPR, 2021.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. ArXiv, abs/19 11 1008731. 2019.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparamettrizzatoon exacerbates spurious correlations. ArXiv, hiss/2055004345 2020.
Marten Scheffer, Stephen R. Carpenter, Jonathan A. Foley, Carl Folke, and Brian H. Walker. Catas- trophic shifts ecosystems. Nature, 413:591-596, 2001.
Burr Settles. Active learning literature survey. Technical report, University of Wisconsin-Maddisn Department of Computer Sciences, 2009.
Burr Settles and Mark Craven. An analysis of active learning strategies for sequence labeling tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pp. 1070-1079, 2008
12

---

Under review as a conference paper at ICLR 2022
Artem Shelmanov, Dmitri Puzyrev, Lyubov Kupriyanova, Denis Belyakov, Daniil Larionov, Nikita Khromov, Olga Kozlova, E. Artemova, Dmitry V. Dylov, and Alexander Panchenko. Active learning for sequence tagging with deep pre-trained models and bayesian uncertainty estimates. In EACI, 2021.
Alex Tamkin, Miles Brundage, Jack Clark, and Deep Ganguli. Understanding the capabilities, limitations, and societal impact of large language models. ArXiv, abs/2102.02503, 2021.
Jessica Taylor, Eliezer Yudkowsky, Patrick La and Andrew Critch. Alignment for advanced machine learning systems. 2020.
Laurens van der Maaten and Geoffrey E. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579- 2008.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference Learning Representations, 2018.
Mark P. Woodward and Chelsea Finn. Active one-shot learning. ArXiv, abs/ less/7002.6559, 2017.
Mike Wu and Noah D. Goodman. A simple framework for uncertainty in contrastive learning. ArXiv, abs/2010.02038, 2020.
Michelle Yuan, Hsuan-Tien Lin, and Jordan L. Boyd-Graber. Cold-start active learning through self-supervised language modeling. In EMNLP, 2020.
A
CODE RELEASE
Our code and training scripts will be released at /tlttps///redctee
B ADDITIONAL EXPERIMENTAL DETAILS
Here we provide some additional experimental details.
All BiT runs use the same default settings specified in the BiT paper (Kolesnikov al. 2020 and the BiT GitHub repo:
com peoogle-reeearcinn transfer
Settings that are constant across all BiT runs:
1. Optimizer SGD with momentum 0.9
2. Learning Rate Base learning rate 0.003. Linear warm up to this rate, then staircase decay.
Exact schedule depends dataset size, but for our few shot setting, this means: (1) linear warm up in the first 100 steps to 0.003, then (2) decay 10 fold every 100 steps; (3) after 500 steps, stop and move the next acquisition round.
3. Data Augmentation repo/utill s/ /t////asstssssooeeeeeeee for details. Also available BiT repo
Random cropping and flipping.
See our
4. Batch Size 32 when training, split into gradient accumulation microbatches of size 8.
5. Early stopping condition When the training loss reaches below 0.001 times the original training loss.
Settings that varied between image datasets:
Size of Initial Seed Set 40 for Waterbirds, 40 for Treeperson, 182 for iWildCam, 20 for CIFARIO, 8 for Camelyon17
2. Size of Training Pool Entire training set for Waterbirds, Treeperson, Camelyon17. A random subset of 5000 examples and 12000 examples (re-drawn for each acquisition) from the entire training set of 10 and iWildCam, respectively.
3. Number of Acquired Examples 320 for Waterbirds, 320 for Treeperson, 1456 for iWild- Cam, 180 for CIFAR10, 128 for Camelyon17
13

---

Under review as a conference paper at ICLR 2022
4. Number of Examples Acquired Each Acquisition 5 for Waterbirds, 5 for Treeperson, 20 for iWildCam, for CIFARIO, for Camelyon17.
Settings used for Amazon RoBERTa-Large runs:
1. Optimizer- AdamW with default hyperparameters (31 0.99, B2 0.999, weight decay =0.1).
2. Learning Rate LR 1e-6.
Batch Size 2 (due to memory considerations).
Early Stopping Condition When the training loss reaches below 0.001 times the original training loss or when 4000 gradient steps have been taken (whichever comes first).
4. Size of Initial Seed S
5. Size of Training Pool A random subset of 2000 examples (re-drawn for each acquisition) from the entire training set.
6. Number of Acquired Examples - 90.
7. Number of Examples Acquired Each Acquisition 2.
C TREEPERSON DATASET
The Treeperson dataset composed images from Visual Genome Krishna al. 2016 with different compositions of detected objects.
Training set composition by subclass:
Person and Building: 3700
Person and Tree: 370
No Person and Building: 370
No Person and Tree: 3700
The validation set contains 498 examples of each subclass.
The following annotated objects were used to form the different subclasses:
Person: person, people, man, men, woman, women
Building: building, buildings
Tree: tree, trees, Icaf, leaves, grass
The training set and validation set were drawn randomly from qualifying images Visual Genome's training set and validation set, respectively.
D EARLY STOPPING CONDITION
At the outset of this work, we explored if we could identify a heuristic for stopping training when there was no validation set present. We compared how the BiT model would perform if stopped the finetuning step based off of when the validation accuracy plateaued versus when the training loss decayed to 0.001 of its original value.
We smaller experiment than the standard Waterbirds parameters described in Appendix B Namely, our seed set was of size 32, we acquired 64 examples on top of that, and we acquired one example time. We also defined the validation accuracy as having plateaued the fifth time did not increase.
We twelve paired experiments where for twelve different randomized seed sets, performed the Waterbirds experiment four times with random sampling and stopping when validation aceu- racy plateaued, with random sampling and stopping when training loss decayed, with uncertainty
14

---

Under review as a conference paper at ICLR 2022
sampling and stopping when validation accuracy plateaued, and with uncertainty sampling and stop- ping when training loss decayed.
We found that these experiments achieved:
Random sampling Stop from validation accuracy; average accuracy 69. 199.11, average
duration 3.5 hours
2. Random sampling + Stop from training loss: average accuracy = 69.93%, average duration
38 minutes
3. Uncertainty sampling + Stop from validation accuracy: average accuracy 83.64%, aver- age duration 4.5 hours
4. Uncertainty sampling + Stop from training loss: average accuracy = 85.62%, average du- ration 74 minutes
Thus, we concluded that the by stopping our finetuning step just by waiting for the training loss to decay to 0.001 of its original values, could achieve comparable if better accuracies, spend fraction of the time, and remove the need for labeled validation set.
E VISION TRANSFORMER MODEL.
Results for Treeperson when using ViT-B/16 are presented in Figure 8 proves upon random sampling, the gains are not as large as for BiT, perhaps because ViT was pretrained for almost 8x fewer epochs.
While active learning im-
0.65
Random
Random
Uncertainty
Uncertainty
0.63
Accuracy
0.62
0.38
0.61
0.37
0.36
0.59
0.58
0.35
50
100
150
200
250
300
350
50
100
150
200
250
300
350
Total Examples
Total Examples
(a) VIT
(b) ViT
Figure 8: Active learning improves upon random sampling for Treeperson when using ViT- B/16. However, the gains are large as for BiT, perhaps because ViT was pretrained for almost 8x fewer epochs.
F DOSE RESPONSE
Results for the Waterbirds dose-response experiment are presented in Figure Even when the training and test distributions are the same, active learning allows the pretrained BiT model to out- perform random sampling. However, as the train and test distribution diverge, the benefit of active learning increases, indicating the importance.
G EFFECT OF MODEL SCALING AND PRETRAINING ON ACQUISITION
For the Waterbirds model scaling experiment, we track the examples each model acquires. These are presented in Figure II The acquisition patterns of all the pretrained models look fairly similar- subclasses. However, the pretrained model is unable to capture that distinction, and is only able to upsam- ple images with a water background, resulting worse performance. A summary of final class acquisition ratios is available in Figure 12
they upsample both minority
15

---

Under review as a conference paper at ICLR 2022
Train Set Random
0.4
Uncertainty
Frequency Of Class
0.3
0.2
0.1
0.0
No People
No People
People
People + Buildings
+ Trees
+ Buildings
+ Trees
Image Class
Figure 9: Acquisitions for Treeperson using VIT. The ViT model requested labels for minority
classes at a significantly lower rate than did the BiT model.
Accuracy on validation test (50% correct background)
Uncertainty
1.0
Random
4.XX
1 X
XXX
0.9
XXXX
X
X
0.8
0.7
0.6
50
60
70
80
90
Percentage of training images with correct background
Figure 10: Waterbirds background mismatch dose-response experiment. As the train and test distribution diverge, the benefit that active learning provides increases.
16

---

Under review as a conference paper at ICLR 2022
200
200
Pretrained
(h) Pretrained BiT-M-R50x1
(c) Pretrained BiT-M-R101xI
(d) Non-pretrained ResNet R50x
Figure 11: All pretrained models acquire disambiguating subgroups much efficiently than their non-pretrained counterparts. The pretrained models do not simply oversample based on the the bird the background; instead they oversample disambiguating examples which have mismatched backgrounds. By contrast, the non-pretrained model only oversamples images with water background, and accordingly less able perform well on the balanced validation set.
Expected
BIT-S-R50x1
BIT-M-R50x1
200
BITMM-101x1
BIT-S R50x1, No Pre-train
Acquisttion Frequency
BiT-M-R50x1, No Pre-train
Landbird)LandBG
LandbirdWWaterßG
Waterbird/LandBG
Weterbird/WaterBG
Class
Figure 12: Pretrained models ask for labels of images with mismatched backgrounds, while non-pretrained models do not.
H VISUALIZATION OF IMAGE EMBEDDINGS OF PRETRAINED BIT-M
As seen in Figure 13b when performing t-SNE on the Waterbirds image embeddings, we find that the 1-SNE of the pretrained model has much more structure than that of the unpretrained model. In the unpretrained model, the image embeddings of the landbird/land background, landbird/water background, and waterbird/water background classes are distributed uniformly about the center of the t-SNE projection. However, observe much more structure in the pretrained model's t-SNE. In particular, the most noticible difference that the landbird/water background and waterbird/land background classes can be found near the other landbird and waterbird images, respectively. This shows that even without any finetuning, the pretrained model already has learned features to help process the type of bird that appears in the image.
17

---

Under review as a conference paper at ICLR 2022
In the t-SNE plots, the black stars represent the embeddings of the first 10 images that were quired by each model. For the pretrained model, the acquired examples very clearly fall along the waterbird- boundary in the projected feature space. However, in the unpretrained model, the acquired examples are distributed randomly. This suggests that the features acquired during pretraining enables them to select examples that fall near the decision boundary of new unseen task.
-20
-20
andiars fand
landiisia
nnterbindwweter and Woter
waterbbriiitte
-00
-02
waterbirsiond
(a) I-SNE of Waterbirds image embeddings from pre- (b) I-SNE of Waterbirds image embeddings from trained Bit-M
non-pretrained Bit-M
Figure 13: Without any finetuning, pretrained models already embed images into a useful feature space.
I ROLE OF LABEL IMBALANCE
We attempt isolate the role of label imbalance by investigating CIFAR-10 (Krizhevsky 2009). widely-used balanced dataset where task ambiguity is not known to be a common problem. We train uncertainty- and random- -sampling models on both the original CIFAR-I0 10 dataset, as well as an imbalanceed variant where half of the classes have 90% of their examples removed. As shown Figure 14] random sampling slightly outperforms uncertainty sampling on the original CIFAR-10 10 dataset, but uncertainty sampling performs better in unbalanced settings, both when the validation dataset is imbalanced as well as when it balanced.
6.00
8.70
670
672
5.100
680
6.00
310
100
(b) (unbalanced validation)
(c) CUFAA--0-uubalanced
(a) CIFAR-10
(balanced validation)
Figure 14: Accuracy CIFARIO as more samples are acquired with uncertainty random acqui- sition. Shaded regions represent 95% Cls (Gaussian approx.). All runs are with the pretrained model BiT-M-R50x1. (a): The usual CIFAR10. (b) and (c): CIFAR10 10 where the training set has 5 out of 10 classes from which 90% samples are removed. (b) is accuracy on validation split with the same distribution train. (c) is on balanced validation split.
18


Reviews:
[{"review_id": "rEDildbz3HQ", "reviewer": "Reviewer_bL9o", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "To begin with, these are interesting observations and likely have practical implications. The strength of the paper are the empirical reductions in sample complexity and increase in performance. However, in my assessment, these results are fairly preliminary. While in two domains, the image classification results are clearly stronger than the text classification results (but no real explanation for the reason — I guess it is that the ‘task ambiguity’ is consistent from train to test (?)). If this is the hypothesis, it seems that one could perform a ‘dose-response’ like analysis of background shift prevalence between train/test and characterize the strength of this relationship, which seems important as this is the primary justification of why AL with pretrained models is more powerful. I have a potentially simpler hypothesis, that these improved representations remove a lot of the cold-start problems with AL that have previously been mitigated with pre-clustering, careful seed selection, etc. Basically, the AL selection can focus on fitting the conditional distribution associated with classification and not be concerned with the joint distribution of cold-start problems. The compressed representation has lower variance within coordinates and selecting ‘difficult’ examples is more likely to be fruitful (i.e., pathological cases are mitigated). Of course, I am just speculating and my hypothesis is neither orthogonal nor contradictory to the one proposed — but there is no careful empirical analysis or any analytical analysis to justify the proposed hypothesis either. \n\nOn a more practical level, I would also expect more AL querying functions as this is largely an empirical papers and I see no reason why uncertainty sampling is guaranteed to be superior (although it is intuitively appealing that pathological cases probably disappear). \n\nFrom a scholarly perspective, there has been recent related work at least in the NLP space (which I am more familiar with):\n[Yuan, Lin & Boyd-Graber, Cold-start Active Learning through Self-supervised Language Modeling, EMNLP20]\n[Margatina, Barrault & Aletras, Bayesian Active Learning with Pretrained Language Models, 2021]\n[Shelmanov, et al., Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimates, EACL21]\nThere are more, but this gives some good seeds to follow-up on.\n\nOverall, the writing is clear, but I would recommend a few things: (1) put an algorithm somehwere to verify if you are fine-tuning the embeddings or the overall model and bring a bit more of the model details into the paper from the appendices and (2) change the title as it is misleading; something like “Pretrained Models Make Good Active Learners”. \n\nIn summary, the preliminary observations are interesting and notable in some cases. There are enough ‘additional’ experiments to demonstrate that the authors have a promising path to a theory. However, to be a more impactful finding, I lean toward better contextualization, more experiments that vary the active learning querying function, and more empirical/analytical results to support a clear theory. Interesting, but I don’t believe ready for acceptance in ICLR.\n", "summary_of_the_paper": "The authors describe interesting empirical observations regarding using uncertainty sampling to select examples to fine-tune models that use pretrained embeddings and provide some hypotheses regarding the reasons for these performance improvements. Specifically, (1) from a methodological perspective, they propose using uncertainty sampling (i.e., least confident selection) to select examples for fine-tuning image/NLP pretrained models and (2) from an empirical perspective, they use Waterbirds/Treeperson/iWildCam2020-WILDS for image classification and Amazon-WILDS for review star prediction based on text and compare with random sampling — noting that these are settings where there is known covariate shift between train/test with semantic meaning to induce interpretable spurious associations (e.g., background in images). The proposed method works overall, especially on the image datasets, and they also dig into the types of examples selected — noting that they align with expected ‘difficult’ examples (depending on the setting).", "summary_of_the_review": "The authors make some interesting and potentially impressive observations regarding the performance of ‘simple’ active learning (i.e., uncertainty sampling) in the context of pre-trained models — showing positive results both on image and text classification problems. Additionally, they show that biasing toward difficult examples for selection may be correlated with being able to ignore spurious features (e.g., background in images). However, the experiments are limited to ‘least confidence’ uncertainty sampling, there is non-negligible missing contextualization wrt related work, and there isn’t a strong justification for the observed performance improvements. Thus, my assessment is that this submission should be rejected in its current form.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hrMZdCfV2bW", "reviewer": "Reviewer_uqL6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I find the paper and the direction to be interesting and I think the paper is well written and well motivated. However, a major problem the paper faces is that there is not enough experiments in the paper to validate the claims made by the authors.\n\nIn the experiments, the authors show the results of pre-training on all the datasets they used and it is clear that active learning performs better than random sampling however on experiments comparing pre-trained models to un-pretrained ones, the authors only show experiments on two datasets. They report a failure case for one dataset in which the model performance is not distinguishable for neither pre-trained or un-pretrained models. \n\nFocusing on the two experiments, in Figure 8, active learning actually does perform better than random sampling even if the relative performance on the experiment is not as good as that of the pre-trained models. In addition, from the plots active learning performance is indistinguishable to random performance on Pretrained BiT-S-R50x1 and only gets marginally better on the other pre-trained models as more examples are added. \nOn both Figure 7 and 8, we observe that random sampling on pre-trained models outperforms active learning on the un-pretrained counterpart. A possible explanation is that the model has only seen few examples to learn good features that are representatives of the class. This explains why pre-training does better since it is trained on a large pool of unlabeled data and fine-tuned on the specific task hence it learns richer/better feature for model training.\n\nGiven the above limitations, it is hard to verify the main claims of the paper. Additional experiments on different datasets including text data will be needed. Also, it will be helpful to conduct experiments in settings where the un-pretrained models performs reasonably well on the task when trained with just the seed labels. Lastly, it is important to consider different architecture models for the pre-trained models and not just variants of the BiT model.\n\nMinor suggestion:\nIn low-shot settings, entropy sampling may be a better query strategy for the un-pretrained model.", "summary_of_the_paper": "The authors set out to investigate if active learning is an emergent property of pre-training. That is if running active learning with pre-trained models gives better result than using the same models without pre-training. They run several experiments on different text and image datasets first showing that active learning performs better than random sampling on pre-trained models and secondly that pre-trained models perform better than un-pretrained ones for active learning.", "summary_of_the_review": "It's an interesting paper but the authors need more empirical validation to establish some of the claims. See detailed comments in main review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZZ5avPVTkb4", "reviewer": "Reviewer_KGpV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I wanted to see two additional directions of investigations.\n1. First, does the choice of active learning heuristic matter? My intuition is that it probably does not, but it would be nice to see a comparison between uncertainity sampling heuristic and another information seeking heuristic such as entropy sampling. The direction of investigating other active learning techniques (such as query-by-committe or expected model change) also remain open.\n\n2. All the pre-trained models considered are supervised pre-trained models. What happens if we use a large, unsupervised pre-trained model (for example a masked language language model)? Do the same results hold true?\n", "summary_of_the_paper": "This paper investigates if using large, pretrained models in an active learning setup helps achieve better performance with lesser data when compared to using randomly sampled data. In order to conduct this investigation the authors study the empirical performance of large pre-trained models on some image datasets and a text dataset. In both cases large pre-trained model is finetuned on a small amount of seed data and then an active learning procedure is used (in this paper the AL procedure is an uncertainty sampling procedure) to collect more data. The datasets are chosen to illustrate several conceptual issues (i) distinguishing causal from spurious correlations (ii) measuring robustness to distribution shifts (iii) role of data imbalance. \n\nExperiments are performed to show that using an active learning procedure indeed helps improve performance using only a small amount of actively labeled training dataset.  The paper is well written and the results are convincing and insightful. ", "summary_of_the_review": "I like the key question and the experimental setup of this paper and would recommend an accept. It would have been better had the authors considered another AL method, along with uncertainty sampling to investigate the relative impact of pre-training and the active learning methodology in data savings. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QiYLLQB1dzz", "reviewer": "Reviewer_Gazs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n\nThis paper is clearly written. It describes experiment setup and results very clearly. The advantage of pretraining is also clearly established. \n\nWeaknesses: \n\nFor a purely empirical paper (which is totally fine for me), that the depth of investigation is a bit lacking. I would like to see more comprehensive results to be convinced that the performance gap is indeed due to pre-training vs. no pre-training, instead of certain hyper-parameter choices happen to play well with the pre-trained models. Some specific items that I would encourage the authors to provde are: \n\n1. More acquisition functions should be studied. While uncertainty-based acquisition function is perhaps the most popular, many others have been recently proposed. For a recent list, see the related work section of [1]. \n\n2. It is not clear which part of the pre-training is most helpful for active learning. It is known that pre-trained models are very efficient at learning from few examples, but they seem to struggle here, especially in Fig. 2, where the random baseline makes very small progress even with hundreds of data points (which seems to be rare for pre-trained models). I suspect that this is at least partly due to the issue below, i.e. the mismatch between training and test distribution, so some clarification would be appreciated. \n\nIn addition, there seems to be a disconnect between the setup described in Fig. 1 and the actual experiments conducted. In the Fig. 1 setup, the existing (small) training set is not sufficient to disambiguate among a set of candidate functions, so the promise of active learning is to find the maximally confusing data point from the unlabeled set and thus achieve efficient disambiguation. Notably, the training and test distributions do not *need* to be different. However, in the experiments, in all three settings, the training and test distributions are intentionally different, so that the models that have undesirable working mechanisms, e.g. relying on spurious correlation, are \"defined\" to be bad. Is this a characterization of the datasets used? If so, it seems that here, active learning with pre-trained models are better not in the typical sense of improving the performance on the training task, but instead helping the model to gain some other behaviors. \n\nInterestingly, [1] shows that the optimal acquisition strategy (w.r.t. the training task) seems to sample quite homogeneously in the input and output space, and a uniformity regularization would actually improve existing acquisition functions. By comparison, Fig. 6 show strong mismatch between the acquired sample distribution and the data distribution. Given that the paper evaluates the \"other behavior\" performance, if my understanding in the paragraph above is correct, I do not find it surprising. Nevertheless, I would like to encourage the authors to experiment with the uniformity regularization (Sec. 6.5 of [1]) and discuss any findings and relationships. \n\n[1] Towards Understanding the Behaviors of Optimal Deep Active Learning Algorithms, AISTATS 2021\n\nMinor issues: \n\n1. Images should be produced as PDF files and embedded into the paper. This ensures that the image is saved in the vector format, rather than bitmap. For matplotlib, this is as easy as changing the suffix from png/jpg to pdf, i.e. `plt.savefig('figure.pdf')`. \n\n2. The texts on most figures are too small. In general, they should approximate match the text size, or at least the footnote size, but some of them are barely readable when printed out. \n\n3. A reference to [2] is recommended in footnote 1 when discussing calibration, since it is the original paper that investigates the confidence calibration problem of neural networks. \n\n[2] On Calibration of Modern Neural Networks, ICML 2017. ", "summary_of_the_paper": "This paper investigates the active learning performance of pre-trained models vs their non-pre-trained counterparts on both vision and NLP tasks. Specifically, the investigation focuses on datasets with spurious correlation, domain shift, and label imbalance. Empirical results generally show that the pre-trained models with the uncertainty acquisition function performs much better than the random baseline and their un-pre-trained counterparts. ", "summary_of_the_review": "Overall, I think this paper is on track to make a good contribution, but would need a more rounded-out experimental execution. Therefore, I am giving a reject recommendation for now, but I'd be happy to increase my score if the authors provide the requested revision. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper shows that active learning is an emergent property of pre-trained models. They show that simple uncertainty sampling improves sample efficiency by 6 times (up to 6x fewer samples for the same accuracy). This is an interesting and important observation that has practical implications. 

Initially, there were various concerns regarding the message of the paper, including the tile and use of uncertainty function in AL and lack of enough experiments that were addressed through rebuttal period. 

However, there are still remaining concerns that lead to the paper not being ready for publication. Namely, 
  (1) Clear discussion on how much of the gains are due to active learning vs pre-training with respect to different cases. it is also worth investigating additional causes for the failure cases.  
  (2) there are many observations here without a clear narrative or theory.
Moreover, making the story more cohesive will strengthen the paper.
Decision: Reject