Revisiting Active Learning in the Era of Vision Foundation Models
Authors: Sanket Rajan Gupte
Josiah Aklilu
Jeffrey J Nirschl
Serena Yeung-Levy
Venue: Accepted by TMLR ()

Abstract: Foundation vision or vision-language models are trained on large unlabeled or noisy data and learn robust representations that can achieve impressive zero- or few-shot performance on diverse tasks. Given these properties, they are a natural fit for _active learning_ (AL), which aims to maximize labeling efficiency. However, the full potential of foundation models has not been explored in the context of AL, specifically in the low-budget regime. In this work, we evaluate how foundation models influence three critical components of effective AL, namely, 1) initial labeled pool selection, 2) ensuring diverse sampling, and 3) the trade-off between representative and uncertainty sampling. We systematically study how the robust representations of foundation models (DINOv2, OpenCLIP) challenge existing findings in active learning. Our observations inform the principled construction of a new simple and elegant AL strategy that balances uncertainty estimated via dropout with sample diversity. We extensively test our strategy on many challenging image classification benchmarks, including natural images as well as out-of-domain biomedical images that are relatively understudied in the AL literature. We also provide a highly performant and efficient implementation of modern AL strategies (including our method) at https://github.com/sanketx/AL-foundation-models.

Published in Transactions on Machine Learning Research (06/2024)
Revisiting Active Learning in the Era of Vision Foundation Models
Sanket. Rajan Gupte*
sanketg@stanford.dd edu
Deportment of Computer Science
Stanford University
Josiah Aklilu"
josakkil@sttnoordd.. edu
Deportment Biomedical Data Science Stanford University
Jeffrey J. Nirschl
jnörschU@stan edu
Department of Pathology
Stanford University
Serena Yeung-Levy
syyeang@stanford.de edu
Department of Biomedical Data Science Stanford University
Reviewed OpenReview: https: openreview. net/
forum? id= 38K83M9bbG
Abstract
Foundation vision or vision-language models are trained on large unlabeled or noisy data and learn robust representations that can achieve impressive zero- or few-shot performance on diverse tasks. Given these properties, they are a natural fit for active learning (AL), which aims maximize labeling efficiency. However, the full potential of foundation mod- els has not been explored in the context of AL, specifically in the low-budget regime. In this work, we evaluate how foundation models influence three critical components of ef- fective AL, namely, 1) initial labeled pool selection, 2) ensuring diverse sampling, and 3) the trade-off between representative and uncertainty sampling. We systematically study how the robust representations of foundation models (DINOv2, OpenCLIP) challenge ex- isting findings in active learning. Our obscrvations inform the principled construction of new simple and elegant AL strategy that balances uncertainty estimated via dropout with sample diversity. We extensively test strategy many challenging image clas- sification benchmarks, including natural images as well as out-of-domain biomedical im- ages that are relatively understudied in the AL literature. We also provide highly per- formant and efficient implementation of modern AL. strategies (including our method) at bttps://giit http://gittuub..oomeeeeeeeeeeeee wtt.s/nkete/////eeeeooocc Soeees/A--ffoooo AL-founnationn modell...
1 Introduction
Foundation models (Oquab et al. 2023, Cherti al.) 2022 Bommasani et al. 2022) have become ubiqui- tous in computer vision. The advent of large vision and vision-language models, pretrained web-scale data corpora, has led to the significant enhancement. of visual representation spaces. Despite differences in pretraining (e.g. contrastive frameworks with language supervision as in Cherti et al. 2022 or vision-only instance discrimination objectives as in Oquab et al. (2023)) ), the shared theme among vision foundation models are robust learned representations that can be effectively leveraged for state-of-the-art performance
"Equal contribution.

---

Published in Transactions on Machine Learning Research (06/2024)
multiple downstream tasks. These learned visual features have direct implications for machine learning paradigms designed to mitigate data paucity challenges, most notably deep active learning.
Active learning (AL) (Sener& & Savarese 2018: Ash et al. 2020 Gal et al., 2017 Kirsch et al. 2022
Aklilu
& Yeung 2022; Hacohen et al. 2022; Yehuda et al.] 2022, Parvanch et al. 2022 Mahmood et al. 2022
is a machine learning framework that addresses the scarcity of labeled data within limited label budget.
It achieves this by iteratively requesting labels from an external oracle, such as a subject matter expert or clinician. Stated differently, the challenge is determining the most beneficial labels to query when constrained by limited labeling budget to maximize model performance. This approach is especially pertinent in practical computer vision applications that necessitate the creation of domain-specific datasets for custom tasks. A well-designed AL strategy can optimize resource allocation and substantially lower acquisition costs in situations where labeled data are scarce and labor- intensive to produce - such as annotating high- resolution histopathology images for tumor classífication.
Previous research has tested the benefit of using pretrained representations during AL by leveraging repre-
sentations of unlabeled data to 1) address the cold-start problem by selecting good initial pool of candidates
for labeling Hacohen et al. (2022); Chen et al. )2023); Chandra et al. (2020), 2) improve query functions
by selecting points that maximize the coverage of the representation space Ychuda et al. (2022): Sener &
Savarese (2018) and, 3) incorporating unlabeled samples in the training process via semi-supervin
learning Gao et al. (2020); Simeoni et al. (202I). However, these studies limit their focus to supervised pretraining
on Image.et or self-supervised pretraining on available unlabeled data, which often proves impractical in
many real-world scenarios. For instance, biomedical imaging datasets exhibit distribution shifts from natural images and may not have sufficient samples for effective self-superviedd representation learning.
Foundation models offer a compelling solution to these challenges. Image embeddings extracted from these models are semantically organized in the representation space, as evidenced by impressive zero- and few-shot performance with simple linear models, even on fine-grained classification datasets. Evidence also suggests they are robust to domain shifts and generalize well to out-of-distribution data. Since these embeddings 'just. work" out of the box, building dataset-specific feature extractors is no longer necessary. For these reasons, we envision that future research in AL will build upon the ever-increasing capabilities Dehghani et al. (2023); ; Sun et al. (2023) of Vision and Vision-Language foundation models to formulate efficient and scala acalable acquisition stralegies. However, there is limited research Tran el al. (2022) on the impact of foundation models on AL. Our work seeks to re-examine the pillars of effective AL strategies in this context, specifically in the low-budget regime in which only handful of images per class can be annotated. We hope our study's compelling results and novel insights will spur the use of foundation models as a 'batteries included' launchpad for future AL research.
Our key contributions are the following:
1. We investigate the impact of large-scale vision foundation models on the four pillars of active learning strategies, specifically in the low-budget regime, and contrast our findings with established results. Our analysis systematically explores the following dimensions of successful AL strategy:
We study the impact of initial pool selection and highlight differences with existing research on the cold-start problem.
We explore the importance of querying diverse samples and demonstrate that with a few simple modifications, poorly performing AL query strategics can match those that explicitly incorpo- rate sample diversity.
We compare representative sampling with uncertainty- sampling and counter the existing notion that incertainty-based query methods are ineffective in the low-budget regime.
We demonstrate the difficulties of leveraging semi-supervised learning as a complementary ap- proach for building accurate models with limited labeled data.
As a direct consequence of the results of our investigation, we construct a simple, performant, and scalable AL strategy, DropQuery. This method, built atop the rich semantic features generated by powerful foundation models, leverages an intelligent strategy for initial pool selection, utilizes
2

---

Published in Transactions on Machine Learning Research (06/2024)
uncertaint--aase criterion to generate pool of query candidates, and selects diverse subset of candidates for annotation. Our strategy outperforms the current state-of-the-art on a diverse collec- tion of datasets, including fine-grained image classification, out-of-distribution biomedical images, and image classification at scale, all of which are relatively understudiod in AL.
2 Background and Related Works
Traditional research in AL for image classification has largely locused on the development of acquisition functions to query samples for labeling by an oracle. These strategies have spanned uncertainty- baseed sampling approaches (Shannon 1948), Bayesian methods (Gal et al.] 2017 Kirsch et al. 2022), strategies
to maximize sample diversity, query by committee strategies (Melville & Mooney 2004), and techniques to estimate the largest update to the model parameters (Ash et al. 2020).
Cncertainty-based methods supply a single measure of uncertainty (epistemic or aleatoric) for unlabeled samples so as to rank them for acquisition. Entropy selects instances with maximum predictive entropy, Uncertainty selects instances with the lowest. confidence, and Margins selects instances with the lowest margin between the highest and second-highest predicted class probabilities. The Bayesian approaches to AL (Gal et al.] 2017 Kirsch et al. 2019 Woo 2023) leverage Bayesian neural networks to model uncertainty and the potential informativeness of unlabeled instances. Bayesian Active Learning by Disagreement BALD (Gal ct al. 2017) aims to query instances that maximize the mutual information gain between model parameters and predictive outputs. Additionally, the PowerBALD (Kirsch el al. (2022) query challenges the traditional top-B acquisition approach by noting the correlation of queried instances between each iteration of AL, which is alleviated by introducing stochasticity the query.
The prior works that have placed an emphasis on diversity sampling argue that the classifier should learn good decision boundaries in the representation space early. In the lens of these diversity-based queries, the AL paradigın can be re-framed as "what are the diverse instances that are representative of the data distribution The Coreset algorithm (Sener & Savarese 2018) can be approximated as solving the k- Center greedy problem (Farahani & Hekmatfar 2009),, where the distance between any instance and its nearest cluster center (the labeled instances from previous AL iterations) is minimized. Following the Coresct query, the recently developed ProbCover query aims avoid the selection of outlier instances and query more representative points. This is done by recasting the AL problem as a max coverage problem Nemhauser et al. 1978).
A hybrid technique, BADGE (Ash et al. 2020). uses an AL query that inspects the gradients of a neural network's final layer relative to the current model's prediction. This identifies unlabeled instances that could induce significant model updates. The underlying intuition is that samples prompting large gradient updates require substantial adjustments, siguifying uncertainly in samples with new features. While this method yields diverse and highly uncertain samples, it is computationally intensive due to the storage of penultimate activations. Alfa-Mix (Parvaneh et al. (202)) solicits unlabeled samples with the greatest classification prediction variability when their reprosentations are lincarly interpolated with labeled instances, suggesting that the model needs to learn new features from unlabeled data points. Typiclust (Hacohen et al. 2022). queries typical instances during the early AL stages leveraging the representation space. Clustering in a semantically meaningful feature space after self-supervised learning ensures maximum diversity sampling, Sampling dense regions within clusters aids in better classifier learning within feature space.
However, in most of the settings considered by these works, model is trained from scratch at each itcration using a random initialization an ImageNet pretrained backbone. The resulting models are often poorly calibrated (Guo et al., 2017) and fail to provide robust estimates of uncertainty. These settings also require relatively large labeling budget as training deep non-lincar models with limited labels can be challenging (Yuan et al. 2020). Consequently, a number of such strategies have been shown to underperform random sampling in the low budget (Simeoni et al. 2021 Hacohen et. al. 2022), or do not perform significantly better than random when strong regularization is applied (Munjal et al. 2022).
Recent methods designed for the low budget regime (Hacohen et al. 2022 Yehuda et al. 2022 take a more
holistic approach to developing an effective AL strategy by leveraging pretrained model embeddings to cluster
3

---

Published in Transactions on Machine Learning Research (06/2024)
features and select representative points, which has been shown to improve the selection of the initial labeled pool. Other approaches aim to exploit unlabeled points using sell-supervised or semi-supervised learning, or a combination of both (Chan et al. 2020 Bai et al. 2021]. Their findings demonstrate that leveraging rich representations significantly improves AL performance in the context of a small labeling budget (Lüth
et al. 2023).
Given these promising results, it is reasonable conjecture that embeddings from large-scale vision trans- formers trained billions of images would amplify the efficacy of these complementary components, con- structing an effective strategy that surpasses simpler baseline consisting of only standalone acquisition function. To comprchensively investigate these facets of AL, we compare the impact of a simple pool initialization strategy that leverages the representation space's implicit structure to sample diverse and rep- resentative points. Further, we investigate different classes of query functions in this context to compare uncertainty-based approaches with typicality- -bsed methods, as the latter have been shown to significantly outperform the former in the low-budget regime. Orthogonal to the query function, we investigate if semi- supervision via label propagation can be an effective aid in improving the performance of an active learner.
Although some recent work has explored the effects of pretraining the AL query (Hacohen et al. 2022; Tamkin et al. 2022,, there has not been a rigorous evaluation of AL queries with respect to developments in foundation models and vision-language pretraining that have yielded highly expressive and rich visual representation spaces. In the following sections, we outline the critical components necessary for an effective AL query in the context of large foundation models.
3 Investigating the pillars of effective active learning strategies
paol UD tame } be a dataset with held-out test set and training pool D poot = {Du U }r consisting of unlabeled instances {a:} for i E (Nu} and labeled instances ((z).V;} for E {. Let f : "xxx3 + Rd be feature extractor such that z; := f(x;) E Id is a feature vector and d is the embedding dimension. Let g(2; Rd - R be a classifier parameterized by 0. The typical AL procedure consists of a series of sequential iterations beginning at t= 0 where the labeled pool Dr is the empty set. At each iteration, a querying function a(Iz; : i (N which given the current model parameters 02, receives as input all unlabeled instances, selects B instances to be labeled by external oracle (e.g. clinical expert). Once labels are acquired, these labeled instances are subsumed in DL and removed from the unlabeled pool, and the model is retrained this slightly larger labeled pool. We simulate the AL procedure by hiding labels yı until queried for a chosen instance 2;.
Formally, we consider the batch AL setting: let D (D pool
In this work, we employ frozen vision-only or Dision-language foundation models as the feature extractor f() and a linear head as the classifier g(.). We only require a single forward pass through the feature extractor to generate embeddings which are saved for use in our various experiments. This enables us to study a wide range of experimental settings for AL efficiently. Unless mentioned otherwise, we use a DINOv2 VIT-g14 (Qquab et al.) 2023 transformer the feature extractor. Simple lincar models trained on features extracted by this model achieve performance that is just shy of state-of-the-art making them a compelling alternative to end-to-end fine-tuning of the backbone with scarce labels. Following the recommendations of (Munjal let al. 2022), we apply strong regularization to our models in the form of weight decay (le-2) and aggressive dropout (0.75). Note that the benchmarks used to evaluate the various AL mothods in this work were not used in the pretraining of the foundation models we investigated (we refer the reader to the data curation processes outlined by Oquab et al. (2023))
In our experiments, we set query budget B of 1 sample per class per iteration. For instance, the CIFAR100 dataset contains images corresponding to 100 classes, the query budget is set to 100 samples. Note that setting B to 1 sample per class per itcration does not necessarily imply that exactly one sample is chosen from each class, since the class labels are unknown at the time of querying. We run our active learning loop for 20 iterations and average our results over random seeds for each experiment. We analyze the following query functions: Random Sampling bascline, Entropy, Uncertainty, Margins, Core-Set (greedy k-centers), BALD, Power BALD, BADGE, ALFA-Mix, Typiclust, and ProbCover. The perereparaneters for these query functions, if any, are taken from the original publication.
4

---

Published in Transactions on Machine Learning Research (06/2024)
3.1 The impact of initial pool selection
We begin our study by measuring the impact of different initial pool selection strategies in AL, particularly in the cold-start scenario where data has been used for training the classifier. Tt. has been shown in the recent literature Hacohen al. (2022) : Yehuda et al. 2022]; Chen et al.] (023) that the initial pool is crucial for establishing a good rudimentary classifier that enjoys performance gains throughout AL. Hacchen et al.) (2022) and Yehuda et al. (2022) also argue that the AL acquisition function need not be decoupled from the initial pool selection. Rather, an AL query should aequire representative samples so the classifier can learn good decision boundaries from the first query. Intuitively, in order for an AL query to be effective, it must rely on the current classifier's 's estimates of uncertainty or informativeness.
To assess initialization strategies, we contrast AL query performance when the initial training pool for the active learner is randomly selected for labeling to a centroid-sceking approach following Pourahmadi et al. (2021), where the initial pool consists of samples from Dy that are the closest to class centers after K-means clustering in the feature space, where B is the number of clusters. Typiclust and ProbCover utilize differ- ent initialization strategics based on sample density and maximizing coverage, respectively, so we hold these queries true to their inilialization approach. Table compares the deltas (A) in the lest set performance of a linear classifier actively trained on foundation model features using a randomly initialized initial pool versus a centroid-based initial pool. Note that a random initial pool is suboptimal to methods that explicitly take advantage of semantically meaningful embedding spaces (i.e. Typiclust or ProbCover) ). The tremendous performance gains in early AT, iterations for uncertainty-basee AI. queries like Entropy sampling enable these methods to surpass queries like Typiclust and ProbCover.
Table I also demonstrates the impact of initial pool selection on AL queries acting on foundation model features. Some queries, like Uncertainty, Entropy, and Coreset, enjoy performance gains throughout AL given a centroid-based basod initialization, a major boost to the random initialization. Intorestingly, some queries like Alfa-Mix and BADGE have deltas that converge within 2-4 iterations, and in later iterations (8-16) 16) we observe higher accuracy with a randomly selected initial pool. Since the Alfa-Mix query crucially relies on interpolation in the feature space, we hypothesize that the separability of foundation model repre- sentation space renders differences between initial pooling strategies negligible. After a few iterations, the robust visual representations from foundation models enable the selection of highly representative samples, which help the classifier establish good class boundaries in few-shot.
The results from this experiment are in contrast to previous preliminary findings from Chandra et al. (2020) that report no significant difference in initial pool selection strategies in the long run. While we also see diminishing returns as progress to later iterations, emphasize that the experimental results in Chandra
ct al. (2020) also showed little to no difference caused by most initialization strategies, even in the very first iteration of AL. This observation does not align with our results in which we see stark differences caused by intelligent selection of the initial labeled pool in the beginning itcrations.
However, this discrepancy can be resolved when laking into account the sizes of the labeling budgets. Our experimental setting the very low-budget regime, and the number of labeled samples in the final iterations of our studics barcly overlaps with the starting budgets studied in Chandra ct al. (2020). We conclude that in the low-budget regime, where only a few examples per class have been acquired, initialization is crucial for AL performance, but in later iterations, as the number of labeled samples grows or as we transition to higher-budget regimes, it is not as significant and randomly selected initial pool works just as well.
3.2 On the importance of diversity in query selection
In line with the initial pool selection, we evaluate how partitioning the reprosentation space via clustering is crucial for AL performance in subsequent iterations, even after the initial query. To conduct these exper- iments, we allow AL queries that use stand-alone measures of uncertainty (i.e. Uncertainty, Entropy, Margins, and BALD) ) to deviate from the top-B acquisition pattern persistent in the AL literature. As noted by Kirsch et al. (2022), top-B acquisition can potentially lead correlation between queried batches for some I and lj, which can hurt performance.
5

---

Published in Transactions on Machine Learning Research (06/2024)
Table 1: Effect of initial pool selection on performance. Test set accuracy using our cnttroddbbsse initial- ization. In parentheses, we show the difference (A) in performance when utilizing our centroid initialization vs. random initialization, where a positive (A) shown in green indicates improvement over random initial- ization. We show AL iterations & for datasets CIFAR100 (Krizhevsky 2009), Food101 (Bossard et al. 2014). ImageNet-1000 (Gansbeke et al., 2020), and Domain (Peng et al. 2019) (from top to bottom) with DINOv2 ViT-g14 as the feature extractor f. For both the Typiclust and ProbCover queries, we report the test accuracy using their own respective initialization strategies for the cold start.
Random
Uncertainty
Eatrogy
Margins
pBALD
BADGE
Alfx mix
Typiclut
ProbCover
CIPAR100
12.4
(+21.4)
72.4 (+214)
72:4 (+2144
12.3 (421,))
72.4 (-201)
72.4 (+20()
72.4 (+21.)
72.4 (-24)
(+260)
65.0
62.3
2
78.1 (+13.6)
76.7 (+14.0)
76.5 (+184)
79.1 (49.2)
(-113)
80.3 (-9.0)
77.7 (+13.2)
70.8 (+7.3)
(+2.7)
80.8
76.2
4
82.5 (+3.7)
80.6 (+5.9)
78.5 (-8.4
640 (+1.4)
82.1 (+1.7)
85.2 (-1.3)
619 (+5.7)
8L6 (-0.3)
83.7 (-0.1
50.5
81.9
(-2.6)
88.3 (+0.4)
86.3
(-0.4)
(-0.1)
(-0.5)
86.5
16
89.2 (10.0)
89.2 (-0.0)
88.8 (-0.8)
90.9 (+0.2)
89.0 (+0.6)
90.9 (-0.1)
88.0 (0.3)
90.7 (.0.2)
90.4 (-0.4)
89.3
80.1
Foo11101
B9:0 (422.3)
06.6 (#22.3)
(+22.3)
(422.3)
09.6
09.8
09:0
09.5
2
74.5 (+10.1)
68.6 (+18.8)
69.1 [+19.5)
73.0 (+10.3)
69.7 (-18.9)
13.7 [-6.6)
69.9 (+18.5)
72.9 (+8.2)
77.1 (+3.0)
79.1
72.3
13.2 (+9.7)
(+13.)
18.1 (+2.7)
72.1 (+9.9)
79.0 (+0.8)
78.7 (+-12)
80.5 (+0.9)
83.1
78.1
83.9 (+1.1)
79.1 (+2.2)
77.9 (-1.)
85.0 (+0.3)
78.0 (+3.7)
85.4 (-0.3)
70.8 (+3.5)
85,5 (+0.2)
85.1 (-0.5)
86.0
81.9
16
87.3 (-0.2)
80.6 (++0.4)
84.3 (-1.6)
89.3 [-0.1]
838 (+1)5)
89.4 [+0.0)
SL7 (10.8)
89.5 (+0.1)
88.8 (-0.3)
81.3
85.1
ImsupeNee-110
80.8 (+26000)
80.8 (+26.0)
80.8 (+26.0)
80.8 (12600)
80.8 (=:2600
80.8 (4:26.0)
80.8 (+26.0)
80.8 (=26.0)
80.8 (1:26.0)
76.7
76.6
55.4 (+5.9)
83.6 (+23.8)
82.5 (+24.7)
56.2 (+7.9)
86.6 (+7.9)
58.0 (-4.1)
84.8 (+8.3)
87.0 (+5.1)
$7.3 (+2.4)
89.5
69.1
(12.0)
86.2 (+11.8)
85.8 (11700)
(-0.1)
87.9 (40.3)
90.6 (-0.1)
(10.9)
89.0 (-0.2)
90.2 (-0.5)
92.3
91.7
91.6 (+0.0)
88.4 (+227)
87.8 (-6.0)
91.5 [.0.7)
89.2 (-0.8)
92.3 (-0.7)
88.0 (-0.5)
92.3 (..3)
98.2 (-0.2)
93.3
92.7
1G
93.0
(1..5
90.5 0.8)
[-0.8]
(-0.1)
(-0.3)
90.4 (-0.7)
98.8 (-0.4)
(-0.1)
93.4
Net
1
88.5 (1 (123.6)
68.5 (1) (1:63.)
68.5 (2.8)
08.5 (123.))
08.5 (1:60)
68.5 (1 23.0)
08.5 (1 (123.))
08.5 23.6)
08.5 (123366
64.8
63.9
11.9 (+10.1)
70.6 (+16.5)
[+19..)
(+10.1)
71.7 (=11.5)
(+7.8)
71.1 (+12.2)
72.6 (+7.4)
13.8 (+2.1)
13.0
73.8
4
76.0 (12.9)
72.9 (8:4)
72.2 [1 13.5)
76.7 (13.3)
73.9 (4.6)
17.1 (1.4)
73.7 (14.5)
76.6 (1-1.2)
76.2 (-0.4)
74.8
77.8
79.6
77.1 (+3.0)
76.5 (-6.)
80.1 (+0.2)
17.5
81.0 (-0.3)
8088 (+0.1)
(-0.3)
76.2
50.6
16
52.2 (40.1)
81.3 (10.9)
80.7 1.7)
$4.0 (+0.0)
81.2 (10.1)
84.0 (-0.0)
80.5 (40.1)
84.3 (40.1)
79.5 (-0.2)
78.0
82.3
We modify these queries to query the cop-(K- (JKKK) B) samples based on their uncertainty metrics (K = 50) and then cluster these samples by k-means into B clusters and select points closest to the cluster centroids. We also experiment with enabling dropout at inference time to add an clement of stochasticity that disrupts the classifier's estimates of uncertainty, allowing for diverse sample selection. We report. the results of our experiments in Table 2 To decouple the influence of the initial pool selection, all queries, including Typiclust and ProbCover use an identical randomly selected pool of initial samples. Our results show that by imposing simple diversity measures, incertainty-based queries like Uncertainty, Entropy, Margins, and BALD surpass the performance of AL strategies that explicitly incorporate diversity in their queries.
3.3 Representative versus uncertainty sampling and the phase transition
Much prior art in AL has placed emphasis on the distinction between sampling the most uncertain instances in Dy for the active learner and sampling diverse, representative instances for establishing well- defined decision boundaries. It is well-known fact that querying diverse but relatively representative instances early on in the AL procedure allows for the classifier to learn the structure of the representation space early in order to make uncertainty estimates in later iterations more reliable (Hacohen et al.) 2022). This trade-off between diversity sampling the low-data regime before uncertainty sampling known as the phase transition (Hacohen et al., 2022).
However, when leveraging large foundation models as feature extractors, observe a phenomenon that is striking deviation from the existing literature. As demonstrated in Table when controlling for random initialization, uncortainty sampling is actually competitive to representative sampling methods like Typi- clust as early as the 2nd AL ileration (CIFAR100, DomainNettRRaal) and even beat these methods in later iterations. Using foundation models in our diverse selection of datasets, we see no evidence of a phase tran- sition and find no support for the notion that uncertainty sampling is ineffoctive low-budget AL. Further, we witness that Typiclust underperforms the uncertainty-bant queries in later iterations (8- 16, Food101 & Relaal)Ne-RRell..
6

---

Published in Transactions on Machine Learning Research (06/2024)
Table 2: Marginal (A) performance for stand-alone uncertainty measures (i.e. softmax uncertainty, predictive entropy, margins sampling, or BALD) when using clustering (Ax) or when using clustering with dropout during inference (As+d). For fairness, we hold all queries to use a randomly selected initial pool. Power BALD, Coresct, BADGE, Alfa-mix, Typiclust, and ProbCover inherently enforce diversity in their respective queries, magnitude of A for better visualization.
do not show As for these methods. The cells are color-coded according to the
Uncertainty Ag
Entropy
Margius Da
BALD
PDALD
Coreset
BADGE
Alli-omix
Typichust
ProbCover
Ace.
Apc.
Ap44
Ace:
Aeta
Anc.
Ag
CIFAR 100
79.6
+13.1
+17.0
79.4
16.6
21.4
80.4
+7.9
+10.5
76.9
+9.5
71.2
64.5
72.5
78.0
73.9
T4.6
88.7
+12.0
+12.0
86.7
-15.9
-6.33
87.1
+11
+1.5
86.8
-0.2
83,9
78.2
81.1
83.9
86.1
82.2
89.9
+5.6
+5.7
89.8
+8.2
+8.4
89.8
+1.8
+1.9
89.3
+4.1
88.4
88.3
90.9
90.8
89.1
16
91.2
+2.2
+1.9
91.4
+3.5
+336
91.4
+0.9
+0.7
91.0
-2.7
90.8
88.3
90.9
90.8
90.0
89.1
Food101
70.3
116.1
19.4
68.8
14.3
18.9
72.3
17.4
19.6
64.0
113.2
67.1
51.5
64.8
74.1
77.0
73.8
81.8
+17.0
+18.3
80.0
-17.6
20..2
81.7
+5.7
+4.2
77.5
+15.3
78.2
63.0
77.5
79.6
82.5
78.6
80.4
+9.3
+9.1
83.7
-11.1
-12.7
86.
+1.5
+1.5
81.3
+10.0
85.6
73.4
85.3
85.6
85.8
82.3
16
88.9
+4.5
+3.6
88.7
+5.9
+-61
89.0
+0.1
-0.4
87.9
+5.6
89.4
80.8
89.4
89.2
87.4
85.3
IcaageNet-100
88.5
+21.6
+28.5
58.4
21.1
-30.5
88.5
+8.5
+9.5
86.7
-8.0
83.9
76.5
81.8
84.9
89.1
87.0
90.6
115.6
116.2
90.4
20.9
21.8
91.2
12.3
12.2
90.2
+2.0
90.8
85.7
89.8
90.7
92.4
91.8
93.3
+7.6
+7.6
92.0
11.3
10.3
93.3
+1.3
+1.1
91.1
+1.6
89.0
82.6
93.4
92.7
16
94.1
+1.1
+3.6
91.2
+1.7
+-.5
94.4
+0.3
+0.2
93.3
+1.9
94.4
91.1
94.1
91.4
93.8
03.8
Domain Net-Real Real
71.5
+12.6
+17.4
71.2
15.4
-00.8
72.3
+7.1
+10.4
68.1
+7.8
65.3
58.8
65.2
71.4
72.4
71.3
79.4
+13.9
+11.9
78.8
19.5
-20.1
79.1
+6.5
+6.6
78.0
-8.7
75.6
69.2
75.4
76.6
75.2
77.3
82.7
+8.3
+8.5
82.5
12.2
12.2
82.4
12.6
+2.5
81.7
+5.4
80.7
76.0
80.7
78.7
77.0
80.4
16
81.6
+12
81.5
+3.8
+3.5
81.4
+0.7
+0.4
8L.5
+3.3
81.0
80.4
79.7
78.3
82.1
3.4
Leveraging unlabeled instances for training the active learner
Even with a good initial query, the classifier g(2; 0) may perform poorly in the low-data regime since there are a limited number of training examples. In cases where the labeling budget is prohibitive, acquiring enough training instances for classifier to establish good decision boundaries early on in AL may be difficult. This motivates the use of unlabeled instances in principled way to ensure that. the classifier maximizes performance even in the low-data regime. However, our experiments find that the efficacy of semi-supervised learning in this setting questionable While we do see initial boost in performance, contrary to Gao et al. (2020) and Simeoni et al. (2021) the gap quickly narrows, and label propagation underperforms the reference query. Propagating labels from uncertain queried samples may cause points across the decision boundary to be assigned to incorrect classes, hampering performance the long run. We point the reader to the Appendix for an in-depth analysis using a popular semi-supervised algorithm.
best, with wide variation across query methods and datasets.
3.5 Summary of results
Our experiments with initial pool selection (we study the importance of intelligently selecting the initial labeled pool by comparing catroid-based initialization with random initialization) motivate the use of a centroid-based initialization strategy to overcome the cold start problem. Additionally, our experiments modifying existing uncertainty-based approaches enforce diversity in query selection, which motivates the use of clustering to select diverse samples during active learning. Our findings also indicate that clustering approach similar to Alfa-Mix selects diverse candidates that span the representation space. Furthermore, our investigation into the previously identified phase transition implicates the need for an uncortainty-based query function instead of represenlative sampling throughout active learning. We revisited the notion that uncertainty-basee queries are outperformed by representative sample selection methods such as Typiclust in the low-budget regime. We find that this is not the case, contradicting this existing notion. Finally, our inquiry into enhancing the active learner with unlabeled instances cautions against the use of semi-supervised learning complementary approach to active learning.

---

Published in Transactions on Machine Learning Research (06/2024)
Based on our findings, an effective AL strategy would initialize the labeled set with representative samples, employ an uncertainty-based query function for shortlisting unlabeled candidates, and select a diverse subset of these candidates.
4 DropQuery, , a simple, effective active learning strategy
The observations from the previous sections directly inform the principled construction of a new AL strategy leveraging the robust visual features of foundation models, which we refer to as DropQuery. A detailed description of the query strategy is provided in Algorithm [ Below, we briefly review the key results from section that motivate the choice of components and the construction of DropQuery.
Centroid-based initial pool selection: Our experimental results in Section 3.1 demonstrated the utility of intelligently selecting the initial pool of candidates to label. Informed by these results and given the semantic clusters characteristic to the latent spaces of foundation models, Drop- Query employs centroid-based initial pool selection strategy to overcome the cold-start problem.
Uncertainy-based sampling approach: Based on our investigation into the tradeoff be- tween uncertainty-baser and representative query sampling in Section 3.3 DropQuery favors an mcasure uncer-
uncertainty-base query strategy. We experiment using dropout perturbations tainty and select candidates for annotation. Given features 2; of an unlabeled instance, we produce M dropout samples of these inputs (p 0.75) to get +,, The current classifier at & then makes predictions on these M samples and the original instance 2;, and measure the consistency of classifier predictions is = g(2;; 0). If more than half of the M predictions are inconsistent, we add 2, to the candidate set Ze- In all of our experiments, we set M =3
Selecting a diverse pool of candidates: Our analysis in Section 3.2 provided evidence favor of choosing a diverse set of candidates annotate. DropQuery leverages this result by taking the candidate set Ze and clustcring it into B clusters, sclecting the point closest to the center of cach cluster. These constitute diverse subset of points to be annolated.
Leveraging unlabeled instances: The results from our investigation in Section 3.4
3.4 did not provide
conclusive evidence favor of adding a semi-supervised learning component to DropQuery, hence we choose to omit this particular variant of label propagation from our strategy.
Algorithm DropQuery
Input: Given unlabeled instances E Le, external oracle o(-), budget B. Output: Queried labels Y = (vi: 1, 1,..., B)
for 2; Z do
(x),..,,
xM + Dropout(z:5p)
4 Apply dropout to the input features
ni
+- Σ [g((2 () = g(2(:0)
Measure inconsistency among classifier predictions
m=1
end for
7. = {z, m > 0.5M Y2 E 7
Use consistency as proxy for uncertainty
C. + K-means Ze, B) where &o {B}
P Cluster points to enforce diversity
S₁ + (z) arg min min,, || - cu where c E C
Y o(Sa)
> Retrieve labels from samples closest to centroids
We take inspiration from works like Ducoffe & Precioso 2017) and Beluch et al. 2008)), which leverage query-by-commiitee to distinguish informative samples. It. is well-known that dropout on the weights of a classifier during inference simulates an ensemble models (Warde-Farley et al. 2014). However, our stratogy a nuanced but critical change from these works. DropQuery more similar spirit to ALFA- Mix, which interpolates unlabeled points with anchors to perturb them and tests the consistency of their class predictions. Applying dropout perturbs features, and those lying close to the decision boundary will
8

---

Published in Transactions on Machine Learning Research (06/2024)
have inconsistent predictions, making them good candidates for querying. Candidate clustering ensures the diversity of selected points, as shown in Table 3 Since label propagation does not help queries in our experiments, we do not consider it a core component.
Table 3: Test accuracy for our method (with random acquisition reference) utilizing dropout at inference time (DQ), with centroid-based initial pool (DQe), and dropout with semi-supervised learning in the form of label propagation (DQsst)-
Random
DQ
DQ,
DQ
Random
DQ
DQ.
DQ..
Random
DQ
DQ.
DQ.a
Romdom
DQ
DQ,
DQ...
CIFAR100
ImageNet- 100
Food101
DomainNet Real
48.1
48.1
72.6
53.3
54.7
54.7
79.1
57.8
47.4
47.4
71.3
49.7
44.8
44.8
08.0
46.8
64.5
81.0
79.2
76.5
89.4
64.4
72.7
76.1
70.6
61.8
75.0
08.9
78.7
87.4
87.7
87.1
86.8
91.7
91.5
90.9
77.2
81.8
82.2
80.9
73.1
79.1
78.7
76.7
86.1
89.8
90.1
89.6
91.0
93.2
93.0
92.8
83.8
86.3
86.3
84.7
79.2
82.4
82.2
80.0
89.2
91.4
91.5
91.0
93.3
94.1
94.1
93.7
87.5
89.5
89.4
86.7
82.1
84.8
84.7
82.1
5
Experimental results
Our strategy's effectiveness is demonstrated on several natural, out-of-domain biomedical, and large-scale image datasets. Initially, we assess DropQuery' performance, maintaining the dataset constant while altering the underlying representation space (see Figure D. Our experiments confirm that larger mod- els generate more resilient visual features, thereby enhancing línear classifier performance in low-budget scenarios. An intriguing phenomenon, observed during iterations 4-16 in Figure reveals that smaller mod- cls produce embeddings that our AL strategy better accommodates in later iterations. The performance benefits of larger models in early iterations (limited data availability)) diminish rapidly, likely due to their expressiveness compared to smaller models' features. Hence, dropout-motivated uncertainty sampling is not as advantageous. We further note that, on average, vision-language foundation models outperform their vision-only counterparts.
@ir.vit:514
din wR.
dine 114
dio. v1.0014
opencip.vtt B16
opercip.vR.. 1.00p.
opencki. M1L
opencils gR.g14
oponcil G14
30
35
30
35
30
25
20
20
20
20
35
15
15
10
10
10
16
16
Itaration
(a) CIFAR100
(b) Food101
(c) ImageNet-100
(d) DomainNet-HH Rea-
Figure 1: Results of our AL strategy on different representation spaces (i.e. DINOv2 (Quab et al. 2023 and OpenCLIP (Cherti al. 2022)). The y-axis is the delta in accuracy between iteration i and i/2. In carly iterations, the improvements to AL query performance are more pronounced for larger models.
5.1 Natural Image Datasets
Our proposed AL strategy is evaluated through comprehensive set of experiments diverse natural im- age datascts sourced from the VTAB+ benchmark (Schuhmann et al. 2022). The VTAB+ benchmark, an expanded version of the VTAB (Zhai et al.) 2020 benchmark, encompasses a range of challenging vi- sual perception tasks including fine-grained image classification and is typically used for assessing zero-shot performance. notable drawback many prior AL studics is the excessive reliance on datascts such as CIFAR100 or TinylmageNet with small image sizes that are not representative of real-world scenarios where AL would be necessary. Therefore, our experiments include several larger, more complex natural image

---

Published in Transactions on Machine Learning Research (06/2024)
coreset probcover
random
uncertainty
bald
margins badge
alfamix typiclust
dropquery
entropy
powerbald
95
85
85
Teat
80
55
Tual
80
75
50
70
40
60
18
(a) Cars
(b) Aircraft
(c) Pets
(d) Places365
80
Teat A09
Test Aze
Text Acc.
60
55
50
40
45
Ioontion
Iteation
Itarationn
Itorafor
(e) Blood Smear
(I) Diabetic Retinopathy
(s) HICBU Hela
(h) HAM 10000
Figure 2: (Top row) Results fine tne-grained natural image classification tasks Stanford Cars Krause et al. 2013). FVGC Aircraft (Maji et al.] 2013), Oxford-IIIT Pets (Parkhi et al. 2012), and the Places365 (?) dalasets. (Bottom row) AL curves for biomedical datasets, including images of peripheral blood smears (Acevedo et al. 2020). retinal fundoscopy (Kaggle & EyePacs 2015), HeLa cell structures (Murphy et al. 2000), and skin dermoscopy skin (Tschandl et al.) 2018), covering pathology, ophthalmology, cell biology, and dermatology domains using various imaging modalities. Additional biomedical datasets are explored in the Appendíx.
datasets that closely mirror real-world scenarios. We describe extensive implementation and training details in the Appendix.
Among the ine-grained natural image classification datasets, Stanford Cars (Krnuee et al. 2013 and Oxford-IIIT Pets (Parkh et al. 2012), our DropQuery outperforms all other AL. queries in each iteration while also outperforming complex query approaches like Alfa-Mix and Typiclust in the low-budget regime in FVGC Aircraft (Maji el al. 2013) (see Figure 2). Our approach, which is agnostic to dataset and model, outperforms the state-of-the-art AL queries, which often necessitate additional hyperparameter tuning given the underlying data distribution. We also test our method on a large-scale dataset with 365 classes, Places365 (Zhou et al. 2017) (which contains approximately 1.8 million images), ), and our strategy beats all other modern AL queries. These results exemplify the scalability of our method on large datasets where AL would be used to probe the unlabeled data space for important samples.
5.2 Out-of-domain datasets
In addition, conduct experiments various biomedical datasets, which significantly deviate from the training distribution of the foundation models discussed in this study. The aim of these experiments is to un- derscore the effectiveness of AL queries when applied to datasets that are out-of-domain or underrepresented in pretraining. This kind of evaluation is especially pertinent in situations where a model trained one institution or sctting deployed in another. An efficient AL query helpful to obtain a minimum number of samples label to fine-tune the model for the new task realistic scenario often neglected in previous studies. illustrates our strategy's performance on challenging out-of-domain data. In real-world contexts,
10

---

Published in Transactions on Machine Learning Research (06/2024)
Random 0.00 4.50
5.05
0.05
3.65
0.00
4.80
0.00
1.40
1.75
1.30
0.00
Random
0.05
Uncertainty
1.60 0.00 3.85
0.00
15
0.25
20
0.05
1,70
1.40
1.75
0.00
Uncertainty
0.20
0.00
0.00
0:00
2:25
0.30
0.00
Entropy
0.05
Entropy
0.05 0.05
0.00
0.00
0.00
0.00
1.65
0:00
0.00
0.20
0.20
0.00
Margins
3.85
4.35
0.20
Margins
1.60
1.15
1.00
0.00
1.20
0.60
2.85
0:25
0.00
1.55
1.55
0.50
0.85
1:90
3.65
0.10
0:00
0:20
3:05
0.05
1:00
1.25
0:80
0.05
Bald
0.65
0.15
0.70
0.10
0.00
0.00
2:665
0.00
0.00
0.90
0.70
0.00
Stretegy
Powerbald
6,00
5.95
6,20
1.45
5.05
0.00
7.15
0.10
3.45
4.25
4.65
0.43
Stotteg
Powerbald
1.25
0.30
0.85
0.05
0.45
0.00
2.00
0.15
0.05
1.00
1.10
0.05
Coreset
0.05
1.50
2.55
0.00
0:20
0.00
0:00
0.00
0.15
0.95
0.30
0.00
Coreset
0.25
0.05
0.00
0.00
0.00
0.00
0::00
0.25
0.00
Badge
5.90
6.25
6.35
1.35
5.05
1.15
2.50
0.00
3:45
4.15
4.65
0.30
Badge
0.05
0.70
0.35
2:85
Alfamix
3.60
5.20
5.35
1.75
4:45
1.10
10
0.95
0.00
2:55
3.15
0.20
1:25
1.00
0:45
Typiclust
5:80
2.50
4.50
0.00
3:10
0.15
Typiclust
0.65 1.05 1.05
0.25
0.80
0:45
2:65
0:20
0.05
0.
1.40
0.05
Probcover
4.65
1.60
3.70
1.20
1.65
0.00
0.00
Probcover
0.40 0.70 0.90 0.30 0.50 0:45 22.50 0:45 0.05
0.20
0.00
0.00
DropQuery
7,25 6.70
7.00 4.50
5.65
5.00
8.00
3.95 Batige
5.15
5.70
0.00
DropQuery 2.20 1.30 1.45
0.65 1.10
1.05 2:95 0.80
0.25
1 25 2.30 0.00
Random
Uncertainty
Entropy
Margins
Bald
Powerbald
Coreset
Alfamix
Typiclust
Probcover
DropQuerry
Random
Uncertainty
Entropy
Margins
Bald
Powerbald
Coreset
Badie
Allamix
Typickest
Prabcover
DropQuery
Strategy
Strategy
Figure 3: Win matrices for all AL strategies investigated in our study evaluated on natural image datasets and out-of-domain biomedical image datasets. (a) CIFAR100, Food101, Imagenet-100, and DomainNet-Rol using DINOv2 VIT-g/14 features and Stanford Cars, FVGC Aircraft, Oxford-III Pets, and Places365 using OpenCLIP VIT-G/14 features (8 total settings). Due to computational costs, ProbCover was not evaluated on Places365, so the max value of cells in the ProbCover row/ row/colume colummn is 7. (b) Blood Smear, Diabetic Retinopathy, ICCBU Hcla, and Skin cancer datascts using DINOv2 VIT-g/14 features (4 total scttings). DropQuery out performmss all other methods on the natural image dalasets and is a strong competitor to all other methods on the biomedical image datasets with statistical significance.
DropQuery excels as a practical, efficient method for querying hard-to-classin samples for improving the model.
5.3 Statistical significance study
In order to verify the empirical strength of DropQuery we furnish our results with statistical significance tests following the procedure in Parvanch et al. (2022) and Ash et al. (2020). We conduct paired f-tests to determine whether a particular AL strategy i outperforms another strategy j. We say that query strategy beats or surpasses strategy at iteration r if
V5μº d
o" 2.776. n¹
aa ag), o"
aa at m2)
where of is the accuracy of strategy at iteration r. This corresponds the p 0.05 threshold for significance. We compute these ci,, for all iterations and sum the number of times strategy surpassed strategy j at each iteration (divided by the total number of iterations) to indicate the score of i beating in a specific datasct-model setting. Figure 3 illustrates two "win" matrices illustrating the proportion of times strategy has a significantly higher (p<0.055) mean accuracy compared to strategy j. For each pair, at each iteration, the mean accuracy for methods i and j are compared using a t-test with p<0..55, and the outcome win, losc, or tie to yield a fraction of wins from (0, 1). This is repeated for all datasots considerod and the values are summed to yield the proportion of wins. For example, Figure DropQuery beats random sampling in 7. 7.25/8 fraction of total AL iterations across the 8 dataset. settings.
11

---

Published in Transactions on Machine Learning Research (06/2024)
5.4 Ablations for DropQuery
5.4.1 Impact of the dropout ratio
We explore the influence of the dropout ratio on both the overall performance of the models and the number of selected candidates at each iteration step. We experiment with range of values - [0.15, 0.30, 0.45, 0.60] in addition to 0.75, the setting used in main experiments. Our findings, highlighted in Appendix Table 1, indicate that higher dropout ratios for the features result in slightly better-performingg models, a consistent trend among datasets. This is perhaps unsurprising, as stronger regularization in the form of harsher dropout ratios would improve the generalizability of models in the low-budget regime. Interestingly, lower dropout ratios lead to fewer samples added to the candidate set. This may be a complementary factor affecting the performance of the model, as a smaller candidate set would likely be less diverse.
5.4.2 Impact of M, the number of dropout iterations
The number of dropout itcrations, M, is tunable hyper-parameter for our query function. We experiment with a range of values [5, 7, 9] in addition to 3, the value used in our main experiments, and present the results in Appendix Table 2. Our findings indicate that this ayper-parameter has a minimal impact on both the performance of the model and the number of selected candidates, with fewer itcrations having slight advantage. The robustness of our models' performance w.r.t M is a desirable trait for an AL query function designed for the low-budget regime since the existence of a validation set for syper-parameter tuning cannot be assumed due to the paucity of labels.
Over the course of our experiments, we found that other query functions could perform better with the right combination of hyperparameters. ProbCover performed better with lower purity levels, and ALFA-Mix performed better with a lower value e. The maximum number of clusters was set arbitrarily for TypiClust, as was the number of neighbors used to calculate typicality. Idcally, these eyper-parameters would have been determined based on the distribution of features, the number of dimensions, and the number of samples in each dataset. These observations strengthen our belief that DropQuery is a strong competitor to existing methods due to its simplicity and excellent performance without tuning M.
6 Conclusions
In this work, we systematically study four critical components of effective active learning in the context of foundation models. Given our observations, we propose new AL strategy, named DropQuery, that combines the benefits of strong initial pool selection by class centroids and a dropout-motivated uncortainty measure for querying unlabeled instances. This method outperforms other AL queries in variety of nat- ural image, biomedical, and large-scale datasets. It poses a paradigm shift to leveraging the properties of foundation model representations in AL.
Our work is a principled investigation focused the interplay of AL and foundation models, and we recognize certain limitations. Firstly, our approach presumes the availability of public foundation models, which may not be universally accessible. Secondly, inherent biases and ethical dilemmas tied to these models could be reflected in our method, possibly leading to biased outcomes or the propagation of harmful stereotypes. Thirdly, our experiments focus on datasets with a relatively even distri- bution of labels, as has been the norm with most works exploring AL queries. Given that long-tailed datasets are quite common in the real world, there is concern that our findings may not generalize to scenarios with heavily imbalanced class labels. This limitation warrants further investigation, particularly in the biomed- ical domain, where class imbalances and long-tailed distributions are widespread. Despite these issues, the growth of foundation models encourages their use in AL, even with out-of -f-maan datasets. Finally, we acknowledge that the semi-supervised learning method implemented in this study is not the only way to leverage unlabeled instances in this manner. Due to our experimental setup involving a fixed representation space generated by frozen foundation model backbones, we are constrained to subset of SSL algorithms that operate in an offline mode. However, we encourage future work investigate further along these lines and emphasize that our recommendation is limited to the use of particular label propagation algorithm. We urge future research to address these concerns while considering the broader ethical implications. ions.
Limitations and Societal Impact
12

---

Published in Transactions on Machine Learning Research (06/2024)
References
Andrea Acevedo, Anna Merino, Santiago Alférez, Ängel Molina, Laura Boldú, and José Rodellar. A dataset of microscopic peripheral blood cell images for development of automatic recognition systems. Data Brief, 10(105474) June 2020.
Josiah Aklilu and Serena Yeung. Alges: Active learning with gradient embeddings for semantic segmentation of laparoscopic surgical images. In Zachary Lipton, Rajesh Ranganath, Mark Sendak, Michael Sjoding, and Serena Yeung (eds.), Proceedings of the 7th Machine Learning for Healthcare Conference, volume 182 of Proceedings of Machine Learning Research, pp. 892-911. PMLR, 05-06 Aug 2022. URL https:
/ //proceedings.. wwr. wtt.:mmm.....e...e.eeeeeeeeeee.e www.ppressseeeeeeeeee 1at.88/kkklll22ee html
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In International Conference on Learning Representations, 2020. URL. httpp:://openrreiieww http:////ppaaaaoeeee.eeeee//r/rp nett/oorummdderrgg2BBB.2
Haoping Bai, Meng Cao, Ping Huang, and Jiulong Shan. Self-supervised semi-supervised learning for data labeling and quality evaluation, 2021.
William Beluch, Tim Genewein, Andreas Nürnberger, and Jan Kohler. The power of ensembles for active learning in image classification. pp. 9368 9377, 06 2018. doi: C0.109/CVVR..008.00000
Rishi Bommasani, Drew A. Hudson, Ebsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Autoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dal- las Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, et al. On the opportumities and risks of foundation models, 2022.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 - mining discriminative components with random forests. In European Conference Computer Vision, 2014.
Yao-Chun Chan, Mingchen Li, and Samet Oymak. On the marginal benefit of active learning: Does self- supervision eat its cake? ICASSP 2021 2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 3455-3459, 2020.
Akshay T. Chandra, Sai Vikas Desai, Chaitanya Devaguptapu, and Vineeth N. Balasubramanian. On initial pools for deep active learning. In Preregister&Neur1P9 2020.
Liangyu Chen, Yutong Bai, Siyu Huang, Yongyi Lu, Bihan Wen, Alan Yuille, and Zongwei Zhou. Making your first choice: To address cold start problem in medical active learning. In Medical Imaging with Deep Learning, 2023. URL //tperr//ppeerevee
Mchdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhnann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for cantrastive Binguageimaage learning, 2022.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Tbrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang, Carlos Riquelme, Matthias Minderer, Joan Puigcerver, Utku Evci, Manoj Kumar, Sjoerd van Steenkiste, Gamaleldin F. Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Patrick Collier, Alexey Gritsenko, Vighnesh Birodkar, Cristina Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov, Filip Pavetíć, Dustin Tran, Thomas Kipf, Mario Lučié, Xiaohua Zhai, Daniel Keysers, Jeremiah Harmsen, and Neil Houlsby. Scaling vision transformers to 22 billion parameters, 2023.
Mélanie Ducoffe and Frédéric Precioso. Active learning strategy for enn combining batchwise dropout and query-by-or boeery--b-oommttee.. In The European Symposium on Artificial Neural Networks, 2017.
Reza Zanjirani Farahani and Masoud Hekmatfar. Facility location: Concepts, Models, algorithms and case sludies. Springer- Verlag Berlin Heidelberg, 2009.
13

---

Published in Transactions on Machine Learning Research (06/2024)
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep Bayesian active learning with image data. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1183 1192. PMLR, 06 11 Aug 2017. URL
htttss://roceeedigss mlr. wttp..press///l/llll.... http:////gaaaaaaiia.aaa html
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels, 2020.
Minglei Gao, Zizhao Zhang, Guo Yu, Sercan Ó. Arik, Larry S. Davis, and Tomas Pfister. Consistency-based semi-supervised active learning: Towards minimizing labeling cost. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, pp. 510-526, Cham, 2020. Springer International Publishing. ISBN 2-0---00---..-..
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, 2017.
Guy Hacohen, Avihu Dekel, and Daphna Weinshall. Active learning on a budget: Opposite strategies suit high and low budgets. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 8175-8195. PMLR, 17-23 Jul 2022. URL
Detts://proceedingg mlr
Kaggle and EyePacs. Kaggle diabetic retinopathy detection, jul 2015. URL https://www.kagg//
Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisi- tion for deep bayesian active learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alcch-BBBu, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL https://prroeed wwwssneurippp. httpprpp.cc//ppppee.eeee/////p/p
pdf
Andreas Kirsch, Sebastian Farquhar, Parmida Atighehchian, Andrew Jesson, Frederic Sannhaud-Charron and Yarin Gal. Stochastic batch acquisition for deep active learning, 2022.
Jonathan Krauso, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fino-grained cate- gorization. In 2013 IEEE International Conference on Computer Vision Workshops, pp. 554 261,, 2013. doi: 10.1109/ICCVW.200 2013.77.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Carsten Tim Lüth, Till J. Bungert, Lukas Klein, and Paul F Jaeger. Navigating the pitfalls of active learning ovaluation: A systematic framework for mcaningful performance asscssment. In Thirty-sewentth Conference on Neural Information Processing Systems, 2023. URL
Rafid Mahmood, Sanja Fidler, and Mare T Law. Low-bulget active learning via wasserstein distance: An integer programming approach. In International Conference on Learning Representations, 2022. URL
attps://openreviee.nettt// wttp://oooo////dddvllllmm Cn235
S. Maji, Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fino-grained visual classification of aircraft. Technical report, 2013.
Prem Melville and Raymond J. Mooney. Diverse ensembles for active learning. In Proceedings of the Twenty First International Conference on Machine Learning, ICML '04, pp. 74, New York, NY, USA, 2004. Association for Computing Machinery. ISBN 1581138385. doi: 00.1145/011533000113388 URL
1015388
Prateck Munjal, Nasir Hayat, Munawar Hayat, Jamshid Sourati, and Shadab Khan. Towards robust and reproducible active learning using neural networks, 2022.
14

---

Published in Transactions on Machine Learning Research (06/2024)
R F Murphy, M V Boland, and M Velliste. Towards a systematics for protein subcelluar location: quantitative description of protein localization patterns and automated analysis of fluorescence microscope images. Proc. Int. Conf. Intell. Syst. Mol. Biol., 8:251 259, 2000.
George Nemhauser, Laurence Wolsey, and M. Fisher. An analysis of approximations for maximizing sub- modular set functions- i. Mathematical Programming, 14:265-200 294, 12 1978. doi:
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Woj- ciech Galuba, Russell Howes, Po-Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bo- janowski. Dinov2: Learning robust visual features without supervision, 2023.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Compuler Vision and Pallern Recognition, 2012.
Amin Parvaneh, Ehsan Abbasnejad, Damien Teney, Gholamreza (Reza) Haffari, Anton van den Hengel, and Javen Qinfeng Shi. Active learning by feature mixing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognátion (CVPR), pp. 12237-12246, June 2022.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Sacnko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Compuler Vision, pp. 1406-1415, 2019.
Kossar Pourahmadi, Parsa Nooralinejad, and Hamed Pirsiavash. A simple baseline for low-budget active lcarning. arxiv preprint arXiv:2110. 12033, 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kun- durthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsov. Laion-ōb: An open large-scale dataset for training next generation image-texi models, 2022.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018 URL https://openreviier wtw.pppwrwweiieeeeeoeerrroaa.a net/formn?id Hlaluk-RW
C. E. Shannon. A mathematical theory of communication. 379 379-423, 1948. doi: 10. 10.1002/j.1538-11 1948.0.1948
The Bell System Technicol Journal, 27(3):
O. Simeoni, M. Budnik, Y. Avrithis, and G. Gravier. Rethinking deep active learning: Using unlabeled data at model training. In 2020 25th International Conference on Pattern Recognation (ICPR), pp. 1220 1227, Los Alamitos, CA, USA, jan 2021. IEEE Computer Society. doi:
URL
1110////.1100///A4886666 2021. 9412716
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip scale, 2023.
Alex Tamkin, Dat Pham Nguyen, Salil Deshpande, Jesse Mu, and Noah Goodman. Active learn- ing helps pretrained models learn the intended task. In Alice H. Oh, Alekh Agarwal, Danielle Bel- grave, and Kyunghyun Cho (eds.), Advonces in Neural Information Processing Systems, 2022. URL http:/peerrvvieeeeeeeeroummmmaaa
Dustin Tran, Jeremiah Liu, Michael W. Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, Neil Band, Tim G. J. Rudner, Karan Singhal, Zachary Nado, Joost van Amers- foort, Andreas Kirsch, Rodolphe Jenatton, Nithum Thain, Honglin Yuan, Kelly Buchanan, Kevin Murphy, D. Sculley, Yarin Gal, Zoubin Ghahramani, Jasper Snoek, and Balaji Lakshminarayanan. Plex: Towards reliability using pretrained large model extensions, 2022.
15

---

Published in Transactions on Machine Learning Research (06/2024)
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The HAM10000 dataset, large collection of multi- source dermatoscopic images of common pigmented skin lesions. Sei. Dala, 5(1):180161, August 2018.
David Warde-Farley, lan J. Goodfellow, Aaron Courville, and Yoshua Bengio. An empirical analysis of dropout in piecewise linear networks, 2014.
Jae Oh Woo. Active learning in bayesian neural networks with balanced entropy learning principle. In The Eleventh International Conference on Learning Representations, 2023. URL https://openrevvew.. net/
7TT
Ofer Yehuda, Avihu Dekel, Guy Hacohen, and Daphna Weinshall. Active learning through a covering lens. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Newral Information Processing Systems, 2022. URL https://opearrevien pest//forunnid-u@pppttte
Michelle Yuan, Hsuan-Tien Lin, and Jordan Boyd-Graber. Cold-start active learning through self-supervised language modeling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7935 7948, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/20200 www.p-main...6... URL htttp:://aclantholog..ooz.
emnlp-main. 637
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucie, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. A large-scale study of representation learning with the visual task adaptation benchmark, 2020.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
16


Reviews:
[{"review_id": "AS5P6T5bBz", "reviewer": "Reviewer_Zp5y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1.\tThis paper carries out deep researches on critical components of active learning.\n\n2.\tThe advantages of the proposed strategy are verified from several angles.\n\n**Weaknesses**\n\n1.\tMy main concern is that the main idea of this article is not clear. After reading the contributions of this article, I think the main contribution is the new strategy. But in fact, not only the title, but also the main part of the article is about effects of the four critical components of effective active learning. I think authors should be clear of what they want to show us mostly and modify the structure of this article.\n\n2.\tIn section 3.2, this paper carries out a large number of experiments and shows us the results. However, this paper only gives experiments settings and lacks the analysis of the experimental results. This is also the case in sections 3.1, 3.3, and 3.4.\n\n3.\tThe findings of the experiment should be clearly stated.\n\n4.\tThe setting of parameter M in Section 4 is not clear. It should be explained why M=3 not something else.\n\n5.\tThere is a lack of comparison with baselines in the experiments in Section 5.2. The results can only tell us that DrouQuery works on biomedical datasets. But is it really better than other existing strategies?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper systematically studies four critical components of effective active learning in the context of Foundation Models. Based on the results of these researches, this paper challenges established research on the cold-start problem and counter the notion that uncertainty-based queries are ineffective in the low-budget regime. Finally, this paper introduces a new AL strategy called DropQuery.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please kindly refer to the above comments.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WHleYo4OY5", "reviewer": "Reviewer_LkyM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. This work studies AL for image classification based on pre-trained large-scale vision/vision-language models. This study can bring insights for applications of vision foundation models in domain-specific, low-data tasks.\n\n2. The evaluated datasets are comprehensive, including both natural images and out-of-domain (medical) images.\n\n3. The comparison between the proposed AL strategy DropQuery and prior methods clearly shows the advantages of DropQuery.\n\n4. The writing is generally clear and easy to follow.\n\nWeaknesses\n1. [Impact of initial pool (in the long run)] In Section 3.1, AL methods starting with a randomly sampled initial pool are compared with those starting from a centroid-based initial pool, and the results “dispel previous preliminary findings Chandra et al. (2020) that emphasize that there is no significant difference in initial pool selection strategies for AL.” However, from the reviewer’s perspective, the conclusions of this work and Chandra et al. (2020) do not seem to contradict each other. In this work, Table 1 indeed demonstrates that the centroid-based initial pool leads to greatly improved performance at the very beginning of AL (e.g., t=1). However, as the number of iterations increases, the benefits diminish or even become negative (e.g., t=8 or 16 in ImageNet-100). The observation from Chandra et al. (2020) is that “Experimental results could not conclusively prove that intelligently sampled initial pools are better for AL than random initial pools *in the long run*,” which do not seem to be very different from the results in Section 3.1. It is suggested to provide some clarification or revise the related claims to avoid misunderstandings of the conclusions in this work.\n\n2. [Utilization of unlabeled instances] In Section 3.4, it is claimed that “the efficacy of *semi-supervised learning* in this setting is questionable at best, with wide variation across query methods and datasets,” based on experiments of (an offline variant of) label propagation (Iscen et al. (2019)). However, this method of utilizing unlabeled instances is just one example of semi-supervised learning, and the conclusion for ineffective semi-supervised learning may not generalize well to other methods, such as Mean Teacher (Tarvainen and Valpola (2017)), self-training (Yalniz et al. (2019)), or Meta Pseudo Labels (Pham et al. (2021)).\n\n3. [Transition from observations to proposal] The connection between the analytical experiments (Section 3) and the proposed new AL strategy (Section 4) seems not very strong. It may be helpful to directly point out which part of DropQuery originates from the observations.\n\nAkshay L Chandra, Sai Vikas Desai, Chaitanya Devaguptapu, and Vineeth N. Balasubramanian. On initial\npools for deep active learning. In Preregister@NeurIPS, 2020.\n\nAntti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NeurIPS, 2017.\n\nI. Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, Dhruv Mahajan. Billion-scale semi-supervised learning for image classification. arXiv, 2019.\n\nHieu Pham, Zihang Dai, Qizhe Xie, Minh-Thang Luong, Quoc V. Le. Meta Pseudo Labels. In CVPR, 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work studies several design choices in Active Learning (AL) in the context of image classification based on frozen vision foundation models (e.g., DINO and CLIP), and proposes a new AL strategy, DropQuery. Through analytical experiments with DINOv2-ViT-g/14, various existing AL methods, and several image classification datasets, this work demonstrates that 1) the initial pool selection greatly influences the early stage of AL; 2) sample diversity and representativity should be emphasized in AL selection; and 3) utilization of semi-supervised learning may not be a necessary option. Based on such observations, this work proposes DropQuery, an effective AL strategy that 1) measures uncertainty of samples via dropout, 2) clusters high-priority samples, and 3) queries labels for samples closest to the cluster centroids. Experiments validate the efficacy of DropQuery on natural images and out-of-domain biomedical images.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Most of the requested changes have been illustrated above. In addition, these minor issues may be fixed:\n\n1. In the paragraph right before Section 3.1, the query budget is set as “1 sample per class per iteration.” Using the word “class” may cause some confusion, because at the time of query, the class labels are unknown. According to later sections, “1 sample per cluster” may be more suitable.\n\n2. The analytical experiments in Section 3 are all performed with DINOv2-ViT-g/14, which may lack generality. It may be helpful to show some more results considering other models and scales.\n\n3. The evaluated datasets do not seem to involve long-tail distributions, which are actually common in real-world scenarios. It is unclear if the proposed clustering-based AL strategy is still helpful on long-tail classification tasks. Some attempts on long-tail classification or discussion may be beneficial.\n\n4. Grammar Issue\n    - [Abstract] “... efficiency, but …” → “... efficiency. But …”\n    - [Introduction] “... where labeled data is scarce and …” → “... where labeled data are scarce and…”\n    - [Sec 3.1] “The results some queries like…” is grammatically incorrect.\n    - [Sec 3.4] “However, in our experiments find that” is grammatically incorrect.\n    - Additional errors throughout the manuscript necessitate thorough proofreading and correction.", "broader_impact_concerns": "No significant adverse effects are apparent from this study. On the contrary, the proposed method enhances the training environment in data-scarce situations, thereby potentially broadening the accessibility of models for individuals who have restricted access to large-scale datasets.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rK1nZOdNbr", "reviewer": "Reviewer_sEZP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper provides meaningful experiment results explaining how foundation models can help in AL. \n\n2. The proposed DropQuery is simple and effective.\n\nWeaknesses:\n1. It would be better to reorganize the paper so that more ablation study results can be presented in the main paper.\n\n2. I wonder if such a method can be applied to supervised trained backbone models? Also, can such a method be applied to non-contrastive SSL backbone models, e.g. MAE? \n\n3. Fig.2 can be improved for better understanding.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper mainly targets active learning. The authors try to explore the usage of foundation models in AL regarding several important component. Based on the finding they propose a new AL strategy which utilizes the inconsistency among predictions of input features applied with different dropout. The authors conduct several experiments to show the effectiveness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weaknesses.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper explores the integration of vision foundation models into active learning (AL) strategies, particularly focusing on low-budget scenarios. The authors evaluate the impact of using foundation models on three crucial aspects of AL: initial labeled pool selection, diversity sampling, and the balance between representativeness and uncertainty in sampling. Their findings inform a newly developed AL strategy, "DropQuery," which combines uncertainty estimation through dropout with sample diversity to enhance AL efficiency. This strategy is extensively tested across various challenging image classification benchmarks, including natural and biomedical images, demonstrating its potential to refine active learning processes in complex image datasets.

The paper is overall solid. The authors thoroughly investigate critical components of effective AL, offering new insights particularly useful for domain-specific, low-data tasks. The investigation of AL with the context of foundation models is interesting and valuable. The proposed AL strategy, DropQuery is simple yet effective across various tests. On the weakness side, several reviewers noted that the main ideas and contributions could be clearer and more distinctively articulated within the paper. The authors were able to address some concerns with their rebuttal, but the paper could further benefit from appropriate rewriting and re-organization. The authors should better address reviewer sEZP's questions with the following changes:
1. Add the MAE backbone experiment besides DINOv2.
2. Re-organize and improve the paper according to the suggestions.
Decision: Accept (poster)