Active Prompt Learning with Vision-Language Model Priors
Authors: Hoyoung Kim
Seokhee Jin
Changhwan Sung
Jaechang Kim
Jungseul Ok
Venue: Accepted by TMLR ()

Abstract: Vision-language models (VLMs) have demonstrated remarkable zero-shot performance across various classification tasks. Nonetheless, their reliance on hand-crafted text prompts for each task hinders efficient adaptation to new tasks. While prompt learning offers a promising solution, most studies focus on maximizing the utilization of given few-shot labeled datasets, often overlooking the potential of careful data selection strategies, which enable higher accuracy with fewer labeled data. This motivates us to study a budget-efficient active prompt learning framework. Specifically, we introduce a class-guided clustering that leverages the pre-trained image and text encoders of VLMs, thereby enabling our cluster-balanced acquisition function from the initial round of active learning. Furthermore, considering the substantial class-wise variance in confidence exhibited by VLMs, we propose a budget-saving selective querying based on adaptive class-wise thresholds. Extensive experiments in active learning scenarios across seven datasets demonstrate that our method outperforms existing baselines.

Published in Transactions on Machine Learning Research (10/2025)
Active Prompt Learning with Vision-Language Model Priors
Hoyoung Kim
hoyoung. kim Spoteechhhccccckk
Graduate School of Artificial Intelligence
POSTECH
Seokhee Jin
jin74498pstecch.oo oc.kr
Graduate School of Artificial Intelligence POSTECH
Changhwan Sung
Shanghhien..unngSppsttee...
Graduate School of Artificial Intelligence
POSTECH
Jacchang Kim
juochanggposstech..cc
Graduate School of Artificial Intelligence
POSTECH
Jungseul Ok
jungseul ok aokkppsttcch. oc.kr
Graduate School of Artificial Intelligence
POSTECH
Reviewed on OpenReview: https: /ttp:e///perreveeee wew/ forum? id- qBeGC=D3IJ
Abstract
Vision-language models (VLMs) have demonstrated remarkable zero-shot performance across various classification tasks. Nonetheless, their reliance on hand-crafted text prompts for each task hinders efficient adaptation to new tasks. While prompt learning offers promising solution, most studies focus maximizing the utilization of given few-shot la- beled datasets, often overlooking the potential of careful data selection strategies, which enable higher accuracy with fewer labeled data. This motivates us to study budget- efficient active prompt learning framework. Specifically, we introduce class-guided cluster- ing that leverages the pre-trained image and text encoders of VLMs, theroby enabling our balanced acquisition function from the initial round of active learning. Furthermore, considering the substantial class-wise variance in confidence exhibited by VLMs, we propose budget-saving selective querying based adaptive dass-wise thresholds. Extensive ex- periments in active learning scenarios across seven dalasets demonstrate that our method outperforms existing baselines.
1 Introduction
Vision-language models (VLMs), such as CLIP (Radford al., 2021) and ALIGN (Jia et al., 2021), have demonstrated impressive zero-shot capabilities across various downstream tasks, including object detec- tion (Du et al., 2022; Feng et al., 2022; Zhong et al., 2022), semantic segmentation (Yi et al., 2023; Ghiasi et al., 2022; Li et al., 2022), and image classification (Radford et al., 2021; Singh et al., 2022; Zhai et al., 2022), by aligning visual and textual information within a shared representation space (Radford et al., 2021; Jia et. al., 2021; Yuan et al., 2021). Nevertheless, as VLMs rely on manually crafted text prompls for each task, which can tine-consuming and labor- intensive,, efficiently adapting VLMs to new tasks remains crucial. Prompt learning has emerged as promising solution, particularly for image classification tasks, al- lowing VLMs to learn taskfsp prompts without the computational burden of directly fine- fino-tuning image and text encoders.

---

Published in Transactions on Machine Learning Research (10/2025)
(a) Class-guided clustering (Section 4.1)
(b) Cluster-balanced sampling (Section 4.2)
Learnable prompts
(CLASSi
Test
Weighted
T2
Fiji
features
dog
plane
K-maan chotering
Image encoder
F-73
Round 1
Round 2 [K
(c) Selective querying (Section 4.3)
M
Class-mise
thresholds
T
=
FC
Á "
(4.F =
Annotation
2)
(A.y=1
heatmap
Class-guidder features
Figure 1: An overview of the proposed framework. (a) Class Class-guided features Fc are obtained by concatenating the image features I with the weighted text features Te, using similarity scores weights. In the heatmaps, Fe focus the guided-classes C (Cat, Dog} than I. (b) K-means clustering is performed on Fc. With increasing K, cluster-balanced sampling becomes available in each round. (c) The confidence scores of previously labeled data (circles) serve as thresholds for new candidates (triangles). If a candidate's confidence exceeds its corresponding threshold, a pseudo- is assigned to conserve the budget, otherwise it is labeled by annotators.
Generally, prompt learning methods have focused on model-centric approaches, modifying prompt archi- (ectures and learning objectives. Specifically, researchers have introduced various prompt types, including text prompts (Zhou et al., 2022b), image-conditioned text prompts (Zhou et al., 2022a) for text encoders, and multimodal prompts that work across both image and text encoders (Khattak et al., 2023a). ). In terms of learming objectives, prompts are initially trained with cross-entropy loss (Zhou et al., 2022b), supple- mented by regularization terms to maintain CLIP's general knowledge (Zhu et al., 2023) and incorporate task-agnostic knowledge (Park et al., 2024), which helps prevent overfitting on specific tasks. However, these model-centric methods mainly focus on leveraging VLM priors to optimize prompts on given few-shot la- beled datasets, often overlooking the potential of data selection to achieve higher accuracy with fewer labeled samples.
In contrast to previous model-centric approaches (Zhou et al., 2022b; Zhu et al., 2023), adopt data-centric perspective by explicitly leveraging VLM priors to select informative data. In this context, active learning offers an alternative by prioritizing the labeling of the most informative images with minimal budgets. Recent data selection crucial
work on active prompt learning (Bang et al., 2024) highlights that class-balande for mitigating the inherent imbalanced knowledge within VLMs. However, this approach merely adheres the conventional few-shot datasets, disregarding the opportunity fully utilize VLM priors. In contrast, we fully exploit VLM priors through our proposed class-guided clustering and selective querying methods.
For a class-guided clustering, we leverage the pre-trained image and text encoder of VLMss In detail, we first derive class-guided features by concatenating two components: (i) image features from the image encoder and (ii) text features computed as a weighted sum of each class's text features, with weights based on similarity scores the image features. We then apply K-means clustering (MacQueen et. al., 1967) on these class-guided fcatures to achieve balanced data selection across clusters. While traditional active learning often encounters the cold-start problem due to a lack of reliable methods evaluate data in the initial round (Mahmood et al., 2021; Chen et al., 2023), our cluster- balanced acquisition function provide the benefit of a warm-start. However, purely diversity-focuse acquisitions may inefficiently allocate budgets to data in which VLMs are already confident.
To address this issue, we introduce a budget-saving selective querying based on adaptive class-wiss thresholds. Since VLMS often exhibit substantial variance confidence across different downstream tasks and even among individual classes (Bang et al., 2024), we implement adaptive class-wise thresholds without adding extra hyperparameters. Specifically, assign pseudo-labels to the data selected by the class-balanced
2

---

Published in Transactions on Machine Learning Research (10/2025)
acquisition when their confidence scores exceed the corresponding threshold. As result, we conserve budget in each round rather than exhausting it entirely.
The proposed framework, illustrated in Figure 1, fully leverages VLM priors to enable efficient adaptation across various classification tasks. Beyond simply employing the two encoders of CLIP (Radford et al., 2021) for clustering (T.i et al., 2024), we further analyze the advantages of incorporating the weighted text features through visualization tools, such as GradCAM (Selvaraju al., 2017) in Figure and T-SNE (Van der Maaten & Hinton, 2008) in Figure 4. Extensive experiments in active learning scenarios demonstrate that our Aluster-balaneee acquisition with selective querying outperforms other baselines.
Our main contributions are summarized as follows:
We propose a budget-cfficient active prompt learning for VLMs, particularly on CLIP, where the class-guided clustering and the selective querying fully leverage VLM priors (Sections 4.1 and 4.3).
We provide in-depth analysos of the class-guided features and clustoring with GradFAM, a variant of GradCAM, and T-SNE, respectively (Figures 2 and 4).
Experiments demonstrate that our method achieve superior budget efficiency and performance across diverse active learning scenarios (Sections 5.2).
We explore the potential for extending of our data-centric approach into existing model-centric prompt learning methods (Section 5.3).
2 Related Work
Prompt learning in vision-language models. To address the inefficiency of fine-tuning all VLM param- eters, CoOp (Zhou et al., 2022b) proposes prompt learning focused on compact prompts for efficient adapta- tion. Subsequent works (Zhon et al., 2022a; Khattak et al., 2023a;b; Park et al., 2024; Li et al., 2023) have further developed CoOp, adopting model-centric approaches such as modifying prompt architectures (Khat- tak et al., 2023a) and learning objectives (Zhu et al., 2023). For instance, MaPle (Khattak et al., 2023a) incorporates multi-modal prompts that jointly consider both VLM encoders, while ProGrad (Zhu et al., 2023) introduces an auxiliary loss term to maintain general knowledge of CLIP. In contrast, PCB (Bang et al., 2024) efficiently adapts VLMs from a data-centric perspective by employing an active learning with a acquisition function. While PCB's acquisition focuses on mitigating the imbalanced prior knowledge of VLMs, fully exploit VLM priors across the proposed method.
pseudo
Active learning in the era of foundation models. We are in an era where foundation models, such as CLIP (Radford et al., 2021), SAM (Kirillov et al., 2023), and GPT-4 (OpenAT, 2023), dominate a wide range of downstream tasks. Their impressive generalization capabilities may imply that the role of active learning is becoming less significant. However, recent studies continue leveraging the prior knowledge embedded in foundation models (Kim al., 2024; Guple et al., 2024; Bayer & Reuter, 2024; Wan et al., 2023) to further enhance budget efficiency with active learning. For instance, ALC (Kim al., 2024) introduces correction queries to refine SAM-generated labels in semantic segmentation tasks, while ActiveLLM (Bayer & Reuter, 2024) addresses the cold start problem by utilizing LLMs in text classification tasks. We fully leverage VLM priors to enhance budget efficiency in active learning for image classification tasks.
Acquisition functions in active learning. Active learning employs various acquisition functions to identify the most informative samples for annotation, aiming to maximize model performance within a constrained budget. These acquisitions are broadly categorized into those focused uncertainty (Asghar et al., 2017; He et al., 2019; Ostapuk et al., 2019; Fuchsgruber et al., 2024), diversity (Sener & Savarese, 2018; Sinha et al., 2019; Ychuda et al., 2022), and both (Ash et al., 2020a; Hwang et al., 2022; Kim et al., 2023; Wang el al., 2019; Hwang et al., 2023). Recent studies show that uncertainly-based acquisitions are more effective with higher budgets, while diversity- based ones perform better with lower budgets (Hacohen et al., 2022; Hacohen & Weinshall, 2023a;b). Building on this, we propose a cluster-balaneee acquisition to adapt VLMs within limited budgets. In addition, we introduce a selective querying with adaptive class-wise thresholds to further conserve budget.
3

---

Published in Transactions on Machine Learning Research (10/2025)
Algorithm Proposed Active Prompt Learning
Require: Image set I, initial prompts to. budget per round B, and the number of rounds R
1: for r r= 1,......., do
2:
Extract class-guided features Vi E I by combining image and weighted text features via (7)
3: Perform K-means clustering on class-guided features
Select representative images from each cluster as candidates via (9)
Compute class-wis, confidence thresholds using previously labeled data via. (11)
Construct dataset D, by assigning pscudo ground-truth labcls via selective querying in (12)
7: Reinitialize and train prompis 4 using datasel D, with the objective in (13)
8: end for
9: return Final dataset Dr and trained prompts tn
3 Preliminaries
For efficient adaptation in vision-languant models (VLMs), we fully leverage their prior knowledge. Before presenting our method, we first outline the structure of VLMs and the relevant priors, in Section 3.1, which are utilized in our method. After that, we describe the basics of prompt learning with fow-shot datasets in Section 3.2.
3.1 VLM Priors
Pre-trained VLMs have shown decent zero-shot performance across various classification tasks. Specifically, the CLIP model 0 (Radford et al., 2021) comprises an image encoder Oung and a text encoder O performing zero-shot inference on an image for a target class C using cosine similarity as:
(op(cs(0mmm(1) Pxxe (lxxo(tt))))))) (tc))/T)
Po =0==,1;;tC) x; Σkec exp(cos(( (0 (I), Pxx (tx))/T)
where the text prompt can be set as "a photo of a ", and represents the concatenation of t and "[CLASS). with the class token replaced by the class name Here, denotes temperature parameter. Considering the highest probability among all classes, we can oblain the pseudo label of an image æ follows:
yo(x: x:; t, C) == arg pax = r;t,C). aEC
(2)
We note that the pseudo label depends on the text prompt and the target class set Section 4, we leverage the pre-trained image and text encoders of CLIP for our class-guided clustering and the pseudo label for our selective querying.
3.2 Prompt Learning in VLMs
Vision-language models (VLMs) contain numerous parameters, which makes fine-tuning on small labeled dataset impractical. Recently, prompt learning has facilitated efficient adaptation in VLMs by freezing the image and text encoders and focusing on learning input prompts. For instance, CoOp (Zhou et al., 2022b) introduces learnable vectors into the text prompt te of class c, replacing the conventional "a photo of a [CLASS]:" text prompts with:
to = [V][[]] ... [Vv][CLASS].
(3)
where each V represents learnable vector and M denotes the number of vectors. These vectors in t are trained dataset D by minimizing the cross-entropy (CE) loss:
B(z.)) ((:,
(4)
In this context, previous prompt learning methods have primarily taken a model-centric perspective, aiming to maximize model performance the given training dataset D. From data-centric perspective, however,
4

---

Published in Transactions on Machine Learning Research (10/2025)
(a) Image
(b)
(c) FC=[Cat,
(d) FC=(Ball
(e) FC=(Red (Red,
Figure 2: GradFAM with various target features. (b) All objects in the image significantly impacts the target image features I. (c-e) With our class-guided features Fe for target features, the heatmap aligns with the
larget classes C. Further details are in the Appendix A.
building datasets requires human labor, making active learning essential for creating information-dense datasets while minimizing interactions with annotators. addition, we note that our data-centric approach is compatible with existing model-centric prompt learning methods.
4 Proposed Active Prompt Learning
Given an unlabeled image set U, we consider a general active learning scenario for image classification tasks, where annotators are asked to label each image U with its accurate label y E 21, Cj. For each round with maximum budget of B, begin by constructing candidate set of B images from U, utilizing class-guided clustering. Fach candidate is then evaluated with wass threshold to determine whether to acquire ground-truth labels from annotators or use pscudo labels to conserve budget. Finally, learnable prompts 4, are trained by prompt learning on the dalaset D, accumulated to round r.
In the followings, we introduce a class-guided clustering, which leverages the pre-trained image and text encoders of CLIP (Section 4.1), and a blaaner-balancod acquisition function seleet candidate images (Soc- tion 4.2). After that, we introduce budget-saving selective querying (Section 4.3). The overall algorithm is described in Algorithm 1.
4.1 Guided Clustering
Conventional active learning often relies on random sampling construct the initial dataset, which can lead to the cold-start problem. In contrast, we fully utilize CLIP's pre-trained image and text encoders for class-guided clustering, which can be applied al each round of the active learning process. To construct class-guided features of an image we first define its image and text features, [(x) and Tc(x), respectively. For the image features, we simply use the image encoder of CLIP as follows:
I(x) == 0ang (x)
(5)
To derive the text features of x, we compute a weighted combination of text features with soft labels as weights:
Te(x;t,-1)= = Σ Po cl x., 1 (tr-
(6)
CEC

---

Published in Transactions on Machine Learning Research (10/2025)
where 1-1 represents the learned vectors from the previous round, and 1,-1,c concatenates t-- with class as described in (3). For the initial round, we set 4--1 as "a photo of ". Finally, we concatenate the image and text features to obtain the class-guided features Fc(x) for image x as:
Fe(x;tr-1) [1(x), (c(=:t,-)))
(7)
After that, we apply the K-means clustering algorithm (MacQueen et al., 1967) on the set of class-guus gass--guidd features across all images to partition them into K clusters.
GradFAM on class-guided features. We modily GradCAM (Selvaraju et al., 2017) into GradFAM (Gradient-weightee Feature Activation Mapping) to visualize the influence of class-guided features an image. GradCAM highlights the degree to which individual pixels contribute to a specific class by analyzing the gradients of the class score with respect to the feature maps. To tailor GradCAM for VLMs, we introduce the concept of target features denoted by Fearget; allowing GradCAM to visualize the influence of target features on an image based on the cosine similarity score defined as cos([(x), Flarge)) Our analysis technique, called GradFAM, visualizes the influence of larget features rather than larget class, offering the advantage of enabling label-free analysis. Figure 2b represents the case where Ftarget [(x). Consistent with CLIP's approach of embodding images and texts into shared space, we observe that the influcnce of the larget image features primarily concentrates on the overall objects within the image. On the other hand, when the target features are guided by the set of classes, i.e. Fearget = Fe(;;t t) where is "a photo of a Figures 2c to 2e distinctly highlight the specific objects corresponding to the guiding class sets. To compute cosine similarity, we concatenate two copies of the same image features to match the dimensionality of the other features. We note that our class-guided features incorporates class-relevant information through the text encoder, potentially resulting in clustering that aligns more closely with the target classifier, described in Figure 4.
4.2 Cluster-Balancee Acquisition Function
For ease of explanation, we first outline the data selection process in the initial round of active learning. For Shuster-balanced acquisition, we set the number of clusters K equal to the budget B, allowing for the selection of one image per cluster. To choose the most representative sample from each cluster C₁, we first calculate its centroid e; as follows:
C =
Σ Fe(a;1).
(8)
C.
ZEC
The closest image of to the centroid is then selected as:
x = arg min [|F(( - cill.
(9)
zEC;
We can construct the set of candidate images for querying:
Q = (2..,
(10)
In the initial round, we consume the entire budget B to request annotations for all candidate images in Q. Here, t is replaced by t,-1, and by 2, for a subsequent round r.
Subsequent rounds with increasing K. To enhance the diversity in the selected data, introduce progressively increasing K based on round r, é.e. K = B x r. This lincar increase in K ensures that at least B clusters remain unlabeled in each round, allowing for the selection of clusters not included in previous rounds. However, due to the inaccuracy of clustering in earlier rounds, samples previously assigned to different clusters may now be classified into the same cluster. To address these cases, we prioritize larger clusters by sorting all unlabeled clusters by size and selecting the top-K clusters. Ablation studies on K are in the Appendix C.
6

---

Published in Transactions on Machine Learning Research (10/2025)
4.3 Selective Querying
In each round r, can allocate the entire budget B to acquire labels for all candidate images in Qr- However, CLIP has demonstrated decent zero-shot performance in downstream classification tasks (Radford et al., 2021) and performs even better with a few labeled samples (Zhou et al., 2022b). Therefore, for a candidate image where CLIP already sufficiently confident in its label, we skip manual labeling and apply a pseudo label to conserve budget.
Since CLIP's knowledge imbalanced across classes in classification tasks (Bang al., 2024), we propose a selective querying with class-wiss wise thresholds. To this end, leverage the confidence scores of images from the previous training dataset 1, In round r, the threshold 6re for class E C is computed as follows:
Cr,o = 1. (2.g=c)ED,-L.e
Σ
poly 1,,t,-1,))
(11)
where 1, represents the subset of the training dataset. labeled as c. Note that thresholds for round depend on prior information, including D,-1 and t--1. Therefore, selective querying is impossible in the initial round. We finally apply pseudo labels to candidates if their confidence exceeds the corresponding threshold. As a result, the dataset D, at round r can be constructed as follows:
D. [ x E 2r, Po x;,r-1,C) C) < G. UD,-1
U ((r x E Ωr, pa x;; x;4-1, 2 $
(12)
where the pseudo label j = ye(rtt, required is actually lower thanks to our selective querying.
t, C) is defined as in (2). Here, we note that [D,] = B x yet the budget
To avoid bias in prompts trained during the previous round, we reinitialize 4, randomly at each round and train prompts by minimizing the following CE loss:
E BLR.NWDD, (o
(13)
Revisiting a unified prompt for prompt learning. Recent studies in prompt learning for VLMs have introduced sophisticated prompt designs, including image-wise (Zhou et. al., 2022a; Yao et al., 2024) and class-wise prompts (Zhou et al,, 2022b; Bang et al., 2024; Yao et al., 2024). However, these prompts are prone to overfitting, particularly in active learning scenarios where only limited number of samples are available. To mitigate this issue, we introduce new similarity measure that incorporates both unified prompt tu and class-wise prompts te = {te c E C). Specifically, we define the cosine similarity between image x and class c as:
cos(Onmg (2)) Oxx (ln,e) + cos(Ommg(z)) Ouxs (rxxm())))x
(14)
where tue concatenates tu with class We demonstrate that incorporating unified prompt is beneficial in active learning scenarios with limited resources and enhances the effectiveness of the proposed selective querying.
5
Experiments
5.1
Experimental Setup
Datasets and implementation details. Following a previous study (Bang et al., 2024), we use seven publicly available image classification datasets: OxfordPets (pet species) (Parkhi et al., 2012), FGVCAireraft (aircraft types) (Maji et al., 2013), Caltech101 (general object categories) (Fei-Fei et al., 2004), Flowers (flower species) (Nilsback & Zisserman, 2008), DTD (texture patterns) (Cimpoi et al., 2014), StanfordCars (car models) (Krause al., 2013), and EuroSAT (satellite land cover types) (Helber al., 2019). In our experiments, employ CLIP ViT-B/32 (Dosovitskiy et al., 2021; Radford et al., 2021) our VLM. At each
7

---

Published in Transactions on Machine Learning Research (10/2025)
Random
Entrupy
CoroSee
BADGE
PCB
CB-SQ (ours)
75
70
90
80
(%)
65
Accuracy (%)
Accuracy (%)
80
60
Ancuracy
70
55
70
60
15
40
50
75
100
25
50
100
25
75
100
50
75
Cumulative Budget (%)
Oumulative Budget (%)
Cumulative Budget (%)
Cumulative Budget (%)
(a) Average
(b) Flowers102
(c) DTD
(d) OxfordPets
35
70
80
90
Accuracy (96)
Ancuracy (%)
Accuracy (5%)
30
Accuracy (%)
80
50
60
20
40
50
30
60
50
75
100
25
50
100
0
25
75
100
50
75
100
Cumulative Budget (%)
Cumulative Budget (%6
Cumulative Budget (%)
Cumulative Budget (%)
(e) StanfordCars
(f) Caltech101
(g) FGVCAircraft
(h) EuroSAT
Figure Effect of the proposed method. (a-h) Our CB+SQ outperforms the other baselines in average
performance across datasets. Experiments with comparable budget are provided in Figure 11.
round r, we reinitialize the Icarnable prompts t,, consisting of 16 vectors, using a Gaussian distribution with mean of 0 and standard deviation of 0.02. Following the training details in CoOp (Zhou et al., 2022b), we train these prompts for 200 epochs per round using the SGD optimizer, initialized with a learning rate of 0.002 and decaying according to a cosine annealing.
5.2 Active Learning Scenario
Baselines. Our cluster-balanced acquisition with selective querying (CB+SQ) is compared with the state- of-the-art (SOTA) active prompt learning method for VLMs, known as pseudo-class balance (PCB) (Bang et al., 2024), which operates based on BADGE (Ash al., 2020b). In addition, we compare with conventional acquisitions commonly used in active learning for classification tasks, including Random, Entropy Holub et al. (2008) and CoreSet (Sener & Savarese, 2018). To ensure fair reproduction of all baseline results, we follow the experimental setting of PCB, leveraging class descriptions from LLMs (Menon & Vondrick, 2022) to train class-wisc text prompts following CoOp (Zhou et al., 2022b).
Evaluation protocol. For a fair comparison, we follow the active learning sconario established in PCB (Bang et al., 2024). Specifically, experiments are conducted over rounds, with the maximum budgel per round set the number of classes, é.e. (C). Since the total budget varies across datasets, we to be fully spent at 100% by the 8th round, with maximum budget of 12.5% per round. Hcre, budget of one indicates that oracle assigns the ground-truth label for single image. Thanks to our selective querying in CB-SQ, we generally consume less budget per round than other baselines from the second round onward. We report the average accuracy over three trials, with shaded regions in the graphs representing the standard deviation.
Effect of proposed method. Figure 3 demonstrates the effectiveness of our proposed acquisition CB+SQ across seven datasets. While other baselines rely on Random acquisition mitigate the cold-start problem, i.e. initial performance degradation compared to Random, our CB+5Q leverages the pre-trained image and
8

---

Published in Transactions on Machine Learning Research (10/2025)
Table 1: Synergy of the proposed acquisition with ezisting prompt learning methods. All baseline methods are trained with 1-shot datasets. Our CB*-based curated datasets enhance the performance of previous model- centric prompt learning methods.
Methods
Flowers102
DTD
OxfordPets
StanfordCars Caltech101
Aireraft
EuroSAT
Average (%)
MaPle
75.23
48.77
85.27
57.30
91.37
18.33
61.67
62.59
+ CB
78.80
46.47
87.23
55.70
90.17
16.00
73.50
63.98
CB
80.70
51.07
88.83
60.47
93.07
20.85
71.07
66.58
PromptSRC
77.60
51.53
89.60
63.67
93.10
18.67
66.93
65.87
+ CB
79.73
50.87
89.80
61.30
92.40
22.03
70.20
66.62
CB"
80.43
56.17
90.13
63.97
93.97
21.27
70.33
68.04
ProMetaR
78.87
52.07
87.63
60.60
92.53
19.50
68.13
65.62
49.10
89.53
59.37
91.80
20.37
71.80
66.06
CB
83.37
55.20
89.53
63.83
93.90
22.10
72.87
ART: 43.2798
ARI: 56.57%
ARII 12. 7.0%
ARD: 37 64%
(a) Image features
(b) Class-guided features
(c) Image features
(d) Class-guided features
Figure 4: T-SNE for class-guided clustering. (a, c) Clustering based solely image features results in clusters that poorly separated. (a) and (c) correspond to the |C| 2 and |C] 20 settings, respectively. (b, d) In contrast, our class-guided clustering leads to more distinct clusters that align with the size of the guiding class set C.
text encoders of CLIP to enable warm-start, leading consistently strong early performance. As shown in Figure 3a, our method shows 19.5%p performance gain over the basclines the first acquisition round. Notably, with only [C queried samples, our CB-SQ already outperforms other baselines trained with 3/C] samples, highlighting its sample efficiency. In addition, our selective querying allows CB+SQ to outperform other baselines while reducing the labeling budget by 17.6%. These gains are consistent across both fine- grained and general datasets, and we also observe the same phenomenon reported in Bang et al. (2024), where conventional acquisition methods such as Entropy and CoreSet underperform even Random sampling.
5.3 Further Analyses
Extensions to SOTA model-centric prompt learning methods. Our cluster-balanced (CB) acquisition effectively selects informative images by utilizing a pre-trained CLIP model. These selected images can directly applied to enhance the performance of SOTA model-centric prompt learning methods, including MaPle (Khattak et al., 2023a), PromptSRC (Khattak al., 2023b), and ProMetal (Park et al., 2024). Table shows that our CB- basdd datasets slightly outperforms the conventional 1-shot datasets, which contain one labeled image per class. More specifically, following the few-shot setting of CoOp (Zhou et al,, 2022b), we assume access to ground-truth labels to construct perfectly class-bale hlass-balanced 1-shot dataset by randomly selecting one sample from each class. However, the 1-shot datasets differ from active learning, which start with unlabeled images. For a fair comparison with them, we construct CB*-base -bseb datasets, where the weighted feat in (6) are replaced with the text features of the ground-truth label. In Table 1, our cB*-basod datasets outperform other basclines, emphasizing the importance of data-centric approaches for efficient adaptation in VLMss.

---

Published in Transactions on Machine Learning Research (10/2025)
75
75
Accuracy (%)
Accuracy (%6
Accuracy (%)
65
6G
72
CB CB-S0Q
-o- CB (C) + CB+SQ (C)
. PCB (C)
PCB+SQ (C)
-0- CB (C) -a- CB+ (C)
CoreSet
Entrope
Entrogy
69
100
Cumulative Budget (%)
Cumulative Budget (%)
Cumulative Budget (%)
(a) Diversity-based methods
(b) Uncertainty-based methods
(c) Prompt types
Figure Ablation studies on selective querying. Each vertical dotted line indicates the effect of selective querying (SQ). U, C, and B denote unifidd prompt, class-wist wise prompts, and both prompts. SQ is more effective for (a) diversity-basod acquisitions, such as our CB and CoreSet, compared to (b) uncortainty-based methods. (c) The benefit of SQ becomes more pronounced when unified prompts (U) are used.
40
Untraiued Trained
Ustraine
Tiiind
Count
Count
20
20
10
0.2
0.6
0.8
0.2
0.4
0.6
0.8
p(y ang fe. C)
P(y (x
(a) Class-wise prompts (tc)
(b) Unified prompt (tw.)
Figure 6: Confidence distributions. Due to overfitting in (a) the class-wise prompls, the overall confidence scores and the mean confidence (dotted line) of trained samples are significantly higher than those obtained with (b) the unified prompt.
Visualization for class-guided clustering. Section 4.1, we analyze the proposed class-guided features using GradFAM, as shown in Figure 2. Here, we further visualize the impact of class-guided features clustering with T-SNE (Van der Maaten & Hinton, 2008) the Waterbirds dataset (Sagawa. et al., 2019). Specifically, the Waterbirds dataset consists of 200 distinct bird species, where each image is labeled by habitat (water, land), background (water, land), and specific species. This allows uss to categorize the dataset into 2, 4, or 200 groups. For improved visual clarity, we analyze a subsample of 20 classes. Figures 4a and 4c shows that the limitation of relying solely on image features in capturing semantic information. contrast, our class-guided features, which incorporate the set of classes C, effectively reflect relevant class information in the clustering results, as illustrated in Figures 4b and 4d. This improvement is also reflected in the Adjusted Rand Index (ARI), where the score increases from 12.74% to 37.64% when moving from image-only features to class-guided features. More details are provided in Appendix C.
Synergy of selective querying with our method. Our selective querying (SQ) synergizes strongly with our method for two key reasons. First, our cluster- cluster-blancee acquisition follows a diversity-driven strategy, which may occasionally select well-understood samples. In such cases, SQ leverages pscudo labels to conserve budget. As shown in Figure 5a, SQ is particularly effective in diversity-based methods such CoreSet and CB, while uncertainty-based approaches like Entropy and PCB show limited improvement in Figure 5h. Second, we
10

---

Published in Transactions on Machine Learning Research (10/2025)
Table 2: Ablation studies on different feature spaces. In the initial round, our class-guidet features (third
row), leveraging both image and text features, demonstrate effectiveness across 7 datascts.
Image Features
Text: Features
Ground-truth
Average Acc. (%)
52. 52.72+0.37
soft
X
18.24
son
X
55.92 40.38
V
18.10.0.44
label
V
58.27 10.43
incorporate a unified prompt, as described in Section 4.3, further enhancing the effectiveness of SQ. Figure 6 shows that the unified prompt results confidence scores for trained samples with less overfitting compared to class-wise prompts, leading to more reliable class-wise thresholds in (11). This observation aligns with the results in Figure 5c, showing that unified prompt is more effective in reducing the labeling budget than class-wise prompts.
Confidence distributions by prompt type. Figure 6 compares the confidence distributions of trained and untrained samples. We first train either class-wise prompts le or a unified prompt ly on the StanfordCars dataset D (Krause et al., 2013). To emulate an active-learning setting with a limited budget, we form a 4-shot training subset D, D. For each class e, we then compute the average confidence score St,e as follows:
St,c
|Dr.cl
Σ ply 1,,,C), (r.y)ED,,
(15)
where Dr. is the portion of Dr that belongs to class c, and is either te or tu- We then plot 50-bin histograms of these scores, using blue for the training subset Dj and red for the remaining data D De. The resulting plots show that class-wise prompts concentrate confidence seen samples, which may indicate overfitting, whereas the unified prompt yields a more balanced confidence distribution across unseen data.
we examine the effect of various feature spaces on our dluster-bllanee acquisition, as an alternative to the proposed class-guided features that leverage both Image and Text features. Table 2 demonstrates that our class-guided features yield 3.2%p and 7.7%p improvement over using only Image and only Text features, respectively. In addition, experiment with conventional few-shot labeled dataset that relies solely on Labels for perfectly class-balanced dataset, which can be impractical in an active learning scenario. For fair comparison, we redefine the weighted text features in (6) as the text features of ground-truth labels, resulting in an average performance increase of 10.2%p.
Effect various feature spaces. In Table 2,
Analyses of constructed datasets. A primary goal of active learning is construct high-quality datasets with minimal labeling budgets. Figure 7 demonstrates that our method builds high-quality datasets with a significantly smaller budget compared to previous methods. Specifically, while previous methods consume the entire budget each round to generate 100% clean dataset, our selective querying conserves budget by using pscudo-labels based on class-wisc thresholds, yet still achieves comparable level of datasct quality. This results in performance on par with previous methods, as shown in Figure: 3. Additionally, we note that as rounds progress, the budget-saving advantage our method becomes more pronounced.
ImageNet experiments. ImageNet (Deng et al., 2009) contains 1.28 M training images, making large- scale active learning computationally demanding. Emam et al. (2021) reports that CoreSet and BADGE are infeasible at this scale, and that PCB, which builds on BADGE, sulfers the same limitation. Our method overcomes this bottleneck by employing a lightweight. K-means clustering step. As shown in Figure 8, the resulting CB+SQ strategy scales efficiently to ImageNet and achieves higher accuracy than all competing baselines.
Base-to-novel generalization. Following previous work (Zhou et al., 2022b), we divide each dataset's classes into base and novel groups. We then use acquisition functions to select subset of unlabeled images
11

---

Published in Transactions on Machine Learning Research (10/2025)
65
100
100
Corroct label ratio (%)
55
Accuracy (%6
Ratio (%)
95
95
45
90
90
83
CRR+SS SSQ
CB+SQ Hnoelinns
CB-SQ
Randonn
Baselinns
25
2
25
30
75
100
Rounde
Rounds
Cumulative Budget (%)
(a) Required budget
(b) Dataset quality
(a) ImageNet
Figure 7: Analyses of constructed datasets. As rounds progress, construct datasets with progressively smaller bud- gets. (b) The quality of the constructed dataset remains consis- tently high, regardless the number rounds.
Figure 8: ImageNet ezperiments. Our approach is applicable to Im- ageNet and outperforms existing methods.
Table 3: Effecliceness on wnseen classes. Our CB shows strong performance novel classes in the initial round across 9 datasets.
Methods
Base
Novel
HM
Random
59.02+1.06
18.94
58.98 38.98+1..14
CB
68.81
63.68+1..0 +1.25
66.15
from the base classes to train prompts, which are subsequently applied to the novel classes for evaluation. In Table 3, we compare our cluster-balanced (CB) acquisition with Random reflecting the cold-start of other acquisitions in Figure: 3. Table demonstrates that our CB acquisition outperforms Random both base and novel groups. Here, HM denotes the harmonic mean of base and novel performance.
Various ablation studies. Figure 9a examines the effect of different prompt types, including unified (U), class-wise (C), and both (B), revealing that our CB consistently outperforms PCB regardless of the prompt type. In addition, we analyze the contribution of the proposed components: (i) selective querying (SQ) and (ii) the unified prompt (C to B), as illustrated in Figures 9b and 9c. Although CoreSet has demonstrated low performance active learning for VLMs (Bang ot al., 2024), this primarily due to its reliance solely on image features. We observe that incorporating our class-guided features can enhance its performance to some extent as shown in Figure 9c, though still fails to surpass PCB. We note that active learning for conventional neural networks with image-only encoders may differ from active learning for VLMs.
6 Conclusion
In this work, we propose an active prompt learning framework that leverages the prior knowledge vision- language models (VLMs) for efficient data acquisition and prompt adaptation. Using the pre-trained image and text encoders of CLIP, we extract class-guided features that combine image embeddings with weighted text embeddings based similarity scores, enabling K-means clustering to select diverse and representative samples. To further enhance budget efficiency, we introduce selective querying strategy with adaptive class-wise thresholds, assigning pseudo-labels to high-confidence samples while requesting annotations for uncertain ones. Experiments across seven datasets and large-scale settings such as ImageNet show that our chuster-blancee acquisition with selective querying reduces annotation costs and achieves superior accuracy compared to baselines. Furthermore, our data-centric approach complements existing cenel-centric prompt learning methods, offering general strategy for scalable VLM adaptation.
12

---

Published in Transactions on Machine Learning Research (10/2025)
75
Accuracy (96)
(56
70
Accuracy (%)
70
63
65
CB (B)
(B)
60
-a- CB (C)
CB4SQ (B)
65
CB (V)
(C)
55
CB-SQ (C)
CB (C)
55
Coreea (F))
PCB (C)
PCB (U)
CoreSet
Cumulative Budget (%)
Cumulative Budget (%)
Camulative Budget (%)
(a) Ablation on prompt types
(b) Ablation on components
(c) Ablation on CoreSet.
Figure 9: Various ablation studies. (a) Our CB outperforms PCB regardless of whether unified prompt (U). wise prompts (C), or both (B) are used.
(b) Even with one component removed, our method still outperforms PCB. (c) The performance of CoreSet improves with our class-guided features.
Limitations and future work. Our framework relies strong pre-trained vision-language models such CLIP, which may limit its effectiveness when applied to weaker domain-speciffe backbones. Moreover, it has so far been validated only on image classification tasks, leaving its applicability to other vision tasks. Future work will explore extending the approach to object detection, semantic segmentation, and real human- in-the-loop settings to assess its broader utility.
Acknowledgements. This work was partly supported by the ITTP grants and the NRF grants funded by Ministry of Science and ICT, Korea (No.RS-2019-11191990 Artificial Intelligence Graduate School Program (POSTECH); No.RS-2021-I121200
2021-T1210739. Development of Artificial Intelligence Innovation Hub; AT based 5G+ Network Data Analytics Functions and Control Technology; No.RS-20244004577580 Al Research Hub Project).
No.RS-
References
Nabiha Asghar, Pascal Poupart, Xin Jiang, and Hang Li. Deep active learning for dialogue generation. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM), 2017.
Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In ICLR, 2020a.
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In International Conference on Learning Representations, 2020b. URL https://openreviie wttpeeewwww.eeeeeoore//rrrttttii /tt./orrdddrrrg2....
Jihwan Bang, Sumyeong Ahn, and Jae-Gil Lee. Active prompt learning in vision language models. In Proceedings of the IEEE/CH CCF Conference on Computer Vision and Pattern Recognition, pp. 27004-27014, 2024.
Markus Bayer and Christian Reuter. Activellm: Large language model-based active learning for textual few-shot scenarios. arXiv preprint arXiv:2405.10800 2024.
Liangyu Chen, Yutong Bai, Siyu Huang, Yongvi Lu, Bihan Wen, Alan Yuille, and Zongwei Zhou. Making your first choice: to address cold start problem in medical active learning. Proceedings of Machine Learning Research nnn, 1:30, 2023.
Mircca Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1006-3613, 2014.
13

---

Published in Transactions on Machine Learning Research (10/2025)
Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint 1XXi::1902003963 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icce, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021. URL https:/ /ttpa////opeeeeeeee/eeee http::////paeeeeeeeeeeeeerrrrrts .ttf:/forum.dddYccoddTTTT
Yu Du, Fangyun Wei, Zih Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open- vocabulary object detection with vision-language model. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pullern Recognilion, pp. 14084-14093, 2022.
Zeyad Ali Sami Emam, Hong-Min Chu, Ping- Yeh Chiang, Wojeiech Czaja, Richard Leapman, Micah Gold- blum, and Tom Goldstein. Active learning at the imagenet scale. arXiv preprinl (rXiv:2111.1288) 2021.
Li Fei-Fei, Rob Fergus, and Pictro Perona. Learning generative visual models from few training examples: An incremental bayesian approach lested on 101 object categories. In 2004 conference on compuler vision and pottern recognition workshop, pp. 178-178. IEFE, 2004.
Chengjian Feng, Yujie Zhong, Zequn Jie, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma. Promptdet: Towards open-vocabulary detection using uncurated images. In European Conference Computer Vision, pp. 701 717. Springer, 2022.
Dominik Fuchsgruber, Tom Wollschläger, Bertrand Charpentier, Antonio Oroz, and Stephan Günnemann. Uncertainty for active learning graphs. In Forty-forst International Conference on Machine Learning, 2024. URL htttps://ppenree http///pnneveieeeeeeee fort:/formmttt borrmmmid=Baee Wrrumiid-BCEtuPPD...
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Y Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In Europeon Conference on Compuler Vision, pp. 340-557. Springer, 2022.
Sanket Rajan Gupte, Josiah Aklilu, Jeffrey J Nirschl, and Screna Yeung-Levy. Revisiting active learning in the era of vision foundation models. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
URL
wwtffoommmiiiusseee id=u8883M9bb.
Guy Hacohen and Daphna Weinshall. How to select which active learning stralegy is best suited for your specific problem and budget. Tn A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 13395 13407. Cur- ran Associates, Inc., 2023a. URL /attss://prottt http::///ooooee...eeeeeoe wwwppppprccppppprpppp httpp/sc...ppppe..eeeee////////e 20000000084422222444443222202223 Paweeeobe--ape----eeee Pwweeeerreeeeeeeeeeeeeeaee
Guy Hacohen and Daphna Weinshall. How to select which active learning strategy is best suited for your specific problem and budget. In Thirly-seventhh Conference on Neural Information Processing Systems, 2023b. URL https://openreviiet wwt.:/eee/ffuun/eee/e 2ow.0orrru....00000A000A
Guy Hacohen, Avihu Dekel, and Daphna Weinshall. Active learning on a budget: Opposite strategies suit high and low budgets. In International Conference on Machine Learning, pp. 8175-8195. PMLR, 2022.
Tao He, Xiaoming Jin, Guiguang Ding, Lan Yi, and Chenggang Yan. Towards better uncertainty sam- pling: Active learning with multiple views for deep convolutional neural network. In IEEE International Conference on Mullimedia and Eapo (ICME), 2019.
Patrick Helber, Benjamin Bischke, Andrcas Dengel, and Damian Borth. Eurosat: novel datasct and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Farth Observations and Remote Sensing, 12(7):2217 2226, 2019.
14

---

Published in Transactions on Machine Learning Research (10/2025)
Alex Holub, Pictro Perona, and Michael C Burl. Eutropy-bascd active learning for object recognition. In 2008 TEEE Computer Sociely Conference on Computer Vision and Paltern Recognition Workshops, pp. 1 8. LEEE, 2008.
Sehyun Hwang, Sohyun Lee, Sungyeon Kim, Jungseul Ok, and Suha Kwak. Combating label distribution shift for active domain adaptation. In ECCV, pp. 549-566. Springer, 2022.
Schyun Hwang, Sohyum Lee, Hoyoung Kim, Minhyeon Oh, Jungseul Ok, and Suha Kwak. Active learning for semantic segmentation with multi-class label query. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 27020-27039. Curran Associates, Inc., 2023. URL http:///00000/000000000000110000
neurips wttpcc/ppppppee... ppape_ffless files/
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana. Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Ducrig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference machine learning, pp. 4904 4916. PMLR, 2021.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19113-19122 2023a.
Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Nascer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15190-15200, 2023b.
Hoyoung Kim, Minhyeon Oh, Sehyun Hwang, Suha Kwak, and Jungseul Ok. Adaptive superpixel for ac- tive learning semantic segmentation. In Proceedings of the IEFE/CVF International Conference on Computer Vision (ICCV), pp. 943-953, October 2023.
Hoyoung Kim, Sehyun Hwang, Suha Kwak, and Jungseul Ok. Active label correction for semantic segmen-- talion with foundation models. In Forty-firtt International Conference Machine Learning, 2024.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedinggs of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object reprosentations for fino-grained catego- rization. In Proceedings of the IEEE international conference compuler vision workshops, pp. 554 561, 2013.
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven se- mantic segmentation. In International Conference on Learning Representations, 2022. URL https: / //openreview wwwparrrriii.eettfoommmre..aaeee mett:/forrumiidddtiiijdddd....
Juncheng Li, Minghe Gao, Longhui Wei, Siliang Tang, Wenqiao Zhang, Mengze Li, Wei Ji, Qi Tian, Tat-Seng Chua, and Yueting Zhuang. Gradient-regulated models. In Proceedings of the IEEE/ /CF International Conference on Computer Vision, pp. 2551-2562, 2023.
learning for generalizable vision-lange languale
Yunfan Li, Peng Hu, Dezhong Peng, Jiancheng Lv, Jianping Fan, and Xi Peng. Image clustering with external guidance. In Forty-first International Conference on Machine Learning, 2024.
James MacQueen et al. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability. Oakland, CA, USA, 1967.
Rafid Mahmood, Sanja Fidler, and Marc T Law. Low-budget active learning via wasserstein distance: An integer programming approach. In International Conference Learning Representations, 2021.
15

---

Published in Transactions on Machine Learning Research (10/2025)
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual 2013.
classification of aircraft. Xiv preprint
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In The Eleventh International Conference on Learning Representations, 2022.
Maria-Elenn Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sirth Indian conference on computer vision, graphics & image processing, pp. 722-729. IEEE, 2008.
OpenAL. Gpt-4 technical report. arXiv preprint arXiv:2303.088 0887774 2023.
Natalia Ostapuk, Jie Yang, and Philippe Cudré-Mauroux. Activelink: deep active learning for link prediction in knowledge graphs. In The World Wide Web Conference (eeeenceeWWW (WWW), 2019.
Jinyoung Park, Juyoon Ko, and Hyunwoo J Kim. Prompt learning via meta-rogularizztion. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 26940 26950, 2024.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. 2012 IEEE conference on computer vision and pattern recognition, pp. 3498 3505. IEEE, 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang, Distributionally robust neural networks. In International Conference on Learning Representations, 2019.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pp. 618 626, 2017.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Leurning Represenlations, 2018.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaune Cousiron, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1663-- 2022.
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. Tn ICCV, 2019.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint 3xXw:23303.1338, 2023.
Yingtao Tian, Chikahiko Suzuki, Tarin Clanuwat, Mikel Bober-Irizar, Alex Lamb, and Asanobu Kitamoto. Kaokore: A pre-modern japanese art facial expression dataset. arXiv preprint arXiv:2002.0885 2020.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(1), 2008
Tianjiao Wan, Kele Xu, Ting Yu, Xu Wang, Dawei Feng, Bo Ding, and Huaimin Wang. A survey of deep active learning for foundation models. Intelligent. Computing, 2:0058, 2023.
Zengmao Wang, Bo Du, Weiping Tu, Lefei Zhang, and Dacheng Tao. Incorporating distribution matching into uncertainty for multiple kernel active learning. IEEE Transactions on Knowledge and Data Engineering (TKDE), 2019.
Hantao Yao, Rui Zhang, and Changshong Xu. Tep: Textual-based class-aware prompt tuning for visual- language model. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recogni- lion, pp. 33438-23444 2024.
16

---

Published in Transactions on Machine Learning Research (10/2025)
Ofer Yehuda, Avihu Dekel, Guy Hacohen, and Daphna Weinshall, Active learning through covering lens. Advances in Neural Information Processing Systems, 35:22354-22333 2022.
Muyang Yi, Quan Cui, Hao Wu, Cheng Yang, Osamu Yoshie, and Hongtao Lu. A simple framework for text-supervised semantic segmentation. In Proceedings of the IREE/CVF Conference Computer Vision and Pattern Recognition, pp. 7071-7080, 2023.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, al. Florence: A new foundation model for computer vision. arXiv preprinl ar 0rx::2111.1144 11.11, 2021.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognétion, pp. 18123-18 18133,, 2022.
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 16793-16803, 2022.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision- language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16816 16825, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337 2348, 2022b.
Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15659-15669, 2023.
17

---

Published in Transactions on Machine Learning Research (10/2025)
A Details of GradFAM
(a) Image
(b)
(c) Dog)
(g) Image
(h)
(d) F(Ball, Prisbee)
(e) R(gy,
(1) F [(rangee Blue)
(i) F(BallBBox)
(k) F(Plint,Wate
(l) F [Rood,
Figure 10: GradFAM visualizations for two image examples under diverse class-guided feature targets.
In Section 4.1, we propose class-guided feal ures and analyze them with our GradFAM, a modified version of GradCAM (Selvaraju et al., 2017). Here, we provide a detailed description of GradFAM and its differences from GradCAM.
Gradient-weighted class activation map (CAM). To highlight the importance of regions in image associated with given class Grad-CAM introduces the cles-diseriminativve localization map as follows:
(x) RUXV
(16)
where U and V are the width and height of the image x.
Let f(z; f) E RlCl represent the output logits of neural network with parameters A, where |C| is the total number of classes. For given class C, the score (x; is defined as follows:
f==
(17)
where fe(2; 0) denotes the e-th logit value, representing the model's confidence for class given the image Grad-CAM computes the gradient of (z; 0) with respect to the activation maps A in the last convolutional layer, where 4 E T2 H represents the activation map of the k-th channel with spatial dimensions W and И. Note that W and H are typically smaller than U and V due to downsampling in the convolutional layers.
The importance weight a for each channel is obtained by applying global average pooling over the gradients as:
W H
By (x;
(18)
OCAM WXH
0
representing the overall importance of the k-th feature map for predicting class c.
The class-discriminative localization map (x) then computed as a weighted combination of the activation maps, followed by upsampling match the dimensions of the input image æ:
TCAM(() == u [ReLU (
c.k (Iociur
(19)
where U denotes the upsampling function.
18

---

Published in Transactions on Machine Learning Research (10/2025)
Random
Eecropy
CocoSes
BADGE
PCB
CB48Q (C)
CB+5QQ (R)
90
70
90
60
(%6
Accuracy (%)
Accuracy (%)
Accuracy (9%
80
50
70
70
40
50
40
60
30
50
25
50
75
100
26
50
75
100
0
25
50
75
100
25
50
75
100
Cumulative Budget (%)
Cumulative Budget (%)
Cumulative Budget (%)
Cumulative Budget (%)
(a) Average
(b) Flowers102
(c) DTD
(d) OxtordPets
70
(%)
60
Accuracy (%
Accuracy (%6
30
Accuracy (%)
70
80
25
55
60
70
15
40
25
50
75
100
25
50
75
100
0
25
50
75
100
25
50
75
100
Cumulative Budget (%)
Cumulative Budget (%)
Cumulative Budget (%)
Cumulative Budget (%)
(6) StanfordCars
(f) Caltech101
(g) PGVCAireraft
(h) EuroSAT
Figure 11: Effect of the proposed method. performance across datasets.
Our CB CB+SQ outperforms the other baselines in average
ARI: 48 000
ARE TO
ARI: 77.60%
ARI: 100%
(a) Image features
(b) Weighted text features
(c) Class Class-guided features with pseudo-labels
(d) Class-guided features with ground-truths
Figure 12: Various clustering methods on OxfordPets dataset. (a) Clustering based solely on image features primarily separates the data into two large groups, corresponding dogs and cats, but fails to captare finer details. (b) Clustering on weighted text features improves Adjusted Rand Index (ARI), but some clusters remain ambiguous. (c) Class-guides clustering with pscudo-labels produces more distinct clusters, aligned with the guided class set [C] = 37. (d) Class-guided clustering with ground truth shows perfect alignment of clusters.
Gradient- feature activation map (FAM). To adapt GradCAM for vision-language models (VLMss,, such CLIP, which consist of image encoder Oung and text encoder Oxxt redefine the weight a with target features Fearget follows:
W
dcos (m0g(()) Franget)
O
k
W
0.hh whh
(20)
US
k=
19

---

Published in Transactions on Machine Learning Research (10/2025)
Randonu
Entrogy
CoreSet
BADGE
PCB
CB- SQ (ours)
84
68
93
(96
Ancuracy (%)
Accuracy (%)
(%)
65
78
89
72
85
59
66
81
25
50
75
100
26
50
75
100
0
25
50
76
100
25
50
75
100
Cumulative Budget (%)
Cumulative Budget (%)
Cumulative Budget (%)
Cumulative Budget (%)
(a) Average
(b) Food101
(c) OxfordPets
(d) FGVCAíreraßt
69
57
(%)
Accuracy (%
Accuracy (%)
(%6
66
65
91
63
60
60
55
0
25
50
75
100
25
50
75
100
0
25
50
75
100
25
50
75
100
Cumulative Budget (%)
Cumulative Budget (%)
Cumulative Budget (%)
Cumulative Buddgtt: (%)
(c) Caltech101
(I) SUN397
(g) Plowers102
(h) DTD
G1
Accuracy (%)
Accuracy (%)
56
65
61
46
25
50
75
100
0
25
50
75
100
Cumulative Budget (%)
Cumulative Budget (%)
(i) StanfordCars
(i) UCF101
Figure 13: Harmonic mean accuracy. (a) Our CB+SQ outperforms the other baselines in average performance across 9 datascts. (b-j) The fluctuation in harmonic mcan performance is attributed to variations in perfor- mance on novel classes, stemming from the limitations of the base prompt learning method, CoOp (Zhou et al., 2022b).
representing the overall important the k-th feature map in determining the cosine similarity between the image features (z) and the target features Frarget- Here, the key difference from the GradCAM's weights in (18) lies in the absence of specific class c. This modification enables the use of GradFAM for more flexible and label-independent analyses, accommodating the multimodal nature of VLMs. Building on this, is then computed as: the target feature discriminative localization map LFAM (z)
(x) = U ReeLU E )
(ROLU
(21)
Our GradFAM can visualize the importance of various target features on the image, including (i) F Ftarget (c)( (it) Fearget = (m)g()) and (iii) Ftarget Fe(x). Specifically, when Forget fxa(c), our GradFAM operates almost identically to the original GradCAM. For Fearget 0mgg(r) due to VLMss being trained
20

---

Published in Transactions on Machine Learning Research (10/2025)
(a) Image features
(b) Image features
(c) Image features
ARE 57.33%
ARE 28.01%
ARE 29.6855
(d) Class-guided features, [C]
(e) Class-guided foatures, [C]
(f) Class-guided fentures, [C] 20
Figure 14: Class-guider clustering on WaterBirds dataset. (a, b, c) Clustering based solely on image features results in clusters that are poorly separated. (d, e, f) In contrast, our class-guided clustering, which incor- porates class information, leads to more distinct clusters that align with the size of the guiding class set C.
ARE: 67. 67.60%
ARI:
ARI: 70.019
ARI: 84.17%
Image features, y
(b) Class-guided features, r (c) Class-guided features,
(d) Class--uided foatures,
Figure 15: Class-guided clustering across different rounds. (a, b) In the initial round, i.e. 0, clustering on class-guide guided features achieves a higher ARI compared to clustering on image features. (c, d) We observe an improvement in ARI performance as the rounds progress.
through contrastive learning to align images and lexts in a shared embedding space, all objects in the image are highlighted. In the case of our class-guided features Fc(x) in (7), the guiding class set effectively highlights the corresponding objects.
B Active Learning with Conserved Budgets
For fair comparisons, we evaluate our CB+SQ alongside with baselines over eight rounds, as illustrated in
Figure However, thanks our selective qucrying in Section 4.3, we conserve budgets for each round. Here, conduct additional experiments using the conserved budget. Figure 11 shows that our CB+SQ outperforms the baselines under various budget scenarios.
21

---

Published in Transactions on Machine Learning Research (10/2025)
Table 4: Examples of guiding class set. To improve the clarity of clustering visualization, we randomly
subsample 20 classes from the total of 200 classes.
Index
Name
Index
Name
Index
Name
Index
Name
019
Gray Catbird
027
Shiny Cowbird
American Crow
Mangrove Cuckoo
046
Gadwall
057
Ruse-breasted Grusbeak
Pigeun Guillemot
Heermann's Gull
069
Rufous Hummingbird
073
Blue Jay
079
Belted Kingfisher
098
Scott Oriole
116
Chipping Sparrow
121
Grasshopper Sparrow
125
Lincoln Sparrow
153
Warbling Vireo
167
Hooded Warbler
172
Nashville Warbler
174
Palm Warbler
185
Bohemian Waxwing
Table 5: Various guiding class sets. Based on two habitats and two backgrounds, we construct class sets
comprising 2 and 4 classes.
C
land background water background
2
land bird on land background land bird on water background
4 water bird land background
water bird water background
Table 6:
Various values of K for K --meons clustering. Our progressively incrcasing K across rounds achieves
the best performance throughtout all rounds.
Methods
Round 1
Round 2
Round 3
Round 4
Round 5
Round 6
Round 7
Round 8
K = B
63.6520.16
66.37 10
67.07
68.3440.46
69.1520.11
69.43
69.96
70.28
K = 8 x B
59.7710.27
64.30
6..33
69.30
70.62
71.92
72.46
72.26
K = r B
63.85
66.91 10.38 40.38
68.92 +0.81
70.49 20.06
71.21 20.18
72.16
72.9410.09
73.34
C Analyses of Class-Guided Clustering
In Figure 4, we visualize the difference between class-guided clustering and conventional clustering basod on image features. In this section, we present additional analyses on various clustering methods across different datasets and rounds.
Class-guided clustering on OxfordPets dataset. In Figure 12, we analyze various clustering methods, including clustcring on (a) image features in (5), (b) weighted text features in (6), (c) class-guided features with pseudo-labels in (7), (d) class-guided features with ground-truth labels, where the weights of weighted text features are replaced to ground-truth labels, i.e. one-hot encodings on labels. Figure 12 illustrates that clustering on class-guided features achieves higher Adjusted Rand Index (ARI) values. Especially, Figure 12d suggests that as the performance of VLMs improves, perfect clustering becomes achievable.
Class-guided clustering on WaterBirds dataset. In Section 5.3 and Figure 4, we introduce the Wa- terBirds dataset to demonstrate the benefits of class-guided clustering, where different classes are guided within the same dataset. Specifically, the Waterbirds dataset comprises 200 distinct bird species, with each image annotated by habitat (water, land), background (water, land), and specific species. For our analyses, we select 20 classes and leverage various label information to separate the subset. into groups of 2, 4, and 20 classes. Tables 4 and provide the detailed class names. We note that text prompts such "a photo of a are prepended to each class C to generate final prompts. As shown in Figure 14, class-guided features based on different sizes of guiding class sets C effectively represent class-specific information.
Cluster-Guided Clustering with Various Rounds. We analyze the impact of class-guided clustering, derived from the zero-shot CLIP model, in comparison to various other clustering methods before initiating active learning. Here, we investigate cluster-guided clustering on the OxfordFlowers dataset, utilizing text
22

---

Published in Transactions on Machine Learning Research (10/2025)
Table Experiments on the ISIC dataset.
Budget (%)
Entropy
CoreSet
BADGE
PCB
CB (ours)
25
57.87
53.69
58.37
51.49
63.38
61.35
59.21
62.10
57.52
64.01
61.65
60.81
63.89
62.10
64.41
100
62.25
60.91
65.59
64.54
65.70
Table 8: Experiments on the KaoKore dalaset.
Budget (%)
Entropy
CoreSet
BADGE
PCB
CB (ours)
51.68
45.75
45.36
50.74
52.62
56.36
53.32
55.58
56.52
56.83
57.73
57.06
59.02
57.30
59.33
100
60.11
59.25
61.01
60.58
61.90
Table 9: Eaperiments on the DTD dalasel with CLIP ViT-L//1-336.
Budget (%)
Entropy
CoreSet
BADGE
PCB
CB (ours)
25
43,00
41.13
44.25
50.49
64.10
50
57.60
52.66
60.74
64.44
68.83
75
65.33
61.70
66.19
68.93
70.90
100
69.41
65.17
69.62
72.60
72.97
Table 10: Experiments on the DTD dataset with (VA00-CLIP----1
(ViT-H/1 H//4)..
Budget (%)
Entropy
CoreSet
BADGE
PCB
CB (ours)
40.17
39.38
44.46
43.87
58.45
52.44
60.09
63.95
68.32
67.18
60.72
67.65
69.11
71.81
71.61
66.77
71.91
72.71
73.92
prompts that evolve with each round. Figure 15 illustrates that as the rounds progress, class-guided clustering forms increasingly well-separated clusters, accompanied by steady increase in ARI.
Effect of Various K. In Section 4.2, set K equal to the budget B in the initial round and introduce linearly increasing K according to round r, i.e. K = r x B. Here, we analyze the effect of this increasing K. Table 6 shows that fixed values of K, whether small (K = B) or large (K = 8 x B), are less effective compared to our incrementally increasing K. Specifically, using a small results multiple samples being selected from the same cluster, lcading redundancy and reduced performance. On the other hand, large K fails to select representative samples during the initial round, resulting in diminished performance.
D Additional Ablation Studies
Extension Non-Natural Image Domains. We conduct additional experiments on the medical dataset ISIC (Codella et al., 2019) in Table and the illustrative dataset KaoKore (Tian al., 2020) in Table 8, While the improvements less pronounced than those class-balanced (CB) acquisition still consistently outperforms strong baselines, indicating that the approach remains effective even when VL.M priors are less directly aligned with the target domain.
main benchmarks in Figure the proposed
Generalization to Other Backbones. We utilize CLIP ViT-B/32 architecture for main experiments. Here, we extend the ovaluation to larger backbones: CL.P VIT-L/14-336 and EVAO1-CLIP---14-p0 (ViT- H/14) (Sun et al., 2023). Tables and 10 show results on the DTD dataset. Here, we follows the same experimental setup with Figure 3, except for the backbone. Increasing model capacity generally improves
23

---

Published in Transactions on Machine Learning Research (10/2025)
Table 11: Effect of dlass-guided features on initial pool selection. the initial round, incorporating class- guided features consistently improves baseline performance across six datasets. indicates that K-meons clustering is employed.
Type
Method
Caltech101
OxtordPets StanfordCars Flowers102
FGVCAireraft
DTD
Avg
Random
62. 42.1040.42
0..9
30.50+0.64
03.3643.01
14.6420.a7
26.35+1.08
38.83 38.83+0.65
Image Features
TypiClust
77.05
58.27=0.17
36.53:0.15
76.6440.18
17.4620.49
41.1140.28
51.18+0.11
Image Features
ProbCover
51.6820
53.02_1.01
26.71:0.16
51.1210.38
16. 18.492034
32.76
38. 18.9622.11
Image Features
ProbCover*
75.34 o1
55.79_0.49
34.36.0007
72.24
16.55
38.77
48.83g0.19
Image Features
DropQuery
70.97
57.43_0.33
37.39
75.30
17.09
42.10g0.34
51.21
Class-guided Peatures
TypiClust
81.08+0.27
71.45-1.67
40.87=00.6
78.83+0.14
18. 10:0.77
40.05+0.74
06-00.66
Clase-guided Features
ProbCover
63.95
62.58_1.91
32.60 32.61.20.00
61.31+0.52
15.81go.15
35.50
:6.99
Claes-gided Features
ProbCover*
78.8621.08
68.61=1.67
39.1110.17
77.07
17.8320.62
36.40
53 53.31:0.29
Class-guided Features
DropQuery
79.85=00.22
70.95-0.40
40.41=+0.63
77.8640.34
17.52=0.36
39.11+0.49
04.28:00110
the final performance when the annotation budget is fully consumed, and CB remains consistently superior to baselines, supporting the generalizability our acquisition strategy across VLM backbones.
Additional Clustering- based Baselines with Class-guided Features. In Table 11, we first evaluate these baselines in the initial round using only image features and find that they mitigate the cold-start issue more effectively than Random. When class-guided features added, all baselines benefit from positive synergy that further enhances performance. Specifically, TypiClust (Hacohen & Weinshall, 2023b), ProbCover (K-means) (Yehuda et al., 2022), and DropQuery (Gupte et al., 2024) employ K-means clustering in the given feature space, selecting samples within each cluster based on typicality, coverage, or proximity to the centroid, respectively. Our method operates in the same manner as DropQuery (Gupte et al., 2024) during initial pool selection, but differs in that we use class-guided features instead of image features. As shown in Table 11 and Figure 9c, the proposed class-guided features can be easily integrated into existing basclines and yield substantial performance gains.
24


Reviews:
[{"review_id": "ElI45aWltA", "reviewer": "Reviewer_9bb1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- This paper introduces a low budget active learning method for image classification tasks with vision language models via prompt tuning. The acquisition function is based on cluster balanced sampling, and selective querying to avoid wasting the annotation budget on less uncertain samples.\n- An important component of this method is cluster balanced sampling which also addresses the cold start problem in active learning. Instead of using just the features from the VLM's image encoder, the authors propose a weighting scheme to incorporate text features as well and combine both to generate class guided features which can then be clustered for balanced sampling.\n- Quantitative experimental results demonstrate that cluster balanced sampling + selective querying outperform a selection of active learning methods in nearly all the tested datasets. The authors also show that their data-centric method can be integrated with model-centric methods to improve performance in the 1-shot setting.\n- Some qualitative results of t-SNE visualizations of the class guided features vs image features alone show that the class guided features are better separated in the latent space. Gradient feature activation maps tend to support this with heatmaps showing a focus on classes of interest. Quantitative ablations confirm these observations.", "claims_and_evidence_yn": "No", "claims_explanation": "The claims are mostly supported by the experimental results presented in the paper, but the baselines chosen for comparison are not the strongest, so I'm unsure if the results will hold up under more stringent conditions.\n\n- **query methods:** CoreSet and BADGE are somewhat older methods and not particularly well suited to the low-budget AL regime. More modern methods like TypiClust [1], ProbCover [2], and DropQuery [3] might be more suitable for baselines, since all of them also address the cold start problem using some form of clustering. At this point it is known that any AL method not mitigating the cold start problem will suffer from diminished early performance, so it is unsurprising that the proposed method does well.\n- **novelty of cluster balanced selection:** Modern AL methods (TypiClust, ALFA-mix [4], DropQuery) utilize cluster based sampling for diversity in their selections. TypiClust is probably the most similar in terms of partitioning the data points into clusters and then sampling from the densest regions (as opposed to centroid in this paper). It may be worth emphasizing that the proposed cluster based selection is novel in the context of active prompt tuning, but not necessarily as an AL method.\n- **class guided features:** In the limit, I would expect the text features to align almost perfectly with the image features, so the computed soft label weights would approach zero for the incorrect classes and one for the correct class, so there would be little information added. While the current method shows quantitative improvements in table 2, it's unclear how much of that improvement would persist when using larger VLMs with better image-text alignment (and therefore approaching the aforementioned limit). Since current experiments were performed with ViT-B/32, it is worth investigating other model sizes like ViT-H/14 or even the giant variants of EVA-CLIP.\n\n[1] [Active Learning on a Budget: Opposite Strategies Suit High and Low Budgets](https://arxiv.org/abs/2202.02794)\n\n[2] [Active Learning Through a Covering Lens](https://arxiv.org/abs/2205.11320)\n\n[3] [Revisiting Active Learning in the Era of Vision Foundation Models](https://arxiv.org/abs/2401.14555)\n\n[4] [Active Learning by Feature Mixing](https://arxiv.org/abs/2203.07034)", "audience_yn": "Yes", "audience_explanation": "There is a lot of interest in tuning VLMs to perform well in low-budget scenarios, and active learning with prompt tuning is certainly a promising approach. I think the proposed method as a whole is of broad interest with multiple experiments probing the efficacy of its individual components. Even though GradFAM is not directly relevant to model performance, it's still an interesting modification and might be of use in other contexts as well, say representation learning.", "requested_changes": "I would like to see a few additional experiments to compare against stronger AL baselines and with larger VLMs.\n\n- It would be good to add some of the modern methods mentioned previously to the results in table 3, specifically those addressing the cold start problem for a fairer comparison, since that seems to be where the majority of gains originate.\n- As an ablation, it would be interesting to see the same experiments conducted with identical initial selections for CB+SQ to control for initialization differences, since centroid based initialization has been shown to improve performance across methods in [3].\n- Repeating the comparison in table 2 with larger ViTs would help to determine if the class-guided features are truly beneficial compared to using just image features.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hj66fwnqGd", "reviewer": "Reviewer_2j62", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to address the data selection in active learning for few-shot prompt tuning for vision language models based on pre-trained prior knowledge. Specifically, it fuses the image-text features using weighted class-guided clustering and querying the samples to annotate based on a class adaptive threshold. The experimental results on standard benchmark demonstrates the effectiveness of the proposed method given the budget constraint.\n\nThe motivation is clear. Frames few-shot prompt tuning as active data selection, tackling cold-start with class-guided clustering.\n\nPerformance: seven datasets plus ImageNet show accuracy gains and label savings over baselines. Cluster sampling plus selective querying saves $\\sim 17\\%$ annotation cost by using reliable pseudo-labels.\n\nAnalysis shows GradFAM and t-SNE confirm the method creates better class separation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The active learning strategy is too simple. See requested changes for more details.", "audience_yn": "Yes", "audience_explanation": "The few-shot prompt tuning task is a little outdated, as the current state-of-the-art VLLMs can perform well as long as the pretraining data has sufficient coverage. There are already some papers exploring active learning for prompt tuning, and, furthermore, the findings from the traditional active learning field can be applied to the vision-language setting with minor modifications.", "requested_changes": "**The task setting scope is too narrow in practice.** Active learning for few-shot prompt tuning is demonstrated primarily on recognition tasks, but whether this method can generalize to broader task categories beyond recognition remains unclear.\n\n**Incomplete evaluation on standard benchmarks.** Why are the complete comparison results on the ImageNet dataset not presented? This omission makes it difficult to assess the method's performance against established baselines on a widely-used benchmark.\n\n**Unexplained performance characteristics.** The proposed method appears to saturate faster than other methods in subfigures (b, d, f, h) of Figure 3. What causes this early saturation, and does this indicate a fundamental limitation of the approach?\n\n**Limited technical novelty.** The proposed class-guided clustering technique shares similar principles with existing semantic feature fusion and augmentation techniques, such as CLIP-GCD [1]. Additionally, while different active learning papers typically propose distinct sampling and querying strategies, the authors do not conduct comprehensive ablations comparing different active labeling strategies, which limits our understanding of the method's core contributions.\n\n**Reference:**\n[1] CLIP-GCD: Simple Language Guided Generalized Category Discovery", "broader_impact_concerns": "Considering all the above factors, I consider that the impact of this work might be a little limited.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5f8FOj3sgS", "reviewer": "Reviewer_WqbA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a data‑centric active‑learning framework that adapts VLMs to downstream datasets with minimal annotation cost. The work uses class-guided feature construction followed by a linearly growing K-means clustering to obtain a cluster‑balanced candidate pool. It also utilizes pseudo-labels to assist in the selective querying process for active learning. They evaluate their proposed approach across seven image‑classification benchmarks, showing higher accuracy than Random, Entropy, CoreSet, BADGE and PCB while reducing human labeling.\n\nStrength:\n\n1. The cold-start problem is well handled by utilizing VLM predictions as priors.\n2. The modified GradFAM for annotation‑free saliency is done well. ARI and t‑SNE analyses show class‑guided clustering is semantically aligned.\n3. The proposed active prompt learning reduces the need for human querying by ~18%.\n4. The evaluation in Figure 3 shows boost in performance across almost all the datasets.\n\nWeakness:\n\n1. The work relies much on the zero-shot priors which doesnot always adapt well to downstream datasets, especially not having natural images. The evaluation of the proposed approach on other types datasets such as medical datasets (ISIC [1]),  illustrative datasets (Kaokore [2]]) and structured datasets (Clevr-Count [1]) is crucial to gauge the contribution of the proposed prompt active learning approach.\n\n2. In this view, the proposed approach also shows reliance on only CLIP model as a supporting VLM. The authors should also highlight the performance using other backbones such as EVA‑CLIP. Does the approach only function well using CLIP? These questions the generalizability of the approach.\n\n3. More details on the robustness analysis needs to be provided regarding the performance tied to linear K‑schedule, prompt re‑initialisation and threshold formula. How does the pseudo-label threshold affect the performance? Does one threshold perform well across all the datasets? Including an ablation study for the changing pseudo-label threshold can give a better clarity.\n\n4. In regard of the ImageNet experiments, the compute overhead is opaque. What is the compute requirement and GPU hours required for K‑means on millions of ImageNet embeddings plus per‑round prompt retraining which is certainly not trivial.\n\n5. The Base-to-novel generalization including Table 3 seems confusing. Where are the results for individual datasets? Does table 3 show only the average of all datasets?\n\n[1] Codella, Noel, et al. \"Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic).\" arXiv preprint arXiv:1902.03368 (2019).\n\n[2] Tian, Yingtao, et al. \"Kaokore: A pre-modern japanese art facial expression dataset.\" arXiv preprint arXiv:2002.08595 (2020).\n\n[3] Johnson, Justin, et al. \"Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are backed by suitable experiments. However, the key contributions of each module, including the GradFAM, need to be portrayed more clearly, including more ablation studies.", "audience_yn": "Yes", "audience_explanation": "The approach couples the essence of prompt learning with active learning. It would be interesting to venture more in this frontier.", "requested_changes": "Please go through the Weakness section to find the detail suggestions and changes. In brief, the suggested changes are as follows:\n\n1. Evaluate the performance on various datasets and using different model backbone.\n2. Threshold Robustness – What happens if initial labeled data are noisy or class confidence is skewed (long‑tail classes)?\n3. Hyperparameter Search Cost—K‑schedule, budget B, prompt length—how sensitive are final results, and are they tuned per dataset?", "broader_impact_concerns": "Nil", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VqLmb501gn", "reviewer": "Reviewer_d98e", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose an active learning method that integrates class-guided clustering and selective querying to efficiently adapt CLIP with minimal annotation budget.\nExperiments across seven datasets show that their approach achieves better performance and budget efficiency than existing baselines. The proposed data-centric approach is also shown to enhance state-of-the-art model-centric prompt learning methods, showing widespread potential.", "claims_and_evidence_yn": "No", "claims_explanation": "I mainly have a concern for the second contribution of GradFAM. \nWhile GradFAM provides a useful visualization of the proposed class-guided features, it appears to be primarily an analytical tool rather than a core methodological contribution. Since it is listed as one of the main contributions of the paper, I would appreciate clarification on the following points:\n - What specific limitations of existing feature analysis methods (e.g., GradCAM or other CLIP-based visual attribution) motivated the design of GradFAM? Could existing methods not be adapted for this purpose?\n - Are there prior works or techniques (e.g., CLIP-GradCAM, Attention Rollout) that have been proposed for feature-level analysis in VLMs? If so, how does GradFAM compare?\n - Does GradFAM provide any analytical or quantitative insight beyond qualitative visualization?", "audience_yn": "Yes", "audience_explanation": "1. The class-guided features seems to be effective to indicate target text regions.\n2. The experiment results show the effectiveness of the proposed method, which can enhance 3 existing methods consistently among 7 benchmarks.\n3. The proposed method shows good generalization to unseen classes in the first round.", "requested_changes": "1. If the author wants to keep the second contribution, please add more supportive evidence, according to my comments above. \n2. In GradFAM, how is the similarity $\\cos(\\mathcal{I}(x), \\mathcal{F}_{\\text{target}})$ computed when the target feature  is the class-guided feature, which is a concatenation of image and text features, resulting in a dimension twice that of $\\mathcal{I}(x)$, making the cosine similarity unclear.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)