UP-DP: Unsupervised Prompt Learning for Data Pre-Selection with Vision-Language Models
Authors: Xin Li
Sima Behpour
Thang Doan
Wenbin He
Liang Gou
Liu Ren
Venue: NeurIPS 2023 poster (2023)
Keywords: Unsupervised prompt learning
UP-DP
Data preselection

Abstract: In this study, we investigate the task of data pre-selection, which aims to select instances for labeling from an unlabeled dataset through a single pass, thereby optimizing performance for undefined downstream tasks with a limited annotation budget. Previous approaches to data pre-selection relied solely on visual features extracted from foundation models, such as CLIP and BLIP-2, but largely ignored the powerfulness of text features. In this work, we argue that, with proper design, the joint feature space of both vision and text can yield a better representation for data pre-selection. To this end, we introduce UP-DP, a simple yet effective unsupervised prompt learning approach that adapts vision-language models, like BLIP-2, for data pre-selection. Specifically, with the BLIP-2 parameters frozen, we train text prompts to extract the joint features with improved representation, ensuring a diverse cluster structure that covers the entire dataset. We extensively compare our method with the state-of-the-art using seven benchmark datasets in different settings, achieving up to a performance gain of 20\%. Interestingly, the prompts learned from one dataset demonstrate significant generalizability and can be applied directly to enhance the feature extraction of BLIP-2 from other datasets. To the best of our knowledge, UP-DP is the first work to incorporate unsupervised prompt learning in a vision-language model for data pre-selection.

UP-DP: Unsupervised Prompt Learning for Data Pre-Selection with Vision-Language Models
Xin Li
Sima Behpour Thang Doan Wenbin He Liang Gou Liu Ren
Bosch Research North America, Bosch Center for Artificial Intelligence (BCAI) (xin.li9, sima.behpour, doang wenbin.he2, liang.gou, liu.ren boss.oocc..oom
Abstract
In this study, we investigate the task of data pre-selection, which aims to select instances for labeling from an unlabeled dataset through a single pass, thereby optimizing performance for undefined downstream tasks with a limited annotation budget. Previous approaches to data pre-selection relied solely on visual features extracted from foundation models, such CLIP and BLIP-2, but largely ignored the powerfulness of text features. In this work, we argue that, with proper design, the joint feature space of both vision and can yield a better representation for data pre-selection. To this end, we introduce UP-DP, a simple yet effective unsupervised prompt learning approach that adapts vision- languae models, like BLIP-2, for data selectiec.on. Specifically, with the BLIP-2 parameters frozen, we train text prompts extract the joint features with improved representation, ensuring diverse cluster structure that covers the entire dataset. We extensively compare our method with the state-of-the-are using seven benchmark datasets different settings, achieving up to performance gain of 20%. Interestingly, the prompts learned from one dataset demonstrate significant generalizability and can be applied directly to enhance the feature extraction of BLIP-2 from other datasets. To the best of our knowledge, UP-DP is the first work to incorporate unsupervised prompt learning in vision-language model for data pre-selection.
Introduction
Data-efficient machine learning, which aims to identify the best subsets of data samples (to either label or not) in order to achieve optimal model performance, has been a crucial research field the In this work, focus new and practical task of data pre-selection for data-efficient visual object recognition (Fig [a). The goal of data pre-selection is select instances for labeling from unlabeled dataset through single pass to maximize model performance for unknown downstream vision tasks (e.g., no knowledge about prediction categories, shown in Fig [b)) given a limited annotation budget. This task motivated by two practical needs: at the data acquisition stage, we want to collect minimal data to support potentially diverse downstream tasks (c.g.. classification, action recognition, or even detection for videos or images in Fig b) and training paradigms (c.g. active learning. semi- sumi-supervised learning, supervised learning in Tc): the annotation stage, we aim to achieve good performance with fewer human annotations over the pre-selected data.
of data-hungry deep learning [12 31 24
Traditionally, tackle the challenge of data efficiency learning, there two main approaches:
Semi-supervised learning (SSL) G. 35 21 and Active Learning (AL) 34 T0 2 SSL aims to address the problem of label scarcity by leveraging limited quantity of labeled data in conjunction with a more abundant pool of unlabeled data to enhance the model's performance. By contrast, AL approaches start with an initial of labeled data and select the most informative data point to label
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

a
Selected
labeled
Instances
Labeling
Instances
Supervised learning
Dataset Pre-selection
Classifier
Data
Unlabeled
Definition
Instances
(e... lsibll resuirermente))
Semi- suppi-uuer learning
Classifier
Active learning
Classifler
OxfordPeta Dataset
Muttimmddal Feeture Promot
Mutti modal Feature Random Prompt
Image Feeture Only
Figure Our data pre-selection (a) novel task data efficiency learning, and different from previous tasks, such as semi-supervised learning and active learning, which needs both seed labels and downstream task information (c). Our intuition demonstrated by the UMAP visualization (d) of the features extracted from BLIP-2 on the OxfordPets training dataset, with each color representing the ground truth. evident that BLIP-2 with an appropriate prompt (purple box) can provide visually superior multimodal features compared to unimodal features (green box), which can serve as a good starting point for data pre-selection.
in an effort to maximize model performance with a limited label budget. However, existing AL methods need initial labeled set to begin with and need to train series of models.
Nonetheless, the task of data pre- selection poses some unique challenges that SSL and AL, as mentioned above, fail to adequately address. In data pre-selection, we do not have initial labeled data or know the specific downstream task. Take object detection as an example: we do not know what categories of objects need to be detected. However, both current SSL. and AL methods need both initial labeled data and specific learning tasks. Therefore, this requires a method to select diverse and representative instances to cover the entire dataset with minimum number of required labels. To achieve this, it is essential to acquire semantically meaningful, high-quality features.
Fortunately, recent advances in foundational Vision-Language (V-L) models, such as CLIP 32 and BLIP-2 23 hold significant promise in addressing this problem. These models acquire compre- hensive semantic knowledge by learning from vast collection of image-text pairs. Consequently, they possess the ability to extract meaningful features from diverse input modalities. While there exists a numerous of studies that adapt V-L models for various tasks like few-shot classification [48). semi-supervised learning [46], and selective labeling [40, they often place emphasis on unimodal features, primarily vision, and inadvertently overlook the potential contributions of multimodal features encompassing both vision and language.
In this paper, we argue that by carefully designing the prompt, the multimodal features extracted by V-L models can offer more powerful representation for data pre-selection. Figure I illustrates the unimodal and multimodal feature representations obtained from BLIP-2, with different colors representing various ground truth labels. In particular, when the class information of downstream tasks is unknown, the unimodal features become mixed, making it challenging to identify specific classes. In contrast, when coupled with an appropriate prompt, multimodal features exhibit more scattered and distinct distribution, allowing for improved discrimination among classes. This enhanced discrimination is crucial helping users identify the most representative and diverse instances during the data pre-selection process.
However, the process of designing an appropriate prompt can be demanding and time-consuming, often requiring trial and errors [48] Moreover, current automatic prompt learning approaches typically rely small set of labeled or pseudo labeled data, making them impractical for data pre-selection when the downstream tasks are undefined. To address this challenge, we propose a novel approach: pure unsupervised prompt learning. which aims to enhance the multimodal features extracted from V-L models for data pre-sclection. Our contributions include:
48 47 19
We introduce UP-DP, an innovative Unsupervised Prompt learning for enhanced Data Pre-selection in Vision-Language models. To the best of our knowledge, UP-DP is the

---

pioneering work that incorporates unsupervised prompt learning in a vision- vision-language model specifically for data pre-selection.
We demonstrate the robust generality of the learned prompt across different datasets, which significantly enhances feature extraction when applied in a alug-and-play manner.
We extensively benchmark our method using various architectural settings, achieving supe- rior performance compared other competitors on image classification datasets.
2 Related Works
2.1 Data Efficiency Learning
Semi-supervised Learning (SSL. combines labeled and unlabeled data to enhance learning, and can be seen unsupervised learning with added labeled data. SSL approaches are divided mainly by how they leverage large amounts of unlabeled data. Consistency Regularization methods [33 42 maintain model output stability under realistic perturbations like data augmentation, with UDA 42 being an example that extends supervised data augmentation to boost SSL performance. Pseudo- labeling methods 4 41 5 use high confidence model predictions to generate pseudo-labels for unlabeled data, training them jointly with labeled data. For example, SoftMatch (5] addresses the erantity-qualite trade-off in pseudo-labeling using a truncated Gaussian function weight samples by confidence. Transfer dearning-based methods such as SimCLRv2 [6] involve supervised fine-tuning after unsupervised pre-training unlabeled datasets, demonstrating the effectiveness of large models for SSL..
However, the success of these methods is based the assumption of stratified sampling 4 35). which requires equal sampling of each known class, a constraint that is often unattainable in practice. More importantly, our data pre-selection task even does NOT have any assumption for downstream tasks and also does NOT know any information about class categories.
Active Learning (AL) actively selects high-value instances for labeling during the iterative training process to enhance predictive performance, offering realistic alternative the stratified sampling setting of SSL. AL strategies generally fall into two categories: (1) methods [15] 44 that select informative instances using scoring function such as uncertainty and (2) methods
that
choose diverse instances representing the dataset's 's domain. Some research [30, addresses both aspects, such as ALFA-MiX [30], which evaluates label variability for perturbed instances and ensures diversity by clustering and selecting centroids.
However, current AL methods have limitations. They often require an initial labeled set, which is inefficient in low-label settings, and involve multiple rounds of labeling and training with a human annotator, making the process consuminn.. On the other hand, the requested labels are tightly associated with the training objective, necessitating distinct instances for each task. In contrast, our data pre-selection task aims for single-pass labeling applicable to any future, unknown downstream task, minimizing human effort and overall cost.
Unsupervised Selective Labeling (USL) [40] is the most relevant approach to ours. They proposed an SSL pipeline that avoids the unrealistic setting of stratified sampling. Specifically, they first apply unsupervised feature learning (e.g.. MoCov2 (7) map data into discriminative feature space. Then, they select instances for labeling for maximum representativeness and diversity in a single pass. Finally, with a strong initialization from the first step, they apply SSL to the labeled data and the remaining unlabeled data. Although their method achieves great performance and even beats stratified sampling in some settings, it is not suitable for data pre- selection. Using the same model and weights for both the selection step and the downstream task step is not a valid assumption.
2.2 Vision-Language Foundation Models
Vision-language models. pre-trained on large-scale image-text pairs, excel in visual representation learning. A taxonomy of V-L models (20] can be based on two aspects: (1) the expressiveness of both modalities in terms of parameters and computation, and (2) the extent of interaction within a Deep Neural Network (DNN). Farly Visual Semantic Embedding (VSE) models, such as VSE++ [3] and SCAN 37 use separate feature extractors for image and text modalities, with the image extractor being more complex. They represent similarity using dot products or shallow attention layers.
3

---

Select Points from Each Cluster
Feature Extractor
Labeling
Downstream
Tasks
Salected Instances
Unsupervised Prompt Leariing
Feature wio Prompt
WIO Prompt
Prompt
(b) Unsupervised Prompt Learning
L
Attraction
Repulsion
Instance-level Contrastive Learning
Learaale Context
gr(O)
Inslance
Fh:
[V]
Mh
Augmented
Clustering
MLPs
f0
BLIP-2
Rr(-)
Cluser Lows
Embaddings
L
Cluster-level Contrastive Learning
Prompt Generalization for Other Datasets
Multimodal Embeddinge
Learndd Context
From Pets
[P]
Better Rapresantation
BLIP-2
Flowers Images Dalaset
Feature with PromptFeature wo Prompt
Figure 2: An Overview of Our Approach. (a) workflow for data pre-selection, (b) training process for unsupervised prompt learning, and (c) generalization of learned prompts across various datasets.
38 for each modality but maintains shallow interaction. Recent models [20] 22 39] use deep transformers for inter-modal interactions, enhancing performance in complex vision- language tasks. However, pre-training becomes computationally expensive with large feature extractors and interaction models. BLIP-2 [23] introduces querying transformer (interaction) that utilizes pre-trained image and language models and captures robust interaction between image and text. The interaction two modalities allows
In contrast, CLIP 32 uses comparably complex feature extractors (transformers
effectively manage the multimodal features extracted from the V-L model merely by modifying the input text (prompt). However, determining the optimal prompt for data pre-selection remains open question. In this work, designed novel approach of generating text prompts to extract multimodal features without any additional information.
Adaptation Vision-Language Foundation Models
Recent advances in multimodal pre-training models have shown great success in classic vision tasks 32]. prompting interest in more efficient adaptation models.
To avoid the time-consuming process of fine-tuning an over- parameterized V-L model, two main strategies have recently gained popularity: (I) V-L Adapters [16 45). and (2) Prompt learnin [48 19]
The first category of adapters methods, like CLIP CLIP-Adapter 16 and Tip-Adapter [45). introduce lightweight networks learn refined features for classification tasks. On the other hand, CoOp proposes continuous prompt optimization strategy for image recognition, inspired by prefix-tuning for language models [17]. However, these methods, including Unsupervised Prompt Learning (UPL)
19), mainly target classification tasks and require labels pscudo labels, making them ill-suited for data pre-selection tasks.
Our work differs from existing literature as we approach downstream tasks with zero prior knowledge. We demonstrate the adaptation of the V-L model to extract superior features an unsupervised manner.
3 Methodology
overview of our approach shown in Figure 2 First, we formally define the data pre-selection task. Next, provide brief reviews on BLIP-2 and present unsupervised prompt learning

---

framework, which assists BLIP-2 in extracting improved multimodal features for data pre-selection. Finally, we describe how use the adapted BLIP-2 to select instances for downstream tasks.
3.1 Data Pre-selection
Figure 2a shows the data pre-selection workflow. Suppose that we have an enormous dataset D consisting of d instances and an annotation budget of l. Our task is to select / (1 < d) instances for labeling, so that undefined downstream tasks trained on such a partially labeled dataset produce the best performance. Formally, let D = (Ii, yo) denote d pairs of the image I; and its class label yı (yi Ø if the label is unknown). Let Dr. denote size I subset of D with known class labels. Our goal is to select D,, C D to acquire class labels, in order to maximize the performance of an undefined model trained labeled data Dr with without unlabeled data Du D Dr. Our data pre-selection task is challenging, as we do not have any labels to begin with. This means we do NOT know what information will make the undefined downstream task perform the best. Our idea is to select / instances that are not only representative but also diverse enough to cover the entire dataset, preventing premature loss of valuable information prior to label acquisition.
The process depicted in Figure illustrates the detailed data pre-selection process by extracting high- quality and semantically significant features. These features facilitate the clustering of all unlabeled images into well-structured groups Subsequently, we select the most representative instance from each cluster for labeling. This strategy ensures both representativeness and diversity, accomplished by selecting one instance from each cluster. In the upcoming section, we will describe how we adapt BLIP-2 to this objective.
3.2 Unsupervised Prompt Learning
Figure 2p depicts the Unsupervised Prompt Learning approach to extract desirable features. It consists of three cooperatively learned elements: the learnable context of BLIP-2 and two Multilayer Perceptron (MLP) heads for both instance-level and cluster- level transformation. These compo- nents are jointly optimized using un- supervised clustering objectives (25). specifically instance-level and cluster- level contrastive learning. After train- ing, the cluster assignments of all in- stances be easily derived from the predicted soft labels produced by the cluster-level MLP head. When com- bined with the multimodal features ob- tained from the adapted BLIP-2, can effectively identify the most rep- resentative and diverse instances from the entire dataset D.
Unsupervised Prompt Learning Input: Unlabeled dataset D; training epochs E; batch size N; temperature parameters Ty and Tci cluster number M; learnable context V; augmentation set T;
Algorithm I:
BLIP-2 f; MLP heads JI and gc: Dy + []
Output: Labeled subset Dr.
Training
for epoch = 1 to E do
Sample a mini-batch { from D 1N
Sample two augmentations Ta, T3 T Compute instance and cluster representations:
h {TT, (x.), V). hς =[Th(x.), V)
2 (r ) 2 91 (h
C = (h.)), c = (h)) Compute overall loss L Update V, gr, and gC to minimize L
end // Inference for in D do h f(x) // Extract features arg maxgaa((chh) // Compute cluster assignment
Bootstrapping Language-Image end Pre-training known as BLIP-2, Labeling presents an efficient pre-training strategy that leverages existing pre-trained image encoders and large language models, all while remaining frozen. By training a lightweight querying transformer (Q-Former), it specifically bridges the modality gap between images and text. In the representation learning phase, BL.IP-2 connects Q-Former to a frozen image encoder and conducts image-text pair pre-training. With three image-text pre-training objectives (Image- Text Contrastive Learning, groundgd Text Generation, and Image-Tex: Text Matching), they establish
for k = 1 to M do
Select the most representative instance m.
Dj = Dr U Um,
end
5

---

strong connection between image and input text, i.c. forcing the model to extract visual information from the image that is most relevant to the text and automatically generating output multimodal features. This characteristic of BLIP-2 provides us with a new opportunity. By altering different prompts (input text), we can extract distinct multimodal features and find the best to enhance the process of data pre-selection.
Learnable Prompt However, designing an appropriate prompt can be demanding and time- consuming, especially for data pre-selection, when we have zero knowledge of downstream tasks. Following the previous prompt learning approach, we aim to model each context token using a con- tinuous vector that can be learned from data from end to end. Specifically, we define N learnable prompt representation context vectors, denoted as V = {V1, V2. V2....un V.. each having the same dimension the word embedding, to serve the text imput for BLIP-2. In contrast to many previous methodologies that leverage cross-entropy loss as their learning objective with label data, have chosen adopt unsupervised clustering our learning objective with respect to undefined downstream tasks. Since the text encoder is differentiable, which means gradients can be back-po pook-propagated to update the context vectors. It's important to note that the base model of BLIP-2 remains unchanged during the training process.
depicted in Figure 2p.
Instance-level Contrastive Loss Contrastive learning at the instance level is designed to maximize the similarities between positive instance pairs, while simultaneously minimizing those of negative pairs. Given the lack of labels for all the data, we aim to maximize the agreement between differently augmented views of the same instance and treat all other augmented instances negative examples. Specifically, consider a randomly sampled mini-batch of image of N images. Each image, denoted by x;, undergoes different augmentation twice, thus creating two views of the same example: x² and x². Subsequently, these two images are encoded, along with a learnable context V, using BLIP-- f()). generate multimodal representations: hę and h.. Following this, the representations are further transformed via nonlinear transformation MLP (-), known as an instance-level head, which yields z and z!. Let sim (.,-) denote the cosine similarity. Then the instance-level contrastive loss of a positive pair of examples (i,j) E [1 is defined as
exp (sim (z. z1) /TT)
2 - llo
E Exe(a,b) (a.b) Σ/=1 exp (sim (zft. z5) /TI)
where Ty is the temperature scalar. The final loss L, = 2N Σ. (ee + 2)) which is computed across all positive pairs in mini-batch.
Cluster-level Contrastive Loss Similar to instance-level constrative learning, cluster-level constra- tive learning aims maximize the similarities between positive cluster pairs, while minimizing the negative pairs. To construct the cluster representation, we use another cluster-level head (c(+) to map the multimodal representations Thus m-th element of the feature can be interpreted as the instance's 's probability of belonging to the m-th cluster. Formally, suppose with mini-batch of N instances and M predefined clusters, the output of the cluster head is C E TR. N under the first augmentation, thus C% can be interpreted as the probability of instance n under augmentation a being assigned to the cluster m. The i-th column of C can be treated as the representation of the i-th cluster under augmentation Thus, we can from positive pair of cluster by selecting i-th column of C" and C³ noted as ég and é while leaving other 2.M- 2 pair to be negative:
.M -idimensional space, where M equals the number of clusters.
exp (sim (e) ot) /TC) Σke(a.) Σÿn1 exp (sim (og et) /TC)
logg
where TO the cluster-level temperature parameter. To avoid a trivial solution that most instances are assigned to the same cluster, a regularization that encourages the entropy of cluster assignment
probability is added: H(C) Σ. (P(ç)) log P(c)) P(e)) log P(c))] where P(c))
N N Cj,, & {a, b}. The final cluster-level loss is Lo ZM Σ M
Dist (4) + e H(C).

---

Objective Function The optimization is one-stage and end-to no- process, shown in Algorithm The prompt and two heads are simultaneously optimized and the overall objective function the combination of constrastive loss at the instance and cluster level: C = Li + Lc.
Labeling Finally, after training, each instance assigned a cluster number by the adapted BLIP-2 and cluster-level MLP head. First, we directly use the probability predicted by the cluster-level head (c(+) for sampling. In this case, for each cluster, the instance with the highest confidence score is selected. On the other hand we can also select medoid of each cluster to be labeled for the downstream task. We can then select the medoid of each cluster be labeled for the downstream task. Formally given the cluster X5. with N₂ members {x1, X2...XN, J. the medoid m, is the instance that has the minimal average dissimilarity to all instances calculated on the multimodel feature h or z, here we use h from [()) as an example:
NR
m arg min X;EX;
sim(f((; V), (f(x,, V))).
(3)
No
4 Experiments
We assess the effectiveness of our UP-DP method in a downstream task, specifically using se- lected labeled instances for image classification. In particular, we perform the UP-DP on BLIP-2, strategically selecting the most representative and diverse instances for labeling, followed by the linear-probe on CLIP to compare the performance against random baselines and two variations of current state-of of-the-art method: USL [40]. Lastly, we show several intriguing properties of UP-DP such as generalizability, i.e. a learned prompt from a single dataset can be directly applied to enhance the feature quality of other datasets.
Dataset We select seven image classification datasets that are widely used in evaluating the V-L model adaptation approach. These datasets constitute a comprehensive benchmark, covering a diverse set of vision tasks, including the classification of generic objects (Caltech101 [4]), actions (UCF101
(36),, lineggriiedd categories (OxfordPets (29). FGVCAircraft (26,, and Flowers 102 (27). as well as
some specialized tasks such as recognizing texture (DTD S and satellite imagery (EuroSAT (18).
Training Details For the base model, we the best available vision backbone in BLIP-2, which is ViT-G. Previous work (48) on prompt learning has shown that shorter context length can lead to better and more robust performance. Therefore, we initialize the context vectors with fixed length of 4. The two hyperparameters, тι and TC, are set to 0.5 and 1.0, respectively. Training is performed with the Adam optimizer and a learning rate of 0.0003. We optimize our model with a batch size of 256 for a total of 150 epochs on RTX 3090. We set an annotation budget of 200 for all datasets except EuroSAT, which, due to its significantly larger size (13,500 examples) and fewer classes (10), is allocated label budget of 40 and trained for 50 epochs. The rationale for setting uniform annotation budget of 200 images for most datasets is our lack of knowledge about the downstream task (e.g., prediction categories) in the context of data pre-selection. Thus, the annotation budget becomes our sole controllable factor. By taking into account variations in class numbers and dataset sizes, this uniform budget encompasses wide range of downstream task difficulties, averaging between 2 to images per class.
Baseline Methods As described in the related work section, USL is the most relevant approach to ours. Similarly to our method, USL also relies on semantically meaningful feature representations for each instance. We apply USL to both the image features extracted from BLIP-22 and the multimodal features extracted from BLIP-2 with the learned prompts, resulting in two settings: USI.-I and USL-M. Alongside USL, we include random selection as a bascline method for the purpose of sanity check.
Zero-shot Recognition with BLIP-2 In the representation learning stage, the Q-Former performs pre-training using image-text pairs with image-text contrastive learning. Similar to CLIP, after pre-training, this head can produce a similarity score between an input image and text, which can be naturally used for zero-shot recognition. We use the hand-crafted prompt provided by CoOp for evaluating zero-shot performance. As shown in the bottom row of Table we observe that the
7

---

Method
EuroAAT
OxfordPets
Cahech101
PGVCAireaatt
UCFI01
Flowers122
Average
Random
0.221 1 10055
10052
0.202 1 0.016
1.33610.002
0.057 a 0.009
0.121 a 0.015
0.178
USL.1
0.209 + 0.052
0.161 +00000
0.188 a 0.028
0.297
000088
0.948 a 0.008
0.104 a 0.043
0.177
RN50
Durs f()
USL-M
0,289 1 0.044
+ 0.040
0.191
0.239
0.038
0.185 4 0,030
0.011
0.324+0.000
0.286 0.008
0,064 0.010
0.066
0.184
0.108 0.027
0.137 0.003
0.130 a 0.026
0.179
0.240
Ours gr(
0.409 + 0.038
0.214 0.007
0.282 : 0.007
0.324 0.00
0.074 = 0.012
0.179 = 0.009
0.136 = 0.003
0.240
Ours
0,487 + 0.001
0.204
0.013
0,283 + 0.007
0,353 a 0,002
0.076 + 0.004
0.205 a 0.006
0.130 0.001
0.248
Random
0.277 0.062
0.021 0.193
0.139 0.076
0.842 + 0.013
0,093 + 0.014
0.116 0,040
0.153
USL-1
0.257 1 0.038
0.149
0.145 1 0.019
0.263 1 0.012
0.042 1 0.006
0.101 0.017
0,040 1 0.012
0.150
RN101
USL-M
0.200 a 0.030
0.150 0.052
0.144 0.055
0.257 0.000
0.858 a 0.013
0.172 a 0.057
0.111 0.026
0.149
Ours [()
0,350 1.051
0,211 1 0.004
0.250 1 0.017
0.298 1 0.001
0.056 4 0.003
0.146 0.013
0.133
0.005
0.206
Our (
1.0024
10011
0.245
0.00
a 0.070
0.145 0.010
0.133 0.002
0.213
Ours get
0.423 x 0.007
0.177 a 0.012
0.260 a 0.010
0.312 0.002
0.067 : 0.006
0.190 a 0.015
0.120 0.001
0.221
Random
0.332 0.004
0.340 + 0.024
0,286 a 0.049
0.371 + 0.014
0.071 # 0.015
0.150 0.013
0.175
0.024
0.246
USI.1
0.422 + 00.96
0.299 + 0.038
0.275
0.028
0.318 + 0.006
0.074 + 0.006
0.164 0.020
0.179 0.036
0.247
NITB32
Ours [()
USL-M
0.525 0.000
0.027
0.439 a: 0.011
0.033
0.372 : 0.002
0.352 1 0.002
0.310 0.010
0.089 2 0.001
+
0.132
0.214 0.011
0.188 0.025
0.188
0.008
0.255
0.311
Ours
0.557 00.06
0.429 + 0.019
0.379 + 0.002
01355 + 0.005
0.994 + 0.013
01200 + 0.013
0.186 0.006
0.315
Ours gct
0.584
0.380 + 0.020
0.385 0.007
0.392 0.002
0,098 = 0.006
0.234 0.010
0.217
0.003
0.327
Randem
0:482 1 0.099
0.404 1 0.042
0.278 1 0.034
0,330 1: 0.016
0.118 : 0.025
0.174 : 0.024
0.229
0.03
0.288
USI-1
0.504 + 0.100
0.359 + 0.055
0.284 + 0.0S5
0.29% + 0.016
0.108 + 0.010
0,206 = 0.022
0.227
0.024
0.284
VITH14
Ours [()
USL-M
0.505 1 0.070
0:434 0.029
0.005
0.301 0.037
0.003
0.301 0.015
0.001
0.128 0.013
0.116
0.180 0.036
0.253 0.017
0.221
0.206
0.018
0.000
0.296
0.349
Ours (
0.596 a 0.007
0.548 1 0.025
0,394 1 0.003
0,332 1 0.003
0.131 = 0.020
0.247 2 0.005
0,203 2 0.001
0.350
Ours gerl-)
0.634 0.014
0.177 a 0.021
0,403 1 0.009
0,371 a 0,005
0,143 a 0.003
0.287 a 0.010
0.241 a 0.001
0,365
Rundom
0.402 + 0.045
0.421 + 0.027
0.305 0,033
0.334
0.109 + 0.014
0.198
0.035
0.216
0.032
0.284
USL-I
0.029
0.013
0.113 =
0.208
0.243
0.026
0.290
VITG14
Ours f()
USI-M
0.481 000777
0.500
0.582 + 0.008
0.461 + 0.02
0.389 + 0,004
0.305 + 0.038
0.29% + 0.017
0,339 + 020002
0.130 = 0.005
0.133 + 0.013
0.184 0.040
0.254 0.014
0.224 = 0.002
0.238
B.024
0.354
0.300
Ours gr(
0.006
0.336=00000)
0.145
0.252
0.216
0.357
Ours get
0.609 0.019
0.503 0.017
0.403 0.005
0.376 0.004
0.161 = 0.005
0.289 0.015
0.258
0.002
0.371
Zero Shot BLIP-2
(0.100)
0.081 (0.027)
0.123 (0.021)
0.379 (0.010)
0.113 (0.010)
0.070 (0.010)
0.114 (0.010)
0.141
Table 1: Accuracy (%) of linear probe results for CLIP trained with data pre-selected by random,
USL-I, USL-M, and our method. Evaluation is carried out using five different versions of the CLIP vision encoder. The accuracy (%) of zero-shot performance with BLIP-2 is displayed in the bottom row, with the numbers in brackets representing the random guessing baseline. Our UP-DP consistently outperforms other baselines by an average accuracy around 7%.
performance of BLIP-2 is much lower than that of CLIP reported in CoOp. There are two potential reasons for this discrepancy: (1) the hand-crafted prompt for CLIP cannot be generalized for BLIP-2, and (2) the features from BLIP-2 cannot be directly generalized to these datasets, or perhaps combination of both reasons. This highlights the necessity of adapting BLIP-2 for data pre-selection tasks.
To replicate the pre-selection process, which involves separating downstream tasks from data pre-selection, we first employ UP-DP to select the most representative and diverse data points for labeling. We then apply the linear probe method using various versions of the CLIP vision encoder, ranging from ResNet-50 to ViT-G, with the labeled data. Table presenns the main outcomes of UP-DP in seven datasets. Overall, our UP-DP consistently outperforms other baselines with an average accuracy of 6.8% to 7.2%.
Linear Probe Results
Upon further analysis of the table across different datasets, we note that UP-DP significantly out- performs the baseline on ET, yet yields mixed results on 111tech101. This variation is not entirely unforeseen considering BLIP-2's limited efficiency with complex tasks. For instance, in zero-shot setting, it nearly resorts to random guessing (0. 1 accuracy on 10 classes) on the satellite image dataset EUroSAT, whereas it exhibits relatively superior performance (0.397 accuracy on 100 classes) on the standard object recognition dataset, 1altech101. When comparing the average performance of the second and third rows, USL-M consistently outperforms USL-L This consistent performance gap between USL-M and USL-I further validates the effeectiveness of a prompt learned by UP-DP, which assists BLIP-2 in extracting superior multimodal features.
Both USL-M and our method utilize the same multimodal features extracted from BLIP-2 with learned prompts for data selection. Essentially, USL can be seen as initially executing k-means clustering and then picking the most representative point with diversity regularization. Unlike USL-M, we explicitly train a cluster MIP head that can directly assign each instance within the cluster, where the medoid is selected in each cluster for labeling. The performance improvement between USL-M and our method showcases the effectiveness of the simple sampling strategy employed in UP-DP.
8

---

Features
Caltech101
DTD
FGVCAirrrat
Flowers 102
OxfordPets
UCF101
EuroSAT
Average
Caltech 101 Prompt
0.975
0.768
0.485
0.987
0.922
0.885
0.957
0.834
DTD Prompt
0,979
0.809
0.482
0.889
0.878
0,964
0.836
RGVCAireraft Prompt
0,974
0.768
0.518
0.986
0.852
0.862
0.963
0.829
Flowers102 Prompt
0,978
0.806
0,583
0,995
0.946
0.876
0,952
0.864
OxfordPets Prompt
0,976
0.746
0,520
0.945
0,960
0.840
SECFO01_Prompee
0.970
0.766
01423
0.978
0.786
0.863
0,962
0.795
EuroSAT EuroSATTPPrompp Prompt
0,955
0.794
0,415
0.975
0.501
0.865
0,969
0.746
Emppty_ Prompt
0.756
0,501
0.322
0.719
0.492
0.744
01933
0.606
Initi Prompt
0.771
0.534
0.270
0.757
0.421
0.779
0.919
0.592
Image
0.974
0.763
0.349
0.984
0.681
0.875
0.961
0.760
Table 2: Domain Generalization Results of the Learned Prompt. The generalization capability is
evaluated by the accuracy (%) of KNN classification, utilizing either image features or multimodal features extracted from BLIP-2 with various prompts. The baseline is the one with image features (in yellow). We highlight results that differ from the baseline by at least =0.02. Green indicates higher accuracy, while red indicates lower accuracy compared to the baseline. This illustrates the impressive generalizability of the learned prompt: even when the prompts are learned from different datasets, the multimodal features almost consistently outperform the image-only features.
Cluster # 149
Cluster #101
Cluster #
Cluster
Figure 3: Visualization of Instance Features and Cluster Assignments on the OxfordPets Dataset. Color coding represents the cluster assignments determined by our cluster MLP head. The multi- modal feature is extracted from BLIP-2 with learned prompts. (a) At a high level, the features extracted with the learned prompts can distinguish between various types of pets even without the labels. (b) A closer examination of one cluster reveals (c) the capability of the features to capture more detailed information, such as background and pet postures, within the same breed of Shiba Inus.
Domain Generalization of Learned Prompt After establishing the effectiveness of UP-DP within a single dataset, we further demonstrate that the learned prompt has the capability to be applied across multiple datasets. This presents considerably more challenging problem, since the fundamental features can undergo significant changes when dealing with different datasets, such as transitioning from object recognition to texture classification.
Figure illustrates the way of using the prompt learned from one dataset, such as OxfordPets, to extract multimodal features from another dataset, such as Flowers We want to demonstrate that these features exhibit superior quality compared to features extracted directly from BLIP-2, including image features and multimodal features with empty random prompts. We evaluate the feature quality with the testing accuracy achieved by the non-parametric K-Ncarest Neighbors (KNN) classifier.
As presented in Table 2, we use the accuracy of the image feature (highlighted in yellow) as the baseline and identify results that differ from it by at least 2% (green and red). It is evident that all the results obtained from multimodal feature extraction using an undefined prompt are inferior to the image feature. Surprisingly. the multimodal feature results with the prompts learned from other datasets exhibit a significant margin of improvement over the image feature (except for the OxfordPets features extracted using the EuroSAT prompt). This demonstrates the generalizability

---

EuroSAT
OxfordPets
DTD Caltech101
FGVCAireraft UCFI01
Flowers
makes
shark
several learn
diamond
healthy
learn
wood
brigham
learn butterfly offers learn
butterfly
takes
saint
putting saint
learn attends
angel
healthy
laying
add
add
plane
performs
saint
ku
elegant six three
del
speaks
personality
single
chicken three
2020
river
newly
picking
have
posing q
2019
100
physical
adding
november
kate vector
2000
40
provides
gold
holds bee
che
speaks
have
choose
spider
holding attends some
with
get
serves giant
Table Top Nearest Words to Learned Prompts. The selected meaningful words are obtained from
the nearest words of the four context vectors learned by UP-DP. Words related to the dataset indicated in green, while words with an orange background represent high-frequency words occurring across various datasets.
of the learned prompt. The exception for OxfordPets features be attributed the significant difference between the satellite image task and the fine-graines dog and cat recognition task.
Qualitative Study To provide intuitive understanding of the functioning of UP-DP, visualize the learned feature representation and cluster assignment of all instances using 2D projection with UMAP. As depicted in Figure 3. high level, the features extracted from BLIP-2 with learned prompts demonstrate discriminative abilities to distinguish between different types of dogs and cats, even without any label information. Taking closer look at the Shiba Inu group in part (b), we observe that the cluster assignments generated by our cluster-head categorize Shiba Inus into distinct clusters based on background and posture. Notably, in (c), clusters 1449, #101, #55, and #52 reveal Shiba Inus against backgrounds of mountains, grasslands, concrete roads, and homes, respectively. Furthermore, Shiba Inus in the clusters # 144 and #5 are typically seen standing, while those in the cluster #101 are observed lying on the ground. Our qualitative study demonstrates that UP-DP effectively selects diverse representative points from unlabeled data, resulting in improved performance for downstream tasks.
Interpreting the Learned Prompts It is difficulty interpreting the learned prompts due their continuous vector nature. To overcome this challenge, we adopted the CoOp approach and searched vocabulary for words that closely correspond to the learned vectors, using Euclidean distance as a metric. The search results are presented in Table We observed that few words, such "wood" EuroSAT (related to the forest class), certain numbers (potentially representing aircraft codes) and the word "plane" in FGVCAircraft, even "butterfly" in the Flowers dataset, demonstrate some relevance to their respective tasks. However, the majority of the words lack coherent meaning. This leads us to speculate that the learned vectors may encode meanings beyond the scope of the existing vocabulary. Notably, we also discovered shared words, including "learn", "saint", "add", and "attend" across different datasets, which could account for the significant generalizability of the learned prompts across various tasks.
Conclusion
In conclusion, we introduce UP-DP, a novel method that leverages Unsupervised Prompt learning to enhance Data Pre-selection with Vision-Language models. Our approach is specifically designed to achieve data efficiency without requiring prior knowledge of specific downstream tasks. By utilizing both vision and text features of V-L models, UP-DP offers a new pathway towards improving data efficiency. It outperforms existing methods by up to 20% on seven benchmark datasets and demonstrates remarkable generalizability on different datasets. We believe that our research opens promising avenues for future studies in data-efficient machine learning.
10

---

References
[1] Sharat Agarwal, Himanshu Arora, Saket Anand, and Chetan Arora. Contextual diversity for active learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI 16, pages 137-153. Springer, 2020.
[2] Sima Behpour, Anqi Liu, and Brian Ziebart. Active learning for probabilistic structured prediction of cuts and matchings. In International Conference on Machine Learning, pages 563-572. PMLR, 2019.
[3] David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv:1911.09775 2019.
[4] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in neural information processing systems, 32, 2019.
[5] Hao Chen, Ran Tao, Yue Fan, Yidong Wang. Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj. and Marios Savvides. Softmatch: Addressing the quantity-qualle trade-off in semi- supervised learning. arXiv preprint arXiv:2301. 10921, 2023.
[6] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners, Advances in neural information processing systems, 33:22243 22255, 2020.
[7] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint arXiv:2003.044 042997, 2020.
[8] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-360 3003,,63,220 2014.
[9] Thang Doan, Xin Li, Sima Behpour, Wenbin Hc, Liang Gou, and Liu Ren. Hyp-ow: Exploiting hierarchical structure learning with hyperbolic distance enhances open world object detection. arXiv preprint arXiv:2306. 14291, 2023.
[10] Melanie Ducoffe and Frederic Precioso. Adversarial active learning for deep networks: a margin based approach. arXiv preprint arXiv:1802.cm 09841, 2018.
[11] M. Everingham, Van Gool, C. K. I. Williams, J. Winn, and Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303-338, June 2010.
[12] Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 961-970, 2015.
[13] Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improving visual- semantic embeddings with hard negatives. arXiv preprint 1rXiv:1707.056122 2017.
[14] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pages 178-178. IEEE, 2004.
[15] Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International conference machine learning, pages 1 1183-1192. PMLR, 2017.
[16] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma. Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint
2021.
[17] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723, 2020.

---

[18] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):211-22266, 2019.
[19] Tony Huang, Jack Chu, and Fangyun Wei. Unsupervised prompt learning for vision-language models. arXiv preprini arXiv:2204.03649, 2022.
[20] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Asion-and-langenn transformer without convolution or region supervision. In International Conference on Machine Learning, pages 5583-5594. PMLR, 2021.
[21] Junnan Li, Caiming Xiong, and Steven CH Hoi. Comatch: Semi-supervised learning with contrastive graph regularization. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9475-9484, 2021.
[22] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900. PMLR, 2022.
[23] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2:
Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint 1xXiv:331112257 2023.
[24] Xin Li, Hassan Bigher-Ebadian, Stephen Gardner, Joshua Kim, Mohamed Elshaikh, Benjamin Movsas, Dongxiao Zhu, and Indrin J. Chetty, An uncertainty-aware deep learning architecture with outlier mitigation for prostate gland segmentation in radiotherapy treatment planning. Medical Physics, 50(1):311-22, 2023. doi:
15982. URL https
7/aapm. onl wwwinenibaarr....mmooam.a.raalla wwtprrarrrlllll............ wiley. com
1002/00........... 15982
[25] Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. Contrastive clustering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 8547-8555, 2021.
[26] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aireraft. arXiv preprint 1rXiv:1306.5135 2013.
[27] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008
[28] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafranicc, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2334.077193, 2023.
[29] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498-3505, IEEE, 2012.
[30] Amin Parvanch, Ehsan Abbasnejad, Damien Teney, Gholamreza Reza Haffari, Anton Van Den Hengel, and Javen Qinfeng Shi. Active learning by feature mixing. In Proceedings of the IEEE/VV Conference on Computer Vision and Pattern Recognition, pages 12237-12246, 2022.
[31] Omead Pooladzandi, David Davini, and Baharan Mirzasoleiman. Adaptive second order coresets for data-efficient machine learning, 2022.
[32] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8766 0763-8733..10 PMLR, 2021.
[33] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep supi-supe learning. Advances in neural information processing systems, 29, 2016.
12

---

[34] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint 17X8: 00489, 2017.
[35] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. Advances in neural information processing systems, 13:596-608 2020.
[36]
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 2012.
[37] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 2020, Proceedings, Part X, pages 268 285. Springer, 2020.
[38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[39] Wenhui Wang. Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, al. Image foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208. 10088/0444 10422, 2022.
[40] Xudong Wang, Long Lian, and Stella X Yu. Unsupervised selective labeling for more effective semi-supervised learning. In European Conference on Computer Vision, pages 427-445. Springer, 2022.
[41] Xudong Wang, Zhirong Wu, Long Lian, and Stella X Yu. Debiased learning from naturally imbalanced pseudo-labels. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 14647-14657, 2022.
[42] Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoe Le. Unsupervised data aug- mentation for consistency training. Advances in neural information processing systems, 33: 0256-6268, 2020.
[43] Yi Yang. Zhigang Ma. Feiping Nie, Xiaojun Chang, and Alexander G Hauptmann. Multi-class active learning by uncertainty sampling with diversity maximization. International Journal of Computer Vision, 113:113-127, 2015.
[44] Donggeun Yoo and In So Kweon. Learning loss for active learning. In Proceedings of the IEEÉ/CVF conference on computer vision and pattern recognition, pages 93- 13-102, 2019.
[45] Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.0399 2021.
[46] Zhiqi Zhang, Pan Ji, Nitin Bansal, Changjiang Cai, Qingan Yan, Xiangyu Xu, and Yi Xu. Clip-flow: Contrastive learning by (s} supervied iterative pseudo {l} abeling for optical flow estimation. arXiv preprint ar 14383, 2022.
[47] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learn- ing for vision- -anguaee models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[48] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for lasion-l models. International Journal of Computer Vision, 233-9243377- 2022.
13

---

Broader Impact
Our research, UP-DP, new method for data pre-selection, holds substantial positive societal impacts. By incorporating unsupervised prompt learning into Assio-languuage models for data pre-selection, our approach improves the efficiency of data labeling and reduces the computational resources required for machine learning tasks. By selecting subset of instances from a large unlabeled dataset for labeling, reduces the time and effort required for manual annotation, thereby alleviating the burden of laborious labeling tasks. This frees up human resources for more complex and creative tasks, rather than repetitive labeling work. Moreover our method democratizes access to machine learning technology. By optimizing performance with limited annotation budget, UP-DP enables individuals or organizations with limited computational resources to train effective models. In conclusion, our work not only advances the field of data pre-selection but also has the potential to positively impact society by improving labeling efficiency, democratizing access to machine learning, and encouraging the efficient of computational resources.
Limitations
Our work UP-DP has demonstrated promising results in enhancing the process of data pre-selection. However, it is necessary discuss its limitations provide a comprehensive perspective. One limitation lies in the nature of the learned prompts. These prompts are continuous vectors, the interpretability of which not straightforward. Unlike hand-crafted prompts that can be directly understood by humans, the meaning of continuous prompts remains opaque. This lack of transparency hinders our understanding of why and how these prompts improve the feature extraction capability of vision- lsiinnlangguage models like BLIP-2 for dataset pre-selection. Furthermore, it is even more challenging to explain the generalizability of learned prompts across different datasets, as they cover diverse set of vision tasks. Another concern is the potential for unfairness and bias in model outcomes. UP-DP operates under a specific constraint: it selects a limited number of instances for labeling from an extensive unlabeled dataset. This strategy optimizes computational efficiency and resource allocation; however, also introduces the risk of selection bias. If the annotation budget is extremely limited, the pre-selected data can be unrepresentative or skewed towards certain characteristics, which can inadvertently lead to biased models.
14

---

Appendix
Section
A
demonstrates the data preselection result on semantic segmentation task.
Section
presents the ablation study on the context length.
Section
shows the performance under large annotation budget settings.
Section visualizes the selected instances from different methods.
15

---

A Semantic Segmentation
Beyond the realm of classification tasks, have also ventured into assessing the performance of semantic segmentation models when trained with data pre-selected via our proposed method. To elucidate, we utilized the Pascal VOC [IT] dataset specifically for the semantic segmentation task, employing various adaptations of the DINOv2 model (28) to serve as our segmentation framework. The dataset comprised total of 1454 training instances, while we constrained our annotation budget to mere 100. As depicted in Table 4 our methodology consistently outperforms other baseline approaches in segmentation tasks, showcasing its robustness and effectiveness."
Model
Random
USL-I
USL-M
Ours
Dinov2_Vii VITS
54.9
56.4
58.1
58.5
Dinov2 ViTB
55.9
57.3
58.3
58.8
Dinov2_ ViTL
55.2
54.2
53.7
57.1
Dinov2_ VITG
47.9
51.9
51.4
53.6
Average
53.5
55.0
55.4
57.0
Table 4: Experiment on Semantic Segmentation Task. The mloU of results for Dinov2 trained with data pre-selected by random, USL-I, USL-M, and our method.
16

---

B
Context length
How many context tokens should be used? And is it better to have more context tokens? The results presented in Table indicate that employing shorter context length yields better overall performance. To investigate this hyperparameter, following the setting in CoOp, we conducted experiments on the benchmark datasets using lengthy context of 16. As illustrated in Table [5] the shorter context length consistently outperforms the longer length across all datasets and settings.
Given that our data pre-selection aims to identify instances that can be generalized to unknown downstream tasks, it becomes crucial to mitigate the overfitting caused by the longer trainable vector on the BLIP-2. This finding aligns with the analysis in CoOp, where it is argued that although having more context can potentially improve the performance of the current model, comes the expense of generalization.
Context Length
EuroSAT
OxfordPets
Caltech101
PGVCAireaat
UCF101
Flowers102
Average
16
0.359 a 0.037
0.207 a: 0.006
0.246 a 0.022
0.312 a 0.002
0.058 2 0.009
0.104 a 0.015
0,210
RN50
0.456 0.040
0,229 0.002
0.011
0.324 + 0.001
0.066 0.001
0.184 0.010
0.137 0.003
0.240
16
0.335 + 0.035
0.188 = 0.017
0.226 0.025
0.286 = 0.003
0.049 = 0.015
0.032
0.087 + 0.015
0.186
RN101
4
0,350 + 0.051
0,211 0.004
0.250
0.017
0.298 + 0.001
0,056 0.003
0,146 + 0.013
0.133 0.005
0.206
16
0.006
0.023
0.320
0,077 0.010
0
0.185
0.285
VITB32
4
0.525 # 0.010
0.439 = 0.011
0.372 + 0.002
0.382 = 0.002
0.089 = 0.001
0,214 + 0.011
0.188 # 0.008
0,311
VITH14
16
0.494 0.007
0.478 0.018
0,348 0.01
0.318 0.004
0,101 0,010
0.249 0.018
0,200 0.011
0.313
4
0.577 + 0.011
0.567 0.005
0.392 0.003
0.335 # 0.001
0.116 0.001
0.253 0.017
0.206 + 0.000
0.349
VITGI4
16
0.501 + 0.006
0.507 = 0.015
0.352 a: 0.009
0.324 a 0.007
0.108 = 0.009
0,251 = 0.019
0.210 a 0.010
0.322
4
0.560 0.024
0.582 0.008
0.389 0.004
0.339 0.002
0.130 0.005
0.254 0.014
0.224 0.002
0.354
Table 5: Impact of Context Length. Accuracy (%) of linear probe results for CLIP trained with data
pre-selected by our method with different numbers of learnable context.
We also conducted an experiment without updating prompts, resulting in context length of zero The learning probe results are presented Table These findings clearly demonstrate the necessity of training the prompts. The motivation behind training the prompts/contexts is to enable the model to extract features more suited to specific datasets. If we only train gr and ge without adjusting the prompts/contexts, the image features would remain static, potentially slowing convergence and lowering performance.
RN50
RN101
ViTB32
VITH14
VITGI4
Average
Random
0.119 + 0.021
0.086 = 0.018
0.150 = 0.035
0.153 = 0.027
0.178 +: 0.028
0.137
BLIP-2 w/o Prompt
0.136 0.005
0.113=000
0.166
0.008
0.176 : 0.003
0.177
0.002
0.153
BLIP-2
0.183 0.011
0.17100.00
0.239 0.007
0.259 0.004
0.256 0.006
0.221
Table 6: Result of Training Without Prompts. Accuracy (%) of linear probe results for CLIP,
trained on the DTD dataset with data pre-selection conducted both with and without prompts. The annotation budget was set at 100.
17

---

C Annotation Budget
In this section, we examine the performance of data pre-selection under various annotation budgets. We conduct a linear probe experiment on CLIP, repeating it with 50% more of the available budget. Specifically, we use 60 labeled instances for EuroSAT and 300 labeled instances for all other datasets. The results are presented in Table 7 The methods (Ours and USL-M) which utilize the multimodal features extracted with learned prompt, continue to outperform other methods, showcasing the effectiveness of our unsupervised prompt learning. It is worth noting that the gain in performance not as substantial as in scenarios with much fewer budgets before. This can be attributed to the fact that, when there is sufficient label budget, such as 10% of the entire training dataset (Table g random selection can serve as a strong bascline. In contrast, our method becomes more valuable in low-budget settings.
Method
EuroSAT
OxfordPets
Caltech101
FGVCAireaf
UCF101
Flowars
Averago
Random
0.323 = 0.069
0.292 # 0.023
0.270=0.000 0.008
0.362 = 0.010
0.062 0.005
0.175 = 0.041
0.151 = 0.011
0.234
RN50
USL-I
0.389 + 0.042
0.254 a 0.041
0.298
0.032
0
0.017
0.349 = 0.013
0.021
0,070 =: 0.007
0.075
0.188 a 0.014
0.153 = 0.026
2.033
0.234
0.241
Ours /(-)
0.398 = 0.010
0.307 0.007
0.342 a 0.010
0.399 a 0.005
0.084 = 0.001
0.245 0.006
0.187 : 0.007
0.280
Random
0.291 = 0.049
0.251 0.028
0.216 0.031
0.319 = 0.016
0,058
0.012
0.142 0.028
0.136 = 0.018
0.202
RN101
USL-1
0.325 0.047
0.225
0.047
0.205
0.027
0:291
0.066
0.011
0.144 0.014
0.124 - 0.109 = 0.026
0.197
USL-M
0.319 0.062
0.298 0.047
0.213 0.031
0.294 : 0.018
0.06-1 0.018
0.149 1 0.017
0.207
Ouns f())
0.325 s 0.000
0.294 a 0.005
0.010 0.305
0,345 = 0.004
0.077 = 0.000
0.224 = 0.005
0.159 = 0.005
0.247
Random
0.066
0.488 0.070
0,400
0.024
0.085
0.201 0.011
0.270
0.330
VITH22
USL-M
USL-1
0.496 a 0.039
0.582 s 0.054
0,438 0.030
0.524 + 0.034
0.409
0.377
0.017
0.016
0.390 = 0.021
0.393 1 0.017
0.102 = 0.020
0.097 = 0.010
0.223 1 0.023
0,223 = 0.011
0.237 = 0.035
0.263 = 0.028
0.348
0.327
f(-)
0.541
0.488 0.019
0.427
0.001
0.448 0.004
0,104 0.004
0.274 0.003
0.001
0.370
Random
0.505 2 0.1
0,560 a 0.029
0.421
0.024
0.425 : 0.009
0.152 = 0.012
0.282 a 0.027
0.290 = 0.038
0.376
V1TH14
USL-M
USL-1
0.647 s 0.057
0.585 + 0.052
0.518 + 0.036
0,629 0.032
0.435
0.408
0.020
0.017
0.372 0.025
0.369 + 0.015
0.147 = 0.011
0.160 0.027
0,292 0.008
0.301 + 0.026
0.294 = 0.039
0,313 = 0.040
0.377
0.404
Ours f(-)
0.615 0.028
0.6000 0.018
0.446 0.005
0.006
0.148 0.009
0.008
0.001
0.413
Random
0.567 a 0.062
0.592 a 0.039
0.440
0.026
0.413 a 0.008
0.165 2 0.026
0.279 0.023
0,341 a 0.023
0.400
VITG14
USL-1
0.578 + 0.044
0.546 0.032
0.411
0.017
0.372
0.019
0.157 0.010
0.312 0.025
0.336 = 0.037
0.387
USL-M
0.627 = 0.052
0.658
0.031
D436 0.117
D.571 0.023
0.171 0.023
0,303
0.004
0.316 = 0.034
0.412
Ours /(-)
0.614 1 0.007
0.636 1 0.014
0.445
2 0,007
0.434 = 0.005
0.165 = 0.011
0,326 = 0.009
0,342 2 0.002
0.423
Table 7: Impact of Annotation Budget. Accuracy (%) of linear probe results for CLIP trained
with data pre-selected by random, USL-I, USL-M, and our method after increasing 50% annotation
budget.
Dataset
Classes
Train
Val
Test
Hand-crafted prompt
ETroSAT
10
13,500
5,400
8,100
"a centered satellite photo of [CLASS]."
OxfordPets
37
2,944
736
3,669
"a photo of a [CLASS], a type of pet."
DTD
47
2,820
1,128
1,692
"[CLASS] texture."
Co
100
4,128
1,649
2,465
"a photo of a [CLASS)."
FGVCAireraft
100
3,334
3,333
3,333
"a photo of [CLASS], type of aireraft."
UCF101
101
7,639
1,898
3,783
"a photo of a person doing [CLASS]]"
102wers102
102
4,093
1,633
2,463
"a photo of [CLASS]. a type of flower."
Table 8: Datasets Statistics. The detailed statistics of the 7 datasets and the hand-crafted prompts that are used for BLIP-2 zero-shot learning.
18

---

D Visualization on Selected Instances
Figure ([c) and (d) present the visualizations of the least-40 and top-40 instances, respectively, that were selected using our method on the EuroSAT dataset. We compare these selections with the top 40 sampled images chosen through random selection and USL. In order facilitate understanding. we have organized the images into different rows based on their ground truth labels. Note that it important to have a balanced selection that covers semantic classes However, the randomly selected instances often exhibit significant imbalances. This means that we may end up with, for example, instances from Class while completely missing any images from Class 5. Such imbalances quantified using the KL divergence, which measures the distance between the current sample distribution and a uniform distribution. In this case, the KL divergence is 0.241. contrast, top selected instances not only provide representation from each class but also offer diversity among the selected instances. The KL divergence for our top selections is only 0.047, indicating that they both representative and diverse. On the other hand, the 40 instances that are least likely to be selected primarily consist of outliers. These outliers have the potential to mislead downstream classifiers and introduce noise into the learning process. For instance, the first image in class 0 (AnnualCrop) may appear strikingly similar to images in Class (Forest) and Class 9 SSeaLake), which can be confusing for the classification task.
Class 0
(a) Random
(b) USL KLDiv: 0.096
KLDiv: 0.241
Class 1
Class 2
Class 3
Class 4
Class 5
Class 6
Class 7
Class 8
Class 9
Class 0
(c) UP-DP
Class 1
Least Utility
(d) UP-DP KLDiv 0.047
KLDiv 0.200
Class 2
Class 3
Class 4
Class 5
Class 6
Class 7
Class 8
Class 9
Figure 4: Visualizations of Selected Instances from EuroSAT. Our selection ensures balance and representativeness. comparison, random selection can lead to imbalance and the potential omission of certain classes, particularly in low-budget settings. KLDiv represents the Kullback-Leibler divergence score between the sampling distribution and the normal distribution (stratified sampling). Classes 0-9 correspond AnnualCrop. Forest, Herbaceous VerbaacousVeegeattion. Highway, Industrial, Pasture, PermanentCrop, Residential, River, and SeaLake, respectively.
19


Reviews:
[{"review_id": "xjRrJcgv5Z", "reviewer": "Reviewer_ucTW", "summary": "This paper presents a novel method to perform data preselection for the task of image classification. Data preselection refers to the task of finding the images for annotating labels and then used for training. The paper builds around the powerful visual-language model BLIP, and proposes learnable prompts as inputs to BLIP to help perform unsupervised clustering for data preselection. The paper then presents results on seven image classification benchmarks, showing the superb performance of the proposed method.", "questions": "See above.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The presented method is a novel application of visual language model to data preselection. \n+ I find the proposed learnable prompting in conjunction with unsupervised clustering novel, and as suggested by experiment effective. \n+ The presented method is effective in data preselection, as demonstrated by the comparison against baseline methods.\n", "weaknesses": "- The decision to annotate 200 images per benchmark (LINE 265 - 273) seems arbitrary. Why this number? It would be great if the number can be varied and then plot the model performance accordingly to understand the effect of annotated data set size on model performance.\n- USL-M, which shares the same multimodal features from BLIP-2 as the proposed UP-DP method, isn't really outperforming the baseline USL-L on EuroSAT (Table 1). Such a result contradicts the claimed effectiveness. What is the explanation?\n- No results on linear probe on BLIP-2 using Random Sampling. This is needed in order to showcase the effectiveness of the proposed method. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer ucTW,\n\nWe would like to thank you for your valuable comments. Below is our response to your questions:\n\nQ1: “The decision to annotate 200 images per benchmark (LINE 265 - 273) seems arbitrary. Why this number? It would be great if the number can be varied and then plot the model performance accordingly to understand the effect of annotated data set size on model performance.”\n\nA1: Thanks for your suggestions. We included additional results for annotation budgets from 50 to 300. As demonstrated in Table E, our method consistently outperforms other baselines regardless of the annotation budget size, thereby underscoring the robustness and effectiveness of our approach. \n\nThe reason for uniformly setting it at 200 images for most datasets is that, within the context of data pre-selection, we lack knowledge about the downstream task (e.g., prediction categories). Therefore, our sole controllable factor is the annotation budget. By considering varying class numbers and dataset sizes, this uniform budget spans a wide range of downstream task difficulties, with an average of 2 to 5 images per class.\n\nQ2: “USL-M, which shares the same multimodal features from BLIP-2 as the proposed UP-DP method, isn't really outperforming the baseline USL-L on EuroSAT (Table 1). Such a result contradicts the claimed effectiveness. What is the explanation?”\n\nA2: Thanks for noting the USL-M and USL-L performance on EuroSAT in Table 1. In some instances, USL-L slightly surpasses USL-M, but given the high standard deviation from USL, it's not statistically significant. For instance, in the case of ViTG-14, the p-value is 0.77. Furthermore, when we examine the average performance across all seven datasets, USL-M consistently outperforms USL-L in five different architectures.\n\nQ3:  “No results on linear probe on BLIP-2 using Random Sampling. This is needed in order to showcase the effectiveness of the proposed method.”\n\nA3: Thank you for the suggestions. We've incorporated additional experiments that provide the performance of the linear probe on BLIP-2 (refer to BLIP-2 ViTL and BLIP-2 ViTG in Table C-F). These experiments illustrate that our approach still surpasses other baselines. It's important to note that using a linear probe on BLIP-2 contradicts the premise of data preselection, as we cannot assume that the downstream task uses the same model as the one employed in data pre-selection. This consideration is the primary reason behind our decision to utilize a linear probe on CLIP rather than on BLIP-2 in the main paper."}, {"review_id": "M2zU09tpON", "reviewer": "Reviewer_ZwSf", "summary": "The paper studies the problem of data pre-selection, which aims to select instances for labeling from an unlabeled dataset to enhance performance for downstream tasks with a limited annotation budget. The authors suggest that combining visual and textual features in a joint space can result in a better representation for data pre-selection. They introduce UP-DP, an unsupervised prompt learning approach that adapts vision-language models(specifically BLIP-2) for data pre-selection. The proposed approach outperforms the state-of-the-art on benchmark datasets and exhibits generalizability.", "questions": "1. Is it fair to use the prompt for CLIP model to assess the zero-shot performance of the BLIP-2 model? It's supposed to train the prompt for BLIP-2 from scratch.\n2. Please demonstrate that UP-DP is valid and effective for other tasks like detection and segmentation.\n3. Table 1 requires revision and additional information. It should incorporate a baseline that utilizes only image features extracted from BLIP-2. Also, it is beneficial to compare with more approaches in the field.\n4. Please include more ablation studies. At least, demonstrate the impact of the instance-level and cluster-level contrastive loss.\n5. During the data pre-selection stage, all the features are extracted from BLIP-2 model. However, it remains unclear whether the method is exclusively effective for this specific model. It is recommended that the authors demonstrate the effectiveness of using other vision and language models as well.", "limitations": "The authors need to analysis the limitations of their work.", "rating": 5, "confidence": 2, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a novel approach for data pre-selection that incorporates unsupervised prompt learning in the vision-language model. This approach effectively exploits the multimodal features and enhances the discrimination among classes.\n2. The authors provide a clear motivation for the task of data pre-selection and highlight the unique challenges it poses compared to semi-supervised learning and active learning.\n3. The paper compares with the state-of-the-art on multiple benchmark datasets, demonstrating its effectiveness and superior performance. Also, it highlights the generalizability of the learned prompts across different datasets.", "weaknesses": "1. The authors claim that the purpose of the data pre-selection is to optimize performance for undefined diverse downstream tasks; however, they only conduct experiments on the image classification task. This limited scope is insufficient to demonstrate the effectiveness of UP-DP for various downstream tasks such as detection or segmentation. The paper would benefit from conducting more experiments on other tasks. And on top of this, it would be interesting to investigate the generalizability of the learned prompts across different tasks.\n2. In Table 1, the \"Zero-Shot BLIP-2\" setting is unreasonable. It lacks a justifiable rationale to use the prompt for the CLIP model to evaluate the zero-shot performance of the BLIP-2 model. If the authors intend to use this baseline, they should train the learnable prompt for BLIP-2 from scratch.\n3. In Table 1, the author does not include a baseline that utilizes only image features extracted from BLIP-2. By comparing with this baseline, the authors can demonstrate the efficiency of the proposed method.\n4. In Table 1, the authors compare \"Random\", \"USL-I/M\", but it is suggested to compare with more approaches in the field. Including additional comparisons would strengthen the paper's evaluation and provide a better context for understanding the performance of the proposed approach.\n5. The ablation studies of the proposed method are limited. It would be beneficial to conduct more comprehensive experiments to analyze the performance of UP-DP under different settings. For instance,  the authors should provide a detailed analysis of the impact of the instance-level and cluster-level contrastive loss.\n6. All the experiments are carried out using the BLIP-2 model during the data pre-selection stage. However, it remains unclear whether the proposed method is exclusively effective for this particular model. It is important to consider other visual-language models, such as CLIP, to establish the efficacy of the approach.\n7. The paper contains several grammar issues. For example, on page 5, line 191, it states \"presents an efficient pre-training,\" and on page 6, line 237, it states \"Thus we can from a positive pair.\" These sentences require revision for improved clarity and grammatical correctness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer ZwSf,\n\nWe would like to thank you for your valuable comments. Below is our response to your questions:\n\nQ1: “Is it fair to use the prompt for CLIP model to assess the zero-shot performance of the BLIP-2 model? It's supposed to train the prompt for BLIP-2 from scratch.” [Question context: “It lacks a justifiable rationale to use the prompt for the CLIP model to evaluate the zero-shot performance of the BLIP-2 model. If the authors intend to use this baseline, they should train the learnable prompt for BLIP-2 from scratch.”]\n\nA1: As discussed between lines 280 to 289, our primary goal isn't to benchmark against BLIP-2's zero-shot performance. Instead, we aim to emphasize the necessity of adapting BLIP-2 for data pre-selection. This is especially relevant for certain datasets like EuroSAT, which is an Out-Of-Distribution (OOD) dataset for BLIP-2, leading it to make random classifications under the zero-shot settings.\n\nUsing the prompt designed from CLIP to BLIP-2 is suitable since these prompts aren’t model-specific but dataset-specific and are crafted by human experts for general zero-shot classification. As shown in Appendix C, Table 6, For example, the prompt of OxfordPets dataset is “a photo of a [CLASS], a type of pet.”\n\nQ2: “Please demonstrate that UP-DP is valid and effective for other tasks like detection and segmentation.”\n\nA2: Thank you for your insightful suggestion. In response, we have conducted additional experiments specifically for the segmentation task utilizing the PASCAL VOC dataset. As demonstrated in Table B, our UP-DP method indeed outperform other baselines in semantic segmentation tasks. More details can be found in the \"global\" response to all reviewers.\n\nQ3: “Table 1 requires revision and additional information. It should incorporate a baseline that utilizes only image features extracted from BLIP-2. Also, it is beneficial to compare with more approaches in the field.”\n\nA3: Thank you for the suggestions. USL-I in Table 1 is the baseline that uses image features extracted from BLIP-2, the detail is described in lines 276-279 and 307-308, which shows that our method can outperform the method only using image features.\n\nWe certainly recognize the importance of comprehensive comparisons with various approaches. However, as elaborated in the Related Works section, other data efficiency approaches like active learning-based methods don't align with the specific requirements of data selection in our context, as they usually necessitate an initial labeled set and predefined downstream models. To our best understanding, USL is the most relevant approach for comparison in this study. \n\nQ4: “Please include more ablation studies. At least, demonstrate the impact of the instance-level and cluster-level contrastive loss” \n\nA4: We greatly appreciate your request for additional ablation studies, particularly regarding the impact of both instance-level and cluster-level contrastive losses. Following your valuable suggestion, we are currently conducting more in-depth ablation studies that delve into the following key aspects: \n\n1.\tDifferent Sampling Strategies (Table A)\n2.\tDifferent Versions of BLIP-2 (Table C)\n3.\tBalance and Influence of Loss Weights (Table D)\n4.\tImpact of Varied Annotation Budgets (Table E)\n5.\tBLIP-2 Training with and without Prompt (Table F)\n6.\tDifference Length of Learned Prompt (Table 4 in Appendix)\n\nIn Table D, as illustrated, we've experimented with a range of weight ratios, spanning from 1:3 to 3:1, for two distinct losses within different versions of BLIP-2. The consistency of the results highlights the significance of both loss components during training. Moreover, your suggestion proved fruitful; employing a 3:1 ratio instead of a 1:1 ratio, as showcased in Table D, has demonstrated further performance enhancement for our approach.\n\nQ5: “During the data pre-selection stage, all the features are extracted from BLIP-2 model. However, it remains unclear whether the method is exclusively effective for this specific model. It is recommended that the authors demonstrate the effectiveness of using other vision and language models as well.”\n\nA5: We appreciate the recommendation to explore the effectiveness of our method with various vision and language models. The primary reason for utilizing BLIP-2 is its strong interaction between vision and language parts at a fine-grained level (namely, the tokens of image patches, and text). Through slight training adjustments to the prompt, we were able to efficiently enhance the extracted features, thereby making BLIP-2 a suitable model for data pre-selection. However, regarding “old” vision-language (V-L) models like CLIP, the interaction is limited to a cosine similarity score between the embeddings of the whole image and the full text, and this prevents us from extracting multi-modal features with fine-grained interaction between image patches and text tokens. Although we would like to extend our method to more advanced V-L models beyond BLIP-2, it currently represents the most recent and suitable V-L model.\n"}, {"review_id": "15FoebZD3X", "reviewer": "Reviewer_soCe", "summary": "This paper presents an unsupervised approach for data preselection, which aims to select instances for labeling from an unlabeled dataset in a single pass. The authors leverage the text features in multimodal models, specifically BLIP2, to enhance the representation for data preselection. They argue that a well-designed joint feature space of vision and text can yield improved results. To achieve this, they train text prompts to extract joint features with enhanced representation, ensuring a diverse cluster structure that covers the entire dataset. The authors employ two loss functions, namely instance-level contrastive and cluster-level contrastive, to train the learnable text prompts. These loss functions encourage the adapted multimodal model's joint representation space to be more diverse and well-separated, suitable for clustering. Experimental results on seven different datasets, along with a comparison against three baselines, demonstrate the effectiveness of the proposed approach.", "questions": "Following questions address the weaknesses and limitations mentioned earlier. Asking the authors these questions could help improve the paper and provide a more comprehensive understanding of the proposed approach:\n\n1. Could you compare your proposed method when built on top of CLIP as well? Including a comparison with CLIP as a baseline would provide valuable insights into the effectiveness of your approach and its advantages over a widely used vision and language model.\n\n2. Could you provide some visualizations, such as attention weight visualizations, to explain why the trained prompts lead to better data preselection and generalization? Visualizations would enhance the interpretability of your method and shed light on the mechanisms behind its improved performance.\n\n3. Have you replicated your proposed method on other variants of the BLIPv2 model? It would be insightful to see how your approach's performance changes when applied to BLIPv2 models with varying levels of expressiveness, particularly when the model size decreases. This analysis would help assess the scalability and adaptability of your method to different model configurations.", "limitations": "There are not any limitations. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I believe the strength of the paper is as follows:\n\n- The integration of vision and language models for data pre-selection holds promise due to the added benefits of leveraging text modality. \n\n- This paper introduces a novel approach that promotes diverse and clustered representations, addressing limitations in the current state-of-the-art BLIP2 model through the lens of prompt learning. \n\n- The proposed method is lightweight and more efficient in terms of training costs. The paper makes a significant contribution, evident in its clear presentation and compelling results.", "weaknesses": "While the paper's results are strong, there are two notable aspects that could be addressed:\n\n1. Missing CLIP baseline: Comparing the proposed approach with a CLIP baseline would provide a fundamental point of reference. Since the authors employ contrastive loss functions and prompt learning, which can be applied to any vision and language model, including CLIP as a baseline would enhance the comparative analysis.\n\n2. Lack of integrability: A limitation of the prompt tuning method is the lack of interpretability. While prompt tuning improves model performance, it does not provide a clear explanation of why and how the model works in the combined language model embedding space. Addressing this limitation would enhance the understanding and justification of the proposed approach.\n\n3. Performance/training-time trade-off: The paper utilizes the best-performing model of BLIPV2, which has over 7 billion parameters. However, it does not explore the performance and training-time trade-off with different BLIPV2 models. Investigating the performance of the proposed approach on BLIPV2 models with fewer parameters would provide insights into its scalability and suitability for models of varying sizes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer soCe,\n\nWe would like to thank you for your valuable comments. Below is our response to your questions:\n\nQ1: “Could you compare your proposed method when built on top of CLIP as well? Including a comparison with CLIP as a baseline would provide valuable insights into the effectiveness of your approach and its advantages over a widely used vision and language model.\"\n\nA1: Thank you for the suggestion to include a comparison with CLIP as a baseline. The decision not to utilize CLIP for data pre-selection in our study stems from the specific characteristics of CLIP's interaction between vision and language components, as outlined in our response to reviewer ZwSf (A5).\n\nQ2: “Could you provide some visualizations, such as attention weight visualizations, to explain why the trained prompts lead to better data preselection and generalization? Visualizations would enhance the interpretability of your method and shed light on the mechanisms behind its improved performance.”\n\nA2: Thank you for your insightful recommendation. In fact, we have already taken steps in this direction by providing visualizations and analyses of the learned prompts, as detailed in Appendix A, Table 3. We observed that certain learned words like \"wood\" in the EuroSAT dataset (related to the forest class), specific numbers in FGVCAircraft (possibly representing aircraft codes), and words such as \"plane\" and \"butterfly\" in different contexts demonstrate relevance to their respective tasks. However, many of the learned words lack a coherent connection to the tasks, leading us to hypothesize that the encoded meanings may extend beyond the existing vocabulary's scope. Interestingly, we also found shared words across different datasets, including \"learn,\" \"saint,\" \"add,\" and \"attend.\" These commonalities could explain the substantial generalizability of the learned prompts across various tasks.\n\nQ3: “Have you replicated your proposed method on other variants of the BLIPv2 model? It would be insightful to see how your approach's performance changes when applied to BLIPv2 models with varying levels of expressiveness, particularly when the model size decreases. This analysis would help assess the scalability and adaptability of your method to different model configurations”\n\nA3: Thank you for the valuable suggestion. We conducted additional experiments with different versions of BLIP-2 (v1 built with ViT-L and v2 built with ViT-G). The results are presented in Tables C and D. Our method outperforms other baselines using both versions, with the larger BLIP-2 v2 (with ViT-G) demonstrating the best performance.\n\nEven though BLIP-2 boasts over 7 billion parameters, our method exclusively trains a few specific components: the parameters of prompts (4 vectors), instance-head, and cluster-head. Consequently, the count of trainable parameters remains below 0.25 million, necessitating less than 60 minutes on a single GPU (RTX 3090) for training. The variation of BLIP-2 will not alter the trainable parameters, ensuring straightforward scalability.\n\nHere are more details regarding the variants of BLIP-2. It comprises two variants for the image encoder component: ViT-G and ViT-L, as well as four variants for the large language models side: OPT_2.7B, OPT_6.7B, FlanT5_XL, and FlanT5_XXL. For our data pre-selection task, we only require the Image encoder component without the language generation part, which narrows our options down to BLIP-2 with ViT-G and ViT-L."}, {"review_id": "TYQrDWeuWW", "reviewer": "Reviewer_uJMt", "summary": "The paper addresses data pre-selection (akin to active learning) problem using the highly successful vision-language models (VLMs) of late. In relation to existing approaches, the proposed approach has a few advantages, e.g., no need to have a small initial set of labeled data, no need to have multiple rounds of selection, labeling and retraining, once selected the data can be used for multiple future, unknown downstream tasks etc. The authors start with a BLIP-2 model and an unlabeled set of data $D$. The BLIP-2 model is extended to have learnable context/prompt and a few MLPs, namely instance-level and cluster-level heads. The instance-level head is employed to produce a contrastive training between two views of each unlabeled instance. Cluster-level head, on the other hand, first assigns cluster memberships to the instances and then helps to train the model with a cluster-level contrastive loss. After training for a few epochs using these two losses, cluster-level MLP along with the learned contexts/prompts are used to get the cluster assignment of unlabeled data (which can come from different downstream dataset). Finally, the medoids from each cluster form the representative selections for active learning. Experiments performed on linear probing and domain generalization show the efficacy of the proposed approach over the state-of-the-arts on benchmark datasets.", "questions": "Questions are already asked above (in the ‘Weaknesses’ section). Those are mostly queries for further clarification. Here in addition, let me list a few presentation related issues (typos mainly).\n - Line 32, 78, Figure 1 caption: Will these be ‘data efficient learning’ instead of ‘data efficiency learning’?\n - Line 231: ‘map’ -> ‘maps’\n - Line 244: ‘optimizing’ -> ‘optimization’\n - Line 246: ‘combine’ -> ‘combination’\n - In Figure 2a, which feature extractor is used? Is it anything different from BLIP-2?", "limitations": "The limitations are described well in the paper. At the same time, the authors tried to address the limitation in the supplementary material.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The use of VLMs for unsupervised active learning is appreciable. VLMs. Now-a-days, are known for good zero-shot transfer. The already well-learned representations can and did help the active learning cause.\n2. The use of learnable contexts/prompts has been shown to be useful for few-shot transfer. The use of these for active learning is interesting.\n3. The use of cluster-level contrastive loss cleverly avoids the use of any initial labeled set that is required in traditional cluster level losses in getting the initial clustering (akin to group contrastive loss in semisupervised literature e.g., [a]).\n4. Experimental analysis and ablations show the efficacy of the proposed approach compared to sota approaches and the importance of different components of the approach as well.\n\n[a] Singh et al., Semi-supervised action recognition with temporal contrastive learning, CVPR 2021.", "weaknesses": "1. One important ablation that could be useful is running the approach without the learnable prompts/contexts. What I mean is updating only $g_I$ and $g_C$ but not employing $V$ in Algorithm 1. This will help gauge the importance of the contexts/prompts vis-à-vis the instance and cluster level MLPs. Does ‘Initi_Prompt’ row in Table 2 do this?\n2. Line 260+: This is more of a clarification query. When the datasets are described, I don’t see any mention of which dataset is used for pre selection. I am assuming these 7 datasets are downstream task datasets. The question is coming from Table 1. While in Table 2, it seems that the first column tells what is the dataset on which the prompts are learnt, in table 1, it is not clear. Is it that the learning is done on the same datasets on which the linear probing performances are shown for Table 1?\n3. Line274+: I am not getting what is meant by 'with the learned prompts' in the baseline using USL. Does it mean everything else here is same as BLIP-2, but in addition a few prompts are learned also?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer uJMt,\n\nWe would like to thank you for your valuable comments. Below is our response to your questions:\n\nQ1: “One important ablation that could be useful is running the approach without the learnable prompts/contexts. What I mean is updating only and but not employing in Algorithm 1. This will help gauge the importance of the contexts/prompts vis-à-vis the instance and cluster level MLPs. Does ‘Initi_Prompt’ row in Table 2 do this?.”\n\nA1: Thank you for the insightful suggestion to run an ablation study without the learnable prompts/contexts. We have conducted the suggested study, and the results are presented in Table F. These findings clearly demonstrate the necessity of training the prompts. The motivation behind training the prompts/contexts is to enable the model to extract features more suited to specific datasets. If we only train g_I and g_C without adjusting the prompts/contexts, the image features would remain static, potentially slowing convergence and lowering performance.\n\nQ2: “Line 260+: This is more of a clarification query. When the datasets are described, I don’t see any mention of which dataset is used for pre selection. I am assuming these 7 datasets are downstream task datasets. The question is coming from Table 1. While in Table 2, it seems that the first column tells what is the dataset on which the prompts are learnt, in table 1, it is not clear. Is it that the learning is done on the same datasets on which the linear probing performances are shown for Table 1?”\n\nA2: Thank you for bringing up this clarification query. Your assumption is correct. In Table 1, the same dataset is used both for pre-selection and evaluation in the linear probing experiments. Let me explain the process using the DTD dataset as an example: We first train a prompt for BLIP-2 using images from the DTD dataset. This trained prompt is then used to extract features and conduct data pre-selection within the same DTD dataset. Finally, we perform a linear probe to evaluate the quality of the selected instances within the DTD dataset itself. Table 2 serves a different purpose, as it evaluates the generalization of our method by using prompts learned from one dataset to make pre-selections in another. For instance, the intersection of Caltech101_Prompt and DTD means that we use the prompt learned from the Caltech101 dataset to extract features from the DTD dataset and select instances within DTD.\n\nQ3: “Line274+: I am not getting what is meant by 'with the learned prompts' in the baseline using USL. Does it mean everything else here is same as BLIP-2, but in addition a few prompts are learned also?”\n\nA3: To perform the USL method, it requires image features. USL-I uses the image features extracted directly from the BLIP-2 model's image part without any additional training. USL-M uses multimodal features that have been learned through our proposed methods."}, {"review_id": "B1fPZZgiso", "reviewer": "Reviewer_X1Uz", "summary": "This paper investigates the task of data pre-selection by learning a better representation from the joint feature space of both vision and text in an unsupervised manner. The paper focuses on training text prompts to extract joint features with enhanced representation, specifically with the BLIP-2 parameters kept fixed. The aim is to achieve a diverse cluster structure that encompasses the entire dataset.", "questions": "Please refer to the weaknesses part.", "limitations": "Please refer to the weaknesses part.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**[New task]** This paper tackles data pre-selection for labelling without accessing the information of downstream tasks, which is quite new to the community.\n\n**[Well-illustrated figures]** The figures shown in this paper are clear enough for better understanding.\n\n**[Good presentation]** The paper is well-written and easy to follow.\n", "weaknesses": "**[Unconvincing illustration]** In Figure 1, BLIP-2 is pre-trained with prompt and image together. This explains why using only image features yields poor performance. Consequently, the evidence presented does not convincingly demonstrate the superiority of multimodal features.\n\n**[Need in-depth analysis]** (i) It is unclear why the self-trained model can be used for sample selection. (ii) The motivation of medoid selection is not given. It would be nice to see the rationale. \n\n**[Missed ablation studies]** The ablation study for two hyperparameters are not given.\n\n**[Disorganized reference format]** Please reformat the references as per some published papers.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer X1Uz,\n\nWe would like to thank you for your valuable comments. Below is our response to your questions:\n\nQ1: “BLIP-2 is pre-trained with prompt and image together. This explains why using only image features yields poor performance. Consequently, the evidence presented does not convincingly demonstrate the superiority of multimodal features.”\n\nA1: You're correct in noting that BLIP-2 is pre-trained using both prompts and images, and I appreciate your concern regarding the demonstration of the superiority of multimodal features. The results presented in Figure 1d and Tables 1 & 2 already illustrate the superiority of using multimodal features.\n\nQuantitatively, Table 1 demonstrates that our multimodal approach (\"Ours\") surpasses the use of Image features (“USL-I”) in data pre-selection tasks. Table 2's KNN classification results further emphasize the superior quality of multimodal features. Specifically, utilizing Flowers102_Prompt for multimodal extraction outperforms image features by an impressive 10% in absolute value. Moreover, Figure 1d visually reinforces this advantage. It clearly displays how multimodal features offer a more distinct and scattered distribution compared to image features. This distinction significantly enhances our ability to differentiate between classes, ultimately improving the task of data preselection and KNN classification.\n\nAdditionally, our method not only offers improved multimodal features but also provides an enhanced joint sampling strategy. These improvements collectively enhance the performance of the data pre-selection task, as demonstrated in Table A in the rebuttal.\n\nQ2: “It is unclear why the self-trained model can be used for sample selection. The motivation of medoid selection is not given. It would be nice to see the rationale.”\n\nA2: As demonstrated in recent literature [1][2], the first step in data selection is to obtain lower-dimensional and semantically meaningful features, which are usually acquired by models [3][4][5] using self-supervised learning. Consequently, self-trained models play a pivotal role in the data selection task.\n\nIn our approach, as shown in Table 1, certain datasets (e.g. EuroSAT) prove to be effectively out-of-distribution (OOD) for BLIP-2, resulting in random guessing during zero-shot performance. To address this, we employ contrastive learning techniques to fine-tune BLIP-2's feature representation specifically for these previously unseen datasets. The enhanced feature representation subsequently leads to more accurate clustering, facilitating the identification of representative instances for the entire dataset.\n\nThe concept of utilizing a medoid (or centroid) to identify a representative sample within a cluster is a widely adopted practice [6][7]. This approach is recognized for its effectiveness in selecting a sample that lies at the center of the cluster, ensuring a faithful representation. Furthermore, as highlighted in Table A, we integrate the output of the cluster head to directly sample the representative instance from each cluster, thereby adding an additional dimension to our approach. Further details can be found in the \"global\" response to all reviewers.\n\nQ3: “The ablation study for two hyperparameters are not given.”\n\nA3: We appreciate your observation regarding the concern about the ablation study for the two hyperparameters. We recognize this aspect and have addressed it by conducting the ablation study (Table D). Further details can be found in our response to reviewer ZwSf (A4).\n\n[1] Wang, Xudong, Long Lian, and Stella X. Yu. \"Unsupervised selective labeling for more effective semi-supervised learning.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[2] Xia, Xiaobo, et al. \"Moderate coreset: A universal method of data selection for real-world data-efficient deep learning.\" The Eleventh International Conference on Learning Representations. 2022.\n\n[3] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International conference on machine learning. pp. 1597–1607. PMLR (2020)\n\n[4] He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9729–9738 (2020)\n\n[5] Oord, Aaron van den, Yazhe Li, and Oriol Vinyals. \"Representation learning with contrastive predictive coding.\" arXiv preprint arXiv:1807.03748 (2018).\n\n[6] Max Welling. Herding dynamical weights to learn. In ICML, pp. 1121–1128, 2009.\n\n[7] Sorscher, Ben, et al. \"Beyond neural scaling laws: beating power law scaling via data pruning.\" Advances in Neural Information Processing Systems 35 (2022): 19523-19536.\n"}]

Author rebuttal:
We extend our gratitude to all the reviewers for their meticulous comments and constructive suggestions. We are heartened by the reviewers' keen interest in our work and their recognition of its novelty, both in terms of the task and methodology.

We wish to emphasize the significance of our approach. Our proposed method not only enhances the feature representation of the dataset but also trains a cluster head capable of jointly selecting the most representative instances while encompassing the entire dataset for data pre-selection. We substantiate this through our supplementary experiment, Table A, included in the attached one-page PDF.

Appreciating the reviewers' input, we achieved further performance gains through the requested ablation studies. By exploring various sampling strategies and fine-tuning hyperparameters for the two losses, we elevated our method's performance, as evidenced in Table A and D.

Moreover, in response to reviewers' suggestions, we have conducted an experiment on the semantic segmentation task utilizing the Pascal VOC dataset [1]. We have also added more ablation studies for the classification task. These experiment results are available in the attached one-page PDF, organized as follows:

1. UP-DP Sampling Variant with Learned Heads (Table A), as requested by Reviewer X1Uz.

2. Semantic Segmentation Task Experiment (Table B), as requested by Reviewer ZwSf.

3. Impact of Different Versions of BLIP-2 (Table C), as requested by Reviewer soCe.

4. Linear Probe Result with BLIP-2 (Tables C-F), as requested by Reviewer ucTW.

5. Impact of Weights between Instance-level and Cluster-level Loss (Table D), as requested by Reviewers ZwSf and X1Uz.

6. Impact of Annotation Budget (Table E), as requested by Reviewer ucTW.

7. Impact of Training without Prompt (Table F), as requested by Reviewer uJMt.

Here is additional information about these experiments:

Table A: Beyond employing multimodal features from 'f' to select medoids to post unsupervised prompt learning, we explored multiple variants. Table A reveals that 'g_I' utilizes the projected feature from the instance head to identify medoids, while 'g_C' employs cluster probability predicted by the cluster-level head. This approach selects instances with the highest confidence score for each cluster, resulting in a significant performance enhancement.

Table B: We employed the Pascal VOC dataset for the semantic segmentation task, employing the variants of the latest DINOv2 [2] as the segmentation models. The total number of training instances was 1454, and we set the annotation budget at 100. Table B demonstrates our method's consistent superiority over other baselines in segmentation tasks.

Tables C-F: To accommodate time constraints, we chose the relatively challenging DTD dataset (with the fewest instance) for our ablation study. The annotation budget for Tables C, D, and F was set at 100, with a sampling strategy utilizing the distribution peak from cluster output by  'g_C', as previously mentioned."

[1] Everingham, Mark, et al. "The pascal visual object classes challenge: A retrospective." International journal of computer vision 111 (2015): 98-136.

[2] Oquab, Maxime, et al. "Dinov2: Learning robust visual features without supervision." arXiv preprint arXiv:2304.07193 (2023).


Meta-review: The submission studies data pre-selection (i.e., select instances for labeling from an unlabeled dataset), to optimize performance for undefined downstream tasks with a limited annotation budget. Prior work relied solely on visual features. This work shows that both vision and text can yield a better representation for data pre-selection. Reviewers raised some initial concerns related to missing ablation studies. The authors provided those results. Reviewers arrived at an unanimous borderline accept recommendation. AC sees no reason to overturn an unanimous albeit borderline accept recommendation.
Decision: Accept (poster)