Title: UP-DP: Unsupervised Prompt Learning for Data Pre-Selection with Vision-Language Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Unsupervised prompt learning

UP-DP

Data preselection
Abstract: In this study, we investigate the task of data pre-selection, which aims to select instances for labeling from an unlabeled dataset through a single pass, thereby optimizing performance for undefined downstream tasks with a limited annotation budget. Previous approaches to data pre-selection relied solely on visual features extracted from foundation models, such as CLIP and BLIP-2, but largely ignored the powerfulness of text features. In this work, we argue that, with proper design, the joint feature space of both vision and text can yield a better representation for data pre-selection. To this end, we introduce UP-DP, a simple yet effective unsupervised prompt learning approach that adapts vision-language models, like BLIP-2, for data pre-selection. Specifically, with the BLIP-2 parameters frozen, we train text prompts to extract the joint features with improved representation, ensuring a diverse cluster structure that covers the entire dataset. We extensively compare our method with the state-of-the-art using seven benchmark datasets in different settings, achieving up to a performance gain of 20\%. Interestingly, the prompts learned from one dataset demonstrate significant generalizability and can be applied directly to enhance the feature extraction of BLIP-2 from other datasets. To the best of our knowledge, UP-DP is the first work to incorporate unsupervised prompt learning in a vision-language model for data pre-selection.

UP-DP: Unsupervised Prompt Learning for Data Pre-Selection with Vision-Language Models

Abstract

In this study, we investigate the task of data pre-selection, which aims to select instances for labeling from an unlabeled dataset through a single pass, thereby optimizing performance for undefined downstream tasks with a limited annotation budget. Previous approaches to data pre-selection relied solely on visual features extracted from foundation models, such as CLIP and BLIP-2, but largely ignored the powerfulness of text features. In this work, we argue that, with proper design, the joint feature space of both vision and text can yield a better representation for data pre-selection. To this end, we introduce UP-DP, a simple yet effective unsupervised prompt learning approach that adapts vision-language models, like BLIP-2, for data pre-selection. Specifically, with the BLIP-2 parameters frozen, we train text prompts to extract the joint features with improved representation, ensuring a diverse cluster structure that covers the entire dataset. We extensively compare our method with the state-of-the-art using seven benchmark datasets in different settings, achieving up to a performance gain of 20%. Interestingly, the prompts learned from one dataset demonstrate significant generalizability and can be applied directly to enhance the feature extraction of BLIP-2 from other datasets. To the best of our knowledge, UP-DP is the first work to incorporate unsupervised prompt learning in a vision-language model for data pre-selection.

1 Introduction

Data-efficient machine learning, which aims to identify the best subsets of data samples (to either label or not) in order to achieve optimal model performance, has been a crucial research field in the era of data-hungry deep learning [12, 31, 24, 40, 9]. In this work, we focus on a new and practical task of data pre-selection for data-efficient visual object recognition (Fig.1-a). The goal of data pre-selection is to select instances for labeling from an unlabeled dataset through a single pass to maximize model performance for unknown downstream vision tasks (e.g., no knowledge about prediction categories, shown in Fig.1-b), given a limited annotation budget. This task is motivated by two practical needs: at the data acquisition stage, we want to collect minimal data to support potentially diverse downstream tasks (e.g., classification, action recognition, or even detection for videos or images in Fig.1-b) and training paradigms (e.g. active learning, semi-supervised learning, supervised learning in Fig.1-c); at the annotation stage, we aim to achieve good performance with fewer human annotations over the pre-selected data.

Traditionally, to tackle the challenge of data efficiency learning, there are two main approaches: Semi-supervised learning (SSL) [3, 35, 21] and Active Learning (AL) [34, 10, 2]. SSL aims to address the problem of label scarcity by leveraging a limited quantity of labeled data in conjunction with a more abundant pool of unlabeled data to enhance the model's performance. By contrast, AL approaches start with an initial set of labeled data and select the most informative data point to label

Selected

labeled

Instances

Labeling

Instances

Supervised learning

Dataset

Classifier

Pre-selection

Data

Unlabeled

TaskDefinition

Instances

(e.g.,label requirements)

Semi-supervised learning

Classifier

Activelearning

Classifier

OxfordPets

Dataset

withLearnedPrompt

Multi-modalFeature

withRandomPrompt

Multi-modalFeature

ImageFeatureOnly

Labeling

in an effort to maximize model performance with a limited label budget. However, existing AL methods need an initial labeled set to begin with and need to train a series of models.

Nonetheless, the task of data pre-selection poses some unique challenges that SSL and AL, as mentioned above, fail to adequately address. In data pre-selection, we do not have initial labeled data or know the specific downstream task. Take object detection as an example: we do not know what categories of objects need to be detected. However, both current SSL and AL methods need both initial labeled data and specific learning tasks. Therefore, this requires a method to select diverse and representative instances to cover the entire dataset with a minimum number of required labels. To achieve this, it is essential to acquire semantically meaningful, high-quality features.

Fortunately, recent advances in foundational Vision-Language (V-L) models, such as CLIP [32] and BLIP-2 [23], hold significant promise in addressing this problem. These models acquire comprehensive semantic knowledge by learning from a vast collection of image-text pairs. Consequently, they possess the ability to extract meaningful features from diverse input modalities. While there exists a numerous of studies that adapt V-L models for various tasks like few-shot classification [48], semi-supervised learning [46], and selective labeling [40], they often place emphasis on unimodal features, primarily vision, and inadvertently overlook the potential contributions of multimodal features encompassing both vision and language.

In this paper, we argue that by carefully designing the prompt, the multimodal features extracted by V-L models can offer a more powerful representation for data pre-selection. Figure 1 illustrates the unimodal and multimodal feature representations obtained from BLIP-2, with different colors representing various ground truth labels. In particular, when the class information of downstream tasks is unknown, the unimodal features become mixed, making it challenging to identify specific classes. In contrast, when coupled with an appropriate prompt, multimodal features exhibit a more scattered and distinct distribution, allowing for improved discrimination among classes. This enhanced discrimination is crucial in helping users identify the most representative and diverse instances during the data pre-selection process.

However, the process of designing an appropriate prompt can be demanding and time-consuming, often requiring trial and errors [48]. Moreover, current automatic prompt learning approaches [48, 47, 19] typically rely on a small set of labeled or pseudo labeled data, making them impractical for data pre-selection when the downstream tasks are undefined. To address this challenge, we propose a novel approach: pure unsupervised prompt learning, which aims to enhance the multimodal features extracted from V-L models for data pre-selection. Our contributions include:

We introduce UP-DP , an innovative Unsupervised Prompt learning for enhanced Data Pre-selection in Vision-Language models. To the best of our knowledge, UP-DP is the

pioneering work that incorporates unsupervised prompt learning in a vision-language model specifically for data pre-selection.

We demonstrate the robust generality of the learned prompt across different datasets, which significantly enhances feature extraction when applied in a plug-and-play manner.

We extensively benchmark our method using various architectural settings, achieving superior performance compared to other competitors on 7 image classification datasets.

2 Related Works

2.1 Data Efficiency Learning

Semi-supervised Learning (SSL) combines labeled and unlabeled data to enhance learning, and can be seen as unsupervised learning with added labeled data. SSL approaches are divided mainly by how they leverage large amounts of unlabeled data. Consistency Regularization methods [33, 42] maintain model output stability under realistic perturbations like data augmentation, with UDA [42] being an example that extends supervised data augmentation to boost SSL performance. Pseudolabeling methods [4, 41, 5] use high confidence model predictions to generate pseudo-labels for unlabeled data, training them jointly with labeled data. For example, SoftMatch [5] addresses the quantity-quality trade-off in pseudo-labeling using a truncated Gaussian function to weight samples by confidence. Transfer learning-based methods such as SimCLRv2 [6] involve supervised fine-tuning after unsupervised pre-training on unlabeled datasets, demonstrating the effectiveness of large models for SSL.

However, the success of these methods is based on the assumption of stratified sampling [4, 35], which requires equal sampling of each known class, a constraint that is often unattainable in practice. More importantly, our data pre-selection task even does NOT have any assumption for downstream tasks and also does NOT know any information about class categories.

Active Learning (AL) actively selects high-value instances for labeling during the iterative training process to enhance predictive performance, offering a realistic alternative to the stratified sampling setting of SSL. AL strategies generally fall into two categories: (1) methods [15, 44] that select informative instances using a scoring function such as uncertainty and (2) methods [34, 43] that choose diverse instances representing the dataset's domain. Some research [30, 1] addresses both aspects, such as ALFA-Mix [30], which evaluates label variability for perturbed instances and ensures diversity by clustering and selecting centroids.

However, current AL methods have limitations. They often require an initial labeled set, which is inefficient in low-label settings, and involve multiple rounds of labeling and training with a human annotator, making the process time-consuming. On the other hand, the requested labels are tightly associated with the training objective, necessitating distinct instances for each task. In contrast, our data pre-selection task aims for single-pass labeling applicable to any future, unknown downstream task, minimizing human effort and overall cost.

Unsupervised Selective Labeling (USL) [40] is the most relevant approach to ours. They proposed an SSL pipeline that avoids the unrealistic setting of stratified sampling. Specifically, they first apply unsupervised feature learning (e.g., MoCov2 [7]) to map data into a discriminative feature space. Then, they select instances for labeling for maximum representativeness and diversity in a single pass. Finally, with a strong initialization from the first step, they apply SSL to the labeled data and the remaining unlabeled data. Although their method achieves great performance and even beats stratified sampling in some settings, it is not suitable for data pre-selection. Using the same model and weights for both the selection step and the downstream task step is not a valid assumption.

2.2 Vision-Language Foundation Models

Vision-language models, pre-trained on large-scale image-text pairs, excel in visual representation learning. A taxonomy of V-L models [20] can be based on two aspects: (1) the expressiveness of both modalities in terms of parameters and computation, and (2) the extent of interaction within a Deep Neural Network (DNN). Early Visual Semantic Embedding (VSE) models, such as VSE++ [13] and SCAN [37], use separate feature extractors for image and text modalities, with the image extractor being more complex. They represent similarity using dot products or shallow attention layers.

(a)DataPre-selectionWorkflow

SelectPointsfromEachCluster

Feature

Labeling

Downstream

Extractor

Tasks

SelectedInstances

UnsupervisedPrompt

Feature w/o Prompt

Learning

W/OPrompt

FeaturewithPrompt

(b) Unsupervised Prompt Learning

L1

Attraction

↑→

Repulsion

Instance-level Contrastive;

Learnable Context

g1()

Z

Instance

Loss

Learning

[V] [V]2

[V]

↑

Clustering

Augmented

hMLPs

LossL

f()

↑

BLIP-2

gc()

C

Cluster

Multimodal

Loss

Embeddings

Lc

Cluster-levelContrastiveLearning

(c)Prompt Generalizationfor OtherDatasets

Multimodal Embeddings

Learned Context

[V]i [v]2

[V]n

FromPets

Better

BLIP-2

Representation

Flowers Images

Dataset

FeaturewithPromptFeaturew/oPrompt

In contrast, CLIP [32] uses comparably complex feature extractors (transformers [38]) for each modality but maintains a shallow interaction. Recent models [20, 22, 39] use deep transformers for inter-modal interactions, enhancing performance in complex vision-language tasks. However, pre-training becomes computationally expensive with large feature extractors and interaction models. BLIP-2 [23] introduces a querying transformer (interaction) that utilizes pre-trained image and language models and captures robust interaction between the image and text. The interaction of two modalities allows us to effectively manage the multimodal features extracted from the V-L model merely by modifying the input text (prompt). However, determining the optimal prompt for data pre-selection remains an open question. In this work, we designed a novel approach of generating text prompts to extract multimodal features without any additional information.

2.3 Adaptation of Vision-Language Foundation Models

Recent advances in multimodal pre-training models have shown great success in classic vision tasks [32], prompting interest in more efficient adaptation models.

To avoid the time-consuming process of fine-tuning an over-parameterized V-L model, two main strategies have recently gained popularity: (1) V-L Adapters [16, 45], and (2) Prompt learning [48, 19]. The first category of adapters methods, like CLIP-Adapter [16] and Tip-Adapter [45], introduce lightweight networks to learn refined features for classification tasks. On the other hand, CoOp [48] proposes a continuous prompt optimization strategy for image recognition, inspired by prefix-tuning for language models [17]. However, these methods, including Unsupervised Prompt Learning (UPL) [19], mainly target classification tasks and require labels or pseudo labels, making them ill-suited for data pre-selection tasks.

Our work differs from existing literature as we approach downstream tasks with zero prior knowledge. We demonstrate the adaptation of the V-L model to extract superior features in an unsupervised manner.

3 Methodology

An overview of our approach is shown in Figure 2. First, we formally define the data pre-selection task. Next, we provide brief reviews on BLIP-2 and present our unsupervised prompt learning

framework, which assists BLIP-2 in extracting improved multimodal features for data pre-selection. Finally, we describe how to use the adapted BLIP-2 to select instances for downstream tasks.

3.1 Data Pre-selection

Figure 2a, shows the data pre-selection workflow. Suppose that we have an enormous dataset D consisting of d instances and an annotation budget of l . Our task is to select l ( l ≪ d ) instances for labeling, so that undefined downstream tasks trained on such a partially labeled dataset produce the best performance. Formally, let D = ( x i , y i ) d i =1 denote d pairs of the image x i and its class label y i ( y i = ∅ if the label is unknown). Let D L denote a size l subset of D with known class labels. Our goal is to select D L ⊂ D to acquire class labels, in order to maximize the performance of an undefined model trained on labeled data D L with or without unlabeled data D U = D \ D L . Our data pre-selection task is challenging, as we do not have any labels to begin with. This means we do NOT know what information will make the undefined downstream task perform the best. Our idea is to select l instances that are not only representative but also diverse enough to cover the entire dataset, preventing premature loss of valuable information prior to label acquisition.

The process depicted in Figure 2a illustrates the detailed data pre-selection process by extracting highquality and semantically significant features. These features facilitate the clustering of all unlabeled images into well-structured groups. Subsequently, we select the most representative instance from each cluster for labeling. This strategy ensures both representativeness and diversity, accomplished by selecting one instance from each cluster. In the upcoming section, we will describe how we adapt BLIP-2 to this objective.

3.2 Unsupervised Prompt Learning

Figure 2b depicts the Unsupervised Prompt Learning approach to extract desirable features. It consists of three cooperatively learned elements: the learnable context of BLIP-2 and two Multilayer Perceptron (MLP) heads for both instance-level and clusterlevel transformation. These components are jointly optimized using unsupervised clustering objectives [25], specifically instance-level and clusterlevel contrastive learning. After training, the cluster assignments of all instances can be easily derived from the predicted soft labels produced by the cluster-level MLP head. When combined with the multimodal features obtained from the adapted BLIP-2, we can effectively identify the most representative and diverse instances from the entire dataset D .

Bootstrapping Language-Image Pre-training known as BLIP-2, presents an efficient pre-training strategy that leverages existing pre-trained image encoders and large language models, all while remaining frozen. By training a lightweight

querying transformer (Q-Former), it specifically bridges the modality gap between images and text. In the representation learning phase, BLIP-2 connects Q-Former to a frozen image encoder and conducts image-text pair pre-training. With three image-text pre-training objectives (Image-Text Contrastive Learning, Image-grounded Text Generation, and Image-Text Matching), they establish a

strong connection between image and input text, i.e. forcing the model to extract visual information from the image that is most relevant to the text and automatically generating output multimodal features. This characteristic of BLIP-2 provides us with a new opportunity. By altering different prompts (input text), we can extract distinct multimodal features and find the best to enhance the process of data pre-selection.

Learnable Prompt However, designing an appropriate prompt can be demanding and timeconsuming, especially for data pre-selection, when we have zero knowledge of downstream tasks. Following the previous prompt learning approach, we aim to model each context token using a continuous vector that can be learned from data from end to end. Specifically, as depicted in Figure 2b, we define N learnable prompt representation context vectors, denoted as V = { v 1 , v 2 , ... v n } , each having the same dimension as the word embedding, to serve as the text input for BLIP-2. In contrast to many previous methodologies that leverage cross-entropy loss as their learning objective with label data, we have chosen to adopt unsupervised clustering as our learning objective with respect to undefined downstream tasks. Since the text encoder is differentiable, which means gradients can be back-propagated to update the context vectors. It's important to note that the base model of BLIP-2 remains unchanged during the training process.

Instance-level Contrastive Loss Contrastive learning at the instance level is designed to maximize the similarities between positive instance pairs, while simultaneously minimizing those of negative pairs. Given the lack of labels for all the data, we aim to maximize the agreement between differently augmented views of the same instance and treat all other augmented instances as negative examples. Specifically, consider a randomly sampled mini-batch of image of N images. Each image, denoted by x i , undergoes different augmentation twice, thus creating two views of the same example: x a i and x b i . Subsequently, these two images are encoded, along with a learnable context V , using BLIP-2 f ( · ) , to generate multimodal representations: h a i and h b i . Following this, the representations are further transformed via a nonlinear transformation MLP g I ( · ) , known as an instance-level head, which yields z a i and z b i . Let sim ( ., . ) denote the cosine similarity. Then the instance-level contrastive loss of a positive pair of examples ( i, j ) ∈ [1 , N ] is defined as

where τ I is the temperature scalar. The final loss L I = 1 2 N ∑ N i ( ℓ a i + ℓ b i ) , which is computed across all positive pairs in a mini-batch.

Cluster-level Contrastive Loss Similar to instance-level constrative learning, cluster-level constrative learning aims to maximize the similarities between positive cluster pairs, while minimizing the negative pairs. To construct the cluster representation, we use another cluster-level head g C ( · ) to map the multimodal representations to a M -dimensional space, where M equals the number of clusters. Thus m -th element of the feature can be interpreted as the instance's probability of belonging to the m -th cluster. Formally, suppose with mini-batch of N instances and M predefined clusters, the output of the cluster head is C a ∈ R N × M under the first augmentation, thus C a n,m can be interpreted as the probability of instance n under augmentation a being assigned to the cluster m . The i -th column of C a can be treated as the representation of the i -th cluster under augmentation a . Thus, we can from a positive pair of cluster by selecting i -th column of C a and C b noted as ˆ c a i and ˆ c b i while leaving other 2 M -2 pair to be negative:

where τ C is the cluster-level temperature parameter. To avoid a trivial solution that most instances are assigned to the same cluster, a regularization that encourages the entropy of cluster assignment probability is added: H ( C ) = -∑ M i [ P (ˆ c a i ) log P (ˆ c a i ) + P (ˆ c b i ) log P (ˆ c b i )] , where P (ˆ c k i ) = 1 N ∑ N t =1 ˆ C k ti , k ∈ { a, b } . The final cluster-level loss is L C = 1 2 M ∑ M i =1 ( ˆ ℓ a i + ˆ ℓ b i ) -H ( C ) .

Objective Function The optimization is a one-stage and end-to-end process, shown in Algorithm 1. The prompt and two heads are simultaneously optimized and the overall objective function is the combination of constrastive loss at the instance and cluster level: L = L I + L C .

Labeling Finally, after training, each instance is assigned a cluster number by the adapted BLIP-2 and cluster-level MLP head. First, we directly use the probability predicted by the cluster-level head g C ( · ) for sampling. In this case, for each cluster, the instance with the highest confidence score is selected. On the other hand we can also select medoid of each cluster to be labeled for the downstream task. We can then select the medoid of each cluster to be labeled for the downstream task. Formally given the cluster X k with N k members { x 1 , x 2 ... x N k } , the medoid m k is the instance that has the minimal average dissimilarity to all instances calculated on the multimodel feature h or z , here we use h from f ( · ) as an example:

4 Experiments

We assess the effectiveness of our UP-DP method in a downstream task, specifically using selected labeled instances for image classification. In particular, we perform the UP-DP on BLIP-2, strategically selecting the most representative and diverse instances for labeling, followed by the linear-probe on CLIP to compare the performance against random baselines and two variations of current state-of-the-art method: USL [40]. Lastly, we show several intriguing properties of UP-DP such as generalizability, i.e. a learned prompt from a single dataset can be directly applied to enhance the feature quality of other datasets.

Dataset We select seven image classification datasets that are widely used in evaluating the V-L model adaptation approach. These datasets constitute a comprehensive benchmark, covering a diverse set of vision tasks, including the classification of generic objects (Caltech101 [14]), actions (UCF101 [36]), fine-grained categories (OxfordPets [29], FGVCAircraft [26], and Flowers102 [27]), as well as some specialized tasks such as recognizing texture (DTD [8]) and satellite imagery (EuroSAT [18]).

Training Details For the base model, we use the best available vision backbone in BLIP-2, which is ViT-G. Previous work [48] on prompt learning has shown that a shorter context length can lead to better and more robust performance. Therefore, we initialize the context vectors with a fixed length of 4. The two hyperparameters, τ I and τ C , are set to 0.5 and 1.0, respectively. Training is performed with the Adam optimizer and a learning rate of 0.0003. We optimize our model with a batch size of 256 for a total of 150 epochs on RTX 3090. We set an annotation budget of 200 for all datasets except EuroSAT, which, due to its significantly larger size (13,500 examples) and fewer classes (10), is allocated a label budget of 40 and trained for 50 epochs. The rationale for setting a uniform annotation budget of 200 images for most datasets is our lack of knowledge about the downstream task (e.g., prediction categories) in the context of data pre-selection. Thus, the annotation budget becomes our sole controllable factor. By taking into account variations in class numbers and dataset sizes, this uniform budget encompasses a a wide range of downstream task difficulties, averaging between 2 to 5 images per class.

Baseline Methods As described in the related work section, USL is the most relevant approach to ours. Similarly to our method, USL also relies on semantically meaningful feature representations for each instance. We apply USL to both the image features extracted from BLIP-2 and the multimodal features extracted from BLIP-2 with the learned prompts, resulting in two settings: USL-I and USL-M. Alongside USL, we include random selection as a baseline method for the purpose of a sanity check.

Zero-shot Recognition with BLIP-2 In the representation learning stage, the Q-Former performs pre-training using image-text pairs with image-text contrastive learning. Similar to CLIP, after pre-training, this head can produce a similarity score between an input image and text, which can be naturally used for zero-shot recognition. We use the hand-crafted prompt provided by CoOp for evaluating zero-shot performance. As shown in the bottom row of Table 1, we observe that the

performance of BLIP-2 is much lower than that of CLIP reported in CoOp. There are two potential reasons for this discrepancy: (1) the hand-crafted prompt for CLIP cannot be generalized for BLIP-2, and (2) the features from BLIP-2 cannot be directly generalized to these datasets, or perhaps a combination of both reasons. This highlights the necessity of adapting BLIP-2 for data pre-selection tasks.

Linear Probe Results To replicate the pre-selection process, which involves separating downstream tasks from data pre-selection, we first employ UP-DP to select the most representative and diverse data points for labeling. We then apply the linear probe method using various versions of the CLIP vision encoder, ranging from ResNet-50 to ViT-G, with the labeled data. Table 1 presents the main outcomes of UP-DP in seven datasets. Overall, our UP-DP consistently outperforms other baselines with an average accuracy of 6 . 8% to 7 . 2% .

Upon further analysis of the table across different datasets, we note that UP-DP significantly outperforms the baseline on EuroSAT, yet yields mixed results on Caltech101. This variation is not entirely unforeseen considering BLIP-2's limited efficiency with complex tasks. For instance, in a zero-shot setting, it nearly resorts to random guessing (0.111 accuracy on 10 classes) on the satellite image dataset EuroSAT, whereas it exhibits a relatively superior performance (0.397 accuracy on 100 classes) on the standard object recognition dataset, Caltech101. When comparing the average performance of the second and third rows, USL-M consistently outperforms USL-I. This consistent performance gap between USL-M and USL-I further validates the effectiveness of a prompt learned by UP-DP, which assists BLIP-2 in extracting superior multimodal features.

Both USL-M and our method utilize the same multimodal features extracted from BLIP-2 with learned prompts for data selection. Essentially, USL can be seen as initially executing k-means clustering and then picking the most representative point with diversity regularization. Unlike USL-M, we explicitly train a cluster MLP head that can directly assign each instance within the cluster, where the medoid is selected in each cluster for labeling. The performance improvement between USL-M and our method showcases the effectiveness of the simple sampling strategy employed in UP-DP.

Cluster # 149

Cluster #101

Cluster#5

Cluster # 52

Domain Generalization of Learned Prompt After establishing the effectiveness of UP-DP within a single dataset, we further demonstrate that the learned prompt has the capability to be applied across multiple datasets. This presents a considerably more challenging problem, since the fundamental features can undergo significant changes when dealing with different datasets, such as transitioning from object recognition to texture classification.

Figure 2c illustrates the way of using the prompt learned from one dataset, such as OxfordPets, to extract multimodal features from another dataset, such as Flowers102. We want to demonstrate that these features exhibit superior quality compared to features extracted directly from BLIP-2, including image features and multimodal features with empty or random prompts. We evaluate the feature quality with the testing accuracy achieved by the non-parametric K-Nearest Neighbors (KNN) classifier.

As presented in Table 2, we use the accuracy of the image feature (highlighted in yellow) as the baseline and identify results that differ from it by at least 2% (green and red). It is evident that all the results obtained from multimodal feature extraction using an undefined prompt are inferior to the image feature. Surprisingly, the multimodal feature results with the prompts learned from other datasets exhibit a significant margin of improvement over the image feature (except for the OxfordPets features extracted using the EuroSAT prompt). This demonstrates the generalizability

of the learned prompt. The exception for OxfordPets features can be attributed to the significant difference between the satellite image task and the fine-grained dog and cat recognition task.

Qualitative Study To provide an intuitive understanding of the functioning of UP-DP, we visualize the learned feature representation and cluster assignment of all instances using 2D projection with UMAP. As depicted in Figure 3a, at high level, the features extracted from BLIP-2 with learned prompts demonstrate discriminative abilities to distinguish between different types of dogs and cats, even without any label information. Taking a closer look at the Shiba Inu group in part (b), we observe that the cluster assignments generated by our cluster-head categorize Shiba Inus into distinct clusters based on background and posture. Notably, in (c), clusters #149 , #101 , #5 , and #52 reveal Shiba Inus against backgrounds of mountains, grasslands, concrete roads, and homes, respectively. Furthermore, Shiba Inus in the clusters #149 and #5 are typically seen standing, while those in the cluster #101 are observed lying on the ground. Our qualitative study demonstrates that UP-DP effectively selects diverse representative points from unlabeled data, resulting in improved performance for downstream tasks.

Interpreting the Learned Prompts It is difficulty to interpreting the learned prompts due to their continuous vector nature. To overcome this challenge, we adopted the CoOp approach and searched the vocabulary for words that closely correspond to the learned vectors, using Euclidean distance as a metric. The search results are presented in Table 3. We observed that a few words, such as "wood" in EuroSAT (related to the forest class), certain numbers (potentially representing aircraft codes) and the word "plane" in FGVCAircraft, even "butterfly" in the Flowers102 dataset, demonstrate some relevance to their respective tasks. However, the majority of the words lack coherent meaning. This leads us to speculate that the learned vectors may encode meanings beyond the scope of the existing vocabulary. Notably, we also discovered shared words, including "learn", "saint", "add", and "attend" across different datasets, which could account for the significant generalizability of the learned prompts across various tasks.

5 Conclusion

In conclusion, we introduce UP-DP, a novel method that leverages Unsupervised Prompt learning to enhance Data Pre-selection with Vision-Language models. Our approach is specifically designed to achieve data efficiency without requiring prior knowledge of specific downstream tasks. By utilizing both vision and text features of V-L models, UP-DP offers a new pathway towards improving data efficiency. It outperforms existing methods by up to 20% on seven benchmark datasets and demonstrates remarkable generalizability on different datasets. We believe that our research opens promising avenues for future studies in data-efficient machine learning.

References

Sharat Agarwal, Himanshu Arora, Saket Anand, and Chetan Arora. Contextual diversity for active learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI 16 , pages 137-153. Springer, 2020.

Sima Behpour, Anqi Liu, and Brian Ziebart. Active learning for probabilistic structured prediction of cuts and matchings. In International Conference on Machine Learning , pages 563-572. PMLR, 2019.

David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv:1911.09785 , 2019.

David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in neural information processing systems , 32, 2019.

Hao Chen, Ran Tao, Yue Fan, Yidong Wang, Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj, and Marios Savvides. Softmatch: Addressing the quantity-quality trade-off in semisupervised learning. arXiv preprint arXiv:2301.10921 , 2023.

Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems , 33:22243-22255, 2020.

Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 , 2020.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3606-3613, 2014.

Thang Doan, Xin Li, Sima Behpour, Wenbin He, Liang Gou, and Liu Ren. Hyp-ow: Exploiting hierarchical structure learning with hyperbolic distance enhances open world object detection. arXiv preprint arXiv:2306.14291 , 2023.

Melanie Ducoffe and Frederic Precioso. Adversarial active learning for deep networks: a margin based approach. arXiv preprint arXiv:1802.09841 , 2018.

M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision , 88(2):303-338, June 2010.

Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 961-970, 2015.

Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improving visualsemantic embeddings with hard negatives. arXiv preprint arXiv:1707.05612 , 2017.

Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop , pages 178-178. IEEE, 2004.

Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International conference on machine learning , pages 1183-1192. PMLR, 2017.

Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:2110.04544 , 2021.

Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723 , 2020.

Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 12(7):2217-2226, 2019.

Tony Huang, Jack Chu, and Fangyun Wei. Unsupervised prompt learning for vision-language models. arXiv preprint arXiv:2204.03649 , 2022.

Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In International Conference on Machine Learning , pages 5583-5594. PMLR, 2021.

Junnan Li, Caiming Xiong, and Steven CH Hoi. Comatch: Semi-supervised learning with contrastive graph regularization. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9475-9484, 2021.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping languageimage pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning , pages 12888-12900. PMLR, 2022.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping languageimage pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 , 2023.

Xin Li, Hassan Bagher-Ebadian, Stephen Gardner, Joshua Kim, Mohamed Elshaikh, Benjamin Movsas, Dongxiao Zhu, and Indrin J. Chetty. An uncertainty-aware deep learning architecture with outlier mitigation for prostate gland segmentation in radiotherapy treatment planning. Medical Physics , 50(1):311-322, 2023. doi: https://doi.org/10.1002/mp.15982. URL https: //aapm.onlinelibrary.wiley.com/doi/abs/10.1002/mp.15982 .

Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. Contrastive clustering. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 8547-8555, 2021.

Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Finegrained visual classification of aircraft. arXiv preprint arXiv:1306.5151 , 2013.

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing , pages 722-729. IEEE, 2008.

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 , 2023.

Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition , pages 3498-3505. IEEE, 2012.

Amin Parvaneh, Ehsan Abbasnejad, Damien Teney, Gholamreza Reza Haffari, Anton Van Den Hengel, and Javen Qinfeng Shi. Active learning by feature mixing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12237-12246, 2022.

Omead Pooladzandi, David Davini, and Baharan Mirzasoleiman. Adaptive second order coresets for data-efficient machine learning, 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. Advances in neural information processing systems , 29, 2016.

Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489 , 2017.

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semisupervised learning with consistency and confidence. Advances in neural information processing systems , 33:596-608, 2020.

Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 , 2012.

Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X , pages 268-285. Springer, 2020.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442 , 2022.

Xudong Wang, Long Lian, and Stella X Yu. Unsupervised selective labeling for more effective semi-supervised learning. In European Conference on Computer Vision , pages 427-445. Springer, 2022.

Xudong Wang, Zhirong Wu, Long Lian, and Stella X Yu. Debiased learning from naturally imbalanced pseudo-labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14647-14657, 2022.

Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. Advances in neural information processing systems , 33: 6256-6268, 2020.

Yi Yang, Zhigang Ma, Feiping Nie, Xiaojun Chang, and Alexander G Hauptmann. Multi-class active learning by uncertainty sampling with diversity maximization. International Journal of Computer Vision , 113:113-127, 2015.

Donggeun Yoo and In So Kweon. Learning loss for active learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 93-102, 2019.

Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.03930 , 2021.

Zhiqi Zhang, Pan Ji, Nitin Bansal, Changjiang Cai, Qingan Yan, Xiangyu Xu, and Yi Xu. Clip-flow: Contrastive learning by { s } emi-supervised iterative pseudo { l } abeling for optical flow estimation. arXiv preprint arXiv:2210.14383 , 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision , 130(9):2337-2348, 2022.

Broader Impact

Our research, UP-DP, a new method for data pre-selection, holds substantial positive societal impacts. By incorporating unsupervised prompt learning into vision-language models for data pre-selection, our approach improves the efficiency of data labeling and reduces the computational resources required for machine learning tasks. By selecting a subset of instances from a large unlabeled dataset for labeling, it reduces the time and effort required for manual annotation, thereby alleviating the burden of laborious labeling tasks. This frees up human resources for more complex and creative tasks, rather than repetitive labeling work. Moreover our method democratizes access to machine learning technology. By optimizing performance with a limited annotation budget, UP-DP enables individuals or organizations with limited computational resources to train effective models. In conclusion, our work not only advances the field of data pre-selection but also has the potential to positively impact society by improving labeling efficiency, democratizing access to machine learning, and encouraging the efficient use of computational resources.

Limitations

Our work UP-DP has demonstrated promising results in enhancing the process of data pre-selection. However, it is necessary to discuss its limitations to provide a comprehensive perspective. One limitation lies in the nature of the learned prompts. These prompts are continuous vectors, the interpretability of which is not straightforward. Unlike hand-crafted prompts that can be directly understood by humans, the meaning of continuous prompts remains opaque. This lack of transparency hinders our understanding of why and how these prompts improve the feature extraction capability of vision-language models like BLIP-2 for dataset pre-selection. Furthermore, it is even more challenging to explain the generalizability of learned prompts across different datasets, as they cover a diverse set of vision tasks. Another concern is the potential for unfairness and bias in model outcomes. UP-DP operates under a specific constraint: it selects a limited number of instances for labeling from an extensive unlabeled dataset. This strategy optimizes computational efficiency and resource allocation; however, it also introduces the risk of selection bias. If the annotation budget is extremely limited, the pre-selected data can be unrepresentative or skewed towards certain characteristics, which can inadvertently lead to biased models.

Appendix

Section A demonstrates the data preselection result on semantic segmentation task.

Section B presents the ablation study on the context length.

Section C shows the performance under large annotation budget settings.

Section D visualizes the selected instances from different methods.

A Semantic Segmentation

Beyond the realm of classification tasks, we have also ventured into assessing the performance of semantic segmentation models when trained with data pre-selected via our proposed method. To elucidate, we utilized the Pascal VOC [11] dataset specifically for the semantic segmentation task, employing various adaptations of the DINOv2 model [28] to serve as our segmentation framework. The dataset comprised a total of 1454 training instances, while we constrained our annotation budget to a mere 100. As depicted in Table 4, our methodology consistently outperforms other baseline approaches in segmentation tasks, showcasing its robustness and effectiveness."

B Context length

How many context tokens should be used? And is it better to have more context tokens? The results presented in Table 5 indicate that employing a shorter context length yields better overall performance. To investigate this hyperparameter, following the setting in CoOp, we conducted experiments on the 7 benchmark datasets using a lengthy context of 16. As illustrated in Table 5, the shorter context length consistently outperforms the longer length across all datasets and settings.

Given that our data pre-selection aims to identify instances that can be generalized to unknown downstream tasks, it becomes crucial to mitigate the overfitting caused by the longer trainable vector on the BLIP-2. This finding aligns with the analysis in CoOp, where it is argued that although having more context can potentially improve the performance of the current model, it comes at the expense of generalization.

We also conducted an experiment without updating prompts, resulting in a context length of zero. The learning probe results are presented in Table 6. These findings clearly demonstrate the necessity of training the prompts. The motivation behind training the prompts/contexts is to enable the model to extract features more suited to specific datasets. If we only train g I and g C without adjusting the prompts/contexts, the image features would remain static, potentially slowing convergence and lowering performance.

C Annotation Budget

In this section, we examine the performance of data pre-selection under various annotation budgets. We conduct a linear probe experiment on CLIP, repeating it with 50% more of the available budget. Specifically, we use 60 labeled instances for EuroSAT and 300 labeled instances for all other datasets. The results are presented in Table 7. The methods (Ours and USL-M) which utilize the multimodal features extracted with learned prompt, continue to outperform other methods, showcasing the effectiveness of our unsupervised prompt learning. It is worth noting that the gain in performance is not as substantial as in scenarios with much fewer budgets before. This can be attributed to the fact that, when there is a sufficient label budget, such as 10% of the entire training dataset (Table 8), random selection can serve as a strong baseline. In contrast, our method becomes more valuable in low-budget settings.

D Visualization on Selected Instances

Figure 4(c) and (d) present the visualizations of the least-40 and top-40 instances, respectively, that were selected using our method on the EuroSAT dataset. We compare these selections with the top 40 sampled images chosen through random selection and USL. In order to facilitate understanding, we have organized the images into different rows based on their ground truth labels. Note that it is important to have a balanced selection that covers all semantic classes. However, the randomly selected instances often exhibit significant imbalances. This means that we may end up with, for example, 9 instances from Class 7 while completely missing any images from Class 5. Such imbalances are quantified using the KL divergence, which measures the distance between the current sample distribution and a uniform distribution. In this case, the KL divergence is 0.241. In contrast, our top selected instances not only provide representation from each class but also offer diversity among the selected instances. The KL divergence for our top selections is only 0.047, indicating that they are both representative and diverse. On the other hand, the 40 instances that are least likely to be selected primarily consist of outliers. These outliers have the potential to mislead downstream classifiers and introduce noise into the learning process. For instance, the first image in class 0 (AnnualCrop) may appear strikingly similar to images in Class 1 (Forest) and Class 9 (SeaLake), which can be confusing for the classification task.

Class

0

(a) Random

(b) USL

KLDiv: 0.241

KLDiv: 0.096!

Class 1

Class 2

Class 3

Class 4

Class 5

Class 6

Class 7

Class 8

Class 9

Class 0

(c) UP-DP

Class 1

Least Utility

(d) UP-DP

KLDiv : 0.200!

Class 2

KLDiv : 0.047!

Class 3

Class 4

Class 5

Class 6

Class 7

Class 8

Class 9

Peer reviews:
comments: SUMMARY:
This paper investigates the task of data pre-selection by learning a better representation from the joint feature space of both vision and text in an unsupervised manner. The paper focuses on training text prompts to extract joint features with enhanced representation, specifically with the BLIP-2 parameters kept fixed. The aim is to achieve a diverse cluster structure that encompasses the entire dataset.

STRENGTHS:
**[New task]** This paper tackles data pre-selection for labelling without accessing the information of downstream tasks, which is quite new to the community.

**[Well-illustrated figures]** The figures shown in this paper are clear enough for better understanding.

**[Good presentation]** The paper is well-written and easy to follow.

WEAKNESSES:
**[Unconvincing illustration]** In Figure 1, BLIP-2 is pre-trained with prompt and image together. This explains why using only image features yields poor performance. Consequently, the evidence presented does not convincingly demonstrate the superiority of multimodal features.

**[Need in-depth analysis]** (i) It is unclear why the self-trained model can be used for sample selection. (ii) The motivation of medoid selection is not given. It would be nice to see the rationale. 

**[Missed ablation studies]** The ablation study for two hyperparameters are not given.

**[Disorganized reference format]** Please reformat the references as per some published papers.

QUESTIONS:
Please refer to the weaknesses part.

LIMITATIONS:
Please refer to the weaknesses part.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses data pre-selection (akin to active learning) problem using the highly successful vision-language models (VLMs) of late. In relation to existing approaches, the proposed approach has a few advantages, e.g., no need to have a small initial set of labeled data, no need to have multiple rounds of selection, labeling and retraining, once selected the data can be used for multiple future, unknown downstream tasks etc. The authors start with a BLIP-2 model and an unlabeled set of data $D$. The BLIP-2 model is extended to have learnable context/prompt and a few MLPs, namely instance-level and cluster-level heads. The instance-level head is employed to produce a contrastive training between two views of each unlabeled instance. Cluster-level head, on the other hand, first assigns cluster memberships to the instances and then helps to train the model with a cluster-level contrastive loss. After training for a few epochs using these two losses, cluster-level MLP along with the learned contexts/prompts are used to get the cluster assignment of unlabeled data (which can come from different downstream dataset). Finally, the medoids from each cluster form the representative selections for active learning. Experiments performed on linear probing and domain generalization show the efficacy of the proposed approach over the state-of-the-arts on benchmark datasets.\n\nSTRENGTHS:\n1. The use of VLMs for unsupervised active learning is appreciable. VLMs. Now-a-days, are known for good zero-shot transfer. The already well-learned representations can and did help the active learning cause.\n2. The use of learnable contexts/prompts has been shown to be useful for few-shot transfer. The use of these for active learning is interesting.\n3. The use of cluster-level contrastive loss cleverly avoids the use of any initial labeled set that is required in traditional cluster level losses in getting the initial clustering (akin to group contrastive loss in semisupervised literature e.g., [a]).\n4. Experimental analysis and ablations show the efficacy of the proposed approach compared to sota approaches and the importance of different components of the approach as well.\n\n[a] Singh et al., Semi-supervised action recognition with temporal contrastive learning, CVPR 2021.\n\nWEAKNESSES:\n1. One important ablation that could be useful is running the approach without the learnable prompts/contexts. What I mean is updating only $g_I$ and $g_C$ but not employing $V$ in Algorithm 1. This will help gauge the importance of the contexts/prompts vis-à-vis the instance and cluster level MLPs. Does ‘Initi_Prompt’ row in Table 2 do this?\n2. Line 260+: This is more of a clarification query. When the datasets are described, I don’t see any mention of which dataset is used for pre selection. I am assuming these 7 datasets are downstream task datasets. The question is coming from Table 1. While in Table 2, it seems that the first column tells what is the dataset on which the prompts are learnt, in table 1, it is not clear. Is it that the learning is done on the same datasets on which the linear probing performances are shown for Table 1?\n3. Line274+: I am not getting what is meant by 'with the learned prompts' in the baseline using USL. Does it mean everything else here is same as BLIP-2, but in addition a few prompts are learned also?\n\nQUESTIONS:\nQuestions are already asked above (in the ‘Weaknesses’ section). Those are mostly queries for further clarification. Here in addition, let me list a few presentation related issues (typos mainly).\n - Line 32, 78, Figure 1 caption: Will these be ‘data efficient learning’ instead of ‘data efficiency learning’?\n - Line 231: ‘map’ -> ‘maps’\n - Line 244: ‘optimizing’ -> ‘optimization’\n - Line 246: ‘combine’ -> ‘combination’\n - In Figure 2a, which feature extractor is used? Is it anything different from BLIP-2?\n\nLIMITATIONS:\nThe limitations are described well in the paper. At the same time, the authors tried to address the limitation in the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an unsupervised approach for data preselection, which aims to select instances for labeling from an unlabeled dataset in a single pass. The authors leverage the text features in multimodal models, specifically BLIP2, to enhance the representation for data preselection. They argue that a well-designed joint feature space of vision and text can yield improved results. To achieve this, they train text prompts to extract joint features with enhanced representation, ensuring a diverse cluster structure that covers the entire dataset. The authors employ two loss functions, namely instance-level contrastive and cluster-level contrastive, to train the learnable text prompts. These loss functions encourage the adapted multimodal model's joint representation space to be more diverse and well-separated, suitable for clustering. Experimental results on seven different datasets, along with a comparison against three baselines, demonstrate the effectiveness of the proposed approach.\n\nSTRENGTHS:\nI believe the strength of the paper is as follows:\n\n- The integration of vision and language models for data pre-selection holds promise due to the added benefits of leveraging text modality. \n\n- This paper introduces a novel approach that promotes diverse and clustered representations, addressing limitations in the current state-of-the-art BLIP2 model through the lens of prompt learning. \n\n- The proposed method is lightweight and more efficient in terms of training costs. The paper makes a significant contribution, evident in its clear presentation and compelling results.\n\nWEAKNESSES:\nWhile the paper's results are strong, there are two notable aspects that could be addressed:\n\n1. Missing CLIP baseline: Comparing the proposed approach with a CLIP baseline would provide a fundamental point of reference. Since the authors employ contrastive loss functions and prompt learning, which can be applied to any vision and language model, including CLIP as a baseline would enhance the comparative analysis.\n\n2. Lack of integrability: A limitation of the prompt tuning method is the lack of interpretability. While prompt tuning improves model performance, it does not provide a clear explanation of why and how the model works in the combined language model embedding space. Addressing this limitation would enhance the understanding and justification of the proposed approach.\n\n3. Performance/training-time trade-off: The paper utilizes the best-performing model of BLIPV2, which has over 7 billion parameters. However, it does not explore the performance and training-time trade-off with different BLIPV2 models. Investigating the performance of the proposed approach on BLIPV2 models with fewer parameters would provide insights into its scalability and suitability for models of varying sizes.\n\nQUESTIONS:\nFollowing questions address the weaknesses and limitations mentioned earlier. Asking the authors these questions could help improve the paper and provide a more comprehensive understanding of the proposed approach:\n\n1. Could you compare your proposed method when built on top of CLIP as well? Including a comparison with CLIP as a baseline would provide valuable insights into the effectiveness of your approach and its advantages over a widely used vision and language model.\n\n2. Could you provide some visualizations, such as attention weight visualizations, to explain why the trained prompts lead to better data preselection and generalization? Visualizations would enhance the interpretability of your method and shed light on the mechanisms behind its improved performance.\n\n3. Have you replicated your proposed method on other variants of the BLIPv2 model? It would be insightful to see how your approach's performance changes when applied to BLIPv2 models with varying levels of expressiveness, particularly when the model size decreases. This analysis would help assess the scalability and adaptability of your method to different model configurations.\n\nLIMITATIONS:\nThere are not any limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of data pre-selection, which aims to select instances for labeling from an unlabeled dataset to enhance performance for downstream tasks with a limited annotation budget. The authors suggest that combining visual and textual features in a joint space can result in a better representation for data pre-selection. They introduce UP-DP, an unsupervised prompt learning approach that adapts vision-language models(specifically BLIP-2) for data pre-selection. The proposed approach outperforms the state-of-the-art on benchmark datasets and exhibits generalizability.\n\nSTRENGTHS:\n1. The paper introduces a novel approach for data pre-selection that incorporates unsupervised prompt learning in the vision-language model. This approach effectively exploits the multimodal features and enhances the discrimination among classes.\n2. The authors provide a clear motivation for the task of data pre-selection and highlight the unique challenges it poses compared to semi-supervised learning and active learning.\n3. The paper compares with the state-of-the-art on multiple benchmark datasets, demonstrating its effectiveness and superior performance. Also, it highlights the generalizability of the learned prompts across different datasets.\n\nWEAKNESSES:\n1. The authors claim that the purpose of the data pre-selection is to optimize performance for undefined diverse downstream tasks; however, they only conduct experiments on the image classification task. This limited scope is insufficient to demonstrate the effectiveness of UP-DP for various downstream tasks such as detection or segmentation. The paper would benefit from conducting more experiments on other tasks. And on top of this, it would be interesting to investigate the generalizability of the learned prompts across different tasks.\n2. In Table 1, the \"Zero-Shot BLIP-2\" setting is unreasonable. It lacks a justifiable rationale to use the prompt for the CLIP model to evaluate the zero-shot performance of the BLIP-2 model. If the authors intend to use this baseline, they should train the learnable prompt for BLIP-2 from scratch.\n3. In Table 1, the author does not include a baseline that utilizes only image features extracted from BLIP-2. By comparing with this baseline, the authors can demonstrate the efficiency of the proposed method.\n4. In Table 1, the authors compare \"Random\", \"USL-I/M\", but it is suggested to compare with more approaches in the field. Including additional comparisons would strengthen the paper's evaluation and provide a better context for understanding the performance of the proposed approach.\n5. The ablation studies of the proposed method are limited. It would be beneficial to conduct more comprehensive experiments to analyze the performance of UP-DP under different settings. For instance,  the authors should provide a detailed analysis of the impact of the instance-level and cluster-level contrastive loss.\n6. All the experiments are carried out using the BLIP-2 model during the data pre-selection stage. However, it remains unclear whether the proposed method is exclusively effective for this particular model. It is important to consider other visual-language models, such as CLIP, to establish the efficacy of the approach.\n7. The paper contains several grammar issues. For example, on page 5, line 191, it states \"presents an efficient pre-training,\" and on page 6, line 237, it states \"Thus we can from a positive pair.\" These sentences require revision for improved clarity and grammatical correctness.\n\nQUESTIONS:\n1. Is it fair to use the prompt for CLIP model to assess the zero-shot performance of the BLIP-2 model? It's supposed to train the prompt for BLIP-2 from scratch.\n2. Please demonstrate that UP-DP is valid and effective for other tasks like detection and segmentation.\n3. Table 1 requires revision and additional information. It should incorporate a baseline that utilizes only image features extracted from BLIP-2. Also, it is beneficial to compare with more approaches in the field.\n4. Please include more ablation studies. At least, demonstrate the impact of the instance-level and cluster-level contrastive loss.\n5. During the data pre-selection stage, all the features are extracted from BLIP-2 model. However, it remains unclear whether the method is exclusively effective for this specific model. It is recommended that the authors demonstrate the effectiveness of using other vision and language models as well.\n\nLIMITATIONS:\nThe authors need to analysis the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper presents a novel method to perform data preselection for the task of image classification. Data preselection refers to the task of finding the images for annotating labels and then used for training. The paper builds around the powerful visual-language model BLIP, and proposes learnable prompts as inputs to BLIP to help perform unsupervised clustering for data preselection. The paper then presents results on seven image classification benchmarks, showing the superb performance of the proposed method.

STRENGTHS:
+ The presented method is a novel application of visual language model to data preselection. 
+ I find the proposed learnable prompting in conjunction with unsupervised clustering novel, and as suggested by experiment effective. 
+ The presented method is effective in data preselection, as demonstrated by the comparison against baseline methods.

WEAKNESSES:
- The decision to annotate 200 images per benchmark (LINE 265 - 273) seems arbitrary. Why this number? It would be great if the number can be varied and then plot the model performance accordingly to understand the effect of annotated data set size on model performance.
- USL-M, which shares the same multimodal features from BLIP-2 as the proposed UP-DP method, isn't really outperforming the baseline USL-L on EuroSAT (Table 1). Such a result contradicts the claimed effectiveness. What is the explanation?
- No results on linear probe on BLIP-2 using Random Sampling. This is needed in order to showcase the effectiveness of the proposed method.

QUESTIONS:
See above.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-25