SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Objective Function The optimization is a one-stage and end-to-end process, shown in Algorithm 1. The prompt and two heads are simultaneously optimized and the overall objective function is the combination of constrastive loss at the instance and cluster level:  $\mathcal{L} = \mathcal{L}_I + \mathcal{L}_C$ .
Labeling Finally, after training, each instance is assigned a cluster number by the adapted BLIP-2 and cluster-level MLP head. First, we directly use the probability predicted by the cluster-level head  $g_{C}(\cdot)$  for sampling. In this case, for each cluster, the instance with the highest confidence score is selected. On the other hand we can also select medoid of each cluster to be labeled for the downstream task. We can then select the medoid of each cluster to be labeled for the downstream task. Formally given the cluster  $\mathbf{X}_k$  with  $N_{k}$  members  $\{\mathbf{x}_1,\mathbf{x}_2\dots \mathbf{x}_{N_k}\}$ , the medoid  $\mathbf{m}_k$  is the instance that has the minimal average dissimilarity to all instances calculated on the multimodel feature  $\mathbf{h}$  or  $\mathbf{z}$ , here we use  $\mathbf{h}$  from  $f(\cdot)$  as an example:
$$
\mathbf {m} _ {k} = \arg \min  _ {\mathbf {x} _ {i} \in \mathbf {X} _ {k}} \frac {1}{N _ {k}} \sum_ {j = 1} ^ {N _ {k}} (1 - \operatorname {s i m} (f (\mathbf {x} _ {i}, \mathbf {V}), (f (\mathbf {x} _ {j}, \mathbf {V}))). \tag {3}
$$
We assess the effectiveness of our UP-DP method in a downstream task, specifically using selected labeled instances for image classification. In particular, we perform the UP-DP on BLIP-2, strategically selecting the most representative and diverse instances for labeling, followed by the linear-probe on CLIP to compare the performance against random baselines and two variations of current state-of-the-art method: USL [40]. Lastly, we show several intriguing properties of UP-DP such as generalizability, i.e. a learned prompt from a single dataset can be directly applied to enhance the feature quality of other datasets.
Dataset We select seven image classification datasets that are widely used in evaluating the V-L model adaptation approach. These datasets constitute a comprehensive benchmark, covering a diverse set of vision tasks, including the classification of generic objects (Caltech101 [14]), actions (UCF101 [36]), fine-grained categories (OxfordPets [29], FGVCAircraft [26], and Flowers102 [27]), as well as some specialized tasks such as recognizing texture (DTD [8]) and satellite imagery (EuroSAT [18]).
Training Details For the base model, we use the best available vision backbone in BLIP-2, which is ViT-G. Previous work [48] on prompt learning has shown that a shorter context length can lead to better and more robust performance. Therefore, we initialize the context vectors with a fixed length of 4. The two hyperparameters,  $\tau_{I}$  and  $\tau_{C}$ , are set to 0.5 and 1.0, respectively. Training is performed with the Adam optimizer and a learning rate of 0.0003. We optimize our model with a batch size of 256 for a total of 150 epochs on RTX 3090. We set an annotation budget of 200 for all datasets except EuroSAT, which, due to its significantly larger size (13,500 examples) and fewer classes (10), is allocated a label budget of 40 and trained for 50 epochs. The rationale for setting a uniform annotation budget of 200 images for most datasets is our lack of knowledge about the downstream task (e.g., prediction categories) in the context of data pre-selection. Thus, the annotation budget becomes our sole controllable factor. By taking into account variations in class numbers and dataset sizes, this uniform budget encompasses a wide range of downstream task difficulties, averaging between 2 to 5 images per class.
**Baseline Methods** As described in the related work section, USL is the most relevant approach to ours. Similarly to our method, USL also relies on semantically meaningful feature representations for each instance. We apply USL to both the image features extracted from BLIP-2 and the multimodal features extracted from BLIP-2 with the learned prompts, resulting in two settings: USL-I and USL-M. Alongside USL, we include random selection as a baseline method for the purpose of a sanity check.
Zero-shot Recognition with BLIP-2 In the representation learning stage, the Q-Former performs pre-training using image-text pairs with image-text contrastive learning. Similar to CLIP, after pre-training, this head can produce a similarity score between an input image and text, which can be naturally used for zero-shot recognition. We use the hand-crafted prompt provided by CoOp for evaluating zero-shot performance. As shown in the bottom row of Table 1, we observe that the
performance of BLIP-2 is much lower than that of CLIP reported in CoOp. There are two potential reasons for this discrepancy: (1) the hand-crafted prompt for CLIP cannot be generalized for BLIP-2, and (2) the features from BLIP-2 cannot be directly generalized to these datasets, or perhaps a combination of both reasons. This highlights the necessity of adapting BLIP-2 for data pre-selection tasks.
Linear Probe Results To replicate the pre-selection process, which involves separating downstream tasks from data pre-selection, we first employ UP-DP to select the most representative and diverse data points for labeling. We then apply the linear probe method using various versions of the CLIP vision encoder, ranging from ResNet-50 to ViT-G, with the labeled data. Table 1 presents the main outcomes of UP-DP in seven datasets. Overall, our UP-DP consistently outperforms other baselines with an average accuracy of  $6.8\%$  to  $7.2\%$ .
Upon further analysis of the table across different datasets, we note that UP-DP significantly outperforms the baseline on EuroSAT, yet yields mixed results on Caltech101. This variation is not entirely unforeseen considering BLIP-2's limited efficiency with complex tasks. For instance, in a zero-shot setting, it nearly resorts to random guessing (0.111 accuracy on 10 classes) on the satellite image dataset EuroSAT, whereas it exhibits a relatively superior performance (0.397 accuracy on 100 classes) on the standard object recognition dataset, Caltech101. When comparing the average performance of the second and third rows, USL-M consistently outperforms USL-I. This consistent performance gap between USL-M and USL-I further validates the effectiveness of a prompt learned by UP-DP, which assists BLIP-2 in extracting superior multimodal features.
Both USL-M and our method utilize the same multimodal features extracted from BLIP-2 with learned prompts for data selection. Essentially, USL can be seen as initially executing k-means clustering and then picking the most representative point with diversity regularization. Unlike USL-M, we explicitly train a cluster MLP head that can directly assign each instance within the cluster, where the medoid is selected in each cluster for labeling. The performance improvement between USL-M and our method showcases the effectiveness of the simple sampling strategy employed in UP-DP.
Features	Caltech101	DTD	FGVCAircraft	Flowers102	OxfordPets	UCF101	EuroSAT	Average
Caltech101_Prompt	0.975	0.768	0.485	0.987	0.922	0.885	0.957	0.834
DTD_Prompt	0.979	0.809	0.482	0.990	0.889	0.878	0.964	0.836
FGVCAircraft_Prompt	0.974	0.768	0.518	0.986	0.852	0.862	0.963	0.829
Flowers102_Prompt	0.978	0.806	0.583	0.995	0.946	0.876	0.952	0.864
OxfordPets_Prompt	0.976	0.746	0.520	0.989	0.945	0.871	0.960	0.840
UCF101_Prompt	0.970	0.766	0.423	0.978	0.786	0.863	0.962	0.795
EuroSAT_Prompt	0.955	0.794	0.415	0.975	0.501	0.865	0.969	0.746
Empty_Prompt	0.756	0.501	0.322	0.719	0.492	0.744	0.933	0.606
Initi_Prompt	0.771	0.534	0.270	0.757	0.421	0.779	0.919	0.592
Image	0.974	0.763	0.349	0.984	0.681	0.875	0.961	0.760
Table 2: Domain Generalization Results of the Learned Prompt. The generalization capability is evaluated by the accuracy (\%) of KNN classification, utilizing either image features or multimodal features extracted from BLIP-2 with various prompts. The baseline is the one with image features (in yellow). We highlight results that differ from the baseline by at least  $\pm 0.02$ . Green indicates higher accuracy, while red indicates lower accuracy compared to the baseline. This illustrates the impressive generalizability of the learned prompt: even when the prompts are learned from different datasets, the multimodal features almost consistently outperform the image-only features.
Figure 3: Visualization of Instance Features and Cluster Assignments on the OxfordPets Dataset. Color coding represents the cluster assignments determined by our cluster MLP head. The multimodal feature is extracted from BLIP-2 with learned prompts. (a) At a high level, the features extracted with the learned prompts can distinguish between various types of pets even without the labels. (b) A closer examination of one cluster reveals (c) the capability of the features to capture more detailed information, such as background and pet postures, within the same breed of Shiba Inus.
Domain Generalization of Learned Prompt After establishing the effectiveness of UP-DP within a single dataset, we further demonstrate that the learned prompt has the capability to be applied across multiple datasets. This presents a considerably more challenging problem, since the fundamental features can undergo significant changes when dealing with different datasets, such as transitioning from object recognition to texture classification.
Figure 2c illustrates the way of using the prompt learned from one dataset, such as OxfordPets, to extract multimodal features from another dataset, such as Flowers102. We want to demonstrate that these features exhibit superior quality compared to features extracted directly from BLIP-2, including image features and multimodal features with empty or random prompts. We evaluate the feature quality with the testing accuracy achieved by the non-parametric K-Nearest Neighbors (KNN) classifier.
As presented in Table 2, we use the accuracy of the image feature (highlighted in yellow) as the baseline and identify results that differ from it by at least  $2\%$  (green and red). It is evident that all the results obtained from multimodal feature extraction using an undefined prompt are inferior to the image feature. Surprisingly, the multimodal feature results with the prompts learned from other datasets exhibit a significant margin of improvement over the image feature (except for the OxfordPets features extracted using the EuroSAT prompt). This demonstrates the generalizability
EuroSAT	OxfordPets	DTD	Caltech101	FGVCAircraft	UCF101	Flowers102
makes	shark	several	learn	diamond	healthy	learn
wood	brigham	learn	butterfly	offers	learn	butterfly
takes	saint	putting	saint	learn	atends	angel
healthy	laying	add	add	plane	performs	saint
ku	elegant	six	three	del	speaks	personality
single	chicken	three	2020	river	newly	picking
have	posing	q	2019	100	physical	adding
november	kate	vector	2000	40	provides	gold
holds	bee	che	speaks	have	choose	spider
holding	atends	some	with	get	serves	giant
Table 3: Top Nearest Words to Learned Prompts. The selected meaningful words are obtained from the nearest words of the four context vectors learned by UP-DP. Words related to the dataset are indicated in green, while words with an orange background represent high-frequency words occurring across various datasets.
of the learned prompt. The exception for OxfordPets features can be attributed to the significant difference between the satellite image task and the fine-grained dog and cat recognition task.
Qualitative Study To provide an intuitive understanding of the functioning of UP-DP, we visualize the learned feature representation and cluster assignment of all instances using 2D projection with UMAP. As depicted in Figure 3a, at high level, the features extracted from BLIP-2 with learned prompts demonstrate discriminative abilities to distinguish between different types of dogs and cats, even without any label information. Taking a closer look at the Shiba Inu group in part (b), we observe that the cluster assignments generated by our cluster-head categorize Shiba Inus into distinct clusters based on background and posture. Notably, in (c), clusters #149, #101, #5, and #52 reveal Shiba Inus against backgrounds of mountains, grasslands, concrete roads, and homes, respectively. Furthermore, Shiba Inus in the clusters #149 and #5 are typically seen standing, while those in the cluster #101 are observed lying on the ground. Our qualitative study demonstrates that UP-DP effectively selects diverse representative points from unlabeled data, resulting in improved performance for downstream tasks.
Interpreting the Learned Prompts It is difficult to interpreting the learned prompts due to their continuous vector nature. To overcome this challenge, we adopted the CoOp approach and searched the vocabulary for words that closely correspond to the learned vectors, using Euclidean distance as a metric. The search results are presented in Table 3. We observed that a few words, such as "wood" in EuroSAT (related to the forest class), certain numbers (potentially representing aircraft codes) and the word "plane" in FGVCAircraft, even "butterfly" in the Flowers102 dataset, demonstrate some relevance to their respective tasks. However, the majority of the words lack coherent meaning. This leads us to speculate that the learned vectors may encode meanings beyond the scope of the existing vocabulary. Notably, we also discovered shared words, including "learn", "saint", "add", and "attend" across different datasets, which could account for the significant generalizability of the learned prompts across various tasks.
In conclusion, we introduce UP-DP, a novel method that leverages Unsupervised Prompt learning to enhance Data Pre-selection with Vision-Language models. Our approach is specifically designed to achieve data efficiency without requiring prior knowledge of specific downstream tasks. By utilizing both vision and text features of V-L models, UP-DP offers a new pathway towards improving data efficiency. It outperforms existing methods by up to  $20\%$  on seven benchmark datasets and demonstrates remarkable generalizability on different datasets. We believe that our research opens promising avenues for future studies in data-efficient machine learning.
Our research, UP-DP, a new method for data pre-selection, holds substantial positive societal impacts. By incorporating unsupervised prompt learning into vision-language models for data pre-selection, our approach improves the efficiency of data labeling and reduces the computational resources required for machine learning tasks. By selecting a subset of instances from a large unlabeled dataset for labeling, it reduces the time and effort required for manual annotation, thereby alleviating the burden of laborious labeling tasks. This frees up human resources for more complex and creative tasks, rather than repetitive labeling work. Moreover our method democratizes access to machine learning technology. By optimizing performance with a limited annotation budget, UP-DP enables individuals or organizations with limited computational resources to train effective models. In conclusion, our work not only advances the field of data pre-selection but also has the potential to positively impact society by improving labeling efficiency, democratizing access to machine learning, and encouraging the efficient use of computational resources.
Our work UP-DP has demonstrated promising results in enhancing the process of data pre-selection. However, it is necessary to discuss its limitations to provide a comprehensive perspective. One limitation lies in the nature of the learned prompts. These prompts are continuous vectors, the interpretability of which is not straightforward. Unlike hand-crafted prompts that can be directly understood by humans, the meaning of continuous prompts remains opaque. This lack of transparency hinders our understanding of why and how these prompts improve the feature extraction capability of vision-language models like BLIP-2 for dataset pre-selection. Furthermore, it is even more challenging to explain the generalizability of learned prompts across different datasets, as they cover a diverse set of vision tasks. Another concern is the potential for unfairness and bias in model outcomes. UP-DP operates under a specific constraint: it selects a limited number of instances for labeling from an extensive unlabeled dataset. This strategy optimizes computational efficiency and resource allocation; however, it also introduces the risk of selection bias. If the annotation budget is extremely limited, the pre-selected data can be unrepresentative or skewed towards certain characteristics, which can inadvertently lead to biased models.
- Section A demonstrates the data preselection result on semantic segmentation task.
- Section B presents the ablation study on the context length.
- Section C shows the performance under large annotation budget settings.
- Section D visualizes the selected instances from different methods.
A Semantic Segmentation
Beyond the realm of classification tasks, we have also ventured into assessing the performance of semantic segmentation models when trained with data pre-selected via our proposed method. To elucidate, we utilized the Pascal VOC [11] dataset specifically for the semantic segmentation task, employing various adaptations of the DINOv2 model [28] to serve as our segmentation framework. The dataset comprised a total of 1454 training instances, while we constrained our annotation budget to a mere 100. As depicted in Table 4, our methodology consistently outperforms other baseline approaches in segmentation tasks, showcasing its robustness and effectiveness."
Model	Random	USL-I	USL-M	Ours
Dinov2_ViT	54.9	56.4	58.1	58.5
Dinov2_ViT	55.9	57.3	58.3	58.8
Dinov2_ViT	55.2	54.2	53.7	57.1
Dinov2_ViT	47.9	51.9	51.4	53.6
Average	53.5	55.0	55.4	57.0
Table 4: Experiment on Semantic Segmentation Task. The mIoU of results for Dinov2 trained with data pre-selected by random, USL-I, USL-M, and our method.
B Context length
How many context tokens should be used? And is it better to have more context tokens? The results presented in Table 5 indicate that employing a shorter context length yields better overall performance. To investigate this hyperparameter, following the setting in CoOp, we conducted experiments on the 7 benchmark datasets using a lengthy context of 16. As illustrated in Table 5, the shorter context length consistently outperforms the longer length across all datasets and settings.
Given that our data pre-selection aims to identify instances that can be generalized to unknown downstream tasks, it becomes crucial to mitigate the overfitting caused by the longer trainable vector on the BLIP-2. This finding aligns with the analysis in CoOp, where it is argued that although having more context can potentially improve the performance of the current model, it comes at the expense of generalization.
Context Length	EuroSAT	OxfordPets	DTD	Caltech101	FGVCAircraft	UCF101	Flowers102	Average
RN50	16	0.359 ± 0.037	0.207 ± 0.006	0.246 ± 0.022	0.312 ± 0.002	0.058 ± 0.009	0.181 ± 0.014	0.104 ± 0.015	0.210
4	0.456 ± 0.040	0.229 ± 0.002	0.283 ± 0.011	0.324 ± 0.001	0.066 ± 0.001	0.184 ± 0.010	0.137 ± 0.003	0.240
RN101	16	0.335 ± 0.035	0.188 ± 0.017	0.226 ± 0.025	0.286 ± 0.003	0.049 ± 0.015	0.132 ± 0.012	0.087 ± 0.015	0.186
4	0.350 ± 0.051	0.211 ± 0.004	0.250 ± 0.017	0.298 ± 0.001	0.056 ± 0.003	0.146 ± 0.013	0.133 ± 0.005	0.206
ViTB32	16	0.462 ± 0.006	0.400 ± 0.023	0.320 ± 0.014	0.336 ± 0.007	0.077 ± 0.010	0.212 ± 0.019	0.185 ± 0.013	0.285
4	0.525 ± 0.010	0.439 ± 0.011	0.372 ± 0.002	0.352 ± 0.002	0.089 ± 0.001	0.214 ± 0.011	0.188 ± 0.008	0.311
ViTH14	16	0.494 ± 0.007	0.478 ± 0.018	0.348 ± 0.011	0.318 ± 0.004	0.101 ± 0.010	0.249 ± 0.018	0.200 ± 0.011	0.313
4	0.577 ± 0.011	0.567 ± 0.005	0.392 ± 0.003	0.335 ± 0.001	0.116 ± 0.001	0.253 ± 0.017	0.206 ± 0.000	0.349
ViTG14	16	0.501 ± 0.006	0.507 ± 0.015	0.352 ± 0.009	0.324 ± 0.007	0.108 ± 0.009	0.251 ± 0.019	0.210 ± 0.010	0.322
4	0.560 ± 0.024	0.582 ± 0.008	0.389 ± 0.004	0.339 ± 0.002	0.130 ± 0.005	0.254 ± 0.014	0.224 ± 0.002	0.354
We also conducted an experiment without updating prompts, resulting in a context length of zero. The learning probe results are presented in Table 6. These findings clearly demonstrate the necessity of training the prompts. The motivation behind training the prompts/contexts is to enable the model to extract features more suited to specific datasets. If we only train  $g_{I}$  and  $g_{C}$  without adjusting the prompts/contexts, the image features would remain static, potentially slowing convergence and lowering performance.
Table 5: Impact of Context Length. Accuracy (%) of linear probe results for CLIP trained with data pre-selected by our method with different numbers of learnable context.
RN50	RN101	ViTB32	ViTH14	ViTG14	Average
Random	0.119 ± 0.021	0.086 ± 0.018	0.150 ± 0.035	0.153 ± 0.027	0.178 ± 0.028	0.137
BLIP-2 w/o Prompt	0.136 ± 0.005	0.113 ± 0.010	0.166 ± 0.008	0.176 ± 0.003	0.177 ± 0.002	0.153
BLIP-2	0.183 ± 0.011	0.171 ± 0.008	0.239 ± 0.007	0.259 ± 0.004	0.256 ± 0.006	0.221
Table 6: Result of Training Without Prompts. Accuracy (%) of linear probe results for CLIP, trained on the DTD dataset with data pre-selection conducted both with and without prompts. The annotation budget was set at 100.
C Annotation Budget
In this section, we examine the performance of data pre-selection under various annotation budgets. We conduct a linear probe experiment on CLIP, repeating it with  $50\%$  more of the available budget. Specifically, we use 60 labeled instances for EuroSAT and 300 labeled instances for all other datasets. The results are presented in Table 7. The methods (Ours and USL-M) which utilize the multimodal features extracted with learned prompt, continue to outperform other methods, showcasing the effectiveness of our unsupervised prompt learning. It is worth noting that the gain in performance is not as substantial as in scenarios with much fewer budgets before. This can be attributed to the fact that, when there is a sufficient label budget, such as  $10\%$  of the entire training dataset (Table 8), random selection can serve as a strong baseline. In contrast, our method becomes more valuable in low-budget settings.