SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xin Li Sima Behpour Thang Doan Wenbin He Liang Gou Liu Ren
Bosch Research North America, Bosch Center for Artificial Intelligence (BCAI)
{xin.li9, sima_behpour, thang.doan, wenbin.he2, liang.gou, liu. ren} @us.bosch.com
In this study, we investigate the task of data pre-selection, which aims to select instances for labeling from an unlabeled dataset through a single pass, thereby optimizing performance for undefined downstream tasks with a limited annotation budget. Previous approaches to data pre-selection relied solely on visual features extracted from foundation models, such as CLIP and BLIP-2, but largely ignored the powerfulness of text features. In this work, we argue that, with proper design, the joint feature space of both vision and text can yield a better representation for data pre-selection. To this end, we introduce UP-DP, a simple yet effective unsupervised prompt learning approach that adapts vision-language models, like BLIP-2, for data pre-selection. Specifically, with the BLIP-2 parameters frozen, we train text prompts to extract the joint features with improved representation, ensuring a diverse cluster structure that covers the entire dataset. We extensively compare our method with the state-of-the-art using seven benchmark datasets in different settings, achieving up to a performance gain of  $20\%$ . Interestingly, the prompts learned from one dataset demonstrate significant generalizability and can be applied directly to enhance the feature extraction of BLIP-2 from other datasets. To the best of our knowledge, UP-DP is the first work to incorporate unsupervised prompt learning in a vision-language model for data pre-selection.
Data-efficient machine learning, which aims to identify the best subsets of data samples (to either label or not) in order to achieve optimal model performance, has been a crucial research field in the era of data-hungry deep learning [12, 31, 24, 40, 9]. In this work, we focus on a new and practical task of data pre-selection for data-efficient visual object recognition (Fig.1-a). The goal of data pre-selection is to select instances for labeling from an unlabeled dataset through a single pass to maximize model performance for unknown downstream vision tasks (e.g., no knowledge about prediction categories, shown in Fig.1-b), given a limited annotation budget. This task is motivated by two practical needs: at the data acquisition stage, we want to collect minimal data to support potentially diverse downstream tasks (e.g., classification, action recognition, or even detection for videos or images in Fig.1-b) and training paradigms (e.g. active learning, semi-supervised learning, supervised learning in Fig.1-c); at the annotation stage, we aim to achieve good performance with fewer human annotations over the pre-selected data.
Traditionally, to tackle the challenge of data efficiency learning, there are two main approaches: Semi-supervised learning (SSL) [3, 35, 21] and Active Learning (AL) [34, 10, 2]. SSL aims to address the problem of label scarcity by leveraging a limited quantity of labeled data in conjunction with a more abundant pool of unlabeled data to enhance the model's performance. By contrast, AL approaches start with an initial set of labeled data and select the most informative data point to label
Figure 1: Our data pre-selection (a) is a novel task in data efficiency learning, and different from previous tasks, such as semi-supervised learning and active learning, which needs both seed labels and downstream task information (c). Our intuition is demonstrated by the UMAP visualization (d) of the features extracted from BLIP-2 on the OxfordPets training dataset, with each color representing the ground truth. It is evident that BLIP-2 with an appropriate prompt (purple box) can provide visually superior multimodal features compared to unimodal features (green box), which can serve as a good starting point for data pre-selection.
in an effort to maximize model performance with a limited label budget. However, existing AL methods need an initial labeled set to begin with and need to train a series of models.
Nonetheless, the task of data pre-selection poses some unique challenges that SSL and AL, as mentioned above, fail to adequately address. In data pre-selection, we do not have initial labeled data or know the specific downstream task. Take object detection as an example: we do not know what categories of objects need to be detected. However, both current SSL and AL methods need both initial labeled data and specific learning tasks. Therefore, this requires a method to select diverse and representative instances to cover the entire dataset with a minimum number of required labels. To achieve this, it is essential to acquire semantically meaningful, high-quality features.
Fortunately, recent advances in foundational Vision-Language (V-L) models, such as CLIP [32] and BLIP-2 [23], hold significant promise in addressing this problem. These models acquire comprehensive semantic knowledge by learning from a vast collection of image-text pairs. Consequently, they possess the ability to extract meaningful features from diverse input modalities. While there exists a numerous of studies that adapt V-L models for various tasks like few-shot classification [48], semi-supervised learning [46], and selective labeling [40], they often place emphasis on unimodal features, primarily vision, and inadvertently overlook the potential contributions of multimodal features encompassing both vision and language.
In this paper, we argue that by carefully designing the prompt, the multimodal features extracted by V-L models can offer a more powerful representation for data pre-selection. Figure 1 illustrates the unimodal and multimodal feature representations obtained from BLIP-2, with different colors representing various ground truth labels. In particular, when the class information of downstream tasks is unknown, the unimodal features become mixed, making it challenging to identify specific classes. In contrast, when coupled with an appropriate prompt, multimodal features exhibit a more scattered and distinct distribution, allowing for improved discrimination among classes. This enhanced discrimination is crucial in helping users identify the most representative and diverse instances during the data pre-selection process.
However, the process of designing an appropriate prompt can be demanding and time-consuming, often requiring trial and errors [48]. Moreover, current automatic prompt learning approaches [48, 47, 19] typically rely on a small set of labeled or pseudo labeled data, making them impractical for data pre-selection when the downstream tasks are undefined. To address this challenge, we propose a novel approach: pure unsupervised prompt learning, which aims to enhance the multimodal features extracted from V-L models for data pre-selection. Our contributions include:
- We introduce UP-DP, an innovative Unsupervised Prompt learning for enhanced Data Pre-selection in Vision-Language models. To the best of our knowledge, UP-DP is the
pioneering work that incorporates unsupervised prompt learning in a vision-language model specifically for data pre-selection.
- We demonstrate the robust generality of the learned prompt across different datasets, which significantly enhances feature extraction when applied in a plug-and-play manner.
- We extensively benchmark our method using various architectural settings, achieving superior performance compared to other competitors on 7 image classification datasets.
2.1 Data Efficiency Learning
Semi-supervised Learning (SSL) combines labeled and unlabeled data to enhance learning, and can be seen as unsupervised learning with added labeled data. SSL approaches are divided mainly by how they leverage large amounts of unlabeled data. Consistency Regularization methods [33, 42] maintain model output stability under realistic perturbations like data augmentation, with UDA [42] being an example that extends supervised data augmentation to boost SSL performance. Pseudolabeling methods [4, 41, 5] use high confidence model predictions to generate pseudo-labels for unlabeled data, training them jointly with labeled data. For example, SoftMatch [5] addresses the quantity-quality trade-off in pseudo-labeling using a truncated Gaussian function to weight samples by confidence. Transfer learning-based methods such as SimCLRv2 [6] involve supervised fine-tuning after unsupervised pre-training on unlabeled datasets, demonstrating the effectiveness of large models for SSL.
However, the success of these methods is based on the assumption of stratified sampling [4, 35], which requires equal sampling of each known class, a constraint that is often unattainable in practice. More importantly, our data pre-selection task even does NOT have any assumption for downstream tasks and also does NOT know any information about class categories.
Active Learning (AL) actively selects high-value instances for labeling during the iterative training process to enhance predictive performance, offering a realistic alternative to the stratified sampling setting of SSL. AL strategies generally fall into two categories: (1) methods [15, 44] that select informative instances using a scoring function such as uncertainty and (2) methods [34, 43] that choose diverse instances representing the dataset's domain. Some research [30, 1] addresses both aspects, such as ALFA-Mix [30], which evaluates label variability for perturbed instances and ensures diversity by clustering and selecting centroids.
However, current AL methods have limitations. They often require an initial labeled set, which is inefficient in low-label settings, and involve multiple rounds of labeling and training with a human annotator, making the process time-consuming. On the other hand, the requested labels are tightly associated with the training objective, necessitating distinct instances for each task. In contrast, our data pre-selection task aims for single-pass labeling applicable to any future, unknown downstream task, minimizing human effort and overall cost.
Unsupervised Selective Labeling (USL) [40] is the most relevant approach to ours. They proposed an SSL pipeline that avoids the unrealistic setting of stratified sampling. Specifically, they first apply unsupervised feature learning (e.g., MoCov2 [7]) to map data into a discriminative feature space. Then, they select instances for labeling for maximum representativeness and diversity in a single pass. Finally, with a strong initialization from the first step, they apply SSL to the labeled data and the remaining unlabeled data. Although their method achieves great performance and even beats stratified sampling in some settings, it is not suitable for data pre-selection. Using the same model and weights for both the selection step and the downstream task step is not a valid assumption.
2.2 Vision-Language Foundation Models
Vision-language models, pre-trained on large-scale image-text pairs, excel in visual representation learning. A taxonomy of V-L models [20] can be based on two aspects: (1) the expressiveness of both modalities in terms of parameters and computation, and (2) the extent of interaction within a Deep Neural Network (DNN). Early Visual Semantic Embedding (VSE) models, such as  $\mathrm{VSE}++$  [13] and SCAN [37], use separate feature extractors for image and text modalities, with the image extractor being more complex. They represent similarity using dot products or shallow attention layers.
Figure 2: An Overview of Our Approach. (a) workflow for data pre-selection, (b) training process for unsupervised prompt learning, and (c) generalization of learned prompts across various datasets.
In contrast, CLIP [32] uses comparably complex feature extractors (transformers [38]) for each modality but maintains a shallow interaction. Recent models [20, 22, 39] use deep transformers for inter-modal interactions, enhancing performance in complex vision-language tasks. However, pre-training becomes computationally expensive with large feature extractors and interaction models. BLIP-2 [23] introduces a querying transformer (interaction) that utilizes pre-trained image and language models and captures robust interaction between the image and text. The interaction of two modalities allows us to effectively manage the multimodal features extracted from the V-L model merely by modifying the input text (prompt). However, determining the optimal prompt for data pre-selection remains an open question. In this work, we designed a novel approach of generating text prompts to extract multimodal features without any additional information.
2.3 Adaptation of Vision-Language Foundation Models
Recent advances in multimodal pre-training models have shown great success in classic vision tasks [32], prompting interest in more efficient adaptation models.
To avoid the time-consuming process of fine-tuning an over-parameterized V-L model, two main strategies have recently gained popularity: (1) V-L Adapters [16, 45], and (2) Prompt learning [48, 19]. The first category of adapters methods, like CLIP-Adapter [16] and Tip-Adapter [45], introduce lightweight networks to learn refined features for classification tasks. On the other hand, CoOp [48] proposes a continuous prompt optimization strategy for image recognition, inspired by prefix-tuning for language models [17]. However, these methods, including Unsupervised Prompt Learning (UPL) [19], mainly target classification tasks and require labels or pseudo labels, making them ill-suited for data pre-selection tasks.
Our work differs from existing literature as we approach downstream tasks with zero prior knowledge. We demonstrate the adaptation of the V-L model to extract superior features in an unsupervised manner.
An overview of our approach is shown in Figure 2. First, we formally define the data pre-selection task. Next, we provide brief reviews on BLIP-2 and present our unsupervised prompt learning
framework, which assists BLIP-2 in extracting improved multimodal features for data pre-selection. Finally, we describe how to use the adapted BLIP-2 to select instances for downstream tasks.
3.1 Data Pre-selection
Figure 2a, shows the data pre-selection workflow. Suppose that we have an enormous dataset  $D$  consisting of  $d$  instances and an annotation budget of  $l$ . Our task is to select  $l$  ( $l \ll d$ ) instances for labeling, so that undefined downstream tasks trained on such a partially labeled dataset produce the best performance. Formally, let  $D = (x_i, y_i)_{i=1}^d$  denote  $d$  pairs of the image  $x_i$  and its class label  $y_i$  ( $y_i = \emptyset$  if the label is unknown). Let  $D_L$  denote a size  $l$  subset of  $D$  with known class labels. Our goal is to select  $D_L \subset D$  to acquire class labels, in order to maximize the performance of an undefined model trained on labeled data  $D_L$  with or without unlabeled data  $D_U = D \setminus D_L$ . Our data pre-selection task is challenging, as we do not have any labels to begin with. This means we do NOT know what information will make the undefined downstream task perform the best. Our idea is to select  $l$  instances that are not only representative but also diverse enough to cover the entire dataset, preventing premature loss of valuable information prior to label acquisition.
The process depicted in Figure 2a illustrates the detailed data pre-selection process by extracting high-quality and semantically significant features. These features facilitate the clustering of all unlabeled images into well-structured groups. Subsequently, we select the most representative instance from each cluster for labeling. This strategy ensures both representativeness and diversity, accomplished by selecting one instance from each cluster. In the upcoming section, we will describe how we adapt BLIP-2 to this objective.
3.2 Unsupervised Prompt Learning
Figure 2b depicts the Unsupervised Prompt Learning approach to extract desirable features. It consists of three cooperatively learned elements: the learnable context of BLIP-2 and two Multilayer Perceptron (MLP) heads for both instance-level and cluster-level transformation. These components are jointly optimized using unsupervised clustering objectives [25], specifically instance-level and cluster-level contrastive learning. After training, the cluster assignments of all instances can be easily derived from the predicted soft labels produced by the cluster-level MLP head. When combined with the multimodal features obtained from the adapted BLIP-2, we can effectively identify the most representative and diverse instances from the entire dataset  $D$ .
Bootstrapping Language-Image Pre-training known as BLIP-2, presents an efficient pre-training strategy that leverages existing pre-trained image encoders and large language models, all while remaining frozen. By training a lightweight
querying transformer (Q-Former), it specifically bridges the modality gap between images and text. In the representation learning phase, BLIP-2 connects Q-Former to a frozen image encoder and conducts image-text pair pre-training. With three image-text pre-training objectives (Image-Text Contrastive Learning, Image-grounded Text Generation, and Image-Text Matching), they establish a
Algorithm 1: Unsupervised Prompt Learning
Input: Unlabeled dataset  $D$ ; training epochs  $E$ ; batch size  $N$ ; temperature parameters  $\tau_{I}$  and  $\tau_{C}$ ; cluster number  $M$ ; learnable context  $\mathbf{V}$ ; augmentation set  $T$ ; BLIP-2  $f$ ; MLP heads  $g_{I}$  and  $g_{C}$ ;  $D_{L} \gets []$
Output: Labeled subset  $D_{L}$
// Training
Sample a mini-batch  $\{\mathbf{x}_i\}_{i = 1}^N$  from  $D$
Sample two augmentations  $T_{a}, T_{b} \sim T$
// Compute instance and cluster representations:
$$
\mathbf {h} _ {i} ^ {a} = f \left(T _ {a} \left(\mathbf {x} _ {i}\right), \mathbf {V}\right), \mathbf {h} _ {i} ^ {b} = f \left(T _ {b} \left(\mathbf {x} _ {i}\right), \mathbf {V}\right)
$$
$$
\mathbf {z} _ {i} ^ {a} = g _ {I} \left(\mathbf {h} _ {i} ^ {a}\right), \mathbf {z} _ {i} ^ {b} = g _ {I} \left(\mathbf {h} _ {i} ^ {b}\right)
$$
$$
\mathbf {c} _ {i} ^ {a} = g _ {C} \left(\mathbf {h} _ {i} ^ {a}\right), \mathbf {c} _ {i} ^ {b} = g _ {C} \left(\mathbf {h} _ {i} ^ {b}\right)
$$
Compute overall loss  $\mathcal{L}$
Update  $\mathbf{V}$ ,  $g_{I}$ , and  $g_{C}$  to minimize  $\mathcal{L}$
// Inference
for  $\mathbf{x}$  in  $D$  do
$\mathbf{h} = f(\mathbf{x})$  // Extract features
$c = \arg \max (g_C(\mathbf{h}))$  // Compute cluster assignment
// Labeling
for  $k = 1$  to  $M$  do
Select the most representative instance  $\mathbf{m}_k$
$$
D _ {L} = D _ {L} \cup \mathbf {m} _ {k}
$$
strong connection between image and input text, i.e. forcing the model to extract visual information from the image that is most relevant to the text and automatically generating output multimodal features. This characteristic of BLIP-2 provides us with a new opportunity. By altering different prompts (input text), we can extract distinct multimodal features and find the best to enhance the process of data pre-selection.
Learnable Prompt However, designing an appropriate prompt can be demanding and time-consuming, especially for data pre-selection, when we have zero knowledge of downstream tasks. Following the previous prompt learning approach, we aim to model each context token using a continuous vector that can be learned from data from end to end. Specifically, as depicted in Figure 2b, we define  $N$  learnable prompt representation context vectors, denoted as  $\mathbf{V} = \{\mathbf{v}_1, \mathbf{v}_2, \dots, \mathbf{v}_n\}$ , each having the same dimension as the word embedding, to serve as the text input for BLIP-2. In contrast to many previous methodologies that leverage cross-entropy loss as their learning objective with label data, we have chosen to adopt unsupervised clustering as our learning objective with respect to undefined downstream tasks. Since the text encoder is differentiable, which means gradients can be back-propagated to update the context vectors. It's important to note that the base model of BLIP-2 remains unchanged during the training process.
Instance-level Contrastive Loss Contrastive learning at the instance level is designed to maximize the similarities between positive instance pairs, while simultaneously minimizing those of negative pairs. Given the lack of labels for all the data, we aim to maximize the agreement between differently augmented views of the same instance and treat all other augmented instances as negative examples. Specifically, consider a randomly sampled mini-batch of image of  $N$  images. Each image, denoted by  $\mathbf{x}_i$ , undergoes different augmentation twice, thus creating two views of the same example:  $\mathbf{x}_i^a$  and  $\mathbf{x}_i^b$ . Subsequently, these two images are encoded, along with a learnable context  $\mathbf{V}$ , using BLIP-2  $f(\cdot)$  to generate multimodal representations:  $\mathbf{h}_i^a$  and  $\mathbf{h}_i^b$ . Following this, the representations are further transformed via a nonlinear transformation MLP  $g_I(\cdot)$ , known as an instance-level head, which yields  $\mathbf{z}_i^a$  and  $\mathbf{z}_i^b$ . Let  $\sin (.,.)$  denote the cosine similarity. Then the instance-level contrastive loss of a positive pair of examples  $(i,j) \in [1,N]$  is defined as
$$
\ell_ {i} ^ {a} = - \log \frac {\exp (\operatorname {s i m} \left(\mathbf {z} _ {i} ^ {a} , \mathbf {z} _ {i} ^ {b}\right) / \tau_ {I})}{\sum_ {k \in \{a , b \}} \sum_ {j = 1} ^ {N} \exp (\operatorname {s i m} \left(\mathbf {z} _ {i} ^ {a} , \mathbf {z} _ {j} ^ {k}\right) / \tau_ {I})}, \tag {1}
$$
where  $\tau_{I}$  is the temperature scalar. The final loss  $\mathcal{L}_I = \frac{1}{2N}\sum_i^N (\ell_i^a +\ell_i^b)$ , which is computed across all positive pairs in a mini-batch.
Cluster-level Contraitive Loss Similar to instance-level constrative learning, cluster-level constrative learning aims to maximize the similarities between positive cluster pairs, while minimizing the negative pairs. To construct the cluster representation, we use another cluster-level head  $g_{C}(\cdot)$  to map the multimodal representations to a  $M$ -dimensional space, where  $M$  equals the number of clusters. Thus  $m$ -th element of the feature can be interpreted as the instance's probability of belonging to the  $m$ -th cluster. Formally, suppose with mini-batch of  $N$  instances and  $M$  predefined clusters, the output of the cluster head is  $\mathbf{C}^a \in \mathbb{R}^{N \times M}$  under the first augmentation, thus  $\mathbf{C}_{n,m}^a$  can be interpreted as the probability of instance  $n$  under augmentation  $a$  being assigned to the cluster  $m$ . The  $i$ -th column of  $\mathbf{C}^a$  can be treated as the representation of the  $i$ -th cluster under augmentation  $a$ . Thus, we can from a positive pair of cluster by selecting  $i$ -th column of  $\mathbf{C}^a$  and  $\mathbf{C}^b$  noted as  $\hat{\mathbf{c}}_i^a$  and  $\hat{\mathbf{c}}_i^b$  while leaving other  $2M - 2$  pairs to be negative:
$$
\hat {\ell} _ {i} ^ {a} = - \log \frac {\exp \left(\operatorname* {s i m} \left(\hat {\mathbf {c}} _ {i} ^ {a} , \hat {\mathbf {c}} _ {i} ^ {b}\right) / \tau_ {C}\right)}{\sum_ {k \in \{a , b \}} \sum_ {j = 1} ^ {N} \exp \left(\operatorname* {s i m} \left(\hat {\mathbf {c}} _ {i} ^ {a} , \hat {\mathbf {c}} _ {j} ^ {k}\right) / \tau_ {C}\right)}, \tag {2}
$$
where  $\tau_{C}$  is the cluster-level temperature parameter. To avoid a trivial solution that most instances are assigned to the same cluster, a regularization that encourages the entropy of cluster assignment probability is added:  $H(\mathbf{C}) = -\sum_{i}^{M}\left[P(\hat{\mathbf{c}}_{i}^{a})\log P(\hat{\mathbf{c}}_{i}^{a}) + P(\hat{\mathbf{c}}_{i}^{b})\log P(\hat{\mathbf{c}}_{i}^{b})\right]$ , where  $P(\hat{\mathbf{c}}_i^k) = \frac{1}{N}\sum_{t = 1}^{N}\hat{\mathbf{C}}_{ti}^k,k\in \{a,b\}$ . The final cluster-level loss is  $\mathcal{L}_C = \frac{1}{2M}\sum_{i = 1}^{M}(\hat{\ell}_i^a +\hat{\ell}_i^b) - H(\mathbf{C})$