SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Cristina Menghini
cristina_menghini@brown.edu
Andrew Delworth
adelwort@cs.brown.edu
Stephen H. Bach
sbach@cs.brown.edu
Fine-tuning vision-language models (VLMs) like CLIP to downstream tasks is often necessary to optimize their performance. However, a major obstacle is the limited availability of labeled data. We study the use of pseudolabels, i.e., heuristic labels for unlabeled data, to enhance CLIP via prompt tuning. Conventional pseudolabeling trains a model on labeled data and then generates labels for unlabeled data. VLMs' zero-shot capabilities enable a "second generation" of pseudolabeling approaches that do not require task-specific training on labeled data. By using zero-shot pseudolabels as a source of supervision, we observe that learning paradigms such as semi-supervised, transductive zero-shot, and unsupervised learning can all be seen as optimizing the same loss function. This unified view enables the development of versatile training strategies that are applicable across learning paradigms. We investigate them on image classification tasks where CLIP exhibits limitations, by varying prompt modalities, e.g., textual or visual prompts, and learning paradigms. We find that (1) unexplored prompt tuning strategies that iteratively refine pseudolabels consistently improve CLIP accuracy, by 19.5 points in semi-supervised learning, by 28.4 points in transductive zero-shot learning, and by 15.2 points in unsupervised learning, and (2) unlike conventional semi-supervised pseudolabeling, which exacerbates model biases toward classes with higher-quality pseudolabels, prompt tuning leads to a more equitable distribution of per-class accuracy. The code to reproduce the experiments is at BatsResearch/menghini-neurips23-code.
Large pre-trained vision-language models (VLMs) [31, 43, 17] achieve remarkable accuracy without task-specific training but still require adaptation for optimal performance. Prompt-tuning [13, 18] is an approach to efficiently enhance VLMs performance on downstream tasks by learning inputs to the model. While learning prompts with a few labeled data can yield significant improvements [48, 2], a broader range of learning settings such as semi-supervised, transductive zero-shot, and unsupervised learning are still underexplored. All of these settings share access to unlabeled data, and the versatile zero-shot classification abilities of VLMs make pseudolabeling a natural approach to leveraging it. This paper investigates how the use of out-of-the-box pseudolabels assigned by CLIP can contribute to improving CLIP's own performance. To this end, we conduct an extensive exploration of learning scenarios by varying prompt modalities, learning paradigms, and training strategies. We present empirical evidence showcasing the effectiveness of iterative prompt-training strategies that leverage CLIP-based pseudolabels, regardless of learning paradigms and prompt modalities, resulting in significant improvements in CLIP's image classification performance across different settings.
Pseudolabels are heuristic labels assigned by a model to unlabeled data, which are leveraged to further train the model [20]. Successful training with pseudolabels relies on two factors: the quality
Figure 1: Our design space to explore the effect of leveraging pseudolabels in a unified way across prompt modalities, learning paradigms, and training strategies. The green (dashed) path has already been explored [15], while the red (solid) lines are the unexplored combinations for prompt tuning.
of the labels and how they are used during training. To address the first, conventional methods assign labels to instances with high-confidence predictions [36]. For pseudolabeling using CLIP, Huang et al. propose to select the most confident samples for each class [15], mitigating CLIP's bias [38] and miscalibration [22] (see Section 3). To assign pseudolabels, we rely on this approach and address the second point by exploring how to make the best use of them. We design a broad space of analysis considering three dimensions: prompt modalities, which are the model inputs we learn; learning paradigms, which define the data we have available; and training strategies, which describe the process used to optimize performance (Figure 1).
Research on prompt tuning has demonstrated that training strategies used for learning prompts in one modality can be transferred to learning prompts in a different modality. For instance, Visual Prompt Tuning [18] was originally designed to effectively fine-tune large vision models but can be adapted to efficiently fine-tune CLIP using the same training strategy as standard textual prompt tuning [48, 34, 44]. On the contrary, different learning paradigms with limited labeled data typically require distinct approaches specifically tailored to extract information from the available data [27, 12]. However, we observe that this changes by using VLM's generated pseudolabels. Unlike conventional pseudolabeling approaches that bootstrap off labeled data and are used as semi-supervised learning techniques [36, 3, 40], VLMs can generate pseudolabels in any learning setting. This offers a significant advantage, expanding the scope of pseudolabeling beyond semi-supervised learning, and making it a promising approach for other settings, such as transductive zero-shot and unsupervised learning. By using CLIP-based pseudolabels as a source of supervision, we can view these settings as optimizing the same loss function, which is simply a weighted sum of the errors on labeled data, if available, and pseudolabeled data. Given that we can express different settings as the same problem, we can propose training strategies, i.e., the way of using pseudolabels, that suit them all.
By standardizing the training strategies across various prompt modalities and learning settings, we can conduct experiments on different applications of pseudolabels for various combinations of prompt modalities, learning paradigms, and training strategies, as illustrated in Figure 1. To the best of our knowledge, only one potential path has been explored thus far; specifically, fine-tuning textual prompts in an unsupervised learning context using a few pseudolabels [15]. Rather than relying on a fixed set of pseudolabels, we propose iterative training techniques that allow for the ongoing refinement and expansion of the pool of pseudolabeled data used during training. With each iteration, we progressively enhance CLIP's pseudolabeling ability, allowing us to extend the set of pseudolabeled data while maintaining the high quality of the initial pseudolabels.
We conduct experiments on six tasks where CLIP has been observed to underperform [31], such as satellite-image classification, flower-species identification, and texture-image recognition, among others. Our findings reveal that iterative approaches effectively fine-tune prompts irrespective of their modality and learning paradigms. Recent studies have identified the "Matthew effect" as a potential issue for semi-supervised models that use pseudolabels [49, 38]. This phenomenon causes models to perform well on classes with accurate pseudolabels but poorly on those with inaccurate ones, thereby reinforcing the model's original bias towards certain classes. Our analysis reveals that using pseudolabels generated by CLIP for prompt-tuning with iterative strategies not only improves CLIP's overall performance but also corrects its natural bias towards certain classes.
We summarize the main takeaways of our work:
- General purpose zero-shot learners used as general purpose pseudolabelers open the opportunity to develop training strategies that leverage pseudolabeled data beyond semi-supervised learning. We point out that different learning paradigms, such as semi-supervised, transductive zero-shot, and unsupervised learning, can be all considered as special cases of a single objective function, by using pseudolabels as a source of supervision.
- We demonstrate that simple iterative training strategies for refining pseudolabels are highly effective approaches for limited-label prompt tuning. In fact, regardless of the prompt modality and learning setting, these strategies improve CLIP, by on average 19.5 points in semi-supervised learning, 28.4 in transductive zero-shot learning, and 15.2 in unsupervised learning.
- We show that prompts learned with iterative strategies help mitigate the "rich get richer, poor get poorer" effect observed in semi-supervised approaches leveraging pseudolabels. By redistributing the quality of pseudolabels across different classes, we observe a "Robin Hood effect" where the extremely rich classes' accuracy stays the same or decreases, while poorer classes get richer, leading to a more equitable distribution of per-class accuracy.
Vision-language models Vision-language models such as CLIP [31], ALIGN [17], and Florence [43] are models that align images and text. We focus on CLIP, which is composed of two components: a text encoder,  $\psi$ , and an image encoder,  $\phi$ , which are jointly trained using a contrastive loss to learn a multi-modal embedding space which aligns the representations of similar text and image inputs. This pre-training enables CLIP to perform zero-shot image classification. Given an image  $x$  and a set of classes  $\mathcal{Y} = \{y_1,\dots,y_C\}$ , CLIP classifies  $x$  by measuring the similarity between the image representation  $z = \phi (x)$  and each class representation  $w_{i} = \psi (\pi_{i})$ , based on their cosine distance in the shared embedding space. Here,  $\pi_{i}$  is a natural language prompt such as “a photo of a  $[\mathrm{CLASS}_i]$ ", where  $\mathrm{CLASS}_i$  is the specific class name, such as “orange dahlia,” “forest” or “Boeing 737”. The image  $x$  gets assigned to the class with the highest similarity score. In this work, we study how to learn better prompts that enhance CLIP by leveraging pseudolabels.
Prompt tuning Prompt tuning is a technique that enhances the practical application of large pretrained models like CLIP [31] and GPT [32, 7]. It involves providing task-specific information to the model during inference through textual or visual inputs, leading to improved performance on downstream tasks [1, 33, 6, 7]. While discrete prompts are manually crafted natural language descriptions of classes that guide the model, they may not yield optimal results [47]. Soft prompting [21, 24], on the other hand, optimizes prompts as continuous vectors. These can be optimized by backpropagating through the frozen pre-trained model, resulting in better performance. Soft prompts can be learned for various modalities, e.g., text or image, [48, 13, 17, 2, 44, 19] and applications [34, 27, 12, 28] by training on a small number of labeled examples per class. If only unlabeled data is accessible, it is possible to learn textual soft prompts by leveraging CLIP-based pseudolabels [15]. Expanding on this concept, we further investigate the use of pseudolabels across a broader range of prompt modalities and learning approaches, and we introduce unexplored training strategies to leverage pseudolabels more effectively.
Learning from pseudolabels Pseudolabeling is the practice of assigning labels to unlabeled data based on the prediction of a model [20]. Then, pseudolabels are used to improve the performance of the model itself. There are different ways to obtain and use pseudolabels and each impacts the final predictions of the model [41, 45, 16, 35]. Some approaches use confidence thresholds [36, 3, 40] and others average predictions from multiple augmentations [4]. Pseudolabeling is a semi-supervised learning technique, and it is rarely used in transductive zero-shot learning [42, 5, 25]. Applying such techniques requires a few labeled examples related to the target task to learn a baseline model capable of pseudolabeling. However, this limitation has been overcome by VLMs, which are capable of pseudolabeling examples without task-specific training. The conventional pseudolabeling scheme based on confidence threshold is not effective if we assign pseudolabels based on CLIP. In fact CLIP is miscalibrated [22] and has imbalanced predictions [38] which may induce noise in the pseudolabels. An alternative approach selects the top-K most confident examples per class to improve performance [15]. In our analysis, we rely on this scheme (Section 3).
3 Design space
Our analysis encompasses the design space consisting of various combinations of prompt modalities, learning paradigms, and training strategies (Figure 1). Within this space, two key components remain constant: the pseudolabeling scheme and a unified loss function. This section begins by introducing these components and subsequently delves into a comprehensive discussion of each dimension within the design space to be explored.
Pseudolabeling scheme The use of CLIP to generate pseudolabels has been investigated in [15]. Given unlabeled data  $X_{u}$  with target classes  $\{y_{1},\dots,y_{C}\}$ , the goal is to assign labels to data points in which the model is most confident. Typically, pseudo labeling schemes use a confidence threshold  $(P(y|x) > \tau)$  to select instances to pseudolabel. However, this approach does not work well for CLIP due to its miscalibration [22] and imbalanced predictions [38]. Instead, one can use a top-K pseudo labeling approach, where the top-K most confident examples per class are used as pseudolabeled data [15]. The pseudolabel assignment consists of (1) computing the similarity scores of each datapoint with classes' textual prompts, and (2) select for each class the  $K$  datapoints with the highest similarity score to the class. In this way, we always get  $K$  pseudolabels per class, effectively addressing the natural bias in CLIP's pseudolabels [38]. In Appendix A.1, we provide more details about pseudolabel assignment corner cases.
This top-K pseudolabeling scheme is applicable to unlabeled data, regardless of the availability of labeled data. As a result, we can extend the use of pseudolabels to any learning setting that involves unlabeled data. We observe that by treating pseudolabeled examples as true labeled data, we can view all learning settings as optimizing the same objective function.
Unified objective function Consider a  $C$ -class image classification task, where  $X_{L}$  and  $Y_{L}$  represent the image representations and labels of the labeled data, and  $X_{U}$  and  $\tilde{Y}_{U}$  denote the image representations and pseudolabels for the unlabeled data. We define a loss function that combines two cross-entropy losses, one accounting for the error on the labeled data points and the other accounting for the error on pseudolabeled data:
$$
\mathcal {L} = \mathcal {L} _ {C E} \left(X _ {L}, Y _ {L}\right) + \lambda \mathcal {L} _ {C E} \left(X _ {U}, \tilde {Y} _ {U}\right) \tag {1}
$$
where  $\gamma$  and  $\lambda$  define the training balance between the errors on labeled and pseudolabeled data.
3.1 Prompt modalities
Learning prompts is the process of training a set of vectors  $\mathrm{P} = [\mathrm{p}]_1\ldots [\mathrm{p}]_K$  that are prepended to the textual or visual inputs of the encoders within the CLIP architecture. By prepending these vectors to specific inputs, we can learn textual prompts, visual prompts, or multimodal prompts when applying a set of vectors to both inputs simultaneously. We provide a technical and detailed explanation in Appendix A.1.
In our exploration, we consider all three types of prompts. The efficacy of prompts can vary depending on the task. Text prompt tuning may be most beneficial when image features are well-separated by class but may not be aligned with the corresponding textual prompt. Visual prompts rearrange the image features within the projection space, and it has the potential to improve CLIP when the pre-trained image features are not well separated by class. Finally, multimodal prompts allows for beneficial interaction between the two separate modalities, which might lead to both separable visual features, and text classifiers that are well-aligned with the corresponding visual features.
3.2 Learning paradigms
By adjusting the values of parameters  $\gamma$  and  $\lambda$  and using the appropriate sets of labeled and pseudolabeled data, the unified objective loss can be customized for each learning paradigm. We note that the redundancy in the use of parameters is for notation clarity in descriptions of the different strategies below. One can simply use  $\lambda$  as balancing factor between labeled and pseudolabeled data.
Semi-supervised learning In the semi-supervised learning (SSL) scenario we have access to a limited number of labeled data for all the target classes  $D_{L} = \{(x,y)\}$  where  $x$  is an input feature and  $y\in \mathcal{V} = [C]$  is the corresponding label. In addition, we have access to unlabeled data  $X_{U} = \{x\}$ , where  $x$  is an image in the target domain  $\mathcal{V}$ . From  $X_{U}$ , we get  $\mathcal{D}_{PL} = \{(x,\tilde{y})\}$ , where  $\tilde{y}\in [C]$  is  $x$ 's pseudolabel. When using the unified loss in this setting, we set  $\gamma$  to  $|\mathcal{D}_{PL}| / |\mathcal{D}_L|$ . As  $|\mathcal{D}_L|$  is much smaller than  $|\mathcal{D}_{PL}|$ ,  $\gamma$  acts as an upweighting factor for the few-labeled instances, thus counterbalancing the learning effect of pseudolabels  $(\lambda = 1)$ .
Transductive zero-shot learning In transductive zero-shot learning (TRZSL), we are provided with labeled data  $D_{L} = \{(x,y)\}$  for some target classes  $S$  (referred to as seen classes), where  $x$  represents input features, and  $y \in [S]$  is the corresponding label. Additionally, we have access to unlabeled data  $X_{U} = \{x\}$  for a disjoint set of classes  $U$  (referred to as unseen classes). Using  $X_{U}$ , we obtain  $\mathcal{D}_{PL} = (x,\tilde{y})$ , where  $\tilde{y} \in [U]$  denotes the pseudolabels for  $x$ . The value of  $\lambda$  in the unified loss is set to  $|\mathcal{D}_L| / |\mathcal{D}_{PL}|$ , which makes the weight of the pseudolabel loss equivalent to that of the labeled data  $(\gamma = 1)$ . This is necessary because an imbalance in the number of labeled and pseudolabeled samples can result in a skewed training distribution, leading to better performance on seen classes while the performance on unseen classes may either remain stagnant or degrade. Studying this setting is interesting beyond transductive zero-shot learning. In fact, it has the potential to generalize to scenarios where the target task involves unseen classes, while the seen classes consist of auxiliary labeled data from the same domain but different task [30].
Unsupervised learning In the unsupervised learning (UL) setting, we have access only to unlabeled data  $X_{U} = \{x\}$ , from which we obtain  $\mathcal{D}_{PL} = (x,\tilde{y})$ , where  $\tilde{y}\in [C]$  denotes the pseudolabel for  $x$ . In this case,  $\gamma$  is set to 0, as there is no labeled data, and  $\lambda = 1$ . The use of this setting was initially explored in [15], who leveraged a few pseudolabels per class to learn textual prompts. In this paper, we build on their work by investigating a variety of training strategies and prompt modalities.
Supervised learning In supervised learning (SL), we are only provided with labeled data  $D_{L} = (x,y)$ , where  $x$  represents an input feature, and  $y \in [C]$  is the corresponding label. If we set  $\lambda$  to 0, the unified loss function is equivalent to the objective functions of default prompt-tuning approaches that optimize the prompts using a few labeled instances per target class. This setting is not strictly part of our design space. However, we will refer to it to define baselines in Section 4.
3.3 Training strategies
The unified objective function enables the development of training strategies broadly applicable across various learning paradigms. We explore three distinct learning strategies to effectively use pseudolabels in this context. The first strategy uses pseudolabels in a static manner. The other two strategies, which are unexplored for prompt tuning, involve the dynamic use of pseudolabeled data.
Few-pseudolabels (FPL) We select  $K$  pseudolabels per target class, resulting in a pseudolabeled dataset of size  $K \cdot C$ . We learn the prompts by minimizing the objective function via backpropagation through CLIP's encoders. This strategy aligns with Unsupervised Prompt Learning (UPL) in [15]. We refer to it as few-pseudolabels (FPL) to encompass its applicability for learning prompts of diverse modalities across learning paradigms.
Iterative Refinement of FPL (IFPL) Similar to FPL, we obtain the top-K pseudolabels for each target class. These pseudolabels are then used to train a new task-specific prompt. After completing the training, we use the learned prompt to compute the top-K pseudolabels per class again. Subsequently, we reinitialize the prompt and repeat this entire process for a total of  $I$  iterations. With this iterative approach, if training with the initial pseudolabel set leads to an improvement in the model's performance, the model itself can become a more effective pseudolabeler, refining the pseudolabels in each subsequent iteration.
Grow and Refine Iteratively Pseudolabels (GRIP) Although IFPL can improve the quality of the  $K \times C$  pseudolabels used for training, it still limits learning to a few examples per target class. To overcome this constraint, we explore a method similar to IFPL, but with a key difference. In each iteration, we progressively increase the value of  $K$ . Specifically, during the  $i$ -th iteration, we use
$K = (i \times \frac{|X_U|}{I}) / C$  of the unlabeled data to perform the steps in the iterative process. GRIP maintains class balance by selecting the top-K samples at each iteration, with  $K$  increasing progressively. Similar to IFPL, both prompts and pseudolabels are reinitialized with every iteration, in order to avoid accumulating errors from earlier iterations. In other words, learning progresses from pseudolabels to new prompts to new pseudolabels, and so on. The rationale behind this strategy is that as the model's accuracy in generating pseudolabels improves, we can increase the total number of pseudolabels without introducing excessive noise.
We explore the design space outlined in Section 3 to understand the effectiveness of leveraging pseudolabels for limited-label prompt tuning. We show that (1) iterative strategies significantly improve CLIP's performance across prompt modalities and learning settings, (2) using CLIP-based pseudolabels with iterative strategies induces a more equitable distribution of per-class accuracy.
Datasets We conduct the analysis on six tasks, covering specialized and fine-grained domains, where CLIP shows deficiencies [31]. We call this set of tasks FRAMED, and it includes Flowers102 [29], RESICS45 [9], FGVC-Aircraft [26], MNIST [11], EuroSAT [14], and DTD [10]. For each dataset we use the training and test splits provided in [23]. For the transductive zero-shot learning setting we randomly generate three splits of seen and unseen classes with a 62-38 ratio. Further details are in Appendix A.2.
Baselines To evaluate the effectiveness of the training strategies described in Section 3.3, we compare the performance of CLIP when queried with the learned soft prompts to CLIP zero-shot with default prompts such as "a photo of a [CLASS]." In addition, we compare with default supervised prompt-tuning baselines, for which we only use the available labeled data: CoOp [48] for textual prompts, VPT [18] for visual prompts, and UPT [44] for multimodal prompts. We defer to Appendix A.1 the technical details of these methods.
Evaluation metrics We assess the performance of each method by measuring the accuracy of the test set, averaging the results over five runs. In the case of TRZSL, we report the harmonic of the accuracies of seen and unseen classes to account for their potentially imbalanced performance [39].