Enhancing CLIP with CLIP: Exploring Pseudolabeling for Limited-Label Prompt Tuning
Authors: Cristina Menghini
Andrew Delworth
Stephen Bach
Venue: NeurIPS 2023 poster (2023)
Keywords: vision-language models
prompt-tuning
pseudolabels
self-training
TL;DR: We explore the use of CLIP's pseudolabels to enhance CLIP's performance via prompt-tuning with limited labeled data.

Abstract: Fine-tuning vision-language models (VLMs) like CLIP to downstream tasks is often necessary to optimize their performance. However, a major obstacle is the limited availability of labeled data. We study the use of pseudolabels, i.e., heuristic labels for unlabeled data, to enhance CLIP via prompt tuning. Conventional pseudolabeling trains a model on labeled data and then generates labels for unlabeled data. VLMs' zero-shot capabilities enable a ``second generation'' of pseudolabeling approaches that do not require task-specific training on labeled data. By using zero-shot pseudolabels as a source of supervision, we observe that learning paradigms such as semi-supervised, transductive zero-shot, and unsupervised learning can all be seen as optimizing the same loss function. This unified view enables the development of versatile training strategies that are applicable across learning paradigms. We investigate them on image classification tasks where CLIP exhibits limitations, by varying prompt modalities, e.g., textual or visual prompts, and learning paradigms. We find that
(1) unexplored prompt tuning strategies that iteratively refine pseudolabels consistently improve CLIP accuracy, by 19.5 points in semi-supervised learning, by 28.4 points in transductive zero-shot learning, and by 15.2 points in unsupervised learning, and (2) unlike conventional semi-supervised pseudolabeling, which exacerbates model biases toward classes with higher-quality pseudolabels, prompt tuning leads to a more equitable distribution of per-class accuracy. The code to reproduce the experiments is at https://github.com/BatsResearch/menghini-neurips23-code.

Enhancing CLIP with CLIP: Exploring
Pseudolabeling for Limited-Label Prompt Tuning
Cristina Menghini Brown University cristina_menghin iarpriiiiirooom.di..
Andrew Delworth
Stephen H. Bach Brown University sbachêcs brown edu
Brown University adelwort@cs, brown edu
Abstract
Fine-tuning vision-language models (VLMs) like CLIP to downstream tasks is often necessary to optimize their performance. However, major obstacle is the limited availability of labeled data. We study the use of pseudolabels, i.e., heuristic labels for unlabeled data, to enhance CLIP via prompt tuning. Conven- tional pseudolabeling trains a model on labeled data and then generates labels for unlabeled data. VLMs' zero-shot capabilities enable "second generation" of pseudolabeling approaches that do not require task-specific training on labeled data. By using zero-shot pseudolabels as source of supervision, observe that learning paradigms such as semi-supervised, transductive zero-shot, and un- supervised learning can all be seen as optimizing the same loss function. This unified view enables the development of versatile training strategies that are ap- plicable across learning paradigms. We investigate them image classification tasks where CLIP exhibits limitations. by varying prompt modalities, e.g., textual or visual prompts, and learning paradigms. We find that (1) unexplored prompt tuning strategies that iteratively refine pseudolabels consistently improve CLIP accuracy, by 19.5 points semi-supervised learning, by 28.4 points transductive zero-shot learning, and by 15.2 points in unsupervised learning, and (2) unlike conventional semi-supervised pseudolabeling. which exacerbates model biases toward classes with higher-qualin pseudolabels, prompt tuning leads to more equitable distribution of per-class accuracy. The code to reproduce the experiments is at BaisRescarch/meenghin htwprcccccennnni----uuui----oooo
Introduction
Large pre-trained vision-language models (VLMs) 31 43 17 achieve remarkable accuracy without
task-specific training but still require adaptation optimal performance. Prompt-tuning [13 [8]
an approach to efficiently enhance VLMS performance on downstream tasks by learning inputs to the model. While lcarning prompts with a few labeled data can yield significant improvements [48 2],
broader range of learning settings such as semi-supervised, transductive zero-shot, and unsupervised learning are still underexplored. All of these settings share access to unlabeled data, and the versatile zero-shot classification abilities of VLMs make pseudolabeling natural approach to leveraging it. This paper investigates how the use of out-of-the-bbx box pseudolabels assigned by CLIP can contribute to improving CLIP's scenarios by varying prompt modalities, learning paradigms, and training strategies. We present empirical evidence showcasing the effectiveness of iterative prompt-training strategies that leverage CLIP-based pseudolabels, regardless of learning paradigms and prompt modalities, resulting in significant improvements in CLIP's image classification performance across different settings.
performance. To this end, conduct extensive exploration of learning
Pseudolabels are heuristic labels assigned by a model unlabeled data, which are leveraged to further train the model [00 Successful training with pscudolabels relies on two factors: the quality
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Textual Prompes (CoOp)
Semi-supervised learning (SSL.)
Few- www-Pseudoolbelss (FPI.)
Transductive zcro-shot learning (TRZSL)
Visual Prompls (VPT)
llerativv F'ww-Pseeucooi Pseuckolablls (IFLL)
Grow and Refine Iteratively Pscudolabels (GRIP)
Multimodal Prompts (UPT)
Unsupervised learning (LL)
Prompt
Leurning
Training
Modalities
Parudigms
Strutegies
Figure 1: Our design space to explore the effect of leveraging pseudolabels in a unified way across prompt modalities, learning paradigms, and training strategies. The green (dashed) path has already been explored [5]. while the red (solid) lines are the unexplored combinations for prompt tuning.
of the labels and how they are used during training. To address the first, conventional methods assign labels to instances with high-confidence predictions [36]. For pseudolabeling using CLIP, Huang et al. propose to select the most confident samples for each class [15), mitigating CLIP's bias (38) and miscalibration [22] (see Section To assign pseudolabels, we rely on this approach and address the second point by exploring how to make the best use of them. We design a broad space of analysis considering three dimensions: prompt modalities, which are the model inputs we learn; learning paradigms, which define the data we have available; and training strategies, which describe the process used to optimize performance (Figure
Research on prompt tuning has demonstrated that training strategies used for learning prompts in one modality can be transferred to learning prompts in different modality. For instance, Visual Prompt Tuning [18] was originally designed to effectively fine-tune large vision models but can be adapted to efficiently fine-tune CLIP using the same training strategy as standard textual prompt
tuning 48 44 On the contrary, different learning paradigms with limited labeled data typically require distinct approaches specifically tailored to extract information from the available data [27 12 However, we observe that this changes by using VLM's generated pseudolabels. Unlike conventional pseudolabeling approaches that bootstrap off labeled data and are used as semi-supervised learning techniques [36]] 3 40). VLMs can generate pseudolabels in any lcarning setting. This offers a
significant advantage, expanding the scope of pseudolabeling beyond semi-supervised learning, and making it a promising approach for other settings, such as transductive zero-shot and unsupervised learning. By using CLIP- pseudolabels as source of supervision, optimizing the same loss function, which is simply a weighted sum of the errors on labeled data, available, and pseudolabeled data. Given that we can express different settings as the same problem, we can propose training strategies, i.e., the way of using pseudolabels, that suit them all.
view these settings as
By standardizing the training strategies across various prompt modalities and learning settings, we can conduct experiments on different applications of pscudolabels for various combinations of prompt modalities, learning paradigms, and training strategies, illustrated in Figure I To the best of our knowledge, only one potential path has been explored thus far; specifically. fine-tuning textual prompts in unsupervised learning context using few pseudolabels (15). Rather than relying on a fixed set of pseudolabels, we propose iterative training techniques that allow for the ongoing refinement and expansion of the pool of pseudolabeled data used during training. With each iteration, we progressively enhance CLIP's pseudolabeling ability, allowing us to extend the set of pseudolabeled data while maintaining the high quality of the initial pseudolabels.
We conduct experiments on six tasks where CLIP has been observed to underperform [1I, such as satellite-imal image classification, flower-species identification, and texture-image recognition, among others. Our findings reveal that iterative approaches effectively fine-tune prompts irrespective of their modality and learning paradigms. Recent studies have identified the "Matthew effect" potential issue for semi- semi-supervised models that use pseudolabels 49 38 This phenomenon causes models to perform well classes with accurate pseudolabels but poorly on those with inaccurate ones, thereby reinforcing the model's original bias towards certain classes. Our analysis reveals that using pseudolabels generated by CLIP for prompt-tuning with iterative strategies not only improves CLIP's overall performance but also corrects its natural bias towards certain classes.
We summarize the main takeaways of our work:

---

General purpose zero-shot learners used as general purpose pseudolabelers open the opportu- nity to develop training strategies that leverage pseudolabeled data beyond semi-supervised learning. We point out that different learning paradigms, such as ami-supervised, transduc- tive zero-shot, and unsupervised learning, can be all considered as special cases of a single objective function, by using pseudolabels as source of supervision.
We demonstrate that simple iterative training strategies for refining pseudolabels are highly effective approaches for limited-label prompt tuning. In fact, regardless of the prompt modality and learning setting, these strategies improve CLIP, by on average 19.5 points in semi-supervised learning, 28.4 in transductive zero-shot learning, and 15.2 in unsupervised learning.
We show that prompts learned with iterative strategies help mitigate the "rich get richer, poor get poorer" effect observed in semi-supervised approaches leveraging pseudolabels. By redistributing the quality of pseudolabels across different classes, we observe a "Robin Hood effect" where the extremely rich classes' accuracy stays the same decreases, while poorer classes get richer, leading to a more equitable distribution of per-class accuracy.
2 Background and related work
Vision-language models Vision-language models such CLIP BI. ALIGN 17). and Flo- rence [4 are models that align images and text. We focus on CLIP, which is composed of two components: text encoder, vi, and image encoder, o, which are jointly trained using contrastive loss to learn a multi-modal embedding space which aligns the representations of similar text and image inputs. This pre-training enables CLIP to perform zero-shot image classification. Given an image x and a set of classes y = {V11,.., VC ). CLIP classifies x by measuring the similarity between the image representation = = <fx) and each class representation w; = (Tta), based on their cosine distance in the shared embedding space. Here, Tú is a natural language prompt such as "a photo of [CLASS,]', where CLASS, is the specific class name, such as "orange dahlia," "forest" or "Boeing 737". The image gets assigned to the class with the highest similarity score. In this work, we study how to learn better prompts that enhance CLIPP by leveraging pseudolabels.
Prompt tuning Prompt tuning is a technique that enhances the practical application of large pre- trained models like CLIP (31] and GPT [32 77 It involves providing task-specific information to the model during inference through textual or visual inputs, leading to improved performance on down- stream tasks D| 6, 77. While discrete prompts are manually crafted natural language descriptions of classes that guide the model, they may not yield optimal results 447. Soft prompting [21 24], on the other hand, optimizes prompts continuous vectors. These can be optimized by backpropagating through the frozen pre-trained model, resulting in better performance. Soft prompts can be learned
44 19] and applications [3 27 12 28 by training on a small number of labeled examples per class. If only unlabeled data is accessible, it is possible to learn textual soft prompts by leveraging CLIP-based pseudolabels [15]. Expanding
for various modalities, e.g., text image, [48 13
17
2
this concept, we further investigate the use of pseudolabels across a broader range of prompt modalities and learning approaches, and we introduce unexplored training strategies to leverage pseudolabels more effectively.
Learning from pseudolabels Pseudolabeling is the practice of assigning labels to unlabeled data based on the prediction of model 20 Then, pseudolabels are used to improve the performance of
the model itself. There are different ways to obtain and use pseudolabels and each impacts the final
predictions of the model [41 45 16 35). Some approaches use confidence thresholds [36] 40) and
others average predictions from multiple augmentations [4). Pscudolabeling is semi-supervise
learning technique, and rarely used in transductive zero-shot learning 42 25 Applying
such techniques requires a few labeled examples related to the target task to learn baseline model capable of pseudolabeling. However, this limitation has been overcome by VLMs, which are capable of pseudolabeling examples without task-specific training. The conventional pseudolabeling scheme based on confidence threshold is not effective if we assign pseudolabels based on CLIP. In fact CLIP is miscalibrated [22) and has imbalanced predictions (38) which may induce noise in the pseudolabels. An alternative approach selects the top-K most confident examples per class to improve performance [15]. In our analysis, we rely on this scheme (Section 3
3

---

3 Design space
Our analysis encompasses the design space consisting of various combinations of prompt modalities, learning paradigms, and training strategies (Figure Tp. Within this space, two key components remain constant: the pseudolabeling scheme and unified loss function. This section begins by introducing these components and subsequently delves into comprehensive discussion of each dimension within the design space to be explored.
Pseudolabeling scheme The use of CLIP to generate pseudolabels has been investigated in [55 Given unlabeled data X₂ with target classes (y1...., BC). the goal is to assign labels to data points in which the model is most confident. Typically, pseudo labeling schemes use confidence threshold (P(ylz) > r) to select instances to pseudolabel. However, this approach does not work well for CLIP due to its miscalibration [22] and imbalanced predictions (38]. Instead, one can use top-K pseudo labeling approach, where the top-K most confident examples per class are used as pseudolabeled data [S. The pseudolabel assignment consists of (1) computing the similarity scores of each datapoint with classes' textual prompts, and (2) select for each class the K datapoints with the highest similarity score to the class, In this way. we always get K pseudolabels per class, effectively addressesing the natural bias in CLIP's pseudolabels [38] In Appendix A.1 we provide more details about pseudolabel assignment corner cases.
This top-K pseudolabeling scheme is applicable to unlabeled data, regardless of the availability of labeled data. As result, we can extend the use of pseudolabels to any learning setting that involves unlabeled data. We observe that by treating pseudolabeled examples as true labeled data, we can view all learning settings as optimizing the same objective function.
Unified objective function Consider a C-class image classification task, where X and Y2 rep- resent the image representations and labels of the labeled data, and Xy and Yú denote the image representations and pseudolabels for the unlabeled data. We define loss function that combines two cross-entropy losses, one accounting for the error on the labeled data points and the other accounting for the error pscudolabeled data:
C = CCE(L,, YL) + ACCE(XU, Yv)
where y and À define the training balance between the errors on labeled and pseudolabeled data.
3.1 Prompt modalities
Learning prompts is the process of training set of vectors P [p] ... [p]x k that are prepended to the textual or visual inputs of the encoders within the CLIP architecture. By prepending these vectors to specific inputs, we can learn textual prompts, visual prompts, or multimodal prompts when applying set of vectors to both inputs simultaneously. We provide technical and detailed explaination in Appendix
In our exploration, we consider all three types of prompts. The efficacy of prompts can vary depending on the task. Text prompt tuning may be most beneficial when image features are sell-separated by class but may not be aligned with the corresponding textual prompt. Visual prompts rearrange the image features within the projection space, and it has the potential to improve CLIP when the pre-trained image features are not well separated by class. Finally, multimodal prompts allows for beneficial interaction between the two separate modalities, which might lead to both separable visual features, and text classifiers that are well-aligned with the corresponding visual features.
3.2 Learning paradigms
By adjusting the values of parameters y and X and using the appropriate sets of labeled and pseudola- beled data, the unified objective loss can be customized for each learning paradigm. We note that the redundancy the use of parameters is for notation clarity descriptions of the different strategies below. One can simply use À as balancing factor between labeled and pseudolabeled data.

---

Semi-supervised learning In the semi-supervised learning (SSL) scenario we have access to limited number of labeled data for all the target classes D,. (2, y)} where x is an input feature and y E y = [C] is the corresponding label. In addition, we have access to unlabeled data X = {*) where x: is an image the target domain y. From Xv, we get Drz = {(x, ()), where ğ E C is z's pseudolabel. When using the unified loss in this setting, we set y to [Dpzl/|D) As [Py)
is much smaller than [Ppz) Y acts as an upweighting factor for the few-labeled instances, thus counterbalancing the learning effect of pseudolabels (A=1).
Transductive zero-shot learning In transductive zero-shot learning (TRZSL), we are provided with labeled data D, = {(1, 3)} for some target classes S (referred to as seen classes), where represents input features, and y E [S] is the corresponding label. Additionally, we have access to unlabeled data Xe = {z} for disjoint set of classes U (referred to as unseen classes). Using Xu. we obtain Drl
(x. fi) where E [U] denotes the pseudolabels for æ. The value of X in the unified loss is set to [Dr.I/[Drr.] which makes the weight of the pseudolabel loss equivalent to that of the labeled data (7- = 1). This is necessary because an imbalance in the number of labeled and pseudolabeled samples can result skewed training distribution, leading to better performance on seen classes while the performance on unseen classes may either remain stagnant or degrade. Studying this setting is interesting beyond transductive zero-shot learning. In fact, it has the potential to generalize to scenarios where the target task involves unseen classes, while the seen classes consist of auxiliary labeled data from the same domain but different task (30]
Unsupervised learning In the unsupervised learning (UL) setting, we have access only to unlabeled data X = {x)) from which we obtain Drz = (x. 5). where E [C] denotes the pseudolabel for x. In this case, y is set to 0, as there is no labeled data, and = 1. The use of this setting was initially explored in [15], who leveraged a few pseudolabels per class to learn textual prompts. In this paper,
we build on their work by investigating a variety of training strategies and prompt modalities.
Supervised learning In supervised learning (SL), we are only provided with labeled data Dr (z, y), where I' represents an input feature, and y E [C] is the corresponding label. If we set À to O, the unified loss function is equivalent to the objective functions of default prompt-tuning approaches that optimize the prompts using few labeled instances per target class. This setting is not strictly part of our design space. However, we will refer
define baselines in Section
3.3 Training strategies
The unified objective function enables the development of training strategies broadly applicable across various learning paradigms. We explore three distinct learning strategies to effectively use pseudolabels in this context. The first strategy uses pseudolabels in static manner. The other two strategies, which are unexplored for prompt tuning, involve the dynamic use of pseudolabeled data.
Few-pseudolabed (FPL) We select K pseudolabels per target class, resulting in a pseudolabeled dataset of size K C. We learn the prompts by minimizing the objective function via backpropagation through CLIP's encoders. This strategy aligns with Unsupervised Prompt Learning (UPL) in (15). We refer to it as sow-pseudolabess (FPL) to encompass its applicability for learning prompts of diverse modalities across learning paradigms.
Iterative Refinement of FPL (IFPL) Similar to FPL, we obtain the top-K pseudolabels for each target class. These pseudolabels are then used to train new task-specific prompt. After completing the training, we use the learned prompt to compute the top-K pseudolabels per class again. Subsequently, we reinitialize the prompt and repeat this entire process for total of / iterations. With this iterative approach, if training with the initial pscudolabel set leads to an improvement in the model's 's performance, the model itself can become more effective pseudolabeler, refining the pseudolabels in each subsequent iteration.
Grow and Refine Iteratively Pseudolabels (GRIP) Although IFPL can improve the quality of the K x C' pseudolabels used for training, it still limits learning to few examples per target class. To overcome this constraint, we explore a method similar to IFPI., but with a key difference. In each iteration, we progressively increase the value of K. Specifically, during the i-th iteration, we use
5

---

K - ) of the unlabeled data to perform the steps in the iterative process. GRIP maintains class balance by selecting the top-K samples at each iteration, with K increasing progressively. Similar to IFPI., both prompts and pseudolabels are reinitialized with every iteration, in order to avoid accumulating errors from earlier iterations. other words, learning progresses from pseudolabels to new prompts to pseudolabels, and on. The rationale behind this strategy is that the model's accuracy in generating pseudolabels improves, we can increase the total number of pscudolabels without introducing excessive noise.
Experiments
We explore the design space outlined in Section to understand the effectiveness of leveraging pseudolabels for limited-label prompt tuning. We show that (1) iterative strategies significantly improve CLIP's performance across prompt modalities and learning settings, (2) using CLIP-based pseudolabels with iterative strategies induces a more equitable distribution of per-class accuracy.
Datasets We conduct the analysis on six tasks, covering specialized and fine- aiee-griind domains, where CLIP shows deficiencies [3]] We call this set of tasks FRAMED, and it includes Flowers 102 22). RESICS45 191 FGVC-Aircratt 26). MNIST 11. Eu [1), and DTD [TO]. For each dataset we use the training and test splits provided in (23] For the transductive zero-shot learning setting we randomly generate three splits of seen and unseen classes with a 62-38 ratio.
Further details are in Appendix A.2
Baselines To evaluate the effectiveness of the training strategies described in Section 3.3 we compare the performance of CLIP when queried with the learned soft prompts to CLIP zero-shot with default prompts such as "a photo of [CLASS]." In addition, we compare with default supervised trompt-tuning baselines, for which we only use the available labeled data: CoOp [48] for textual prompts, VPT [[8 for visual prompts, and UPT 44 for multimodal prompts. We defer to Appendix/A A. the technical details of these methods.
We assess the performance of each method by measuring the accuracy of the test set, averaging the results over five runs. In the case of TRZSI,, we report the harmonic of the accuracies of seen and unseen classes to account for their potentially imbalanced performance [39].
Evaluation metrics
Training settings For all experiments, datasets, and learning strategies, we use ViT-B/32 as the vision backbone. For both visual and textual prompt learning, we set the prefix size to 16 48 [8]. Multimodal prompts have length 8 [4]] We use SGD) as the optimizer and train for 1 150 epochs. We use warmup epochs at a learning rate of 0.0001, and then set the learning rate to I, which is decayed by the cosine annealing rule. For textual and visual prompt learning, 7 7= 0.1, while for multimodal prompt learning, 1 = 0.01. In SSL, we use labeled samples per class to assess the impact of pseudolabels in the scenario of very few labeled data and abundant unlabeled data. The number of iterations / 10. FPL and IFPL have the number of pseudolabels per class fixed to 16 since it is indicated as the optimal K in the previous research pseudolabeling with CLIP [15). In general, K is hyperparameter that may require optimization in practical cases. We decide to be consistent with the literature and apply this fixed value of K in order to reduce the addition of more confounding factors in our analysis.
4.1 Exploring the design space
GRIP consistently enhances CLIP across prompt modalities and learning settings Table I reports the performance of GRIP, the best performing among the training strategies in Section 3.3 compared to CLIP and prompt-tuning baselines. Overall, GRIP consistently improves the performance of CLIP and the baselines across prompt modalities and learning settings. By tuning textual prompts, the average improvement over CLIP is 20.7 points in SSL, 14.9 in UL, and 32.4 in TRZSL, while the improvement on CoOp 9.6 points SSL, and 26.6 in TRZSL Similar results for the visual prompts show that GRIP improves CLIP by 18.2 points in SSL, 15.7 in UL. and 30.8 in TRZSL, and VPT by 12.9 points in SSI., and 20.8 in TRZSI. We note that CoOp and VPT applied to the SSL setting correspond to learning only the labeled data, and we do not run them in the UL setting as

---

Textual prompts
Flowers1022
RESIC545
FGVCAiremaf
Mathod
SSL.
UL
TRZSI.
SSL.
UL
TRZSI.
SSL.
UL
TRZSI.
CLIP
63.6 3T.6C..co
63. 03.400.00
5-1. 18..0
51.466..00
17.5%
17. 17.86n.00
CoOp
76.76
63. 02.22
58.5310.81
63.37
14.915,22
21.70h.oz
GRIP
83.6..s
69.84.um .06
86.26..00
74.11..ss
70.55
81.07..00
16.98o.82
12.22
26.08
A CLIP
1 19.03
6.17
1 22.86
1 19.63
16.07
1 26.61
0.6.
2.36
1 8.22
CoOp
6.84
1 23.04
1 15.58
1 17.70
2.07
1 4.38
MNIST
EurOSAT
DTD
CLIP
25. 20..00
20.770.00
352.880.00
30.54g.00
43.240.00
43.45%.00
CoOp
56. 56.42g.06
21. 21.15g.00
59.51
49.688..s
37. 106.45
46.3g.03
GRIP
71.78g,50
27.88
74.06..00
58.662.84
57.21
92.330.00
56.070.ss
46.09
65.30b.01
CLIP
t 16.68
1 12.78
1 53.29
25.78
21133
16.79
12.83
2.85
21.85
A CoDp
t 15.36
1 52.91
0.85
1 42.65
1 16.97
19.00
Visual prompts
Flowers 102
RESICS45
FGVCAireraft
Method
SSL
UL
TRZSI.
SSL.
UL
TRZSI.
SSL
UL
TRZSI.
CLIP
63.67o.00
63.40g.00
54.48%,00
54.466.00
17.580.0
17. 86
VPT GRIP
63.73, .22
64.71g.00
60.80,.05
G7. 67.066.00
17.760.09
26.69
67,95
63.09
77.18,
71.22
68.430
82.19
19,430.5
17.51m.6
26. 20.00
A CLIP
14228
L. 0.58
1 13.78
1 16.74
1 13.95
1 27.73
188
0.00
1 8.56
VPT
14222
1 12.47
1 10:42
15:13
1.67
4 0.27
MNIST
EuroSAT
DTD
CLIP
25.106.c0
20.77g.00
32. 880.08
30.54..00
43. 43.244.00
43. $3.455..0n
VPT
44.5
25.51a.os
17.131.34
62.24a.m2
36.112.17
16.106.01
GRIP
69.666.61
68.04,.11
69.54..01
63.483.00
63.68s.4 42
96.97..00
54.574.66
50.510.90
62.78%.00
A CLIPP
1 44.56
1 42.94
1 48.77
1 33060
30.80
1 66.43
1 11.33
7.27
19.33
VPT
t 27.14
1 44.03
f 16.35
t 34.73
1 18.16
18.62
Table 1: For each learning paradigm, we compare the accuracy of GRIP with CLIP zero-shot (ViT- B/32), CoOp, and VPT. Results are for SSL., UL, and TRZSL on FRAMED, We average the accuracy on seeds and report the standard deviation. METHOD is the difference between the accuracy of GRIP and METHOD. We note that for UL we can not apply CoOp and VPT since no labeled data is available.
there is no labeled data. Results are similar for multimodal prompts. We defer them to Appendix A.3 due to space constraints.
No prompt modality is clearly superior Using pseu- dolabels dynamically is beneficial for each modality. How- ever, determining the clear superiority of one prompt modality over the other is challenging, as it depends on the specific tasks. For example, visual prompts work better for EuroSAT, while textual prompts excel in Flowers 102.
Balance Accuracy
100
Coow/eP ITL PTL GrUP
Accuracy (%)
60
40-
Despite intuitive explanations (Section the scientific consensus remains elusive [44) Hence, we prefer to em-
phasize that the dynamic use of pseudolabels consistently improves performance for each prompt modality, without declaring one modality definitively better than the other.
-100
-50
1.00
150
200
Class Balance (%)
Figure 2: Balance of seen and unseen accuracies vs. model's overall accuracy. Points close O indicate good balance. Negatives represent better accuracy for the seen classes.
Unsupervised learning is equivalent or more robust than learning with very few shots The accuracy of GRIP when applied to the fully unsupervised setting is either higher or equivalent the accuracy of VPT, which is trained using two labeled instances per class [Table[]]. This shows that pseudolabeled data can substitute very few labeled examples for prompt tuning. However, the significant improvement of GRIP over CoOp and VPT in the semi-supervised setting (see Table I suggests that leveraging unlabeled data through pseudolabeling is advantageous in scenarios where labeled data is searce but there is an abundance of unlabeled data.
7

---

Textual prompts
Flowers1002
RESICS45
DID
Method
SSL
UL
TRZSL
SSL
UL
TRZSL
SSL
UL
TRZSL
FPL
75. 96n.7 r
65.67o.25
80.97o.00
68, 18,
63..07..5
72. 11.00
37 .0.10..45
11.966.
16.30.0
IFPL GRIP
78, 75
69.56..0
82.00
70.52,
64.11o.s
75.51p.00
55.24u.
47.77,.15
o9
83.600.4s
69.84.0s
86.260.00
74.11a.c
70.55p.58
81.070.00
56.07
46.0%
65.30k.
IFPL.
1 2.72
3.80
1 1.11
2.33
1 1.07
1 3.4
1 18.14
1 2.81
1 12.84
A GRIP
1 7.64
4.17
5:29
5.89
7.44
8.96
18:97
1.13
19.00
Visual prompts
FPL
67.03k.05
65.50%..1
71.944.00
65. 14..44.
62.24p.22
67 67.85p.00
47 47.601.09
47.69c.4z 48
52.43o.00
IFPL
68.69
66.12n.4s
76.9 16.91..0
67. 11
62 93,
73 73.53g.00
51.65u.79
50.34
57. 06
GRIP
67.95
63.09
77.18..0
71.22,,7
68.433..1
82.19p.00
54.57
50.51g.9
62.78.
A IFPL
t 1.06
0.38
1 4.97
1.97
1 0.09
5.568
t 1.05
2.65
5.433
GRIP
1 0.92
3.41
1. 5.24
6.08
7 6.19
1 14.34
1 6.97
2.82
10.35
Multimodal prompts
FPL
72.5
68.26).38
77.476.0
62.841.0
(2.322
71 71-133..0
13.712.19
44.85
54.86
IFPL
73.14
65.39
81.47%.00
70.60
63 63.699..3
16 16.044.55
53.211.21
47.59 .os
43.17e
GRIP
74.562.02
64.82,.s
82.01o.on
73.78
69.37a.s1
82.17
54.07
47.376.20
63.42..on
IFPL A GRIP
f 1.91
0.13
1 4.00
7.76
1 1.37
4 25.30
9.5
2.2.4
11.60
1 2.02
10.41
1 4.51
1 10.81
7 7.05
1 10.71
10.36
1.522
8,50
Table 2: For each learning paradigm, we compare FPL., IFPL., and GRIP on Flowers 102, RESICS45, and DTD, for all the learning settings SSI., UL., TRZSL. We average across 5 runs and report the standard deviation. A METHOD is the difference between the accuracy of FPL and METHOD.
Transductive zero-shot learning effectively transfers knowledge In the TRZSL setting, GRIP improves over CLIP and the baselines by a large margin (Table M. Figure 2 displays the balance of seen and unseen classes of each method alongside its accuracy. The class balance is (aCCunseen aCCseen) //CCaeens where values close to zero indicate a good balance, negative values indicate better accuracies for classes, and positive values indicate better accuracies for unseen classes. Methods employing iterative usage of pseudolabels maintain a good balance, as opposed CoOp/VPT and FPL. This balance in accuracy is likely a combined effect of the quality of the pseudolabels and the transfer of knowledge from the seen to the unseen classes. The latter point is significant because it implies that even if we only possess unlabeled data for specific tar-
landions
c..
% unabcled cota
of unfateled data
torstions
nerations
Amany
20
23 30 20
% www.tblddddeoo caln
35 unfnbeled data
berstions
Berations
32
1.0
Aecumcy
get task, we can still use labeled data from related classes (30] within the same domain to enhance CLIP's performance.
20 to 15 110
1s unbbeledddoo cata
There is trade-off between quality and quan- tity of pseudolabels Table[]show the perfor- mance of CLIP employing prompts learned with different training strategies, all leveraging pseu- dolabels (Section 3.3) Iterative strategies are more effective than FPL which, similarly to (15), use static set of a few pseudolabels for one iteration.
(a) Flowers102
(b) RESICS45
Evolution of pseudolabels accuracy during training. The rows refer to SSL, UL, and TRZSL, in order. IFPL. refers to the top x-axis, while CLIP and GRIP the bottom.
Figure 3:
On Flowers 102, RESICS45, and DTD, IFPL improves on average FPL by 5.6 points in SSL, 1.7 in UL, and 5.6 in TRZSL. GRIP boosts the performance even more by average 7.8 points in SSL, 3.1 in UL, and 9.7 in TRZSL. Results on the other tasks are comparable or larger and we report them in Appendix[A.3 due space constraints.
Figure 3 shows the progression of pseudolabels quality for the iterative learning of textual prompts. IFPL maintains fixed set of 16 pseudolabels, improving their quality with each iteration (top x-axis).
8

---

On the other hand, GRIP and CLIP expand pseudolabels by incorporating additional decile of unlabeled data in each iteration (bottom x-axis), Initially, GRIP maintains accuracy, but as it nears completion the quality tends to decrease, while a larger dataset with good-quality pseudolabels becomes available.
Comparing GRIP and CLIP, GRIP's expanded pseudolabels exhibit superior quality and performs bet- ter (Table D. Even though IFPL's pseudolabel accuracy surpasses GRIP in the final iteration. GRIP's overall performance remains better due to training on a larger number of pseudolabels (Table 2 This suggests that numerous, slightly noisier pseudolabels can yield better results, highlighting a trade-off and offering insights for future approaches.
GRIP benefits adaptation even for larger im-
Textual prompts
age encoders We measure how much the ef- fect of the iterative strategies changes if we con- sider larger pre-trained image encoder. In Ta- ble 3] we report the average improvements of GRIP on CLIP for Flowers102, RESICS45, and DTD. The magnitude of improvements slightly decreases when using larger image encoder. However, we still see significant benefits for both modalities. The smaller relative improve- ments with respect to smaller visual encoders align with our expectations. Larger encoders possess a stronger base knowledge, making it relatively more challenging attain further im- provements on top of it. In Table I0 we break
SSL
UL
TRZSI
Aeg. CLIP (VIT-B/2)) Avg. CLIP (VIT-L/14)
17.461283
8.366 as
23.772.51
15 15.85
8. 8.16
19.96g
Visual prompts
SSL
UL
TRZSL
Avg. CLIP (VIT-B/32) Avg. CLIP (VIT-L/14)
10. 78.
6.88
20.28
7.61
4.89-
16. 16.14
Table 3:
Average improvement of GRIP with dif- ferent backbones on Flowers 102, RESICS45, and DTD. A CLIP is the difference between the accu- racy of GRIP and CLIP. Alongside the average, we provide the minimum improvement across tasks.
down the accuracy of CLIP with different backbones. The performance of the larger backbonn is higher, indicating higher quality pseudolabels.
4.2 The Robin Hood effect
Although training models with pseudolabels can lead to good performance, it can also result in biased predictions and gen- crate disparate impacts on sub- uub-populations, i.c., the "Matthew effect" [49 8). Particularly, the use of pscudolabels can lead to improved performance in well-behaved (high accuracy) classes but can cause stagnation or decreased performance in poorly behaved (low accuracy) classes. As we explore the use of pseu- dolabels, we investigate how the accuracy of the analyzed ap- proaches distributes across classes. Figure 4 show an opposite scenario from typical SSL. The solid line represents the sorted per-class accuracies of CLIP. The arrows indicate the per-class accuracies of GRIP. For all learning paradigms, the iterative training strategies increase the accuracy of classes where CLIP not proficient, while maintaining or decreasing the accuracy of initially well-behaved classes. This effect, which we call the "Robin Hood effect," is very interesting because shows how CLIP can mitigate its own bias toward certain classes by learning from itself.
SSL
SSL
0.5
0.5
40
40
UL
UL
1.0
1.0
CUP
CUP
0.0
0.0
20
TRZSL
TRZSL
0.5
CUP
CUP
20
40
20
40
To understand the roots of the Robin Hood effect, we examine two factors: (1) the role of pseudolabels generated by CLIP, and (2) the role of prompt tuning. To disentangle these factors, we explore the variation per-class accuracy of basic linear classifier trained on CLIP's ViT-B/32 image representation.
(a) FPL
(b) GRIP
Figure 4: Improvements of FPL and GRIP on CLIP's per-class accu- racies (RESICS45). The x-axis is the ranked class index, while y-axis the accuracy.
"Second generation" pseudolabels are good treatment for class disparity We train the linear classifier in the SSL setting on 2 labeled examples per class and pseudolabels. The pseudolabels are obtained through conventional methods, where a threshold of .95 is applied, or by using CLIP to generate 16 pscudolabels per class.

---

We find that both approaches yield similar overall accuracies. However, we observe the "Matthew effect" when using the first approach. In contrast, when using CLIP-based pseudolabels, the class disparity of the regressor trained solely on seen classes is reduced. Particularly, we see significant improvement on initially poor classes, together with significant diminishing of the accuracy of well-behaved classes. We observe a clear manifestation of the "Robin Hood effect." We present plots illustrating this effect in Appendix A.4 A.4
Prompt tuning retains the accuracy of already rich classes better than linear probing To evaluate the role of prompt tuning in the "Robin Hood effect," train linear classifier and textual prompts in the UL setting using GRIP's training strategy. Comparing the per-class acurracies of the two approaches, GRIP on prompts shows average improvement of 22.85 points for the poor classes across tasks, along with a slight average decrease of 0.3 points for the rich classes. On the other hand, linear probing determines 14.42 points improvement for the poor classes, but it results in an average decrease 9.39 points in accuracy for the rich classes (Appendix A..4
5 Conclusions
We show that prompt tuning using pseudolabels generated by CLIP itself is a successful approach to enhance CLIP across various learning settings. Training strategies that iteratively refine pseudolabels turn out to be effective ways of leveraging pscudolabeled data. These approaches not only enhance CLIP's accuracy but also mitigate model biases toward certain classes. We hope this work lays solid groundwork for reducing reliance on labeled data when adapting pre-trained vision-language models like CLIP to new tasks.
Limitations The effectiveness of the training strategies examined in this paper depends on both the strategies themselves and the quality of pseudolabels. The latter is particularly crucial. If CLIP performs poorly on task, we may struggle to obtain a reliable set of pseudolabels to begin with, potentially diminishing CLIP's performance. Despite this potential risk, we have not observed any relevant failure of GRIP, even in tasks where CLIP's initial accuracy is extremely low (such as FGVCAircraft). Also, the pseudolabeling strategy we adopt involves selecting K pseudolabels per class, which can create a strong assumption about the distribution of the training data we attempt to cover all unlabeled data. During the final iteration, it is as if we assume uniform class balance.
Another important consideration is the efficiency of the explored methods. Repeating the training process multiple times brings impressive improvements at the cost of non-negligible increase of computation time. At each iteration, we generate pseudolabels for the unlabeled data from scratch. While we parallelized the pseudolabeling procedure to cut some of the cost, reducing those for the iterative training presents more significant challenges. We decided to mainly focus on the analysis of qualitative and quantitative effects of pscudolabels in prompt tuning. Future research should address budget constraints and investigate optimal stopping criteria for the iterative process, considering the possibility of reaching a plateau or decreased pseudolabel quality after a certain point, maximize efficiency while maintaining performance.
Acknowledgments and Disclosure of Funding
We are thankful to our reviewers for the fruitful and insightful discussions that contributed to the refinement of the paper. We also thank Reza Esfandiarpoor and Zheng-Xin Yong for the comments on our drafts. This material is based on research sponsored by Defense Advanced Research Projects Agency (DARPA) and Air Force Research Laboratory (AFRL) under agreement number FA8750- 19- 2-1006. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of Defense Advanced Research Projects Agency (DARPA) and Air Force Research Laboratory (AFRL) or the U.S. Government. We gratefully acknowledge support from Google and Cisco. Disclosure: Stephen Bach is an advisor to Snorkel AI, a company that provides software and services for data-centric artificial intelligence.
10

---

References
[1] Stephen Bach, Victor Sanh, Zheng Xin Yong. Albert Webson, Colin Raffel, Nihal V. Nayak. Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Fries, Maged Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir Radev, Mike Tian-jian Jiang, and Alexander Rush. PromptSource: An integrated development environment and repository for natural language prompts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. Association for Computational Linguistics. 2022.
[2] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint htth:2203.177744 2022.
[3] David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring. In International Conference on Learning Representations, 2020.
[4] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems, 2019.
[5] Liu Bo. Qiulei Dong, and Zhanyi Hu. Hardness sampling for self celf-training based transductive zero-shot learning. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages I 16494-166550 16503, 2021.
[6]
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong. Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri. Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isahelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Fric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R'e Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and 2021.
risks of foundation models. ArXiv, 0is/2108.0722
[7] Tom Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
[8] Baixu Chen, Junguang Jiang, Ximei Wang, Pengfei Wan, Jianmin Wang, and Mingsheng Long. Debiased teaf-nraining for semi-supervised learning. In Advances in Neural Information Processing Systems, 2022.
[9] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art Proceedings of the IEEE, 2017.
11

---

[10] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed. and Andrea Vedaldi. Describing textures in the wild. 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2013.
[11] Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 2012.
[12] Yunhe Gao, Xingjian Shi, Yi Zhu, Hongya Wang. Zhiqiang Tang, Xiong Zhou, Mu Li, and Dim- itris Metaxas. Visual prompt tuning for test-time domain adaptation. ArXiv, abs/22 10./2200,,481,, 2022.
[13] Chunjiang Ge, Rui Huang, Mixue Xie, Zihang Lai, Shiji Song, Shuang Li, and Gao Huang. Domain adaptation via prompt learning. ArXiv, abs/2202.06687, 2022.
[14] Patrick Helber, Benjamin Bischke, Andreas R. Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. 2017.
[15] Hao Huang, Jack Chu, and Fangyun Wei. Unsupervised prompt learning for vision- vision-language models. ArXiv, abs/2204.03649,, 2022.
[16] Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondřej Chum. Label propagation for deep semi-supervised learning. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5065-5074, 2019.
[17] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoc V. Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- risionlanguuage representation learning with noisy text supervision. In International Conference on Machine Learning, 2021.
[18) Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV 2022: 17th European Conference on Computer Vision, 2022.
[19] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi-modal prompt learning. ArXiv, abs/2210.03117, 2022.
[20] Dong-Hyun Lee. Pseudo-label: The simple and efficient sumi-supe learning method for deep neural networks. ICML 2013 Workshop Challenges in Representation Learning (WREPL), 2013.
[21] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021.
[22] Will Vine, Benjamin Pikus, Pranav Vishnu Raja, and Fernando Amat. Enabling calibration in the zero-shot inference of large vision-language models. In ICIR 2023 Workshop on Pitfalls of limited data and computation for Trustworthy MI., 2023.
[23] Chunyuan Li, Haotian Liu, Liunian Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang, Ping Jin, Houdong Hu, Zicheng Liu, Yong Jae Lec, and Jianfeng Gao. Elevater: A benchmark and toolkit for evaluating sunguage-augmente visual models. In Advances in Neural Information Processing Systems, 2022.
[24] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the l 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), 2021
[25) Bo Liu, Lihua Hu, Qiulei Dong, and Zhanyi Hu. An iterative co-training transductive framework for zero shot learning. IEEE Transactions Image Processing, 30:6943-6956, 2021.
[26] Subhransu Maji, Esa Rahtu, Juho Kannala, Mathew B. Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. ArXiv, abs/1306.5151, 2013.
12

---

[27] Shu Manli, Nie Weili. Huang De-An, Yu Zhiding, Goldstein Tom, Anandkumar Anima, and Xiao Chaowei. Test-time prompt tuning for zero-shot generalization in -ainulla models. In NewrlPS, 2022.
[28] Nihal V. Nayak, Peilin Yu, and Stephen Bach. Learning to compose soft prompts for composi- tional zero-shot learning. In The Eleventh International Conference on Learning Representa- tions, 2023.
[29] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing. 2008
[30] Wasu Piriyakulkij. Cristina Menghini, Ross Briden, Nihal V. Nayak, Jeffrey Zhu, Elahch Raisi, and Stephen H. Bach. TAGLETS: A system for automatic semi-supervised learning with auxiliary data. In Conference on Machine Learning and Systems (MLSys), 2022.
[31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional Conference on Machine Learning, 2021.
[32] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Iya Sutskever. Language models are unsupervised multitask learners. 2019.
[33] Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Techan. Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022.
[34] Sheng Shen, Shijia Yang, Tianjun Zhang, Bohan Zhai, Joseph E. Gonzalez, Kurt Keutzer, and Trevor Darrell. Multitask vision-language prompt tuning. arXiv preprint arX1v:2211.11720, 2022.
[35] Weiwei Shi, Yihong Gong, C. Ding, Zhiheng Ma, Xiaoyu Tao, and Nanning Zheng. Transductive semi-supervised deep learning using min-max features. In European Conference on Computer Vision, 2018.
[36] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. In Advances in Neural Information Processing Systems, 2020.
[37] Ximeng Sun, Ping Hu, and Kate Saenko. Dualcoop: Fast adaptation to multi-label recognition with limited annotations. ArXiv, 2022.
[38] Xudong Wang, Zhi-Li Wu, Long Lian, and Stella X. Yu. Debiased learning from naturally imbalanced pseudo-labels. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[39] Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning - the good, the bad and the ugly. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[40] Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoe Le.. Unsupervised data aug- mentation for consistency training. In Advances in Neural Information Processing Systems, 2020.
[41] Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi-super learning with dynamic thresholding. In International Conference on Machine Learning, 2021.
13

---

[42] Yunlong Yu, Zhong Ji, Xi Li, Jichang Guo, Zhongfei Zhang, Haibin Ling, and Fei Wu. Trans- ductive zero-shot learning with self-training dictionary approach. IEEE Transactions on Cybernetics, 48(10):2908-2) 2919,, 2018.
[43] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel C. F. Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, and Pengchuan Zhang. Florence: A new foundation model for computer 2021.
vision. ArXiv,
[44] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning. ArXiv, abs/2210.07225. 2022.
[45] Bowen Zhang. Yidong Wang. Wenxin Hou, HAO WU, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi- supervsper learning with curriculum pseudo labeling. In Advances in Neural Information Processing Systems, 2021.
[46] X. Zhang, Yusuke Iwasawa, Yutaka Matsuo, and Shixiang Shane Gu. Domain prompt learning for efficiently adapting clip to unseen domains. 2021.
[47] Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. 2021.
[48] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision-language models. International Journal of Computer Vision, 2021.
[49] Zhaowei Zhu, Tianyi Luo, and Yang Liu. The rich get richer: Disparate impact of semi- supervised learning. In International Conference on Learning Representations, 2022.
14

---

A Appendix
We include here extra information that supports the results presented in the main body of the paper.
Reproducibility We have provided the code to run the experiments supplementary material for the submission. However, we plan to release it as an open repository upon acceptance.
A.1 Trainable Prompts
Text Prompt Tuning The primary objective of text prompt tuning is to improve the alignment between the class token and the image features extracted by the image encoder. This is achieved by adding learnable vectors, i.e., prefix, before the CLASS token to create contextualized representation. Specifically, the sequence
[V] -.- [V] (CLASS)
fed into the textual encoder, where each vector [V]m (m E ........ has the same dimension as word embeddings, and M is hyperparameter that determines the length of the prefix.
the first work explore continuous prompts for VLMs. Follow-up works have experimented with different training strategies to enhance the generalizability of the learned prompts while preserving the core concept of continuous vector tuning [34 12 27 46
Context Optimization (CoOp) 48
13 37
Tuning the text prefix vector changes the resulting linear weight vectors w. = (p,),, while leaving the image features unchanged. Therefore, text prompt tuning may be most beneficial when image features cell-separated by class but may not be aligned with the corresponding textual prompt. Conversely, text prompt tuning may not be as effective when the image features are poorly separated, in specialized novel domains where CLIP may lack sufficient training data.
Visual Prompt Tuning Instead of tuning the text prompts, one can also tune the inputs of the vision encoder. In this case, a learnalle visual prefix is prepended to the image tokens as input to the image transformer follows:
i [p] [p) [4] [4p
where p represents sequence of K learnable prefix vectors, and [T] [1] the image tokens from the corresponding patches of the input images. The new sequence î is the input to the image encoder é
Visual Prompt Tuning (VPT) was introduced in the context of efficiently adapting pre-true mre-trained vision transformers to downstream tasks [18. However, the approach has since been applied in the context of VLM 34
Whereas text prompt tuning does not alter the image features, visual prompt tuning does. By rearranging the image features within the projection space, VPT has the potential to improve CLIP when the image features are not well separated by class, such as in specialized domains.
Multimodal Prompt Tuning The previous approaches are unimodal, as they either involve modi- fying the text visual input, but never both. This choice may be suboptimal does not allow the flexibility to dynamically adjust both representations on a downstream task. Recently, multimodal prompt tuning has been introduced [4] 19]. We focus on Unified Prompt Tuning (UPT) [44] which essentially learns a tiny neural network to jointly optimize prompts across different modalities. UPT learns set of prompts U is transformed as follows:
UT Uv] E Rddxa with length n, where Ur E RRdxer .Uv E RRd
U' SA(U) LN(U)
Ô =FFFN FFN (LN (U')) + + +UN
where SA is the self-attention operator, LN is the layer normalization operator, and FFN is feed forward network. After transformation, we obtain Ù used text prompt, and Uf is to be used as a visual prompt.
Ur. Uv] E 12dx", such that U is to be
15

---

Num. classes (12)
Num.
Nom. classes (CC)
Size training
Aug. labeled data per class
Size
P0owers102
21:40
RESICS45
27
18
6300
110
25200
FGVC Aireraf
100
62
38
6667
53
3333
MNIST
60000
1696
10000
EuroAAT
27000
2200
DTD
29
18
3760
1880
Table 4: For each dataset report the number of classes, the number of seen and unseen classes in the TRZSL setting, the size of training data (including both labeled and unlabeled data), the average number of labeled examples per class, and the size of the test set which is the same across learning paradigms. We recall that we use the datasets gathered by the recent ELEVATER [23] benchmark for vision-language models.
The author of UPL argue that self-attention allows for beneficial interaction between the two separate modalities, which leads to both separable visual features, and text classifiers that are well-aligned with the corresponding visual features 44
0.11
1330
Prompts initialization We initialize textual and visual prompts from normal distribution of mean 0 and variance 0.02. We note that learn shallow visual prompts by modi- fying only the input to the image encoder. Multimodal prompts are initialized from a uniform distribution. We found that the latter was not working properly for textual and visual prompts.
(a) Flowers
0.11
231
Additional training settings 64.
For training, the batch size is
(h) RESICS45
Additional details about pseudolabels assignment It can happen that if K is too large and the unlabeled dataset is smaller than x C, we cannot assign K samples per class. In this case, can reduce K accordingly. Also, the same sample might get assigned to the pseudolabel lists of different classes. It rarely happens in experiments cases. Hoever, this is characteristic of the pseudolabeling strategy proposed [I5], and it can be object of study for future work motivated by the effectiveness of self-training.
2104
Farbic Calknne
(c) FGiC-Airrraft
A.2 Datasets details
(d) MNIST
We use six datasets from specialized or fine-grained domains. Here we provide a description of each of them. Table 4 we report the details about the number of classes and data available for each dataset. For each dataset, we also show CLIP's prediction distribution over classes Figure 5
Karkec
(c) EuroSAT
Flowers102 29 It is dataset collecting images for 102 flower categories commonly occurring in the United King- dom. For each class we have between 40 and 258 images. Figure 5 shows that CLIP's predictions are skewed toward certain classes, which are predicted more often than what would expect according to the real class distribution on the test set.
sa1
Torted Cara FERK
DTD
RESICS45 This publicly available benchmark for Remote Sensing Image Scene Classification. It collects 45 kind of scenes. Figure
Figure 5: For each dataset we show the distribution of CLIP's predic- tions over classes on the test set. The blue dots represent the true class distribution.
(5) (5b shows that CLIP predicts more often a subset of classes.
16

---

Multimodal prompts
10o
RESICS45
FGVCAireraft
Method
SSL
UL
TRZSI.
SSL.
UL
TRZSL
SSL.
UL
TRZSL.
C1JIP
63.67c.00
63.40,.00
54.48
54.46%
17.58g.00
17.86..00
UPT
68.03
61 06..04
02.84
58.79..0
11.13
15.89
GRIP
74.56..02
61.82
82.01
73.68
39.377
82.17
17.36p
11.730.08
17.85
CLIP
t 110.80
1 1.15
1 18.61
1 19.2
1 14.89
27.71
4 0.22
2.25
4 0.01
A UPT
6.53
20.96
10.84
22.38
6.23
1 1.96
MNIST
EuroSAT
DTD
CLIP
25.10k
20.77a
32.88
30.544
43.244.00
43.45m..0
UPT
64.44,.05
63.50h.
68.85
60.43u.o4
43.712.18
36.91a.04
GRIP
65.94..95
68.18
73.75g.4
60.38..77
61.52..04
95.52
54.07
47.37
63.42o.00
A CLIP UPT
1 40.84
43.08
1 52.988
1 27.5
1 28.61
61.98
1 10.83
1 4.13
19.97
1 2.35
10.16
33.00
1 10.36
26.51
Table 5: For each learning paradigm, we compare the accuracy of GRIP with CLIP zero-shot (ViT- B/32), and UPT. Results are for SSL, UL, and TRZSL on FRAMED. We average the accuracy on 5 seeds and report the standard deviation. A METHOD is the difference between the accuracy of GRIP and METHOD. We note that for UL we can not apply UPL since no labeled data is available.
It describes the fine-grained task of categorizing aircraft. We consider the task of classifying aircrafts into 100 variants. Also for this task, CLIP assigns images to reduced set of classes (Figure
PGVC-Aireraft [26]
MNIST database of handwritten digits. The digits are size-normalized and centered in a fixed-size image. We observe that CLIP never predicts out of 10 classes (Figure 5c).
MNIST [
EuroSAT represents the task of categorizing satellite images of scenes. consists of 10 classes. In Figure 5c we show CLIP's predictions distribution over the classes.
EuroSAT [14
DTD [10] DTD stands for Describable Textures Dataset. It is an evolving collection of textural images in the wild, and it is annotated relying on human-centric attributes, inspired by the perceptual properties textures. The zero-shot CLIPP predictions show the model's bias toward certain classes
(Figure 5f).
A.3 Experiments
In this section, we report tables and plots that complement the results presented in Section 4
The effect of GRIP on multimodal prompts Table 5 the improvements of GRIP on CLIP and Unified Prompt Tuning (UPL) 44]. Similar to the results in Table GRIP consistently improves CLIP with respect to the baselines. The improvements learning, 14.8 in unsupervised learning, and 30.7 in transductive zero-shot learning. While GRIP outperforms UPL by 4.7 in semi-supervisee learning, and 19.5 in transductive zero-shot learning.
CLIP by 18.2 in semi-supervised
Comparison across iterative strategies In Table 6 we report a comparison between FPL and the iterative strategies (IFPL and GRIP) on MNIST, EuroSAT, and FGVC-Aircraft. Results the other tasks can be found in the main body of the paper Section 4.1 While GRIP largely and consistently outperforms FPL by on average 16.7 points in accuracy, IFPL is not robust and it leads to performances that are inferior to FPL by on average 4.4 points in accuracy.
The evolving accuracy of dynamic pseudolabels Frerrettss the evolution of pseudolabels accuracy during training for all datasets, but Flowers 102 and RESICS45 presented in Figure 3 We observe that the accuracy of the pseudolabels characterizes the overall performance of the models reported in Table 6 For instance, IFPL for EuroSAT in the TRZSL setting is highly variable, explaining the low average accuracy of the model on the test set (Table Similarily, for MNIST in the TRZSI we observe that after the first iteration, the pseudolabels get very noisy. When we increase the amount of pseudolabeled data, the accuracy CL.IP not necessarily constant. This because as we increase K, effectively selecting pseudolabels with lower similarities to the
17

---

Textual prompts
MNIST
EuroSAT
FGVCAineraft
Method
SSL
UL
TRZSL
SSL
UL
TRZSL
SSL
UL
TRZSL
FPL
66.061.10
40.033
9.73
62.155
48. 48.961.0
53.70go
20.02..77
17.55c.r 16.62a.07
IFPL
59. 44.44
28.942.06
0.00p.00
61.281.59
26.46a.26
14.36ga.71
18.00o.55
13. 800.7
21.72..17
GRIP
71.78a.99
67.88
74,06.
38.662.84
57.21
92.33
16,98us2
15. 22
26.08
A IFPL
6.92
J 11.00
9.733
0.77
7..00
1399.
2.0.2
2.2.2
1 4.117
GRIP
5.72
27.85
66.33
3.39
1.255
38.63
1 3.04
1.40
1,53
Visual prompts
42.84.6.80
39.622g.s sa
31.82
$2 22.477..3
48.79a.c9
68.6814.74
20.14..20
18.28
16.28o.a 46
IFPL
52. 62.91a.99
37.176.2r
38. 28.21
57.85a.52
32.52.0.00
48, 48.1311.18
18.770.4
16. 36..57
19.296..
GRIP
69.66,.51
68.04, 11
69.54; 31
63.483.09
63.68g.12
96.97o.77
19.43..0
17.51g.61
26.420.30
IFPL A GRIP
10.07
2.45
6.56
5.388
16.27
20.55
1 1.37
1.99
3.001
26.82
28.422
37.72
11.01
14.89
28 29
J 0.71
0.77
10.14
Table 6: For each learning paradigm, we compare FPL. IFPL, and GRIP on MNIST, EurOSAT, and FGVCAircraft We average across 5 runs and report the standard deviation. A METHOD is the difference between the accuracy of FPL and METHOD.
horattans
heration
Ierrations
Itorations
Accuricy
Actureyy
26
50
03
70
90
06
100
20
26
43
50
05
70
90
100
10
26
50
95
90
00
200
20
00
68
06
20
uniabeled
unlaboled
2013
olod
d3ta
Banulions
Danaliess
Itarations
30
ACCIPAY
Rorations
Iterattons
Itorations
Iterattons
Accunry
50
303
20
102
(a) DTD
(b) EuroSAT
MNIST
(d) FGVC-Aireraft
Figure 6: We plot the evolution of
accuracy during training. The rows refer to SSL, UL. and TRZSI, in order. IFPL refers to the top x-axis, while CLIP and GRIP to the bottom.
classes, resulting in a reduction in their accuracy, as shown in the plot. This observation aligns with previous findings in [15].
GRIP performance on transductive zero-shot learning We show how the effectiveness of GRIP consistent over the three random splits of seen and unseen classes which we randomly generated. gathers The splits are reported in T Table & gathers the accuracy of seen and unseen classes, along with the harmonic mean for all three splits using textual prompts. Beyond the consistent improvement induced by GRIP training strategy, we observe that the accuracy of GRIP on the seen classes is often lower than the accuracy of CoOp on the same set of classes. During training, for large lambda, the loss component of unlabeled data (unseen classes) is the first to decrease, while the loss the seen classes reduces later. Thus, we hypothesize that extra training steps might be needed to complete the learning the labeled data.
18

---

A.4 The Robin Hood effect
The Robin Hood effect on all tasks For each dataset, we provide the per-class accuracy distribution of GRIP compared with CLIP, Figure @ The Robin Hood effect characterizes all the tasks. We observe that for GRIP the increase in overall aceuracy corresponds to consistent improvements in the predictions of initially poor classes. By comparing Figure 7 with Figure g we see that GRIP reinforces the Robin Hood effect already visible when using FPL. certain cases.
The importance of good quality pseudolabels to mitigate the Matthew effect in SSL In the SSL setting. we train logistic regression on top the visual feature extracted by CLIP's image encoder (ViT-B/32). In Figure we show the per-class accuracy of the final model trained by combining labeled data with either pseudolabels assigned with the conventional scheme (threshold at .95) or 16 -L.e-ggee pseudolabels. We compare the two distribution with the per-class accuracy of the model trained solely on the few labeled examples per class (2 instances).
The different impact of prompt tuning and linear probing on the Robin Hood effect We investigate if there any difference in the Robin Hood effect when adapting CLIP via prompt tuning or linear probing. We train both relying the iterative training strategy that grows the set of pseudolabels at each iteration by using the top-K K scheme (Section
3}. We consider the UL setting.
Among the set of target classes, we distinguish between poor and rich classes. A class is poor, if CLIP's accuracy on that class is lower than its overall accuracy on the task. Otherwise, the class considered rich. Table 7 the accuracy of the two approaches, and the accuracy on the poor and rich classes, while highlighting the average effect with respect to CLIP. Training with prompt tuning retains more knowledge of the rich classes than linear probing. Prompt tuning reduces the accuracy on the rich classes by on average 0.3 points, while linear probing has an average deterioration of 9,4. the reduction of accuracy of rich classes is average 30 times larger than the reduction observed using prompts. Overall, GRIP works better than linear probing. We note that the lower accuracy of linear probing is characterized by worse ability to correctly predict the rich classes, i.c.. "rich get poorer." " This is surprising, as we would have expected the errors to concentrate on the poor classes compared to CLIP.
SSL
SSL
SSL
SSL
SSL
1.0
1,0
0.5
0.5
0.50
CUP
CLIP
0.0
0.0
0.0
0.00
0.0
UL
UL
UL
UL
1.0
1.0
1.0
3.0
0.75
0.5
0.50
CUP
CLIP
0.0
0.0
0.0
0.00
0.0
100
100
TRZSL
TRZSL
TRZSL
TRZSL
TRZSL
1.0
1.0
1.0
1.0
0.75
0.25
CUP
CUP
CLIP
CLIP
CUP
0.0
0.0
0.0
0.00
0.0
100
100
5
20
(a) Flowers 102
(b) FiVC-Aireraft
(c) MNIST
(d) EuroSAT
(c) DTD
Figure 7: Per-class accuracy of FPL compared to CLIP's per-class accuracy on Flowers 102, FGVC-
Aircraft, MNIST, EuroSAT, DTD. X-axis is the ranked class index, while the y-axis is the accuracy.
19

---

SSL
SSL
SSL
SSL
SSL
1.0
1.0
0.75
0.5
05
0.5
0.5
0.25
CUP
CUP
CLIP
CUP
100
100
5
20
40
UL
UL
UL
UL
UL
0.5
0,5
0.5
0.5
0.25
0.0
100
TRZSL
TRZSL
TRZSL
TRZSL
TRZSL
1.0
0.5
0.5
0.5
0.50
0.5
0.0
0.00
100
(a) Flowers 102
(b) FGVC-Aireraft
(c) MNIST
(d) FuroSAT
(e) DTD
Figure 8: Per-class accuracy of GRIP compared to CLIP's per-class accuracy Flowers 102, FGVC- Aircraft, MNIST, EuroSAT, and DTD. X-axis is the ranked class index, while the y-axis is the accuracy.
(a) Flowers102 (b) RESICS45
(c) FGVCA
(d) MNIST
(e) EuroSAT
(1 DTD
Figure 9: Per-class accuracy of logistic classifier using conventional pseudolabels (first row) and CLIP-based pseudolabels (second row). The solid orange line represents the per-elass accuracy of logistic regression trained on 2-shots per class. X-axis the ranked class index, while the y-axis is the accuracy. We present results for Flowers102, RESICS45, NGVC-Airerafi MNIST, EuroSAT, and DTD, in order.
20

---

Flowers102
RESICS45
FGVC-Aircraft
MNIST
EuroSAT
DTD
Avg.
Linear probe (LP)
41.01
58.79
61.94
50.52
51.37
10.17
GRIP
46.09
70.55
69.84
57.21
67.88
15.22
Rich CLIP
67.81
75.47
85.16
65.26
65.14
45.93
Rich LP
52.87
69.01
79.55
67.53
50.34
29.12
Rich GRIP
56.05
78.81
86.40
71.73
77.84
31.95
A
14.92
6.47
1 5.61
2.26
14.79
16.81
9.39
GRIP
4 11.76
3.33
1.21
6.46
1 12.70
13.98
0.33
Poor CLIP
25.63
35.60
27.98
11.10
3.18
5.35
Poor LP
26.50
42.77
36.25
28.34
56.76
4.77
Poor GRIP
35.03
56.85
42.82
39.88
65.08
6,31
A
1 0.87
7.18
1 8.27
1.7.24
1: 53.58
0.58
1 14.43
A GRIP
1 9.4
1 21.26
1 14.84
1.28.88
1 61.9
0.96
1 22.86
Table 7: For each task we report the overall accuracy of linear probing (LP) and GRIP textual along with the accuracy on poor and rich classes. МETHHD the difference between the accuracy CLIP and METHOD. For overall evaluation of the difference between linear probing and prompt tuning, we report average difference of LP and GRIP with respect to CLIP on poor and rich classes.
21

---

Split 1
Flowers102
RESICS45
FGVCAireraft
Method
U
H
S
U
S
H
CLIP
641.66,,.0
62.56..00
63. Ag.00
54.85g.00
04
541.66y.00
16.27p.ov
19.79b.0
17.86
CoOp
91.52...3
48.352.96
63.2 63.22, no
84.661.01
50.73
63.372.23
34.181.56
16.28..m
21.70a.45
GRIP
90.31u.51
82.57
86.26
82.680.47
79.53u.72
81.076.3
22.25
31.51..0
26.08
A CL.IP
26.05
20.01
22.86
1 27.83
t 25.45
1 26.61
> 5.98
11.72
8.22
A CoOp
J 1.21
34.22
23.04
1.98
28.8
17.7
4. 11.93
15.23
4.38
MNIST
BuroSAT
DTD
CLIP
31.74..00
15.43p.00
20.770.00
22.33g.00
48.30.00
30.5.lo.00
42.5g.00
44.446.
43.45p.00
CoOp GRIP
94.685.64
15.437.75
21.15:g.11
8..91
46.02g.23
58.644..6
69.67
34.813.44
46.32.92
95.13
60.63..44
74.066.29
91.755.6a
92.91
92.330.70
68.26p.69
62.61
65.30
CLIP
1 63.39
45.2
1 53.29
1 69.42
1 44.61
1 61.79
1.27.6
18.17
1-28.5
CoOp
0.45
45.2
52.91
8.84
1 46.89
7 13.69
1.41
1 27.8
1 19.00
Split 2
Flowers102
RESICS45
FGVCAireraft
Method
S
U
H
S
U
H
S
H
CLIP
65.38,.00
60. 60.644.00
62.92p oo
59.5p
47 00
52.55v.00
17.30p.00
18.12,.00
17.70p.00
CoOp
91.8
47.75g.56
62.773
86.54,.0l oz
48.003.01
61.70,.1
33.59
19.57,
24.63
GRIP
88.84o.75
70.932.08
78.86
84.47a.41
84.09
84,28o.ra
22.130.24
28.32o.3
24.84b.05
A CLIP
1 23.46
1 10:29
15.94
1.27 83
1 25.45
1 26.66
> 4.88
0 10.20
7,14
A CoOp
2.96
23.18
1 16.09
2.07
1 36,09
1 22.58
4 11.46
8,75
0.21
MNIST
EuroSAT
DTD
CLIP CoOp GRIP
15.99u.00
39.18p.00
22.71g.0 00
32.47
13.00
32.78v.00
45. 43..000
39.72,.om
42.39
90.613.02
18.77,, 12
30.29:2.a9
86.433.20
47. 47.1611.17
60.53s.42
70.4_.90
32.534.58
44.424.63
95.71
97.50
96.59
91.080.02
92.02n.os
91.550.47
66. 66.699.s3
56.19
60.99..0
A CLIP
85.12
50.76
79.32
08.61
1 58.92
1 18.77
21226
16.47
18.6
A CoOp
6.11
1 77.20
57.19
4.65
1 44.86
1 331.2
3.71
23.66
1 16.57
Split 3
Flowers102
RESICS45
FGVCAircraft
Method
S
U
H
S
U
H
S
U
H
CLIP
68,29).00
57.26g.oo
62. 62.28
56.02
52 02.32..00
541.10g.00
17.55p.o0
17.71o.00
17. 17.63g.00
CoOp
91.52..35
48.35
23..22,..
87.612.17
43. 43.644.97
58. 14.12
37.771.92
16.463.2n
22.77
GRIP
90.090.53
69.002.44
78.13
85.19
75.58
80.07
22.07
28.726.76
24.95
A CLIP
1 21.8
1 11.75
1 15.85
1 29.17
1 23.26
1: 25.97
4.52
$11.0
1 7.32
CoOp
1.43
20.65
14.91
2.42
1 31.94
1 21.93
15.70
12.26
2.18
MNIST
Eu
DTD
CLIP
10.59o.no
46. 46.740.00
17.27o.
41.47a.00
19. 19.600.00
26. 26.62o.n0
45.52a.o0
39.58o.no
42.34n.00
CoOp
80, 80.6g.co
26.312.s8 s8
39.416.61
79.339.a7
43.3812.4 49
55.06s.02
70.53
24.94s.a7
36.63
GRIP
95.8
96.06
95.93
90.57a.13
94.25,.11
92.37
67. 67.28p.74
58.94.
62.81
A CLIP
1 79.81
56.88
73.22
49.1
t 74.65
+ 05.75
11.766
19.36
1 20.47
A CoOp
5.20
77.29
65.64
1 11.24
1 50.87
1 37.31
3.25
34.00
1 26.18
Table 8: In the TRZSL settings, for each dataset and split, we compare the accuracy of GRIP textual with CLIP zero-shot (ViT-B/32), and CoOp. Results show the accuracy on seen (8) and unseen classes (U), and the harmonic mean (H). We average the accuracy on 5 seeds and report the standard deviation. METHOD the difference between the accuracy of GRIP and METHOD.
22

---

Appl
Seia clases (5)
Unson class (21)
persiis shlvltel, piac
flower,
castolaug bells wupplogess spox tiill flewer
pondiss Sagigan lonell pocket odlid.
bearded lis posiet foon fget fly, lasian ose eae foierr
pagle conelioess perasiaz illy
of plant senicet pousts comnnn dankdfie negolla, foogline
bellou Towes gize white onan Iyy, etkdwike sneet willian pades pilos, aljice soo helly
concilia warge
denais aduuian.
boggainvillls ouly liyeed catlen
king protes cycaaee. fitillrs odiftnics pwy, wild gping deversoos, azaflonet mec zrpe aracc pie pentone.
defielll stoed lil,, moiggtt
biteee tale liustall zarrs
eisecr,
moniddadd, momin glar, saan tipp
dazy bokeo
caallos
secrcce, Hads semr. ofGrepetsun vasllowe, zacass, scacs, water aly. Btm spple, losa, I8; colembere, los Hppeaairun baln,
Hac. caration ave poper globe bolk explas murated
penrala, vullionee: tladsbery ubs ire lds, pou
misi
RESICISS
back piics. mendiionn ratwwyy soncce rallay.
stpon, stap exontung
thermal porer soacce, mer arpiow abnd. trdgs.
cheparnl, chunch amsie fimiand stadim.
rendemal cleal, dense
welhand.
gill barlee, industta anda,
mosatiion meston, hachall damond parking let, arrgg task
lsce. matam renseriaal
Iconss coun, commercen ancs minti bome park
everpeos restmnall famiand
19- Sprites Challonge NAI 311-000, 1-A-15 B-37m, 733-200 AMMI Paloo 2000
710-300 757 247-190 747-300 757-300 757- 100 700-320 767-410) 7TT,333, AT10-10 A510-200 A540-5000
DR-40 MIDART CRE-200 TRI 168 Takom M05.
MEL-BL DEC-10, B-TK, Glchal Eopross Gaffenam IV
An-12 Bixiny 7IT, C-E3
540,
DC-9.30
TAT 300, DC A3I0 AUR 42, CRU 300 Hork TT.
DH D#C & DHC.& 1000
Filker 100, ATR--2, PI--2,, 5319, 707 SOIL ASI8, A220, BAE IES, 747.200 ERT 135, 737 800.
DHC 8.300, Dosti 325, E. E-1ML E.15
757 Besdlocul
Couss 172. 300,
Enforas Lesncy (00) Eandialt::
298, MD
737-000.
Gelfareun
737 300, DEC-11, DC A36C, C-7,, 767 BAE 146 200
MD-11, Menollas, Model B200, Sash 2000, Tostado Te-154
MNIST
0,3
1.6,7
IneoSA
vedautaal heildinge communal busiddegs, brun or crabeshinnd Like or usa highoaay or neal ansual zmip lasd. posure losd
perrazett crop lane, rasdennal buidng home or spertrants
hurviul fecked, cntred, hempy specklied tednly, god trod cryusline, medhal, rigieggel ploard beireed perimnuul panslld woflled deitd namrs gaey
Hotdhy
cripect rerrly frockst filly gromed
Ivaryyosmhed insrlaedd, lassliz, nattldd poiday poiee
Spdt
perce
conzfiinct crese, hebop bespenaties nos, quar tntls and peratae comatite fmsilary prape byainnt, mesica seer morkobeed poteattia
play,
vindiirnet suneeser
Inc byy, pretabbue frmee
com popty, sneet villaa.
mui cane flower, great mannreen
pes
deep mangold
mnttdd Haddbary aituczs fragzipan, castleya cuseella, casna liss passas fower, wild pares atcriks
buzereep commen daraktiee.
geneiimn enagge cabis alheitaahs cilifenis popp, mioupemaan beanded int.
gentis, bellone faress zarrs thom apple memisg sloy. ispancee
slote fowz pesastas lily balooun daay talp. lil,
woter ill, Jon,
foughnn gas yollow dallis plic yr prlmrose alow hells anichoko mny ahyyllyydontti
dematicc colurtiiee the rellow magedlia hegoortnn, Stoeali, bloder
petsnis,
grogee pratela
cupiaagon girlen
RESICSAS
rabway statin, stowingg polace, leech, conserrii a.d.
siplase arper mannloon claganal chush
DeCMAais purking bit joddcetial spene
coclaggala Gasdend
ship.
istnnd
baschall Gammedd thenaal poser station industial area.
Josl, temaon Joien welasd, intreetion. lake.
guff course meelow, goord uadk Redd accage tark. circclar Senvard feres, belgge
freeway see
home park overasss
POVC-Aiccah
AS21, ME SIL 237, DC-S, Falon 500, Seab 340, 707 200
737 500.0000 747 300, 757.200, 777 200
F.A-15, DC-A SR-2), DC-3, Saah 2000.
177-300, A30054, A325 A340 200, A340-000.
Folkkr
A310,
737-800, C-4T, Damici 325,
ATR-72 BAE
Cosau
737.
Challseger
337-000
525,
560, CRI
Yik 42, Hank Tİ. Folkar 103, DHC (00, Gulfieuus IV.
DH 42, DRC-1. L. E-170
Model B300, Entluer Legres 600, CRI,000 A330.200 767-400.
E-INI E-15 EMB-120 ERJ BRI 145,
1-19 Heeing 2E,, 732-406 251.3000 151:3000 Beecharati 1903, BAI:: 1-6-360 257-200 1A-28 DIIC 200-22 An-11, A130--00 Cl--Ii 347-200, XIR-22 ASDS, DC-00, 147- 100L A348-500
Gatiuman V. I. I.1011, MD-11, MD--7, AI1-- Monohner, Sprars, limaad, To 155, 1o-154
BauSAT
beuilland or dladhladd dies indestial beiklngss corrrnsiid buildings lake or soa fired penreceet cro laed
anecal land, lightwyy road pestre lndd reédestdd buiMings heres or opanneens
pelia
lnsed
banded Hlader spailee
cotneited
perfocted, polnuee plastl. valled, besiei. winilal pailes gaacy nested, grid, stoliei, kustio, miidl coshudeed Siokie. dleqased, gioored, stacaed, filly
cnsked, staned, cysialise, suutieed stapol feekee, wores diprssci. intelacd aecilee natkei, matied
Spit1
cape flwerr back- theyy calliesun abote theess waite war Blys cydtaes,
talp.
cat's yelloe plate firee
sxplaager. fangipan tutnncp common danktion.
pepple concfooe el fre liy, filllars
Dippestron, colurkine, spring moess helfs
Hac sper babeton dayy.
girper, grope byscish. prace
surfower.
the
diger Byy soa lily. megaela, terker me. great meserwon.
oby Tppel cilless.
coreeita, malios moming glny koas porps carleya splira
aflian.
oward lhy. wallfower, petiais. paace, pearpeeim bidlop yelow fallis
syple
mortgrd hiiwar
bellor
eeglish marigeki, king posis, andea fraginn wasremes
jepersso animers sirdfooor. pasnis mos, posaic
Madhurry ilkk harded lnss duadd mesion sour. orange fablis
My
parras ahhebtsh
Boarr
beegsmiflls becralla, Narkyt flsor,
RESICSAS
salwey, parking lit welaail reedow, bullo
sipent beech, dig.
loree pok strege duntral power
bridge.
dhaporal dunch spuse
hochall dummnd, crzela famiasss butkchal cam attrahbo commanic SEEa. milaay statie, tames. frreet restangsiar tentand
rendennal.
doien, sesniccount Inceway
goll counse, groand trac fits, intrrcction overpeas
AX-12, 737-00, 1-16A-1, BAE: 166-300 MD-30, 1-T Carhnooo DV, DS--DD 737-90) 71T-300
713-200 71T-690, 192-300
Boongg 717, 347-105 Sah Czorra
307-200 757-396 155-900,
IAID-125 MIX-1, 207-30 Cruess SAL, A2T8--00L E-195, 773-50L Takler SL ATT8-12
A390-2200 A340--00 410-000
BAE Ht-3300 booker 26, Fhicce 900
ATR-4E, C-UTI Caunn 218, CRA-70, CRI 4000
Paloce 2000 Spaifm, A340-200 UX-S A340-1001 Boocherft
DCA DC-S DCC-630 DEI-K, IIK-A DILC-6-30 EMBB-120
Typhoss,
Lopacy
PN-3X Mil-n,, Yak-2, DRLL- 231--90 ATRIL Modil 11200 EXU 125, SM-- 237-300
Gamal Exprrro 11-15
201-320 DC-10, Domia 526, A30084
L- Memiieer Seah 2201 Temuda, 15-154
BauSAT
flee hightsy modd pesnae lasd, penreseet cno land, frost restecti bridings or bemes opatnentss
aceoed coop land oike
bouilaad or shndiandd indusiial bulldigg conreercid buildngs
plazed polas- Jeted sprnkled goowel, luital, bronked ingoggul spinaled hosted sorfled cmnhhaceed heMlyy eroon deard pornes. worth trikked tred pateled, laztler, ructted amaindd sady, enaiee
wisklsl, bosoyonded dispecid.
bixalo, stäted, colnciited.
cractad, orpedt cryooiliee sarty thoms fedeed windt inlly geary. geal. evertood, medert packyy permeard
Table 9: For each dataset, we report the class names of seen and unseen classes in each of the splits used for TRZSL.
23

---

Textual prompts
Flowers102
RESICS45
DTD
Method
SSL
UL
TRZSL
SSL
UL
TRZSL
SSL
UL
TRZSL
CLIP (ViT-B/32)
63.67
63.67
63.40
54.48
54.48
54.46
43.24
43.24
43.45
GRIP (ViT-B/32)
83.60
69.84
86.26
74.11
70.55
81.07
56.07
46.09
65.30
CLIP (ViT-L/14)
73.98
73.98
73.05
62.67
62.67
62.13
52.45
52.45
51.61
GRIP (VIT-L/14)
94.21
82.33
96.18
81.53
76.86
86.88
60.91
54.40
64.92
Visual prompts
CLIP (ViT-B/32)
63.67
63.67
63.40
54.48
54.48
54.46
43.24
43.24
43.45
GRIP (ViT-B/32)
67.95
63.09
77.18
71.22
68.43
82.19
54.57
50.51
62.78
CLIP (VIT-L/14)
73.98
73.98
73.05
62.67
62.67
62.13
52.45
52.45
51.61
GRIP (ViT-L/14)
78.68
73.50
85.85
77.53
76.00
86.63
55.72
54.27
62.74
Table 10: Performance of CLIP and GRIP with different backbones Flowers102, RESICS45, and DTD, for all the learning settings SSL, UL. TRZSL.
24


Reviews:
[{"review_id": "jJUGIi0F7P", "reviewer": "Reviewer_Qe43", "summary": "This paper explores the concept of prompt tuning in the context of limited labeled data. The authors propose a unified objective function that encompasses three different learning paradigms, and investigate three distinct training strategies for leveraging pseudolabels. The experimental results on six datasets demonstrate the effectiveness of the proposed GRIP strategy. The findings of this study highlight the value of utilizing pseudolabels generated by CLIP itself to enhance its performance in diverse learning scenarios.", "questions": "* I am little confused that does `line 217` means reinitializing the prompts at the begining of each iteration? Or does it mean only reinitializing the set of pseudolabels while the prompts are kept?\n* Is the process of pseudolabeling done online or offline? If it is done offline, is it time-consuming when there is a large amount of unlabeled data?\n* Are the abbreviations TZSL mentioned in `line 182` and TRZSL mentioned later referring to the same concept? This might be a bit confusing.\n* The experimental results in Table 1 show significant improvements compared to CLIP or other baselines. However, there is a slight decrease in some cases. Perhaps the authors could try to explain the reasons behind this? For example, it could be related to the dataset or other factors.\n* I think there might be a typo error in `line 290`. It should be Table 2 instead of Table 5.\n* Have you validate the effectiveness on datasets with more categories (e.g, ImageNet-1K), because the quality of pseudo labels might decrease when the category space get larger.", "limitations": "I think the authors have adequately discussed the limitations of their research.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* this paper is well-written with a clear structure, making it easy to understand\n* this paper extensively explores a board design space, including the prompt modality, the learning paradigm and the training strategy\n* the authors conduct extensive expriments to verify the effectiveness of this method", "weaknesses": "* The writing of the article is not clear in some details, such as the typo error in line 290 and inconsistencies in abbreviations.\n* please see questions listed below", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. Below we address all your questions.\n\n\n> Does line 217 means reinitializing the prompts at the beginning of each iteration? Or does it mean only reinitializing the set of pseudolabels while the prompts are kept?”\n\n\nThat’s correct. In line 217, we say that after each iteration we recompute the pseudolabels and then reinitialize the prompts (both in IFPL and GRIP). The reason behind this choice is to avoid accumulating errors from previous iterations.\n\n> Is the process of pseudolabeling done online or offline? If it is done offline, is it time-consuming when there is a large amount of unlabeled data?\n\nAt each iteration, we generate pseudolabels for the unlabeled data from scratch. As the reviewer points out, the computation time becomes non-negligible with large amounts of unlabeled data.  We mitigated this issue via parallelization. We did not explore more sophisticated solutions, which we believe could become a topic of interest for future research, given the significant gain in performance obtained using pseudolabels dynamically. \n\nWe thank the reviewer for raising this important question. We will add a brief discussion to the limitations section.\n\n> The experimental results in Table 1 show significant improvements compared to CLIP or other baselines. However, there is a slight decrease in some cases. Perhaps the authors could try to explain the reasons behind this? \n\nThe few cases of slight decreases in performance in Table 1 happen for different reasons. For EuroSAT (SSL), the prompt modality in use matters. Indeed, adapting textual prompts with just a few labeled examples (CoOp) is enough to learn as much as we are able to learn with GRIP. This does not happens for visual prompts. The prompt modality has an impact also for Flowers102, indeed under the same settings, using textual prompts bring significant improvement. For FGVCAircraft, the initial set of pseudolabels has very low accuracy for SSL and UL (see Figure 6 in the Appendix) thus the learning process might become ineffective or even damage performance. \n\n> Have you validate the effectiveness on datasets with more categories (e.g, ImageNet-1K), because the quality of pseudo labels might decrease when the category space get larger.\n\nIn our paper, we focused on datasets where CLIP performed poorly, primarily due to domain shift with the training data, as well as domain-specific tasks [31, Section 3.1.5, and Figure 5]. Consequently, we excluded larger datasets such as ImageNet-1K, which were either too similar to CLIP's training data distribution or represented very general domains. Exploring datasets with more categories could be an interesting avenue for future research."}, {"review_id": "hRzLnMXuuO", "reviewer": "Reviewer_1u3c", "summary": "The paper explores the use of CLIP for pseudo-labeling for various tasks such as SSL and on various datasets. Overall - while not being very surprising - the results can clearly outperform prior work (that is not using CLIP it has to be said) - and thus showing the potential of CLIP for such tasks.", "questions": "Overall the paper has a clear point and it does a good job making that point reasonably clear. As said above, after reading the paper I am not surprised that this can work, but I am left without an answer what the limits are of CLIP pseudo-labeling, \n\n", "limitations": "limitation discussion ok for me - except the point of the limits of CLIP pseudolabeliing as mentioned above", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Clearly, the authors show what they set out to do: CLIP pseudo-labeling can improve via pseudo-labeling on a variety of tasks and datasets. That is indeed nice to see.\n\nThe positive experiments are good - doing this for a variety of tasks and on a variety of datasets. \n", "weaknesses": "While the improvements are good - they are not surprising either. CLIP is trained on a wide variety of data and thus it is clear that pseudo-labeling using CLIP should help in many cases\n\nIn my view the main weakness of the paper is that it is not showing the limits of CLIP pseudo-labeling - while most of us do not know which exact data is used to train CLIP - the datasets used seem still reasonably close. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. Below we address all your questions.\n\n> The results can clearly outperform prior work (that is not using CLIP it has to be said) - and thus showing the potential of CLIP for such tasks.\n\nWe would like to clarify that  the baselines and comparisons described in LL237-242 are using CLIP. The wide range in performances show the benefits of pseudolabeling and the importance of the choices within the design space we explore.\n\n> While the improvements are good - they are not surprising either. CLIP is trained on a wide variety of data and thus it is clear that pseudo-labeling using CLIP should help in many cases\n\nBecause of the remarkable zero-shot ability of CLIP, we agree about the possibility of getting sufficiently good pseudolabels. However, we respectfully disagree about the clear benefit we can derive from them. Indeed, in our paper we show that the naive and static usage of top-K pseudolabels per-class, as proposed in [15], does not fully unlock their potential. On the contrary, dynamic training strategies show good improvements.\n\n> In my view the main weakness of the paper is that it is not showing the limits of CLIP pseudo-labeling while most of us do not know which exact data is used to train CLIP - the datasets used seem still reasonably close.\n\nWe agree that exploring the limitations of a method is also important. In this work, we were surprised to find that pseudolabeling can be effective for improving CLIP on such a wide range of tasks. We ran our experiments on domain-specific tasks where CLIP exhibited poor performance, primarily attributed to domain shift issues, as highlighted by CLIP's authors [31, Section 3.1.5, and Figure 5]. It was surprising to see that pseudolabels can bring improvements even in cases where CLIP has low baseline performance, e.g., EuroSAT, DTD, and FGVCAircraft.\n\nWe know that pseudolabeling will fail if CLIP’s initial performance is not much better than random guessing or adversarially biased. We have discussed this in the paper’s limitations section. But given the range of datasets and design choices already considered, we had to leave searching for additional datasets on which it fails for future work. Perhaps datasets with very different sensors, like medical imaging, would be even more challenging."}, {"review_id": "Yjk4dokpxj", "reviewer": "Reviewer_kGE6", "summary": "In this paper authors extend empirical study of self-training for the case when pseudo-labels are generated by models (CLIP) in zero-shot regime (models are trained on unlabeled data with respect to a downstream task, but can be used for zero-shot prediction for the downstream task). Authors investigate self-training in case of visual-language models (CLIP) with tuning visual/text/visual-text prompts with pseudo-labels generated in the zero-shot regime by CLIP model. They formulate also different training regimes (unsupervised, semi-supervised, transductive zero-shot) in terms of simpler supervised training where part of data (or all) just has pseudo-labels. Also it is investigated how we could bootstrap further performance in self-training doing rounds of training while updating the pseudo-labels. In the end authors show that self-training is effective and improve across different settings and datasets. Interesting results are obtained in terms of debiasing original CLIP model when self-training is done afterwards.", "questions": "- lines 50-53: I would not agree that VLM are trained on unlabeled data. We do mining of audio-visual pairs, so in that sense it is really unlabeled. On the other hand, yes, downstream task is different and we can use zero-shot inference to generate pseudo-label. However one could consider that task of pairing text and audio is more general than pairing class name and image. I would be here more concrete in phrasing.\n- There are a lot of repetitions of the same sentences / phrases along the text, e.g. lines 60-69 (other - lines 134-144) appear couple of times. I would reorganize the text to refer to the previous parts rather than repeat almost same ways in introduction / results / sections.\n- typo line 96: \"such as such as\"\n- lines 179 and 187 - selection of $gamma$ and $\\alpha$ seems to be equivalent providing 1:1 balancing between labeled / unlabeled data. I don't get why then expression are made different if in the end mathematically optimization is the same.\n- IFPL: what happens if for some class we don't have K samples? say no any examples are predicted as class 0 for MNIST, what happens then?\n- How about IFPL variant but when we increase K over time, still taking top only pseudo-labels? This is I assume is different from the GRIP. \n- It is not clear if in GRIP classes for pseudo-labels are still balanced, and also how do we select which part of unlabeled set to take? Do we take every iteration top pseudo-labels increasing the number of them or we just increase randomly the data size and take whatever pseudo-labels will be there? Do we still re-initialize prompt every iteration here? Do we reuse or regenerate pseudo-labels for the part of data used in previous iteration here?\n- Why we don't train continuously prompt when we re-generate pseudo-labels? Why do we not fine-tune the whole model also but only prompting (if for efficiency purpose - ok, but interesting to see if we need to finetune the whole model)?\n- lines 308-311 - this is actually known fact. E.g. for fixmatch/remixmatch we do different augmentations, and they are the key to make self-training to work, so particular type of noise in data/labels make self-training to work.\n- Figure 3 (and similar in Appendix) is not clear. I got what is x-axis only after reading it several times. Either better notation or better caption is needed. Maybe \"refers to the top x-axis (number of iterations) while .. to the bottom a-axis (amount of unlabeled data)\".\n- I still don't get why GRIP over iterations can become worse than IFPL. Also maybe it is worth to have ablation where we do IFPL but for every iteration K is increased so that we increase amount of unlabeled data involved though we do this over the most confident every time still.\n- Robin Hood effect: is it because we do balancing of classes in self-training? I guess here this plays a huge role, assuming that some classes are unrepresented in CLIP pretraining + we know that in self-training different classes have different pace of learning, so that balancing can resolve issues on under representative classes or hard ones.\n- line 345 - what is \"conventional pseudo-labeling\"?\n- Table 5: typo - UPL -> UPT\n- lines 664-665 - is it because of the CLIP bias itself and the way classes are balanced in fine-tuning?\n- lines 692-693 - maybe not surprising as we balanced examples per class?", "limitations": "Limitations are discussed after conclusion sections. Formulation sounds reasonable to me.\n\n**Update: after rebuttal and discussion with authors the score is update from 6-weak accept to 7-accept and contribution from 2 to 3.**", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- extensive empirical study of self-training for the CLIP-based models and various configurations and settings\n- demonstrating that CLIP-style models can be used as seed model for self-training in variety of settings bringing significant boost for prompt finetuning\n- empirical demonstration how CLIP de-biasing can be done via self-training", "weaknesses": "- absence of any ablations on balancing between labeled / unlabeled data\n- absence of ablation on the number of samples selected for every class in self-training (why K=16? is it important? why balanced across classes?), also it is not clear what is the effect of balanced selection here especially on the de-biasing the CLIP (if de-biasing happens only because of this balancing then it is very obvious/expected result in my opinion)\n- GRIP exact explanation / how it works should be extended in the text (many ambiguities in the current description)\n- Table 1 and other Tables: I don't get why for CLIP TRZSL is similar to SSL and UL while for self-training in any combination TRZSL becomes way better than SSL and UL?\n- Missed discussion on the results that text prompt alone gives superior results compared to text+visual prompt or visual-alone prompt\n\n**Summary** There is no novelty or any impactful design of the self-training algorithm itself, though extensive empirical study is done for zero-shot learners (CLIP) used to pseudo-label data. The particular choice of hyper-parameters (like balancing pseudo-labels per class) demonstrate significant improvement for the prompt fine-tuning task as well as de-biasing original model (latter I found particularly important and interesting).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. Below we address all your questions that space allows. We do not seem to be able to reply to our own rebuttal to answer more. Please reply if you would like us to answer the remaining questions in our own reply.\n\n> Absence of any ablations on balancing between labeled / unlabeled data\n\nWe started experimenting without balancing labeled and unlabeled data. This yielded unsatisfactory results  since the training excessively focused on either of the two sets of data. So, we tried balancing the two. This strategy turned out to robustly work for all the 162 combinations of datasets, prompt modalities, learning settings, and training strategies we explored. We observed impressive quantitative and qualitative performance improvements, while eliminating the need for a hyperparameter search to optimize results. We decided not to confound the conclusions  with balance optimization due to our focus on the design space overviewed in Figure 1.\n\n> Absence of ablation on the number of samples selected for every class in self-training (why K=16? is it important? why balanced across classes?)\n\nWe set K=16 since it is indicated as the optimal K in the previous research on pseudolabeling with CLIP [15]. In general, K is an hyperparameter that that may require optimization in practical cases. We decided to be consistent with the literature and applied this fixed value of K in order to focus on the design space overviewed in Figure 1.\n\nThe balance across classes also derives from [15]. About that, we note that this is an easy and effective way to avoid pseudolabels distribution skewed toward certain classes.\n\n> Table 1 and other Tables: I don't get why for CLIP TRZSL is similar to SSL and UL while for self-training in any combination TRZSL becomes way better than SSL and UL?\n\nCLIP's performance similarity across TRZSL, SSL, and UL is due to evaluation on the same test set. While SSL and UL get the same scores, TRZSL accuracy differs since it is the harmonic mean between seen and unseen class accuracies. This helps recalibrate the overall score, particularly when the model performs poorly on one set of classes.\n\nObserving significantly larger scores for TRZSL compared to SSL and UL is expected.  This difference arises because in TRZSL, a portion of the target classes is provided with labeled data, which contributes to a larger ground truth being available to the model compared to SSL and UL settings.\n\n> Missed discussion on the results that text prompt alone gives superior results compared to text+visual/visual prompt\n\nUsing pseudolabels dynamically is beneficial for each modality. However, determining the clear superiority of one prompt modality over the other is challenging, as it depends on the specific tasks. For example, visual prompts work better for EuroSAT, while textual prompts excel in Flowers102. Despite intuitive explanations (Section 3.1), the scientific consensus remains elusive [44]. Hence, we prefer to emphasize that the dynamic use of pseudolabels consistently improves performance for each prompt modality, without declaring one modality as definitively better than the other.\n\n> The selection of lambda and gamma seems to be equivalent providing 1:1 balancing between labeled / unlabeled data. I don't get why then expression are made different if in the end mathematically optimization is the same.\n\nYes, they are equivalent. We will simplify the presentation and just leave the one weighting the unlabeled data.\n\n> IFPL: what happens if for some class we don't have K samples? say no any examples are predicted as class 0 for MNIST, what happens then?\n\nThe top-K pseudolabeling strategy we adopt, originally studied in [15], consists of (1) computing the similarity scores of each datapoints with classes’ textual prompts, and  (2) select for each class the K datapoints with the highest similarity score to the class. In this way, we always get K pseudolabels per class. Of course, it can happen that if K is too large and the unlabeled dataset is smaller than K*|number of classes|. In this case, we suggest reducing K.\n\n> How about IFPL variant but when we increase K over time, still taking top only pseudo-labels? This is I assume is different from the GRIP.\n\nBased on our understanding, it appears that the variant of IFPL proposed by the reviewer is the same as GRIP. In IFPL, top-K pseudolabels are recomputed at each iteration while keeping K fixed. In contrast, in GRIP, K increases with each iteration and we assign pseudolabels still following the top-K schema.\n\n> It is not clear if in GRIP classes for pseudo-labels are still balanced, and also how do we select which part of unlabeled set to take? [...] Do we reuse or regenerate pseudo-labels for the part of data used in previous iteration here?\n\nGRIP maintains class balance by selecting the top-K samples at each iteration, with K increasing progressively. Similar to IFPL, both prompts and pseudolabels are reinitialized with every iteration, in order to avoid accumulating errors from earlier iterations. In other words, learning progresses from pseudolabels to new prompts to new pseudolabels, and so on.\n\n> Why we don't train continuously prompt when we re-generate pseudo-labels? Why do we not fine-tune the whole model [...]?\n\nWe reinitialized prompts at each iteration to avoid error accumulation from previous steps. The focus of this work is on parameter-efficient adaptation of CLIP, thus we did not try out the use of pseudolabels to fine-tune the entire model. Additionally, it is worth noting that the limited amount of initial training data might not be sufficient to meaningfully fine-tune such a large model.\n\nWe will clarify this point in the revised version of the paper.\n\n> I still don't get why GRIP over iterations can become worse than IFPL. [...].\n\nAs reported in Table 2, GRIP either outperforms or performs comparatively with IFPL. The suggestion of the reviewer is correct and it corresponds to GRIP."}, {"review_id": "eHiuWp7EY7", "reviewer": "Reviewer_sr7c", "summary": "The authors provide compelling evidence that underlines the power of a repetitive prompt-training approach, which leverages CLIP-based pseudo labels. Regardless of the learning model (SSL, TZSL, UL) or the type of prompt (text, visual), this strategy significantly enhances the image classification capabilities of CLIP across multiple settings. Moreover, by using the Top-K pseudo-labeling approach, they ensure a balanced distribution of pseudo-labeled training samples for each class, thereby effectively addressing the inherent bias of CLIP pseudo labels.", "questions": "- In Table 1 and Table 2, there's a noticeable drop in some experimental results under the UL setting. I'm curious as to the cause of this decrease.\n- Within Table 2, the outcomes for the TRZSL learning setting in RESICS45 and DTD, under the \"Multimodal prompts\" category, show a decline. I'm interested in understanding why this is so.\n- In the experimental results presented in Table 3, where \"GRIP benefits adaptation even for larger image encoders,\" there seems to be a reduced rate of improvement when using a larger image encoder. I wonder the reason or analysis behind this observation.", "limitations": "I already pointed out in the weakness part.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Various learning paradigms, including semi-supervised, transductive zero-shot, and unsupervised learning, can all be viewed as unique instances of a single objective function when pseudolabels are used as a form of guidance.\n-  As evidenced by Table 1and Table 2, the experimental results demonstrate a significant boost in performance, indicating that the proposed pseudo-labeling strategy holds considerable merit.", "weaknesses": "The overall structure or layout of the manuscript needs to be further refined.\n- Table 5 (\"There is a trade-off between quality and quantity of pseudolabels\") is mentioned in L290, but Table 5 does not exist in the main manuscript. \n- Table 2 was not mentioned at all in the body text. \n- Since Fig 3 and Fig 5 are too small to be seen properly, efforts are needed to improve readability.\n- The formulas of L157-L158 should be specified as an equation in the form of (1), but it was not.\n\nMethodologically, the proposed approach is too naive. \n- Setting the trade-off parameters in unified objective function (L157-L158), e.g., gamma, and lambda, is too heuristic, and there is lack of the analysis. \n- Selecting the Top-K most confident samples by class is too naive. It would be nice to select a confident sample by improving class diversity through the application of techniques such as information maximization loss [1]. And it would be good to present comparison results with other attempts for class diversity.\n\n[1] Liang, Jian, Dapeng Hu, and Jiashi Feng. \"Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation.\" ICML 2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. Below we address all your questions.\n\nAbout the layout and typos we will address all the points in the final version of the paper.\n\n> Setting the trade-off parameters in unified objective function (L157-L158), e.g., gamma, and lambda, is too heuristic, and there is lack of the analysis.\n\nIndeed the trade-off parameters we defined are heuristic, but they have proven to be effective across all 162 combinations of tasks, prompt modalities, learning paradigms, and training strategies that we extensively investigated.\n\nAlthough having a theoretical analysis is always desirable, it is worth emphasizing that empirically setting hyperparameters to balance the terms of a loss function is the common practice in machine learning. Often, hyperparameter optimization can be resource-intensive, whereas in our approach, equally weighting labeled and pseudolabeled data yields remarkable improvements and obviates the need for an exhaustive hyperparameter search. \n\n> Selecting the Top-K most confident samples by class is too naive. It would be nice to [..] other attempts for class diversity.\n\nWe respectfully disagree in considering simplicity a weakness, especially after observing the quantitative (impressive performance improvement) and qualitative advantages (Robin Hood effect) of using the top-K approach for pseudolabeling. Since we set our goal to explore how to use CLIP’s pseudolabels in a variety of low-resource settings for prompt tuning, we deliberately decided to focus on the design space overviewed in Figure 1. Using more complex procedures for other aspects of the experiments risks confounding the results.\n\n> In Table 1 and Table 2, there's a noticeable drop in some experimental results under the UL setting. I'm curious as to the cause of this decrease.\n\nThe drops in performance under the unsupervised learning (UL) setting typically correspond to a poor initial assignment of pseudolabels, perpetuated through the iterative process. Indeed, in UL pseudolabeled data is the only source of supervision. However, this phenomenon can also be attributed to the prompt modality, as we can observe improvements under the same setting where a prompt of different modality is trained.\n\n> Within Table 2, the outcomes for the TRZSL learning setting in RESICS45 and DTD, under the \"Multimodal prompts\" category, show a decline. I'm interested in understanding why this is so.\n\nWe speculate that during the first training iteration, the learned prompts produce low-quality pseudolabels, leading to suboptimal learning in subsequent iterations. In contrast, GRIP - which increases the set of pseudolabels over time - significantly outperforms the baseline in these cases. \n\nUnderstanding why this happens for multimodal prompts but not for textual or vision prompts is challenging. In the literature, the behavior of different prompt modalities is typically associated with the task’s characteristics [44]. However, the prompt tuning literature lacks a definitive scientific consensus on the effectiveness of one modality over another.\n\n> In the experimental results presented in Table 3, where \"GRIP benefits adaptation even for larger image encoders,\" there seems to be a reduced rate of improvement when using a larger image encoder. I wonder the reason or analysis behind this observation.\n\nThe smaller relative improvements with respect to smaller visual encoders align with our expectations. Larger encoders possess a stronger base knowledge, making it relatively more challenging to attain further improvements on top of it.\n\nIn Table 10 (additional PDF for rebuttal, which we will add to an appendix), we report the accuracy of CLIP with different backbones. To clarify, the performance of the larger backbone is higher, indicating higher quality pseudolabels. Table 3 is included to show that techniques like GRIP generalize across backbones, but it is unsurprising that the gains are a bit smaller with a larger backbone because it is a stronger starting point.\n\n"}, {"review_id": "wMJz893f63", "reviewer": "Reviewer_qyTk", "summary": "This paper is about the exploration of using pseudolabels for VLMs on different downstream tasks. Specifically, the authors experimented CoOp, VPT and UPT with pseudolabels generated from CLIP on semi-supervised learning, transductive zero-shot learning and unsupervised learning. They proposed three training strategies, namely FPL, IFPL and GRIP, where the difference is mainly at whether the pseudolabels are static or dynamic. Experiments on 6 datasets show significant improvement.", "questions": "- The authors stated that the proposed IFPL and GRIP are \"novel and proposed here for the first time for prompt tuning\". However, aren't these two methods similar in spirit to self-training? What are the differences?\n-  According to Fig 3, the accuracy of pseudolabels are only marginally increasing for IFPL, and decreasing for GRIP as the training proceeds. Is it necessary then to train over 10 iterations? What are the detailed improvements for each iteration? \n- Why is it that CLIP's pseudolabel accuracy decreases in Fig 3? Shouldn't zero-shot CLIP remain an approximately constant performance irrespective of the number of iteration and the number of unlabeled data?\n- When using a larger image encoder, why does the performance decrease? If a larger backbone is used, wouldn't the quality of pseudolabels be higher? This further deepens my concerns on the proposed method.", "limitations": "Yes, the authors have addressed the limitations.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Writing is good. The work is presented in a logical way and is easy to follow.\n- Performance improvement is impressing.\n- Experiments are thorough.", "weaknesses": "- Novelty is limited. The use of CLIP's pseudolabels are not new and the proposed training strategies are also widely used in self-training methods. \n- I agree with the authors' statement that \"If CLIP performs poorly on a task, we may struggle to obtain a reliable set of pseudolabels to begin with, potentially diminishing CLIP’s performance.\" Therefore, it is unintuitive why GRIP achieves such a big improvement even on datasets like MNIST where CLIP fails. One explanation I can think of is that the quality of the pseudolabels are drastically increasing as the training proceeds. However, results from Fig 3 seems to show the opposite. I am mainly concerned about this and believe further analysis is needed to explain what is the exact reason for the success of the proposed method.\n- I'm also concerned with the computational cost since a total of 10 iterations are used. Authors should include this when comparing with other methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. Below we address all your questions.\n\n> Novelty is limited. The use of CLIP's pseudolabels are not new and the proposed training strategies are also widely used in self-training methods.\n\nWe respectfully disagree that novelty is limited. Our study centers on conducting a comprehensive exploration of a wide design space that includes prompt modalities, learning paradigms, and training strategies. As shown in Figure 1, this design space has been an underexplored topic in the literature. If there are other papers that have considered these choices in the context of vision-language models like CLIP, we would be grateful for pointers. As also discussed above in the general comment, this exploration leads to novel and useful findings about prompt tuning across a wide range of settings like UL and TRZSL including improved performance and the Robin Hood effect.\n\n> I agree with the authors' statement that \"If CLIP performs poorly on a task, we may struggle to obtain a reliable set of pseudolabels to begin with, potentially diminishing CLIP’s performance.\" Therefore, it is unintuitive [...] I am mainly concerned about this and believe further analysis is needed to explain what is the exact reason for the success of the proposed method.\n\nThe concerns raised by the reviewer about MNIST can be clarified in Figure 6 of the Appendix. As the plots display, for SSL and UL the accuracy of GRIP pseudolabels is pretty high and maintained throughout the iterations. Moreover, its value is close to the accuracy of MNIST reported in Table 1.  As for TRZSL, the accuracy is close to the one of CLIP and in absolute value reflects the overall accuracy reported in Table 1. We explain the fact that GRIP accuracy is higher than CLIP, even with comparable pseudolabels accuracy, with the trade-off between quality and quantity that we extensively cover in Section 4.1, page 7, and the Appendix.\n\n> I'm also concerned with the computational cost since a total of 10 iterations are used. Authors should include this when comparing with other methods.\n\nAnd\n\n> According to Fig 3, the accuracy of pseudolabels are only marginally increasing for IFPL, and decreasing for GRIP as the training proceeds. Is it necessary then to train over 10 iterations? What are the detailed improvements for each iteration?\n\nRepeating the training process multiple times brings impressive improvements at the cost of a non-negligible increase of computation time. While we parallelized the pseudolabeling procedure to cut some of the cost, reducing those for iterative training presents more significant challenges. We decided to mainly focus on the analysis of qualitative and quantitative effects of pseudolabels in prompt tuning, as overviewed in Figure 1.\n\nFuture research should address budget constraints and, as suggested by the reviewer, investigate optimal stopping criteria for the iterative process, considering the possibility of reaching a plateau or decreased pseudolabel quality after a certain point, to maximize efficiency while maintaining performance.\n\nDue to the importance of the topic, we will add a discussion of computational time and cost in the limitations section of the paper.\n\n> The authors stated that the proposed IFPL and GRIP are \"novel and proposed here for the first time for prompt tuning\". However, aren't these two methods similar in spirit to self-training? What are the differences?\n\nIFPL and GRIP are novel in the context of prompt tuning and CLIP-based pseudolabels since they had not been previously investigated. However, it is fair to say that they are similar in spirit to the common self-training approach. The key distinction is the pseudolabel assignment method: using a top-K strategy instead of a confidence threshold. Despite the subtlety of this difference, it significantly impacts the final prediction’s quality. Our experiments demonstrate that prompt tuning with IFPL and GRIP achieves a more equitable distribution of class accuracies.\n\nIf considered misleading, we can replace “novel” with “unexplored”.\n\n> Why is it that CLIP's pseudolabel accuracy decreases in Fig 3? Shouldn't zero-shot CLIP remain an approximately constant performance irrespective of the number of iteration and the number of unlabeled data?\n\nWhen we increase the amount of pseudolabeled data, the accuracy of CLIP is not necessarily supposed to remain constant. As we increase K, we are effectively selecting pseudolabels with lower similarities to the classes, resulting in a reduction in their accuracy, as shown in the plot. This observation aligns with previous findings in [15].\n\n\n> When using a larger image encoder, why does the performance decrease? If a larger backbone is used, wouldn't the quality of pseudolabels be higher? This further deepens my concerns on the proposed method.\n\nWe would like to clarify that when using a larger encoder, GRIP still substantially improves the performance across all learning paradigms.  The smaller relative improvements with respect to smaller visual encoders align with our expectations. Larger encoders possess a stronger base knowledge, making it relatively more challenging to attain further improvements on top of it.\n\nIn Table 10 (additional PDF for rebuttal, which we will add to an appendix), we report the accuracy of CLIP with different backbones. To clarify, the performance of the larger backbone is higher, indicating higher quality pseudolabels as the reviewer says. Table 3 is included to show that techniques like GRIP generalize across backbones, but it is unsurprising that the gains are a bit smaller with a larger backbone because it is a stronger starting point.\n"}]

Author rebuttal:
We thank all the reviewers for their time and valuable feedback. Addressing your concerns during the discussion phase will significantly enhance the paper.

We clarify common questions here and address your reviews individually below.


### Novelty

Review qyTk expressed concern about the novelty of our training strategies using CLIP-based pseudolabels for prompt tuning. However, our paper's core novelty lies in the extensive exploration of an underexplored design space for prompt tuning with CLIP, which has been generally appreciated by all reviewers.

The design space includes prompt modalities, learning paradigms, and training strategies (Figure 1), that define 27 possible paths to explore, among which only 1 was previously investigated in literature [15]. We applied the same versatile training strategies to all the settings, based on the observations that (1) VLMs' zero-shot capabilities extend the usability of pseudolabels to any limited-label data scenario, and (2) paradigms such as semi-supervised, transductive zero-shot, and unsupervised learning can all be seen as optimizing the same loss function, by using zero-shot pseudolabels as a source of supervision.

While pseudolabeling with CLIP and pseudolabeling in general are not new, our perspective brings novel and meaningful contributions, particularly in the applicability of pseudolabels and their use to tailor CLIP for specialized domains with limited labeled resources.

### Naivety

While in our analysis we vary the prompt modalities, learning paradigms, and training strategies, we do not vary the rule for assigning pseudolabels and stick to the top-K strategy [15].  Reviewer sr7c criticized the usage of a too-naive rule. Although more sophisticated pseudolabeling and training strategies have been devised, we deliberately decided to stick to simple methods and have a better focus on the the three dimensions of the design space overviewed in Figure 1. It is impressive to see how this simple strategy leads to quantitative (impressive performance improvement) and qualitative advantages (Robin Hood effect), recognized by all reviewers.


### Clarification on training strategies

Reviewers Qe43 and kGE6 suggested to improve the clarity of the descriptions of  iterative strategies (IFPL and GRIP). In the manuscript, we will do that by adding explicit step-by-step explanations. 

For IFPL, we begin by obtaining the top-K pseudolabels for each target class. These pseudolabels are then used to train a new task-specific prompt. After completing the training, we use the acquired prompt to compute the top-K pseudolabels per class again. Subsequently, we reinitialize the prompt and repeat this entire process for a total of I iterations.

As for the GRIP method, it shares similarities with IFPL, but with a key difference. In each iteration, we progressively increase the value of K. Specifically, during the ith iteration, we use K= i/I-th of the unlabeled data to perform the steps in the iterative process.


### Typos

We did our best to proofread the paper before the submission. However, we missed some minor typos which were highlighted by the reviewers, e.g., Table 5 in L290 should be Table 2. We resolved them. Thank you for pointing them out.

Meta-review: This paper proposes to use CLIP for pseudo-labeling in various tasks such as SSL and on various datasets. Concerns were originally raised regarding unclear presentations, missing comparisons and discussion with alternative methods.

The authors provided a rebuttal and after the discussion period, 4 out 5 reviewers vote for acceptance. The AC believes the results are meaningful for future research using CLIP and recommends acceptance, and encourages the authors to revise paper accordingly.
Decision: Accept (poster)