Title: Enhancing CLIP with CLIP: Exploring Pseudolabeling for Limited-Label Prompt Tuning
Venue: NeurIPS 2023
Accepted: yes
Keywords: vision-language models

prompt-tuning

pseudolabels

self-training
Abstract: Fine-tuning vision-language models (VLMs) like CLIP to downstream tasks is often necessary to optimize their performance. However, a major obstacle is the limited availability of labeled data. We study the use of pseudolabels, i.e., heuristic labels for unlabeled data, to enhance CLIP via prompt tuning. Conventional pseudolabeling trains a model on labeled data and then generates labels for unlabeled data. VLMs' zero-shot capabilities enable a ``second generation'' of pseudolabeling approaches that do not require task-specific training on labeled data. By using zero-shot pseudolabels as a source of supervision, we observe that learning paradigms such as semi-supervised, transductive zero-shot, and unsupervised learning can all be seen as optimizing the same loss function. This unified view enables the development of versatile training strategies that are applicable across learning paradigms. We investigate them on image classification tasks where CLIP exhibits limitations, by varying prompt modalities, e.g., textual or visual prompts, and learning paradigms. We find that
(1) unexplored prompt tuning strategies that iteratively refine pseudolabels consistently improve CLIP accuracy, by 19.5 points in semi-supervised learning, by 28.4 points in transductive zero-shot learning, and by 15.2 points in unsupervised learning, and (2) unlike conventional semi-supervised pseudolabeling, which exacerbates model biases toward classes with higher-quality pseudolabels, prompt tuning leads to a more equitable distribution of per-class accuracy. The code to reproduce the experiments is at https://github.com/BatsResearch/menghini-neurips23-code.

Enhancing CLIP with CLIP: Exploring Pseudolabeling for Limited-Label Prompt Tuning

Abstract

Fine-tuning vision-language models (VLMs) like CLIP to downstream tasks is often necessary to optimize their performance. However, a major obstacle is the limited availability of labeled data. We study the use of pseudolabels, i.e., heuristic labels for unlabeled data, to enhance CLIP via prompt tuning. Conventional pseudolabeling trains a model on labeled data and then generates labels for unlabeled data. VLMs' zero-shot capabilities enable a 'second generation' of pseudolabeling approaches that do not require task-specific training on labeled data. By using zero-shot pseudolabels as a source of supervision, we observe that learning paradigms such as semi-supervised, transductive zero-shot, and unsupervised learning can all be seen as optimizing the same loss function. This unified view enables the development of versatile training strategies that are applicable across learning paradigms. We investigate them on image classification tasks where CLIP exhibits limitations, by varying prompt modalities, e.g., textual or visual prompts, and learning paradigms. We find that (1) unexplored prompt tuning strategies that iteratively refine pseudolabels consistently improve CLIP accuracy, by 19.5 points in semi-supervised learning, by 28.4 points in transductive zero-shot learning, and by 15.2 points in unsupervised learning, and (2) unlike conventional semi-supervised pseudolabeling, which exacerbates model biases toward classes with higher-quality pseudolabels, prompt tuning leads to a more equitable distribution of per-class accuracy. The code to reproduce the experiments is at BatsResearch/menghini-neurips23-code.

1 Introduction

Large pre-trained vision-language models (VLMs) [31, 43, 17] achieve remarkable accuracy without task-specific training but still require adaptation for optimal performance. Prompt-tuning [13, 18] is an approach to efficiently enhance VLMs performance on downstream tasks by learning inputs to the model. While learning prompts with a few labeled data can yield significant improvements [48, 2], a broader range of learning settings such as semi-supervised, transductive zero-shot, and unsupervised learning are still underexplored. All of these settings share access to unlabeled data, and the versatile zero-shot classification abilities of VLMs make pseudolabeling a natural approach to leveraging it. This paper investigates how the use of out-of-the-box pseudolabels assigned by CLIP can contribute to improving CLIP's own performance. To this end, we conduct an extensive exploration of learning scenarios by varying prompt modalities, learning paradigms, and training strategies. We present empirical evidence showcasing the effectiveness of iterative prompt-training strategies that leverage CLIP-based pseudolabels, regardless of learning paradigms and prompt modalities, resulting in significant improvements in CLIP's image classification performance across different settings.

Pseudolabels are heuristic labels assigned by a model to unlabeled data, which are leveraged to further train the model [20]. Successful training with pseudolabels relies on two factors: the quality

Prompt

Modalities

Textual Prompts (CoOp)

Visual Prompts (VPT)

Multimodal Prompts (UPT)

Learning

Paradigms

Semi-supervised learning (SSL)

Transductive zero-shot learning

(TRZSL)

Unsupervised learning (UL)

Training

Strategies

Few-Pseudolabels (FPL)

Iterative Few-Pseudolabels (IFPL)

Pseudolabels (GRIP)

of the labels and how they are used during training. To address the first, conventional methods assign labels to instances with high-confidence predictions [36]. For pseudolabeling using CLIP, Huang et al. propose to select the most confident samples for each class [15], mitigating CLIP's bias [38] and miscalibration [22] (see Section 3). To assign pseudolabels, we rely on this approach and address the second point by exploring how to make the best use of them. We design a broad space of analysis considering three dimensions: prompt modalities, which are the model inputs we learn; learning paradigms, which define the data we have available; and training strategies, which describe the process used to optimize performance (Figure 1).

Research on prompt tuning has demonstrated that training strategies used for learning prompts in one modality can be transferred to learning prompts in a different modality. For instance, Visual Prompt Tuning [18] was originally designed to effectively fine-tune large vision models but can be adapted to efficiently fine-tune CLIP using the same training strategy as standard textual prompt tuning [48, 34, 44]. On the contrary, different learning paradigms with limited labeled data typically require distinct approaches specifically tailored to extract information from the available data [27, 12]. However, we observe that this changes by using VLM's generated pseudolabels. Unlike conventional pseudolabeling approaches that bootstrap off labeled data and are used as semi-supervised learning techniques [36, 3, 40], VLMs can generate pseudolabels in any learning setting. This offers a significant advantage, expanding the scope of pseudolabeling beyond semi-supervised learning, and making it a promising approach for other settings, such as transductive zero-shot and unsupervised learning. By using CLIP-based pseudolabels as a source of supervision, we can view these settings as optimizing the same loss function, which is simply a weighted sum of the errors on labeled data, if available, and pseudolabeled data. Given that we can express different settings as the same problem, we can propose training strategies, i.e., the way of using pseudolabels, that suit them all.

By standardizing the training strategies across various prompt modalities and learning settings, we can conduct experiments on different applications of pseudolabels for various combinations of prompt modalities, learning paradigms, and training strategies, as illustrated in Figure 1. To the best of our knowledge, only one potential path has been explored thus far; specifically, fine-tuning textual prompts in an unsupervised learning context using a few pseudolabels [15]. Rather than relying on a fixed set of pseudolabels, we propose iterative training techniques that allow for the ongoing refinement and expansion of the pool of pseudolabeled data used during training. With each iteration, we progressively enhance CLIP's pseudolabeling ability, allowing us to extend the set of pseudolabeled data while maintaining the high quality of the initial pseudolabels.

We conduct experiments on six tasks where CLIP has been observed to underperform [31], such as satellite-image classification, flower-species identification, and texture-image recognition, among others. Our findings reveal that iterative approaches effectively fine-tune prompts irrespective of their modality and learning paradigms. Recent studies have identified the 'Matthew effect' as a potential issue for semi-supervised models that use pseudolabels [49, 38]. This phenomenon causes models to perform well on classes with accurate pseudolabels but poorly on those with inaccurate ones, thereby reinforcing the model's original bias towards certain classes. Our analysis reveals that using pseudolabels generated by CLIP for prompt-tuning with iterative strategies not only improves CLIP's overall performance but also corrects its natural bias towards certain classes.

We summarize the main takeaways of our work:

General purpose zero-shot learners used as general purpose pseudolabelers open the opportunity to develop training strategies that leverage pseudolabeled data beyond semi-supervised learning. We point out that different learning paradigms, such as semi-supervised, transductive zero-shot, and unsupervised learning, can be all considered as special cases of a single objective function, by using pseudolabels as a source of supervision.

We demonstrate that simple iterative training strategies for refining pseudolabels are highly effective approaches for limited-label prompt tuning. In fact, regardless of the prompt modality and learning setting, these strategies improve CLIP, by on average 19.5 points in semi-supervised learning, 28.4 in transductive zero-shot learning, and 15.2 in unsupervised learning.

We show that prompts learned with iterative strategies help mitigate the "rich get richer, poor get poorer" effect observed in semi-supervised approaches leveraging pseudolabels. By redistributing the quality of pseudolabels across different classes, we observe a 'Robin Hood effect' where the extremely rich classes' accuracy stays the same or decreases, while poorer classes get richer, leading to a more equitable distribution of per-class accuracy.

2 Background and related work

Vision-language models Vision-language models such as CLIP [31], ALIGN [17], and Florence [43] are models that align images and text. We focus on CLIP, which is composed of two components: a text encoder, ψ , and an image encoder, ϕ , which are jointly trained using a contrastive loss to learn a multi-modal embedding space which aligns the representations of similar text and image inputs. This pre-training enables CLIP to perform zero-shot image classification. Given an image x and a set of classes Y = { y 1 , ..., y C } , CLIP classifies x by measuring the similarity between the image representation z = ϕ ( x ) and each class representation w i = ψ ( π i ) , based on their cosine distance in the shared embedding space. Here, π i is a natural language prompt such as 'a photo of a [CLASS i ]' , where CLASS i is the specific class name, such as 'orange dahlia,' 'forest' or 'Boeing 737'. The image x gets assigned to the class with the highest similarity score. In this work, we study how to learn better prompts that enhance CLIP by leveraging pseudolabels.

Prompt tuning Prompt tuning is a technique that enhances the practical application of large pretrained models like CLIP [31] and GPT [32, 7]. It involves providing task-specific information to the model during inference through textual or visual inputs, leading to improved performance on downstream tasks [1, 33, 6, 7]. While discrete prompts are manually crafted natural language descriptions of classes that guide the model, they may not yield optimal results [47]. Soft prompting [21, 24], on the other hand, optimizes prompts as continuous vectors. These can be optimized by backpropagating through the frozen pre-trained model, resulting in better performance. Soft prompts can be learned for various modalities, e.g., text or image, [48, 13, 17, 2, 44, 19] and applications [34, 27, 12, 28] by training on a small number of labeled examples per class. If only unlabeled data is accessible, it is possible to learn textual soft prompts by leveraging CLIP-based pseudolabels [15]. Expanding on this concept, we further investigate the use of pseudolabels across a broader range of prompt modalities and learning approaches, and we introduce unexplored training strategies to leverage pseudolabels more effectively.

Learning from pseudolabels Pseudolabeling is the practice of assigning labels to unlabeled data based on the prediction of a model [20]. Then, pseudolabels are used to improve the performance of the model itself. There are different ways to obtain and use pseudolabels and each impacts the final predictions of the model [41, 45, 16, 35]. Some approaches use confidence thresholds [36, 3, 40] and others average predictions from multiple augmentations [4]. Pseudolabeling is a semi-supervised learning technique, and it is rarely used in transductive zero-shot learning [42, 5, 25]. Applying such techniques requires a few labeled examples related to the target task to learn a baseline model capable of pseudolabeling. However, this limitation has been overcome by VLMs, which are capable of pseudolabeling examples without task-specific training. The conventional pseudolabeling scheme based on confidence threshold is not effective if we assign pseudolabels based on CLIP. In fact CLIP is miscalibrated [22] and has imbalanced predictions [38] which may induce noise in the pseudolabels. An alternative approach selects the top-K most confident examples per class to improve performance [15]. In our analysis, we rely on this scheme (Section 3).

3 Design space

Our analysis encompasses the design space consisting of various combinations of prompt modalities, learning paradigms, and training strategies (Figure 1). Within this space, two key components remain constant: the pseudolabeling scheme and a unified loss function. This section begins by introducing these components and subsequently delves into a comprehensive discussion of each dimension within the design space to be explored.

Pseudolabeling scheme The use of CLIP to generate pseudolabels has been investigated in [15]. Given unlabeled data X u with target classes { y 1 , ..., y C } , the goal is to assign labels to data points in which the model is most confident. Typically, pseudo labeling schemes use a confidence threshold ( P ( y | x ) > τ ) to select instances to pseudolabel. However, this approach does not work well for CLIP due to its miscalibration [22] and imbalanced predictions [38]. Instead, one can use a top-K pseudo labeling approach, where the top-K most confident examples per class are used as pseudolabeled data [15]. The pseudolabel assignment consists of (1) computing the similarity scores of each datapoint with classes' textual prompts, and (2) select for each class the K datapoints with the highest similarity score to the class. In this way, we always get K pseudolabels per class, effectively addressesing the natural bias in CLIP's pseudolabels [38]. In Appendix A.1, we provide more details about pseudolabel assignment corner cases.

This top-K pseudolabeling scheme is applicable to unlabeled data, regardless of the availability of labeled data. As a result, we can extend the use of pseudolabels to any learning setting that involves unlabeled data. We observe that by treating pseudolabeled examples as true labeled data, we can view all learning settings as optimizing the same objective function.

Unified objective function Consider a C -class image classification task, where X L and Y L represent the image representations and labels of the labeled data, and X U and ˜ Y U denote the image representations and pseudolabels for the unlabeled data. We define a loss function that combines two cross-entropy losses, one accounting for the error on the labeled data points and the other accounting for the error on pseudolabeled data:

where γ and λ define the training balance between the errors on labeled and pseudolabeled data.

3.1 Prompt modalities

Learning prompts is the process of training a set of vectors P = [p] 1 . . . [p] K that are prepended to the textual or visual inputs of the encoders within the CLIP architecture. By prepending these vectors to specific inputs, we can learn textual prompts, visual prompts, or multimodal prompts when applying a set of vectors to both inputs simultaneously. We provide a technical and detailed explaination in Appendix A.1.

In our exploration, we consider all three types of prompts. The efficacy of prompts can vary depending on the task. Text prompt tuning may be most beneficial when image features are well-separated by class but may not be aligned with the corresponding textual prompt. Visual prompts rearrange the image features within the projection space, and it has the potential to improve CLIP when the pre-trained image features are not well separated by class. Finally, multimodal prompts allows for beneficial interaction between the two separate modalities, which might lead to both separable visual features, and text classifiers that are well-aligned with the corresponding visual features.

3.2 Learning paradigms

By adjusting the values of parameters γ and λ and using the appropriate sets of labeled and pseudolabeled data, the unified objective loss can be customized for each learning paradigm. We note that the redundancy in the use of parameters is for notation clarity in descriptions of the different strategies below. One can simply use λ as balancing factor between labeled and pseudolabeled data.

Semi-supervised learning In the semi-supervised learning (SSL) scenario we have access to a limited number of labeled data for all the target classes D L = { ( x, y ) } where x is an input feature and y ∈ Y = [ C ] is the corresponding label. In addition, we have access to unlabeled data X U = { x } , where x is an image in the target domain Y . From X U , we get D PL = { ( x, ˜ y ) } , where ˜ y ∈ [ C ] is x 's pseudolabel. When using the unified loss in this setting, we set γ to |D PL | / |D L | . As |D L | is much smaller than |D PL | , γ acts as an upweighting factor for the few-labeled instances, thus counterbalancing the learning effect of pseudolabels ( λ =1).

Transductive zero-shot learning In transductive zero-shot learning (TRZSL), we are provided with labeled data D L = { ( x, y ) } for some target classes S (referred to as seen classes), where x represents input features, and y ∈ [ S ] is the corresponding label. Additionally, we have access to unlabeled data X U = { x } for a disjoint set of classes U (referred to as unseen classes). Using X U , we obtain D PL = ( x, ˜ y ) , where ˜ y ∈ [ U ] denotes the pseudolabels for x . The value of λ in the unified loss is set to |D L | / |D PL | , which makes the weight of the pseudolabel loss equivalent to that of the labeled data ( γ = 1 ). This is necessary because an imbalance in the number of labeled and pseudolabeled samples can result in a skewed training distribution, leading to better performance on seen classes while the performance on unseen classes may either remain stagnant or degrade. Studying this setting is interesting beyond transductive zero-shot learning. In fact, it has the potential to generalize to scenarios where the target task involves unseen classes, while the seen classes consist of auxiliary labeled data from the same domain but different task [30].

Unsupervised learning In the unsupervised learning (UL) setting, we have access only to unlabeled data X U = { x } , from which we obtain D PL = ( x, ˜ y ) , where ˜ y ∈ [ C ] denotes the pseudolabel for x . In this case, γ is set to 0, as there is no labeled data, and λ = 1 . The use of this setting was initially explored in [15], who leveraged a few pseudolabels per class to learn textual prompts. In this paper, we build on their work by investigating a variety of training strategies and prompt modalities.

Supervised learning In supervised learning (SL), we are only provided with labeled data D L = ( x, y ) , where x represents an input feature, and y ∈ [ C ] is the corresponding label. If we set λ to 0, the unified loss function is equivalent to the objective functions of default prompt-tuning approaches that optimize the prompts using a few labeled instances per target class. This setting is not strictly part of our design space. However, we will refer to it to define baselines in Section 4.

3.3 Training strategies

The unified objective function enables the development of training strategies broadly applicable across various learning paradigms. We explore three distinct learning strategies to effectively use pseudolabels in this context. The first strategy uses pseudolabels in a static manner. The other two strategies, which are unexplored for prompt tuning, involve the dynamic use of pseudolabeled data.

Few-pseudolabels (FPL) We select K pseudolabels per target class, resulting in a pseudolabeled dataset of size K · C . We learn the prompts by minimizing the objective function via backpropagation through CLIP's encoders. This strategy aligns with Unsupervised Prompt Learning (UPL) in [15]. We refer to it as few-pseudolabels (FPL) to encompass its applicability for learning prompts of diverse modalities across learning paradigms.

Iterative Refinement of FPL (IFPL) Similar to FPL, we obtain the top-K pseudolabels for each target class. These pseudolabels are then used to train a new task-specific prompt. After completing the training, we use the learned prompt to compute the top-K pseudolabels per class again. Subsequently, we reinitialize the prompt and repeat this entire process for a total of I iterations. With this iterative approach, if training with the initial pseudolabel set leads to an improvement in the model's performance, the model itself can become a more effective pseudolabeler, refining the pseudolabels in each subsequent iteration.

Grow and Refine Iteratively Pseudolabels (GRIP) Although IFPL can improve the quality of the K × C pseudolabels used for training, it still limits learning to a few examples per target class. To overcome this constraint, we explore a method similar to IFPL, but with a key difference. In each iteration, we progressively increase the value of K . Specifically, during the i -th iteration, we use

K = ( i × | X U | I ) /C of the unlabeled data to perform the steps in the iterative process. GRIP maintains class balance by selecting the top-K samples at each iteration, with K increasing progressively. Similar to IFPL, both prompts and pseudolabels are reinitialized with every iteration, in order to avoid accumulating errors from earlier iterations. In other words, learning progresses from pseudolabels to new prompts to new pseudolabels, and so on. The rationale behind this strategy is that as the model's accuracy in generating pseudolabels improves, we can increase the total number of pseudolabels without introducing excessive noise.

4 Experiments

We explore the design space outlined in Section 3 to understand the effectiveness of leveraging pseudolabels for limited-label prompt tuning. We show that (1) iterative strategies significantly improve CLIP's performance across prompt modalities and learning settings, (2) using CLIP-based pseudolabels with iterative strategies induces a more equitable distribution of per-class accuracy.

Datasets We conduct the analysis on six tasks, covering specialized and fine-grained domains, where CLIP shows deficiencies [31]. We call this set of tasks FRAMED, and it includes F lowers102 [29], R ESICS45 [9], FGVCA ircraft [26], M NIST [11], E uroSAT [14], and D TD [10]. For each dataset we use the training and test splits provided in [23]. For the transductive zero-shot learning setting we randomly generate three splits of seen and unseen classes with a 62-38 ratio. Further details are in Appendix A.2.

Baselines To evaluate the effectiveness of the training strategies described in Section 3.3, we compare the performance of CLIP when queried with the learned soft prompts to CLIP zero-shot with default prompts such as ' a photo of a [CLASS] .' In addition, we compare with default supervised prompt-tuning baselines, for which we only use the available labeled data: CoOp [48] for textual prompts, VPT [18] for visual prompts, and UPT [44] for multimodal prompts. We defer to Appendix A.1 the technical details of these methods.

Evaluation metrics We assess the performance of each method by measuring the accuracy of the test set, averaging the results over five runs. In the case of TRZSL, we report the harmonic of the accuracies of seen and unseen classes to account for their potentially imbalanced performance [39].

Training settings For all experiments, datasets, and learning strategies, we use ViT-B/32 as the vision backbone. For both visual and textual prompt learning, we set the prefix size to 16 [48, 18]. Multimodal prompts have length 8 [44]. We use SGD as the optimizer and train for 150 epochs. We use 5 warmup epochs at a learning rate of 0.0001, and then set the learning rate to l , which is decayed by the cosine annealing rule. For textual and visual prompt learning, l = 0 . 1 , while for multimodal prompt learning, l = 0 . 01 . In SSL, we use 2 labeled samples per class to assess the impact of pseudolabels in the scenario of very few labeled data and abundant unlabeled data. The number of iterations I is 10 . FPL and IFPL have the number of pseudolabels per class fixed to 16 since it is indicated as the optimal K in the previous research on pseudolabeling with CLIP [15]. In general, K is a hyperparameter that may require optimization in practical cases. We decide to be consistent with the literature and apply this fixed value of K in order to reduce the addition of more confounding factors in our analysis.

4.1 Exploring the design space

GRIP consistently enhances CLIP across prompt modalities and learning settings Table 1 reports the performance of GRIP, the best performing among the training strategies in Section 3.3, compared to CLIP and prompt-tuning baselines. Overall, GRIP consistently improves the performance of CLIP and the baselines across prompt modalities and learning settings. By tuning textual prompts, the average improvement over CLIP is 20.7 points in SSL, 14.9 in UL, and 32.4 in TRZSL, while the improvement on CoOp is 9.6 points in SSL, and 26.6 in TRZSL. Similar results for the visual prompts show that GRIP improves CLIP by 18.2 points in SSL, 15.7 in UL, and 30.8 in TRZSL, and VPT by 12.9 points in SSL, and 20.8 in TRZSL. We note that CoOp and VPT applied to the SSL setting correspond to learning only on the labeled data, and we do not run them in the UL setting as

Visual prompts

Method

CLIP

VPT

GRIP

∆

CLIP

∆

VPT

CLIP

VPT

GRIP

∆

CLIP

∆

0

.

FGVCAircraft

UL

58

24

0

.

17

.

↓

00

-

51

0

.

61

0

.

07

-

DTD

00

-

50.51

0

.

99

↑

7

.

27

-

TRZSL

17

0

.

.

86

00

26.69

0

26

.

42

↑

↓

43

.

.

00

0

.

00

8

.

56

0

.

27

45

0

.

00

44

.

16

62.78

0

.

0

↑

19

01

.

.

↑

18

00

33

.

62

TRZSL

63

0

.

.

40

00

64

.

71

77.18

0

0

↑

13

.

00

.

.

↑

20

12

00

78

.

.

77

47

0

25

.

51

69.54

.

00

0

0

↑

48

.

05

.

01

.

↑

44

77

.

03

SSL

54

60

.

80

71.22

.

1

.

65

0

↑

16

.

.

↑

10

47

.

13

63.48

77

74

.

42

32

.

1

.

34

3

↑

30

.

.

↑

16

09

60

.

35

RESICS45

UL

48

0

.

00

-

68.43

0

↑

13

.

.

61

95

-

EuroSAT

88

0

.

00

-

63.68

3

↑

30

.

-

TRZSL

54

0

.

.

46

00

67

.

06

82.19

0

.

0

↑

27

00

.

.

↑

30

15

00

73

.

.

54

13

0

62

.

24

96.97

.

00

0

.

0

↑

66

02

.

.

↑

34

00

43

.

73

SSL

17

17

.

76

.

0

19.43

.

68

0

.

5

↑

1

.

85

↑

1

.

67

43

36

.

41

54.57

.

2

.

17

4

↑

11

.

.

↑

18

86

33

.

16

VPT

SSL

63

63

.

1

.

73

67.95

.

52

1

2

.

↑

28

4

.

↑

4

.

22

25

42.53

10

.

14

69.66

13

.

5

.

↑

44

.

↑

27

51

56

.

14

Flowers102

UL

67

0

00

.

63

-

.

↓

09

0

.

55

0

58

.

-

MNIST

.

00

-

68.04

1

↑

42

.

11

.

-

there is no labeled data. Results are similar for multimodal prompts. We defer them to Appendix A.3, due to space constraints.

No prompt modality is clearly superior Using pseudolabels dynamically is beneficial for each modality. However, determining the clear superiority of one prompt modality over the other is challenging, as it depends on the specific tasks. For example, visual prompts work better for EuroSAT, while textual prompts excel in Flowers102. Despite intuitive explanations (Section 3.1), the scientific consensus remains elusive [44]. Hence, we prefer to emphasize that the dynamic use of pseudolabels consistently improves performance for each prompt modality, without declaring one modality as definitively better than the other.

Unsupervised learning is equivalent or more robust than learning with very few shots The accuracy of GRIP when applied to the fully unsupervised setting is either higher or equivalent to the accuracy of VPT, which

100

50

0

50

100

150

200

Class Balance (%)

20

40

60

80

100

Accuracy (%)

Class Balance vs Accuracy

CoOp/VPT

FPL

IFPL

GRIP

is trained using two labeled instances per class (Table 1). This shows that pseudolabeled data can substitute very few labeled examples for prompt tuning. However, the significant improvement of GRIP over CoOp and VPT in the semi-supervised setting (see Table 1) suggests that leveraging unlabeled data through pseudolabeling is advantageous in scenarios where labeled data is scarce but there is an abundance of unlabeled data.

94

.

42

80

0

Transductive zero-shot learning effectively transfers knowledge In the TRZSL setting, GRIP improves over CLIP and the baselines by a large margin (Table 1). Figure 2 displays the balance of seen and unseen classes of each method alongside its accuracy. The class balance is ( acc unseen -acc seen ) /acc seen , where values close to zero indicate a good balance, negative values indicate better accuracies for seen classes, and positive values indicate better accuracies for unseen classes. Methods employing an iterative usage of pseudolabels maintain a good balance, as opposed to CoOp/VPT and FPL. This balance in accuracy is likely a combined effect of the quality of the pseudolabels and the transfer of knowledge from the seen to the unseen classes. The latter point is significant because it implies that even if we only possess unlabeled data for a specific target task, we can still use labeled data from related classes [30] within the same domain to enhance CLIP's performance.

There is a trade-off between quality and quantity of pseudolabels Table 2 shows the performance of CLIP employing prompts learned with different training strategies, all leveraging pseudolabels (Section 3.3). Iterative strategies are more effective than FPL which, similarly to [15], use a static set of a few pseudolabels for one iteration.

10

20

30

40

50

60

70

80

90 100

% of unlabeled data

0.4

0.6

0.8

1.0

CLIP

IFPL

GRIP

1

2

3

4

5

6

7

8

9

10

Iterations

10

20

30

40

50

60

70

80

90 100

% of unlabeled data

0.4

0.6

0.8

1.0

CLIP

IFPL

GRIP

1

2

3

4

5

6

7

8

9

10

Iterations

10

20

30

40

50

60

70

80

90 100

% of unlabeled data

0.4

0.6

0.8

1.0

CLIP

IFPL

GRIP

1

2

3

4

5

6

7

8

9

10

Iterations

10

20

30

40

50

60

70

80

90 100

% of unlabeled data

0.4

0.6

0.8

1.0

CLIP

IFPL

GRIP

1

2

3

4

5

6

7

8

9

10

Iterations

10

20

30

40

50

60

70

80

90 100

% of unlabeled data

0.4

0.6

0.8

1.0

CLIP

IFPL

GRIP

1

2

3

4

5

6

7

8

9

10

Iterations

(a) Flowers102

10

20

30

40

50

60

70

80

90 100

% of unlabeled data

0.4

0.6

0.8

1.0

CLIP

IFPL

GRIP

1

2

3

4

5

6

7

8

9

10

Iterations

(b) RESICS45

On Flowers102, RESICS45, and DTD, IFPL improves on average FPL by 5.6 points in SSL, 1.7 in UL, and 5.6 in TRZSL. GRIP boosts the performance even more by on average 7.8 points in SSL, 3.1 in UL, and 9.7 in TRZSL. Results on the other tasks are comparable or larger and we report them in Appendix A.3 due to space constraints.

Figure 3 shows the progression of pseudolabels quality for the iterative learning of textual prompts. IFPL maintains a fixed set of 16 pseudolabels, improving their quality with each iteration (top x-axis).

On the other hand, GRIP and CLIP expand pseudolabels by incorporating an additional decile of unlabeled data in each iteration (bottom x-axis). Initially, GRIP maintains accuracy, but as it nears completion the quality tends to decrease, while a larger dataset with good-quality pseudolabels becomes available.

Comparing GRIP and CLIP, GRIP's expanded pseudolabels exhibit superior quality and performs better (Table 1). Even though IFPL's pseudolabel accuracy surpasses GRIP in the final iteration, GRIP's overall performance remains better due to training on a larger number of pseudolabels (Table 2). This suggests that numerous, slightly noisier pseudolabels can yield better results, highlighting a trade-off and offering insights for future approaches.

GRIP benefits adaptation even for larger image encoders We measure how much the effect of the iterative strategies changes if we consider a larger pre-trained image encoder. In Table 3, we report the average improvements of GRIP on CLIP for Flowers102, RESICS45, and DTD. The magnitude of improvements slightly decreases when using a larger image encoder. However, we still see significant benefits for both modalities. The smaller relative improvements with respect to smaller visual encoders align with our expectations. Larger encoders possess a stronger base knowledge, making it relatively more challenging to attain further improvements on top of it. In Table 10, we break

down the accuracy of CLIP with different backbones. The performance of the larger backbone is higher, indicating higher quality pseudolabels.

4.2 The Robin Hood effect

Although training models with pseudolabels can lead to good performance, it can also result in biased predictions and generate disparate impacts on sub-populations, i.e., the 'Matthew effect' [49, 8]. Particularly, the use of pseudolabels can lead to improved performance in well-behaved (high accuracy) classes but can cause stagnation or decreased performance in poorly behaved (low accuracy) classes. As we explore the use of pseudolabels, we investigate how the accuracy of the analyzed approaches distributes across classes. Figure 4 show an opposite scenario from typical SSL. The solid line represents the sorted per-class accuracies of CLIP. The arrows indicate the per-class accuracies of GRIP. For all learning paradigms, the iterative training strategies increase the accuracy of classes where CLIP is not proficient, while maintaining or decreasing the accuracy of initially well-behaved classes. This effect, which we call the 'Robin Hood effect,' is very interesting because it shows how CLIP can mitigate its own bias toward certain classes by learning from itself.

To understand the roots of the Robin Hood effect, we examine two factors: (1) the role of pseudolabels generated by CLIP, and (2) the role of prompt tuning. To disentangle these factors, we explore the variation in per-class accuracy of a basic linear classifier trained on CLIP's ViT-B/32 image representation.

'Second generation' pseudolabels are a good treatment for class disparity Wetrain the linear classifier in the SSL setting

0

20

40

0.0

0.5

1.0

SSL

CLIP

0

20

40

0.0

0.5

1.0

SSL

CLIP

0

20

40

0.0

0.5

1.0

UL

CLIP

0

20

40

0.0

0.5

1.0

UL

CLIP

0

20

40

0.0

0.5

1.0

TRZSL

CLIP

(a) FPL

0

20

40

0.0

0.5

1.0

TRZSL

CLIP

(b) GRIP

on 2 labeled examples per class and pseudolabels. The pseudolabels are obtained through conventional methods, where a threshold of .95 is applied, or by using CLIP to generate 16 pseudolabels per class.

We find that both approaches yield similar overall accuracies. However, we observe the 'Matthew effect' when using the first approach. In contrast, when using CLIP-based pseudolabels, the class disparity of the regressor trained solely on seen classes is reduced. Particularly, we see a significant improvement on initially poor classes, together with a significant diminishing of the accuracy of well-behaved classes. We observe a clear manifestation of the 'Robin Hood effect.' We present plots illustrating this effect in Appendix A.4.

Prompt tuning retains the accuracy of already rich classes better than linear probing To evaluate the role of prompt tuning in the 'Robin Hood effect,' we train a linear classifier and textual prompts in the UL setting using GRIP's training strategy. Comparing the per-class acurracies of the two approaches, GRIP on prompts shows an average improvement of 22.85 points for the poor classes across tasks, along with a slight average decrease of 0.3 points for the rich classes. On the other hand, linear probing determines a 14.42 points improvement for the poor classes, but it results in an average decrease of 9.39 points in accuracy for the rich classes (Appendix A.4).

5 Conclusions

We show that prompt tuning using pseudolabels generated by CLIP itself is a successful approach to enhance CLIP across various learning settings. Training strategies that iteratively refine pseudolabels turn out to be effective ways of leveraging pseudolabeled data. These approaches not only enhance CLIP's accuracy but also mitigate model biases toward certain classes. We hope this work lays a solid groundwork for reducing reliance on labeled data when adapting pre-trained vision-language models like CLIP to new tasks.

Limitations The effectiveness of the training strategies examined in this paper depends on both the strategies themselves and the quality of pseudolabels. The latter is particularly crucial. If CLIP performs poorly on a task, we may struggle to obtain a reliable set of pseudolabels to begin with, potentially diminishing CLIP's performance. Despite this potential risk, we have not observed any relevant failure of GRIP, even in tasks where CLIP's initial accuracy is extremely low (such as FGVCAircraft). Also, the pseudolabeling strategy we adopt involves selecting K pseudolabels per class, which can create a strong assumption about the distribution of the training data if we attempt to cover all unlabeled data. During the final iteration, it is as if we assume a uniform class balance.

Another important consideration is the efficiency of the explored methods. Repeating the training process multiple times brings impressive improvements at the cost of a non-negligible increase of computation time. At each iteration, we generate pseudolabels for the unlabeled data from scratch. While we parallelized the pseudolabeling procedure to cut some of the cost, reducing those for the iterative training presents more significant challenges. We decided to mainly focus on the analysis of qualitative and quantitative effects of pseudolabels in prompt tuning. Future research should address budget constraints and investigate optimal stopping criteria for the iterative process, considering the possibility of reaching a plateau or decreased pseudolabel quality after a certain point, to maximize efficiency while maintaining performance.

Acknowledgments and Disclosure of Funding

We are thankful to our reviewers for the fruitful and insightful discussions that contributed to the refinement of the paper. We also thank Reza Esfandiarpoor and Zheng-Xin Yong for the comments on our drafts. This material is based on research sponsored by Defense Advanced Research Projects Agency (DARPA) and Air Force Research Laboratory (AFRL) under agreement number FA8750-192-1006. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of Defense Advanced Research Projects Agency (DARPA) and Air Force Research Laboratory (AFRL) or the U.S. Government. We gratefully acknowledge support from Google and Cisco. Disclosure: Stephen Bach is an advisor to Snorkel AI, a company that provides software and services for data-centric artificial intelligence.

References

Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Fries, Maged Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir Radev, Mike Tian-jian Jiang, and Alexander Rush. PromptSource: An integrated development environment and repository for natural language prompts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations . Association for Computational Linguistics, 2022.

Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.17274 , 2022.

David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring. In International Conference on Learning Representations , 2020.

David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems , 2019.

Liu Bo, Qiulei Dong, and Zhanyi Hu. Hardness sampling for self-training based transductive zero-shot learning. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 16494-16503, 2021.

Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel J. Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R'e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. ArXiv , abs/2108.07258, 2021.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems , 2020.

Baixu Chen, Junguang Jiang, Ximei Wang, Pengfei Wan, Jianmin Wang, and Mingsheng Long. Debiased self-training for semi-supervised learning. In Advances in Neural Information Processing Systems , 2022.

Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE , 2017.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. 2014 IEEE Conference on Computer Vision and Pattern Recognition , 2013.

Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine , 2012.

Yunhe Gao, Xingjian Shi, Yi Zhu, Hongya Wang, Zhiqiang Tang, Xiong Zhou, Mu Li, and Dimitris N. Metaxas. Visual prompt tuning for test-time domain adaptation. ArXiv , abs/2210.04831, 2022.

Chunjiang Ge, Rui Huang, Mixue Xie, Zihang Lai, Shiji Song, Shuang Li, and Gao Huang. Domain adaptation via prompt learning. ArXiv , abs/2202.06687, 2022.

Patrick Helber, Benjamin Bischke, Andreas R. Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 2017.

Hao Huang, Jack Chu, and Fangyun Wei. Unsupervised prompt learning for vision-language models. ArXiv , abs/2204.03649, 2022.

Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondˇ rej Chum. Label propagation for deep semi-supervised learning. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5065-5074, 2019.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, YunHsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning , 2021.

Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV 2022: 17th European Conference on Computer Vision , 2022.

Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. ArXiv , abs/2210.03117, 2022.

Dong-Hyun Lee. Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks. ICML 2013 Workshop : Challenges in Representation Learning (WREPL) , 2013.

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , 2021.

Will LeVine, Benjamin Pikus, Pranav Vishnu Raja, and Fernando Amat. Enabling calibration in the zero-shot inference of large vision-language models. In ICLR 2023 Workshop on Pitfalls of limited data and computation for Trustworthy ML , 2023.

Chunyuan Li, Haotian Liu, Liunian Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang, Ping Jin, Houdong Hu, Zicheng Liu, Yong Jae Lee, and Jianfeng Gao. Elevater: A benchmark and toolkit for evaluating language-augmented visual models. In Advances in Neural Information Processing Systems , 2022.

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , 2021.

Bo Liu, Lihua Hu, Qiulei Dong, and Zhanyi Hu. An iterative co-training transductive framework for zero shot learning. IEEE Transactions on Image Processing , 30:6943-6956, 2021.

Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Finegrained visual classification of aircraft. ArXiv , abs/1306.5151, 2013.

Shu Manli, Nie Weili, Huang De-An, Yu Zhiding, Goldstein Tom, Anandkumar Anima, and Xiao Chaowei. Test-time prompt tuning for zero-shot generalization in vision-language models. In NeurIPS , 2022.

Nihal V. Nayak, Peilin Yu, and Stephen Bach. Learning to compose soft prompts for compositional zero-shot learning. In The Eleventh International Conference on Learning Representations , 2023.

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing , 2008.

Wasu Piriyakulkij, Cristina Menghini, Ross Briden, Nihal V. Nayak, Jeffrey Zhu, Elaheh Raisi, and Stephen H. Bach. TAGLETS: A system for automatic semi-supervised learning with auxiliary data. In Conference on Machine Learning and Systems (MLSys) , 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , 2021.

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.

Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations , 2022.

Sheng Shen, Shijia Yang, Tianjun Zhang, Bohan Zhai, Joseph E. Gonzalez, Kurt Keutzer, and Trevor Darrell. Multitask vision-language prompt tuning. arXiv preprint arXiv:2211.11720 , 2022.

Weiwei Shi, Yihong Gong, C. Ding, Zhiheng Ma, Xiaoyu Tao, and Nanning Zheng. Transductive semi-supervised deep learning using min-max features. In European Conference on Computer Vision , 2018.

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semisupervised learning with consistency and confidence. In Advances in Neural Information Processing Systems , 2020.

Ximeng Sun, Ping Hu, and Kate Saenko. Dualcoop: Fast adaptation to multi-label recognition with limited annotations. ArXiv , 2022.

Xudong Wang, Zhi-Li Wu, Long Lian, and Stella X. Yu. Debiased learning from naturally imbalanced pseudo-labels. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning - the good, the bad and the ugly. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017.

Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. In Advances in Neural Information Processing Systems , 2020.

Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi-supervised learning with dynamic thresholding. In International Conference on Machine Learning , 2021.

Yunlong Yu, Zhong Ji, Xi Li, Jichang Guo, Zhongfei Zhang, Haibin Ling, and Fei Wu. Transductive zero-shot learning with a self-training dictionary approach. IEEE Transactions on Cybernetics , 48(10):2908-2919, 2018.

Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel C. F. Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, and Pengchuan Zhang. Florence: A new foundation model for computer vision. ArXiv , abs/2111.11432, 2021.

Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning. ArXiv , abs/2210.07225, 2022.

Bowen Zhang, Yidong Wang, Wenxin Hou, HAO WU, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. In Advances in Neural Information Processing Systems , 2021.

X. Zhang, Yusuke Iwasawa, Yutaka Matsuo, and Shixiang Shane Gu. Domain prompt learning for efficiently adapting clip to unseen domains. 2021.

Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. 2021.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision , 2021.

Zhaowei Zhu, Tianyi Luo, and Yang Liu. The rich get richer: Disparate impact of semisupervised learning. In International Conference on Learning Representations , 2022.

A Appendix

We include here extra information that supports the results presented in the main body of the paper.

Reproducibility We have provided the code to run the experiments as supplementary material for the submission. However, we plan to release it as an open repository upon acceptance.

A.1 Trainable Prompts

Text Prompt Tuning The primary objective of text prompt tuning is to improve the alignment between the class token and the image features extracted by the image encoder. This is achieved by adding learnable vectors, i.e., prefix , before the CLASS token to create a contextualized representation. Specifically, the sequence

is fed into the textual encoder, where each vector [V] m ( m ∈ 1 , . . . , M ) has the same dimension as word embeddings, and M is a hyperparameter that determines the length of the prefix.

Context Optimization (CoOp) [48] was the first work to explore continuous prompts for VLMs. Follow-up works have experimented with different training strategies to enhance the generalizability of the learned prompts while preserving the core concept of continuous vector tuning [34, 12, 27, 46, 13, 37].

Tuning the text prefix vector changes the resulting n linear weight vectors w i = ψ ( p i ) , while leaving the image features unchanged. Therefore, text prompt tuning may be most beneficial when image features are well-separated by class but may not be aligned with the corresponding textual prompt. Conversely, text prompt tuning may not be as effective when the image features are poorly separated, as in specialized or novel domains where CLIP may lack sufficient training data.

Visual Prompt Tuning Instead of tuning the text prompts, one can also tune the inputs of the vision encoder. In this case, a learnable visual prefix is prepended to the image tokens as input to the image transformer as follows:

where p represents a sequence of K learnable prefix vectors, and [I] 1 . . . [I] P are the image tokens from the corresponding P patches of the input images. The new sequence ˆ I is the input to the image encoder ϕ .

Visual Prompt Tuning (VPT) was introduced in the context of efficiently adapting pre-trained vision transformers to downstream tasks [18]. However, the approach has since been applied in the context of VLM [34].

Whereas text prompt tuning does not alter the image features, visual prompt tuning does. By rearranging the image features within the projection space, VPT has the potential to improve CLIP when the image features are not well separated by class, such as in specialized domains.

Multimodal Prompt Tuning The previous approaches are unimodal, as they either involve modifying the text or visual input, but never both. This choice may be suboptimal as it does not allow the flexibility to dynamically adjust both representations on a downstream task. Recently, multimodal prompt tuning has been introduced [44, 19]. We focus on Unified Prompt Tuning (UPT) [44] which essentially learns a tiny neural network to jointly optimize prompts across different modalities. UPT learns a set of prompts U = [ U T , U V ] ∈ R d × n with length n , where U T ∈ R d × n T , U V ∈ R d × n V . U is transformed as follows:

where SA is the self-attention operator, LN is the layer normalization operator, and FFN is a feed forward network. After transformation, we obtain ˆ U = [ ˆ U T , ˆ U V ] ∈ R d × n , such that ˆ U T is to be used as a text prompt, and ˆ U V is to be used as a visual prompt.

The author of UPL argue that self-attention allows for beneficial interaction between the two separate modalities, which leads to both separable visual features, and text classifiers that are well-aligned with the corresponding visual features [44].

Prompts initialization We initialize textual and visual prompts from a normal distribution of mean 0 and variance 0.02. We note that we learn shallow visual prompts by modifying only the input to the image encoder. Multimodal prompts are initialized from a uniform distribution. We found that the latter was not working properly for textual and visual prompts.

Additional training settings For training, the batch size is 64.

Additional details about pseudolabels assignment It can happen that if K is too large and the unlabeled dataset is smaller than K × C , we cannot assign K samples per class. In this case, we can reduce K accordingly. Also, we the same sample might get assigned to the pseudolabel lists of different classes. It rarely happens in our experiments cases. Hoever, this is a characteristic of the pseudolabeling strategy proposed in [15], and it can be an object of study for future work motivated by the effectiveness of self-training.

A.2 Datasets details

We use six datasets from specialized or fine-grained domains. Here we provide a description of each of them. In Table 4, we report the details about the number of classes and data available for each dataset. For each dataset, we also show CLIP's prediction distribution over classes Figure 5.

Flowers102 [29] It is a dataset collecting images for 102 flower categories commonly occurring in the United Kingdom. For each class we have between 40 and 258 images. Figure 5a shows that CLIP's predictions are skewed toward certain classes, which are predicted more often than what we would expect according to the real class distribution on the test set.

RESICS45 [9] This is a publicly available benchmark for Remote Sensing Image Scene Classification. It collects 45 kind of scenes. Figure 5b shows that CLIP predicts more often a subset of classes.

0

20

40

60

80

100

Ranked Class Index

0.000

0.005

0.010

0.015

0.020

0.025

0.030

0.035

Frequency

True

CLIP

(a) Flowers102

0

10

20

30

40

Ranked Class Index

0.00

0.01

0.02

0.03

0.04

0.05

0.06

Frequency

True

CLIP

(b) RESICS45

0

20

40

60

80

100

Ranked Class Index

0.00

0.01

0.02

0.03

0.04

0.05

0.06

Frequency

True

CLIP

(c) FGVC-Aircraft

0

2

4

6

8

Ranked Class Index

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Frequency

True

CLIP

(d) MNIST

0

2

4

6

8

Ranked Class Index

0.0

0.1

0.2

0.3

0.4

Frequency

True

CLIP

(e) EuroSAT

0

10

20

30

40

Ranked Class Index

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

Frequency

True

CLIP

(f) DTD

FGVC-Aircraft [26] It describes the fine-grained task of categorizing aircraft. We consider the task of classifying aircrafts into 100 variants. Also for this task, CLIP assigns images to a reduced set of classes (Figure 5c).

MNIST [11] MNIST is a database of handwritten digits. The digits are size-normalized and centered in a fixed-size image. We observe that CLIP never predicts 6 out of 10 classes (Figure 5c).

EuroSAT [14] EuroSAT represents the task of categorizing satellite images of scenes. It consists of 10 classes. In Figure 5e, we show CLIP's predictions distribution over the classes.

DTD [10] DTD stands for Describable Textures Dataset. It is an evolving collection of textural images in the wild, and it is annotated relying on human-centric attributes, inspired by the perceptual properties of textures. The zero-shot CLIP predictions show the model's bias toward certain classes (Figure 5f).

A.3 Experiments

In this section, we report tables and plots that complement the results presented in Section 4.

The effect of GRIP on multimodal prompts Table 5 shows the improvements of GRIP on CLIP and Unified Prompt Tuning (UPL) [44]. Similar to the results in Table 1, GRIP consistently improves CLIP with respect to the baselines. The improvements on CLIP are by 18.2 in semi-supervised learning, 14.8 in unsupervised learning, and 30.7 in transductive zero-shot learning. While GRIP outperforms UPL by 4.7 in semi-supervised learning, and 19.5 in transductive zero-shot learning.

Comparison across iterative strategies In Table 6, we report a comparison between FPL and the iterative strategies (IFPL and GRIP) on MNIST, EuroSAT, and FGVC-Aircraft. Results on the other tasks can be found in the main body of the paper Section 4.1. While GRIP largely and consistently outperforms FPL by on average 16.7 points in accuracy, IFPL is not robust and it leads to performances that are inferior to FPL by on average 4.4 points in accuracy.

The evolving accuracy of dynamic pseudolabels Figure 6 represents the evolution of pseudolabels accuracy during training for all datasets, but Flowers102 and RESICS45 presented in Figure 3. We observe that the accuracy of the pseudolabels characterizes the overall performance of the models reported in Table 6. For instance, IFPL for EuroSAT in the TRZSL setting is highly variable, explaining the low average accuracy of the model on the test set (Table 6). Similarily, for MNIST in the TRZSL we observe that after the first iteration, the pseudolabels get very noisy. When we increase the amount of pseudolabeled data, the accuracy of CLIP is not necessarily constant. This is because as we increase K, we are effectively selecting pseudolabels with lower similarities to the

10

20

30

40

50

60

70

80

90 100

% of unlabeled data

0.4

0.6

0.8

1.0

CLIP

IFPL

GRIP

1

2

3

4

5

6

7

8

9

10

Iterations

10

20

30

40

50

60

70

80

90 100

% of unlabeled data

0.4

0.6

0.8

1.0

CLIP

IFPL

GRIP

1

2

3

4

5

6

7

8

9

10

Iterations

10

20

30

40

50

60

70

80

90 100

% of unlabeled data

0.4

0.6

0.8

1.0

CLIP

IFPL

GRIP

1

2

3

4

5

6

7

8

9

10

Iterations

10

20

30

40

50

60

70

80

90 100

% of unlabeled data

0.1

0.2

0.3

0.4

CLIP

IFPL

GRIP

1

2

3

4

5

6

7

8

9

10

Iterations

10

20

30

40

50

60

70

80

90 100

% of unlabeled data

0.4

0.6

0.8

1.0

CLIP

IFPL

GRIP

1

2

3

4

5

6

7

8

9

10

Iterations

10

20

30

40

50

60

70

80

90 100

% of unlabeled data

0.4

0.6

0.8

1.0

CLIP

IFPL

GRIP

1

2

3

4

5

6

7

8

9

10

Iterations

10

20

30

40

50

60

70

80

90 100

% of unlabeled data

0.2

0.4

0.6

0.8

1.0

CLIP

IFPL

GRIP

1

2

3

4

5

6

7

8

9

10

Iterations

10

20

30

40

50

60

70

80

90 100

% of unlabeled data

0.1

0.2

0.3

0.4

CLIP

IFPL

GRIP

1

2

3

4

5

6

7

8

9

10

Iterations

10

20

30

40

50

60

70

80

90 100

% of unlabeled data

0.4

0.6

0.8

1.0

CLIP

IFPL

GRIP

1

2

3

4

5

6

7

8

9

10

Iterations

(a) DTD

10

20

30

40

50

60

70

80

90 100

% of unlabeled data

0.4

0.6

0.8

1.0

CLIP

IFPL

GRIP

1

2

3

4

5

6

7

8

9

10

Iterations

(b) EuroSAT

10

20

30

40

50

60

70

80

90 100

% of unlabeled data

0.2

0.4

0.6

0.8

1.0

CLIP

IFPL

GRIP

1

2

3

4

5

6

7

8

9

10

Iterations

(c) MNIST

10

20

30

40

50

60

70

80

90 100

% of unlabeled data

0.2

0.4

0.6

0.8

CLIP

IFPL

GRIP

1

2

3

4

5

6

7

8

9

10

Iterations

(d) FGVC-Aircraft

classes, resulting in a reduction in their accuracy, as shown in the plot. This observation aligns with previous findings in [15].

GRIP performance on transductive zero-shot learning We show how the effectiveness of GRIP is consistent over the three random splits of seen and unseen classes which we randomly generated. The splits are reported in Table 9. Table 8 gathers the accuracy of seen and unseen classes, along with the harmonic mean for all three splits using textual prompts. Beyond the consistent improvement induced by GRIP training strategy, we observe that the accuracy of GRIP on the seen classes is often lower than the accuracy of CoOp on the same set of classes. During training, for large lambda, the loss component of unlabeled data (unseen classes) is the first to decrease, while the loss on the seen classes reduces later. Thus, we hypothesize that extra training steps might be needed to complete the learning on the labeled data.

A.4 The Robin Hood effect

The Robin Hood effect on all tasks For each dataset, we provide the per-class accuracy distribution of GRIP compared with CLIP, Figure 8. The Robin Hood effect characterizes all the tasks. We observe that for GRIP the increase in overall accuracy corresponds to consistent improvements in the predictions of initially poor classes. By comparing Figure 7 with Figure 8, we see that GRIP reinforces the Robin Hood effect already visible when using FPL in certain cases.

The importance of good quality pseudolabels to mitigate the Matthew effect in SSL In the SSL setting, we train a logistic regression on top of the visual feature extracted by CLIP's image encoder (ViT-B/32). In Figure 9, we show the per-class accuracy of the final model trained by combining labeled data with either pseudolabels assigned with the conventional scheme (threshold at .95) or 16 CLIP-generated pseudolabels. We compare the two distribution with the per-class accuracy of the model trained solely on the few labeled examples per class (2 instances).

The different impact of prompt tuning and linear probing on the Robin Hood effect We investigate if there is any difference in the Robin Hood effect when adapting CLIP via prompt tuning or linear probing. We train both relying on the iterative training strategy that grows the set of pseudolabels at each iteration by using the topK scheme (Section 3). We consider the UL setting.

Among the set of target classes, we distinguish between poor and rich classes. A class is poor , if CLIP's accuracy on that class is lower than its overall accuracy on the task. Otherwise, the class is considered rich . Table 7 reports the accuracy of the two approaches, and the accuracy on the poor and rich classes, while highlighting the average effect with respect to CLIP. Training with prompt tuning retains more knowledge of the rich classes than linear probing. Prompt tuning reduces the accuracy on the rich classes by on average 0.3 points, while linear probing has an average deterioration of 9.4. the reduction of accuracy of rich classes is on average 30 times larger than the reduction observed using prompts. Overall, GRIP works better than linear probing. We note that the lower accuracy of linear probing is characterized by a worse ability to correctly predict the rich classes, i.e., 'rich get poorer.' This is surprising, as we would have expected the errors to concentrate on the poor classes compared to CLIP.

0

100

0.0

0.5

1.0

SSL

CLIP

0

100

0.0

0.5

1.0

SSL

CLIP

0

5

0.0

0.5

1.0

SSL

CLIP

0

5

0.00

0.25

0.50

0.75

SSL

CLIP

0

20

40

0.0

0.5

1.0

SSL

CLIP

0

100

0.0

0.5

1.0

UL

CLIP

0

100

0.0

0.5

1.0

UL

CLIP

0

5

0.0

0.5

1.0

UL

CLIP

0

5

0.00

0.25

0.50

0.75

UL

CLIP

0

20

40

0.0

0.5

1.0

UL

CLIP

0

100

0.0

0.5

1.0

TRZSL

CLIP

(a) Flowers102

0

100

0.0

0.5

1.0

TRZSL

CLIP

(b) FGVC-Aircraft

0

5

0.0

0.5

1.0

TRZSL

CLIP

(c) MNIST

0

5

0.00

0.25

0.50

0.75

TRZSL

CLIP

(d) EuroSAT

0

20

40

0.0

0.5

1.0

TRZSL

CLIP

(e) DTD

0

100

0.0

0.5

1.0

SSL

CLIP

0

100

0.0

0.5

1.0

SSL

CLIP

0

5

0.0

0.5

1.0

SSL

CLIP

0

5

0.00

0.25

0.50

0.75

SSL

CLIP

0

20

40

0.0

0.5

1.0

SSL

CLIP

0

100

0.0

0.5

1.0

UL

CLIP

0

100

0.0

0.5

1.0

UL

CLIP

0

5

0.0

0.5

1.0

UL

CLIP

0

5

0.00

0.25

0.50

0.75

UL

CLIP

0

20

40

0.0

0.5

1.0

UL

CLIP

0

100

0.0

0.5

1.0

TRZSL

CLIP

(a) Flowers102

0

100

0.0

0.5

1.0

TRZSL

CLIP

(b) FGVC-Aircraft

0

5

0.0

0.5

1.0

TRZSL

CLIP

(c) MNIST

0

5

0.00

0.25

0.50

0.75

TRZSL

CLIP

(d) EuroSAT

0

20

40

0.0

0.5

1.0

TRZSL

CLIP

(e) DTD

0

25

50

75

100

0.0

0.2

0.4

0.6

0.8

1.0

Logistic

0

10

20

30

40

0.0

0.2

0.4

0.6

0.8

1.0

Logistic

0

25

50

75

100

0.0

0.2

0.4

0.6

0.8

1.0

Logistic

0

2

4

6

8

0.0

0.2

0.4

0.6

0.8

1.0

Logistic

0

2

4

6

8

0.0

0.2

0.4

0.6

0.8

1.0

Logistic

0

20

40

0.0

0.2

0.4

0.6

0.8

1.0

Logistic

0

25

50

75

100

0.0

0.2

0.4

0.6

0.8

1.0

Logistic

(a) Flowers102

0

10

20

30

40

0.0

0.2

0.4

0.6

0.8

1.0

Logistic

(b) RESICS45

0

25

50

75

100

0.0

0.2

0.4

0.6

0.8

1.0

Logistic

(c) FGVCA

0

2

4

6

8

0.0

0.2

0.4

0.6

0.8

1.0

Logistic

(d) MNIST

0

2

4

6

8

0.0

0.2

0.4

0.6

0.8

1.0

Logistic

(e) EuroSAT

0

20

40

0.0

0.2

0.4

0.6

0.8

1.0

Logistic

(f) DTD

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is about the exploration of using pseudolabels for VLMs on different downstream tasks. Specifically, the authors experimented CoOp, VPT and UPT with pseudolabels generated from CLIP on semi-supervised learning, transductive zero-shot learning and unsupervised learning. They proposed three training strategies, namely FPL, IFPL and GRIP, where the difference is mainly at whether the pseudolabels are static or dynamic. Experiments on 6 datasets show significant improvement.\n\nSTRENGTHS:\n- Writing is good. The work is presented in a logical way and is easy to follow.\n- Performance improvement is impressing.\n- Experiments are thorough.\n\nWEAKNESSES:\n- Novelty is limited. The use of CLIP's pseudolabels are not new and the proposed training strategies are also widely used in self-training methods. \n- I agree with the authors' statement that \"If CLIP performs poorly on a task, we may struggle to obtain a reliable set of pseudolabels to begin with, potentially diminishing CLIP’s performance.\" Therefore, it is unintuitive why GRIP achieves such a big improvement even on datasets like MNIST where CLIP fails. One explanation I can think of is that the quality of the pseudolabels are drastically increasing as the training proceeds. However, results from Fig 3 seems to show the opposite. I am mainly concerned about this and believe further analysis is needed to explain what is the exact reason for the success of the proposed method.\n- I'm also concerned with the computational cost since a total of 10 iterations are used. Authors should include this when comparing with other methods.\n\nQUESTIONS:\n- The authors stated that the proposed IFPL and GRIP are \"novel and proposed here for the first time for prompt tuning\". However, aren't these two methods similar in spirit to self-training? What are the differences?\n-  According to Fig 3, the accuracy of pseudolabels are only marginally increasing for IFPL, and decreasing for GRIP as the training proceeds. Is it necessary then to train over 10 iterations? What are the detailed improvements for each iteration? \n- Why is it that CLIP's pseudolabel accuracy decreases in Fig 3? Shouldn't zero-shot CLIP remain an approximately constant performance irrespective of the number of iteration and the number of unlabeled data?\n- When using a larger image encoder, why does the performance decrease? If a larger backbone is used, wouldn't the quality of pseudolabels be higher? This further deepens my concerns on the proposed method.\n\nLIMITATIONS:\nYes, the authors have addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors provide compelling evidence that underlines the power of a repetitive prompt-training approach, which leverages CLIP-based pseudo labels. Regardless of the learning model (SSL, TZSL, UL) or the type of prompt (text, visual), this strategy significantly enhances the image classification capabilities of CLIP across multiple settings. Moreover, by using the Top-K pseudo-labeling approach, they ensure a balanced distribution of pseudo-labeled training samples for each class, thereby effectively addressing the inherent bias of CLIP pseudo labels.\n\nSTRENGTHS:\n- Various learning paradigms, including semi-supervised, transductive zero-shot, and unsupervised learning, can all be viewed as unique instances of a single objective function when pseudolabels are used as a form of guidance.\n-  As evidenced by Table 1and Table 2, the experimental results demonstrate a significant boost in performance, indicating that the proposed pseudo-labeling strategy holds considerable merit.\n\nWEAKNESSES:\nThe overall structure or layout of the manuscript needs to be further refined.\n- Table 5 (\"There is a trade-off between quality and quantity of pseudolabels\") is mentioned in L290, but Table 5 does not exist in the main manuscript. \n- Table 2 was not mentioned at all in the body text. \n- Since Fig 3 and Fig 5 are too small to be seen properly, efforts are needed to improve readability.\n- The formulas of L157-L158 should be specified as an equation in the form of (1), but it was not.\n\nMethodologically, the proposed approach is too naive. \n- Setting the trade-off parameters in unified objective function (L157-L158), e.g., gamma, and lambda, is too heuristic, and there is lack of the analysis. \n- Selecting the Top-K most confident samples by class is too naive. It would be nice to select a confident sample by improving class diversity through the application of techniques such as information maximization loss [1]. And it would be good to present comparison results with other attempts for class diversity.\n\n[1] Liang, Jian, Dapeng Hu, and Jiashi Feng. \"Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation.\" ICML 2020\n\nQUESTIONS:\n- In Table 1 and Table 2, there's a noticeable drop in some experimental results under the UL setting. I'm curious as to the cause of this decrease.\n- Within Table 2, the outcomes for the TRZSL learning setting in RESICS45 and DTD, under the \"Multimodal prompts\" category, show a decline. I'm interested in understanding why this is so.\n- In the experimental results presented in Table 3, where \"GRIP benefits adaptation even for larger image encoders,\" there seems to be a reduced rate of improvement when using a larger image encoder. I wonder the reason or analysis behind this observation.\n\nLIMITATIONS:\nI already pointed out in the weakness part.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper authors extend empirical study of self-training for the case when pseudo-labels are generated by models (CLIP) in zero-shot regime (models are trained on unlabeled data with respect to a downstream task, but can be used for zero-shot prediction for the downstream task). Authors investigate self-training in case of visual-language models (CLIP) with tuning visual/text/visual-text prompts with pseudo-labels generated in the zero-shot regime by CLIP model. They formulate also different training regimes (unsupervised, semi-supervised, transductive zero-shot) in terms of simpler supervised training where part of data (or all) just has pseudo-labels. Also it is investigated how we could bootstrap further performance in self-training doing rounds of training while updating the pseudo-labels. In the end authors show that self-training is effective and improve across different settings and datasets. Interesting results are obtained in terms of debiasing original CLIP model when self-training is done afterwards.\n\nSTRENGTHS:\n- extensive empirical study of self-training for the CLIP-based models and various configurations and settings\n- demonstrating that CLIP-style models can be used as seed model for self-training in variety of settings bringing significant boost for prompt finetuning\n- empirical demonstration how CLIP de-biasing can be done via self-training\n\nWEAKNESSES:\n- absence of any ablations on balancing between labeled / unlabeled data\n- absence of ablation on the number of samples selected for every class in self-training (why K=16? is it important? why balanced across classes?), also it is not clear what is the effect of balanced selection here especially on the de-biasing the CLIP (if de-biasing happens only because of this balancing then it is very obvious/expected result in my opinion)\n- GRIP exact explanation / how it works should be extended in the text (many ambiguities in the current description)\n- Table 1 and other Tables: I don't get why for CLIP TRZSL is similar to SSL and UL while for self-training in any combination TRZSL becomes way better than SSL and UL?\n- Missed discussion on the results that text prompt alone gives superior results compared to text+visual prompt or visual-alone prompt\n\n**Summary** There is no novelty or any impactful design of the self-training algorithm itself, though extensive empirical study is done for zero-shot learners (CLIP) used to pseudo-label data. The particular choice of hyper-parameters (like balancing pseudo-labels per class) demonstrate significant improvement for the prompt fine-tuning task as well as de-biasing original model (latter I found particularly important and interesting).\n\nQUESTIONS:\n- lines 50-53: I would not agree that VLM are trained on unlabeled data. We do mining of audio-visual pairs, so in that sense it is really unlabeled. On the other hand, yes, downstream task is different and we can use zero-shot inference to generate pseudo-label. However one could consider that task of pairing text and audio is more general than pairing class name and image. I would be here more concrete in phrasing.\n- There are a lot of repetitions of the same sentences / phrases along the text, e.g. lines 60-69 (other - lines 134-144) appear couple of times. I would reorganize the text to refer to the previous parts rather than repeat almost same ways in introduction / results / sections.\n- typo line 96: \"such as such as\"\n- lines 179 and 187 - selection of $gamma$ and $\\alpha$ seems to be equivalent providing 1:1 balancing between labeled / unlabeled data. I don't get why then expression are made different if in the end mathematically optimization is the same.\n- IFPL: what happens if for some class we don't have K samples? say no any examples are predicted as class 0 for MNIST, what happens then?\n- How about IFPL variant but when we increase K over time, still taking top only pseudo-labels? This is I assume is different from the GRIP. \n- It is not clear if in GRIP classes for pseudo-labels are still balanced, and also how do we select which part of unlabeled set to take? Do we take every iteration top pseudo-labels increasing the number of them or we just increase randomly the data size and take whatever pseudo-labels will be there? Do we still re-initialize prompt every iteration here? Do we reuse or regenerate pseudo-labels for the part of data used in previous iteration here?\n- Why we don't train continuously prompt when we re-generate pseudo-labels? Why do we not fine-tune the whole model also but only prompting (if for efficiency purpose - ok, but interesting to see if we need to finetune the whole model)?\n- lines 308-311 - this is actually known fact. E.g. for fixmatch/remixmatch we do different augmentations, and they are the key to make self-training to work, so particular type of noise in data/labels make self-training to work.\n- Figure 3 (and similar in Appendix) is not clear. I got what is x-axis only after reading it several times. Either better notation or better caption is needed. Maybe \"refers to the top x-axis (number of iterations) while .. to the bottom a-axis (amount of unlabeled data)\".\n- I still don't get why GRIP over iterations can become worse than IFPL. Also maybe it is worth to have ablation where we do IFPL but for every iteration K is increased so that we increase amount of unlabeled data involved though we do this over the most confident every time still.\n- Robin Hood effect: is it because we do balancing of classes in self-training? I guess here this plays a huge role, assuming that some classes are unrepresented in CLIP pretraining + we know that in self-training different classes have different pace of learning, so that balancing can resolve issues on under representative classes or hard ones.\n- line 345 - what is \"conventional pseudo-labeling\"?\n- Table 5: typo - UPL -> UPT\n- lines 664-665 - is it because of the CLIP bias itself and the way classes are balanced in fine-tuning?\n- lines 692-693 - maybe not surprising as we balanced examples per class?\n\nLIMITATIONS:\nLimitations are discussed after conclusion sections. Formulation sounds reasonable to me.\n\n**Update: after rebuttal and discussion with authors the score is update from 6-weak accept to 7-accept and contribution from 2 to 3.**", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The paper explores the use of CLIP for pseudo-labeling for various tasks such as SSL and on various datasets. Overall - while not being very surprising - the results can clearly outperform prior work (that is not using CLIP it has to be said) - and thus showing the potential of CLIP for such tasks.

STRENGTHS:
Clearly, the authors show what they set out to do: CLIP pseudo-labeling can improve via pseudo-labeling on a variety of tasks and datasets. That is indeed nice to see.

The positive experiments are good - doing this for a variety of tasks and on a variety of datasets.

WEAKNESSES:
While the improvements are good - they are not surprising either. CLIP is trained on a wide variety of data and thus it is clear that pseudo-labeling using CLIP should help in many cases

In my view the main weakness of the paper is that it is not showing the limits of CLIP pseudo-labeling - while most of us do not know which exact data is used to train CLIP - the datasets used seem still reasonably close.

QUESTIONS:
Overall the paper has a clear point and it does a good job making that point reasonably clear. As said above, after reading the paper I am not surprised that this can work, but I am left without an answer what the limits are of CLIP pseudo-labeling,

LIMITATIONS:
limitation discussion ok for me - except the point of the limits of CLIP pseudolabeliing as mentioned above
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the concept of prompt tuning in the context of limited labeled data. The authors propose a unified objective function that encompasses three different learning paradigms, and investigate three distinct training strategies for leveraging pseudolabels. The experimental results on six datasets demonstrate the effectiveness of the proposed GRIP strategy. The findings of this study highlight the value of utilizing pseudolabels generated by CLIP itself to enhance its performance in diverse learning scenarios.\n\nSTRENGTHS:\n* this paper is well-written with a clear structure, making it easy to understand\n* this paper extensively explores a board design space, including the prompt modality, the learning paradigm and the training strategy\n* the authors conduct extensive expriments to verify the effectiveness of this method\n\nWEAKNESSES:\n* The writing of the article is not clear in some details, such as the typo error in line 290 and inconsistencies in abbreviations.\n* please see questions listed below\n\nQUESTIONS:\n* I am little confused that does `line 217` means reinitializing the prompts at the begining of each iteration? Or does it mean only reinitializing the set of pseudolabels while the prompts are kept?\n* Is the process of pseudolabeling done online or offline? If it is done offline, is it time-consuming when there is a large amount of unlabeled data?\n* Are the abbreviations TZSL mentioned in `line 182` and TRZSL mentioned later referring to the same concept? This might be a bit confusing.\n* The experimental results in Table 1 show significant improvements compared to CLIP or other baselines. However, there is a slight decrease in some cases. Perhaps the authors could try to explain the reasons behind this? For example, it could be related to the dataset or other factors.\n* I think there might be a typo error in `line 290`. It should be Table 2 instead of Table 5.\n* Have you validate the effectiveness on datasets with more categories (e.g, ImageNet-1K), because the quality of pseudo labels might decrease when the category space get larger.\n\nLIMITATIONS:\nI think the authors have adequately discussed the limitations of their research.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}