Exploring Weak-to-Strong Generalization for CLIP-based Classification
Authors: Jinhao Li
Sarah Monazam Erfani
Lei Feng
James Bailey
Feng Liu
Venue: Accepted by TMLR ()

Abstract: Aligning large-scale commercial models with user intent is crucial to preventing harmful outputs. Current methods rely on human supervision but become impractical as model complexity increases. When models surpass human knowledge, providing accurate feedback becomes challenging and inefficient.
A novel solution proposed recently is using a weaker model to supervise a stronger model. This concept leverages the ability of weaker models to perform evaluations, thereby reducing the workload on human supervisors. 
Previous work has shown the effectiveness of weak-to-strong generalization in the context of language-only models. Extending this concept to vision-language models leverages these insights, adapting the proven benefits to a multi-modal context.
In our study, we explore weak-to-strong generalization for CLIP-based classification. We propose a method, \emph{class prototype learning} (CPL), which aims to enhance the classification capabilities of the CLIP model, by learning more representative prototypes for each category.
Our findings indicate that, despite using a simple loss function under weak supervision, CPL yields robust improvements in targeted scenarios, particularly when pretraining is limited. Extensive experiments demonstrate that our approach is effective under these settings, achieving a 3.67\% improvement over strong baseline methods.

Published in Transactions on Machine Learning Research (05/2025)
Exploring Weak-to-Strong Generaizationn for CLIP-based Classification
Jinhao Li
Sichao.i@@@@mmmebb.. edu.
School of Computing and Information Systems University of Melbournem, Austrakia
Sarah M. Erfani
surah.crfanii@wmimcchh edu.
School Computing and Information Systems University of Melbourne, Australia.
Lei Feng
fenglei@seu. edu.con
School of Computer Science and Engineering Southeast University, China.
James Bailey
baileyj@unimelb.coo edu.oo
School of Computing and Information Systems University of Melbowrne, Australia.
Feng Liu
feng. liu. @unimelll. edu.
School of Computing and Information Systems University of Melbourne, Australia.
Reviewed OpenReview: https: // openreview. net/ forum? id= quE8gDDegf
Abstract
Aligning large-scale commercial models with user intent is crucial to preventing harmful outputs. Current methods rely on human supervision but become impractical as model complexity increascs. When models surpass human knowledge, providing accurato feedback becomes challenging and inefficient. novel solution proposed recently is using a weaker model to supervise stronger model. This concept leverages the ability of weaker models to perform evaluations, thereby reducing the workload human supervisors. Previous work has shown the effectiveness of weak-to-strong generalization in the context of language-only models. Extending this concept. to vision-language models leverages these insights, adapting the proven benefits to a multi-modal context. In our study, we explore weak-to-strong goneralization for CLIP-based classification. We propose a mothod, class prototype learning (CPL), which aims to enhance the classification capabilities of the CLIP model, by learning more representative prototypes for each category. Our findings indicate that, despite using a simple loss function under weak supervision, CPL yields robust improvements in targeted scenarios, particularly when pretraining is limited. Extensive experiments demonstrate that our approach is effective under these settings, achieving 3.67% improvement over strong baseline methods.
1 Introduction
Large language models (LLMs), such as GPT 4o (Achiam el al., 2023), Claude 3 (Anthropic, 2024) and Gemini 1.5 (Reid et al., 2024), have made significant strides enhancing performance across a spectrum of natural language processing tasks. However, despite their successes, ensuring that these models align with human expectations and intentions remains formidable challenge (Burns al., 2023). Increasing the size of language models does not necessarily improve their ability to follow user intent, they still produce

---

Published in Transactions on Machine Learning Research (05/2025)
untruthful, toxic, or unhelpful outputs, indicating a lack of alignment with their users (Ouyang et al., 2022). Alignment with user intent is crucial for deploying these models effectively in practice (Bai et al., 2022). Traditional alignment techniques often rely heavily on human supervision (Christiano et al., 2017; Stiennon et al., 2020; Ouyang ct al., 2022; Glaese et al., 2022; Bai et al., 2022), requiring evaluators to provide feodback on model outputs. However, as the complexity and intricacy of model outputs increase, the feasibility and scalability of this approach diminishes (Burns et al., 2023). As a result, there a need to effectively align LLMs with human values without overly burdening human evaluators.
A recent study (Burns et al., 2023) explored novel approach known as weak-to-strong generalization to address the challenge of aligning strong models with human feedback. This strategy leveragos weaker model to supervise a more robust one, presenting a promising method to enhance model aligument. The study by Burns et al. (2023) demonstrates the effectiveness of this strong models with knowledge generated by their weaker counterparts consistently improves performance. For example, in natural language processing tasks, finctuning GPTT4 with supervision from a GPT-2-level model significantly enhances GPT-4's performance. This approach highlights the viability of weak-to-strong learning a solution for better model alignment, demonstrating that even weaker models can provide valuable guidance for improving stronger models. While the technique proves effective, applying it to Vision-Language Models (VLMs) is far from straightforward. VLMs face unique challenges in aligning complex multimodal tasks, making it essential to thoroughly explore the method's Is applicability and limitations in this context. Unlike in natural language tasks, where rext-based guidance can be more straightforward, VLMs must align both visual and textual information, making supervision from weaker models significantly more challenging. The complexity of managing two distinct modalities introes difficulties in ensuring coherent. feedback across image and text domains, necessitating a more nuanced approach when adapting weak-to-strong gencralization to VLMs. Our goal is to rigorously investigate the weak-to-strong paradigm within VLMs, as this problem extends beyond mere adaptation of previous work.
learning approach, where finetuning
In this study, we explore weak-to-strong gencralization for CLIP-based classification, recognizing it as a crucial starting point in VLMs. Existing VLMs (Radford et al., 2021; Jia et al., 2021) take classification as the basic task evaluate the alignment of images and texts. Also in our task setting, classification makes it casier for to design simulation experiments and establish a benchmark. In this context, we introduce a method called class prototype learning (CPL). CPL involves generating class prototypes that encapsulate the characteristics of each class using weak supervision. This method effectively mitigates the false signals typically generated by weak supervision, thereby showcasing superior performance. Moreover, when compared to conventional methods for adapting VLMs to downstream tasks, such as prompt tuning (Zhou et al., 2022b; Jia et al., 2022), our CPL. approach proves to be more efficient. This efficiency arises from the fact that CPL climinates the need to employ a text encoder during the fine-tuning phase. By streamlining the adaptation process, CPL offers a more resource-effective solution while maintaining high-performance levels.
We conduct extensive experiments to evaluate the performance of the proposed method, CPL, using the DomainNet dataset (Peng et al., 2019), which includes six diverse visual domains. The dataset is divided into training and test sets, and the experiment involves training weak models, generating weak supervision sets, fine-tuning strong models with weak supervision, and comparing the results to strong model training with ground truth labels. Various baselines are used for comparison. Our results show that the CPL achieves the highest average accuracy across all domains, significantly outperforming other methods. In particular, CPL shows substantial improvements for challenging domains like Infograph and handles dlomain-specific features effectively, despite CLIP's lower zero-shot performance in domains like QuickDraw. This illustrates the robustness and effectiveness of CPL in weak-to-strong generalization scenarios.
We summarize the main contributions of our work:
(i) Exploring weak-to-strong generalization for CLIP-based classification: Previous work (Burns et al., 2023; Guo et al., 2024) has shown the effectiveness of weak-to-strong generalization in L.MMs. Extending this concept to VLMs leverages these insights, adapting the proven benefits to multi-modal context.
2

---

Published in Transactions on Machine Learning Research (05/2025)
(ii) Proposing CPL:
We present method that effectively leverages class prototype representations through weak supervision to enhance the dassification performance of VLMSs such as CLIP (Radford et al., 2021).
(iii Conducting simulation experiments: We design a simulation experiment within the VLMs framework based on DomainNet (Peng et al., 2019) to study this problem, and establish benchmark in this context. Our experiment resulted in 3.67% improvement over baseline methods.
2 Related Work
Vision-language models. VLMs integrate visual and textual information, enabling multifaceted under- standing and interaction with multimodal content. CLIP (Radford et al., 2021) exemplifies this approach, leveraging contrastive learning to align images with textual descriptions effectively. This model demonstrates robust zero-shot capabilities, where it can recognize images or concepts it was not explicitly trained on. The effectiveness of CLIP and similar models, such as ALIGN (Jia et al., 2021), Flamingo (Alayrac et al., 2022), BLIP (Li et al., 2022) and Llava (Liu et al., 2023), arises from their ability generalize from vast amounts of web-collected data, learning nuanced, multinodal representations that are applicable across various lasks and domains.
Vision-language prompt tuning. Research has also locused improving prompt-based learning and fine-tuning methods, such as CoOp (Zhou et al., 2022b) and CoCoOp (Zhou et al., 2022a), which adapt VLMs more effectively to specific tasks by learning customized prompt strategies. CoOp transforms static text prompts into dynamic, Iearnable components. This allows prompts to adjust during training, aligming model responses with task-specífic needs, and improving performance, especially in zero-shot or few-shot. settings. Following CoOp, several studies Lu et al. (2022); Sun al. (2022); Derakhshani et al. (2023); Zhu et al. (2023); Gao et al. (2024) have advanced prompt tuning to enhance model performance.
Knowledge distillation. Knowledge distillation (Hinton et al., 2015; Ahn et al., 2019; Zhao et al., 2022; Jin et al., 2023) is an effective model compression technique in which a smaller, more efficient student model learns from a larger, more complex Leacher model. The conventional method for knowledge distillation involves training the student model to minimize the difference between its predicted probability distribution and that of the teacher model, often measured using Kullback-Leibler (KL) divergence. However, weak-to-strong generalization offers an alternative by having strong models supervised by weaker models.
Weak-to-strong generalization. The concept of weak-to-strong generalization, initially introduced by Burns al. (2023), presents a promising approach for aligning super-intelligent models with human values. This study emphasizes the significance of the issue and provides experimental evidence to support its feasibility. Building on this framework, Guo et al. (2024) introduces a dynamically adjusted confidence loss and demonstrates the effectiveness of their method in the context of visual foundation models. Therefore, based on those previous work, we explore the to-ston generalization for VLMs.
3 Preliminaries
In this section, we outline the preliminary studies considered in this paper.
CLIP-like vision-language models. The CLIP model (Radford et al., 2021) employs a vision encoder fision and a text encoder fiext which jointly learn map visual inputs x; and textual inputs t, into feature embeddings r; frision (x.) and r, (tyx respectively. These embeddings are projected into a shared latent space where their similarity is measured by cosine similarity, cos(r,, By maximizing the similarity of positive pairs (y), and minimizing the similarity of negative pairs sampled from the dataset, CLIP optimizes the contrastive loss function:
N
log exp(cos(Tí,, T/T) + N
N
exp(cos(r,nn Tt,)//) Ek= (sppcco((rs..55)./.
LCLP N
Σ log n=l
A=1 exp(cos(ri,,TT.))
N
3

---

Published in Transactions on Machine Learning Research (05/2025)
Class protolype learning
Test Data
photo of a {xc)
Text Encoder
Initializee
C₁
C2
Ck:
Drest
{
Inference
lext
fentoo (z))
Labeled Set
y = fu
cos
Prediction
Image Encoder
cos
Task
C
Image
1.6
0.8
-3.2
Unlabeled Data
Strong Model (CLIP)
(ni221
Strong logits
CCPL
-0.5
2.8
1.2
Trainable
Frozen
Weak Model
Weak logits
Figure Overview of the weak-to-strong process for enhancing strong model performance using weak model supervision. Unlabeled data from given task is fed into both strong model (CLIP) and weak model. The strong model uses image encoder generate image features (ri), which are compared with learnable class protolypes (Ci.:, C2....., Ck.)) through cosine similarity to produce strong logits. Concurrently, the weak model generates weak logits from the same data. Our alignment loss (LePl in Eq. 5) is computed between the strong logits (based on the prototype matrix C) and weak logits. For test data, the image features (r() extracted from the strong model f are compared with the learned prototype matrix C* to make predictions, aiming to improve the strong model's classification performance in the given task.
where N is the batch size, (in; denotes the index pairs of positive examples, and 7 is temperature parameter. This contrastive learning approach enables CLIP to achieve remarkable zero-shot classification performance across various tasks, leveraging its pretrained representations and zj without task-specific training.
CLIP linear probs. The standard method to fine-tune pre-trained VLMs, e.g., CLIP, involves training linear classifier on the feature representations extracted from these pre-trained models. This approach mirrors how Radford et al. (2021) evaluated the transferability CLIP, treating pre-trained models primarily feature extractors. This method is generally classification heads need to be trained. The formula is:
efficient. because only the parameters of the additional
(1) where Minsoon( (x) denotes the feature representation extracted from the pre-trained CLIP model for an input æ, W represents the weights of the linear classifier, b is the bias term, and á is the predicted probability distribution over the classes. The parameters W and b are learned during the training process the downstream task's labelled data. This approach leverages the rich feature representations Iearned by CLIP during its pre-training phase, enabling efficient and effective adaptation to new tasks with minimal additional training. In this method, fiext not used during this training process.
p softmax(W (W.. (xr) + b)
CLIP prompt tuning. A recent mainstream approach more effectively adapt VLMs involves learning customized prompts (Zhou et al., 2022b). This method fine-tunes the input prompts that guide the model's attention and feature extraction processes. Mathematically, this approach can be represented as:
f cos( (otsoon (æ)), fiexc ((t (
(2)
4

---

Published in Transactions on Machine Learning Research (05/2025)
where frision (x) denotes the feature representation extracted from the vision encoder for an input æ, and Rext (f4 denotes the feature representation extracted from the text encoder for a set of prompts (4) where t, = { U2...., Uk. {classmame:)) with U1, U2,.... representing the learned prompt vectors and {classname;) being the target class name. The function cos represents the cosine similarity between the vision and text feature representations. The parameters of the prompt vectors 1/ ..,, Uk are learned during the training process, enabling the model to better align the vision and language features for the specific downstream task. Unlike in the previous method, fiex is utilized during the training.
4 Weak-to-Strony Learning for CLIP-based Classification
In this section, we first introduce the problem formulation and describe our proposed method CPL. Additionally, the overall procodure is shown in Figure 1, and the algorithm shown in Algorithm 1.
Problem formulation. In this paper, we consider a scenario involving a weakly pre-trained model fr and strongly pre-trained model fe, where fe generally exhibits better generalization due to more parameters extensive training data. In this paper, we consider f to be VLM model, c.g., CLIP (Radford et al., 2021), while ! is a vision model. Given a target lask, we have a dataset consisting of unlabeled samples D = (>,);= m labeled test samples' Drest = =[<]", y:) and label set y = = where k represents the number of category and each yi represent one semantic label. We apply the weak model f to D to generale predictions, which we refer to as weak supervision, represented by a weakly supervised dataset D (x:)) The task of weak-to-strong generalization to fine-tune the strong model f with the weakly supervised dataset D to enhance its classification capabilities on the test dataset Drest-
Class prototype learning. Empirical evidence (Figure 2a and 2b) indicates that previous VLM fine-tuning approaches, including lincar probs and prompt tuning, applied in weak-to-strong generalization often result in strong models overfitling to the weak models. Consequently, this leads to the strong models performing close to the weak models on test sets. To address this, we aim to learn the set of class prototypes as a matrix C e Rxxd, where & is the total number of classes and each row C.,, E R Ixd is the class prototype for cach class i based on the feature embeddings of training images belonging to that class, which encapsulate the characteristies of each class. The prototype representation C., for each class i can be initialized by the text embedding corresponding to a textual description of the class label. For instance, C, could be initialized with the text embedding of 'a photo of (label}' extracted by CLIP text encoder, where label represents the class label name.
When presented with an input image x E Rixwxe in D, we compute its visual feature embedding (asiou())) where frision (z) E Id Then, the cosine similarity between the image embedding fvision (x) and each class prototype C, is calculated as the logits za. Mathematically, the unnormalized logit for i-th class (i.e., the i-th clement in 2) regarding æ computed as:
Ciee vision (x)
2 (Ci.,, x))
(3)
||Cr,: (isiion (x)
where this cosine similarity operation measures the alignment between the image and class centroids, providing measure of the image's association with each class. Subsequently, a softmax function can be applied to the logits to obtain class probabilities.
Weak-to-strong alignment. The ultimate aim of weak-to-strong alignment is to elicit the capabilities of much stronger model using weak supervision from weaker model (Burns et al., 2023). Unlike knowledge distillation, where the stronger model serves as the teacher and the weaker model as the student, weak-to- strong alignment reverses these roles. Here, the weaker model acts as the teacher guiding the stronger model. A straightforward approach to this challenge is to use knowledge distillation methods (Hinton et al., 2015) to make the strong model's behavior agree with that of the weak model. Most logit-based KD methods utilize the KL divergence, which quantifies the amount of information lost when approximating one probability
This test set only used for testing phrase.
5

---

Published in Transactions on Machine Learning Research (05/2025)
Algorithm 1: Weak-to-strong Generalization for VLMs
Input An unlabeled set: D = {=,))_ii a test set: Deast = { ) a label set: y = { i== strong model: (-); learnable class prototypes: C' E Rexd; maximum epochs: Tiaxi alignment loss function: (cP.(:)).
1: Obtain frision (.), fieee()) (+) from f*(:):
2: Initialize class prototypes C where Ci = ftext ("a photo of a {y:}");
for 1 to Taax do
3: Fetch mini-batch in D;
4: Compute the average loss -= [8] EmEs (crL(C,
5: Update class prototypes C' using Adam (Kingma Ba, 2014) and the average loss C;
end
6: Get learned class prototypes C";
7: Obtain test feature embeddings {r:)==1 {Svision (x Datne
8: Compute predicted logits {z, ) where 2; cos( C", ri);
9: Compute prediction y = (arg max, 2 1im
Output
distribution with another. Therefore, for each æ in D, given the logits of the weak model, fm and those of the strong model xt (using Eq. 3), we convert them into the softened probability vector pm and p². The i-th value of p" or pd computed by softmar function with a temperature hyperparameter т, which is denoted by
p. (x) = (2)(2(1))/))
Σ exp(2) (CC.....)) x) = exp
(4)
Σラ exp( (C,,,
Thus, the loss value each x D realized by minimizing the KL divergence between softened probability vectors of weak and strong models, which definod as:
P. (C..) 1 (C., x))
= (I, (1=)
Σ pe (Ci,)) log
(5)
We demonstrate the overall algorithm in Algorithm 1. The algorithm for weak to-strong generalization in VLMs begins by initializing class prototypes using text embeddings from the strong model. During training, mini-batches of unlabeled data are processed to obtain feature embeddings and generate logits from both the strong and weak models. The alignment loss between these logits computed update the class prototypes iteratively. Once training complete, the learned class prototypes are used compute feature embeddings from the test data, generate logits through cosine similarity, and predict the labels by selecting the class with the highest logit value.
5 Experiments
In this section, we evaluate the performance of our method by a series of experiments and varions ablation studies. The implementation details can be found in Appendix
Datasets. In exploration of weak-to-strong scenarios, turn to the challenging and relatively large dataset: DomainNet (Peng et al., 2019). Comprising six diverse domains, each housing 345 categories of common objects, DomainNet offers rich landscape for analysis. These domains encompass a range of visual styles and sources: Clipart, leaturing a collection of clipart images; Infograph, presenting infographic images with specific objects; Painting, showcasing artistic renditions of objects in the form of paintings; Quickdraw, housing drawings from the popular game 'Quick Draw!' by worldwide players; Real, encompassing photographs and real-world images; and Sketch, containing sketches of various objects. Refer to Table for detailed statistics into each domain.

---

Published in Transactions on Machine Learning Research (05/2025)
Experimental setup. In our experiments, each domain within the DomainNet dataset is treated as an individual task, resulting in a total of 6 tasks under consideration. To investigate weak-to-strong generalization within the setting of VLMs, we design these steps to simulate the problem:
(1) Dataset splitting: Referring to Table 2, each domain is divided into a training set Drrain and a test set Drest- The test set Dust is further partitioned into Phold and D D'eaa: comprising 80% and 20% of Drest respectively. (2) Create the weak model: The training data Puain is utilized to fine- tune the weak model, employing ground truth labels. Evaluation occurs in Postt termed as weak performance. (3) Weak supervision set generation: The weak supervision set Phola is generated by the weak model from Phold, replacing ground labels with logits produced by the weak model. (4) Strong model training with weak supervisor: Initially, Photd is split into 80% and 20% portions for strong model fine-tuning and parameter tuning, respectively. The strong model is then fine-tuned in the holdout training set. The final performance is assessod Prest: labeled as st-rong performance. (5) Strong model training with ground truth labels as cciling: Finally, the strong model undergoes fine-tuning on Dhold (with ground truth labels) to represent strong ceiling performance.
Baselines. In exploring the weak-to-strong problem within the VLM setting, we investigate different fine-tuning strategies. Initially, Radford et al. (2021) assessed CLIP's transferability via linear probs (LP) across many datasets. Subsequent research focused on tertual prompling (TP) (Zhou et al., 2022b), where a learnable prompt is learned from a small target dataset. This method is data-efficient and demonstrates good gencralization effects. Prompt tuning has emerged as a popular method for adapting VLMs to downstream tasks (Wu et al., 2023). Thus, we adopt linear probs and prompt tuning as our foundational fine-tuming strategies within the realm of weak-to-strong generalization. In addition, we compare our method with the following learning strategies:
(1) Cross entropy (CE): Utilized in studies by (Radford et al., 2021; Zhou et al., 2022b), cross-entropy measures the disparity between one-hot ground truth label distribution and model prediction probability. It serves as a straightforward baseline for this task.
(2) Knowledge distillation (KD) (Hinton et al., 2015) transfer knowledge from a strong model to a smaller one, serving as a fundamental baseline due to its simplicity and effectiveness.
(3) Auxiliary confidence loss (AuxConf) is proposed by Burns et al. (2023), which excels in balancing direct learning from the weak model with the inherent capacity of the strong model. (4) Adaptive Confidence loss (odl) is introduced by Guo et al. (2024) that dynamically adjusts weights based on confidence levels, enabling the strong model to discern when to prioritize its predictions or follow the guidance of the weak model.
Implementation details. In this section, we provide an overview of the implereentatio details regarding our proposed method and comparative baseline methods on simulation experiments. The code is mainly based on Pytorch and the Huggingface library. We employed ResNet and ViT as the weak model and CLIP as the strong model, for our task. The evaluation is performed in five random seeds. During training, we used a test batch size of 2048 for evaluation. The weak model was trained for 3 epochs with batch size of 512 and learning rate of 1e-3, whereas the strong model underwent 10 epochs with the same batch size and a learning rate of 1c-2. The learning rate was adjusted dynamically, and a warm-up ratio of 0.1 was utilized. We also ensured the loading of the best model at the end of training based on the validation set, All our experiments are conducted using a single A100 GPU with 40GB of memory, supported by 8 CPU workers and 64GB of RAM.
Experiment results. The results presented in Table 1 provide a comprehensive evaluation of various methods across multiple domains within the DomainNet dataset. Each method's efficacy is assessed based on its accuracy in six distinct domains: Clipart, Infograph, Painting, Quickdraw, Real, and Sketch. Notably, our proposed method (CPL) exhibits remarkable performance, achieving the highest average accuracy of 64.74%. This significs substantial improvement over bascline methods, with CPL, outperforming the best-performing bascline by notable margins, showcasing gains of 2. 2.41%, 10.11%, 3.19%, -0.19%, 1.41%, and 3. 3.67% across respective domains.
7

---

Published in Transactions on Machine Learning Research (05/2025)
Table 1: Performance on DomainNet datasets across different methods and styles. This table showcases the results accuracy (%) of various methods on different styles within the DomainNet datasets, including Clipart, Infograph, Painting, Quickdraw, Real, and Sketch. The average performance across all styles is also listed. The compared methods include CE+LP, KD+LP, AuxConf+LP, AdaptConf+IP, CE+ KD+TP, AuXConf+TP, and AdapIConf+TP, with CPL yiekding the highest performance in most categories. The final row, A, represents the improvement margin of CPL over other methods. CPL is used as the strong ceiling performance, which is the best among the IP,, TP, and CPL
DomainNet
Method
Avg.
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
Weak
67.15
31.71
67.90
46.70
85.10
52.26
Strong Ceiling
74.27
50.84
72.24
49.92
85.34
66.64
CE+LP
66.97
30.55
64.90
45.59
82.69
55.28
57.66
KD-LP
70.69
35.78
68.28
48.15
83,66
59.88
61.07
AuxConf+LP
67.41
18.37
66.92
30.59
84.00
56.79
54.01
AdaptConf+IIP
70.68
33.78
67.86
47.80
83.84
60.18
60.69
CEFTP
62.02
29.25
62.6 62.68
44.82
81.16
52.51
55.41
KD+TP
69.57
36.03
68.37
47.34
83.70
59.93
60.82
AuxConf+TP
69.20
20.97
68.61
43.60
83.92
58.49
57.47
AdaptConf++PP
69.97
35.96
68.18
47.42
83.59
59.95
60.85
CPL (Ours)
73.10
46.14
71.80
47.96
85.41
64.01
64.74
A
2.41
10.11
3.19
-0.19
1.41
3.83
3.67
In the domain of QuickDraw, it is evident that CLIP demonstrates a lower zero-shot ability, suggesting significant disparities between the data distribution in QuickDraw and the CLIP training data. This observation underscores the challenge of generalizing CLIP to the QuickDraw domain effectively. Surprisingly, in such cascs, the straightforward KD approach emerges as the most effective method, outperforming more sophisticated techniques. This phenomenon suggests that the inherent structure of the KD method enables it to leverage available information optimally, leading to superior performance despite the substantial dissimilarities between the CLIP and QuickDraw domains.
In the context of the Infograph domain, the weak model exhibits notably inferior performance compared to all other domains. Conversely, our proposed method demonstrates the most substantial performance improvement, showcasing a significant gain in accuracy as compared to both the weak model and other competing methods. This highlights the effectiveness of our approach in addressing the challenges specific to the Infograph domain, where the weak model struggles to generalize effectively. The considerable performance gain achieved by our method underscores its ability capture and leverage domain-specific features, resulting in improved accuracy and robustness in handling Infograph data.
As shown in Table 4, our BLIP-based method (CLIP (CPL)) consistently outperforms all baselines across the Nom datasct. It achieves the highest average accuracy, with noticcable improvements over strong adaptation methods such as Adapt Conf+TP and AuxConf+TP. These results demonstrate the effectiveness of our approach leveraging BLIP representations for improved domain generalization.
A Ablaioon on different CLIP variants. Table compares our method against a wide range of baselines using different CLIP backbones (ViT-B/16 and VIT-L/16) on the Clipart domain. Our approach achieves the best performance across both variants. These results confirm the effectiveness and robust of our method under both low- and high-capacity model settings, outperforming bascline methods.
Ablation on OfficeHome. As shown in Table 6, our method achieves the best performance across all four domains of the OfficeHome dataset, outperforming existing baselines by a clear margin. Specifically, attains an average accuracy of 75.36%, improving over the best prior method (AdaptConf+TTP) by 2.16%. The consistent gains across domains Art (+2.22%), Clipart (+2.20%), Product (+1.67%), and Weal-World (+2.57%))hightt highlight the robustness and generalization ability of approach.
8

---

Published in Transactions on Machine Learning Research (05/2025)
Table DomainNet statistics. This table provides statistics for the DomainNet dataset (Peng et al., 2019) across different styles: Clipart, Infograph, Painting, Quickdraw, Real, and Sketch. It includes the number of classes (#Classes), the number of training samples (#Triin,, the number of test samples (#Test), and the total number of samples (#Total) for each style. Each style has 345 classes, with varying numbers of training and test samples.
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
Classes
345
345
345
345
345
345
Train
33525
36023
50416
120750
120906
48212
# Test
14604
15582
21850
51750
52041
20916
Total
48,129
51,605
72,266
172,500
172,947
69,128
Table 3: Performance Comparison of Different Weak Models. This table presents the results in accuracy (%) of various methods applied to weak models, including Resnet-18, Resnet-20 Resnet-34, Cvt-13, and Convnext-tiny-224. The average performance across all models is also provided. The strong ceiling performance is given for reference. The methods compared CE+LP, KD+LP, AuxConf+LP, AdaptConf-LP,, CE+TP, KD+TP, AuXConf+TP, and AdapIConf(+TP, with CPL showing the best performance. The final row, A, indicates the improvement margin of CPL over other methods.
Weak Models
Method
Avg.
Resnet-18 18
Resnet-26
Resnet-34
Cvt-13
Convnext-tiny-22
Weak
55.22
59.96
51.33
69.47
Strong Ceiling
74.27
CE+LP
64.28
64.19
64.72
62.19
69.5
64.98
KD+LP
66.80
67.26
68.53
65.89
71.53
68.00
AuxConf
67.02
66.21
62.52
71.10
66.61
AdaptConf
67.72
68.13
68.83
66.95
71.42
68.61
CE+TP
62.22
63.35
64.04
61.02
67.71
63.67
KD+TP
65.71
66.43
67.34
64.50
69.31
66.66
AuxConf+TP
66.39
67.04
67.47
66.04
69.56
67.30
AdaptCouf+TP
65.91
66.69
67.42
65.31
69.04
66.87
CPL. (BLIP)
72.25
71.84
72.06
71.91
72.47
72.11
A
4.53
3.71
3.23
4.96
0.94
3.50
Ablation on different weak supervision. Table illustrates our approach to various forms of weak supervision across different models in detail, such as Resnet models (He et al., 2016) (Resnet-18, Resnet-26, Resnet-34), Cvt-13 (Wu et al., 2021), and Convnext-tin--22. (Liu al., 2022). The experiment was conducted the DomainNet Clipart domain, revealing diverse range of performances from different weak models, with accuracy scores spanning from 55.2% to 69.47%. Notably, our method consistently achieved the best. weak-to-strong generalization performance among all the weak models tested, closely approximating the strong ceiling performance, which was benchmarked at 74.27%.
Our method's superior performance is evident across various weak supervision techniques. The results show that while other methods improved performance varying degrees, none matched the consistency and high performance of our method. For instance, our approach significantly performmdd the baseline weak supervision models, achieving top accuracy scores such as 72.25% for Resnet-18, 71.84% for Resnet-26, 72.06% for Resnet-34, 71.91% for Cvt-13, and 72.47% for Convnext-tiny-224, On average, our method achieved a performance gain of 3.5%, underscoring its superior ability to enhance model accuracy through improved weak supervision techniques.
The performance gains highlight the incremental improvements our method brings compared other approaches. These improvements range from 0.94% to 4.96%, demonstrating our method's ability consistently push model performance closer to the strong ceiling benchmark. From this table, it is evident.

---

Published in Transactions on Machine Learning Research (05/2025)
Table 4: Performance on DomainNet datasets across different methods and styles. This table showcases the results accuracy (%) of various methods on different styles within the DomainNet datasets, including Clipart, Infograph, Painting, Quickdraw, Real, and Sketch. The average performance across all styles is also listed. The compared methods include CE+LP, KD+LP, AuxConf+LP, AdaptConf+IP, CE+ KD+TP, AuXConf+TP, and AdaptConf+TPP, with CPL yiekding the highest performance in most categories. The final row, A, represents the improvement margin of CPL over other methods. CPL is used as strong ceiling performance, which the best among the I.P, TP, and CPL.
DomainNet
Method
Avg.
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
Weak
60.23
28.45
63.12
42.98
80.56
50.34
Strong Ceiling
75.12
52.78
73.65
50.89
86.12
67.45
CE+LP
61.78
27.69
61.34
40.21
81.23
52.10
54.06
KD-LP
68.12
33.47
65.89
46.21
83.02
57.45
59.36
AuxConf+LP
65.89
19.45
63.78
32.10
82.34
54.67
53.37
AdaptConf+IIP
69.23
31.89
66.14
44.75
83.47
58.12
58.93
CEFTP
59.67
27.12
60.45
39.88
79.89
50.76
52.96
KD+TP
67.34
34.23
66.10
45.12
82.78
58,23
59.30
TuxConf+TP
66.78
21.10
65.23
41.89
82.90
55.89
55.63
Tdap
67.89
33.12
65.45
44.12
83.10
57.67
58.56
CPL (Ours)
72.45
45.23
70.12
47.89
85.23
63.45
64.06
A
3.22
11.00
3.98
1.68
1.76
5.22
3.84
Table 5: Comparison across CLIP variants (VIT-B/16 and VIT-L/16) on the Clipart domain.
Accuracy (%) is reported for each method. Our method achieves the best performance across both variants, confirming its robustness.
Backbone Weak
Strong Ceiling
CE L.P
KD L.
AuxConf L.P.
AdagstConf/ LP
CE /PP
KD| TP
AuxConf TP
AdaotCoonf TP
CPL (Ours)
VTT-D/16
76,80
67.89
71.12
68.56
70.12
72.94
13.78
75.12
VIT-L/16
00:78
80.12
60.34
71.12
73:23
75.34
72.45
74.23
76.45
77.89
79.45
Comparison of Train/Accuracy Metrics Over Steps
Comparison of Test/Accuracy Metrics Over Steps
0.7
0.6
Accuracy (9%)
0.5
Accuracy
Colling Performance ly=
Perforance (y-0 6713)
AurCordsTP
0.
AuxCrtfTTP
OuTs AZCCONtLLP
Ours
0.1
AUxCont+LF
Step (log scale)
Step (log scale) (b) Test Accuracy.
(a) Train Accuracy.
Figure 2: Comparison of train and test accuracy metrics over training steps. It shows the comparison of train (a) and test. (b) accuracy metrics for different methods include AuXConf+TP, Ours, and AuxConf+LP, "Ours" demonstrates the highest accuracy, nearing the ceiling performance (y = 0.7427) and surpassing weak performance (y = 0.6715) in both the training and testing phases.
training steps. Methods
that weak-to-stee teak-to-strong generalization is feasible the VLMs setting. By utilizing supervision from weak models, our strong model has attained results that close to the ceiling performance.
10

---

Published in Transactions on Machine Learning Research (05/2025)
Table Performance on OfficeHome domains (Art, Clipart, Product, Real-World) Accuracy (%) is reported for each domain and averaged across them. Our method (CLIP (CPL)) achieves the best results across all domains.
Method
Art
Clipart
Product
Real- World
Avg.
Wcak
54.78
50.45
63.12
66.95
Strong Ceiling
79.10
65.34
85.01
87
CE+LP
62.10
60.12
72.05
76.90
KD+LP
63.80
61.55
74.45
78.22
69.00
AuxConf LP
65.23
75.78
79.90
ConptConf
66.89
62.78
77.01
81.12
71.45
CE+TP
64.89
61.45
75.56
79.05
70.11
KD+TP
66.23
62.56
76.45
80.56
71.02
AuxConf+TP
67.89
64.12
77.89
81.78
72.42
Adapt
68.23
64.78
78.67
83.10
73.20
CLIP (CPL)
70.45
66.98
80.34
85.67
75.36
2.22
1.67
2.57
2.16
Table Average Performance Comparison of Different Tuning Methods in Accutacy (%).
Method
Performance
Text Encoder
70.34
C
74.42
Ablation on different tuming methods. We have conducted an ablation study to compare the performance of tuning versus tuning the text encoder. The results have been shown in Table 7. Our findings indicate that tuning C yields better performance than tuning the text encoder. The study by Wu et al. (2023) shows that prompt tuning for VL.Ss is more robust to noisy labels compared to fine-tuning.
Analysis of method. In Figures 2a and 2a, we demonstrate the training and test. accuracy over training steps for our method compared to two baseline methods, AuxConf+TP and AuxConf+LP. The training accuracy plot shows that all methods eventually converge our method shows rapid and consistent improvement, achieving high training accuracy more quickly than the other methods. The baseline methods, AuxConf+TP and AuxConf=L.P, also improve but at different rates, with AuxConf+TP showing a steadier progression and AuxConf+LP catching up later the process. In the test accuracy plot, the differences between the mothods become more pronounced. While AuxConf+ and AuxConf+LP exhibit similar weak performance levels, struggling to surpass certain threshold, our method showcases significantly better performance. It not only achieves higher test accuracy but also maintains this performance consistently over the steps, closely approaching the ceiling performance.
accuracy of around 0.6. Specifically,
6
Limitation
Since the core problem aim to address in this research has not yet emerged, we currently lack access to superintelligence models. Although our experiments rely simulations, these simulated scenarios do not fully replicate the complexities of the actual challenge we anticipate. Consequently, there exists a significant gap between our simulated experiments and the real-world problem. This discrepancy implies that the methods demonstrating success in our current simulations may not necessarily prove effective when applied to the final real-world task. Therefore, while our current research provides valuable insights and progress, it remains crucial to acknowledge these limitations and continue refining our approaches better align with the ultimate goal of weak-to-strong alignment.
11

---

Published in Transactions on Machine Learning Research (05/2025)
7 Conclusion
In conclusion, traditional alignment techniques for LLMs, which rely heavily human supervision, such RLHF, face significant challenges due to the intricacy of model outpuls and the inelficiency of requiring substantial human feedback. To address this, a novel approach has recently been proposed where weaker model supervises a much stronger one. Extending this concept to VLMs leverages these insights, adapting the proven benefits to a multi-modal context. Hence, we introduced a method called CPL, which effectively enhances the classification capabilities of VL.Ms with weak supervision. Our simulation experiments validate the effectiveness of this weak- weak-to-strong approach. Extensive experimental results demonstrate that our method significantly improves performance across various benchmarks. These results underscore the potential of weak supervision powerful tool in the alignment, offering promising avenue for future research and application.
Acknowledgments
This research was supported by The University of Melbourne's Research Computing Services and the Petascale Campus Initiative. FL supported by the Australian Research Council (ARC) with the grant number DE240101089, LP240100101, DP230101540 and the NSF&CSIRO Responsible Al program with grant number 2303037.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almcida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint wrXiv:2303.08774, 2023.
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9163-99 9171, 2019.
Bean-Bapliste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Tain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:23716 1--3766--3776, 2022.
AI Authropic. The claude model family: Opus, sonnet, haiku. Claude-3 Model Card, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. ar preprint ar arxi::2204, 055662, 2022.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arxiv:2312 2...322.0000 2.0990, 2023.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in nearal information processing systems, 30, 2017.
Mohammad Mahdi Derakhshani, Enrique Sanchez, Adrian Bulat, Victor G Turrisi da Costa, Cees GM Snock, Georgios Tzimiropoulos, and Brais Martincz. Bayesian prompt learning for imagc-languago model generalization. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pp- 15237-15246, 2023.
Peng Gao, Shijic Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better languae models with feature adapters. International Journal of Computer Vision, 132(():581150)) 2024.
12

---

Published in Transactions on Machine Learning Research (05/2025)
Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arxio preprint arXiw:2209.143: 14375, 2022.
Jianyuan Guo, Hanting Chen, Chengeheng Wang, Kai Han, Chang Xu, and Yunhe Wang. Vision super- aligument: Weak-to- generalization for vision foundation models. arxiv preprint or 4rx 03749, 2024.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference on compuler vision and paltern recognilion, pp. 770 778, 2016.
GeofTrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge a neural network. arxie preprinl arXiv:1500 503.02591, 2015.
Chao Jia, Yinfci Yang, Ye Xia, Yi-Ting Chen, Zarana Parckh, Hicu Pham, Quoe Lc, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text. supervision. In International conference on machine learning, pp. 1904-4916. PMLR, 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference Computer Vision, pp. 709 727. Springer, 2022.
Ying Jin, Jiaqi Wang, and Dahua Lin. Multi-level logit distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp.
2023.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint wrXiv:141269888 2014.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-inage pre-training for
unified dision-language understanding and generation. In International conference on machine learning, pp. 12888-12904 PMLR, 2022.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310, 03744, 2023.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pp. 11976, 2022.
Yuming Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 5206 5215, 2022.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feodback. Advances in neural information processing systems, 35:27730-27744, 2022.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Sacnko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406-1415, 2019.
Alcc Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.
PMLR, 2021.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicap,, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXie preprint (r3.::2403.055:00 2024.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Newral Information Processing Systems, 33:3008 3021, 2020.
Ximeng Sun, Ping Hu, and Kate Saenko. Dualcoop: Fast adaptation to multi-label recognition with limited annotations. Advances in Neural Information Processing Systems, 35:30569-3000 305822 2022.
Cheng- En Wu, Yu Tian, Haichao Yu, Heng Wang, Pedro Morgado, Yu Hen Hu, and Linjie Yang. Why is prompt tuning for vision-language models robust to noisy labels? In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 15488 15497, 2023.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In Proceedings of the LEEE/CVE international conference on computer vision, pp. 22-31, 2021.
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CV /VF Conference on compuler vision and pattern recognition, pp. 11953 11962, 2022.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision- language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16816 16825, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Lin. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337 2348, 2022b.
Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang, Prompt-aligned gradient for prompt tuning. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 15659-15669, 2023.
14


Reviews:
[{"review_id": "I63C73VGFr", "reviewer": "Reviewer_dHPD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength and weakness\n\n### Strength\n1. In section 4, the author firstly propose the overfitting problem existing in Weak-to-strong generalization when adapting the fine-tuning and prompt-tuning method. This experiment is essential and convincing which result in the propose of CPL.\n\n2. The CPL initilize the class prototype initialized by the text embedding corresponding to a textual description of the class label and train the class prototybe by by minimizing the KL divergence between softened probability vectors of weak and strong models.\n\n3. The algorithm 1 is very clear and convincing.\n\n4. The main experiments are demostrated in Table 2 and its CPL method demostrate the advancement of the method.\n\n\n### Weakness\n1. This paper is based on [1], however, its main methods are based on the pre-trained large language model. Recent researches on multi-modal large language models (VLMs) such as BLIP, LLaVA has the similar architecture/language-based compareing to CLIP. Thus, why this paper dose not dig this phenonmeno in those models? \n\n2. Lack of comprehensive experiments: in the section 5, it uses the ResNet and ViT-based CLIP as the weak and strong models while there are various version of CLIP models such as ViT b/16, ViT b/32 and ViT L/16. Thus, using the gereral ResNet and ViT is not so convincing. Probably, there exists a more suitable comparison set for the expriments.\n\n\n\n[1] Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390, 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## Summary Of Contributions\n1. This paper is based on [1] where it proposes viability of weak-to-strong learning as a solution for better model alignment, demonstrating that even weaker models can provide valuable guidance for improving stronger models. This paper aims in extending this method into the area of VLM. \n\n2. In order to do that, this papaer proposes class prototype learning (CPL) which involves generating class prototypes that encapsulate the characteristics of each class using weak supervision.\n\n3. This paper shows that the CPL achieves the highest average accuracy across all domains, significantly outperforming other methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "nothing particular to be changed", "broader_impact_concerns": "ther is no Broader Impact Concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MWetnarKlf", "reviewer": "Reviewer_fvRS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The method achieves strong performance on the DomainNet dataset.\n2. The work seems to be the first to explore weak-to-strong generalization for the CLIP model\n\nWeaknesses:\n\n1. There is a lack of novelty in this method. Prior work has already introduced the use of text encoders to build prototypes. In that regard what would be the unique contribution of this method specifically in designing the prototypes?\n\n2. There are multiple prior works that have used knowledge distillation with logits from a teacher model to enhance the performance of a CLIP model [4,5,6]. Again I feel there is a lack of novelty in this aspect.\n\n3. The results are limited to only domain net. For evaluating CLIP on downstream classification tasks, it is standard practice to evaluate a diverse range of datasets. If the method improves the generalization of CLIP, it should be shown through a diverse range of out-of-distribution datasets. Right now the evaluation is essentially in-distribution. Evaluations commonly used include training on ImageNet and evaluating on StanfordCars, DTD, OxfordPets, SUN397, EuroSAT, Food101, Caltech101, Flower102, Aircraft, and UCF101. Also, evaluations include training on Imagenet and then evaluating ImageNet variants such as ImageNet-V2, ImageNet-A, ImageNet-R, and ImageNet-Sketch.\n\n4. The overall setting of the paper is quite unclear. Labels are available to train a weak model but not a strong model. If labels are available anyway, why not just use D_train to train the strong model? Why does one even have to go through the whole process of training a weaker model on D_train, then producing logits on a smaller set of unlabelled data, and then using those logits to train the strong model? Just use the D_train to train the strong model if labels are available on it anyway.\n\n[1]\tY. Wang, et. al., ‘Exploring Vision-Language Models for Imbalanced Learning’, IJCV, 2023.\n\n[2]\tJ.-X. Shi, et. al., ‘Parameter-Efficient Long-Tailed Recognition’, arXiv preprint, arXiv:2309.10019, 2023.\n\n[3]\tJ. Li, et. al., ‘Masked Unsupervised Self-training for Label-Free Image Classification’, in ICLR, 2023.\n\n[4]\tM. U. Khattak, et. al., ‘Self-regulating Prompts: Foundational Model Adaptation without Forgetting’, in ICCV, 2023.\n\n[5]\tT. Huang, et. al., ‘Unsupervised Prompt Learning for Vision-Language Models’, arXiv preprint, arXiv:2204.03649, 2022.\n\n[6]\tZ. Li, et. al., ‘Promptkd: Unsupervised prompt distillation for vision-language models’, in CVPR, 2024.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work aims to improve the classification performance of CLIP models using a technique termed as CPL (Class Prototype Learning), where a strong CLIP model is enhanced using a weaker model given unlabelled training data. First, several representative prototypes are initialized using a text encoder, which is then used to calculate the logits of the strong model. The prototypes are then optimized by aligning the strong logits with the weak ones. Results are presented on the DomainNet dataset.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It is important to specify what the novelty of each part of the method is. For example, the prototypes based on the text encoder are already proposed in previous work. Similarly, knowledge distillation to improve CLIP through a teacher model is also quite extensively used already. So the authors need to exactly specify what has been done already, what they use as is from previous work, and what they contribute in each aspect of the method.\n\n2. The results are quite limited, and a departure from the standard benchmarks widely used to show the generalization of CLIP. For example, rather than creating new splits out of DomainNet, authors could have simply worked with ImageNet. They can train the weak model on a subset of the imagenet trainset. Then utilize the same remaining train set as the hold set. And then evaluate all the standard benchmarks used by various other works (datasets mentioned in the previous \"weaknesses\" section).\n\n3. The authors need to justify the overall setting itself. Why would one want to use a holdout set with logits from a weak model, when the training labels for D_train are already available? I believe this requirement limits the application of the method in a real-world setting.\n\n4. A minor thing but it should be made clear from the start the work only focuses on CLIP. VLMs now include works like Llava or even generative models such as text-image diffusion models. It is important to use specific terminology to avoid overstating claims.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BhaM25dxCu", "reviewer": "Reviewer_Nvqx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n**Topic**: The paper explores a relatively new area, weak-to-strong generalization, specifically within Vision-Language Models (VLMs). Extending this concept from language models to VLMs is a valuable contribution.\n\n**Class Prototype Learning (CPL) Method**: The paper proposes a novel method, CPL, which aims to improve classification in CLIP models by learning representative prototypes. The method is shown to be more efficient than traditional prompt tuning as it eliminates the need for a text encoder during fine-tuning.\n\n**Experimental Results**: The experiments on the DomainNet dataset demonstrate that CPL achieves significant improvements over baseline methods, particularly in challenging domains. The 3.67% improvement is a quantifiable result.\n\n**Weakness**\n\n**CLIP-Based Task**: I wonder whether you can extend your framework to more tasks, instead of restricting the task to CLIP classification.\n\n**Simulation Experiments:** The study relies on simulation experiments within the VLMs framework based on DomainNet, which may not fully reflect real-world scenarios.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigated one task in weak-to-strong generalization. Specifically, the contribution of the paper lies in three folds:\n\n(i) Exploring weak-to-strong generalization for CLIP-based classification\n\n(ii) Proposing CPL that effectively leverages class prototype representations\nthrough weak supervision to enhance the classification performance of VLMs.\n\n(iii) Conducting simulation experiments and gaining outstanding performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the **weakness** part. I appreciate it if you could do more experiments and extend your framework to much broader tasks.", "broader_impact_concerns": "I do not have any concerns on this side.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission was reviewed by three experts who provided constructive suggestions and comments. The authors responded in detail to each point raised. After carefully reviewing the discussion, the reviewers’ comments, and the submission itself, I believe this work will be of interest to the TMLR community, as it represents one of the earliest attempts to introduce the weak-to-strong learning paradigm in the context of Vision-Language Models (VLMs). However, I suggest the authors consider the following points in the revision:

- 1. As suggested by Reviewer dHPD, additional VLMs (e.g., BLIP) should be included in the main paper. Some results are already discussed, but should be incorporated directly into the main content.
- 2. Reviewer dHPD also pointed out alternative choices for weak and strong models. While the discussion includes two additional strong models, the Problem Formulation in Section 4 defines the weak model as a vision encoder. Given this setup, could the authors also consider using a smaller CLIP variant (with fewer parameters) as the weak model? This simulates the usage of using GPT-2 for GPT-4, both are language models.
- 3. Reviewers Nvqx and fvRS asked for clarification on why the focus is solely on DomainNet rather than more general datasets. While the authors did not show results on other datasets, they provided a reasonable explanation: *“We want to simulate a scenario where the training data is limited in pretraining, and a weak-to-strong setting [1] is intended to improve a specific capability of the strong model that currently underperforms. Since the zero-shot performance of CLIP is already very strong on ImageNet, we choose DomainNet. The zero-shot performance of CLIP on some domains within DomainNet is below 50% accuracy.”*

    I suggest clarifying this motivation more explicitly in the Abstract and Introduction, as the current presentation may give the impression that weak-to-strong learning leads to universal performance gains. Specifically, ***please consider narrowing the scope of the submission to reflect its focus on targeted improvements under limited pretraining conditions.*** Additionally, while optional, including results on more general datasets such as CIFAR or iWilds would help strengthen the broader relevance of the findings.

- 4. Step 4 in Algorithm 1 could be improved by directly including the full Equation (Eq. 5), as the notation $f^w(\cdot)$ is not mentioned in the steps.

In summary, demonstrating that weak-to-strong learning can help improve specific capabilities of VLMs (e.g., CLIP) is an interesting contribution for the TMLR community. Therefore, I recommend accept for this paper. Please consider the above suggestions in the revisions.
Decision: Accept (poster)