Delving into Weakly Supervised Learning with Pre-Trained Models

Authors: Ming Li, Wei Wang, Masashi Sugiyama
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Weakly supervised learning, positive-unlabeled learning, unlabeled-unlabeled learning, pre-trained models.

Abstract:
Weakly supervised learning (WSL) is a popular machine learning paradigm in recent years that aims to learn a classifier with incomplete, imprecise, or inaccurate supervision. Existing WSL approaches have mainly focused on designing different loss functions or training strategies and then training models from scratch. In this paper, we first empirically show that a zero-shot baseline based on the Contrastive Language-Image Pre-Training (CLIP) model with class descriptions empowered by GPT-4o can outperform previous state-of-the-art methods trained from scratch on various WSL problems. Therefore, this motivates us to fine-tune pre-trained models to further improve the performance. However, our additional experiments show that naive use of existing WSL losses degrades performance due to severe overfitting exacerbation and feature degeneration problems. To address these problems, we propose a novel weakly supervised fine-tuning approach using dual classification heads that are trained synergistically by alternately distilling reliable supervision and performing efficient model fine-tuning. Theoretically, we prove the consistency and convergence rate of the proposed risk estimator. Empirically, extensive experiments on benchmark datasets of different WSL problems validate the effectiveness of the proposed approach against state-of-the-art competitors. The code is provided at https://github.com/ICLR2025-6897/WSFT_code.

Decision: reject
Reviewer scores: 5, 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1730096244094

thread_type:
official_review

<thread_object>
creator: reviewer_wu3a

<content>
summary:
This paper proposes a WSFT approach to effectively fine-tune pre-trained models for various WSL problems. WSFT encapsulates reliable supervision distillation and efficient model fine-tuning seamlessly supported by theoretical guarantees. Extensive experiments on various WSL problems and benchmark datasets validate the effectiveness of WSFT.

strengths:
This paper proposes a novel weakly supervised fine-tuning approach using dual classification heads that are trained synergistically
by alternately distilling reliable supervision and performing efficient model fine-tuning. Theoretically, the authors prove the consistency and convergence rate of the proposed risk estimator.

weaknesses:
1. The zero-shot baselines exhibit better performance in few pairs setting in table 2, which brings the concerns of effectiveness of WSFT in few shot scenarios. It would be better to provide additional analysis or experiments to examine WSFT's performance in other few pairs settings.
2. More recent methods are expected in comparison. Most of the compared apporaches seems to be out-of-data (2021 or before in table 2 and table 3).

questions:
1. In table 1, the results of WSFT on Oxford-IIIT Pet-b is worse than CVIR and DistPU. Further analysis is expected to explain this phenomenon.
2. In table 2 and table 3, experiment results with more recent approaches should be presented.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730573449476

thread_type:
official_review

<thread_object>
creator: reviewer_p8yg

<content>
summary:
The authors proposed a WSFT method for effectively fine-tuning pre-trained models for various WSL problems. Specifically, WSFT uses dual classification heads to alternately combine the reliability of supervision distillation and the efficiency of model fine-tuning, and theoretically proves the correctness of its methods. Finally, the authors conducted extensive experiments on various WSL problems and benchmark datasets, validating the effectiveness of the proposed method compared to SOTA WSL methods.

strengths:
The authors' method has a solid theoretical proof with a detailed proof process. The main theme of the article is clear. It first finds that the original weakly supervised training is not as good as the performance of the Clip model, then further analyzes the problem in depth, introduces new theories to solve it, and ultimately proves the correctness and effectiveness of the method through both theory and experiments.

weaknesses:
An overview figure could be added to make the entire process clearer.

questions:
(1) For binary classification tasks, is the conclusion related to the division of the dataset categories? How much impact does directly using GPT for division of the datasets have on the results, and has any verification been done for other combinations? How to consider the impact of the division of positive and negative samples on the results? 

(2) Regarding question 1, can a simple verification of the effect on multi-class classification be conducted?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730762130889

thread_type:
official_review

<thread_object>
creator: reviewer_tjxa

<content>
summary:
The paper investigates the potential of leveraging pre-trained models for weakly supervised learning (WSL).  The authors first demonstrate that a zero-shot baseline using a CLIP model with GPT-4o enriched class descriptions surpasses existing state-of-the-art WSL methods trained from scratch. Recognizing the potential for further improvement through fine-tuning, they identify limitations with existing unbiased and corrected risk estimators, noting issues with overfitting exacerbation and feature degeneration.

To address these issues, they propose a novel "Weakly Supervised Fine-Tuning (WSFT)" approach. WSFT utilizes dual classification heads: one trained on a corrected risk estimator to distill reliable supervision, and another trained via empirical risk minimization on the subset of data with high-confidence predictions from the first head. This iterative process allows for efficient model fine-tuning. The authors provide theoretical guarantees for the consistency and convergence rate of their proposed risk estimator. Finally, they present extensive empirical results on various WSL problems (PU learning, Pcomp learning, and UU learning) across benchmark datasets (CIFAR-100, EuroSAT, Oxford-IIIT Pet, Caltech-101), showing that WSFT consistently outperforms existing SOTA WSL approaches. In summary, the paper’s main contributions are: demonstrating the power of pre-trained models for WSL, proposing the novel WSFT algorithm with theoretical grounding, and empirically validating the effectiveness of WSFT across a range of WSL problems and datasets.

strengths:
+ This paper tackles the largely unexplored intersection of pre-trained models and weakly supervised learning, showing the untapped potential of existing pre-trained models for WSL.

+ The paper is written clearly and concisely, with a logical flow that guides the reader through the motivation, methodology, theoretical analysis, and experimental results.

weaknesses:
- While the paper presents extensive experiments across various benchmark datasets, the choice of datasets could be expanded to include more diverse and challenging scenarios, e.g., ImageNet and other variants.

- This paper does not explain how to generalize to multiple-class setting, especially what changes need to be made in the loss objective.

questions:
- How robust is WSFT to varying levels of label noise and class imbalance?

- Given the connection to self-training, have you considered including a self-training baseline using the same pre-trained model and data augmentation strategies for a direct comparison?  This would help isolate the specific benefits of the dual head approach and supervision distillation in WSFT.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730840035554

thread_type:
official_review

<thread_object>
creator: reviewer_jyaz

<content>
summary:
This paper explores the zero-shot ability of CLIP on weakly-supervised learning. They propose a two-step algorithm for WSL by generating a pseudo label first. The experiments show improvement in various settings of WSL.

strengths:
1. The author provides mathematical proof for their algorithm.
2. The experiments are detailed and show significant improvement to SOTA.

weaknesses:
1. The writing is not clear.
2. Putting Sec. 3.1 in the approach section is questionable to me. See Q1.
3. Possible missing related work. See Q2.

questions:
1. What is the difference between the zero-shot CLIP baseline introduced in L159 and the zero-shot performance on CIFAR-100 on the official CLIP GitHub repo (https://github.com/openai/CLIP)?

2. What is the author's comment on the comparison to [1]? To me, semantic segmentation, which can be viewed as a per-pixel image classification problem, is more complicated than image classification, which is the focus of this paper. It is mentioned that this paper can be extended to the "multi-class classification problem" in L276.

3. What is the formulation of PU learning? How is it compared to UU learning? This should be introduced before using PU learning as a major topic in L201. PU learning seems to be a special case for UU learning as mentioned in L223.

4. I am confused about Sec. 3.2, which is about the motivation of this paper. Does the depicted problem only occur in PU learning? In UU learning? Or both? The author explains the problem by giving PU learning as an example but later talks about UU learning in Sec. 3.3.

5. What is CRE $\hat R_1$ in L239? Is it the same as Eq 3, which is $\hat R_{CUU}$?

6. In L244, what does it mean "could contain all the unlabeled data whose true labels are not accessible to the learning algorithm"? Is there unlabeled data whose true labels are accessible? How can one access the labels of unlabeled data?

7. I think the writing of Sec. 3.3 can be improved. Especially, the definition of supervision distillation and model fine-tuning should be explained or summarized first in L237 at the start of the paragraph. Is the first step, i.e. supervision distillation, from L239 to L244, or L239 to L257? What is model-finetuning then?

8. Are the "class priors" in Sec. 4.3 and 4.4 the same as $\pi_{T_e}$?

9. The formulation of Pcomp should be explained somewhere, especially the meaning of "unlabeled pairwise observations were provided" in L402.

10. The legend in Fig. 2(d) is not visible.

11. The notation $C$ is first used in Eq. 2. The author also uses $C_g$ and $C_l$ for various constants in the theoretical analysis. I recommend distinguishing them. Additionally, $D$ is used in Eq. 2 and also as some datasets.

[1] CLIP is Also an Efficient Segmenter: A Text-Driven Approach for Weakly Supervised Semantic Segmentation. [CVPR 2023]
</content>
</thread_object>

</thread>
<thread>
creation date:
1732374991746

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We would like to thank the area chairs and reviewers for their efforts and suggestions in reviewing our paper. Their helpful comments have greatly enhanced our work.  

We have revised the manuscript according to the reviewers' suggestions and marked them in blue in the new version of the manuscript. The main differences include

- We revised Section 3.2 to present our methodology more clearly.

- We discussed the difference between our work and weakly supervised semantic segmentation in Section 5. 

- We included the experiments on ImageNet-100 in Appendix E.

- We included the experiments on CIFAR-100 with a different class partition in Appendix E.

- We added an overall pipeline of WSFT in Appendix A.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper explores using pretrained CLIP for weakly supervised learning problem. They shows that CLIP with GPT4o can help surpass existing WSL methods trained from scratch. In addition, they propose Weakly Supervised Fine-Tuning approach with theoretical grounding to effectively fine-tune pre-trained models for various WSL problems.

This paper received mixed reviews, with one positive and three negative scores. The authors are encouraged to improve the formatting for better readability and consider incorporating a broader range of experiments to substantiate their claim that "our proposal can handle any kind of weak supervision generated based on MCD or CCN." At this stage, the paper is not sufficiently prepared for publication.