Kati8/ICLR_2023_2025 huggingface dataset:
authors: Sumyeong Ahn
authors: Siqi Liang
authors: Jiayu Zhou
reviews: 
Comments: This paper introduces a robust training method called PoND, which utilizes a complementary approach across various types of prompts. The authors also explore the behavior of pre-trained VLMs under different prompts, including those crafted by humans and LLMs with visual characteristics. Lastly, the proposed PoND method outperforms existing prompt learning methods with noisy labels (e.g., CoOp and PTNL).
Strengths: 1. The proposed method PoND introduces a novel robust training process into prompt learning with noisy labels. Specifically, PoND consists of three steps for each iteration: distinguishing the _clean_ and _noisy_ samples in a training iteration (epoch), re-labeling noisy labels, and using GCE loss to traing models. Moreover, the good experiment results verify the proposed method. Besides, the experiment analysis and ablation study are sufficient.

2. I like the tables and figures of this paper. They are clear and.
Weaknesses: The performance of PoND is impressive compared to the current state-of-the-art approach, PTNL. However, its primary weakness is the computation cost because it need additional steps to clean or denoise noisy labels. As shown in Table 11, the training times for CoOp, PTNL, and PoND on the DTD dataset are 4m 7.725s, 4m 14.607s, and 5m 41.95s, respectively. This indicates that PoND's training time is over 38% longer than Vanilla, which could hinder its broader application. And it would be beneficial to provide more details on computational costs for large datasets like ImageNet.

Furthermore, two of the three proposed steps in PoND are existing methods, namely GMM estimation and GCE loss. Thus, the contribution may not be substantial enough for the whole community.
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 3
reviews: 
Comments: The paper introduces PoND, a prompt learning method for vision-language models under noisy labels. The paper first analyses the effectiveness of various prompts (human-crafted, synonym-based, description-based) for zero-shot classification, revealing that each prompt has specific strengths across classes. Building on this, PoND classifies samples as clean or noisy using Gaussian mixture models and assigns pseudo-labels to noisy samples with high confidence. Experimental results across 11 datasets show PoND's good performance in maintaining robustness under noise compared to existing methods.
Strengths: 1. The discovery that different prompts—Human, Synonym, and Description—each excel in specific classes under noisy labels is interesting and adds depth to prompt engineering in vision-language models.

2. The idea of PoND is interesting, which combines prompt expertise with GMM and high-confidence pseudo-labeling, demonstrating a robust approach to noise handling.

3. Extensive experiments across 11 datasets confirm PoND’s superior performance, especially in high-noise scenarios.
Weaknesses: 1. The paper spends significant space demonstrating that different prompts—human-crafted, synonym-based, and description-based—offer distinct advantages across datasets and categories, which is intuitive and already discussed in prior works; however, the implementation and advantages of each module in the proposed method are less clearly detailed. Also the choice of the three selected prompts seems somewhat ad hoc. 

2. In the first module of the method (set distinguishing), the authors do not provide comparative results between different expert models, only presenting results without this module.

3. In the re-labeling step, the authors rely on CoOp to generate pseudo-labels but do not attempt comparisons with newer prompt learning methods, e.g. Shuvendu Roy and Ali Etemad. Consistency-guided prompt learning for vision-language models. ICLR 2024.

4. The paper lacks comparative experiments on clean datasets, such as using only the expert model in Module 1 for direct prediction results.

5. Paper writing is sloppy, e.g. it uses a non-standard citation style, table conventions are specified in Appendix D making each individual table a hard to parse puzzle, the tiny font also does not help. 

6. The Related Work section is unfocused and does not highlight the novelty of the proposed method well, especially in light of prior prompt learning methods and methods for learning with noisy labels. Also note LNL is used but never properly defined. A division into two topics (prompt learning and learning with noisy labels) would make the section much easier to consume.
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: The paper tackles the issue of noisy labels in datasets when adapting pretrained Vision-Language Models (VLMs) to target domains with noisy training data. It presents a method using prompt techniques to differentiate between clean and noisy data, relabeling the latter based on confidence predictions. The relabeled data is combined with clean data to train domain-specific learnable prompts, aiming to improve classification accuracy.
Strengths: The paper addresses the practical challenge of noisy training data in image classification tasks.

The study explores the effects of various prompts and introduces synonyms as a novel data augmentation technique.

Detailed ablation studies are conducted to verify the effectiveness of the proposed method.
Weaknesses: 1. The overall contribution is limited. The effectiveness of using synonyms is intuitive, and correcting mislabeled samples is not novel [1]. The use of four types of prompts to identify incorrect labels, claimed to be better than zero-shot identification, resembles prompt ensemble. A comparison study with simple prompt ensembles should be included.

2. The experiments should be significantly extended. It's surprising to see only two methods compared in the experiment section (CoOp and PTNL). More methods should be included, such as CLIP pseudolabeling methods [2], traditional denoising methods applied to CLIP [1, 3], and advanced CLIP transfer learning methods [4, 5].

3. The additional cost (e.g., inference time, monetary cost) of querying GPT is not discussed, which cannot be ignored.

4. The "syn" prompt (Line 200) is confusing. Since CLIP is known for its bag-of-words feature [6], it may not distinguish "This is a photo of {CLS}. It is not a {ANT}." from "This is a photo of {ANT}. It is not a {CLS}.", raising doubts about the method's effectiveness.

5. Citations should be in parentheses (use \citep{}) when the authors or publication are not included in the sentence.

6. The paper is somewhat hard to follow. For example, the introduction lacks sufficient information for readers to understand the complex teaser figure.

[1] Learning from Massive Noisy Labeled Data for Image Classification. CVPR 2015.

[2] Enhancing CLIP with CLIP: Exploring Pseudolabeling for Limited-Label Prompt Tuning. NeurIPS 2023.

[3] CleanNet: Transfer Learning for Scalable Image Classifier Training With Label Noise. CVPR 2018.

[4] Self-regulating Prompts: Foundational Model Adaptation without Forgetting. ICCV 2023.

[5] Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinement. ICCV 2023.

[6] When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It? ICLR 2023.
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 2
reviews: 
Comments: This paper enhances classification fine-tuning performance by leveraging the zero-shot classification capability under a noisy labeled training dataset. Specifically, authors introduce a new training method called PoND, which employs a complementary approach across different types of prompts, leveraging the expertise of each class. The proposed method outperforms prior approaches across 11 real-world datasets.
Strengths: - The paper is well written and easy to follow.
- The results provide the valuable hints that what's the better way to deploy the pretrained vision-language model.
- The paper inspire people to think about more effective prompt design.
Weaknesses: - Although the paper states “Vision-Language Models” in its title, the experiments are only performed on CLIP models. It would be great to see similar findings for other vision-language models like DeCLIP, FILIP, CLOOB, CyCLIP, etc. Given that the paper is more of an analysis paper instead of a methodology paper, I would expect authors to verify their claims on other VLMs. Does the size of models effect the conclusions presented in this paper? A more thorough comparison and analysis should be included in the paper.
- How is this paper different from Rethinking the Value of Prompt Learning for Vision-Language Models? Isn't the conclusion opposite to this paper? It would be good discuss this paper in the lens of prompt learning vs classifier finetuning for VLMs.
- How does the proposed method work with respect to distribution shifts across domains, e.g., applying POND in ImageNet while testing on ImageNet-V2 or ImageNet-Sketch?
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 3