Title: Vision-Language Models are Strong Noisy Label Detectors
Venue: NeurIPS 2024
Accepted: yes
Keywords: label-noise learning

sample selection

semi-supervised learning
Abstract: Recent research on fine-tuning vision-language models has demonstrated impressive performance in various downstream tasks. However, the challenge of obtaining accurately labeled data in real-world applications poses a significant obstacle during the fine-tuning process. To address this challenge, this paper presents a Denoising Fine-Tuning framework, called DeFT, for adapting vision-language models. DeFT utilizes the robust alignment of textual and visual features pre-trained on millions of auxiliary image-text pairs to sieve out noisy labels. The proposed framework establishes a noisy label detector by learning positive and negative textual prompts for each class. The positive prompt seeks to reveal distinctive features of the class, while the negative prompt serves as a learnable threshold for separating clean and noisy samples. We employ parameter-efficient fine-tuning for the adaptation of a pre-trained visual encoder to promote its alignment with the learned textual prompts. As a general framework, DeFT can seamlessly fine-tune many pre-trained models to downstream tasks by utilizing carefully selected clean samples. Experimental results on seven synthetic and real-world noisy datasets validate the effectiveness of DeFT in both noisy label detection and image classification. Our source code can be found in the supplementary material.

Vision-Language Models are Strong Noisy Label Detectors

Abstract

Recent research on fine-tuning vision-language models has demonstrated impressive performance in various downstream tasks. However, the challenge of obtaining accurately labeled data in real-world applications poses a significant obstacle during the fine-tuning process. To address this challenge, this paper presents a De noising F ineT uning framework, called DEFT, for adapting vision-language models. DEFT utilizes the robust alignment of textual and visual features pretrained on millions of auxiliary image-text pairs to sieve out noisy labels. The proposed framework establishes a noisy label detector by learning positive and negative textual prompts for each class. The positive prompt seeks to reveal distinctive features of the class, while the negative prompt serves as a learnable threshold for separating clean and noisy samples. We employ parameter-efficient fine-tuning for the adaptation of a pre-trained visual encoder to promote its alignment with the learned textual prompts. As a general framework, DEFT can seamlessly fine-tune many pre-trained models to downstream tasks by utilizing carefully selected clean samples. Experimental results on seven synthetic and real-world noisy datasets validate the effectiveness of DEFT in both noisy label detection and image classification tasks. Our source code is available at https://github.com/HotanLee/DeFT .

1 Introduction

Vision-language models pre-trained on large-scale image-text pairs, such as Contrastive LanguageImage Pretraining (CLIP) [29], have gained widespread adoption in various machine learning tasks including few-shot learning [53, 52], multi-label learning [33, 14], and long-tail recognition [7, 30]. Recently, CLIP has shown impressive generalization performance to many downstream tasks without the need for adaptation [36, 31]. Zero-shot CLIP leverages the strong alignment of learned visual and textual features and classifies unseen images by comparing the similarity of image embedding with textual class prompts. Despite the good zero-shot performance, fine-tuning becomes necessary when the data distribution of downstream tasks significantly deviates from the CLIP training source. Two popular fine-tuning paradigms are commonly employed for adaptation, i.e., full fine-tuning (FFT), which modifies all model parameters, and parameter-efficient fine-tuning (PEFT), which fixes the pre-trained parameters while adding a few learnable parameters for adaptation. While promising, fine-tuning CLIP necessitates perfectly labeled datasets which may not be readily available in many real-world tasks, thereby hindering their broader applicability.

To tackle this issue, this paper investigates the fine-tuning of CLIP utilizing imperfectly labeled datasets, i.e., datasets with noisy labels. Intuitively, direct adaptation using noisy labels can signifi-

cantly degrade the performance. To mitigate the negative impact of noisy labels, researchers have proposed various approaches for learning with noisy labels, including sample selection techniques [10, 44, 43] and noise-robust learning methods [37, 27]. However, the exploration of this problem in the context of CLIP adaptation remains limited. This paper aims to fill this research gap by demonstrating that CLIP can be leveraged as effective noisy label detectors, capitalizing on their strong alignment between visual and textual features. To ensure the generality, we follow the basic idea in mainstream studies of learning with noisy labels. Firstly, we identify potentially mislabeled samples, and subsequently, we adapt the model parameters using carefully selected clean samples.

In the first phase, we propose to construct a noisy label detector by learning positive and negative textual prompts for each class. The positive prompt aims to uncover distinguishable features specific to the class, while the negative prompt functions as a learnable 'threshold' for identifying noisy labels. Our noisy label detector can be derived from the similarity between embeddings of a training image and positive/negative prompts associated with its assigned class. If the image exhibits a higher similarity score with the positive prompt than the negative prompt, it is considered a correctly labeled sample, i.e., a clean sample. This design draws inspiration from previous studies that have shown the robustness of prompt tuning to noisy labels, particularly in the presence of high noise ratios [41]. To optimize the positive prompt, we align its representation with image features from the corresponding class during training. For the negative prompts, we introduce a novel negative learning objective. It is important to note that calculating similarities between images and textual prompts requires a strong alignment between these two modalities in the downstream task. To achieve this, we utilize PEFT, such as Visual Prompt Tuning (VPT) [18], for the adaptation of the visual encoder. This decision is based on a key finding that FFT can distort pre-trained feature representations when noisy labels are present, which is discussed in Section 3.2.

In the second phase, we can adapt the pre-trained model after acquiring carefully selected clean samples. It is worth noting that not only can we employ pre-trained CLIP for adaptation, but we can also utilize other pre-trained visual backbones such as ResNet [12] and MAE [11]. While the model adaptation phase may seem unnecessary since we can leverage the learned textual prompts for zero-shot classification without further fine-tuning, we emphasize its necessity by the improved performance on curated downstream datasets, particularly in fine-grained classification tasks. In this step, we learn a linear classifier and fully fine-tune the visual encoders using only selected clean samples. The rationale behind this approach is twofold: 1) the linear classifier tends to generalize better than textual class prompts, and 2) full fine-tuning can effectively adapt model parameters to address significant domain shifts, surpassing the capabilities of PEFT. In our implementation, the training is conducted for only 10 epochs, resulting in a minimal increase in computation cost.

Our main contributions can be summarized as follows:

We propose DEFT, a simple yet effective framework for learning with noisy labels. DEFT offers several compelling advantages over existing methods: instance-dependent (no information required from entire training data), robust to various types of noisy labels, and generalizable to many pre-trained models.

We conduct extensive experiments and show that DEFT achieves superior performance in both noisy label detection and image classification tasks on a wide range of synthetic and real-world datasets.

We provide in-depth empirical analysis, providing insights to understand the effectiveness of DEFT. We hope that this work will serve as a springboard for future works on noisy label detection with multi-modal features.

2 Related Work

Learning with Noisy Labels A diverse variety of approaches have been proposed for addressing learning with noisy labels and can be broadly divided into twofold: robust learning and clean sample selection [9, 32]. Due to the susceptibility of traditional cross-entropy loss to noisy labels, several methods adopt noise-robust loss functions [8, 51, 37, 6] and regularization techniques [27, 38, 16] to enhance model robustness. For example, symmetric cross-entropy loss [37] introduces a noise-robust objective function for fast convergence and better performance. Early-learning regularization [27] counteracts the influence of the noisy labels by introducing a regularization term that integrates estimated target probabilities. However, empirical findings suggest that robust model learning faces

the problem of maintaining a balance between robustness and accuracy [51, 35]. Conversely, sample selection approaches strive to select clean samples from noisy datasets using specific criteria. Based on the memorization effect identified in deep neural networks [2], Numerous methods adopt the small-loss strategy that treats samples with small loss as clean ones [10, 15, 25, 19]. Co-teaching [10] selects a fraction of training data in each mini-batch based on the noise ratio, whereas DivideMix [25] fits a Gaussian mixture model to dynamically divide the training samples. Although these methods demonstrate strong performance, they inevitably overlook the significance of clean hard samples with a large loss. Besides, all the aforementioned approaches concentrate exclusively on learning from scratch with label noise. Our proposed approach leverages the pre-trained multi-modal information from visual-language models to construct a noisy label detector, resulting in improved sample selection performance.

Fine-Tuning Vision-Language Models Recent years have witnessed incredible progress in pretrained visual-language models [29, 17, 49]. As a representative method, CLIP [29] employs contrastive pre-training to acquire informative representations from extensive image-text pairs. Despite the remarkable zero-shot performance of CLIP, its sensitivity to hand-crafted textual prompts is significant. Prompt-tuning aims to learn the optimal prompts from target data by replacing the textual templates with adaptable soft prompts, which demonstrates superior performance in the context of few-show learning [53, 52, 20] and multi-label learning [33, 14]. However, further enhancements can be achieved by adapting the visual encoder to accommodate larger datasets in downstream tasks. A standard approach to obtain task-specific representations is fine-tuning all model parameters. While FFT yields significant performance on curated datasets, recent work [1] empirically finds that FFT can result in feature distortion in the presence of massive noisy labels. In contrast, PEFT [48] has been verified to improve generalization capabilities for mitigation of overfitting issues [30], which fix the pre-trained model and introduce a few number of learnable parameters for adaptation, such as VPT [18], LoRA [13], and AdaptFormer [3]. Considering the distinct property of fine-tuning methods, this paper aims to delineate the most effective paradigm for model adaptation on noisy downstream datasets, which remains under-explored yet holds significant promise.

3 Preliminary and Initial Findings

3.1 Zero-Shot CLIP

This paper focuses on a K -class image classification task, where the objective is to categorize an image x ∈ R d and assign it a label y ∈ [ K ] = { 1 , 2 , . . . , K } . Benefiting from large-scale datasets and high-capacity models, CLIP [29] has demonstrated remarkable performance in zero-shot classification by computing the similarity between visual and textual representations. Specifically, with hand-crafted textual prompts such as ' a photo of a [CLS] ', where the class token is replaced by a specific class name, the prediction probability is computed as:

where sim ( · , · ) denotes the cosine similarity between the extracted image features I and text features T , and τ is the softmax temperature.

While zero-shot classification with CLIP is promising, its performance can be further enhanced by adapting CLIP on downstream labeled datasets D = { ( x i , y i ) N i =1 } to incorporate more task-specific representations. In this paper, we investigate the presence of noisy labels in the training dataset D , which is prevalent in real-world applications. We use r ∈ [0 , 1] to denote the noise ratio of a training set, meaning that N × r training samples are incorrectly labeled, i.e., the assigned label y differs from the ground-truth label y ∗ . Since y ∗ is not accessible, our goal is to fine-tune the CLIP model using training labels y by eliminating the negative influence of noisy labels. Considering the feature distortion induced by noisy labels [2], the key challenge lies in obtaining distinguishable yet robust representations when adapting CLIP on noisy datasets.

3.2 Fine-tuning CLIP on Downstream Datasets

To discern the most effective method for CLIP adaptation, we conduct empirical studies on various datasets as shown in Figure 1. Specifically, we utilize three fine-tuning approaches to adapt the

pre-trained CLIP model and compare their performance on both noisy and clean datasets, including 1) FFT which updates the entire model parameters, 2) VPT which fixes pre-trained model parameters and prepends a small subset of extra learnable parameters to the visual encoder during fine-tuning, and 3) VLPT (Vision-Language Prompt Tuning) which integrates both visual and textual learnable prompts into the fixed pre-trained model for fine-tuning. For FFT and VPT, we learn an additional linear classifier, while VLPT directly utilizes the learned textual prompts for classification. We present our three primary findings in the following.

VPT benefits representation learning in the presence of massive noisy labels. Intuitively, utilizing FFT for CLIP adaptation yields improved performance by leveraging its substantial capacity to incorporate task-specific representations. However, the efficacy of FFT diminishes notably when applied to datasets containing noisy labels, as shown in Figure 1a. This decline is attributed to the pronounced distortion of representations with an escalating noise ratio [1]. In contrast, VPT benefits representation learning when adapting CLIP on noisy data, particularly under high levels of noise. As only a small set of parameters is introduced, VPT effi-

































 























(b) Clean Datasets

ciently retains the generalization ability from image-text pre-training while enhancing classification performance on downstream tasks, making it a robust and effective fine-tuning approach for handling noisy downstream datasets.

Textual classifier is robust to noisy labels. Classification with learnable textual prompts (e.g., CoOp [53]) leverages the multi-modal information in pre-trained vision-language models and enhances the alignment of visual and textual representations on downstream tasks. Recent literature has substantiated its robustness with a frozen visual encoder in the context of few-shot learning [41]. From Figure 1a, we observe that VLPT with a textual classifier (green line) consistently surpasses VPT with a traditional linear classifier for classification (orange line), especially under severe noise. The improvement in performance across diverse noise ratios further affirms the robustness of learnable textual prompts in mitigating the impact of label noise for model adaptation.

FFT enhances visual recognition on clean datasets. Although previous findings show that FFT is more susceptible to noisy labels compared to VPT and VLPT, Figure 1b demonstrates its enhanced discriminative ability when training data is clean. This superiority is particularly significant on fine-grained datasets such as Stanford-Cars and CUB-200-2011, with an average improvement of 2.81% against VPT. It is important to note that VLPT exhibits the worst performance on clean datasets. This is primarily due to the implicit regularization of pre-trained textual information when tuning the context of textual prompts, as explained in the recent study [41].

4 The Denoising Fine-tuning Framework

Based on the findings in Section 3.2, we now present a novel denoising fine-tuning framework, DEFT, which comprises two pivotal phases. In the first phase, DEFT learns dual textual prompts to separate clean and noisy samples while adapting the visual encoder utilizing PEFT methods. In the second phase, DEFT re-adapts the pre-trained model using FFT, leveraging the curated clean samples to further boost visual recognition performance. Figure 2 illustrates our proposed framework.

4.1 Identifying Noisy Labels with Dual Prompts

Given the empirically demonstrated robustness of textual prompts to label noise, we intend to harness this property to identify corrupted samples in noisy downstream datasets. The initial strategy is to select a subset of samples with pronounced similarity between their visual and textual representations and designate them as clean. However, this method relies on either prior knowledge of the noise ratio or the manual setting of thresholds, which restricts its practical applicability. Inspired by recent works [33, 14, 36], we utilize dual textual prompts instead of a single prompt to overcome these limitations. Concretely, we design a class-specific pair of positive and negative prompts for the textual encoder,

Text

Encoder

Visual Encoder

PEFT Modules

Pre-trained

Models

Classifier

Noisy Dataset

Clean Subset

cat

dog

Positive Prompt

Negative Prompt

cat

dog

cat

Learnable prompt

Data flow

Learnable network

dog +

dog

-

Full Fine-tune

Phase 1: Noisy Label Detection

Phase 2: Model Adaptation

>

denoted as prompt + k and prompt -k , respectively:

Here, [ V ] m is the word vector of context (with dimension 512 in CLIP), M is the length of context, and [ CLS ] k is the k -th provided category name. The positive prompt aims to uncover distinguishable features by maximizing the similarity between the image features and their corresponding text features, while the negative prompt serves as a learnable sample-dependent similarity threshold to select clean data. Specifically, the learnable threshold ϕ i for the i -th training sample with label y i = k is formulated as:

which represents the cosine similarity between the extracted image features and negative text features. Based on this threshold, the clean subset of dataset D can be constructed as follows:

The proposed selection strategy surpasses the conventional loss-based approaches in two aspects: 1) It is more practical by utilizing data-driven thresholds thus eliminating the requirement for prior knowledge like noise ratio, and 2) the integration of text modality enhances its robustness to label noise, making it capable of identifying challenging hard noise. Nevertheless, the primary dilemma lies in the optimization of positive and negative prompts using noisy downstream datasets.

4.2 Optimization for Noisy Label Detector

It can be an intricate task to optimize the threshold ϕ i in an end-to-end manner since it is indirectly involved in the forward propagation. However, we can still optimize the associated parameters by formulating the clean probability of the i -th image with y i = k as:

Obviously, selecting samples with the clean probability p clean ik > 0 . 5 is equivalent to the specified criteria in Eq. (5). In this way, the original threshold-based selection approach can be transformed into K binary classification tasks to determine whether a sample is clean or not. Concretely, given a sample ( x i , y i ) from noisy datasets D , we can employ the dual prompts of class y i as a binary classifier to identify label noise. If the classifier outputs a positive prediction of x i , it is categorized as a clean sample.

To optimize the noisy label detector, we need to construct positive and negative training samples for binary classification, i.e., images with correctly and wrongly assigned labels. Inspired by the prior work on negative learning [21], we designate each image with a randomly picked complementary label ¯ y to form negative samples. For positive samples, we initiate with the original supervision in

D and refine it with pseudo labels ˆ y generated by Eq. (1). The training loss is then formulated as follows:

where p clean i = { p clean ik } K k =1 is the clean probability and ℓ nll ( · , · ) is the negative log-likelihood loss where ℓ nll ( p i , y ) = -log p iy .

Additionally, to enhance task-specific representations of the noisy label detector, we harness PEFT for the adaptation of the visual encoder, considering its robustness under massive noisy labels compared with FFT, as per our previous finding (see Figure 1a). This is achieved by maximizing the alignment between image embeddings and their corresponding positive textual features:

Note that L sim is exclusively computed on D clean after the warm-up stage to further eliminate the impact of label noise on representations. With the objective L = L dp + L sim , DEFT effectively sieves out noisy samples with learned thresholds while uncovering distinguishable visual representations on downstream tasks.

4.3 Model Adaptation using Clean Data

Although the learned positive textual prompt can be readily employed for classification, its performance may be suboptimal on curated clean datasets, as demonstrated in our previous finding (see Figure 1b). To mitigate this problem, we introduce the model adaptation phase which learns a linear classifier using selected clean samples, i.e., D clean . Moreover, our findings also indicate that FFT outperforms PEFT on clean datasets, so we remove the PEFT modules and fully fine-tune the pre-trained model parameters for model adaptation. We minimize the classic cross-entropy loss ℓ ce ( x , y ) = -log exp( z y ) ∑ K k =1 exp( z k ) , where z = { z k } K k =1 represents the logits predicted by the linear classifier. It is noteworthy that with the clean subset of training data constructed by Eq. (5), the second phase can be applied universally to a wide range of pre-trained models, regardless of their backbones. The versatility of our approach is demonstrated in the experiments.

5 Experiment

We now proceed to the experiments, demonstrating the advantages of our approach in noisy label detection and image classification on both synthetic and real-world datasets.

5.1 Experimental Settings

Synthetic Datasets We first evaluate the performance of DEFT on four image classification benchmarks by synthesizing noisy labels with varying noise types and ratios. Specifically, we conduct experimental analyses on widely-used CIFAR-100 [23] and Tiny-ImageNet [42], as well as two fine-grained datasets Stanford-Cars [22] and CUB-200-2011 [34]. Given the noise transition matrix T ∈ [0 , 1] K × K with T ij denoting the probability of the ground-truth y ∗ = i being flipped into a corrupted label y = j , we introduce two types of label noise: 1) T ij = p ( y = j | y ∗ = i ) , where the corruption probability is assumed to be only dependent on the true label, e.g., the symmetric noise generated by replacing y ∗ with a random label for a given proportion of training samples [10, 25], and 2) T ij = p ( y = j | y ∗ = i, x ) , which is a more realistic assumption considering the impact of instance x in the label corruption process, e.g., the instance-dependent noise proposed in [45]. We conduct extensive experiments on each dataset with noise ratio r ∈ { 0 . 2 , 0 . 4 , 0 . 6 } for symmetric noise and r ∈ { 0 . 2 , 0 . 3 , 0 . 4 } for instance-dependent noise.

Real-World Datasets We further examine the performance of DEFT on three real-world noisy label datasets: 1) CIFAR-100N [39] ( r ≈ 0 . 4 ) is a variant of CIFAR-100 with real-world human annotations from Amazon's Mechanical Turk, 2) Clothing1M [47] ( r ≈ 0 . 4 ) is a large-scale dataset consisting of 1 million clothing images of 14 classes collected from online shopping websites, and

3) WebVision [26] ( r ≈ 0 . 2 ) contains 2.4 million images crawled from Flickr and Google using the 1,000 concepts in ImageNet ILSVRC12. Following previous works [25, 19, 24], we experiment on the first 50 classes of the Google image subset.

Evaluation Metrics We use precision and recall metrics to evaluate the selection of clean samples. A higher precision indicates a greater proportion of actual clean samples in D clean , while a higher recall means that more clean samples are identified from the noisy dataset. For image classification, We report the 'best' test accuracy across all training iterations and the 'last' test accuracy at the end of training.

Implementation Details We adopt the pre-trained CLIP [29] with the Transformer as the text encoder and the ViT-B/16 as the image encoder. We use the SGD optimizer with a momentum of 0.9, a weight decay of 5 × 10 -4 , and a batch size of 64. We run 10 epochs for both the noisy label detection phase and the model adaptation phase with learning rates 3 × 10 -2 and 5 × 10 -4 , respectively. In the noisy label detection phase, we employ VPT [18] and CoOp [53] to adapt visual encoder and textual encoder respectively, and perform model warm-up for 1 epoch on all datasets. When identifying noisy labels on real-world datasets, we augment the condition in Eq. (5) with a constraint that the training label should be consistent with its predicted label. This is due to the existence of more complex noise patterns in real-world tasks. All experiments are conducted on a single NVIDIA GeForce RTX 3090.

5.2 Performance for Noisy Label Detection

We conduct a thorough evaluation of DEFT to justify its effectiveness in detecting noisy labels. For this purpose, we simulate various types and ratios of label noise on four benchmark datasets. We compare our approach with two other sample selection strategies: 1) Label-match strategy, where a training sample is deemed clean if its given label matches that assigned by the CLIP zero-shot classifier, and 2) Small-loss strategy, which selects a proportion (noise ratio) of confident samples in each mini-batch, as commonly used in previous studies [10, 25, 19]. The precision and recall of sample selection are reported in Table 1. The results show that our proposed noisy label detector outperforms the compared strategies, with significant improvements observed in all dataset settings. The trivial label-match strategy tends to be conservative, leading to low selection recall of clean samples. In contrast, the small-loss strategy and DEFT achieve a better trade-off between precision and recall, making better use of training samples. Moreover, leveraging the informative multimodel information in the pre-trained vision-language model, DEFT surpasses the small-loss strategy

in detecting noisy labels, particularly under severe noise settings and fine-grained classification tasks. For instance, in Tiny-ImageNet with 60% symmetric noise, DEFT demonstrates significant enhancements of 4.58% and 4.55% in precision and recall, respectively, as well as an improvement of 9.78% and 14.42% in CUB-200-2011. Additionally, DEFT reduces the need for prior knowledge of noise ratios, making it a practical and effective approach to detect label noise in real-world tasks.

5.3 Performance for Image Classification

We evaluate the performance of DEFT in image classification tasks against four baselines, including CE (Cross-Entropy loss), SCE [37], ELR [27], and GMM [25]. To further verify the effectiveness of our method on real-world datasets, we also make comparisons with some recent sample selectionbased works including RoLT[40], UNICON[19], LongReMix[4], and ProMix[46]. We utilize FFT for adapting the pre-trained CLIP model for all approaches. More details can be found in the Appendix.

Results on Synthetic Datasets Table 2 presents the results on four synthetic noisy datasets. Compared with CE, the adoption of noise-robust loss functions such as ELR and SCE improves the classification performance on CIFAR-100 and Tiny-ImageNet under certain noisy label settings, e.g., ELR achieves the best performance on CIFAR-100 with 40% instance-dependent noise. However, these methods are not always effective and may even worsen the performance under varying types and ratios of noise, compared to the sample selection paradigm like GMM and DEFT. While GMM outperforms ELR and SCE in most cases, its performance degrades dreadfully on fine-grained datasets. Nevertheless, DEFT retains the most robust performance and generally advances all compared methods, especially on Stanford-Cars and CUB-200-2011. For example, DEFT boosts the test accuracy



























(a) CIFAR-100

























(b) Tiny-ImageNet

























(c) Stanford-Cars

























(d) CUB-200-2011

by an average of 4.34% on Stanford-Cars compared to the best results of comparison methods. The results demonstrate the effectiveness of our proposed approach in tackling both symmetric and instance-dependent label noise.

Results on Real-world Datasets The results reported in Table 3 demonstrate the superiority of our method on real-world datasets, as DEFT surpasses other methods by a significant margin on all datasets. Notably, while most compared methods show little improvement on Clothing1M due to its fine-grained nature, both ELR and DEFT demonstrate substantial improvement. However, ELR exhibits a performance decrease on WebVision. Moreover, directly fine-tuning pre-trained CLIP models with cross-entropy (CE) achieves competitive results on WebVision, yet DEFT elevates this performance further, showcasing its ability to combat label noise in practical situations.

5.4 Further Analyses

Necessity of Model Adaptation In this experiment, we emphasize the necessity of the model adaptation phase in DEFT. Figure 3 demonstrates the test accuracy of DEFT without the model adaptation phase (w/o adap.), adaptation utilizing PEFT and FFT. Results show that fully fine-tuning the model on selected clean samples yields the best classification performance, especially on finegrained datasets, such as Stanford-Cars and CUB-200-2011. The results unveil two primary insights: 1) the noisy label detector in DEFT exhibits strong capabilities in detecting label noise, yielding a nearly clean subset from noisy downstream datasets, and 2) employing FFT for model adaptation is more effective in mitigating significant domain shifts between downstream data and pre-training data, particularly evident in fine-grained datasets.

DEFT for Various Pre-trained Models It is noteworthy that DEFT can seamlessly integrate with various pre-trained visual backbones during the model adaptation phase. To demonstrate the versatility of our proposed denoising fine-tuning framework, we conduct experiments on Clothing1M and apply FFT to a range of pre-trained models, including ViT-B/16 [5], ResNet-50 [12], ConvNeXt-T [28], and MAE-VIT-B [11]. The results are presented in Table 4, where GCE [51] employs a noise-robust loss function akin to SCE. As a recent work, TURN [1] is an improved version of GMM, which applies linear probing to initialize the classifier and then uses FFT for adaptation. Compared with previous methods, DEFT exhibits the best performance on average. In particular, DEFT outperforms TURN by 4.51% and 3.27% when adopting ResNet-50 and MAE-ViT-B as the target pre-trained models, respectively.

6 Conclusion

In this work, we delve into a new landscape for learning with noisy labels, departing from the classic single-modal toward a multi-modal regime. By learning dual textual prompts, we construct a new noisy label detection approach, which offers several compelling advantages: robust to various types of label noise, generalizable to many pre-trained models, and does not require the dynamics of training samples. We investigate the effectiveness of DEFT on a wide range of synthetic and real-world datasets, showing its superior performance in both noisy label detection and image classification tasks. Lastly, we demonstrate the advantage of parameter-efficient fine-tuning over full fine-tuning over noisy label detection. We hope our work will inspire future research toward multi-modal noisy label detection.

Acknowledgments and Disclosure of Funding

This work was supported by the National Science Foundation of China (62206049, 62225602), the Key Program of Jiangsu Science Foundation (BK20243012), and the Big Data Computing Center of Southeast University. We would like to thank anonymous reviewers for their constructive suggestions.

References

Sumyeong Ahn, Sihyeon Kim, Jongwoo Ko, and Se-Young Yun. Fine tuning pre trained models for robustness under noisy labels. arXiv preprint arXiv:2310.17668 , 2023.

Devansh Arpit, Stanisaw Jastrzbski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning , pages 233-242. PMLR, 2017.

Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems , 35:16664-16678, 2022.

F. R. Cordeiro, Vasileios Belagiannis, Ian Reid, and Gustavo Carneiro. Longremix: Robust learning with high confidence samples in a noisy label environment. Pattern Recognition , 133:109013, 2023.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Lei Feng, Senlin Shu, Zhuoyi Lin, Fengmao Lv, Li Li, and Bo An. Can cross entropy loss be robust to label noise? In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence , pages 2206-2212, 2021.

Kai Gan and Tong Wei. Erasing the bias: Fine-tuning foundation models for semi-supervised learning. In Proceedings of the 41st International Conference on Machine Learning , pages 14453-14470, 2024.

Aritra Ghosh, Himanshu Kumar, and P Shanti Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI conference on artificial intelligence , volume 31, 2017.

Bo Han, Quanming Yao, Tongliang Liu, Gang Niu, Ivor W Tsang, James T Kwok, and Masashi Sugiyama. A survey of label-noise representation learning: Past, present and future. arXiv preprint arXiv:2011.04406 , 2020.

Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems , 31, 2018.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000-16009, 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

Ping Hu, Ximeng Sun, Stan Sclaroff, and Kate Saenko. Dualcoop++: Fast and effective adaptation to multi-label recognition with limited annotations. arXiv preprint arXiv:2308.01890 , 2023.

Jinchi Huang, Lie Qu, Rongfei Jia, and Binqiang Zhao. O2u-net: A simple noisy label detection approach for deep neural networks. In Proceedings of the IEEE/CVF international conference on computer vision , pages 3326-3334, 2019.

Ahmet Iscen, Jack Valmadre, Anurag Arnab, and Cordelia Schmid. Learning with neighbor consistency for noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4672-4681, 2022.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning , pages 4904-4916. PMLR, 2021.

Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision , pages 709-727, 2022.

Nazmul Karim, Mamshad Nayeem Rizve, Nazanin Rahnavard, Ajmal Mian, and Mubarak Shah. Unicon: Combating label noise through uniform selection and contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9676-9686, 2022.

Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19113-19122, 2023.

Youngdong Kim, Junho Yim, Juseung Yun, and Junmo Kim. Nlnl: Negative learning for noisy labels. In Proceedings of the IEEE/CVF international conference on computer vision , pages 101-110, 2019.

Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei. Collecting a large-scale dataset of fine-grained cars. Second Workshop on Fine-Grained Visual Categorization , 2013.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of feature from tiny images. Master's Thesis , 2009.

Hao-Tian Li, Tong Wei, Hao Yang, Kun Hu, Chong Peng, Li-Bo Sun, Xun-Liang Cai, and Min-Ling Zhang. Stochastic feature averaging for learning with long-tailed noisy labels. In Proceedings of the International Joint Conference on Artificial Intelligence , pages 3902-3910, 2023.

Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In International Conference on Learning Representations , 2020.

Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint arXiv:1708.02862 , 2017.

Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda. Early-learning regularization prevents memorization of noisy labels. Advances in Neural Information Processing Systems , 2020.

Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11976-11986, 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763, 2021.

Jiang-Xin Shi, Tong Wei, Zhi Zhou, Jie-Jing Shao, Xin-Yan Han, and Yu-Feng Li. Long-tail learning with foundation model: Heavy fine-tuning hurts. In Proceedings of the 41st International Conference on Machine Learning , pages 45014-45039, 2024.

Yang Shu, Xingzhuo Guo, Jialong Wu, Ximei Wang, Jianmin Wang, and Mingsheng Long. Clipood: Generalizing clip to out-of-distributions. In International Conference on Machine Learning , 2023.

Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Ximeng Sun, Ping Hu, and Kate Saenko. Dualcoop: Fast adaptation to multi-label recognition with limited annotations. Advances in Neural Information Processing Systems , 35:30569-30582, 2022.

Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. Technical report , 2011.

Deng-Bao Wang, Yong Wen, Lujia Pan, and Min-Ling Zhang. Learning from noisy labels with complementary loss functions. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 10111-10119, 2021.

Hualiang Wang, Yi Li, Huifeng Yao, and Xiaomeng Li. Clipn for zero-shot ood detection: Teaching clip to say no. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1802-1812, 2023.

Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF international conference on computer vision , pages 322-330, 2019.

Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 13726-13735, 2020.

Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In International Conference on Learning Representations , 2022.

Tong Wei, Jiang-Xin Shi, Wei-Wei Tu, and Yu-Feng Li. Robust long-tailed learning under label noise. arXiv preprint arXiv:2108.11569 , 2021.

Cheng-En Wu, Yu Tian, Haichao Yu, Heng Wang, Pedro Morgado, Yu Hen Hu, and Linjie Yang. Why is prompt tuning for vision-language models robust to noisy labels? In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15488-15497, 2023.

Jiayu Wu, Qixiang Zhang, and Guoxi Xu. Tiny imagenet challenge. Technical report , 2017.

Zhi-Fan Wu, Tong Wei, Jianwen Jiang, Chaojie Mao, Mingqian Tang, and Yu-Feng Li. Ngc: A unified framework for learning with open-world noisy data. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 62-71, 2021.

Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, and Masashi Sugiyama. Sample selection with uncertainty of losses for learning with noisy labels. In International Conference on Learning Representations , 2021.

Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. Advances in Neural Information Processing Systems , 33:7597-7610, 2020.

Ruixuan Xiao, Yiwen Dong, Haobo Wang, Lei Feng, Runze Wu, Gang Chen, and Junbo Zhao. Promix: Combating label noise via maximizing clean sample utility. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence , pages 4442-4450, 2023.

Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR , 2015.

Bruce XB Yu, Jianlong Chang, Haixin Wang, Lingbo Liu, Shijie Wang, Zhiyu Wang, Junfan Lin, Lingxi Xie, Haojie Li, Zhouchen Lin, et al. Visual tuning. arXiv preprint arXiv:2305.06061 , 2023.

Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 , 2022.

Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages 1-9, 2022.

Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems , 31, 2018.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16816-16825, 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision , 130(9):2337-2348, 2022.

A Appendix

A.1 Additional Results on Noisy Label Detection

Impact of CLIP Backbones Table 1 has demonstrated the superiority of DEFT in detecting noisy labels based on CLIP ViT-B/16. To evaluate the impact of different visual backbones on noise detection, we re-implement all methods using CLIP ViT-B/32 as the backbone and report the results in Table 5. Notably, while the small-loss strategy demonstrates comparable performance on noisy CIFAR-100 and Tiny-ImageNet, its efficacy significantly diminishes on fine-grained datasets such as Stanford-Cars and CUB-200-2011, which exhibits its sensitivity to visual backbones. In contrast, DEFT maintains consistently superior performance on both ViT-B/16 and ViT-B/32, underscoring its resilience across different pre-trained visual backbones.

Results under Severe Noise We conduct experiments on synthetic noisy datasets with significant label noise to further validate the robustness of the proposed noisy label detector. It is noteworthy that two adjustments are implemented in order to train a more effective model: 1) the learning rate is adjusted from 3 × 10 -2 to 1 × 10 -2 , and 2) For DEFT, a small weight (specifically, 0.25) is introduced to the positive loss component in Eq. (7) to mitigate the impact of noisy pseudo labels in high levels of noise. Table 6 reports the precision, recall, and their harmonic mean (F1-score) for noise detection, where DEFT exhibits a robust capability in identifying severe noise. For example, a remarkable improvement in the F1-score is achieved on CUB-200-2011. These results strongly reaffirm the superior performance of DEFT across various noise ratios.

GMMfor Noisy Label Detection DivideMix [25] introduces the use of a Gaussian Mixture Model (GMM) for the dynamic separation of clean and noisy samples with an adjustable threshold. Table 7 summarizes the performance of GMM and DEFT in detecting label noise across diverse datasets and noise settings. To fine-tune the pre-trained model for GMM, we independently apply PEFT and FFT. The results reveal that DEFT consistently exhibits superior noise detection capability, particularly evident in fine-grained datasets with high noise ratios, such as Stanford-Cars with 60% symmetric noise and CUB-200-2011 with 40% instance-dependent noise. Intriguingly, implementing GMM with PEFT yields better noise detection when compared to utilizing FFT. This difference can be attributed to the potentially unstable training process of FFT, where the loss value may undergo significant fluctuations due to an abundance of tunable parameters.

Noisy Label Detection on CIFAR-100N The CIFAR-100N dataset [39], which serves as a realworld benchmark by providing CIFAR-100 with human-annotated noisy labels, allows us to evaluate

the efficacy of DEFT for noise detection in practical scenarios. The results presented in Table 8 reveal the distinctive performance characteristics of various sample selection strategies. For example, the label-match strategy demonstrates a propensity for being greedy and yields the highest precision, whereas the GMM and recent sample selection-based works tends to be conservative and thus achieve a higher recall. Nevertheless, DEFT stands out with the highest F1-score across all strategies, striking a judicious balance between precision and recall in the identification of real-world noisy labels.

Sensitivity of Hyperparameters We explore the sensitivity of our noise label detector to various hyperparameter settings, specifically the length of the visual/textual prompt and the number of training epochs. Table 9 presents the F1-score on CIFAR-100 with 40% symmetric noise. The results indicate that the length of the visual prompt has a marginal impact on the noisy label detection. For the textual prompt length, once the context length exceeds 8, the performance is enhanced slightly as the length increases. Additionally, variations in the number of training epochs have minimal impact on performance. In order to strike a balance between effectiveness and efficiency, we set the visual prompt length to 20, the textual prompt length to 16, and the training epoch to 10 in all experiments.

A.2 Additional Results on Image Classification

Robustness of Various PEFT techniques To further substantiate the advantages of PEFT over FFT for adapting visual backbones in the presence of massive noisy labels, we conduct a comprehensive evaluation of various PEFT techniques for image classification, including 1) Visual Prompt Tuning (VPT) [18] which prepends learnable prompts to the input at each layer, 2) Low-Rank Adapter (LoRA) [13] which injects trainable rank decomposition matrices into each layer, 3) AdaptFormer [3] which replaces the original Multi-Layer Perceptron (MLP) block with AdaptMLP that incorporates a trainable bottleneck module, and 4) Bias-terms Fine-tuning (BitFit) [50] which is a sparse fine-





























(a) VPT





























(b) LoRA





























(c) AdaptFormer





























(d) BitFit

tuning method where only the bias-terms of the model are being modified. The results are shown in Figure 4. Notably, nearly all PEFT methods outperform FFT when the noise ratio exceeds 0.4, and the performance gap becomes more pronounced with the increase in the noise ratio. These results comprehensively underscore the efficacy of PEFT over FFT in mitigating the distortion of representations when adapting visual backbones under severe noise.

Applying PEFT for Model Adaptation Table 10 presents the test accuracy obtained by applying PEFT to adapt the pre-trained CLIP model. There are three noteworthy observations in comparison to the results in Table 2. Firstly, previous label-noise learning methods consistently demonstrate their efficacy in handling noisy datasets compared to the vanilla methods (CE), showcasing a clear improvement in performance across various datasets and noise settings. Secondly, despite the bad performance of GMM on fine-grained datasets in Table 2, it achieves the best performance among

Algorithm 1: The Proposed DEFT Framework

baselines when utilizing PEFT to adapt the pre-trained model, possibly due to the more stable training process with fewer learnable parameters. Lastly, DEFT continues to outperform all compared methods in almost all cases when applying PEFT during the model adaptation phase (DEFT w/ PEFT), and its performance can be further enhanced by replacing PEFT with FFT (DEFT w/ FFT), confirming the validity of the proposed framework.

A.3 Additional Implementation Details

Pseudo-code for the Proposed Framework DEFT is a two-phase framework consisting of a noisy label detection phase and a model adaptation phase. The pseudo-code is presented in Algorithm 1.

Implementation of the State-of-the-arts Approaches To verify the effectiveness of DEFT on real-world datasets, we compare our methods with several sample selection-based state-of-the-art approaches including RoLT[40], UNICON[19], LongReMix[4], and ProMix[46]. Notably, these methods utilize CNN-like network backbones in their original implementations. To ensure a fair comparison, we implement a two-phase evaluation approach: we first run the source code to obtain the clean subset, then fully fine-tune the pre-trained CLIP model using these selected clean samples. We opt not to directly replace the model backbone because these methods incorporate various training techniques such as co-training, mixup, and contrastive learning. Removing the influence of these techniques is crucial to ensure a valid comparison.

Hyperparameter Settings for PEFT Modules In the noisy label detection phase, we employ VPT [18] to adapt the visual encoder by default. In all experiments, we consistently set the length of the visual prompt to 20. For LoRA and AdaptFormer, we establish the bottleneck dimension as 8.

Configurations of Pre-trained Models In the model adaptation phase, we apply FFT to a range of models that are pre-trained on ImageNet-1K. For a fair comparison, we follow the optimizer settings outlined in [1] and initialize each model with the corresponding pre-trained weights obtained from HuggingFace 1 . The specific configurations are described in Table 11.

A.4 Limitations and Broader Impacts

Limitations Despite DEFT's superior performance compared to the existing methods, it exhibits certain limitations and there are several unexplored research avenues. For example, the current algorithm only deals with noisy image-label pairs in single-label multi-class classification tasks. A promising avenue for future research would be to generalize DEFT to handle noisy image-text pairs or multi-label classification settings.

Broader Impacts This study falls within the domain of weakly supervised learning, a learning paradigm that aims to achieve superior performance while reducing labeling costs. Consequently, as this technique gains efficacy and wider adoption, the necessity for extensive data annotation may get diminished, potentially contributing to a rise in unemployment among data annotation professionals.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We provide a clear overview of the research problem and outline the main contributions in the introduction.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Section A.4.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The complete details of the implementation can be found in Section 5.1 and Section A.3.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our source code can be found via the link in the abstract.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specify all the necessary details in Section 5.1.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The paper does not report error bars following previous studies.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper provides the type of compute workers (GPU) in Section 5.1.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have reviewed the NeurIPS Code of Ethics and the research conforms with them in every respect.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We analyse the potential societal impacts in Section A.4.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The licenses for existing assets are properly respected.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The code introduced in the paper is well documented.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects.

Peer reviews:
comments: SUMMARY:
This paper proposes a novel method for learning with noisy labels leveraging pre-trained foundation models. The method is motivated by new findings that prompt learning is more robust to noisy labels when fine-tuning CLIP. The paper then designs a simple detector by learning both positive and negative prompts for each class. This paper showcases strong empirical results on multiple datasets, outperforming many existing methods.

STRENGTHS:
1.	The proposed method is well-motivated. The effectiveness of CLIP is often overlooked in previous works for learning with noisy labels, and this paper examines it thoroughly and finds a good way to adapt CLIP in downstream tasks even with noisy labels.
2.	This paper makes an important contribution that prompt learning is robust to robust to noisy labels, while full fine-tuning is better on clean datasets. Based on this observation, the paper proposes a two-stage method.
3.	This paper proposes a novel idea by jointly learning positive and negative prompts for classes, and leverage the negative learning loss to optimize learnable prompts.
4.	Extensive experiments on both synthetic and real-world datasets show that the proposed method consistently outperforms existing baselines.

WEAKNESSES:
1.	It is unclear why positive and negative prompts can help detect noisy labels. It is suggested to provide some visualization examples.

QUESTIONS:
1.	Can examples detected as noisy label help improve the performance?

LIMITATIONS:
The authors have adequately addressed the limitations and potential negative societal impact of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on improving the finetuning performance of pretrained vision models by removing the noisy labeled data. Specifically the authors propose a two-stage method: the first stage is to learn a noise detector by prompt learning of the text encoder in CLIP. The second stage is to full-finetune the pretrained models (including CLIP and other pretrained models) . In the experiment, the authors include both synthetic datasets and real datasets and compare their model with different 6 different methods. They also use the filtered dataset to finetune different pretrain models.\n\nSTRENGTHS:\n1. The paper is well writen and easy to follow.\n2. The analysis of the relationship between noisy data and finetune methods is very insightful (Figure 1)\n3. The proposed two-stage method is reasonable and results in good performance compared to other baselines.\n4. It is great to show that the filtered data works for both CLIP models and non-CLIP models even though the filtering is done by the model based on CLIP.\n\nWEAKNESSES:\n1. The optimization part is a bit unclear. (See Questions Below)\n2. The proposed noisy detector cannot be reusable when working on a different dataset. Each downstream dataset needs a specific trained detector.\n3. Noisy label problem is a more severe problem during the pretraining than the finetuning. But this paper only focuses on the finetuning, which makes the research scope narrow in this case.\n\nQUESTIONS:\n1. I am confused about the optimization for the positive prompts. Are positive prompts learnt together with negative prompts and Visual PEFT by using $L_{dp} + L_{sim}$? Or are they learned separately?\n2. Can a noisy detector learned on one dataset transferred to another dataset? For example, the noisy detector learned on ImageNet can be used to detect noise in CIFAR100 dataset since there are a great number of class overlapping in these two datasets?\n\nLIMITATIONS:\nThe authors have included the discussion of limitations in the supplementary.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for detecting noisy samples using vision-language models (CLIP). The main idea is to efficiently adapt (via prompt tunning) the clip model on noisy data and use this adapted model to select clean data. In the second stage, the clean data can be used to fully fine-tune a backbone model. The paper proposes the following contributions: a) using two learnable text prompts (positive and negative) to avoid using a threshold when detecting noise b) using negative learning [40] c) using a two-stage approach to adapt to the noisy distribution (first select clean samples then fully fine-tune using them). Experiments are done on multiple datasets with synthetic (CIFAR-100, Tiny-ImageNet, Stanford-Cars, CUB-200-2011) or real noisy labels (CIFAR-100N, Clothing1M, WebVision).\n\nSTRENGTHS:\n* S1. Good direction of efficiently adapting CLIP models for noise detection.\n* S2. The general setting is sound, of using efficiently fine-tuned CLIP to select clean data and then fine-tuning using it.\n* S3. Good results on multiple noisy datasets.\n\nWEAKNESSES:\n* W1. There needs to be more fair comparisons with baselines that are trained in the same settings and use similar models. \n\n* W2. Table 1 seems to compare DEFT which has a fine-tuning stage on the noisy dataset with the initial clip model that is not trained or adapted at all. Is this the case? Is the small-loss baseline using the initial CLIP model, without any adaptation to the current dataset? \n\n* W2.2. More appropriate baselines would be CLIP models that are adapted in standard ways to the current noisy dataset. Thus, one aspect that needs to be ablated is the fine-tuning method (none, fully fine-tuning, efficient fine-tuning, etc). Then, given a CLIP model, the second aspect that needs ablation is the selection method. Here the paper already compares against zero-shot and small-loss selection, but using the initial, pre-trained CLIP. The same should be done using an adapted clip. \n\n* W2.3. Another simple selection method that, like DEFT doesn’t require a threshold would be to select samples with $sim(I_i, T_k) > th$ (where th=0) where $T_k$ is the text features corresponding to the correct class. This selection should be different from the zero-shot selection because the threshold is applied directly on the similarity of the image and correct class features, without applying softmax, thus without taking into account if the correct class is the class predicted by the zero-shot model.\n\n* W3. Table 2 and Table 3 compare against multiple methods for training in noisy settings, starting with the same visual backbone, i.e. the visual part of CLIP. Is this correct? The question arises, is DEFT using additional information since it does the selection using both the visual and textual part of CLIP? Thus, the DEFT has a possible unfair advantage.\n\n* W4. What is the motivation for using two text prompts (positive and negative)? If only one prompt is used (as usual), we can still make a prediction and optimize this prediction to be the correct class and then apply a fixed threshold (>0.5). Why shouldn’t the simple approach work and what benefits does the dual prompt bring?\n\nQUESTIONS:\n* Is the $L_{sim}$ loss (Eq.8) used for the adaptation in the second stage, or is it used after the selection in phase 1? If the latter, would this mean that this loss updates the PEFT module and the learnable prompts, and then the final model is used in the second stage?\n\nLIMITATIONS:\nThe method is not sufficiently evaluated against fair baselines, see weak points for more details.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a Denoising Fine-Tuning (DEFT) framework to address the challenge of noisy labels in vision-language models, particularly focusing on models like CLIP. The DEFT framework leverages the robust alignment of textual and visual features pre-trained on extensive image-text pairs to filter out noisy labels. This is achieved by learning class-specific positive and negative textual prompts. Positive prompts highlight distinctive class features, while negative prompts act as thresholds to differentiate between clean and noisy samples. The framework uses parameter-efficient fine-tuning (PEFT) to adapt the visual encoder to align with the textual prompts. Extensive experiments on synthetic and real-world noisy datasets demonstrate that DEFT significantly improves both noisy label detection and image classification performance.\n\nSTRENGTHS:\n1. This paper proposes to combining textual and visual prompts for noisy label detection, which enhances the robustness of vision-language models to label noise. \n\n2. The framework's generalizability to various pre-trained models and its parameter efficiency make it a versatile solution. \n\n3. The experimental validation on multiple datasets, including real-world noisy data, provides strong evidence of the method's effectiveness. \n\n4. The use of PEFT to maintain the generalization ability of pre-trained models while adapting to specific tasks is particularly noteworthy.\n\nWEAKNESSES:\n1. The heavy reliance on pre-trained models may limit the framework's applicability in scenarios where such models are not available or suitable. \n\n2. The discussion on the practical implementation and potential limitations in different real-world settings is relatively limited. \n\n3. The computational overhead associated with maintaining and fine-tuning dual prompts could be a concern in resource-constrained environments.\n\nQUESTIONS:\n1. How does DEFT perform when applied to pre-trained models with varying degrees of generalizability and domain relevance?\n\n2. Can DEFT be adapted to scenarios with extremely high noise ratios or highly imbalanced datasets, and what modifications would be necessary to maintain its effectiveness?\n\n3. What are the potential trade-offs between the computational overhead of DEFT and its performance gains, and how can the framework be optimized for deployment in resource-constrained environments?\n\nLIMITATIONS:\nThe authors have discussed the limitations of this paper, and there is no negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}