Vision-Language Models are Strong Noisy Label Detectors

Authors: tong  wei
hao-tian  li
chun-shu  li
jiang-xin  shi
yu-feng  li
min-ling  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 06:03:18

Abstract: Recent research on fine-tuning vision-language models has demonstrated impressive performance in various downstream tasks. However, the challenge of obtaining accurately labeled data in real-world applications poses a significant obstacle during the fine-tuning process. To address this challenge, this paper presents a Denoising Fine-Tuning framework, called DeFT, for adapting vision-language models. DeFT utilizes the robust alignment of textual and visual features pre-trained on millions of auxiliary image-text pairs to sieve out noisy labels. The proposed framework establishes a noisy label detector by learning positive and negative textual prompts for each class. The positive prompt seeks to reveal distinctive features of the class, while the negative prompt serves as a learnable threshold for separating clean and noisy samples. We employ parameter-efficient fine-tuning for the adaptation of a pre-trained visual encoder to promote its alignment with the learned textual prompts. As a general framework, DeFT can seamlessly fine-tune many pre-trained models to downstream tasks by utilizing carefully selected clean samples. Experimental results on seven synthetic and real-world noisy datasets validate the effectiveness of DeFT in both noisy label detection and image classification. Our source code can be found in the supplementary material.

Main paper: 
 introduction: 
 Vision-language models pre-trained on large-scale image-text pairs, such as Contrastive Language-Image Pretraining (CLIP) [29] , have gained widespread adoption in various machine learning tasks including few-shot learning [53, 52] , multi-label learning [33, 14] , and long-tail recognition [7, 30] . Recently, CLIP has shown impressive generalization performance to many downstream tasks without the need for adaptation [36, 31] . Zero-shot CLIP leverages the strong alignment of learned visual and textual features and classifies unseen images by comparing the similarity of image embedding with textual class prompts. Despite the good zero-shot performance, fine-tuning becomes necessary when the data distribution of downstream tasks significantly deviates from the CLIP training source. Two popular fine-tuning paradigms are commonly employed for adaptation, i.e., full fine-tuning (FFT), which modifies all model parameters, and parameter-efficient fine-tuning (PEFT), which fixes the pre-trained parameters while adding a few learnable parameters for adaptation. While promising, fine-tuning CLIP necessitates perfectly labeled datasets which may not be readily available in many real-world tasks, thereby hindering their broader applicability.To tackle this issue, this paper investigates the fine-tuning of CLIP utilizing imperfectly labeled datasets, i.e., datasets with noisy labels. Intuitively, direct adaptation using noisy labels can signifi-cantly degrade the performance. To mitigate the negative impact of noisy labels, researchers have proposed various approaches for learning with noisy labels, including sample selection techniques [10, 44, 43] and noise-robust learning methods [37, 27] . However, the exploration of this problem in the context of CLIP adaptation remains limited. This paper aims to fill this research gap by demonstrating that CLIP can be leveraged as effective noisy label detectors, capitalizing on their strong alignment between visual and textual features. To ensure the generality, we follow the basic idea in mainstream studies of learning with noisy labels. Firstly, we identify potentially mislabeled samples, and subsequently, we adapt the model parameters using carefully selected clean samples.In the first phase, we propose to construct a noisy label detector by learning positive and negative textual prompts for each class. The positive prompt aims to uncover distinguishable features specific to the class, while the negative prompt functions as a learnable "threshold" for identifying noisy labels. Our noisy label detector can be derived from the similarity between embeddings of a training image and positive/negative prompts associated with its assigned class. If the image exhibits a higher similarity score with the positive prompt than the negative prompt, it is considered a correctly labeled sample, i.e., a clean sample. This design draws inspiration from previous studies that have shown the robustness of prompt tuning to noisy labels, particularly in the presence of high noise ratios [41] . To optimize the positive prompt, we align its representation with image features from the corresponding class during training. For the negative prompts, we introduce a novel negative learning objective. It is important to note that calculating similarities between images and textual prompts requires a strong alignment between these two modalities in the downstream task. To achieve this, we utilize PEFT, such as Visual Prompt Tuning (VPT) [18] , for the adaptation of the visual encoder. This decision is based on a key finding that FFT can distort pre-trained feature representations when noisy labels are present, which is discussed in Section 3.2.In the second phase, we can adapt the pre-trained model after acquiring carefully selected clean samples. It is worth noting that not only can we employ pre-trained CLIP for adaptation, but we can also utilize other pre-trained visual backbones such as ResNet [12] and MAE [11] . While the model adaptation phase may seem unnecessary since we can leverage the learned textual prompts for zero-shot classification without further fine-tuning, we emphasize its necessity by the improved performance on curated downstream datasets, particularly in fine-grained classification tasks. In this step, we learn a linear classifier and fully fine-tune the visual encoders using only selected clean samples. The rationale behind this approach is twofold: 1) the linear classifier tends to generalize better than textual class prompts, and 2) full fine-tuning can effectively adapt model parameters to address significant domain shifts, surpassing the capabilities of PEFT. In our implementation, the training is conducted for only 10 epochs, resulting in a minimal increase in computation cost.Our main contributions can be summarized as follows:• We propose DEFT, a simple yet effective framework for learning with noisy labels. DEFT offers several compelling advantages over existing methods: instance-dependent (no information required from entire training data), robust to various types of noisy labels, and generalizable to many pre-trained models. • We conduct extensive experiments and show that DEFT achieves superior performance in both noisy label detection and image classification tasks on a wide range of synthetic and real-world datasets. • We provide in-depth empirical analysis, providing insights to understand the effectiveness of DEFT. We hope that this work will serve as a springboard for future works on noisy label detection with multi-modal features. 
related work: 
 Learning with Noisy Labels A diverse variety of approaches have been proposed for addressing learning with noisy labels and can be broadly divided into twofold: robust learning and clean sample selection [9, 32] . Due to the susceptibility of traditional cross-entropy loss to noisy labels, several methods adopt noise-robust loss functions [8, 51, 37, 6] and regularization techniques [27, 38, 16] to enhance model robustness. For example, symmetric cross-entropy loss [37] introduces a noise-robust objective function for fast convergence and better performance. Early-learning regularization [27] counteracts the influence of the noisy labels by introducing a regularization term that integrates estimated target probabilities. However, empirical findings suggest that robust model learning faces the problem of maintaining a balance between robustness and accuracy [51, 35] . Conversely, sample selection approaches strive to select clean samples from noisy datasets using specific criteria. Based on the memorization effect identified in deep neural networks [2] , Numerous methods adopt the small-loss strategy that treats samples with small loss as clean ones [10, 15, 25, 19] . Co-teaching [10] selects a fraction of training data in each mini-batch based on the noise ratio, whereas DivideMix [25] fits a Gaussian mixture model to dynamically divide the training samples. Although these methods demonstrate strong performance, they inevitably overlook the significance of clean hard samples with a large loss. Besides, all the aforementioned approaches concentrate exclusively on learning from scratch with label noise. Our proposed approach leverages the pre-trained multi-modal information from visual-language models to construct a noisy label detector, resulting in improved sample selection performance.Fine-Tuning Vision-Language Models Recent years have witnessed incredible progress in pretrained visual-language models [29, 17, 49] . As a representative method, CLIP [29] employs contrastive pre-training to acquire informative representations from extensive image-text pairs. Despite the remarkable zero-shot performance of CLIP, its sensitivity to hand-crafted textual prompts is significant. Prompt-tuning aims to learn the optimal prompts from target data by replacing the textual templates with adaptable soft prompts, which demonstrates superior performance in the context of few-show learning [53, 52, 20] and multi-label learning [33, 14] . However, further enhancements can be achieved by adapting the visual encoder to accommodate larger datasets in downstream tasks. A standard approach to obtain task-specific representations is fine-tuning all model parameters. While FFT yields significant performance on curated datasets, recent work [1] empirically finds that FFT can result in feature distortion in the presence of massive noisy labels. In contrast, PEFT [48] has been verified to improve generalization capabilities for mitigation of overfitting issues [30] , which fix the pre-trained model and introduce a few number of learnable parameters for adaptation, such as VPT [18] , LoRA [13] , and AdaptFormer [3] . Considering the distinct property of fine-tuning methods, this paper aims to delineate the most effective paradigm for model adaptation on noisy downstream datasets, which remains under-explored yet holds significant promise.3 Preliminary and Initial Findings 
zero-shot clip: 
 This paper focuses on a K-class image classification task, where the objective is to categorize an image x ∈ R d and assign it a label y ∈ [K] = {1, 2, . . . , K}. Benefiting from large-scale datasets and high-capacity models, CLIP [29] has demonstrated remarkable performance in zero-shot classification by computing the similarity between visual and textual representations. Specifically, with hand-crafted textual prompts such as "a photo of a [CLS]", where the class token is replaced by a specific class name, the prediction probability is computed as:EQUATIONwhere sim(•, •) denotes the cosine similarity between the extracted image features I and text features T , and τ is the softmax temperature.While zero-shot classification with CLIP is promising, its performance can be further enhanced by adapting CLIP on downstream labeled datasets D = {(x i , y i ) N i=1 } to incorporate more task-specific representations. In this paper, we investigate the presence of noisy labels in the training dataset D, which is prevalent in real-world applications. We use r ∈ [0, 1] to denote the noise ratio of a training set, meaning that N × r training samples are incorrectly labeled, i.e., the assigned label y differs from the ground-truth label y * . Since y * is not accessible, our goal is to fine-tune the CLIP model using training labels y by eliminating the negative influence of noisy labels. Considering the feature distortion induced by noisy labels [2] , the key challenge lies in obtaining distinguishable yet robust representations when adapting CLIP on noisy datasets.To discern the most effective method for CLIP adaptation, we conduct empirical studies on various datasets as shown in Figure 1 . Specifically, we utilize three fine-tuning approaches to adapt the pre-trained CLIP model and compare their performance on both noisy and clean datasets, including 1) FFT which updates the entire model parameters, 2) VPT which fixes pre-trained model parameters and prepends a small subset of extra learnable parameters to the visual encoder during fine-tuning, and 3) VLPT (Vision-Language Prompt Tuning) which integrates both visual and textual learnable prompts into the fixed pre-trained model for fine-tuning. For FFT and VPT, we learn an additional linear classifier, while VLPT directly utilizes the learned textual prompts for classification. We present our three primary findings in the following. VPT benefits representation learning in the presence of massive noisy labels. Intuitively, utilizing FFT for CLIP adaptation yields improved performance by leveraging its substantial capacity to incorporate task-specific representations. However, the efficacy of FFT diminishes notably when applied to datasets containing noisy labels, as shown in Figure 1a . This decline is attributed to the pronounced distortion of representations with an escalating noise ratio [1] . In contrast, VPT benefits representation learning when adapting CLIP on noisy data, particularly under high levels of noise. As only a small set of parameters is introduced, VPT efficiently retains the generalization ability from image-text pre-training while enhancing classification performance on downstream tasks, making it a robust and effective fine-tuning approach for handling noisy downstream datasets. Textual classifier is robust to noisy labels. Classification with learnable textual prompts (e.g., CoOp [53] ) leverages the multi-modal information in pre-trained vision-language models and enhances the alignment of visual and textual representations on downstream tasks. Recent literature has substantiated its robustness with a frozen visual encoder in the context of few-shot learning [41] . From Figure 1a , we observe that VLPT with a textual classifier (green line) consistently surpasses VPT with a traditional linear classifier for classification (orange line), especially under severe noise. The improvement in performance across diverse noise ratios further affirms the robustness of learnable textual prompts in mitigating the impact of label noise for model adaptation. FFT enhances visual recognition on clean datasets. Although previous findings show that FFT is more susceptible to noisy labels compared to VPT and VLPT, Figure 1b demonstrates its enhanced discriminative ability when training data is clean. This superiority is particularly significant on fine-grained datasets such as Stanford-Cars and CUB-200-2011, with an average improvement of 2.81% against VPT. It is important to note that VLPT exhibits the worst performance on clean datasets. This is primarily due to the implicit regularization of pre-trained textual information when tuning the context of textual prompts, as explained in the recent study [41] . 
the denoising fine-tuning framework: 
 Based on the findings in Section 3.2, we now present a novel denoising fine-tuning framework, DEFT, which comprises two pivotal phases. In the first phase, DEFT learns dual textual prompts to separate clean and noisy samples while adapting the visual encoder utilizing PEFT methods. In the second phase, DEFT re-adapts the pre-trained model using FFT, leveraging the curated clean samples to further boost visual recognition performance. Figure 2 illustrates our proposed framework.Given the empirically demonstrated robustness of textual prompts to label noise, we intend to harness this property to identify corrupted samples in noisy downstream datasets. The initial strategy is to select a subset of samples with pronounced similarity between their visual and textual representations and designate them as clean. However, this method relies on either prior knowledge of the noise ratio or the manual setting of thresholds, which restricts its practical applicability. Inspired by recent works [33, 14, 36] , we utilize dual textual prompts instead of a single prompt to overcome these limitations. Concretely, we design a class-specific pair of positive and negative prompts for the textual encoder, denoted as prompt + k and prompt - k , respectively:EQUATIONHere, [V ] m is the word vector of context (with dimension 512 in CLIP), M is the length of context, and [CLS] k is the k-th provided category name. The positive prompt aims to uncover distinguishable features by maximizing the similarity between the image features and their corresponding text features, while the negative prompt serves as a learnable sample-dependent similarity threshold to select clean data. Specifically, the learnable threshold ϕ i for the i-th training sample with label y i = k is formulated as:EQUATIONwhich represents the cosine similarity between the extracted image features and negative text features.Based on this threshold, the clean subset of dataset D can be constructed as follows:EQUATIONThe proposed selection strategy surpasses the conventional loss-based approaches in two aspects:1) It is more practical by utilizing data-driven thresholds thus eliminating the requirement for prior knowledge like noise ratio, and 2) the integration of text modality enhances its robustness to label noise, making it capable of identifying challenging hard noise. Nevertheless, the primary dilemma lies in the optimization of positive and negative prompts using noisy downstream datasets.It can be an intricate task to optimize the threshold ϕ i in an end-to-end manner since it is indirectly involved in the forward propagation. However, we can still optimize the associated parameters by formulating the clean probability of the i-th image with y i = k as:EQUATIONObviously, selecting samples with the clean probability p clean ik > 0.5 is equivalent to the specified criteria in Eq. (5) . In this way, the original threshold-based selection approach can be transformed into K binary classification tasks to determine whether a sample is clean or not. Concretely, given a sample (x i , y i ) from noisy datasets D, we can employ the dual prompts of class y i as a binary classifier to identify label noise. If the classifier outputs a positive prediction of x i , it is categorized as a clean sample.To optimize the noisy label detector, we need to construct positive and negative training samples for binary classification, i.e., images with correctly and wrongly assigned labels. Inspired by the prior work on negative learning [21] , we designate each image with a randomly picked complementary label ȳ to form negative samples. For positive samples, we initiate with the original supervision in D and refine it with pseudo labels ŷ generated by Eq. ( 1). The training loss is then formulated as follows:EQUATIONwherep clean i = {p clean ik } K k=1is the clean probability and ℓ nll (•, •) is the negative log-likelihood loss where ℓ nll (p i , y) = -log p iy .Additionally, to enhance task-specific representations of the noisy label detector, we harness PEFT for the adaptation of the visual encoder, considering its robustness under massive noisy labels compared with FFT, as per our previous finding (see Figure 1a ). This is achieved by maximizing the alignment between image embeddings and their corresponding positive textual features:EQUATIONNote that L sim is exclusively computed on D clean after the warm-up stage to further eliminate the impact of label noise on representations. With the objective L = L dp +L sim , DEFT effectively sieves out noisy samples with learned thresholds while uncovering distinguishable visual representations on downstream tasks.Although the learned positive textual prompt can be readily employed for classification, its performance may be suboptimal on curated clean datasets, as demonstrated in our previous finding (see Figure 1b ). To mitigate this problem, we introduce the model adaptation phase which learns a linear classifier using selected clean samples, i.e., D clean . Moreover, our findings also indicate that FFT outperforms PEFT on clean datasets, so we remove the PEFT modules and fully fine-tune the pre-trained model parameters for model adaptation. We minimize the classic cross-entropy loss ℓ ce (x, y) = -log exp(zy) K k=1 exp(z k ) , where z = {z k } K k=1 represents the logits predicted by the linear classifier. It is noteworthy that with the clean subset of training data constructed by Eq. ( 5), the second phase can be applied universally to a wide range of pre-trained models, regardless of their backbones. The versatility of our approach is demonstrated in the experiments. 
experiment: 
 We now proceed to the experiments, demonstrating the advantages of our approach in noisy label detection and image classification on both synthetic and real-world datasets.Synthetic Datasets We first evaluate the performance of DEFT on four image classification benchmarks by synthesizing noisy labels with varying noise types and ratios. Specifically, we conduct experimental analyses on widely-used CIFAR-100 [23] and Tiny-ImageNet [42] , as well as two fine-grained datasets Stanford-Cars [22] and CUB-200-2011 [34] . Given the noise transition matrix T ∈ [0, 1] K×K with T ij denoting the probability of the ground-truth y * = i being flipped into a corrupted label y = j, we introduce two types of label noise: 1) T ij = p(y = j | y * = i), where the corruption probability is assumed to be only dependent on the true label, e.g., the symmetric noise generated by replacing y * with a random label for a given proportion of training samples [10, 25] , and 2) T ij = p(y = j | y * = i, x), which is a more realistic assumption considering the impact of instance x in the label corruption process, e.g., the instance-dependent noise proposed in [45] . We conduct extensive experiments on each dataset with noise ratio r ∈ {0.2, 0.4, 0.6} for symmetric noise and r ∈ {0.2, 0.3, 0.4} for instance-dependent noise.We further examine the performance of DEFT on three real-world noisy label datasets: 1) CIFAR-100N [39] (r ≈ 0.4) is a variant of CIFAR-100 with real-world human annotations from Amazon's Mechanical Turk, 2) Clothing1M [47] (r ≈ 0.4) is a large-scale dataset consisting of 1 million clothing images of 14 classes collected from online shopping websites, and On each dataset, we compare the Precision (%) and Recall (%) of DEFT with CLIP labelmatch and small-loss to evaluate the clean sample selection performance. ∆ is the difference between the performance of DEFT and small-loss.3) WebVision [26] (r ≈ 0.2) contains 2.4 million images crawled from Flickr and Google using the 1,000 concepts in ImageNet ILSVRC12. Following previous works [25, 19, 24] , we experiment on the first 50 classes of the Google image subset.Evaluation Metrics We use precision and recall metrics to evaluate the selection of clean samples. A higher precision indicates a greater proportion of actual clean samples in D clean , while a higher recall means that more clean samples are identified from the noisy dataset. For image classification, We report the "best" test accuracy across all training iterations and the "last" test accuracy at the end of training.We adopt the pre-trained CLIP [29] with the Transformer as the text encoder and the ViT-B/16 as the image encoder. We use the SGD optimizer with a momentum of 0.9, a weight decay of 5×10 -4 , and a batch size of 64. We run 10 epochs for both the noisy label detection phase and the model adaptation phase with learning rates 3×10 -2 and 5×10 -4 , respectively. In the noisy label detection phase, we employ VPT [18] and CoOp [53] to adapt visual encoder and textual encoder respectively, and perform model warm-up for 1 epoch on all datasets. When identifying noisy labels on real-world datasets, we augment the condition in Eq. ( 5) with a constraint that the training label should be consistent with its predicted label. This is due to the existence of more complex noise patterns in real-world tasks. All experiments are conducted on a single NVIDIA GeForce RTX 3090.We conduct a thorough evaluation of DEFT to justify its effectiveness in detecting noisy labels. For this purpose, we simulate various types and ratios of label noise on four benchmark datasets. We compare our approach with two other sample selection strategies: 1) Label-match strategy, where a training sample is deemed clean if its given label matches that assigned by the CLIP zero-shot classifier, and 2) Small-loss strategy, which selects a proportion (noise ratio) of confident samples in each mini-batch, as commonly used in previous studies [10, 25, 19] . The precision and recall of sample selection are reported in Table 1 . The results show that our proposed noisy label detector outperforms the compared strategies, with significant improvements observed in all dataset settings. The trivial label-match strategy tends to be conservative, leading to low selection recall of clean samples. In contrast, the small-loss strategy and DEFT achieve a better trade-off between precision and recall, making better use of training samples. Moreover, leveraging the informative multimodel information in the pre-trained vision-language model, DEFT surpasses the small-loss strategy in detecting noisy labels, particularly under severe noise settings and fine-grained classification tasks. For instance, in Tiny-ImageNet with 60% symmetric noise, DEFT demonstrates significant enhancements of 4.58% and 4.55% in precision and recall, respectively, as well as an improvement of 9.78% and 14.42% in CUB-200-2011. Additionally, DEFT reduces the need for prior knowledge of noise ratios, making it a practical and effective approach to detect label noise in real-world tasks.We evaluate the performance of DEFT in image classification tasks against four baselines, including CE (Cross-Entropy loss), SCE [37] , ELR [27] , and GMM [25] . To further verify the effectiveness of our method on real-world datasets, we also make comparisons with some recent sample selectionbased works including RoLT [40] , UNICON [19] , LongReMix [4] , and ProMix [46] . We utilize FFT for adapting the pre-trained CLIP model for all approaches. More details can be found in the Appendix. The results demonstrate the effectiveness of our proposed approach in tackling both symmetric and instance-dependent label noise.The results reported in Table 3 demonstrate the superiority of our method on real-world datasets, as DEFT surpasses other methods by a significant margin on all datasets. Notably, while most compared methods show little improvement on Clothing1M due to its fine-grained nature, both ELR and DEFT demonstrate substantial improvement. However, ELR exhibits a performance decrease on WebVision. Moreover, directly fine-tuning pre-trained CLIP models with cross-entropy (CE) achieves competitive results on WebVision, yet DEFT elevates this performance further, showcasing its ability to combat label noise in practical situations.Necessity of Model Adaptation In this experiment, we emphasize the necessity of the model adaptation phase in DEFT. Figure 3 demonstrates the test accuracy of DEFT without the model adaptation phase (w/o adap.), adaptation utilizing PEFT and FFT. Results show that fully fine-tuning the model on selected clean samples yields the best classification performance, especially on finegrained datasets, such as Stanford-Cars and CUB-200-2011. The results unveil two primary insights: 1) the noisy label detector in DEFT exhibits strong capabilities in detecting label noise, yielding a nearly clean subset from noisy downstream datasets, and 2) employing FFT for model adaptation is more effective in mitigating significant domain shifts between downstream data and pre-training data, particularly evident in fine-grained datasets.DEFT for Various Pre-trained Models It is noteworthy that DEFT can seamlessly integrate with various pre-trained visual backbones during the model adaptation phase. To demonstrate the versatility of our proposed denoising fine-tuning framework, we conduct experiments on Clothing1M and apply FFT to a range of pre-trained models, including ViT-B/16 [5] , ResNet-50 [12] , ConvNeXt-T [28] , and MAE-VIT-B [11] . The results are presented in Table 4 , where GCE [51] employs a noise-robust loss function akin to SCE. As a recent work, TURN [1] is an improved version of GMM, which applies linear probing to initialize the classifier and then uses FFT for adaptation. Compared with previous methods, DEFT exhibits the best performance on average. In particular, DEFT outperforms TURN by 4.51% and 3.27% when adopting ResNet-50 and MAE-ViT-B as the target pre-trained models, respectively. 
conclusion: 
 In this work, we delve into a new landscape for learning with noisy labels, departing from the classic single-modal toward a multi-modal regime. By learning dual textual prompts, we construct a new noisy label detection approach, which offers several compelling advantages: robust to various types of label noise, generalizable to many pre-trained models, and does not require the dynamics of training samples. We investigate the effectiveness of DEFT on a wide range of synthetic and real-world datasets, showing its superior performance in both noisy label detection and image classification tasks. Lastly, we demonstrate the advantage of parameter-efficient fine-tuning over full fine-tuning over noisy label detection. We hope our work will inspire future research toward multi-modal noisy label detection. 
 
 Appendix: A.1 Additional Results on Noisy Label Detection Impact of CLIP Backbones Table 1 has demonstrated the superiority of DEFT in detecting noisy labels based on CLIP ViT-B/16. To evaluate the impact of different visual backbones on noise detection, we re-implement all methods using CLIP ViT-B/32 as the backbone and report the results in Table 5 . Notably, while the small-loss strategy demonstrates comparable performance on noisy CIFAR-100 and Tiny-ImageNet, its efficacy significantly diminishes on fine-grained datasets such as Stanford-Cars and CUB-200-2011, which exhibits its sensitivity to visual backbones. In contrast, DEFT maintains consistently superior performance on both ViT-B/16 and ViT-B/32, underscoring its resilience across different pre-trained visual backbones.We conduct experiments on synthetic noisy datasets with significant label noise to further validate the robustness of the proposed noisy label detector. It is noteworthy that two adjustments are implemented in order to train a more effective model: 1) the learning rate is adjusted from 3×10 -2 to 1×10 -2 , and 2) For DEFT, a small weight (specifically, 0.25) is introduced to the positive loss component in Eq. ( 7) to mitigate the impact of noisy pseudo labels in high levels of noise. Table 6 reports the precision, recall, and their harmonic mean (F1-score) for noise detection, where DEFT exhibits a robust capability in identifying severe noise. For example, a remarkable improvement in the F1-score is achieved on CUB-200-2011. These results strongly reaffirm the superior performance of DEFT across various noise ratios.GMM for Noisy Label Detection DivideMix [25] introduces the use of a Gaussian Mixture Model (GMM) for the dynamic separation of clean and noisy samples with an adjustable threshold. Table 7 summarizes the performance of GMM and DEFT in detecting label noise across diverse datasets and noise settings. To fine-tune the pre-trained model for GMM, we independently apply PEFT and FFT.The results reveal that DEFT consistently exhibits superior noise detection capability, particularly evident in fine-grained datasets with high noise ratios, such as Stanford-Cars with 60% symmetric noise and CUB-200-2011 with 40% instance-dependent noise. Intriguingly, implementing GMM with PEFT yields better noise detection when compared to utilizing FFT. This difference can be attributed to the potentially unstable training process of FFT, where the loss value may undergo significant fluctuations due to an abundance of tunable parameters.Noisy Label Detection on CIFAR-100N The CIFAR-100N dataset [39] , which serves as a realworld benchmark by providing CIFAR-100 with human-annotated noisy labels, allows us to evaluate 8 reveal the distinctive performance characteristics of various sample selection strategies. For example, the label-match strategy demonstrates a propensity for being greedy and yields the highest precision, whereas the GMM and recent sample selection-based works tends to be conservative and thus achieve a higher recall. Nevertheless, DEFT stands out with the highest F1-score across all strategies, striking a judicious balance between precision and recall in the identification of real-world noisy labels.We explore the sensitivity of our noise label detector to various hyperparameter settings, specifically the length of the visual/textual prompt and the number of training epochs. Table 9 presents the F1-score on CIFAR-100 with 40% symmetric noise. The results indicate that the length of the visual prompt has a marginal impact on the noisy label detection. For the textual prompt length, once the context length exceeds 8, the performance is enhanced slightly as the length increases. Additionally, variations in the number of training epochs have minimal impact on performance. In order to strike a balance between effectiveness and efficiency, we set the visual prompt length to 20, the textual prompt length to 16, and the training epoch to 10 in all experiments.Robustness of Various PEFT techniques To further substantiate the advantages of PEFT over FFT for adapting visual backbones in the presence of massive noisy labels, we conduct a comprehensive evaluation of various PEFT techniques for image classification, including 1) Visual Prompt Tuning (VPT) [18] which prepends learnable prompts to the input at each layer, 2) Low-Rank Adapter (LoRA) [13] which injects trainable rank decomposition matrices into each layer, 3) AdaptFormer [3] which replaces the original Multi-Layer Perceptron (MLP) block with AdaptMLP that incorporates a trainable bottleneck module, and 4) Bias-terms Fine-tuning (BitFit) [50] which is a sparse fine- Applying PEFT for Model Adaptation Table 10 presents the test accuracy obtained by applying PEFT to adapt the pre-trained CLIP model. There are three noteworthy observations in comparison to the results in Table 2 . Firstly, previous label-noise learning methods consistently demonstrate their efficacy in handling noisy datasets compared to the vanilla methods (CE), showcasing a clear improvement in performance across various datasets and noise settings. Secondly, despite the bad performance of GMM on fine-grained datasets in Table 2 , it achieves the best performance amongLimitations Despite DEFT's superior performance compared to the existing methods, it exhibits certain limitations and there are several unexplored research avenues. For example, the current algorithm only deals with noisy image-label pairs in single-label multi-class classification tasks. A promising avenue for future research would be to generalize DEFT to handle noisy image-text pairs or multi-label classification settings.Broader Impacts This study falls within the domain of weakly supervised learning, a learning paradigm that aims to achieve superior performance while reducing labeling costs. Consequently, as this technique gains efficacy and wider adoption, the necessity for extensive data annotation may get diminished, potentially contributing to a rise in unemployment among data annotation professionals.38th Conference on Neural Information Processing Systems (NeurIPS 2024).https://huggingface.co/ 


Decision: The paper introduces DeFT, a Denoising Fine-Tuning framework that effectively adapts vision-language models to handle noisy labeled data. The key innovation of this work lies in the use of positive and negative textual prompts to enhance the alignment of visual and textual features, which helps in detecting and mitigating the impact of noisy labels. DeFT's versatility in being applicable to various pre-trained models, along with its parameter-efficient approach, is well-supported by strong experimental results on both synthetic and real-world noisy datasets. All reviewers agree that the paper provides significant contributions and is ready for publication. The AC agrees with this assessment.

Reviews: review: paper_summary: paper_summary: The paper introduces a Denoising Fine-Tuning (DEFT) framework to address the challenge of noisy labels in vision-language models, particularly focusing on models like CLIP. The DEFT framework leverages the robust alignment of textual and visual features pre-trained on extensive image-text pairs to filter out noisy labels. This is achieved by learning class-specific positive and negative textual prompts. Positive prompts highlight distinctive class features, while negative prompts act as thresholds to differentiate between clean and noisy samples. The framework uses parameter-efficient fine-tuning (PEFT) to adapt the visual encoder to align with the textual prompts. Extensive experiments on synthetic and real-world noisy datasets demonstrate that DEFT significantly improves both noisy label detection and image classification performance.
strength_weakness: strength_weakness: 1. The heavy reliance on pre-trained models may limit the framework's applicability in scenarios where such models are not available or suitable. 

2. The discussion on the practical implementation and potential limitations in different real-world settings is relatively limited. 

3. The computational overhead associated with maintaining and fine-tuning dual prompts could be a concern in resource-constrained environments.
questions: questions: 1. How does DEFT perform when applied to pre-trained models with varying degrees of generalizability and domain relevance?

2. Can DEFT be adapted to scenarios with extremely high noise ratios or highly imbalanced datasets, and what modifications would be necessary to maintain its effectiveness?

3. What are the potential trade-offs between the computational overhead of DEFT and its performance gains, and how can the framework be optimized for deployment in resource-constrained environments?
limitations: limitations: The authors have discussed the limitations of this paper, and there is no negative societal impact.
paper_summary: paper_summary: The paper proposes a method for detecting noisy samples using vision-language models (CLIP). The main idea is to efficiently adapt (via prompt tunning) the clip model on noisy data and use this adapted model to select clean data. In the second stage, the clean data can be used to fully fine-tune a backbone model. The paper proposes the following contributions: a) using two learnable text prompts (positive and negative) to avoid using a threshold when detecting noise b) using negative learning [40] c) using a two-stage approach to adapt to the noisy distribution (first select clean samples then fully fine-tune using them). Experiments are done on multiple datasets with synthetic (CIFAR-100, Tiny-ImageNet, Stanford-Cars, CUB-200-2011) or real noisy labels (CIFAR-100N, Clothing1M, WebVision).
strength_weakness: strength_weakness: * W1. There needs to be more fair comparisons with baselines that are trained in the same settings and use similar models. 

* W2. Table 1 seems to compare DEFT which has a fine-tuning stage on the noisy dataset with the initial clip model that is not trained or adapted at all. Is this the case? Is the small-loss baseline using the initial CLIP model, without any adaptation to the current dataset? 

* W2.2. More appropriate baselines would be CLIP models that are adapted in standard ways to the current noisy dataset. Thus, one aspect that needs to be ablated is the fine-tuning method (none, fully fine-tuning, efficient fine-tuning, etc). Then, given a CLIP model, the second aspect that needs ablation is the selection method. Here the paper already compares against zero-shot and small-loss selection, but using the initial, pre-trained CLIP. The same should be done using an adapted clip. 

* W2.3. Another simple selection method that, like DEFT doesn’t require a threshold would be to select samples with $sim(I_i, T_k) > th$ (where th=0) where $T_k$ is the text features corresponding to the correct class. This selection should be different from the zero-shot selection because the threshold is applied directly on the similarity of the image and correct class features, without applying softmax, thus without taking into account if the correct class is the class predicted by the zero-shot model.

* W3. Table 2 and Table 3 compare against multiple methods for training in noisy settings, starting with the same visual backbone, i.e. the visual part of CLIP. Is this correct? The question arises, is DEFT using additional information since it does the selection using both the visual and textual part of CLIP? Thus, the DEFT has a possible unfair advantage.

* W4. What is the motivation for using two text prompts (positive and negative)? If only one prompt is used (as usual), we can still make a prediction and optimize this prediction to be the correct class and then apply a fixed threshold (>0.5). Why shouldn’t the simple approach work and what benefits does the dual prompt bring?
questions: questions: * Is the $L_{sim}$ loss (Eq.8) used for the adaptation in the second stage, or is it used after the selection in phase 1? If the latter, would this mean that this loss updates the PEFT module and the learnable prompts, and then the final model is used in the second stage?
limitations: limitations: The method is not sufficiently evaluated against fair baselines, see weak points for more details.
paper_summary: paper_summary: This paper focuses on improving the finetuning performance of pretrained vision models by removing the noisy labeled data. Specifically the authors propose a two-stage method: the first stage is to learn a noise detector by prompt learning of the text encoder in CLIP. The second stage is to full-finetune the pretrained models (including CLIP and other pretrained models) . In the experiment, the authors include both synthetic datasets and real datasets and compare their model with different 6 different methods. They also use the filtered dataset to finetune different pretrain models.
strength_weakness: strength_weakness: 1. The optimization part is a bit unclear. (See Questions Below)
2. The proposed noisy detector cannot be reusable when working on a different dataset. Each downstream dataset needs a specific trained detector.
3. Noisy label problem is a more severe problem during the pretraining than the finetuning. But this paper only focuses on the finetuning, which makes the research scope narrow in this case.
questions: questions: 1. I am confused about the optimization for the positive prompts. Are positive prompts learnt together with negative prompts and Visual PEFT by using $L_{dp} + L_{sim}$? Or are they learned separately?
2. Can a noisy detector learned on one dataset transferred to another dataset? For example, the noisy detector learned on ImageNet can be used to detect noise in CIFAR100 dataset since there are a great number of class overlapping in these two datasets?
limitations: limitations: The authors have included the discussion of limitations in the supplementary.
paper_summary: paper_summary: This paper proposes a novel method for learning with noisy labels leveraging pre-trained foundation models. The method is motivated by new findings that prompt learning is more robust to noisy labels when fine-tuning CLIP. The paper then designs a simple detector by learning both positive and negative prompts for each class. This paper showcases strong empirical results on multiple datasets, outperforming many existing methods.
strength_weakness: strength_weakness: 1.	It is unclear why positive and negative prompts can help detect noisy labels. It is suggested to provide some visualization examples.
questions: questions: 1.	Can examples detected as noisy label help improve the performance?
limitations: limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.

Comments: title: Official Comment by the Authors
Response by Authors
Response to rebuttal
comment: Dear Reviewer g1PL,

We appreciate your thorough evaluation and helpful suggestions and comments. In our response, we have provided point-by-point responses to your specific comments. We hope our response addresses all the concerns raised in your review. 

Since the author-reviewer discussion ends soon, we are happy to hear your thoughts if you need additional clarifications from the authors. Thank you very much.

Best
Dear Reviewer FyAK,

Thank you for your thoughtful suggestions and the positive feedback on our work.
The authors address all my concerns and questions. Thanks for the effort in making the rebuttal. I changed my rating from 5 to 6.
Dear Reviewer 6CXy,

We sincerely appreciate your valuable comments and encouraging feedback.

For the adapted CLIP model in W4, the minimum $sim(I_i, T_k)$ is $-0.443$, and the F1-scores for noise detection at different thresholds are presented in the table below. These results indicate that the adapted CLIP model is also sensitive to the choice of threshold. We will add the new baseline in the revised version.


| Threshold |  0.0  |  0.1  |  0.2  |  0.3  |  0.4  |  0.5  |
| --------- | :---: | :---: | :---: | :---: | :---: | :---: |
| F1-score  | 80.77 | 85.72 | 89.89 | 91.93 | 92.75 | 91.81 |
I thank the authors for the response. I appreciate the clarification of the Small-loss baseline and the additional ablations. 

In W2.3. you show that the initial, pre-trained CLIP model is sensible to the threshold and it makes sense. How about the adapted CLIP model, as asked in W4? 

Overall I tend to increase my score to 5.