Vision-Language Models are Strong Noisy Label Detectors
Authors: Tong Wei
Hao-Tian Li
Chun-Shu Li
Jiang-Xin Shi
Yu-Feng Li
Min-Ling Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: label-noise learning
sample selection
semi-supervised learning
Primary area: machine_vision
TL;DR: This paper proposes a denoising fine-tuning framework to adapt vision-language models on noisy downstream tasks.

Abstract: Recent research on fine-tuning vision-language models has demonstrated impressive performance in various downstream tasks. However, the challenge of obtaining accurately labeled data in real-world applications poses a significant obstacle during the fine-tuning process. To address this challenge, this paper presents a Denoising Fine-Tuning framework, called DeFT, for adapting vision-language models. DeFT utilizes the robust alignment of textual and visual features pre-trained on millions of auxiliary image-text pairs to sieve out noisy labels. The proposed framework establishes a noisy label detector by learning positive and negative textual prompts for each class. The positive prompt seeks to reveal distinctive features of the class, while the negative prompt serves as a learnable threshold for separating clean and noisy samples. We employ parameter-efficient fine-tuning for the adaptation of a pre-trained visual encoder to promote its alignment with the learned textual prompts. As a general framework, DeFT can seamlessly fine-tune many pre-trained models to downstream tasks by utilizing carefully selected clean samples. Experimental results on seven synthetic and real-world noisy datasets validate the effectiveness of DeFT in both noisy label detection and image classification. Our source code can be found in the supplementary material.

Vision-Language Models are Strong Noisy Label Detectors
Tong 12,
Hao-Tian Li! Chun-Shu Li
Jiang-Xin Shi
Yu-Feng Li3,4 Min-Ling Zhang
School of Computer Science and Engineering, Southeast University, Nanjing, China
2 Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China
3 National Key Laboratory for Novel Software Technology, Nanjing University, China 'School of Artificial Intelligence, Nanjing University, China {weit, 1int) @seu. edu. cn
Abstract
Recent research on fine-tuning vision-language models has demonstrated impres- sive performance in various downstream tasks. However, the challenge of obtain- ing accurately labeled data in real-world applications poses significant obstacle during the fine-tuning process. To address this challenge, this paper presents a Denoising Fine-Tuning framework, called DEFT, for adapting vision-language models. DEFT utilizes the robust alignment of textual and visual features pre- trained on millions of auxiliary image-text pairs to sieve out noisy labels. The pro- posed framework establishes noisy label detector by learning positive and negative textual prompts for each class. The positive prompt seeks to reveal distinctive fea- tures of the class, while the negative prompt serves as a learnable threshold for sep- arating clean and noisy samples. We employ parameter-efficient fine-tuning for the adaptation of a pre-trained visual encoder to promote its alignment with the learned textual prompts. As general framework, DEFT can seamlessly fine-tune many pre-trained models to downstream tasks by utilizing carefully selected clean sam- ples. Experimental results on seven synthetic and real-world noisy datasets validate the efffcctiveness of DEFT in both noisy label detection and image classification tasks. Our source code is available at hhtpss: //gith/bb com Ho IoootanLee/DDee DeFT
Introduction
Vision-language models pre-trained on large-scale image-text pairs, such as Contrastive Language- Image Pretraining (CLIP) [29], have gained widespread adoption various machine learning tasks including few-shot learning 533 12), multi-label learning [33 14), and long-tail recognition [7] 30).
Recently, CLIP has shown impressive generalization performance to many downstream tasks without the need for adaptation 36 31. Zero-shot CLIP leverages the strong alignment of learned visual and textual features and classifies unseen images by comparing the similarity of image embedding with textual class prompts. Despite the good zero-shot performance, fine-tuning becomes necessary when the data distribution of downstream tasks significantly deviates from the CLIP training source. Two popular fine-tuning paradigms are commonly employed for adaptation, i.e., full fine-tuning (FFT), which modifies all model parameters, and parfmeter-efficien fine- min-ttnnngg (PEFT), which fixes the pre-trained parameters while adding few learnable parameters for adaptation. While promising, fine-tuning CLIP necessitates perfectly labeled datasets which may not be readily available in many real-world tasks, thereby hindering their broader applicability.
To tackle this issue, this paper investigates the tine-tuning of CLIP utilizing imperfectly labeled datasets, i.c., datasets with noisy labels. Intuitively, direct adaptation using noisy labels can signifi-
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

cantly degrade the performance. To mitigate the negative impact of noisy labels, researchers have proposed various approaches for learning with noisy labels, including sample selection techniques
and noise-robust learning methods 37 27 However, the exploration of this problem in the context of CLIP adaptation remains limited. This paper aims to fill this research gap by demonstrating that CLIP can be leveraged as effective noisy label detectors, capitalizing on their strong alignment between visual and textual features. To ensure the generality, we follow the basic idea in mainstream studies of learning with noisy labels. Firstly, we identify potentially mislabeled samples, and subsequently, we adapt the model parameters using carefully selected clean samples.
[10
44 43
In the first phase, we propose to construct a noisy label detector by learning positive and negative textual prompts for each class. The positive prompt aims to uncover distinguishable features specific to the class, while the negative prompt functions as learnable "threshold" for identifying noisy labels. Our noisy label detector can be derived from the similarity between embeddings of a training image and positive/negative prompts associated with its assigned class. If the image exhibits a higher similarity score with the positive prompt than the negative prompt, it is considered a correctly labeled sample, i.e., a clean sample. This design draws inspiration from previous studies that have shown the robustness of prompt tuning to noisy labels, particularly in the presence of high noise ratios [4]. To optimize the positive prompt, we align its representation with image features from the corresponding class during training. For the negative prompts, we introduce a novel negative learning objective. is important to note that calculating similarities between images and textual prompts requires strong alignment between these two modalities in the downstream task. To achieve this, we utilize PEFT, such Visual Prompt Tuning (VPT) [18), for the adaptation of the visual encoder. This decision is based on a key finding that FFT can distort pre-trained feature representations when noisy labels are present, which is diseussed in Section 3.2
In the second phase, we can adapt the pre-trained model after acquiring carefully selected clean samples. It is worth noting that not only can we employ pre-trained CLIP for adaptation, but we can also utilize other pre-trained visual backbones such as ResNet 12] and MAE ΑΑ.. While the model adaptation phase may seem unnecessary since we can leverage the learned textual prompts for zero-shot classification without further fine- fine-tuning, we emphasize its necessity by the improved performance on curated downstream datasets, particularly in fine-gratee tne-ggrined classification tasks. In this step, we learn a linear classifier and fully fine-tune the visual encoders using only selected clean samples. The rationale behind this approach is twofold: 1) the linear classifier tends to generalize better than textual class prompts, and 2) full fine-tuning can effectively adapt model parameters to address significant domain shifts, surpassing the capabilities of PEFT. In our implementation, the training is conducted for only 10 epochs, resulting in a minimal increase in computation cost.
Our main contributions can be summarized as follows:
We propose DEFT, simple yet effective framework for learning with noisy labels. DEFT offers several compelling advantages over existing methods: instance-dependent (no in- formation required from entire training data), robust various types of noisy labels, and generalizable to many pre-trained models.
We conduct extensive experiments and show that DEFT achieves superior performance in both noisy label detection and image classification tasks on a wide range of synthetic and real-world datasets.
We provide in-depth empirical analysis, providing insights to understand the effectiveness of DEFT. We hope that this work will serve as a springboard for future works on noisy label detection with multi- modal features.
2 Related Work
Learning with Noisy Labels A diverse variety of approaches have been proposed for addressing learning with noisy labels and can be broadly divided into twofold: robust learning and clean sample selection g 32. Due to the susceptibility of traditional cross entropy loss to noisy labels, several
methods adopt noise-robust loss functions 8 51, 37 6 and regularization techniques 27 38 [6] to
enhance model robustness. For example, symmetric cross-entropy loss [37 introduces a noise-robust objective function for fast convergence and better performance. Farly-learning regularization [27] counteracts the influence of the noisy labels by introducing regularization term that integrates estimated target probabilities. However, empirical findings suggest that robust model learning faces

---

the problem of maintaining a balance between robustness and accuracy [51 351 Conversely, sample
selection approaches strive to select clean samples from noisy datasets using specific criteria. Based on the memorization effect identified in deep neural networks 22. Numerous methods adopt the
small-loss strategy that treats samples with small loss as clean ones [T0 15 25 19]. Co-teaching [TO] selects a fraction of training data in each mini-batch based on the noise ratio, whereas DivideMix [25] fits a Gaussian mixture model to dynamically divide the training samples. Although these methods demonstrate strong performance, they inevitably overlook the significance of clean hard samples with a large loss. Besides, all the aforementioned approaches concentrate exclusively on learning from scratch with label noise. Our proposed approach leverages the pre- pre-trained multi--mddl information from risual-languae models to construct a noisy label detector, resulting in improved sample selection performance.
Fine-Tuning Vision-I Languale Models Recent years have witnessed incredible progress in pre- trained visual-language models (29 17 49]. As representative method, CLIP [2 employs con- trastive pre-training to acquire informative representations from extensive image-text pairs. Despite the remarkable zero-shot performance of CLIP, its sensitivity to hand-crafted textual prompts is significant. Prompt-tuning aims to learn the optimal prompts from target data by replacing the textual templates with adaptable soft prompts, which demonstrates superior performance in the context of few-show learning [53] 52 20 and multi-label learning [3] [4]. However, further enhancements can be achieved by adapting the visual encoder to accommodate larger datasets in downstream tasks. A standard approach to obtain task-specific representations is fine-tuning all model parameters. While curated datasets, recent work [ empirically finds that FFT
FFT yields significant performance can result in feature distortion in the presence of massive noisy labels. In contrast, PEFT [48] has been verified improve generalization capabilities for mitigation of overfitting issues 30 which fix the pre-trained model and introduce a few number of learnable parameters for adaptation, such as VPT 18], LoRA (13), and AdaptFormer 33. Considering the distinct property of fine-tuning methods, this paper aims to delineate the most effective paradigm for model adaptation on noisy downstream datasets, which remains under-explored yet holds significant promise.
3 Preliminary and Initial Findings
3.1 Zero-Shot CLIP
This paper focuses on a K-class image classification task, where the objective is to categorize an image æ I& and assign it label y [K] 21, K}. Benefiting from large-scale datasets and high-capacity models, CLIP [29] has demonstrated remarkable performance in zero-shot classification by computing the similarity between visual and textual representations. Specifically, with hand-crafted textual prompts such as "a photo of a [CLS] where the class token is replaced by specific class name, the prediction probability is computed as:
exp(sim(1, Tw)//)
ply k
x)
(1)
k= cxp(sim cxp(sim(T,TT)//7 (I,Tw)/T)
where sim(..-) denotes the cosine similarity between the extracted image features I and text features T. and J is the softmax temperature.
While zero-shot classification with CLIP is promising, its performance can be further enhanced by adapting CLIP on downstream labeled datasets D { y:) >" } incorporate more task-specific representations. In this paper, we investigate the presence of noisy labels in the training dataset D, which is prevalent real-world applications. We use E [0, 1] to denote the noise ratio of a training set, meaning that N x training samples are incorrectly labeled, i.c., the assigned label y differs from the ground-truth label y² Since yy is not accessible, our goal is to fine-tune the CLIP model using training labels y by eliminating the negative influence of noisy labels. Considering the feature distortion induced by noisy labels 22, the key challenge lies in obtaining distinguishable yet robust representations when adapting CLIP on noisy datasets.
3.2 Fine-tuning CLIP on Downstream Datasets
To discern the most effective method for CLIP adaptation, we conduct empirical studies on various datasets as shown in Figure [ Specifically, we utilize three fine-tuning approaches to adapt the
3

---

pre-trained CLIP model and compare their performance on both noisy and clean datasets, including FFT which updates the entire model parameters, 2) VPT which fixes pre-trained model parameters and prepends small subset of extra learnable parameters to the visual encoder during fine-tuning, and 3) VLPT Languae Prompt Tuning) which integrates both visual and textual learnable prompts into the fixed pre-trained model for fine-tuning. For FFT and VPT, we learn an additional linear classifier, while VLPT directly utilizes the learned textual prompts for classification. We present our three primary findings in the following.
VPT benefits representation learning in the presence of massive noisy labels. Intuitively, utilizing FFT for CLIP adaptation yields im- proved performance by leveraging its substan- tial capacity to incorporate task-specific repre- sentations. However, the efficacy of FFT dimin- ishes notably when applied to datasets contain- ing noisy labels, as shown in Figure Ia This decline is attributed to the pronounced distortion of representations with an escalating noise ra- tio [. In contrast, VPT benefits representation learning when adapting CLIP on noisy data, par- ticularly under high levels of noise. As only small set of parameters is introduced, VPT effi- ciently retains the generalization ability from image-text pre-training while enhancing classification performance on downstream tasks, making it a robust and effective fine-tuning approach for handling noisy downstream datasets.
T-ragaliel
(a) Noisy Imay-Im
(b) Clean Datasets
Figure 1: Comparison of different fine-tuning methods under (a) various ratios of noisy labels and (b) clean datasets.
Textual classifier is robust to noisy labels. Classification with learnable textual prompts (e.g.. CoOp 53) leverages the multi-modal information in pre-trained vision-language models and en- hances the alignment of visual and textual representations downstream tasks. Recent literature has substantiated its robustness with a frozen visual encoder in the context of few-shot learning [41]. From Figure Ial we observe that VLPT with a textual classifier (green line) consistently surpasses VPT with traditional linear classifier for classification (orange line), especially under severe noise. The improvement performance across diverse noise ratios further affirms the robustness of learnable textual prompts in mitigating the impact of label noise for model adaptation.
FFT enhances visual recognition on clean datasets. Although previous findings show that FFT is more susceptible noisy labels compared VPT and VLPT, Figure Tb demonstrates its enhanced diseriminative ability when training data is clean. This superiority is particularly significant on fine-grained datasets such as Stanford-Cars and CUB-200-2011, with an average improvement of 2.81 1% against VPT. It is important to note that VLPT exhibits the worst performance on clean datasets. This is primarily due the implicit regularization of pre-trained textual information when tuning the context of textual prompts, as explained in the recent study [41]
The Denoising Fine-tuning Framework
Based on the findings Section 3.2 we now present a novel denoising fine-tuning framework, DEFT, which comprises two pivotal phases. In the first phase, DEFT learns dual textual prompts to separate clean and noisy samples while adapting the visual encoder utilizing PEFT methods. In the second phase, DEFT re-adapts the pre-trained model using FFT, leveraging the curated clean samples to further boost visual recognition performance. Figure illustrates our proposed framework.
4.1 Identifying Noisy Labels with Dual Prompts
Given the empirically demonstrated robustness of textual prompts to label noise, we intend to harness this property to identify corrupted samples in noisy downstream datasets. The initial strategy is to select a subset of samples with pronounced similarity between their visual and textual representations and designate them as clean. However, this method relies on either prior knowledge the noise ratio or the manual setting of thresholds, which restricts its practical applicability. Inspired by recent works 33 14 36), we utilize dual textual prompts instead of a single prompt to overcome these limitations.
Concretely, we design class class-spccffic pair of positive and negative prompts for the textual encoder,

---

Data Now
Loarnable prompt
Leamaale network
Frazen natwork
dog"
cal
dog
Cal
Postive Prompi
Text Encoder
dog"
Classifier
Negative Prompt
dog
Pre-trained Models
Visual Encoder
Full Fine-tune
Gat
Modules
Noisy Dalasel
Clean Subset
Phase Noisy Label Detection
Phase 2: Model Adaptation
Illustration of the proposed DEFT framework. Left: We identify noisy labels with learnable dual textual prompts and improve image-text alignment by optimizing PEFT modules. Right: Adapt pre-trained models using FFT on selected clean samples.
Figure 2:
denoted as prompt, and prompt respectively:
prompt,
[CLS]
(2)
prompt,
VV [CLS]]
Here, [.]m m. is the word vector of context (with dimension 512 CLIP), M the length context, and [CLS]& is the k-th provided category name. The positive prompt aims to uncover distinguishable features by maximizing the similarity between the image features and their corresponding text similarity threshold to
features, while the negative prompt serves learnable select clean data. Specifically, the learnable threshold Φ; for the i-th training sample with label k is formulated as:
Φ sim I,, T
which represents the cosine similarity between the extracted image features and negative text features. Based on this threshold, the clean subset of dataset 'D can be constructed as follows:
Delean {(x,, 14)
si
Φs and y: = k}
(5)
The proposed selection strategy surpasses the conventional loss-based approaches in two aspects: 1) It is more practical by utilizing data-driven thresholds thus eliminating the requirement for prior knowledge like noise ratio, and the integration of text modality enhances its robustness to label noise, making it capable of identifying challenging hard noise. Nevertheless, the primary dilemma lies in the optimization of positive and negative prompts using noisy downstream datasets.
4.2 Optimization for Noisy Label Detector
It can be an intricate task to optimize the threshold d; in end-to-ent manner since is indirectly involved in the forward propagation. However, we can still optimize the associated parameters by formulating the clean probability of the i-th image with y/ is as:
exp(sim(1,, T+7)
clean
(6)
exp(simmi..ssmp......e exp(sim(L,. +en TT)/
Obviously, selecting samples with the clean probability pr > 0.5 is equivalent to the specified selection approach can be transformed
criteria in Eq. (5). In this way, the original threshold- into K binary classification tasks to determine whether a sample is clean or not, Concretely, given sample (æ, yı) from noisy datasets D, we can employ the dual prompts of class ye binary classifier to identify label noise. the classifier outputs positive prediction of I; it is categorized clean sample.
To optimize noisy label detector, we need construct positive and negative training samples for binary classification, i.e., images with correctly and wrongly assigned labels. Inspired by the prior work on negative learning (21), we designate each image with randomly picked complementary label to form negative samples. For positive samples, we initiate with the original supervision in
5

---

D and refine it with pseudo labels ÿ generated by Eq. ①T) The training loss is then formulated as follows:
N
Cup
Σ
(ps () + Ent(1 p.
(7)
N
=k= is the clean probability and (nu(;,-) -) is the negative log- -il-iiilloood loss where (P 3) log Piy.
where p.lean {pik 1K
Additionally, to enhance task-specific representations of the noisy label detector, we harness PEFT for the adaptation of the visual encoder, considering its robustness under massive noisy labels compared with FFT, as per our previous finding (see Figure Tap. This is achieved by maximizing the alignment between image embeddings and their corresponding positive textual features:
exp(sim( 1,, TT)TT)
Luim
log
8
k= exp(sim( TT)//)
Note that Lesm is exclusively computed on Delean after the warm-up stage to further eliminate the impact of label noise on representations. With the objective = Lup CSim- DEFT effectively sieves out noisy samples with learned thresholds while uncovering distinguishable visual representations on downstream tasks.
4.3 Model Adaptation using Clean Data
Although the learned positive textual prompt can be readily employed for classification, its per- formance may be suboptimal on curated clean datasets, as demonstrated in our previous finding (see Figure To mitigate this problem, introduce the model adaptation phase which learns a linear classifier using selected clean samples, i... Deleam Morcover, our findings also indicate that FFT outperforms PEFT on clean datasets,
remove the PEFT modules and fully fine-tune the pre-trained model parameters for model adaptation. We minimize the classic cross-entropy loss exp(zy) for (ar v) log ET= exp(=x.' where z { [== represents the logits predicted by the linear classifier. noteworthy that with the clean subset training data constructed by Eq. ()) the second phase can be applied universally to a wide range of pre-trained models, regardless of their backbones. The versatility of our approach demonstrated in the experiments.
Experiment
We now proceed to the experiments, demonstrating the advantages of our approach in noisy label detection and image classification both synthetic and real-world datasets.
5.1 Experimental Settings
Synthetic Datasets We first evaluate the performance of DEFT on four image classification benchmarks by synthesizing noisy labels with varying noise types and ratios. Specifically, conduct experimental analyses on widely-used CIFAR- 100 23) and Tiny-ImageNet [42), as well as two fine-grained datasets Stanford- Caaford-CCars (2 and CUB-200-24 2011 34. Given the noise transition matrix T E [0, 1] K corrupted label y = j. introduce corruption probability is assumed to be only dependent on the true label, e.g., the symmetric noise generated by replacing y" with random label for given proportion of training samples [10 25). and 2) Tij p(y = j y" =,, which is a more realistic assumption considering the impact of instance x in the label corruption process, c.g., the instance- deeenddnd noise proposed in [45). We conduct extensive experiments on each dataset with noise ratio r E {0.2, 0.4, 0.6} for symmetric noise and E (0.2, 0.3, 0.4) for instance-dependen noise.
x with Tij denoting the probability of the ground-truth y = i being flipped into
types of label noise: 1) Ti = p(y = j 1y" = =),
where the
Real- World Datasets We further examine the performance of DEFT on three real-world noisy label datasets: 1) CIFAR-10ON [39 (r Rs 0.4) is variant of CIFAR-100 with real-world human annotations from Amazon's 's Mechanical Turk, 2) Clothing 47 (r 0.4) a large-scale dataset consisting of I million clothing images of 14 classes collected from online shopping websites, and

---

26 (r As 0.2) contains 2.4 million images crawled from Flickr and Google using the we experiment on Sym. 0.2
Sym. 0.4
Syn.. 0.6
Ins. 0.2
Ins. 0.3
Ins. 0.4
Method
Prec.
Roc.
Prec.
Rec.
Prec.
Rec.
Prec.
Rec.
Prec.
Rec.
Prec.
Roc.
CIFAR-100
Label-match
99.83
63.62
99.61
63.85
99.31
63.52
99,93
63.65
99.85
63.72
99.81
63.69
Small-loss
97.24
96.79
95 35.68
94.49
92.93
90.68
95.20
95.46
94.00
92.53
90.33
89.85
DEFT (ours)
99.51
97.77
98.75
97.91
97.04
97.27
98.47
97,88
96.32
97.63
94.08
95.28
1 2.27
0.98
1 3.07
3:42
1.4.11
1 6.50
1 3.27
2.42
1.2.22
5.10
1 3.75
5,43
Image-maaeNNet
Label-mater
99.92
60.81
99.83
60.66
99,91
60.58
99.84
60.53
76
60.47
Small-loss
97.25
96.93
95.33
94.48
92.63
90.89
94,74
95.17
93.66
92.35
90.41
89.71
DEFT (ours)
99.50
96.00
98.78
95.97
97.21
95.44
99.21
96,21
97.80
95.80
95.45
95.77
1 2.25
0.93
1 3.45
1.49
4.58
1 4.55
1 4:47
1.01
1 4.1414
1:3.45
1 5.04
6.06
Stanford-Cars
Label-match
99.97
60.34
99.88 39.86
60.27
99.70
60.71
99.85
60.34
99.82
60.32
99.80
60.25
Small-loss
96.92
96.56
93.71
93.21
89.46
87.79
96,94
97.78
96,72
95,96
95.25
94.48
DEFT (ours)
98.72
99.56
98.98
98.56
98,58
95.62
99.02
99.09
98.96
98.15
98.75
97.71
t 1.80
f 3.00
1 5.27
5.35
9.12
1 7.83
1 2.08
1 1.31
1: 2.24
1.2.1
1 3.50
3.23
CUB-220-2011
Label-match
99,92
53.26
99.74
53.13
99.46
53.02
99,96
53.39
99.96
53.32
99.74
53.69
Small-loss
96.74
96.32
93 93.69
92.84
84,10
82.01
96.91
97.33
96.49
95.59
93.98
93.96
DEFT (ours)
99.04
97.01
96.76
95.60
93.88
96.43
99.15
97.45
97,93
96.85
96.03
97.11
1 2.30
0.00
3.07
2.76
9.788
1 14.442
1 2.24
0.12
1 1.44
1 1.26
1 2.05
3.15
Table 1: On each dataset, we compare the Precision (%) and Recall (%) of DEFT with CLIP label- match and small-loss to evaluate the clean sample selection performance. A is the difference between the performance of DEFT and small-loss.
3) Web 1,000 concepts in ImageNet ILSVRC12. Following previous works [25 19 24 the first 50 classes of the Google image subset.
We use precision and recall metrics to evaluate the selection of clean samples. higher precision indicates a greater proportion of actual clean samples in Delean while higher recall means that more clean samples are identified from the noisy dataset. For image classification, We report the "best" test accuracy across all training iterations and the "last" test accuracy at the end of training.
Evaluation Metrics
Implementation Details We adopt the pre-trained CLIP 29 with the Transformer as the text encoder and the ViT-B/16 as the image encoder. We use the SGD optimizer with a momentum of 0.9, weight decay of 5x 10-4, 14 and batch size of 64. We run 10 epochs for both the noisy label detection phase and the model adaptation phase with learning rates 3x 10 and 5x 10-4, respectively. In the noisy label detection phase, we employ VPT [18 and CoOp 333 to adapt visual encoder and textual encoder respectively, and perform model warm-up for epoch on all datasets. When identifying noisy labels real-world datasets, augment the condition in Eq. (5) with a constraint that the training label should be consistent with its predicted label. This is due to the existence of more complex noise patterns in real-world tasks. All experiments are conducted on single NVIDIA GeForce RTX 3090.
5.2 Performance for Noisy Label Detection
We conduct a thorough evaluation of DEFT to justify its effectiveness in detecting noisy labels. For this purpose, we simulate various types and ratios of label noise on four benchmark datasets. We compare our approach with two other sample selection strategies: 1) Label-match strategy, where a training sample is deemed clean if its given label matches that assigned by the CLIP zero-shot classifier, and Small-loss strategy, which selects proportion (noise ratio) of confident samples in each mini-batch, as commonly used in previous studies [10 25 19 The precision and recall of sample selection are reported in Table [ The results show that our proposed noisy label detector outperforms the compared strategies, with significant improvements observed in all dataset settings. The trivial label-match strategy tends to be conservative, leading to low selection recall of clean samples. In contrast, the small-loss strategy and DEFT achieve better trade-off between precision and recall, making better use of training samples. Moreover, leveraging the informative multi- model information in the pre-trained vision- vasion-language model, DEFT surpasses the small-loss strategy
7

---

Method
Sym. 0.2
Sym. 0.4
Sym. 0.6
Ins. 0.2
Ins. 0.3
Ins. 0.4
CIFAR-100 100
CR
86.71 / 86,70
84,06 / 82.60
81.05 / 77.45
87 .330 / 87.18
84,60 / 83.64
78.41 / 75.66
F..R
86.53 / 86.53
83.66/ 83.66
78.34 78.34
86.61 / 86.61
85.89/ 85.89
85.78 85,78
SCE
86.82/ / 86.82
83.84 83.84
78.90 77.71
86.61 / 86.61
83,99 83.20
80.06 73.45
GMM
88.49 88.49
87.21 / 87.21
85.22 / 85.20
88, 44 88.44
87,95 87.95
82. 14 82.11
DEFT
Ours
89.38 89.35
88.17 / 88.11
85.81 / 85.72
89.38 89.35
88.68 88.68
85.75 85.74
Inny-ImaeeNet
CE
81.77 / 81.08
76.53 76.52
73.17 / 77.46
80.75 / 80.71
78.83 / 78.57
74,80 74.08
ELR
79.40 / 79.40
77.13 77.13
73.74 / 73.74
79.98 79.98
77.13 77.13
73.74 73.74
FFT
SCE
79.23 / 79.23
76.24 76.18
71.76/ 70.62
78.96 / 78.90
77.80/ 77.54
74.47 / 73.25
GMM
81.91 / 81.88
80.37 / 80.37
43.47 /3.47
81,84 / 81.79
81,26 / 81.26
79.01 / 79.01
DEFT
Ours
82.91 82.91
82.48 82.37
80.60 80.59
83.37 83.33
82.69 82.65
80.52 80.49
Stanford-Can Cars
CE
89.75 / 89.74
8 84.89
71.70/ 71.55
89.13/ 89.06
85,94/ 85.92
80.59
80.59
ELR
86.61 / 86.61
76.98 / 76.98
61.58 61.58
84.40 / 84.40
83.11/ 83.11
75.97
75.84
FFT
SCE
91.11 /91.11
87.73 87.45
79.09 / 79.09
90.34 90.34
87.35/ 86.23
83,50 80.69
GMM
90.10/ 90.08
83.14/ 83.10
56,90 56.90
88.15 / 88.10
85.39 / 85.33
78.76 78.72
DEFT
Ours
92.12
90.75 / 90.75
85.72 / 85.45
92.19 / 92.15
90.77 / 99.77
89.74 / 89.68
CUB-200-2000
CR
80.76 / 80.76
73.09/ / 72.87
55.42 / 55.21
80.36 80.25
75.80/ 75.53
69.62 69.62
ELR
77.70/
68.26 68.26
50.17 / 49.88
78.32 78.32
73.16/73.08
63.57 63.34
FFT
SCE
82.81 / 82.74
78.12 77.87
63.31 / 63.31
81.91 81.91
78.31 78.00
71.25 70.95
GMM
75.79 75.77
64.39 / 64.38
42.84 / 42.84
75.73 75.65
69.95 / 69.95
56.13 55.80
DEFT
Ours
83.05 83.03
79.24/ 79.13
73.08 / 73.08
82.53 / 82.50
81.39 / 81.39
79.34/ / 79.24
Table 2: Test accuracy (%) on synthetic datasets with symmetric and instance-dependent label noise.
Dalaset
ELR
SCE
GMM
RoLT
UNICON
LongReMix
ProMix
DEFT (Ours)
CIFAR-10ON
72.41
72.83
72.52
76.06
75.91
77.68
73.94
75.97
79.04
Clothing IM
69.75
72.14
70.49
70.03
70.46
70.38
70.62
70.71
72.44
WebVision
84.64
79.32
82.88
84.88
84.12
84.56
84.96
84.44
85.12
Table 3: Test accuracy (%) datasets with real-world label noise.
in detecting noisy labels, particularly under severe noise settings and fine-grained classification tasks. For instance, in Tiny-ImageNe with 60% symmetric noise, DEFT demonstrates significant enhancements of 4.58% and 4.55% in precision and recall, respectively, as well as an improvement of 9.78% and 14.42% in CUB-200-2011 Additionally, DEFT reduces the need for prior knowledge of noise ratios, making a practical and effective approach to detect label noise in real-world tasks.
5.3 Performance for Image Classification
We evaluate the performance of DEFT in image classification tasks against four baselines, including CE (Cross-Entropy loss), SCE [37]. ELR [27]. and GMM (25). To further verify the effectiveness of our method on real-world datasets, we also make comparisons with some recent sample selection- based works including [oLT UNICON 19), Me) and ProMix (46). We utilize FFT for adapting the pre-trained CLIP model for all approaches. More details can be found in the Appendix.
Results on Synthetic Datasets Table presents the results on four synthetic noisy datasets. Com- pared with CE, the adoption of noise-robust loss functions such ELR and SCE improves the classification performance on CIFAR- 100 and Iny-ImageNee under certain noisy label settings, e.g.. ELR achieves the best performance on CIFAR-100 with 40% eestace--dpendee these methods are not always effective and may even worsen the performance under varying types and ratios of noise, compared to the sample selection paradigm like GMM and DEFT. While GMM out- performs ELR and SCE in most cases, its performance degrades dreadfully on fine-grained datasets. Nevertheless, DEFT retains the most robust performance and generally advances all compared meth- ods, especially on Stanford-Cars and CUB-200-2011. 1. For example, DEFT boosts the test accuracy
noise. However,

---

92
84
50
0.2
06
(a) CIFAR-100
(b) Tiny-ImageNet
(c) Stanford-Cars
(d) CUB-200-2011
Figure 3: Ablation studies. We report the test accuracy across varying noise ratios for the following variants: 1) w/o adap.: DEFT without the model adaptation phase, 2) PEFT: use PEFT for model adaptation phase, and 3) FFT: use FFT for model adaptation phase.
Architecture
CE
GCE
ELR
TURN
DEFT (Ours)
ResNet-50 12
66.02
66.19
66.19
66.31
70.82
MAE- ViT-B
61.31
60.80
61.51
61.96
65.23
ViT-B/16 [
68.98
69.74
68.73
70.28
69.84
ConvNeXt-T 28
68.80
68.92
68.52
69.53
71.68
Table 4: Test accuracy (%) using various pre-trained models Clothing Partial results are sourced from [[]. The best results across all methods are highlighted in bold, with the second-best results indicated by underscores.
by average of 4.34% on Stanford-Cars compared to the best results of comparison methods. The results demonstrate the effectiveness of our proposed approach tackling both symmetric and instance-dependent label noise.
Results on Real-world Datasets The results reported in Table demonstrate the superiority of method real-wer wolld datasets, DEFT surpasses other methods by significant margin all datasets. Notably, while most compared methods show little improvement on Clothing IM due to its ane-grained nature, both ELR and DEFT demonstrate substantial improvement. However, ELR exhibits performance decrease on WebVision Moreover, directly fine-tuning pre-trained CLIP models with cross-entropy (CE) achieves competitive results performance further, showcasing its ability to combat label noise in practical situations.
yet DEFT elevates this
5.4 Further Analyses
Necessity of Model Adaptation In this experiment, emphasize the necessity of the model adaptation phase in DFFT. Figure 3 demonstrates the test accuracy DEFT without the model adaptation phase (w/o adap.), adaptation utilizing PEFT and FFT. Results show that fully fine-tuning the model on selected clean samples yields the best classification performance, especially on fine- grained datasets, such Stanford-Cars and CUB-200-2011 The results unveil two primary insights: 1) the noisy label detector in DEFT exhibits strong capabilities in detecting label noise, yielding a nearly clean subset from noisy downstream datasets, and 2) employing FFT for model adaptation is more effective in mitigating significant domain shifts between downstream data and pre-training data, particularly evident fine-graine mne-grained datasets.
DEFT for Various Pre-trained Models is noteworthy that DEFT can seamlessly integrate with various pre-trained visual backbones during the model adaptation phase. To demonstrate the versatility of our proposed denoising fine-tuning framework, we conduct experiments on Clothing 1M and apply FFT to a range of pre-trained models, including ViT-B/16 (5). ResNet-50 [2]. ConvNeXi-T (28). and MAE VIT-B [TI. The results are presented in Table where GCE [5I] employs noise-robust loss function akin to SCE. As recent work, TURN [ an improved version of GMM, which applies linear probing to initialize the classifier and then uses FFT for adaptation. Compared with previous methods, DEFT exhibits the best performance on average. In particular, DEFT outperforms TURN by 4.51% and 3.27% when adopting ResNet-50 and MAE-ViT-R as the target pre-trained models, respectively.

---

6 Conclusion
In this work, we delve into a new landscape for learning with noisy labels, departing from the classic single-modal toward a multi-modal regime. By learning dual textual prompts, we construct new noisy label detection approach, which offers several compelling advantages: robust to various types of label noise, generalizable to many pre- pre-trained models, and does not require the dynamics of training samples. We investigate the effectiveness of DEFT on a wide range of synthetic and real-world datasets, showing its superior performance in both noisy label detection and image classification tasks. Lastly, we demonstrate the advantage of parameter-efficien fine-tuning over full fine-tuning over noisy label detection. We hope work will inspire future research toward multi-modal noisy label detection.
Acknowledgments and Disclosure of Funding
This work was supported by the National Science Foundation of China (62206049, 62225602), the Key Program Jiangsu Science Foundation (BK20243012), and the Big Data Computing Center of Southeast University. We would like to thank anonymous reviewers for their constructive suggestions.
References
[1] Sumycong Ahn, Sihycon Kim, Jongwoo Ko, and Se- Young Yun. Fine tuning pre trained models for robustness under noisy labels. arXiv preprint arXiv:2310. 17668, 2023.
[2] Devansh Arpit, Stanisaw Jastrzbski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kan- wal. Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, al. A closer look memorization in deep networks. In International conference on machine learning, pages 233-242, -42. PMLR, 2017.
[3] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adapt- former: Adapting vision transformers for scalable visual recognition. Advances Neural Information Processing Systems, 35: 15664-1666667. 2022.
[4] F. R. Cordeiro, Vasilcios Belagiannis, Ian Reid, and Gustavo Cameirr. Longremix: Robust learning with high confidence samples in noisy label environment. Pattern Recognition, 133: 2023.
[5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint 2iiv20000111229 2020.
[6] Lei Feng, Senlin Shu, Zhuoyi Lin, Fengmao Lx, Li Li, and Bo An. Can eross entropy loss be robust label noise? In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 2206-2212, 2021.
[7] Kai Gan and Tong Wei. Erasing the bias: Fine-tuning foundation models for semi-supervised learning. In Proceedings of the 41st International Conference on Machine Learning, pages 14453 14470,, 2024.
[8] Aritra Ghosh, Himanshu Kumar, and P Shanti Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI conference artificial intelligence, volume 31, 2017.
[9] Han, Quanming Yao, Tongliang Liu, Gang Niu, Ivor Tsang, James Kwok, and Masashi Sugiyama. A survey of label-noise representation leaming: Past, present and future. arXiv preprint MxXi:2011.04440 2020.
[10] Bo Han, Quamming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018
[11] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 16000-16009, 2022.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
10

---

[13] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09655 2021.
[14] Ping Hu, Ximeng Sun, Stan Sclaroff, and Kate Saenko. Dualcoopt+: Fast and effective adaptation to multi-label recognition with limited annotations. arXiv preprint arXiv:2308.0/880 2023.
[15] Jinchi Huang, Lie Qu, Rongfei Jia, and Binqiang Zhao. O2u-net: A simple noisy label detection approach for deep neural networks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1326--3344- 2019.
[16] Ahmet Iscen, Jack Valmadre, Anurag Arnab, and Cordelia Schmid. Learning with neighbor consistency for noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4672-4681, 2022.
[17] Chao Jia, Yinfci Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoe Lc, Yun-Hsuan Sung, Zhen Li, and Tom Ducrig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904-4916. PMLR, 2021.
[18] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709-727. 2022.
[19] Nazmul Karim, Mamshad Nayeem Rizve, Nazanin Rahnavard, Ajmal Mian, and Mubarak Shah. Unicon: Combating label noise through uniform selection and contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9676-9686, 2022.
[20] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi modiil prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19122, 2023.
[21] Youngdong Kim, Junho Yim, Juseung Yun, and Junmo Kim. Nlal: Negative learning for noisy labels. In P Proceedings of the IEEE/CVF international conference on computer vision, pages 101-110, 2019.
[22] Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei. Collecting large-scale dataset of fine-grained cars. Second Workshop on Fine-Grained Visual Categorization, 2013.
[23] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of feature from tiny images. Master's Thesis, 2009.
[24] Hao- Taan Li, Tong Wei, Hao Yang, Kun Hu, Chong Peng, Li-Bo Sun, Xun-Liang Cai, and Min- Stochastic feature averaging for lcarning with long-tailed noisy labels. In Proceedings of the International Joint Conference Artificial Intelligence, pages 3902 2910,,90,20 2023.
Min-Ling Ling Zhang.
[25] Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: learning. International Conference on Learning Representations, 2020.
Learning with noisy labels as semi-supervised
[26] Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Lue Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint 11Xxv:100.0088000 2017.
[27] Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernanez-GGranda. Early-learning regu- larization prevents memorization of noisy labels. Advances in Neural Information Processing Systems, 2020.
[28] Zhuang Liu, Hanzi Mao, Chao- Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xic. convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogmition, pages 1996-11986, 2022.
[29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, 2021.
[30] Jiang-Xin Shi, Tong Wei, Zhi Zhou, Jic-Jing Shao, Xin-Yan Han, and Yu-Feng Li. Long-tail learning with foundation model: Hcavy fine-tuning hurts. In Proceedings of the 41st International Conference on Machine Learning, pages 45014-45039, 2024.
[31] Yang Shu, Xingzhuo Guo, Jialong Wu, Ximei Wang, Jianmin Wang, and Mingsheng Long. Clipood: Generalizing clip to out-of-distriuution. In International Conference Machine Learning, 2023.
11

---

[32] Hwanjun Song, Minseok Kim, Dongmin Park. Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
[33] Ximeng Sun, Ping Hu, and Kate Saenko. Dualcoop: Fast adaptation to multi-label recognition with limited annotations. Advances in Neural Information Processing Systems, 35:30569-30555 2022.
[34] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. Technical report, 2011.
[35] Deng-Bac Bao Wang, Yong Wen, Lujia Pan, and Min-Ling Zhang. Learning from noisy labels with comple- mentary loss functions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10111-10119, 2021.
[36] Hualiang Wang. Yi Li, Huifeng Yao, and Xiaomeng Li. Clipn for zero-shot ood detection: Teaching clip say no. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1812-1812, 2023.
[37] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF international conference on computer vision, pages 322-330, 2019.
[38] Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agrecment: A joint training method with co-regularization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13726-13735, 2020.
[39] Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In International Conference on Learning Representations, 2022.
[40] Tong Wei, Jiang-Xin Shi, Wei-Wei Tu, and Yu-Feng Li. Robust long-tailed learning under label noise. arXiv preprint arXiv:2108.I 11569, 2021.
[41] Cheng- En Wu, Yu Tian, Haichao Yu. Heng Wang. Pedro Morgado, Yu Hen Hu, and Linjie Yang. Why is prompt tuning for vision-language models robust to noisy labels? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15488-15497, 2023.
[42] Jiayu Wu, Qixiang Zhang, and Guoxi Xu. Tiny imagenet challenge. Technical report, 2017.
[43] Zhi-Fan Wu, Tong Wei, Jianwen Jiang, Chaojie Mao, Mingqian Tang, and Yu-Feng Li. Nge: A unified framework for learning with open- world noisy data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12-71,22 2021.
[44] Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, and Masashi Sugiyama. Sample selection with uncertainty of losses for learning with noisy labels. In International Conference on Learning Representations, 2021.
[45] Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang. Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent lahel noise, Advances in Neural Information Processing Systems, 33:7597-7610, 2020.
[46] Ruixuan Xiao, Yiwen Dong, Haobo Wang, Lei Feng, Runze Wu, Gang Chen, and Junbo Zhao. Promix: Combating label noise via maximizing clean sample utility. In Proceedings of the Shirry-Second Interna- tional Joint Conference on Artificial Intelligence, pages 4442-4450, 2023.
[47] Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR, 2015.
[48] Bruce XB Yu, Jianlong Chang. Haixin Wang. Lingbo Liu, Shijie Wang. Zhiyu Wang, Junfan Lin, Lingxi Xie. Haojie Li, Zhouchen Lin, et al. Visual tuning. arXiv preprint arXiv:2305. 060661 2023.
[49] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-tex -ext foundation models. arXiv preprint (xX.x:22205009977 2022.
[50] Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-efficient fine-tuning for baassooomer-bbsed masked models In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), pages 1-9, 2022.
[51] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018
12

---

[52] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 16816-16825, 2022.
[53] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning prompt for vision-language models. International Journal of Computer Vision, 130(9):2337-2344 2022.
13

---

A Appendix
A.1 Additional Results on Noisy Label Detection
Impact of CLIP Backbones Table I has demonstrated the superiority of DEFT in detecting noisy labels based on CLIP ViT-B/16. To evaluate the impact of different visual backbones on noise detection, we re-implement all methods using CLIP ViT-B/32 as the backbone and report the results in T Notably, while the small-loss strategy demonstrates comparable performance on noisy CIFAR- 100 and Tiny-ImageNer its efficacy significantly diminishes on fine-grained datasets such as Stanford- Cars and CUB-200-2011, which exhibits its sensitivity to visual backbones. In contrast, DEFT maintains consistently superior performance on both ViT-B/16 and ViT-B/32, underscoring its resilience across different pre-trained visual backbones.
Results under Severe Noise We conduct experiments on synthetic noisy datasets with significant label noise to further validate the robustness of the proposed noisy label detector. It is noteworthy that two adjustments are implemented in order to train a more effective model: 1) the learning rate is adjusted from 3x10 to 1x10-2, and 2) For DEFT, a small weight (specifically, 0.25) is introduced to the positive loss component in Eq. to mitigate the impact of noisy pseudo labels in high levels of noise. Table Table the precision, recall, and their harmonic mean (FI-score) for noise detection, where DEFT exhibits robust capability in identifying severe noise. For example, remarkable improvement in the Fi-score is achieved on CUB-200-2011. These results strongly reaffirm the superior performance of DEFT across various noise ratios.
GMM for Noisy Label Detection DivideMix 25 introduces the use of Gaussian Mixture Model (GMM) for the dynamic separation of clean and noisy samples with adjustable threshold. Table 7 summarizes the performance of GMM and DEFT in detecting label noise across diverse datasets and noise settings. To fine-tune the pre-trained model for GMM, we independently apply PEFT and FFT, The results reveal that DEFT consistently exhibits superior noise detection capability, particularly evident in fine-gran one-grained datasets with high noise ratios, such as Stanford-Cars with 60% symmetric noise and CUB-200-200 with PEFT yields better noise detection when compared utilizing FFT. This difference can be attributed to the potentially unstable training process of FFT, where the loss value may undergo significant fluctuations due to an abundance of tunable parameters.
with 40% instance-dependent noise. Intriguingly, implementing GMM
The CIFAR- dataset (39]. which serves as real- world benchmark by providing CIFAR-100 with human-annotalle noisy labels, allows to evaluate
Noisy Label Detection on CIFAR-10ON
Sym. 0.2
Sym.
Sym.
Sym.
Sym. 0.3
Sym.
Method
Prec.
Rec.
Prec.
Rec.
Prec.
Rec.
Prec.
Rec.
Prec.
Rec.
Prec.
Rec.
CIFAR-100
Label-match
99.82
57.91
99.51
58.13
99.10
58.03
99.87
57.90
99.82
$7.86
99.72
$7.84
Small loss
97.32
96.86
95.43
94,23
92.78
90.46
94.78
95:00
93.33
91.83
89.61
89.12
DEFT (ours)
99.41
97.88
98.51
97.80
96.10
97.74
98.89
97.89
97.00
97.82
93.27
97.76
1 1.02
3.00
3.55
1 3.32
1 7.28
1 4.11
1 2.89
3.67
1.999
3.66
8.61
Tiny- IInageNet
Label-match
99.88
58.02
99.80
58.
99.47
58.26
99.89
58.00
99.82
58.04
99.74
$7.93
Small loss
97.20
96.88
95.44
94.56
92.73
90.89
94.72
95.14
93.36
89.16
DEFT (ours)
99.45
95.31
98.52
95.79
96.08
95.88
99.04
95.71
98.08
95.88
96.16
95.76
1 2.25
J 1.57
3.08
1.23
1 3.33
1..99
f 4.33
1 0.57
4.72
3.83
6.33
6.60
Stanford-Crs
Label-match
99.88
50.14 14
99.51
50.14 14
99.65
51.28
99.81
49.84
99.72
49.79
99.55
49.90
Small-Kost
96.98
96.60
86.51
85.43
41.23
39.92
95.02
95.75
78.25
77.31
63.71
62.67
DEFT (ours)
99.17
99.23
97.40
99.31
94.66
97.68
98.68
98.95
97.75
98.96
95.62
99.33
A
2.19
1 2.63
1 10.89
1 13.88
1 53.43
1 /7.776
1. 3.66
1 3.20
19.50
1:21.65
31.91
36.66
CUB-200-2011
Label-match
49.24
99.67
49.72
99.75
49.38
99.79
50.07
99.71 1
50.13
99.61
50.36
Small-loss
96.12
95.68
87.41
86.35
58.65
56.99
94.42
94.85
86. 66
85.74
77.08
76.77
DEFT (ours)
98.62
97.56
95.15
98.18
94.32
95.85
98.18
97.00
93.57
93.05
95.35
97.41
2.50
1 1.88
7.7.4
t 11.83
1 35.67
1 13.886
1 3.76
1 2.15
6.91
7.31
18.27
20.64
Table 5: Precision and recall of noisy label detection with CLIP VIT-B/32 as the visual backbone.
14

---

Method
Sym. 0.8
0.5
Method
Symmetric
Instance- dependeed
Prec.
Rec.
Fl.
Prec.
Rec.
F1.
0.2
0.4
0.6
0.2
0.3
0.4
CIFAR-
--FRR-100
CIFAR- 100
Label-match
97.94
63,68
77.
99.69
63.86
GMM w/ FFT
97.44
97.56
97.19
97.08
96.99
Small-loss
90.31
83 69
86.87
85.37
86.45
85.91
GMM w/ PEFT
97.82
97.74
96.94
97.08
95.
91.37
DEFT (ours)
95.25
91.60
93.39
92.83
93.08
92.95
(ours)
98.63
98.33
97.15
98.17
96.97
94.68
Tiny ImageNet
Tiny- ImageNet
Label-match
98.74
61.04
75.44
99.68
60.61
75.38
GMM w/ FFT
94,80
95.03
65.
94,
94.42
91.66
Small-Hoss
89.12
84.52
86.76
84.92
85.04
84.98
GMM w/ PEFT
96.56
96.64
96.24
96.01
95.02
92.04
DEFT (ours)
95.97
89.68
92.72
96.19
90.86
93.45
DEFT (ours)
97.72
97.35
96.32
97.69
96.79
95.61
Stanford-Cars
Stanford-Cars
Label-match
98.62
60.52
75.01
99.63
59.49
74.50
GMM w/ FFT
97.81
95
77.00
96,
95.
05
Basall-lsss
81.97
77.39
79.61
88.18
88.77
88.97
GMM w/ PEFT
99.10
98.31
93.22
98.34
96.58
92.46
DEFT (ours)
92.78
95.31
94.03
95.80
99.02
97.38
DEFT (ours)
99.14
98.77
97.08
99.05
98.55
98.23
CUB-2000-001
CUBB-200-2011
Label-match
98.20
53.22
69.03
99.44
53,50
69.57
GMM w/ FFT
91.92
86.56
67.52
91.86
88.58
76.34
Small-Hos
71.23
66.41
68.74
83.81
84.65
84.23
GMM w/ PEFT
97.06
96.57
91.39
97.27
94.45
89.20
DEFT (ours)
90.12
89,94
90.03
95.02
96.47
95.74
DEFT (ours) 98.01
96.18
95.14
98.29
97.39
96.57
Table 6: Precision, recall, and Fi-score of noisy label detection under 80% symmetric noise and 50% instance- dependet noise.
Table 7: Fi-score (%) of noisy label detection. For GMM, we utilize both PEFT and FFT to adapt the pre-trained model.
Label-match
Small-loss
GMM
RoLT
UNICON LongReMix
ProMix
DEFT (Ours)
Precision 91.48
89.37
80.03
75.23
87.05
69.37
75.92
88.
Recall
73.88
89.14
97.05
95.82
85.44
98.67
94,90
92.84
Fl-score
81.74
89.25
87.72
84.29
86.24
81.47
84.36
90.58
Table 8: Noise detection results on CIFAR-10ON,
visual prompt length
textual prompt length
training epoch
Hyperpara.
15
20
30
8
32
5
15
20
F1-score
98.08
98.17
98.33
98.30
93.29
97.14
97.15
97.31
96.65
97.15
97.34
97.38
Table 9: Sensitivity analysis of promt length and training epoch.
the efficacy of DFFT for noise detection in practical scenarios. The results presented in Table reveal the distinctive performance characteristics of various sample selection strategies. For example, the label-match strategy demonstrates a propensity for being greedy and yields the highest precision, whereas the GMM and recent sample selection-basse works tends to be conservative and thus achieve a higher recall. Nevertheless, DEFT stands out with the highest F1-score across all strategies, striking judicious balance between precision and recall in the identification of real-world noisy labels.
Sensitivity of Hyperparameters We explore the sensitivity of our noise label detector to various hyperparameter settings, specifically the length of the visual/textual prompt and the number of training epochs. Table[]present the Fl-score on CIFAR- 100 with 40% symmetric noise, The results indicate that the length of the visual prompt has marginal impact on the noisy label detection. For the textual prompt length, once the context length exceeds 8, the performance is enhanced slightly as the length increases. Additionally, variations in the number of training epochs have minimal impact on performance. In order to strike a balance between effectiveness and efficiency. we set the visual prompt length 20, the textual prompt length to 16, and the training epoch to 10 in all experiments.
A.2 Additional Results on Image Classification
Robustness of Various PEFT techniques To further substantiate the advantages of PEFT over FFT for adapting visual backbones in the presence of massive noisy labels, we conduct a comprehensive evaluation of various PEFT techniques for image classification, including 1) Visual Prompt Tuning (VPT) 18 which prepends learnable prompts to the input at each layer, 2) Low-Rank Adapter (LoRA) 13 which injects trainable rank decomposition matrices into each layer, 3) AdaptFormer 3 which replaces the original Multi-I Layer Perceptron (MLP) block with AdaptMI.P that incorporates trainable bottleneck module, and 4) Bias-terms Fine-tuning (BitFit) [50] which is a sparse fine-
15

---

Method
Sym. 0.2
Sym. 0.4
Sym. 0.6
Ins. 0.2
Ins. 0.3
Ins. 0.4
CIFAR-100
CE ELR
84.86 84.86
82.70/ 82.42
79.32 75.49
81.35 81.29
75.64 / 75.57
68.13 68.13
87.17 / 87.16
85,95
85.89
84.29/ / 84.29
87.05 87.01
86.93 / 86,91
86,58 / 86.57
PEFT
SCE GMM
86.94 86.90
85.24 85.17
82,40 81.66
83 98/ 83.81
81.16/ 79.40
77.43 / 72.30
86.99 86.95
85.53 85.51
84.16/ / 84.16
86.64 86.64
85.06 / 85.06
80.02 / 80.02
PEFT
87.50 87.50
86.69
86.69
84.00
83.88
87.58 87.58
85.68 85.64
83.73 83.71
DEFT
w/ FFT
89.38 89.35
88.17 88. 11
85.81 85.72
89.38 89.35
88.68 88.68
85.75 / 85.74
ImageNee
CE
80.73 80.71
77.99 77.99
74.44 / 73.91
79.58
79.58
75.93 77.89
71.08 7
ELR SCE GMM
83.01 83.01
81.81 81.81
79.34 / 79.34
82.93 82.93
82.28 82.28
8 81.71
PEFT
82.84 82.84
81.27 81.25
78.09 78.09
80.96 80.96
78.96 79.02
75.73 74.57
82.89 82.87
81.74 81.72
80.35 80.35
81.53 81.53
81.79 /81.79
78.61 78.61
w/ PEFT
/3.42
/2.12
80.43 80.43
82,76
82.74
82.63 82.57
81.31 /1.1/81.44
DEFT
w/ FFT
82.91 82.91
82.48/ 82.37
80.60 80.59
83.37 83.33
82.69 82.65
80.52 80.49
Stanford-Cars
CE
75.79
72.30
66,29
54.62
49.09 33.86
74.23 72.72
66.50 64.13
58.85 / 54.31
ELR SCE GMM
87.18
87.18
81.00 81.04
70.44 70.44
86.53 86.53
84.42 84.33
80.09 / 79.93
PEFT
77.95 / 71.84
/5.14
53.84 35.74
77.74/ 70.20
71.02 61.03
65.38 / 51.18
88.97
88.97
85.91 85.60
75.82 75.82
88.05 88.04
85.51 85,39
78.61 / 78.61
w/ PEFT
89.80/ 89.77
86.82 86.77
81.43 80.54
89.72 89.65
88.01 87.80
86.58 86.36
DEFT
w/ FFT
92.13/
/2.12 92.12
90.75 90.75
85.72 / 85.45
92.19/ / 92.15
90.77 / 90.77
89.74 / 89.68
CUB-200-2011
CE
69.35/ 65.22
54.42 / 46.91
43.98 30.77
/ 66.12
59.65 56.35
52.33 / 47.26
ELR
79.00
72.04 771.99
60.34/ / 60.24
78.32 78.29 69.97 62.82
/5.73/75.77
70.78 70.25
PEFT
SCE
65.55 70.12 65.55
61.84 45.31
49.55 / 29.25
63.55 54.28
56.68 45.01
GMM
78.75 78.51
77.72 76.91
64.83 / 64.69
80.10 / 80.10
76.06 / 76.06
67.97 / 67.97
PEFT
80.55 80.41
76.63
766.63
69.71 69.28 .88
80.57 80.50
77.25 77..11
74.42 74.20 79.34 79.24
DEFT
FFT
83.05 83.03
79.24 79.13
73.08 73.08
82.53 82.50
81.39 81.39
Table 10: Test accuracy (%) on synthetic datasets with symmetric and instance dependent label noise. For all approaches, utilize PEFT to adapt the pre- traitrained CLIP model. The results of DEFT when applying FFT for adaptation are also reported.
80
VPT
URA
BIRI
60
60
P-T
(a) VPT
(b) LoRA
(c) AdaptFormer
(d) BitFit
Figure 4: Comparison of different parameter-efficient fine-tuning techniques on Tiny- with various ratios of noisy labels
tuning method where only the bias-terms of the model are being modified. The results are shown in Figure 4 Notably, nearly all PEFT methods outperform FFT when the noise ratio exceeds 0.4, and the performance gap becomes more pronounced with the increase in the noise ratio. These results comprehensively underscore the efficacy of PEFT over FFT in mitigating the distortion of representations when adapting visual backbones under severe noise.
Applying PEFT for Model Adaptation Table 10 presents the test accuracy obtained by applying PEFT to adapt the pre- rrettrained CLIP model. There are three noteworthy observations in comparison to the results in Table 2 Firstly, previous label-noise learning methods consistently demonstrate their efficacy in handling noisy datasets compared to the vanilla methods (CE), showcasing clear improvement in performance across various datasets and noise settings. Secondly, despite the bad performance of GMM on fine-grain fine-grained datasets in Table 2 it achieves the best performance among
16

---

Algorithm 1: The Proposed DEFT Framework
Input: training dataset D = { y: )= PEFT parameters w, pre-trained parameters 0,
warm-up epoch To. PEFT epoch 7i and FFT epoch T2.
// Phase1: Learning Noisy Label Detector with PEFT
2 for / 1, ... To do
Warm-up the pre-trained model on noisy dataset D {1 Wa) 4 end
5 for f To .;. do
Construct the clean subset Delean by Eq. 4 and Eq.
Compute the total loss L = ddp fism by Eq. 7 and Eq. 88
Update current model parameters wt SGD DDelan L, wi
9 end // Phase2: Adapting Model on Clean Data with FFT for 1,2, .... T2 do
Compute the CE loss tee for samples in the clean subset Delean
Update current model parameters 0g SGI(Delaa lee, O2-1)
end
ViT-B/16
ResNet-50
ConvNeXi-T
MAE-ViT-B
Optimizer
SGD
AdamW
Adam W 1x10- 1x10-
AdamW
Learning Rate
1x10-2
1x10-3
1x10-4
Weight Decay
1x10-
1x10-
1x10
Table 11: Configurations of different pre-trained models.
baselines when utilizing PEFT to adapt the pre-trained model, possibly due to the more stable training process with fewer learnable parameters. Lastly, DEFT continues outperform all compared methods in almost all cases when applying PEFT during the model adaptation phase (DBFT w/ PEFT), and its performance confirming the validity of the proposed framework.
be further enhanced by replacing PEFT with FFT (DEFT w/ FFT),
A.3 Additional Implementation Details
Pseudo-code for the Proposed Framework DEFT is a two-phase framework consisting of a noisy label detection phase and model adaptation phase. The pseudo-code is presented in Algorithm
Implementation of the State-of-the-art Approaches To verify the effectiveness of DEFT real-world datasets, we compare our methods with several sample selection- based state-of-the-ar approaches including RoLT[40) UNICON(19), LengReMix(4), and ProMix[46] Notably, these methods utilize CNN-like network backbones in their original implementations. To ensure a fair comparison, we implement a two-phase evaluation approach: the clean subset, then fully fine-tune the pre-trained CLIP model using these selected clean samples. We opt not to directly replace the model backbone because these methods incorporate various training techniques such co-training, mixup, and contrastive learning. Removing the influence of these techniques is crucial to ensure a valid comparison.
first run the source code to obtain
Hyperparameter Settings for PEFT Modules In the noisy label detection phase, we employ VPT 18 to adapt the visual encoder by default. In all experiments, we consistently set the length of the visual prompt to 20. For LoRA and AdaptFormer, we establish the bottleneck dimension as 8.
Configurations of Pre-trained Models the model adaptation phase, we apply FFT range of models that are pre-trained on ImageNet- IK. For fair comparison, we follow the optimizer settings outlined in I and initialize each model with the corresponding pre-trained weights obtained from HuggingFace The specific configurations are described in Table 11
'atpss//nugggngface.o/
17

---

A.4 Limitations and Broader Impacts
Limitations Despite DEFT's superior performance compared to the existing methods. exhibits certain limitations and there are several unexplored research avenues. For example, the current algorithm only deals with noisy image-label pairs in single-label multi-class classification tasks. A promising avenue for future research would be to generalize DEFT to handle noisy image-text pairs or multi-label classification settings.
This study falls within the domain of weakly supervised learning, a learning paradigm that aims to achieve superior performance while reducing labeling costs. Consequently, as this technique gains efficacy and wider adoption, the necessity for extensive data annotation may get diminished, potentially contributing to rise unemployment among data annotation professionals.
Broader Impacts
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide clear overview of the research problem and outline the main contributions in the introduction.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Section A.4
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The complete details of the implementation be found in Section 5. and Section A.3
5. Open access to data and code
Question: Does the paper provide open the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our source code can be found via the link in the abstract.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all the necessary details in Section 5.1
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The paper does not report error bars following previous studies.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: The paper provides the type of compute workers (GPU) in Section 5.1
9. Code Of Ethics
19

---

Question:
Does the research conducted in the paper conform, in every respect. with the
NeurIPS Code of Ethics
Answer: [Yes]
Justification; The authors have reviewed the NeurIPS Code of Ethics and the research conforms with them in every respect.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We analyse the potential societal impacts in Section A.4
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The licenses for existing assets are properly respected.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code introduced in the paper is well documented.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
20


Reviews:
[{"review_id": "jQ3l9V9yh5", "reviewer": "Reviewer_g1PL", "summary": "The paper introduces a Denoising Fine-Tuning (DEFT) framework to address the challenge of noisy labels in vision-language models, particularly focusing on models like CLIP. The DEFT framework leverages the robust alignment of textual and visual features pre-trained on extensive image-text pairs to filter out noisy labels. This is achieved by learning class-specific positive and negative textual prompts. Positive prompts highlight distinctive class features, while negative prompts act as thresholds to differentiate between clean and noisy samples. The framework uses parameter-efficient fine-tuning (PEFT) to adapt the visual encoder to align with the textual prompts. Extensive experiments on synthetic and real-world noisy datasets demonstrate that DEFT significantly improves both noisy label detection and image classification performance.", "questions": "1. How does DEFT perform when applied to pre-trained models with varying degrees of generalizability and domain relevance?\n\n2. Can DEFT be adapted to scenarios with extremely high noise ratios or highly imbalanced datasets, and what modifications would be necessary to maintain its effectiveness?\n\n3. What are the potential trade-offs between the computational overhead of DEFT and its performance gains, and how can the framework be optimized for deployment in resource-constrained environments?", "limitations": "The authors have discussed the limitations of this paper, and there is no negative societal impact.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes to combining textual and visual prompts for noisy label detection, which enhances the robustness of vision-language models to label noise. \n\n2. The framework's generalizability to various pre-trained models and its parameter efficiency make it a versatile solution. \n\n3. The experimental validation on multiple datasets, including real-world noisy data, provides strong evidence of the method's effectiveness. \n\n4. The use of PEFT to maintain the generalization ability of pre-trained models while adapting to specific tasks is particularly noteworthy.", "weaknesses": "1. The heavy reliance on pre-trained models may limit the framework's applicability in scenarios where such models are not available or suitable. \n\n2. The discussion on the practical implementation and potential limitations in different real-world settings is relatively limited. \n\n3. The computational overhead associated with maintaining and fine-tuning dual prompts could be a concern in resource-constrained environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer g1PL,\n\nWe sincerely appreciate the reviewer for the thoughtful feedback. We are encouraged by comments like *a versatile solution* and *strong evidence of the method's effectiveness*. We address your concerns one by one.\n\n> W1. The heavy reliance on pre-trained models may limit the framework's applicability.\n>\n> Q1. How does DEFT perform when applied to pre-trained models with varying degrees of generalizability and domain relevance?\n\n**[prevalence of pre-trained models]** Pre-trained models have become a cornerstone in many tasks due to their ability to capture generalizable features from large datasets, such as multi-label classification [1], long-tailed learning [2], and out-of-distribution detection [3]. This paper focuses on noisy label detection, where the pre-trained CLIP model is available and suitable for our method in almost all scenarios. \n\n**[various pre-trained models]** In Table 4 of the paper, we compared various pre-trained models in addition to CLIP. The results validate the effectiveness of our method.\n\n**[domain relevance]** In addition, we conduct experiments on the MNIST dataset. MNIST consists of monochrome images of handwritten digits and has been verified to have a low domain relevance with CLIP as there is no data overlap with the CLIP pre-training data [4]. Results (F1-score) in the table below exhibit that our method consistently outperforms the small-loss baseline in sample selection performance.\n\n| Method          | *sym.* 0.2 | *sym.* 0.4 | *sym.* 0.6 |\n| --------------- | :--------: | :--------: | :--------: |\n| **Small-loss**  |   97.52   |   96.16   |   94.18   |\n| **DeFT (ours)** | **99.63** | **99.30** | **98.54** |\n\n> W2. The discussion on the practical implementation and potential limitations in different real-world settings is relatively limited.\n\n**[implementation]** We provided the code of this paper in the supplementary material, which contains the practical implementation details of our method on three real-world datasets. We will include more implementation details in the next version of the paper.\n\n**[limitations]** We discussed the limitations in Appendix A.4 of the paper. Specifically, our method may have some potential limitations under different real-world settings. For example, DeFT primarily focuses on the label noise problem in image classification, where the label is a class name. Therefore, it cannot directly handle the noise in image-text pair data, where the label is a text description of the image.\n\n> Q2. Can DEFT be adapted to scenarios with extremely high noise ratios or highly imbalanced datasets?\n\nWe reported the noise detection results under severe noise conditions in Appendix A.1 of the Appendix. To tackle high noise ratios, there are two necessary modifications: 1) the learning rate is adjusted from $3$$\\times$$10^{-2}$ to $1$$\\times$$10^{-2}$, and 2) a smaller weight is assigned to the positive loss component in $L_{dp}$ to mitigate the impact of noisy pseudo-labels. For highly imbalanced datasets, we can employ class-balanced loss functions to replace the standard cross-entropy loss $\\ell_{ce}$ in the model adaptation stage, such as the logit-adjustment loss [5].\n\n> W3. The computational overhead could be a concern in resource-constrained environments.\n>\n> Q3. How can the framework be optimized for deployment in resource-constrained environments\n\nIt is noteworthy that we utilize parameter-efficient fine-tuning (PEFT) techniques to adapt CLIP on downstream datasets, which is both effective and efficient compared to fully fine-tuning, as PEFT is more robust to label noise and requires optimizing much fewer parameters. Even in extremely resource-constrained environments, the proposed framework can still be adjusted in the following ways:\n\n1) **Learning transferable detector on smaller datasets**. In the first stage of DeFT, we can learn a noisy label detector on a small dataset and then transfer it to a larger one for sample selection. For example, the detector trained on Tiny-ImageNet can be used to detect noise in the overlapping classes of the ImageNet dataset.\n2) **Adapting model with smaller backbones**. In the second stage of DeFT, the filtered data can be used with various visual backbones, as shown in Table 4 of the paper. Therefore, we can utilize a smaller model and still achieve good performance with the filtered clean data.\n\n[1] Abdelfattah R, Guo Q, Li X, Wang X, Wang S. Cdul: Clip-driven unsupervised learning for multi-label image classification. ICCV 2023.\n\n[2] Shi J, Wei T, Zhou Z, Han X, Shao J, Li Y. Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts. ICML 2024.\n\n[3] Ming Y, Cai Z, Gu J, Sun Y, Li W, Li Y. Delving into out-of-distribution detection with vision-language representations. NeurIPS 2022.\n\n[4] Radford A, Kim J, Hallacy C, et al. Learning transferable visual models from natural language supervision. ICML 2021\n\n[5] Menon A, Jayasumana S, Rawat A, Jain H, Veit A, Kumar S. Long-tail learning via logit adjustment. ICLR2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OPJztpSz6y", "reviewer": "Reviewer_6CXy", "summary": "The paper proposes a method for detecting noisy samples using vision-language models (CLIP). The main idea is to efficiently adapt (via prompt tunning) the clip model on noisy data and use this adapted model to select clean data. In the second stage, the clean data can be used to fully fine-tune a backbone model. The paper proposes the following contributions: a) using two learnable text prompts (positive and negative) to avoid using a threshold when detecting noise b) using negative learning [40] c) using a two-stage approach to adapt to the noisy distribution (first select clean samples then fully fine-tune using them). Experiments are done on multiple datasets with synthetic (CIFAR-100, Tiny-ImageNet, Stanford-Cars, CUB-200-2011) or real noisy labels (CIFAR-100N, Clothing1M, WebVision).", "questions": "* Is the $L_{sim}$ loss (Eq.8) used for the adaptation in the second stage, or is it used after the selection in phase 1? If the latter, would this mean that this loss updates the PEFT module and the learnable prompts, and then the final model is used in the second stage?", "limitations": "The method is not sufficiently evaluated against fair baselines, see weak points for more details.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* S1. Good direction of efficiently adapting CLIP models for noise detection.\n* S2. The general setting is sound, of using efficiently fine-tuned CLIP to select clean data and then fine-tuning using it.\n* S3. Good results on multiple noisy datasets.", "weaknesses": "* W1. There needs to be more fair comparisons with baselines that are trained in the same settings and use similar models. \n\n* W2. Table 1 seems to compare DEFT which has a fine-tuning stage on the noisy dataset with the initial clip model that is not trained or adapted at all. Is this the case? Is the small-loss baseline using the initial CLIP model, without any adaptation to the current dataset? \n\n* W2.2. More appropriate baselines would be CLIP models that are adapted in standard ways to the current noisy dataset. Thus, one aspect that needs to be ablated is the fine-tuning method (none, fully fine-tuning, efficient fine-tuning, etc). Then, given a CLIP model, the second aspect that needs ablation is the selection method. Here the paper already compares against zero-shot and small-loss selection, but using the initial, pre-trained CLIP. The same should be done using an adapted clip. \n\n* W2.3. Another simple selection method that, like DEFT doesn’t require a threshold would be to select samples with $sim(I_i, T_k) > th$ (where th=0) where $T_k$ is the text features corresponding to the correct class. This selection should be different from the zero-shot selection because the threshold is applied directly on the similarity of the image and correct class features, without applying softmax, thus without taking into account if the correct class is the class predicted by the zero-shot model.\n\n* W3. Table 2 and Table 3 compare against multiple methods for training in noisy settings, starting with the same visual backbone, i.e. the visual part of CLIP. Is this correct? The question arises, is DEFT using additional information since it does the selection using both the visual and textual part of CLIP? Thus, the DEFT has a possible unfair advantage.\n\n* W4. What is the motivation for using two text prompts (positive and negative)? If only one prompt is used (as usual), we can still make a prediction and optimize this prediction to be the correct class and then apply a fixed threshold (>0.5). Why shouldn’t the simple approach work and what benefits does the dual prompt bring?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 6CXy,\n\nWe sincerely appreciate the reviewer for the thoughtful feedback. We are encouraged by comments like *The general setting is sound* and *Good results on multiple noisy datasets*. We address your concerns one by one.\n\n> W1~W2.2. There needs to be more fair comparisons with baselines in Table 1.\n\nTo improve the clarity, we would like further to explain the experimental settings of the two baselines:\n\n1) Zero-shot uses the initial CLIP model to predict labels for the training set. Samples where the given label matches the model's prediction are considered clean. In order to distinguish such an approach from model fine-tuning methods, we rename it as the **\"Label-match\"** strategy in the following discussion.\n2) Small-loss selects a proportion of samples with small loss *during training* as clean samples. Therefore, the Small-loss baseline in Table 1 also utilizes the adapted CLIP model like our method, ensuring a fair comparison.\n\nTo make a fair comparison between the Label-match strategy and our method, we conduct an ablation study on noisy CIFAR-100 based on the CLIP model adapted with parameter-efficient fine-tuning (PEFT). The Table below presents the F1-score for noisy label detection by three methods. It can be seen that our method outperforms the other two baselines under varying noise ratios.\n\n\n| Method                  | *sym.* 0.2 | *sym.* 0.4 | *sym.* 0.6 |\n| ----------------------- | :--------: | :--------: | :--------: |\n| **Label-match w/ PEFT** |   95.64   |   94.54   |   93.06   |\n| **Small-loss w/ PEFT**  |   97.01   |   95.08   |   91.79   |\n| **DeFT (ours) w/ PEFT** | **98.63** | **98.33** | **97.15** |\n\nFurthermore, we agree with the suggestion to perform ablations on the fine-tuning methods. Accordingly, we use a CLIP model fully fine-tuned on noisy CIFAR-100 for sample selection and present the F1-score in the table below. Comparing the results of the two tables, we observe that:\n\n1) Regardless of the fine-tuning method used, our method (DeFT) achieves superior results.\n2) Parameter-efficient fine-tuning (PEFT) performs better under high noise conditions, which is why we choose PEFT for adapting the CLIP model in the noisy label detection stage.\n\n\n| Method                 | *sym.* 0.2 | *sym.* 0.4 | *sym.* 0.6 |\n| ---------------------- | :--------: | :--------: | :--------: |\n| **Label-match w/ FFT** |   96.17   |   93.11   |   85.10   |\n| **Small-loss w/ FFT**  |   97.16   |   94.77   |   89.81   |\n| **DeFT (ours) w/ FFT** | **98.76** | **97.44** | **90.03** |\n\n> W2.3. Select samples with $sim(I_i, T_k) > 0$.\n\nIn practice, selecting samples with $sim(I_i, T_k) > 0$ is not effective. Our experiments on CIFAR-100 with 60% symmetric noise using the initial CLIP model reveal that the minimum $sim(I_i, T_k)$ value among all samples is 0.098. As a result, setting the threshold at $sim(I_i, T_k) > 0$ does not exclude any samples and is thus not useful for filtering out noisy samples. However, when we set the threshold to 0.25, the F1-score for noise detection improved significantly to 87.57%. This indicates that the effectiveness of sample selection based on $sim(I_i, T_k)$ is highly dependent on the choice of threshold.\n\n> W3. Table 2 and Table 3 compare against multiple methods with the same visual backbone. Is DEFT using additional information since it does the selection using both the visual and textual part of CLIP? Thus, the DEFT has a possible unfair advantage.\n\n**All methods in Table 2 and Table 3 use the same backbone**, specifically the visual part of CLIP, and our method indeed leverages additional text information for sample selection. However, we do not view this as a drawback but rather one of the main contributions of this paper. To achieve better noise detection performance than previous unimodal methods, we introduce the text modality within the CLIP model. Through the DeFT framework, we successfully utilize multimodal information to enhance noise detection, achieving superior results compared to prior unimodal approaches. To the best of our knowledge, DeFT is the first method to effectively leverage multimodal information in label-noise learning. Similar explorations have also appeared in other domains, such as multi-label classification [1], semi-supervised learning [2], and out-of-distribution detection [3].\n\n> W4. What is the motivation for using two text prompts (positive and negative)? What benefits does the dual prompt bring?\n\nThe key advantage of using dual textual prompts is that they enable adaptive and instance-specific selection. With a single prompt, a fixed threshold must be manually set. As discussed in the response to W2.3, finding an appropriate fixed threshold is challenging and often requires prior knowledge, such as the noise ratio. However, by using dual prompts, sample selection can be based on the similarity between each sample and the corresponding two prompts, i.e. $sim(I_i, T_k^+) > sim(I_i, T_k^-)$. This is equivalent to learning an adaptive threshold for each sample in a data-driven manner, eliminating the need for manual tuning.\n\n> Q1. Is the $L_{sim}$ loss (Eq.8) used for the adaptation in the second stage, or is it used after the selection in phase 1?\n\nSorry for the confusion. $L_{sim}$ loss is only used in the first stage. With the filtered data, we can adapt any visual backbones using the selected clean data in the second stage. We summarized the proposed method in **Algorithm 1** in the Appendix of the paper.\n\n[1] Abdelfattah R, Guo Q, Li X, Wang X, Wang S. Cdul: Clip-driven unsupervised learning for multi-label image classification. ICCV 2023.\n\n[2] Mo S, Kim M, Lee K, Shin J. S-clip: Semi-supervised vision-language learning using few specialist captions. NeurIPS 2023.\n\n[3] Ming Y, Cai Z, Gu J, Sun Y, Li W, Li Y. Delving into out-of-distribution detection with vision-language representations. NeurIPS 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ynA8nrooJZ", "reviewer": "Reviewer_FyAK", "summary": "This paper focuses on improving the finetuning performance of pretrained vision models by removing the noisy labeled data. Specifically the authors propose a two-stage method: the first stage is to learn a noise detector by prompt learning of the text encoder in CLIP. The second stage is to full-finetune the pretrained models (including CLIP and other pretrained models) . In the experiment, the authors include both synthetic datasets and real datasets and compare their model with different 6 different methods. They also use the filtered dataset to finetune different pretrain models.", "questions": "1. I am confused about the optimization for the positive prompts. Are positive prompts learnt together with negative prompts and Visual PEFT by using $L_{dp} + L_{sim}$? Or are they learned separately?\n2. Can a noisy detector learned on one dataset transferred to another dataset? For example, the noisy detector learned on ImageNet can be used to detect noise in CIFAR100 dataset since there are a great number of class overlapping in these two datasets?", "limitations": "The authors have included the discussion of limitations in the supplementary.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well writen and easy to follow.\n2. The analysis of the relationship between noisy data and finetune methods is very insightful (Figure 1)\n3. The proposed two-stage method is reasonable and results in good performance compared to other baselines.\n4. It is great to show that the filtered data works for both CLIP models and non-CLIP models even though the filtering is done by the model based on CLIP.", "weaknesses": "1. The optimization part is a bit unclear. (See Questions Below)\n2. The proposed noisy detector cannot be reusable when working on a different dataset. Each downstream dataset needs a specific trained detector.\n3. Noisy label problem is a more severe problem during the pretraining than the finetuning. But this paper only focuses on the finetuning, which makes the research scope narrow in this case.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer FyAK,\n\nWe sincerely appreciate the reviewer for the thoughtful feedback. We are encouraged by comments like *The analysis of the relationship between noisy data and finetune methods is very insightful* and *is well-written and easy to follow*. We address your concerns one by one.\n\n> W1. The optimization part is a bit unclear.\n>\n> Q1. Are positive prompts learnt together with negative prompts and Visual PEFT by using $L_{dp} + L_{sim}$? Or are they learned separately?\n\nSorry for the confusion. The positive prompts are **learnt together** with the negative prompts and Visual PEFT by $L_{dp} + L_{sim}$ in the first stage. We summarized the proposed method in **Algorithm 1** in the Appendix of the paper.\n\n> W2. The proposed noisy detector cannot be reusable when working on a different dataset.\n>\n> Q2. Can a noisy detector learned on one dataset transferred to another dataset?\n\nThough the main focus of this paper is to identify noisy labels in the specific downstream data, the proposed noisy label detector can also generalize to a different dataset. To validate this, we use the noisy Tiny-ImageNet dataset ($64$$\\times$$64$ size) to train the noisy label detector and test it on the ImageNet dataset with 40% symmetric noise. The reason we do not use the noisy CIFAR-100 dataset for training is due to the difficulty in constructing a consistent class name mapping between CIFAR-100 and ImageNet (e.g., \"turtle\" in CIFAR-100 is referred to as \"loggerhead\" in ImageNet). \n\nThe experimental results show that the F1-score on the noisy ImageNet dataset reaches 95.16%, which is 10.16% higher than the Zero-shot baseline. Besides, we find that the proposed noisy label detector can also generalize to other types of noisy labels. We construct a test dataset comprising 50% instance-dependent label noise to verify the effectiveness of the detector trained with symmetric noise and achieve an F1-score of 97.88%. These results demonstrate the strong generalization ability of the proposed noisy label detector.\n\n> W3. Noisy label problem is a more severe problem during the pretraining than the finetuning. But this paper only focuses on the finetuning, which makes the research scope narrow in this case.\n\nThe noisy label problem during fine-tuning is significant because it directly impacts the model's performance on specific downstream tasks. To address this challenge, we propose the DeFT framework, which significantly improves the robustness of models against noisy labels, making a valuable contribution to the field. \n\nWhile our focus is on fine-tuning, we acknowledge that exploring the noisy label problem during pretraining is an intriguing avenue for future research. We hope our work provides meaningful insights and promotes further studies on this problem.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3q1ICqXe0c", "reviewer": "Reviewer_mmEa", "summary": "This paper proposes a novel method for learning with noisy labels leveraging pre-trained foundation models. The method is motivated by new findings that prompt learning is more robust to noisy labels when fine-tuning CLIP. The paper then designs a simple detector by learning both positive and negative prompts for each class. This paper showcases strong empirical results on multiple datasets, outperforming many existing methods.", "questions": "1.\tCan examples detected as noisy label help improve the performance?", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe proposed method is well-motivated. The effectiveness of CLIP is often overlooked in previous works for learning with noisy labels, and this paper examines it thoroughly and finds a good way to adapt CLIP in downstream tasks even with noisy labels.\n2.\tThis paper makes an important contribution that prompt learning is robust to robust to noisy labels, while full fine-tuning is better on clean datasets. Based on this observation, the paper proposes a two-stage method.\n3.\tThis paper proposes a novel idea by jointly learning positive and negative prompts for classes, and leverage the negative learning loss to optimize learnable prompts.\n4.\tExtensive experiments on both synthetic and real-world datasets show that the proposed method consistently outperforms existing baselines.", "weaknesses": "1.\tIt is unclear why positive and negative prompts can help detect noisy labels. It is suggested to provide some visualization examples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer mmEa,\n\nWe sincerely appreciate the reviewer for the thoughtful feedback. We are encouraged by comments like *The proposed method is well-motivated* and *This paper makes an important contribution*. We address your concerns one by one.\n\n> W1. It is unclear why positive and negative prompts can help detect noisy labels. \n\nThe primary benefit of utilizing dual textual prompts is their ability to facilitate adaptive and instance-specific selection. In contrast, the conventional Small-loss strategy presents two significant limitations:\n\n1) The Small-loss strategy requires the selection threshold to be manually set. The efficacy of noise detection heavily depends on this threshold, and finding an optimal value typically necessitates prior knowledge, such as the noise ratio in the dataset.\n2) The Small-loss strategy assumes that all samples with high loss values are inherently noisy. However, it overlooks the presence of hard but clean samples that also exhibit high losses. By sorting samples based solely on loss and selecting only those with lower losses as clean, the Small-loss strategy risks overlooking these challenging yet valid instances.\n\nBy leveraging dual prompts, sample selection can be based on the similarity between each sample and the corresponding two prompts, i.e. $sim(I_i, T_k^+) > sim(I_i, T_k^-)$. This is equivalent to learning an adaptive threshold for each sample in a data-driven manner, thereby circumventing the limitations of the Small-loss strategy. \n\n> Q1. Can examples detected as noisy label help improve the performance?\n\nWe can make use of noisy data for further improvement by treating noisy data as unlabeled data and leveraging semi-supervised techniques. Specifically, by simply incorporating FixMatch [1] in the second stage of DeFT, we improve the test accuracy on noisy CIFAR-100 with 60\\% symmetric noise from 85.72\\% to 86.13\\%.\n\n[1] Sohn K, Berthelot D, Carlini N, et al. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. NeurIPS 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces DeFT, a Denoising Fine-Tuning framework that effectively adapts vision-language models to handle noisy labeled data. The key innovation of this work lies in the use of positive and negative textual prompts to enhance the alignment of visual and textual features, which helps in detecting and mitigating the impact of noisy labels. DeFT's versatility in being applicable to various pre-trained models, along with its parameter-efficient approach, is well-supported by strong experimental results on both synthetic and real-world noisy datasets. All reviewers agree that the paper provides significant contributions and is ready for publication. The AC agrees with this assessment.
Decision: Accept (poster)