NoiseGPT: Label Noise Detection and Rectification through Probability Curvature
Authors: Haoyu Wang
Zhuo Huang
Zhiwei Lin
Tongliang Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: Label noise
Multimodal Learning
Large Language Models
Primary area: safety_in_machine_learning

Abstract: Machine learning craves high-quality data which is a major bottleneck during realistic deployment, as it takes abundant resources and massive human labor to collect and label data. Unfortunately, label noise where image data mismatches with incorrect label exists ubiquitously in all kinds of datasets, significantly degrading the learning performance of deep networks. Learning with Label Noise (LNL) has been a common strategy for mitigating the influence of noisy labels. However, existing LNL methods either require pertaining using the memorization effect to separate clean data from noisy ones or rely on dataset assumptions that cannot extend to various scenarios. Thanks to the development of Multimodal Large Language Models (MLLMs) which possess massive knowledge and hold In-Context Learning (ICL) ability, this paper proposes NoiseGPT to effectively leverage MLLMs as a knowledge expert for conducting label noise detection and rectification. Specifically, we observe a \textit{probability curvature} effect of MLLMs where clean and noisy examples reside on curvatures with different smoothness, further enabling the detection of label noise. By designing a token-wise Mix-of-Feature (MoF) technique to produce the curvature, we propose an In-Context Discrepancy (ICD) measure to determine the authenticity of an image-label pair. Subsequently, we repeat such a process to find the best matching pairs to complete our label rectification. Through extensive experiments, we carefully demonstrate the effectiveness of NoiseGPT on detecting and cleansing dataset noise, especially on ILSVRC12, the AUROC of NoiseGPT reached over 0.92. And by integrating with existing methods, the classification performance can be significantly improved on noisy datasets, typically by 22.8\% on 80\% symmetric CIFAR-10 with M-correction. Source code: \url{https://github.com/drunkerWang/NoiseGPT}

NoiseGPT: Label Noise Detection and Rectification through Probability Curvature
Haoyu Wang School of Automation Beijing Institute of Technology haoyu. wang@bit iorttttt.. edu cn
Zhuo Huang
Sydney Al Centre University of Sydney zhuohuang. ai @gmailmlll.... com
Zhiwei Lin" School of Automation Beijing Institute of Technology linzhi wei @bit. edu.. cn
Tongliang Liu' Sydney AI Centre University of Sydney tongliang. liuusydneyy..... edu.. edu. au
Abstract
Machine learning craves high-quality data which is a major bottleneck during realistic deployment, it takes abundant resources and massive human labor collect and label data. Unfortunately, label noise where image data mismatches with incorrect label exists ubiquitously in all kinds of datasets, significantly de- grading the learning performance of deep networks. Learning with Label Noise (LNL) has been common strategy for mitigating the influence of noisy labels. However, existing LNL methods either require pertaining using the memorization effect separate clean data from noisy ones or rely on dataset assumptions that cannot extend to various scenarios. Thanks to the development of Multimodal Large Language Models (MILMMs) which possess massive knowledge and hold In-Context Learning (ICL) ability, this paper proposes NoiseGPT to effectively leverage MLLMs as a knowledge expert for conducting label noise detection and rectification. Specifically, where clean and noisy examples reside on curvatures with different smoothness, further enabling the detection of label noise. By designing token-wiss Mix- of of-Feature (MoF) technique to produce the curvature, we propose an In-Context Discrepancy (ICD) measure to determine the authenticity of an image-label pair. Subsequently, we repeat such a process to find the hest matching pairs to complete label rectification. Through extensive experiments, the effectiveness of NoiseGPT on detecting and cleansing dataset noise, especially ILSVRC12, the AUROC of NoiseGPT reached over 0.92. And by integrat- ing with existing methods, the classification performance can be significantly improved on noisy datasets, typically by 22.8% on 80% symmetric CIFAR-10 with M-correction. Source code: https: thub. com drwnkerwang/Noiieeee..
observe probability curvature effect of MLLMs
carefully demonstrate
Introduction
Contemporary machine learning is greedy for high-quality datasets. However, large-scale datasets such human-annotated ones like ImageNet [1] and COCO [2] internet-downloaded ones like Wision [3] and Instagram Datasets [4] are either resource-consuming untrustworthy. As
"Corresponding author; Zhiwei Lin (linzhiweid) bit.edu.cc) 'Corresponding author: Tongliang Liu (tongliang, liu @sydney.edu.au)
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

result, practitioners often have to spend substantial time to conduct prolonged labeling process and the results could still be undesirable. Noise still ubiquitously exist in almost all kinds of datasets. More importantly, dataset noise has been demonstrated to be significantly harmful for training of Deep Learning Models [5]. Consequently, it is urgent to discover efficient and transferable methodology to identify and rectify dataset noise.
To solve this problem, Learning with Noisy Labels (LNL) aims to improve the robustness of Deep Neural Networks (DNNs) through bridging the noise distribution and clean distribution. Existing methodologies [6, 7] commonly leverage loss correction and loss reweighting techniques, where transition matrix [8, 9]. for example, is used to capture the noise generation process which enables end- to-end optimization without fitting to noisy labels. Meanwhile, based on the observation that DNNs converge faster on clean examples than noisy ones, sample-selectinn velection-based methods [10, 11, 12] divide samples into clean and noisy during training in order to learn from confident examples while exploiting noisy samples [13, 14, 15, 16, 17]. However, it is challenging to estimate accurate noise transition matrix due to the complexity of real-world noise generation process. And the performance of approximation struggles under high-level noise rate without strong dataset assumptions. Consequently, existing methods are largely limited and require to be improved or assisted.
Thanks to the development of MLLMs [18, 19, 20, 21, 22] which have been trained on massive data to effectively fit various real-world data distribution, we propose leverage MLLMs as knowledge experts to help reduce dataset noise. Based on the empirical findings that noisy data are distributed on different MLLMs probability curvature from clean data, we propose an intuitive hypothesis that MIL.Ms are inherently optimized for the matching of image-text pairs. Such hypothesis motivates us to propose NoiseGPT which leverages novel In-Context Discrepancy (ICD) criteria to identify and rectify noisy examples. Particularly, given an example which is in context with its label from dataset, if they match with each other, the MLLM output is stable under perturbation. Conversely, if the image and label are unmatched, the MLLM output will be sensitive to input perturbations. In circumstances where a sample is regarded as noisy, we further leverage CLIP [23] as zero-shot classifier to generate candidate labels. ICD is also applied on these candidates to elect corrected label with best score. Through extensive studies on datasets such as conventional corrupted versions of CIFAR-10, 10, CIFAR- 100 [24], ImageNet ISCVRC2012, as well real-world grounded datasets like CIFAR-N [25], Web Vision [3]. the efficacy of NoiseGPT is rigorously validated. As a zero-shot data cleansing method, NoiseGPT demonstrates its powerful ability to scalably distill significantly cleaner versions of noisy datasets than their original ones. Furthermore, our algorithm can be embedded into other LNL methods to further enhance the noisy learning performance under various LNL scenarios. To conclude, our contributions in this paper are as follows:
We introduce MILMS as machine experts to cope with noisy labels for the first time, potentially mitigating the reliance on human labor.
We propose NoiseGPT to tackle the challenge of label noise by leveraging zero-shot capa- bility of MLLMs. We employ a novel In-Context Discrepancy (ICD) criteria combined with the token-wiss wise Mixture-of- and identify noisy samples. Additionally, these noisy samples are recycled after label panne cation by comparing ICD scores among candidate labels generated by zero-shot classifier CLIP.
(MoF) technique to quantify the possibility discrepancy
We conduct intricate investigations to evaluate the effectiveness of NoiseGPT, Furthermore, we integrate NoiseGPT as an auxiliary data cleansing method alongside existing LNL algorithms to validate its performance improvement. Additionally, we conduct performance analysis comprehensively understand the details of NoiseGPT.
2 Related works
2.1 Learning with noisy labels
Existing LNL methods can be categorized into three types, data cleaning, loss-adjustment based approaches and sample-selection based approaches. Data cleaning endeavors to filter out examples whose labels are likely to be corrupted [26, 27]. Previous works in this branch leverages various methods [28, 29, 30] such as bagging, boosting, K-nearest neighbor, anomaly detection to exclude falsclabeled instances. However, these methods tend to over-clean samples that even true-labeled,

---

resulting in aggravation of shortage of data in many cases. Tendencies of probability curvatures of DNNs [31, 32, 33] during training are also utilized filter noisy examples. However, their robustness is strongly correlative to the training setting. Loss-adjustment based approaches focus on modifying the loss items before updating the DNNs, including loss correction and loss reweighting. Based on the fact that DNNs using Cross Entropy(CH) loss are prone to overfit noisy data [5], substantial researches have been conducted to design a robust corrected loss [34, 35] by leveraging transfer learning [36, 37, 38, 391, where noise transition matrix [8, 40, 41, 42] has been utilized. However, the performance of loss correction is highly dependent on the precisely-estimated transition matrix, which is hard under heavy noise and large number of classes. And the correction errors will be accumulated during training. Loss reweighting, on the other hand, aims at attributing weighted importance to examples in designed training scheme to separate clean and noisy examples [43, 44, 45, 46). Rectifying vectors [47] are also used to guide the classification network through leveraging information from input of the logits and labels. Nonetheless, this approach requires particular reweighting functions and hyperparameters for different noise type and datasets, limiting its practical implementation.
Another line of work, sample-selection, endeavors to identify true-labeled examples from noisy training datasets during training. Researches [48, 49, 13, 50] distinguish clean examples by their early-stage loss in DNNS utilizing memorization effect [51, 52]. Multi-network learning [53, 54, 11] simultancously trains an additional network guide the sample sample-selection. Mix-up [15] employs a semi- -upersupe mixture model [55] to separate clean and noisy examples, which is integrated into multi-networf framework in DivideMix [13]. SELF [56] leverages unsupervised loss from unlabeled examples while maintaining a running average model called mean-teacher [57, 58). Pro- Mix [59] leverages the utilities of clean examples by training balanced and unbiased classifiers in self-supervised framework on separated sub-datasets. Recently, RoCL [60] combines supervised and semi-supervised learning in a two-stage training strategy exploit both selected clean examples and relabeled examples. Despite their achievement, existing sample-selection methods are intrinsically linked to specific classification tasks, incapable to explicitly provide cleaned versions of datasets in various scenarios. To cope with this, our work aims to provide a transferable noise reduction paradigm detect and rectify noisy labels by leveraging the shoo capability of MLLMs.
2.2 Multi- models
Recent years, Large Language Models [61, 62, 63, 64, 65, 66, 67] have been successfully applied across different tasks of natural language processing, Concurrently, the arising of Vision Transformers (ViTs) [68] has significantly advanced the development of visual models. In order to align the representations of image and text, CLIP [23] utilizes unsupervised learning by training separate image and text encoders with a contrastive loss on substantial image-tex pairs. Besides, researches also endeavor to augment LLM with pretrained visual models to obtain multi-modal LLMs, known as MLLMs [69, 70, 71]. InstructBLIP [72] focuses on equippin MLLMS with the capacity to follow human-instructions on natural-language interface. The mPLUG-OWl [73] introduces modularized multi-modal pretraining paradigm to boost its transferability. MMICL [21] proposes a novel context training scheme, allowing the insertion of image features at any position among input text tokens. Based on the powerful zero-shot in-context capability of MLLMs, we propose NoiseGPT, leveraging extensive knowledge acquired from vast multi-modal training examples, detect label noise. Subsequently, we utilize the zero-shot classification capability of CLIP for candidate labels and compare their matching rates with the input image to derive rectified label.
3 Methodology
In this section, we propose NoiseGPT for label noise detection and rectification, as shown in Figure 1. Given classifier and an MLLM model parameterized by & and 0 = {Oene, (ddcc),de) respectively, and small clean exemplar dataset De with several examples per category to provide prompt for MLLMs, we take advantage of intriguing Probability Curvature effeet of MLLM where clean examples x pllann and noisy examples nooisy lead to different prediction discrepancies under perturbation, e.g., E S (anoisy) dxnnoiey; s snoisy) < Ezsteen ~p( sadcem/geeean d s.lean), where d(::-) denotes novel In-Context Discrepancy (ICD) measure and 5 stands for the perturbation of an example under distribution p(ā)x)). Based on such an effect, we can successfully detect whether a given image-label example pair x {x. ū) has clean labels, i.e., whether ÿ matches with x. By
3

---

4o (x.
Va y False
MILM
qo
% molss
Vo i True
X notsy
2 2
geleun
X2
moLey
MoF
(x)
x slean
classtt class82
x gaissy
clessPin
v/o
Classifier
Discrepancy
Retoon
PXIx
Pa
Figure I: We leverage the zero-shot ability of MLLM to examine whether example pair is noisy. To identify the potential noise, NoiseGPT first perturbs a given example z and produces a set of augmented versions è via using novel token-wise fixture-oo-Feattue comparing the Softmax probabilities qo between x and ž, we can calculate an In-Context Discrepancy (ICD) measure further decide the authenticity of the given label
(MoF) technique. Then by
further exploring the probable class candidates of the classifier, we can effectively find the ground truth label by choosing the best-matchie matt-matching category. Hence, our NoiseGPT can assist as a dataset cleanser without human intervention.
Generally, PP includes two stages: 1) Noise detection and 2) Label rectification. Next, we first demonstrate the noise detection process in Section 3.1, and then we carefully elucidate the details of label rectification in Section 3.2.
3.1 Noise Detection
In this section, propose to con- duct noise detection based on the probability curvature effect. Specifi- cally, we observe that clean examples lie on smooth and convex probabil- ity curvature, and noisy examples fall on fluctuated and non-convex curva- ture, as shown in Figure 2. A similar effect has also been found in Mitchell et al. [74]. As a result, under slight perturbation, the probability value would change differently, which al- lows us identify dataset noise. To utilize such an effect, we first con- duct novel token-wist Mixture-of- Feature (MoF) to perturb each exam- ple, then, the In-Context Discrepancy (ICD) measure can be calculated to assist detection.
clean
qo (x,
4a (x,
xe nnisy
R
solly
grlsen
perturbation
Figure 2: We demonstrate the distinctive curvatures clean and noisy examples through an experiment. Ten perturbed exemplars are generated for a clean and a noisy sample from CIFAR- 1-0AR-10 respectively. The MLLM output Softmax prob- ability qe (xlğ) of perturbed clean exemplars xelean ~ p(-) (left) reside within convex region the curvature; While those of noisy exemplars knoisy ~ p(-) (right) tend to cluster around the original point, posing lower higher probability.
Mixture-of-Feature aims to perturbate or augment examples at the feature level through interpo- lation, segmentation, and partial substitution, which avoids changing in the input space it might cause a mismatch between modalities [75]. Since dataset noise owes to the image example possessing some confusing features that resemble other noisy classes, propose to mix the features between given query example r and exemplars =² < De from other noisy classes to inject noisy signals. Specifically, our MoF process is formulated as
ž -(z|z A z([(m= 1)] @ td [[([m 0)], where p(m 1) w,
(I)

---

CIFAR 10 Sym. 50%
CIFAR-100 Sym. 50%:
CIFAR-1ON CIFAR-13ON Wabvision 404
I.SVRC12 Sym. 40W
Density
siae
ICD score under perturbation produced by MLLM
Figure 3: We conduct validation experiments on six noisy datasets using ICD score, namely CIFAR- 10 Sym. 50%,, CIFAR-100 Sym. 50%,, CIFAR-ION with worse labels, CIFAR-100N with noisy labels. Webvision Sym. 40%,, ISLVRCI2 Sym. 40%.. We collect scores of 10,000 clean samples and 10,000 noisy samples from each dataset, and each sample is augmented by 10 perturbed exemplars with MoF weight of 0.5.
where z and zt are the latent representations of imput images x E and x E x correspondingly extracted by the vision encoders of MLLMs, z[[(-)] denotes the indexing operation that selects the elements from z where an indexing function TI(.) holds true, @ stands for element-wise addition, p(.) stands for the MoF function, and m indicates binary mask to select feature elements, which controlled by probability w to trade off between and zt larger w reserves more information from the original input in the mixed feature 2.
Different from the MoF process conducted using intermediate features in Tong et al. [75] which might break the image patterns further causing misalignment with the subsequent LLM decoder, we propose token-wise MoF operation that mixes the features by replacing tokens. Intuitively, MLLMs are sequentialized tokens, each containing specific semantic
the latent embeddings information. Thus, we conduct MoF from the token level, which can correspondingly perturbate semantic parts across different example pairs. In this way, the mixed feature can successfully inherit noisy information and still be aligned with the latent MLLM feature space. After producing mixed features the perturbed versions of original ones z, we combine them with the embeddings of both prompt and label and feed them to the MLLM decoder module to obtain the final prediction as qgstes {8, z, 7.. Next, we leverage the In-Context Discrepancy (ICD) criteria to detect dataset noise.
In-Context Discrepancy is based on the probability curvature effect where perturbation affects the prediction probabilities of clean and noisy examples differently. Formally, our ICD criteria is calculated
d(ss, z. 8, quace (s, z. 1) Ba- Agdce 7),
(2)
where d(::-) denotes the ICD function calculated between two entries, and p(ž|z) is the perturbation distribution for generating ă. Intuitively, MLLMs are convexly optimized to associate visual features with corresponding text labels. an example pair x is clean, i.e., image x and label ÿ are matched, will reside in an extrema where its local probability curvature is smooth and convex. On the other hand, if x a noisy point where the x and ÿ are mismatched, it might fall on unstable and non-convex curvature.
By injecting a little perturbation as done by the previous MoF process, we observe that the MLLM prediction of clean examples will gradually decrease, but for noisy examples. qe would seriously oscillate random value around its original one, as depicted in Figure 2. Therefore, our ICD results for clean examples are always positive and relatively larger than noisy ones, further effectively validating the authenticity of input example pairs. To rescale the ICD score of different examples, we further conduct normalization by dividing the results of Eq. 2 by standard deviation of godse (s, 2, O):
Qgder (8, () Eg--p(
R.
d(ss, z, D1 2,
EE E2 x) [goee (s, ž. E2 xx) Godse (s, 2, 9)
As result, we can effectively divide clean examples and noisy ones based on the distribution of d(2, ž) as illustrated Figure 3. Moreover, the effectiveness of such an effect carefully validated on various datasets under MMICL [76], state-of-the-art MILMM that supports effective image-text in-context learning (ICL) in Section In practice, we set threshold r to decide whether an example pair is noisy not: those with discrepancy scores larger than 7 considered as clean, otherwise. we further conduct label rectification.
3.2 Label Rectification
5

---

Thanks to the previous noise detection pro- cess, we can effectively validate the authen- ticity of given example pair x: = {x, 5)- Moreover, we hope to find potentially cor- rect labels to rectify the noisy ones ÿ- Particularly, we repeat the noise detection process for C' probable noisy categories to find the most likely label y, formally
Algorithm NoiseGPT: noise identification and rectifi-
cation.
Input:
sample a and label from dataset D MLLLM qo- pretrained multi-classifier Jee. perturbation function weight of MoF w, number of perturbations number of candidate labels C, threshold 7
1: Uniformly sample pC examplers with ground truth labels from Deet to construct tiny support set D
2: for x in D do
y A arg max maxxdd(s, z, yi 8, y. pred
x; ~ p(
p(x
[1,
P token-wise MoF
11
(4) where yored indicates the most proba- ble candidate labels of selected by the classifier & where the subscript denotes the j-th entry. Since various datasets have different numbers of classes, it is infeasible traverse all classes. Hence,
4:
j 1 Σ (xi, D)
approximate expectation in
Eq.
5:
d, + q (x 0)
6:
a; -- - (x) /))
7:
d. Vag
normalization
if de > then
we leverage the predictions of classi- fier models such as CLIP [23] and se- lect top-C' classes as the candidate labels
9:
Accept current y as correct label
10:
else
11:
iu qeo & acquire candidate labels
i... yored [argsort (x))] lo.c where
12:
for j E 0,,1, 1..1...... cdo
[argsort(-)o..c finds the index of top-C el-
13:
d, d(xx, 8: x v1 vrr"d) + arg max (d(2, ū: 2, 1
) [1
ements.
14:
Additionally, to ensure the MLLM output mapped certain probability space in order to provide unified measurement for every example, we conduct prompting to restrict the output only binary words, i.e., True or False. Moreover, our prompt leverages the ICL ability of MLLMs by providing both correct and incorrect matching exemplars. Our prompt is shown below:
15:
end for
16: end
17:
end for
Question: This image <IMG_labelee >> shows photo of < >, True or False? Answer: True;
Question: This image <IMG [ > shows photo of < >,
True or False? Answer: False;
Question: This image <I <IMG_ -uerr>> shows photo of <label#i True or False? Answer:
Specifically, for a query example x {IMG_ label4i whose label prediction from classifier i : [argsor((g we choose one image IMG i E Dew match with label label as a True exemplar. Moreover, we choose another image IMG_label E DD where j E {0..... ......... C} and j # as False exemplar. As shown in Huang al. [77], such prompt design can effectively teach MLLMs what kind of image-label combination is True or False. As result, the prediction qo for the query example is based on both the inherent knowledge of MLLMs and the demonstration provided by the prompt. Furthermore, our NoiseGPT restricts qe to binary values which significantly stabilizes the MILM outputs. As revealed by existing studies [77, 78], providing a set of class candidates and asking which one is the ground truth shows sub-optimal performance. The reason is that current MLLMS cannot effectively conduct multi-class classification and it gets easily confused when facing various choices. Therefore, we only require MLLMs to output binary prediction under demonstrative exemplars which unleashes the power of ICL and benefits making trustworthy inferences. We summarize our methodology in Algorithm 1, Next, we empirically validate the proposed NoiseGPT.
4 Experiments
In this section, we first introduce the specifics of our experiment setup. Then we validate the efficacy of our method through an investigation with regard to noise identification and rectification. Furthermore, we undertake quantitative analysis to demonstrate the enhancing effects of NoiseGPT as data cleansing method through comparing contemporary state-of-the-art LNI. models with their combined counterparts with our methodology. Finally, we conduct ablation studies to fully explore the performance of our approach.

---

4.1 Experiments setup
Datasets In our experiments, we leverages re-annotaeed noisy datasets CIFAR-ION and CIFAR- 100N [25] which contain real-world human annotation errors. We also generate noisy versions of CIFAR-I0, CIFAR-100 [24], Web Vision [3] and ImageNet IISVRR2212 for our studies. The details
are as follows:
CIFAR-ION and CIFAR-IOON [25]: We adopt Aggregate, Randl and Worst versions of CIFAR- ION whose noise rates are 9.03%. 17.23% and 40.21% respectively and Noisy-Fine version of CIFAR-10ON whose noise rate is 40.20%.
CIFAR-10 10 and CIFAR-100 [24]] The CIFAR-10 contains 50,000 labeled images 10 different classes, while CIFAR-100 contains 100 classes, each with 500 images. We mannually inject 20%, 50%, 80%, 90% symmetric and 40% asymmetric noise into CIFAR-10 and CIFAR- 100 respectively.
WebVision [3]: We utilize its validation subset which contains 50,000 images for the 40% symmetric noise condition. Moreover, order to verify the capacity of NoiseGPT under the real-world circumstance where samples are collected without careful annotation, we utilize mini- Senti--ebviiionn a subset of Webvision, for noise detection and rectification experiments and test the classification performance on the validation set of Webvision.
ImageNet ILSVRC2012: We utilizes the validation subset which contains 50,000 images and generate symmetric noise for 50% examples in it.
Models Primarily, we leverage original CLIP models [23] as our multi-classifier. For MLLM backbone, we employ MMICL [76] which adopts vision encoder of BLIP-2 [70] and FLAN-T5-XXL [79] as the LLM. For comparison with previous LNL works, we consider two methods Pro-Mix [59] and M-correction [35] which train an 18-layer PreAct Resnet for classification tasks.
Evaluation settings For exemplars that are used in the in-context learning process, select 3 images per category to construct tiny ground- truth support set {xe} to simulate the scarceness of examples in real-world condition. examples are also selected from this support set gen- erate perturbed sample features. Specifically, for each query sample, we construct perturbed features with different perturbing re- sources from (ra). For pseudo labels, we em- ploy the top C 3 prediction of CLIP to con- duct label rectification process.
3.
044
10
Appregane
02
kSVRC12
FPR
Figure 4: The noise detection ROC curves.
4.2 Performance analysis
In this section, aim to demonstrate the radical capability NoiseGPT de- tecting and rectifying label noise. Our experiments are conducted on 6 datasets: CIFAR- 10N Aggregate, Rand1, Worst, CIFAR- 10ON Noisy, Webvisin Sym. 40% and ILSVRC12 Sym. 40%. The results of classification and rectifica- tion are recorded evaluate the perfor- mance.
Table I: Noise detection and rectification performance.
Dataset
Noise type
AUROC
Rectification
Aggregate
0.8333
81.4%
CIFAR-ION
Rand1 0.8467
83.9%
Worst 0.8508
84.9%
CIFAR-10ON Webvision ILSVRC12
Noisy
0.7899
59.3%
Sym. 40%
0.8935
43.2%
Sym. 40%
0.9253
43.3%
Effectiveness of noise detection For evaluation metrics, we utilize Area Under the Receiver Operating Characteristic curve (AUROC) to reflect the noise detection performance. AUROC is commonly used to evaluate the performance of binary classification models. Generally, a higher AUROC score (closer to 1) indicates better discrimination ability, while 0.5 suggests random guessing. For label rectification performance, we compare corrected labels of query examples with their true labels to obtain rectification accuracy.
7

---

Table shows the performance of NoiseGPT in noise detection and rectification, the first column shows AUROC scores and the second shows the label correction accuracy. Figure 4 shows the AUROC curves. Our NoiseGPT achieves over 0.83 and 0.78 AUROC score on CIFAR-100 10N and CIFAR-1001 datasets. Especially for Webvision and IISSVRC1,, their score reaches over 0.89, which demonstrates the efficacy of filtering out noisy examples. In terms of rectification, NoiseGPT achieves over 80% accuracy on CIFAR-I00 ION datasets. sum up, NoiseGPT is evident to detect label noise and further recycle most of the noisy examples by rectifying noisy labels to ensure the dataset quantity and quality.
Comparison with baselines We compare the detection performance with the baselines in dinary-classffication manner with evaluation metrics such as Precision, Recall rate and score with baseline methods. Experiments are conducted on CIFAR-10 sym. 80% dataset, where instances are categorized into 4 types: true-clean, false-clean, true-noisy, and false- noisy. Table 2 shows the results. Note that the detection scores of baselines from their detection modules. And the hyperparameters NoiseGPT are fine-tuned to get the best scores.
Table 2: Detection performance comparison.
Method
Precision
Recall
F1
DivideMix*
94.36%
92.63%
93.49%
Proto-Mix
96.59%
93.78%
94.57%
NoiseGPT 97.80%
94.39%
96.07% %
4.3 Quantitative comparison
While NoiseGPT primarily focuses zero-shot noise detection and rectification, existing research Learning with Noisy Labels (LNL) has delved into training noise-robust Deep Neural Networks (DNNs) for classification tasks. To effectively showcase the advantages of our approach, we integrate NoiseGPT as a data cleansing method with two LNL baselines, namely Pro-Mix and M-correction, and BaaseGPT+Mcooretttin
constituting "NoiseGPT+Pro-MM
respectively. The classification models are trained under the same settings of as specified in papers of baselines.
Table 3: Noise rectification results.
Dataset
CIFAR-10
CIFAR- 100
Noise type
Sym.
Asym.
Sym.
Before
20%
50%
80%
90%
40%
20%
50%
80%
90%
After
7.4%
13.6%
19.3%
24.4%
9.4%
16.1%
28.0%
40.2%
44.6%
NoC
46279
43206
40335
37807
45282
41940
35985
29918
27716
Comparison with classic noisy labels Our experiments are conducted on the CIFAR-10 and CIFAR- 100 datasets, considering varying levels of symmetric and asymmetric noise. Table shows the noise reduction effects of NoiseGPT, which is capable of improving the clean proportion within datasets. Note that the last row shows the number of clean examples after rectification. The improvement is particularly substantial for CIFAR-I0 datasets with high noise rate. And for more challenging datasets of CIFAR- 100, NoiseGPT still keeps its effectiveness across varing noise conditions.
Subsequently, we transfer cleaned datasets into clas- sification training. We compare the performance of "NoiseGPT+" with their baseline counterparts and other representative works this field. Note that we re-produce the aforementioned Proto-Mix and M-correction. Due the lack of detailed training setting information proposed in their papers, some of the re-produced results are not as fine-tuned as what in papers. Nonetheless, we conduct experiments of "NoiseGPT+" with same hyperparameters with their baseline counterparts on each dataset. Thus they objectively demonstrate the effect of NoiseGPT. Table 4 shows experimental results. NoiseGPT poses enhancing effects to LNL. works in most noise conditions. Especially in high noise levels, the im- provement of classification accuracy increased by over 20% and 14% respectively for M-correction and Pro-Mix in CIFAR-10 10 Sym. 0.9.
Table 5: Classification on Webvision.
Method
Accuracy
F-correction
61.12
D2L.
62.68
Co-teaching
63.58
DivideMix
77.32
NoiseGPT+DividMMii
78.10
8

---

Table 4: Classification accuracy comparisons.
Dataset
CIFAR-10
CIFAR-100 100
Noise type
Sym.
Asym.
Sym.
Noise level
20%
50%
80%
90%
40%
20%
50%
80%
90%
Cross-Entronn
82.7
57.9
26.1
16.8
85.0
61.8
37.3
8.8
3.5
F-correction
83.1
59.4
26.2
18.8
87.2
61.4
37.3
9.0
3.4
Co-teaching+
88.2
84.1
45.5
30.1
64.1
45.3
15.5
8.8
Mixup
92.3
77.6
46.7
43.9
66.0
46.6
17.6
8.1
P-correction
92.0
88.7
76.5
58.2
88.5
68.1
56.4
20.7
8.8
Meta-Learning
92.0
88.8
76.1
58.3
89.2
67.7
58.0
40.1
14.3
M-correction*
93.7
92.0
87.6
68.7
91.5
68.6
59.4
47.3
12.9
89.2
93.7
92.6
91.5
92.8
67.0
64.3
58.1
39.4
Pro-Mix*
95.9
94.6
83.1
75.0
80.5
79.6
74.6
55.4
28.9
NoiseGPT+Pro-MMi
96.2
94.9
88.8
89.6
92.3
76.3
71.5
63.9
47.4
Comparison with real-world noisy labels In the real-world situation, some datasets prevailing recently are collected from the Internet, such as Webvision. Thus, they contain label noises that represent different patterns from the symmetric. In this paper, we utilize mini- SeriiiWebviionn, smaller subset of Webvision verify the capacity of NoiseGPT combat label noises of this kind. Due to the lack of ground-trute truth annotations, we directly compare the classification performance with baselines. Following previous works [13], we use the Inception-ResNe v2 [80] as the classification backbone. The The results in Table 5 demonstrate that NPi remains effective real-world noisy datasets like Webvision. The "NoiseGPT4" denotes that the training is first cleaned by NoiseGPT.
4.4 Sensitivity study
Effect of perturbation number Since approximate the expectation in Eq. 2 with a sequence of perturba- tions, theoretically increasing the per- turbation number 7 will make the nor- malized ICD score more robust and effective in distinguishing clean and noisy examples. However, there is
83
conrection actuacy
68
marginal effect when increases to an extent and becomes computationally unworthy. Figure 5 shows the trend of noise rectification performance un- der changing hyperparameter n in attachment. In our experiments, we select the number of perturbations order to balance the computational cost and performance.
58
53
12
number of perturbation
CIFAR-10 Syz
CIFAR-I 100 Sym.809
Figure 5: Trend of performance under changing perturbation number.
4.5 Ablation study
The existence of curvatures different datasets We further demonstrate the quality of possibility curvature by conducting experiments where query examples from noisy dataset are augmented by series of perturbed exemplars with varying perturbation strength. According to Section 3.1, control the perturbation strength by adjusting token-wise MoF weight w, a larger w indicates higher proportion of information coming from query example. We conduct this experiment on four datasets: CIFAR-10 Sym. 50%, CIFAR-100 Sym. 50%, Webvision Sym. 40%, and ILSVRC Sym. 40%.
Figure 6 shows the curvatures of output Softmax probability under changing MoF weights. We calculate the averaged qe (x) of clean and noisy examples in each dataset respectively. It investigated that on for clean examples, the output Softmax probability ggleam(s) (x:) tends to descend as the MoF weight decreases. Conversely however, the curvature of noisy examples fluctuates optionally

---

CIFAR Sym.
CIFAR-100 Sym.
Webvision
ILSVRC12 Sym.
dlaan
ICD score
Perturbation strength
Figure 6: Output possibility curvatures of clean and noisy examples under different perturbation strength.
unconcerned with the change weight w. This phenomenon is confirmed all datasets, firmly backing our method, NoiseGPT, which utilizes the discrepancy of output Softmax probability (x) under perturbation distinguish between clean and noisy examples.
Clean classes that are easier to be considered noisy At- tributing to the fact that clean and noisy examples have differ- ent ICD score distributions, our NoiseGPT capable detect and rectify noise. However, the ICD score distribution does not remain unchanged among differ- ent categories. Some categories of clean examples tend to have relatively higher ICD scores than others, which, in other words, are easier to be mistaken as noisy during the process of NoiseGPT. Figure 7: Clean categories that are easier to be mistaken as noisy. We investigate such categories different datasets, their average ICD scores recorded indication of what extent they tend to be mistaken. Note that for clean examples, lower score indicates easier to be mistaken. Figure 7 shows which classes easy be mistaken for noisy. Further exploration of detection biases are provided in Section 5 in our Appendix.
0958
344
D.70
25
aperiistt
ICD
0.068
0:004
Softcc
CHAAR-10
CIFA-100
Webvision
ILSVRICI2
Conclusion
Contribution In this work, we propose a novel label noise solution via leveraging MILMS as experts to reduce and recycle noisy instances in datasets. Specifically, we investigate the probability curvature of MILMs under input perturbation. Through a token-wise Miture-of-Featte can calculate ICD scores of input examples and divide them into clean and noisy. By conducting extensive quantitative and qualitative experiments on different datasets, our method is validated to sustain effective over varying noise conditions. Moreover, it surpasses previous LNL methods in noise detection and poses substantial potential to cope with other deep learning models improve their performance. In the future, method can be further explored for insights into more probelms like OOD detection, weakly-supervised learning, etc.
technique,
Limitation Despite its adaptability to various datasets and noise levels, the performance of NoiseGPT constrained by the capabilities of the underlying machine expert relies on. Re- search [75] has highlighted the bottleneck effect in vision models within MLLMs. Furthermore, the instruction -iaructioon-ollowinng capability of MLLMs significantly influences the distributions of ICD scores, which closely tied to the confidence of MLLM answers. Enhancing the proficiency of these machine experts can lead to improved performance of NoiseGPT in its tasks.
10

---

References
[1] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-252, 2015.
[2] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitmick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[3] Wen Li, Limin Wang. Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint arXiv:1708.000 02008,02822.222 2017.
[4] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten, Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV), pages 181-196, 2018.
[5] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3): 2021.
[6] Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using noise adaptation layer. In International conference on learning representations, 2016.
[7] Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In Proceedings of the IEEE conference computer vision and pattern recognition. pages 5552-5 5560, 2018.
[8] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1944-1952, 2017.
[9] Hongxin Wei, Huiping Zhuang, Renchunzi Xie, Lei Feng, Gang Niu, Bo An, and Yixuan Li. Logit clipping for robust learning against label noise. 2022.
[10] Hwanjun Song, Minseok Kim, and Jae-Gil Lee. Selfie: Refurbishing unclean samples for robust deep learning. In International conference on machine learning, pages 5907-5915, PMLR, 2019.
[11] Eran Malach and Shai Shalev-Shwartz. Decoupling" when to update" from" how to update". Advances in neural information processing systems, 30, 2017.
[12] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentomet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International conference on machine learning, pages 2304-22 2313. PMLR, 2018.
[13] Junnan Li, Richard Socher. and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. arXiv preprint arXiv:2002.00 07394, 2020.
[14] Haobo Wang, Ruixuan Xiao, Yiwen Dong, Lei Feng, and Junbo Zhao. Promix: Combating label noise via maximizing clean sample utility. arXiv preprint arXiv:2207. 10276, 2022.
[15] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to cuni-suprrissed learning. Advances in neural information processing systems, 32, 2019.
[16] Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018
[17] Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagrecment help generalization against label corruption? In International conference on muchine learning, pages 7164-7173. PMLR, 2019.
[18] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language language understanding and generation. In International conference on machine learning, pages 28888-12000 PMLR, 2022.
[19] Wenliang Dai, Junnan Li, Dongxu Li. Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-pupose vision- vision-language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
11

---

[20] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[21] Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. Mmiel: Empowering vision-language model with multi-modal in- context learning. arXiv preprint arXiv:2309.077 2023.
[22] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengeheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. 14178, 2023.
[23] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[24] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[25] Jiaheng Wei, Zhaowei Zhu, Hao Cheng. Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real- world human annotations. arXiv preprint arXiv:2110.1208, 2021.
[26] Virginia Wheway. Using boosting to detect noisy data. In Advances in Artificial Intelligence. PRICAI 2000 Workshop Reader: FourWorkshaps held at PRICAI 2000 Melbourne, Australia, August 28-September 1. 2000 Revised Pupers 6, pages 123-130. Springer, 2001.
[27] Borut Sluban, Dragan Gamberger, and Nada Lavrač. Ensemble-based noise detection: noise ranking and visual performance evaluation. Data mining and knowledge discovery, 28:265-303, 2014.
[28] Sarah Jane Delany, Nicola Segata, and Brian Mac Namee. Profiling instances in noise reduction. Knowledge- Rased Systems, 31:28-40 2012.
[29] Dragan Gamberger, Nada Lavrac, and Saso Dzeroski. Noise detection and climination in data preprocessing: experiments in medical domains. Applied artificial intelligence, 14(2):205 223, 2000.
[30] Jaree Thongkam, Guandong Xu, Yanchun Zhang, and Fuchun Huang, Support vector machine for outlier detection in breast cancer survivability prediction. In Advanced Web and Network Technologies, and Applications: APWeb 2008 International Workshops: BIDM, IWHDM, and DeWeb Shenyong, China, April 26-28, 2008, Revised Selected Papers 10, pages 99-109. Springer, 2008.
[31] Deepak Ravikumar, Efstathia Soufleri, Abolfazl Hashemi, and Kaushik Roy. Unveiling privacy, memoriza- tion, and input curvature links. arXiv preprint (rXiv:2402.18724 2024.
[32] Isha Garg. Deepak Ravikumar, and Kaushik Roy. Memorization through the lens of curvature of loss function around samples. arXiv preprint Xiw:22307.05831. 2023.
[33] Sheng Liu, Jonathan Niles Weed, Narges Razavian, and Carlos Fernandez-Granda. Early-learning reg- ularization prevents memorization noisy labels. Advances neural information procexsing systems. 33:20331-20342, 2020.
[34] Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rahinovich. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint
2014.
[35] Eric Arazo, Diego Ortego, Paul Albert, Noel O' O'Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In International conference on machine learning, pages 312-321. PMLR, 2019.
[36] Ziming Hong, Zhenyi Wang. Li Shen, Yu Yao, Zhuo Huang, Shiming Chen, Chuanwu Yang. Mingming Gong, and Tongliang Liu. Improving non-transferable representation learning by harnessing content and style. In The Twelfth International Conference on Learning Representations, 2024.
[37] Zhuo Huang, Muyang Li, Li Shen, Jun Yu, Chen Gong, Bo Han, and Tongliang Liu. Winning prize comes from losing tickets: Improve invariant learning by exploring variant parameters for diitt--iisttibutton gencralization. International Journal of Computer Vision, pages 1-19, 2024.
[38] Yexiong Lin, Yu Yao, Xiaolong Shi, Mingming Gong, Xu Shen, Dong Xu, and Tongliang Liu. Cs-isolate: Extracting hard confident examples by content and style isolation. Advances in Neural Information Processing Systems, 36, 2024.
12

---

[39] Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence, 38(3)4447-461, 2015.
[40] Dan Hendrycks, Mantas Mazeika, Dunean Wilson, and Kevin Gimpel. Using trusted data train deep networks on labels corrupted by severe noise. Advances in neural information processing systems, 31, 2018.
[41] Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? Advances in neural information processing systems, 32, 2019.
[42] Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang. Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. Advances in Neural Information Processing Systems, 33:7597-7610,, 2020.
[43] Haw-Shiuan Chang, Erik Learned-Miller and Andrew McCallum. Active bias: Training more accurate neural networks by emphasizing high variance samples. Advances in Neural Information Processing Systems, 30, 2017.
[44] Nikola Konstantinov and Christoph Lampert. Robust learning from untrusted sources, In International conference on machine learning. pages 3488-3498 3498. PMLR, 2019.
[45] Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff Bilmes, Gopinath Chennupati, and Jamal Mohd- Yusof. Combating label noise in deep learning using abstention. arXiv preprint arXiv:1905. 10964, 2019.
[46] Guoqing Zheng, Ahmed Hassan Awadallah, and Susan Dumais. Meta label correction for noisy label learning. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11053-11061 2021.
[47] Haoliang Sun, Chenhui Guo, Qi Wei, Zhongyi Han, and Yilong Yin. Learning to rectify for robust learning with noisy labels. Puttern Recognition, 124:108467, 2022.
[48] Yanyao Shen and Sujay Sanghavi. Learning with bad training data via iterative trimmed loss minimization. In International Conference on Machine Learning, pages 5739-5748. PMLR, 2019.
[49] Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In International conference on machine learning, pages 1062-1070. PMLR, 2019.
[50] Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, and Masashi Sugiyama. Sample selection with uncertainty of losses for learning with noisy labels. arXiv preprint
2021.
[51] Devansh Arpit. Stanislaw Jastrzębski. Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kan- wal, Tegan Maharaj. Asja Fischer, Aaron Courville. Yoshua Bengio, al. A closer look memorization in deep networks. International conference on machine learning, pages 233-242, PMLR, 2017.
[52] Hwanjun Song, Minseok Kim, Dongmin Park, and Jae- Gil Lee. How does early stopping help generalization against label noise? arXiv preprint Xi1::1911.0005) 2019.
[53] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018
[54] Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In International conference on machine learning, pages 7164-7173. PMLR, 2019.
[55] Zhuo Huang, Chao Xue, Bo Han, Jian Yang, and Chen Gong. Universal semispurried leammnn in Neural Information Processing Systems, 34:26714-2666 2021.
learning, Advances
[56] Due Tam Nguyen, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. Self: Learning to filter noisy labels with self-onsemblingg arXiv preprint 2019.
[57] Anti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight- consistency targets improve semi- supi-vipe deep learning results. Advances in neural information processing systems. 30, 2017.
13

---

[58] Mingyu Li. Tao Zhou. Zhuo Huang, Jian Yang. Jie Yang. and Chen Gong. Dynamic weighted adversarial learning for semi-supervised classification under intersectional class mismatch. ACM Transactions on Multimedia Computing, Communications and Applications, 20(4): 1-24, 2024.
[59] Haobo Wang, Ruixuan Xiao, Yiwen Dong, Lei Feng, and Junbo Zhao. Promix: Combating label noise via maximizing clean sample utility. arXiv preprini arXiv:2207.10222 10276, 2022.
[60] Tianyi Zhou, Shengjie Wang, and Jeff Bilmes, Robust curriculum learning: from clean label detection to noisy label self-correction. In International Conference on Learning Representations. 2020.
[61] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1990 1901, 2020.
[62] Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:68 30:681-694, 2020.
[63] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmidt. Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv: 087774 2023.
[64] Hugo Touvron, Thibaut Lavril, Gautier Iracard, Xavier Martinet, Marie-Anne I. Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal. Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv: 2222322211997 13971, 2023.
[65] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra. Prajjwal Bhargava, Shruti Bhosale, et Llama Open foundation and tine-tuned chat models. arXiv preprint arXiv:2307 092888, 2023.
[66] Deyao Zhu, Jun Chen, Xiaogian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.1550/ 2023.
[67] lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716 23736, 2022.
[68] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint XrXiv:2010. 1999, 2020.
[69] Junnan Li. Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- pre-training unified
understanding and generation. In International conference on machine learning. pages I 12888-128 12900. PMLR, 2022.
[70] Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language -aaagg---igeeeee tre-training with frozen image encoders and large language models. In International conference on machine learning. pages 19730-19742. PMLR, 2023.
[71] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[72] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-languape models with instruction tuming. Advances in Neural Information Processing Systems, 36, 2024.
[73] Qinghao Ye. Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengeheng Shi, Yaya Shi, al. mplug-owl: Modularization empowers large language models with multimodality, arXiv preprint arXiv:2304. 14178, 2023.
[74] Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. Detectgpt: Zero-shot -eeeeiategeeertted text detection using probability curvature. In International Conference on Machine Learning, pages
PMLR, 2023.
[75] Shengbang Tong. Zhuang Liu, Yuexiang Zhai, Yi Ma. Yann LeCun, and Saining Xie, Eyes wide shut? exploring the visual shortcomings of multimodal lms. arXiv preprint 2rXiv:440..06209 2024.
[76] Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang. Wenjuan Han, and Baobao Chang. Mmiel: Empowering vision-language model with mudti- in- context learning. arXiv preprint 20Xiv22309.0795. 2023.
14

---

[77] Zhuo Huang, Chang Liu, Yinpeng Dong, Hang Su, Shibao Zheng, and Tongliang Liu. Machine vision therapy: Multimodal large language models enhance visual robustness via denoising in-context learning. arXiv preprint arXiv:2312.02544 2023.
[78] Yuexiang Zhai, Shengbang Tong. Xiao Li. Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma. Investigating the catastrophic forgetting multimodal large language models. arXiv preprint arXiv:2309.1033 2023.
[79] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, al. Scaling instruction-fietuneed language models. Journal of Machine Learning Research, 25(70): 53, 2024.
[80] Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander Alemi. Inception-v4, inception-resace resnet and the impact of residual connections on learning. In Proceedingy of the AAAI conference on artificial intelligence, volume 31, 2017.
15

---

Appendix
In this section, first introduce implementations details and computational usage of our experiment. Then, we demonstrate more supplementary experiments to fully explore the qualities of NoiseGPT.
Implementation details
Among all hyperparameters, MoF weight w, number of perturbation emplars n and number of candidate classes C' are of most significance. Theoretically, larger n avails the ap- proximation in Eq. and for datasets with more complicated classes larger C' is suitable search for appropriate label. Table 6 shows hyperparameters pf our NoiseGPT experiments. Note that in some of the experiments, we fine-tune the threshold for better noise detection performance.
Table 6: NoiscGPT hyperpaamtt.......eeettt....
Hyperparameters
CIFAR-ION
MoF weight
0.5
Number of exemplars per class
3
Number of perturbations per query
10
Threshold
0.7
Number of candidate labels
3
Compute resources
Our noise detection and rectification experiments of NoiseGPT are powered by GeForce RTX 4090, taking up about 28.5 GiB memory in total for CIFAR datasets. The runtime (hours) experiments recorded under
10, C 3, 0.5, and illustrated in Table 7.
Table 7: Runtime of NoiseGPT.
Dataset
CIFAR-10
CIFAR-100
Noise type
Sym.
Asym.
Sym.
Noise level
20%
50%
80%
90%
40%
20%
50%
80%
90%
Runtime (hour)
67.5
94.2
116.4
125.8
88.3
70.2
103.7
126.5
130.9
More classification training performance
In Section 4.3, we have compared the classification performance of "NoiseGPT+" and baselines on CIFAR- Sym/Asym. datasets. Here we expand this experiment on more datasets: CIFAR- -ON Worst and CIFAR- 100N Noisy. Table shows the effects of NoiseGPT as a dataset cleansing method. And Table shows the results of classification accuracy.
Table 8: Noise rectification results of CIFAR-N datasets.
Dataset
CIFAR- 10N
Noise type
Aggregate
Rand1
Worst
Noisy
Noise Rate (before)
9.03%
17.23%
40.21%
40.20%
Noise Rate (after)
7.92%
8.07%
14.84%
28.45%
Clean number
46038
45963
42578
35774
Table 9: Classification accuracy on CIFAR-N datasets.
Dataset
CIFAR-1ON
CIFAR-10ON
Noise type
Aggregate
Randl
Worst
Noisy
M-correction
95.12
94.81
86.09
64.19
NoiseGPT-MMcorrctiin
95.16
94.79
90.36
68.20
ProtoMix
97.95
97.17
95.73
72.84
NoiseGPT+ ProtoMix
97.66
97.48
96.90
73.24
More noise detection and rectification performance
16

---

In section 4.2 we intro- duce AUROC and Label Correction Accuracy as the evaluating metrics for NoiseGPT noise detection and rectification perfor- mance. In order to better understand the improving impacts NoiseGPT poses to classification tasks, we further exhibit the evaluat- ing results of NoiseGPT on CIFAR-10/100 Sym./Asm.. datasets. Table 10 shows the noise detection and rectification results with AUROC score and correction accuracy. Figure shows the ROC curves of noise identification.
Table 10: NoiseGPT performance on CIFAR datasets.
Dataset
Noise type
AUROC
Correction accuracy
Sym. 20%
0.9175
82.8%
Sym. 50%
0.9203
83.4%
CIFAR-10
Sym. 80%
0.9225
84,2%
Sym. 90%
0.9130
84.0%
Asym. 40%
0.8183
84.9%
Sym. 20%
0.8949
56.4%
Sym. 50%
0.8969
59,9%
CIFAR-100
Sym. 80%
0.8903
61.5%
Sym. 90%
0.8848
61.9%
0.6
TPR
CIFAR-10 Sym. 0.2
CIFAR-10 Sym. 0.5
CIFAR-10 Sym. 0.8
CIFAR-10 Sym. 0.9
CIFAR-10 Asym. 0.4
CIFAR- 140 Sym. 0.2
CIFAR-100 Sym. 0.5
CIFAR-100 Sym. 0.8
CIFAR 100 Sym. 0.9
0.4
0.6
0.8
1.0
FPR
Figure 8: The noise detection ROC curves of CIFAR datasets.
Noisy classes that are easier to be considered noisy
In Section 4.5, illustrate classes that clean but easier be mistaken for noisy. Similarly, some categories of noisy examples are easier to be considered clean, obtaining lower ICD scores than their fellows. Contrary clean classes, higher score for noisy examples indicate that they easy be mistaken.
Table 11: Comparison detection bias
Method
0
2
3
5
6
7
8
9
Var
Proto-Mix *
9.8
11.0
9.5
7.8
10.1
8.2
10.6
11.0
10.8
11.1
1.2
NoiseGPT
10.4
10.7
9.0
10.6
10.9
9.0
10.8
8.5
10.0
10.2
0.68
Comparisons on the detection biases with the baseline
The experiments in Section 4.5 indicate that there are essentially biases in the noise detection stage of NoiseGPT, which will lead unbalanced example quantities of different classes after rectification. Similar phenomenon also appears in the basclines. Thus, here we compare the biases in noise
17

---

10
14
ICD score
0.681
beach
patio 0-95--055 0:552
77777
0.740
303
diplons
94
0.501
truck 0220
0.188
CIFAR-1D
CIFAR-100
Webvision
ILSVRC12
Figure Noisy categories that are easier to be mistaken as clean.
detection between NoiseGPT and Mix, and calculate the proportions of 10 classes in selected clean data on CIFAR- 10 sym. 90%. Table shows the results. Our NoiseGPT exhibits significantly lower variance in the example distributions across different classes, indicating reduced bias in noise detection and rectification compared Proto-Mix.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In Abstract and Section I, we discuss about previous LNL methods, and the utilization of MLLMs in work to detect and rectify noisy labels.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Section 5, we point out that our method is constrained by the performance of visual models, as their capability to extract prioritized information limit our NoiseGPT to work on complicated large-revolution datasets. Besides, the effects of prompt settings is also mentioned.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
19

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We mainly propose the assumption of MLLM output probability curvature in Section 3, guiding the definition of ICD criteria, which is further proved with experiments in Section 3.1. The results of experiments in Section 4.5 also back up this theory.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Our method is based on the zero-shot inference of backbone models. Following the experiment settings and hyperparameters mention in Section 4.1, all results of NoiseGPT can be reproduced. As for comparison baseline works, we follow the settings in their original codes and papers, and re-produce some of the results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
20

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: So far have not got ready for the open of our codes, due to time constraint. We will consider it latter.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
public/gun Soaidee//ooeuuumiseeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//ipp. wttpccccpppcccccctccc.cccc.cclct
httppppwwwuuupp....ppppppppppaap guides httpa////guideee//eeuuuubbiiinia /wwo///ooossssiieeeeee ssionnolccc)) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We introduce the effects of hyperparameters in Section 3 and make it clearly demonstrated in our experiment settings in Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to the lack of computational resources, we are not able to conduct statistical researches.
Guidelines:
21

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We introduce the running environment of our experiments and reference execution time of NoiseGPT on different datasets in the Appendix Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /nerrpps...pp/ wwtppblcc//eeecccccccccccceceeee htt.oo/iie/tiiicuiieeeliliiee
Answer: [Yes]
Justification: We fully understand the NeurIPS codes of ethics provided. And we will make it ensured from harmful consequences.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
22

---

Justification: In Section 5 we discuss the positive impacts our work may have on the future researches in the fields noisy labels and machine learning.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
There is no risk our released work could have posed.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual- should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We state the reference of all the models and data clearly in our paper, which can be seen as cites in the main content and detailed in Reference.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
23

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work does not contain such new assets that includes datasets or model frameworks.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work involves no crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work involves no crowdsourcing nor research with human subjects.
Guidelines:
24

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "zfT7Bef27a", "reviewer": "Reviewer_TkfP", "summary": "This paper employs Multimodal Large Language Models to detect noise samples. The In-Context Discrepancy is utilized to quantify the discrepancy between the original and perturbed samples. Additionally, the identified noise samples are integrated with the Pro-Mix and M-correction noise label learning framework. Furthermore, a comparison between the proposed method and existing approaches is conducted.", "questions": "The efficacy of the proposed method is questionable, as Table 3 shows only a limited improvement compared to Pro-Mix*.\n\nThe author is recommended to include a section on \"Sample cleaning\" in the related work to provide a comprehensive coverage of relevant studies.", "limitations": "YES.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "Strengths: 1) The author utilizes Multimodal Large Language Models to detect noise samples.", "weaknesses": "Weakness;\n1) The novelty of the approach is limited. \n2) Some previous works are not properly cited, such as the lack of citation for In-Context Discrepancy. \n3) Several highly relevant works are missing, such as those measuring the In-Context Discrepancy between original and perturbed samples, including recent publications.\n\n[1] Early-learning regularization for preventing memorization of noisy labels. \n[2] Strategies for preventing continuous damage caused by noise during model training.\n [3] A survey on learning from noisy labels with deep neural networks. \n[4] Learning to rectify for robust learning with noisy labels. \n[5] ...\n\n4) The efficacy of the proposed method is questionable, as Table 3 shows only a limited improvement compared to Pro-Mix*.\n\n\nSome typos:\n\"Figure ?? \"\n\n\"50%„ \"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments.\n- Limited novelty: \n  - We make the first contribution to effectively detect and rectify label noise without the need for pre-training, providing a novel direction for learning with label noise, which has never been previously studied (**Reviewer 81A3**), and has the potential to be widely adopted (**Reviewer gxfG**). We hope the Reviewer could further provide on which aspect he thinks our novelty is limited, and we would be happy to try our best to address them.\n\n- Missing previous works & several relevant works.\n  - We will carefully cite all the mentioned works and discuss them in detail in the related works section. Should there be more references to consider, please feel free to let us know, thanks.\n  - We would like to stress that the In-Context Discrepancy is a novel measuring score that is based on contextual prompt information, which can guide and formulate the output. But other measuring score mentioned by the reviewer is mostly based on classification probability. Therefore, our measurement is novel and different from existing works. We will clarify this in the future version.\n\n- Typo.\n  - Thanks for pointing them out, we will carefully address all of them.\n\n- Efficacy.\n  - By comparing to other well-known baseline methods, our method can further enhance the learning performance in most scenarios.\n  - Only under CIFAR-100 dataset with low noise rate, such as 20\\% and 50\\%, the performance of NoiseGPT shows degradation. This is because the performance of NoiseGPT highly relies on the noise detection threshold, which is set fixed as 0.72 throughout Table 3. In fact, under low noise rates, the detection threshold should be higher than that of high noise rates, in order to prevent overconfidently selecting clean examples as noisy ones. Our experimental results in section 4.4 validate such a claim, and we further provide an ablation study in Figure D in the attachment file of the author rebuttal to show that when setting proper threshold, the performance of NoiseGPT can still be effective compared to Pro-Mix.\n  - In challenging scenarios under high noise rates, NoiseGPT is quite effective in achieving 8.5\\% and 18.5\\% performance improvement compared to Pro-Mix under 80\\% and 90\\% noise rate, respectively.\n\n- Related work on Sample Cleaning.\n  - We have carefully provided the discussion of related works as mentioned in the author rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "snmTfyaGxv", "reviewer": "Reviewer_gxfG", "summary": "The paper introduces NoiseGPT, a method that leverages Multimodal Large Language Models (MLLMs) for label noise detection and rectification in datasets. The approach exploits the probability curvature effect observed in MLLMs, where clean and noisy examples exhibit different curvature smoothness under perturbation. NoiseGPT uses a token-wise Mixture-of-Feature (MoF) technique to generate perturbations and an In-Context Discrepancy (ICD) measure to detect label noise and rectifies them. Experiments demonstrate the method's effectiveness in improving dataset quality and enhancing classification performance in various noisy datasets.", "questions": "1. How sensitive is the method to these hyperparameters? The paper could benefit from an analysis of the sensitivity to hyperparameters like the number of perturbations $n$, and the threshold $\\tau$ and more details of how they were chosen.\n\n2. The paper mentions the influence of prompt settings on the reliability of ICD scores. Can the authors elaborate on how different prompt designs impact the performance of NoiseGPT? \n\n3. Is curvature only relevant in MLLM models? I.e. can the baseline be a classification model and do curvature scores in that case perform similarly (see [1]) for mislabeled sample identification?\n\n4. My understanding suggests that the method scales poorly to large datasets since multiple perturbations are needed on MLLM input for rectification. The authors should elaborate on this aspect, discussing potential strategies to mitigate the high computational cost and improve scalability. For instance, are there ways to reduce the number of perturbations required while maintaining detection accuracy? Can the method be parallelized more effectively? Thus the compute requirement and the number of perturbations $n$ and how it affects performance needs discussion.\n\n[1] Garg, et al. \"Memorization through the lens of curvature of loss function around samples.\" arXiv preprint arXiv:2307.05831 (2023).", "limitations": "1. The paper adequately addresses some limitations, such as the reliance on MLLMs' vision capabilities and the effect of prompt settings on ICD scores. However, the computational cost associated with the method is not fully discussed. Since NoiseGPT requires generating multiple perturbations (n perturbations) for each example, this can lead to significant computational overhead, particularly when dealing with large datasets. This overhead may result in slow processing times, as the models involved (e.g., CLIP, large language models) are computationally intensive.\n\n2. It is not clear if the curvature properties leveraged in this paper are restricted to MLLMs, the paper does not provide any evidence for or against it. I.e. do smaller models behave differently or the same?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents a new application of MLLMs for label noise detection and rectification, introducing the concept of probability curvature and leveraging the zero-shot capabilities of models like CLIP for label rectification.\n\n2. The paper includes experiments across multiple datasets, demonstrating the effectiveness of NoiseGPT in both detecting and rectifying label noise.\n\n3. The paper is well-organized and clearly written. The methodology, including the MoF technique and ICD measure, is explained in detail. The inclusion of algorithm pseudocode and detailed experiment settings enhances reproducibility.\n\n4. The results show significant improvements in label noise detection and rectification, which is crucial for training robust deep learning models. The approach has the potential to be widely adopted", "weaknesses": "1. Probability curvature has been previously explored in [1, 2] and [1] specifically for mislabeled detection in vision models, thus reducing the significance of the contribution in the paper, however, one should note [1,2] that these works are limited to vision classification models only. It might warrant discussion in the related work section.\n\n2. It is not clear if the curvature properties leveraged in this paper are restricted to MLLMs, the paper does not provide any evidence for or against it. I.e. do smaller models behave differently or the same? I.e. is a `large' model needed?\n\n3. The paper has missed some similar works, such as [1] Garg et al. (2023) and [2] Ravikumar et al. (2024), which also explore curvature in vision models and mislabelled sample detection. Thus the discussion of these works in the related section should be considered.\n\n4. There needs to be a discussion on the scalability of the method to large-scale datasets, the results in the paper use small subsets of ImageNet and Webvision. Since multiple perturbations are needed on MLLM to detect and correct for label noise, can this method scale to millions and billons of images?\n\n\n\n[1] Garg, et al. \"Memorization through the lens of curvature of loss function around samples.\" arXiv preprint arXiv:2307.05831 (2023). \\\n[2] Ravikumar, et al. \"Unveiling privacy, memorization, and input curvature links.\" arXiv preprint arXiv:2402.18726 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your suggestions. \n\n- Relationship to previous loss curvature works:\n  - The curvature of loss function is a common phenomenon in deep networks, existing works have studied in some fields but none of them have been conducted in LNL. Our work is the first to discover the effectiveness of learning curvature to benefit LNL, which can establish our contribution and novelty in this field.\n  - Moreover, existing curvature studies are based on the classification probability, but our ICD measure considers the knowledge of in input prompt by carefully leveraging the in-context learning ability, which can benefit the effectiveness of the MLLM identification and rectification.\n  - Furthermore, our ICD score does not need pre-training to be used, we only need to conduct zero-shot inference to implement this score, which is much more efficient and straightforward.\n\n- Specification of model for curvature properties:\n  - Our framework which leverages the ICL capability can only be implemented using MLLMs. Maybe in the future when vision models start to show ICL ability, we can also implement NoiseGPT to them.\n  - Although the ICD curvature is restricted to MLLMs, smaller models of this kind also share this property, but with a drop in performance. For comparison, we adopt FLAN-T5-XL as another LLM backbone which has a size of 3B parameters and is smaller than FLAN-T5-XXL (11B). The performance of reducing noise is illustrated in Table A in the attachment file of the author rebuttal.\n\n- Missing references and discussions of the related works:\n  - Thanks for pointing them out, we will complement all the mentioned references, and a discussion is added in the related work section in our author rebuttal due to the character limitation here.\n\n- Discussion on the scalability to large-scale datasets:\n  - The computational cost required for perturbation will not increase as the scale of datasets grows. We select a few exemplars from each class in the dataset to construct a subset D^ex at first. All perturbations of subsequent MLLM input query examples are injected from this subset, and no more exemplars are needed when the scale of the dataset grows.\n  - The computing time of the proposed method grows proportionally to the number of examples in a dataset. Thus, when faced with datasets of a much larger scale, we suggest that adjust the hyperparameter $n$ to reduce the number of perturbations for each input example. As the computing time also shrinks proportionally with the decrease of $n$, since $n$ determines how many times MLLM needs to generate an answer to produce ICD scores. By selecting a relatively smaller $n$, the performance of NoiseGPT can be maintained adequately while saving much of computing time. \n\n- Parameter sensitivity analysis & Effect of perturbation numbers and how to reduce it for efficiency:\n  - Threshold $\\tau$: To further explore the influence of threshold on the overall noise mitigation, we make experiments on CIFAR-100 sym. 20% and 50% with varying τ. Besides, as $\\tau$ only affects the noise detection of NoiseGPT, which is a binary classification scenario. We adopt the ROC curve and AUROC score to better evaluate changes in detection performance along with different thresholds, which reflects the changes in TPR and FPR as the threshold moves from the smallest to the largest. By analysing the ROC curve of a small subset of query examples, a suitable threshold can be inferred for the whole dataset.\n  - Number of perturbations $n$: Theoretically, more perturbations make the normalized ICD score more robust and effective in distinguishing clean and noisy examples. However, there is a marginal effect when $n$ increases to an extent when it becomes computationally unworthy. In Figure A in the attachment, the sensitivity of hyperparameter $n$ is explored. In practice, we select the number of perturbations $n$ by balancing the computational cost and performance. \n\n- Prompt design: We based our prompt settings on previous works [1] and made some modifications, including the sequence and the content.\n  - The proposed prompt includes three sentences, a positive Q&A, a negative Q&A, and a query question whose answer is to be generated. In earlier experiments, we changed the order of the first two sentences which were negative, affirmative, and query. It was observed that MLLM tended to output a lower Softmax possibility of answering “true” for all query examples according to the regularity in former lines of prompts. Thus, we bypass this problem using the ICD score which depends on the relative magnitude of output possibilities between original and perturbated examples, instead of the absolute possibilities of “true” or “false”. \n  - In experiments, we extended the prompts to different sizes in the same format shown in Sec. 3.2. And different sequences of these sentences were also investigated. It turned out that the tendency mentioned in the last paragraph was facilitated.\n  - To completely mitigate the influence of prompts, we assume that the representation editing [2,3] methods could have possibly been a substitution for existing prompt learning methods. We are still working on this idea.\n\n- Baseline using learning curvature with classifier model:\n  - The loss function produced by classification models is related to the discrepancy between the prediction $y’$ and label $y$, which are essentially different from ICD. As a result, it would have been complicated and counterintuitive to investigate in probability changes of DNNs induced by perturbation.\n\n[1] Huang Z, Liu C, Dong Y, et al. Machine vision therapy: Multimodal large language models can enhance visual robustness via denoising in-context learning[C]\n[2] Liu, S., Ye, H., Xing, L. & Zou, J. In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering.\n[3] Kong, L. et al. Aligning Large Language Models with Representation Editing: A Control Perspective.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D0clsTCNgX", "reviewer": "Reviewer_81A3", "summary": "This paper proposes to use large multimodal models (LMM) to detect noisy annotations in image datasets. The NoiseGPT method consists in observing whether a controlled perturbation in the latent representation of the LMM leads to a large modification of the LMM response. Experiments in the paper demonstrate that noisy samples are more sensitive to input perturbations than their clean counterparts. This discrepancy is used to detect noisy samples.\n\nExperiments are proposed where NoiseGPT is used in conjunction with existing noise robust algorithms to correct noise on synthetically corrupted datasets as well as the real world CIFAR-N datasets. When NoiseGPT is used, the classification accuracy of datasets is improved.", "questions": "Why are the experiment settings on Webvision different from previous research ?\n\nDid the authors observe that NoiseGPT struggles to detect web-noisy samples more because they come from the same distribution as the data LMMs are trained on ? (see my last weakness point)", "limitations": "NoiseGPT requires a lot more compute than previous approaches to detect noisy samples.\nThis has been adequately evidenced in the paper (Table 6)", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Using LMMs to detect noisy annotations sound next step in using large models for data curation that to my knowledge has not been previously studied.\n\nThe observation about the sensitivity of noisy samples to noise in the latent space is intuitive and motivates the approach well.\nNoiseGPT going beyond simply prompting the MLM to answer whether the label is clean or noisy strengthens the contribution.\n\nWhen coupled with existing noise-correction algorithms, the NoiseGPT strategy appears to be beneficial to the test accuracy (Tables 3 and 7).\n\nSome results are proposed on the CIFAR-N datasets in Tables 6 and 7 which hints towards the applicability of NoiseGPT to uncontrolled noisy datasets.", "weaknesses": "Although the NoiseGPT idea is relevant to study, I find that the paper severely lacks in insights.\nThere are no comparison into the possible complementarity of the noise detection between NoiseGPT and ProMix/M-correction. The reader is left to wonder whether NoiseGPT is better in every case or if there exist complementarity in the detection.\nThere also lacks a baseline comparison/complementarity of the LMM detection with using CLIP directly (as studied in [1] for example) which would be much more efficient to run (see Table 5).\n\nSecondly, because Webvision’s training set is naturally noisy, I believe it would have been more relevant to evaluate the detection capacities of NoiseGPT on these noisy labels instead of artificially corrupting the validation set. This experiment setting goes against previous label noise research evaluating on Webvision [2]. In general, the relevance of presenting ImageNet and Webvision results in this context is limited since both of their validation set are clean (before the proposed synthetic corruption) and classify the same classes.\n\nFinally, because LMMs are trained mostly trained on (and possibly overfit to) noisy data themselves, I would expect that they might struggle to detect noisy samples in uncurrated datasets such as Webvision in its original setting. I believe studying this supposition would be a nice addition to a revised version.\n\n[1] Liang, et al. \"Combating Label Noise With A General Surrogate Model For Sample Selection.\" arXiv preprint arXiv:2310.10463 (2023).\n\n[2] Ortego, et al. \"Multi-objective interpolation training for robustness to label noise.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your suggestions.\t\n\n- Complementary of noise detection & Baseline of noise detection using CLIP:\n  - To directly compare the effectiveness of our noise detection over the baseline methods such as Pro-Mix and DividMix, we conduct experiments on CIFAR-10 sym. 80% dataset and show the results in Table B in the attachment of the author rebuttal. The compared detection scores are from noise detection modules of baselines. Note that the hyperparameters of NoiseGPT are fine-tuned to get the best scores, which validates the effectiveness of our NoiselGPT.\n  - Moreover, we have made combination experiments on baselines more than M-correction and Pro-Mix, including DivideMix, Cross-Entropy, etc. Experiments show that all the baseline methods perform better after using cleaner datasets produced by NoiseGPT in most cases, verifying the enhancing effect of NoiseGPT. In this paper, we show two combinations that are assumed effective demonstrations of this property. \n  - Meanwhile, there exist cases where complementarity between NoiseGPT and baselines is tested. In fact, MLLM tends to rectify some classes in a dataset more than others, as illustrated in Fig. 6 and Fig. 8. This bias leads to the unbalance in cleaned datasets produced by NoiseGPT, and degrades the performance of classification models in cases like CIFAR-100 sym. 0.2&0.5. To this end, we propose that adjust the threshold τ to prevent overconfident rectifications on clean examples. Results of this method are shown in Figure D in the attachment.\n  - To better demonstrate the efficacy of NoiseGPT, we add an experiment where CLIP ViT-L is leveraged as a label rectifier to assign labels to examples considered noisy. The results are shown in Table A in the attachment in comparison with NoiseGPT on noisy CIFAR datasets.\n\n- Experimental setting Webvision:\n  - Although it would be more relevant to use the training set of Webvision which contains label noise crawled from the Internet, it would be difficult to analyze the performance of detection and rectification without ground-truth labels as reference. Thus, we leverage WebVision to demonstrate the scalability of NoiseGPT over larger datasets with more classes.\n  - For the same reason, there has been hard evidence in experiments indicating diminished detection performance of MLLMs on these datasets. \n  - To demonstrate the capability of NoiseGPT to combat noise web datasets, we visualize the distribution of ICD scores on 20000 examples selected from mini-WebVision[1], which is illustrated in Figure B in the attachment. The overall distribution is similar to what is shown in Figure 3, hopefully indicating the separation of clean and noisy examples.\n\n[1] J. Li, R. Socher, and S.C.H. Hoi. DivideMix: Learning with Noisy Labels as Semi-supervised Learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fSxCP1oKZb", "reviewer": "Reviewer_QCqA", "summary": "This paper proposes NoiseGPT, which utilizes a token-wise Mix-of-Feature (MoF) technique and an In-Context Discrepancy (ICD) measure to determine the noisy samples and find best candidate labels with CLIP and MLLM. The effectiveness of this approach is demonstrated through experiments, particularly on the ILSVRC12 dataset, where NoiseGPT achieves an AUROC of over 0.92.", "questions": "•\tThe paper mentioned they introduce MLLMs as expert to cope with noisy labels for the first time,  is there really no other previous work or recent work？ Can you compare your job with them or some MLLM basic label correction method？", "limitations": "•\tICD score distribution does not remain unchanged among different categories. Some categories of clean examples tend to have relatively higher ICD scores than others.\n\n•\tPerformance of NoiseGPT is constrained by the capabilities of the underlying machine expert it relies on.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tThis paper utilizes MLLM as expert to detect and rectify noisy labels. Specifically, they propose token-wise Mix-of-Feature and In-Context Discrepancy to detect noise, and use CLIP and MLLM with ICL prompt to find best candidate labels.\n\n2.\tExperiments were carried out on various synthetic noise and human annotated noise data sets to demonstrate the performance of NoisyGPT, and further improve the classification accuracy by combining with other LNL methods.", "weaknesses": "1.\tThis paper states that existing LNL methods either require the memorization effect to separate clean data from noisy data or rely on dataset assumptions that cannot extend to various scenarios. However, NoisyGPT seems similar to the former except that it uses LLM as an expert.\n\n2.\tThis paper needs more ablastion study and baselines to demonstrating the performance of each component of NoiseGPT. For example, what's performance if we directly use CLIP as label corrector in Table 1,2? How does NoiseGPT perform alone in Table 3? What will happen if we use perturbation methods other than MoF?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for pointing out perspectives that could be further improved.\n\n- Claims of NoiseGPT:\n  - Although there is a lack of works leveraging MLLMs to mitigate label noise, prior researchers have explored its capability in different fields. Huang [1] proposes MVT to supervise vision models using MLLM predictions in OOD scenarios. However, this method cannot be transferred to label noise detection scenarios as the predictions of MLLMs vary ubiquitously among different classes. It is hard to detect and rectify noisy labels from different classes with a unique threshold. Thus, we investigate and leverage the ICD curvature indicating the tendency of MLLM predictions under perturbations to distinguish between clean and noisy examples. On the other hand, CLIP, as a contrastive multi-modal model has been used to select clean examples according to its confidence[2]. And an adaptive loss is constructed for the classifier subsequently through combination with the transition matrix and the class frequency prior to mitigate the overfitting to label noise during training. Nonetheless, NoiseGPT is still distinctive for its way of utilizing pre-trained MLLMs.  \n  - About the memorization effects. There are significant differences between the memorization effects [3] that previous works use and the properties of MLLMs we leverage in NoiseGPT. Arpit defines memorization effects as behavior exhibited by DNNs to optimize faster on clean data than noise. Based on the fact that DNNs are prone to produce a higher loss for noise during the early stage of training, existing works [4,5,6] leverage an early stopping strategy or recurrently filter out noisy examples to mitigate the label noise. On the other hand, NoiseGPT leverages the generalization capability of MLLMs in a zero-shot manner. The properties of ICD curvature lie in its inherent optimization to associate visual features with corresponding text labels.\n\n\n- Ablation study:\n  - we add an experiment where different numbers of perturbations are tested. The performance of label rectification is shown in Figure A in the attachment file of the author rebuttal. It is noted that there is a marginal effect when $n$ increases to an extent and it would be unworthy since the computing time grows significantly.\n  - We leverage CLIP ViT-L as a label corrector in comparison with MLLM backbones (including a smaller version of LLM) in Table A to demonstrate the superiority of NoiseGPT. \n  - We tried feature interpolation in the earlier experiments instead of the proposed MoF to introduce noisy perturbation into query examples. The distribution of ICD scores would be different this way, which is illustrated in Figure C in the attachment. We make experiments on CIFAR-10N worse label, which turns out that the overlaps between clean and noisy scores of MoF are smaller compared with that of feature interpolation. Thus, it indicates a better capability to distinguish noisy labels. \n\n[1] Huang Z, Liu C, Dong Y, et al. Machine vision therapy: Multimodal large language models can enhance visual robustness via denoising in-context learning[C]//Forty-first International Conference on Machine Learning. 2023.\n[2] Liang, et al. \"Combating Label Noise With A General Surrogate Model For Sample Selection.\" arXiv preprint arXiv:2310.10463 (2023).\n[3] Arpit D, Jastrzębski S, Ballas N, et al. A closer look at memorization in deep networks[C]//International conference on machine learning. PMLR, 2017: 233-242.\n[4] Song H, Kim M, Park D, et al. How does early stopping help generalization against label noise?[J]. arXiv preprint arXiv:1911.08059, 2019.\n[5] Shen Y, Sanghavi S. Learning with bad training data via iterative trimmed loss minimization[C]//International conference on machine learning. PMLR, 2019: 5739-5748.\n[6] Chen P, Liao B B, Chen G, et al. Understanding and utilizing deep neural networks trained with noisy labels[C]//International conference on machine learning. PMLR, 2019: 1062-1070.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for reading and highlighting our paper, including 
- 1)”The paper presents a new application of MLLMs for label noise detection and rectification, introducing the concept of probability curvature…” (R1); 
- 2)”The observation about the sensitivity of noisy samples to noise in the latent space is intuitive and motivates the approach well. NoiseGPT going beyond simply prompting the MLM to answer whether the label is clean or noisy strengthens the contribution.” (R2); 
- 3)”Experiments were carried out on various synthetic noise and human annotated noise data sets to demonstrate the performance of NoisyGPT…” (R3).

They also voiced some valid concerns and put forward some constructive suggestions, which we will further elaborate on below.

- R4 questioned the novelty, R1 mentioned the similarity between probability curvature in DNNs and ICD, and R3 questioned that NoiseGPT as well as previous works leverage the memorization effects. 
  - The utilization of probability curvatures of DNN are behaviors of loss function during training where clean examples tend to have a lower loss, as indicated by memorization effects. However, our method leverages the inherent optimization in MLLMs to associate visual features with corresponding text labels in a zero-shot manner. Thus, the ICD curvature is essentially different from existing probability curvatures and can be proposed as a novel method to mitigate label noise problems.

- R2 and R3 indicated the completeness of performance comparison with CLIP and smaller MLLM model.
  - To this end, we make experiments respectively with CLIP as label corrector and FLAN-T5-XL as LLM backbone of smaller size. Results are shown in Table A.

- R1 and R4 put forward that some relevant works should be mentioned and discussed. These works will be added to the discussion in related works.
  - Existing LNL methods can be categorized into three types, data cleaning, loss-adjustment-based approaches, and sample-selection-based approaches. Data cleaning endeavors to filter out examples whose labels are likely to be corrupted [1,2]. Previous works in this branch leverage various methods [3,4,5] such as bagging, boosting, K-nearest neighbor, and anomaly detection to exclude falsely labeled instances. Nonetheless, these methods tend to over-clean samples that are even true-labeled, resulting in aggravation of shortage of data in many cases. Changes in probability curvatures of DNNs [6,7] during training based on memorization effects are also utilized to filter noisy examples. However, their robustness is strongly correlative to the training setting. Besides, CLIP[9] as a zero-shot classifier is leveraged to rectify noise, which cannot still distinguish noisy examples apart.

- R4 mentioned limited performance improvement of NoiseGPT compared to baseline on CIFAR-100 sym. 20%/50%.
  - The performance is degraded because the performance of NoiseGPT relies highly on the detection threshold, which is set fixed as 0.72 throughout Table 3. In fact, under low noise rates, the detection threshold should be higher than that of high noise rates, in order to prevent overconfidently selecting clean examples as noisy ones. Our experimental results in section 4.4 validate such a claim, and we further provide an ablation study in Figure D in the attachment to show that when setting proper threshold, the performance of NoiseGPT can still be effective compared to Pro-Mix.

- R1 indicated further exploration into the sensitivity of hyperparameters.
  - We add an experiment illustrating the noise rectification performance under different perturbation numbers $n$ in Figure A. The computing time of NoiseGPT can be reduced by selecting a smaller hyperparameter while maintaining the performance.
  - We demonstrate the influence of threshold τ on classification performance on two datasets in Figure D.

- R2 questioned the complementarity between NoiseGPT and baselines.
  - Since baselines like Pro-Mix and DivideMix have noise detection modules of their own, we make a comparison of noise detection in Table B to demonstrate the superiority of NoiseGPT in noise detection and its effectiveness in enhancing the classification performance of baselines.

- R2 questioned the utilization of WebVision dataset in our experiment settings and the performance of NoiseGPT on datasets that MLLM backbones are trained on.
  - We understand that it would be more relevant to use the training set of Webvision which contains label noise crawled from the Internet. However, it would be difficult to quantitively analyze the performance of detection and rectification without ground-truth labels as reference. Thus, to demonstrate the capability of NoiseGPT to filter out noisy examples within a web dataset, we visualize the distribution of ICD scores from mini-WebVision, as illustrated in Figure B. The overall distribution is similar to what is shown in Figure 3, indicating the separation of clean and noisy examples.

[1] Wheway V. Using boosting to detect noisy data[C]
[2] Sluban B, Gamberger D, Lavrač N. Ensemble-based noise detection: noise ranking and visual performance evaluation[J].
[3] Delany S J, Segata N, Mac Namee B. Profiling instances in noise reduction[J].
[4] Gamberger D, Lavrac N, Dzeroski S. Noise detection and elimination in data preprocessing: experiments in medical domains[J].
[5] Thongkam J, Xu G, Zhang Y, et al. Support vector machine for outlier detection in breast cancer survivability prediction[C].
[6] Garg I, Ravikumar D, Roy K. Memorization through the lens of curvature of loss function around samples[J].
[7] Ravikumar D, Soufleri E, Hashemi A, et al. Unveiling privacy, memorization, and input curvature links[J].
[8] Liu S, Niles-Weed J, Razavian N, et al. Early-learning regularization prevents memorization of noisy labels[J].
[9] Liang C, Zhu L, Shi H, et al. Combating Label Noise With A General Surrogate Model For Sample Selection[J].

Meta-review: This manuscript received mixed reviews both pre- and post-rebuttal. After the rebuttal, the AC initiated and discussed with the reviewers. It seems two key concerns remain after the rebuttal, author-reviewer discussion, and post-rebuttal discussion.

1. Reviewer TkfP complains about the novelty, and considers the proposed method as "A+B". During post-rebuttal discussion, reviewer TkfP listed quite some references to support this claim. The AC read through the list and many posts by the authors and by reviewer TkfP. After careful evaluations, the AC feels that the list of references are talking about the theoretical aspects of noise rectification, which does not fall in the scope of this discussion or the purpose of NoiseGPT (this submission). Hence, the AC is with the authors in terms of this disagreement.

2. Reviewer 81A3 was still concerned about WebVision style experiments. The author(s) did provided some experimental results, but they were considered (partly) inappropriate by reviewer 81A3. After reading the materials, the AC is with reviewer 81A3 in this issue. The authors are suggested to conduct experiments following the adivce of reviewer 81A3, and incorprate these results into the paper.
Decision: Accept (poster)