NoiseGPT: Label Noise Detection and Rectification through Probability Curvature

Authors: haoyu  wang
zhuo  huang
zhiwei  lin
tongliang  liu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 09:49:49

Abstract: Machine learning craves high-quality data which is a major bottleneck during realistic deployment, as it takes abundant resources and massive human labor to collect and label data. Unfortunately, label noise where image data mismatches with incorrect label exists ubiquitously in all kinds of datasets, significantly degrading the learning performance of deep networks. Learning with Label Noise (LNL) has been a common strategy for mitigating the influence of noisy labels. However, existing LNL methods either require pertaining using the memorization effect to separate clean data from noisy ones or rely on dataset assumptions that cannot extend to various scenarios. Thanks to the development of Multimodal Large Language Models (MLLMs) which possess massive knowledge and hold In-Context Learning (ICL) ability, this paper proposes NoiseGPT to effectively leverage MLLMs as a knowledge expert for conducting label noise detection and rectification. Specifically, we observe a probability curvature effect of MLLMs where clean and noisy examples reside on curvatures with different smoothness, further enabling the detection of label noise. By designing a token-wise Mix-of Feature (MoF) technique to produce the curvature, we propose an In-Context Discrepancy (ICD) measure to determine the authenticity of an image-label pair. Subsequently, we repeat such a process to find the best matching pairs to complete our label rectification. Through extensive experiments, we carefully demonstrate the effectiveness of NoiseGPT on detecting and cleansing dataset noise, especially on ILSVRC12, the AUROC of NoiseGPT reached over 0.92. And by integrating with existing methods, the classification performance can be significantly improved on noisy datasets, typically by 22.8% on 80% symmetric CIFAR-10 with M-correction.

Main paper: 
 introduction: 
 Contemporary machine learning is greedy for high-quality datasets. However, large-scale datasets such as human-annotated ones like ImageNet [1] and COCO [2] or internet-downloaded ones like WebVision [3] and Instagram Datasets [4] are either resource-consuming or untrustworthy. As a result, practitioners often have to spend substantial time to conduct prolonged labeling process and the results could still be undesirable. Noise still ubiquitously exist in almost all kinds of datasets. More importantly, dataset noise has been demonstrated to be significantly harmful for training of Deep Learning Models [5] . Consequently, it is urgent to discover efficient and transferable methodology to identify and rectify dataset noise.To solve this problem, Learning with Noisy Labels (LNL) aims to improve the robustness of Deep Neural Networks (DNNs) through bridging the noise distribution and clean distribution. Existing methodologies [6, 7] commonly leverage loss correction and loss reweighting techniques, where transition matrix [8, 9] , for example, is used to capture the noise generation process which enables endto-end optimization without fitting to noisy labels. Meanwhile, based on the observation that DNNs converge faster on clean examples than noisy ones, sample-selection-based methods [10, 11, 12] divide samples into clean and noisy during training in order to learn from confident examples while exploiting noisy samples [13, 14, 15, 16, 17] . However, it is challenging to estimate accurate noise transition matrix due to the complexity of real-world noise generation process. And the performance of approximation struggles under high-level noise rate without strong dataset assumptions. Consequently, existing methods are largely limited and require to be improved or assisted.Thanks to the development of MLLMs [18, 19, 20, 21, 22] which have been trained on massive data to effectively fit various real-world data distribution, we propose to leverage MLLMs as knowledge experts to help reduce dataset noise. Based on the empirical findings that noisy data are distributed on different MLLMs probability curvature from clean data, we propose an intuitive hypothesis that MLLMs are inherently optimized for the matching of image-text pairs. Such hypothesis motivates us to propose NoiseGPT which leverages a novel In-Context Discrepancy (ICD) criteria to identify and rectify noisy examples. Particularly, given an example which is in context with its label from dataset, if they match with each other, the MLLM output is stable under perturbation. Conversely, if the image and label are unmatched, the MLLM output will be sensitive to input perturbations. In circumstances where a sample is regarded as noisy, we further leverage CLIP [23] as zero-shot classifier to generate candidate labels. ICD is also applied on these candidates to elect a corrected label with best score. Through extensive studies on datasets such as conventional corrupted versions of CIFAR-10, CIFAR-100 [24] , ImageNet ISCVRC2012, as well as real-world grounded datasets like CIFAR-N [25] , WebVision [3] , the efficacy of NoiseGPT is rigorously validated. As a zero-shot data cleansing method, NoiseGPT demonstrates its powerful ability to scalably distill significantly cleaner versions of noisy datasets than their original ones. Furthermore, our algorithm can be embedded into other LNL methods to further enhance the noisy learning performance under various LNL scenarios. To conclude, our contributions in this paper are as follows:• We introduce MLLMs as machine experts to cope with noisy labels for the first time, potentially mitigating the reliance on human labor.• We propose NoiseGPT to tackle the challenge of label noise by leveraging zero-shot capability of MLLMs. We employ a novel In-Context Discrepancy (ICD) criteria combined with the token-wise Mixture-of-Feature (MoF) technique to quantify the possibility discrepancy and identify noisy samples. Additionally, these noisy samples are recycled after label rectification by comparing ICD scores among candidate labels generated by zero-shot classifier CLIP.• We conduct intricate investigations to evaluate the effectiveness of NoiseGPT. Furthermore, we integrate NoiseGPT as an auxiliary data cleansing method alongside existing LNL algorithms to validate its performance improvement. Additionally, we conduct performance analysis comprehensively understand the details of NoiseGPT.2 Related worksQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: In Abstract and Section 1, we discuss about previous LNL methods, and the utilization of MLLMs in our work to detect and rectify noisy labels.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
learning with noisy labels: 
 Existing LNL methods can be categorized into three types, data cleaning, loss-adjustment based approaches and sample-selection based approaches. Data cleaning endeavors to filter out examples whose labels are likely to be corrupted [26, 27] . Previous works in this branch leverages various methods [28, 29, 30] such as bagging, boosting, K-nearest neighbor, anomaly detection to exclude falselabeled instances. However, these methods tend to over-clean samples that are even true-labeled, resulting in aggravation of shortage of data in many cases. Tendencies of probability curvatures of DNNs [31, 32, 33] during training are also utilized to filter noisy examples. However, their robustness is strongly correlative to the training setting. Loss-adjustment based approaches focus on modifying the loss items before updating the DNNs, including loss correction and loss reweighting.Based on the fact that DNNs using Cross Entropy(CE) loss are prone to overfit noisy data [5] , substantial researches have been conducted to design a robust corrected loss [34, 35] by leveraging transfer learning [36, 37, 38, 39] , where noise transition matrix [8, 40, 41, 42] has been utilized. However, the performance of loss correction is highly dependent on the precisely-estimated transition matrix, which is hard under heavy noise and large number of classes. And the correction errors will be accumulated during training. Loss reweighting, on the other hand, aims at attributing weighted importance to examples in a designed training scheme to separate clean and noisy examples [43, 44, 45, 46] . Rectifying vectors [47] are also used to guide the classification network through leveraging information from input of the logits and labels. Nonetheless, this approach requires particular reweighting functions and hyperparameters for different noise type and datasets, limiting its practical implementation.Another line of work, sample-selection, endeavors to identify true-labeled examples from noisy training datasets during training. Researches [48, 49, 13, 50] distinguish clean examples by their early-stage loss in DNNS utilizing memorization effect [51, 52] . Multi-network learning [53, 54, 11] simultaneously trains an additional network to guide the sample-selection. Mix-up [15] employs a semi-supervised mixture model [55] to separate clean and noisy examples, which is integrated into a multi-network framework in DivideMix [13] . SELF [56] Recent years, Large Language Models [61, 62, 63, 64, 65, 66, 67] have been successfully applied across different tasks of natural language processing. Concurrently, the arising of Vision Transformers (ViTs) [68] has significantly advanced the development of visual models. In order to align the representations of image and text, CLIP [23] utilizes unsupervised learning by training separate image and text encoders with a contrastive loss on substantial image-text pairs. Besides, researches also endeavor to augment LLM with pretrained visual models to obtain multi-modal LLMs, known as MLLMs [69, 70, 71] . InstructBLIP [72] focuses on equipping MLLMs with the capacity to follow human-instructions on a natural-language interface. The mPLUG-Owl [73] introduces a modularized multi-modal pretraining paradigm to boost its transferability. MMICL [21] proposes a novel context training scheme, allowing the insertion of image features at any position among input text tokens. Based on the powerful zero-shot in-context capability of MLLMs, we propose NoiseGPT, leveraging extensive knowledge acquired from vast multi-modal training examples, to detect label noise. Subsequently, we utilize the zero-shot classification capability of CLIP for candidate labels and compare their matching rates with the input image to derive a rectified label.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: In Section 5, we point out that our method is constrained by the performance of visual models, as their capability to extract prioritized information limit our NoiseGPT to work on complicated large-revolution datasets. Besides, the effects of prompt settings is also mentioned.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 In this section, we propose NoiseGPT for label noise detection and rectification, as shown in Figure 1 .Given a classifier and an MLLM model parameterized by ψ and θ = {θ enc , θ ), where d(•; •) denotes a novel In-Context Discrepancy (ICD) measure and x stands for the perturbation of an example under distribution p(x|x). Based on such an effect, we can successfully detect whether a given image-label example pair x = {x, ỹ} has clean labels, i.e., whether ỹ matches with x. By Figure 1 : We leverage the zero-shot ability of MLLM to examine whether an example pair is noisy. To identify the potential noise, NoiseGPT first perturbs a given example x and produces a set of augmented versions x via using a novel token-wise Mixture-of-Feature (MoF) technique. Then by comparing the Softmax probabilities q θ between x and x, we can calculate an In-Context Discrepancy (ICD) measure to further decide the authenticity of the given label ỹ.further exploring the probable class candidates of the classifier, we can effectively find the ground truth label y by choosing the best-matching category. Hence, our NoiseGPT can assist as a dataset cleanser without human intervention.Generally, NoiseGPT includes two stages: 1) Noise detection and 2) Label rectification. Next, we first demonstrate the noise detection process in Section 3.1, and then we carefully elucidate the details of label rectification in Section 3.2. In this section, we propose to conduct noise detection based on the probability curvature effect. Specifically, we observe that clean examples lie on a smooth and convex probability curvature, and noisy examples fall on fluctuated and non-convex curvature, as shown in Figure 2 . A similar effect has also been found in Mitchell et al. [74] . As a result, under slight perturbation, the probability value would change differently, which allows us to identify dataset noise. To utilize such an effect, we first conduct a novel token-wise Mixture-of-Feature (MoF) to perturb each example, then, the In-Context Discrepancy (ICD) measure can be calculated to assist detection.Mixture-of-Feature aims to perturbate or augment examples at the feature level through interpolation, segmentation, and partial substitution, which avoids changing in the input space as it might cause a mismatch between modalities [75] . Since dataset noise owes to the image example possessing some confusing features that resemble other noisy classes, we propose to mix the features between a given query example x and exemplars x ex ∈ D ex from other noisy classes to inject noisy signals. Specifically, our MoF process is formulated asEQUATIONFigure 3 : We conduct validation experiments on six noisy datasets using ICD score, namely CIFAR-10 Sym. 50%" CIFAR-100 Sym. 50%" CIFAR-10N with worse labels, CIFAR-100N with noisy labels, Webvision Sym. 40%" ISLVRC12 Sym. 40%,. We collect scores of 10,000 clean samples and 10,000 noisy samples from each dataset, and each sample is augmented by 10 perturbed exemplars with a MoF weight of 0.5.where z and z ex are the latent representations of input images x ∈ x and x ex ∈ x ex correspondingly extracted by the vision encoders of MLLMs, z[I(•)] denotes the indexing operation that selects the elements from z where an indexing function I(•) holds true, ⊕ stands for element-wise addition, p(•) stands for the MoF function, and m indicates a binary mask to select feature elements, which is controlled by probability w to trade off between z and z ex , larger w reserves more information from the original input in the mixed feature z.Different from the MoF process conducted using intermediate features in Tong et al. [75] which might break the image patterns further causing misalignment with the subsequent LLM decoder, we propose a token-wise MoF operation that mixes the features by replacing tokens. Intuitively, the latent embeddings z of MLLMs are sequentialized tokens, each containing specific semantic information. Thus, we conduct MoF from the token level, which can correspondingly perturbate semantic parts across different example pairs. In this way, the mixed feature z can successfully inherit noisy information and still be aligned with the latent MLLM feature space. After producing mixed features z as the perturbed versions of original ones z, we combine them with the embeddings of both prompt s and label ỹ and feed them to the MLLM decoder module to obtain the final prediction as q θ dec {s, z, ỹ}. Next, we leverage the In-Context Discrepancy (ICD) criteria to detect dataset noise.In-Context Discrepancy is based on the probability curvature effect where perturbation affects the prediction probabilities of clean and noisy examples differently. Formally, our ICD criteria is calculated asEQUATIONwhere d(•; •) denotes the ICD function calculated between two entries, and p(z|z) is the perturbation distribution for generating z. Intuitively, MLLMs are convexly optimized to associate visual features with corresponding text labels. If an example pair x is clean, i.e., image x and label ỹ are matched, it will reside in an extrema where its local probability curvature is smooth and convex. On the other hand, if x is a noisy point where the x and ỹ are mismatched, it might fall on unstable and non-convex curvature.By injecting a little perturbation as done by the previous MoF process, we observe that the MLLM prediction q θ of clean examples will gradually decrease, but for noisy examples, q θ would seriously oscillate to a random value around its original one, as depicted in Figure 2 . Therefore, our ICD results for clean examples are always positive and relatively larger than noisy ones, further effectively validating the authenticity of input example pairs. To rescale the ICD score of different examples, we further conduct normalization by dividing the results of Eq. 2 by a standard deviation of q θ dec (s, z, ỹ):EQUATION)As a result, we can effectively divide clean examples and noisy ones based on the distribution of d(z, z) as illustrated in Figure 3 . Moreover, the effectiveness of such an effect is carefully validated on various datasets under MMICL [76] , a state-of-the-art MLLM that supports effective image-text in-context learning (ICL) in Section 4. In practice, we set a threshold τ to decide whether an example pair is noisy or not: those with discrepancy scores larger than τ are considered as clean, otherwise, we further conduct label rectification.Algorithm 1 NoiseGPT: noise identification and rectification.Input: sample x and label y from dataset D set , MLLM q θ , pretrained multi-classifier qcv, perturbation function p, weight of MoF w, number of perturbations n, number of candidate labels C, threshold τ 1: Uniformly sample ρC examplers with ground truth labels from D set to construct a tiny support set D ex ; 2: for x in D set do 3:xi ∼ p(x|x ex i ), i ∈ [1, n] ▷ token-wise MoF 4: μ ← 1 n n i=1 q θ (xi, ỹ) ▷ approximate expectation in Eq. 2 5: dx ← q θ (xi, ỹ) - μ 6: σ2 x ← 1 n n i=1 (q θ (xi, ỹ) -μ) 2 7: dx ← dx √ σ2 x ▷ normalization 8: if dx > τ then 9:Accept current ỹ as correct label 10: else 11:{y pred j } C 1 ← qcv(x) ▷ acquire candidate labels 12: for j ∈ 0, 1, • • • , c do 13: dj ∼ d(x, ỹ; x, y pred j ), j ∈ [1, C] 14: y ← arg max{ d(z, ỹ; z, y pred j )} C j=0 15:end for 16:end if 17: end for Thanks to the previous noise detection process, we can effectively validate the authenticity of a given example pair x = {x, ỹ}. Moreover, we hope to find potentially correct labels y to rectify the noisy ones ỹ. Particularly, we repeat the noise detection process for C probable noisy categories to find the most likely label y, formallyEQUATIONwhere y pred indicates the most probable candidate labels of x selected by the classifier ψ where the subscript denotes the j-th entry. Since various datasets have different numbers of classes, it is infeasible to traverse all classes. Hence, we leverage the predictions of classifier models such as CLIP [23] and select top-C classes as the candidate labels i.e., y pred = [argsort(g ψ (x))] 0:C where [argsort(•)] 0:C finds the index of top-C elements.Additionally, to ensure the MLLM output is mapped to a certain probability space in order to provide unified measurement for every example, we conduct prompting to restrict the output to only binary words, i.e., True or False. Moreover, our prompt leverages the ICL ability of MLLMs by providing both correct and incorrect matching exemplars. Our prompt is shown below:Question: This image <IMG_label#i > shows a photo of <label#i >, True or False? Answer: True; Question: This image <IMG_label#j > shows a photo of <label#i >, True or False? Answer: False; Question: This image <IMG_query> shows a photo of <label#i >, True or False? Answer: Specifically, for a query example x = {IMG_query, label#i} whose label prediction from classifier is i := [argsort(g ψ (x))] 0 , we choose one image IMG_label#i ∈ D ex to match with label label#i as a True exemplar. Moreover, we choose another image IMG_label#j ∈ D ex where j ∈ {0, • • • , C} and j ̸ = i as a False exemplar. As shown in Huang et al. [77] , such prompt design can effectively teach MLLMs what kind of image-label combination is True or False. As a result, the prediction q θ for the query example is based on both the inherent knowledge of MLLMs and the demonstration provided by the prompt. Furthermore, our NoiseGPT restricts q θ to binary values which significantly stabilizes the MLLM outputs. As revealed by existing studies [77, 78] , providing a set of class candidates and asking which one is the ground truth shows sub-optimal performance. The reason is that current MLLMs cannot effectively conduct multi-class classification and it gets easily confused when facing various choices. Therefore, we only require MLLMs to output binary prediction under demonstrative exemplars which unleashes the power of ICL and benefits making trustworthy inferences. We summarize our methodology in Algorithm 1. Next, we empirically validate the proposed NoiseGPT.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]Justification: We mainly propose the assumption of MLLM output probability curvature in Section 3, guiding the definition of ICD criteria, which is further proved with experiments in Section 3.1. The results of experiments in Section 4.5 also back up this theory.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments: 
 In this section, we first introduce the specifics of our experiment setup. Then we validate the efficacy of our method through an investigation with regard to noise identification and rectification. Furthermore, we undertake a quantitative analysis to demonstrate the enhancing effects of NoiseGPT as data cleansing method through comparing contemporary state-of-the-art LNL models with their combined counterparts with our methodology. Finally, we conduct ablation studies to fully explore the performance of our approach.Datasets In our experiments, we leverages re-annotated noisy datasets CIFAR-10N and CIFAR-100N [25] which contain real-world human annotation errors. We also generate noisy versions of CIFAR-10, CIFAR-100 [24] , WebVision [3] and ImageNet ILSVRC2012 for our studies. The details are as follows:CIFAR-10N and CIFAR-100N [25] : We adopt Aggregate, Rand1 and Worst versions of CIFAR-10N whose noise rates are 9.03%, 17.23% and 40.21% respectively and Noisy-Fine version of CIFAR-100N whose noise rate is 40.20%. CIFAR-10 and CIFAR-100 [24] : The CIFAR-10 contains 50,000 labeled images of 10 different classes, while CIFAR-100 contains 100 classes, each with 500 images. We mannually inject 20%, 50%, 80%, 90% symmetric and 40% asymmetric noise into CIFAR-10 and CIFAR-100 respectively.WebVision [3] : We utilize its validation subset which contains 50,000 images for the 40% symmetric noise condition. Moreover, in order to verify the capacity of NoiseGPT under the real-world circumstance where samples are collected without careful annotation, we utilize mini-Webvision, a subset of Webvision, for noise detection and rectification experiments and test the classification performance on the validation set of Webvision.ImageNet ILSVRC2012: We utilizes the validation subset which contains 50,000 images and generate symmetric noise for 50% examples in it.Models Primarily, we leverage original CLIP models [23] as our multi-classifier. For MLLM backbone, we employ MMICL [76] which adopts vision encoder of BLIP-2 [70] and FLAN-T5-XXL [79] as the LLM. For comparison with previous LNL works, we consider two methods Pro-Mix [59] and M-correction [35] which train an 18-layer PreAct Resnet for classification tasks. Evaluation settings For exemplars that are used in the in-context learning process, we select 3 images per category to construct a tiny groundtruth support set {x e } to simulate the scarceness of examples in real-world condition. examples are also selected from this support set to generate perturbed sample features. Specifically, for each query sample, we construct n = 10 perturbed features with different perturbing resources from {x e }. For pseudo labels, we employ the top C = 3 prediction of CLIP to conduct label rectification process. Effectiveness of noise detection For evaluation metrics, we utilize Area Under the Receiver Operating Characteristic curve (AUROC) to reflect the noise detection performance. AUROC is commonly used to evaluate the performance of binary classification models. Generally, a higher AUROC score (closer to 1) indicates better discrimination ability, while 0.5 suggests random guessing.For label rectification performance, we compare corrected labels of query examples with their true labels to obtain a rectification accuracy.Table 1 shows the performance of NoiseGPT in noise detection and rectification, the first column shows AUROC scores and the second shows the label correction accuracy. Figure 4 2 shows the results. Note that the detection scores of baselines are from their own detection modules. And the hyperparameters of NoiseGPT are fine-tuned to get the best scores.While NoiseGPT primarily focuses on zero-shot noise detection and rectification, existing research in Learning with Noisy Labels (LNL) has delved into training noise-robust Deep Neural Networks (DNNs) for classification tasks. To effectively showcase the advantages of our approach, we integrate NoiseGPT as a data cleansing method with two LNL baselines, namely Pro-Mix and M-correction, constituting "NoiseGPT+Pro-Mix" and "NoiseGPT+M-correction", respectively. The classification models are trained under the same settings of as specified in papers of baselines. Comparison with real-world noisy labels In the real-world situation, some datasets prevailing recently are collected from the Internet, such as Webvision. Thus, they contain label noises that represent different patterns from the symmetric. In this paper, we utilize mini-Webvision, a smaller subset of Webvision to verify the capacity of NoiseGPT to combat label noises of this kind. Due to the lack of ground-truth annotations, we directly compare the classification performance with baselines. Following previous works [13] , we use the Inception-ResNet v2 [80] as the classification backbone.The results in Table 5 demonstrate that NoiseGPT remains effective on real-world noisy datasets like Webvision. The "NoiseGPT+" denotes that the training set is first cleaned by NoiseGPT. Since we approximate the expectation in Eq. 2 with a sequence of perturbations, theoretically increasing the perturbation number n will make the normalized ICD score more robust and effective in distinguishing clean and noisy examples. However, there is a marginal effect when n increases to an extent and becomes computationally unworthy. Figure 5 shows the trend of noise rectification performance under a changing hyperparameter n in the attachment. In our experiments, we select the number of perturbations in order to balance the computational cost and performance.The existence of curvatures on different datasets We further demonstrate the quality of possibility curvature by conducting experiments where query examples from noisy dataset are augmented by a series of perturbed exemplars with varying perturbation strength. According to Section 3.1, we can control the perturbation strength by adjusting token-wise MoF weight w, a larger w indicates higher proportion of information coming from query example. We conduct this experiment on four datasets: CIFAR-10 Sym. 50%, CIFAR-100 Sym. 50%, Webvision Sym. 40%, and ILSVRC Sym. 40%.Figure 6 shows the curvatures of output Softmax probability under changing MoF weights. We calculate the averaged q θ (x) of clean and noisy examples in each dataset respectively. It is investigated that on for clean examples, the output Softmax probability q clean θ (x) tends to descend as the MoF weight w decreases. Conversely however, the curvature of noisy examples fluctuates optionally unconcerned with the change of weight w. This phenomenon is confirmed on all datasets, firmly backing up our method, NoiseGPT, which utilizes the discrepancy of output Softmax probability q θ (x) under perturbation to distinguish between clean and noisy examples. Clean classes that are easier to be considered noisy Attributing to the fact that clean and noisy examples have different ICD score distributions, our NoiseGPT is capable to detect and rectify noise. However, the ICD score distribution does not remain unchanged among different categories. Some categories of clean examples tend to have relatively higher ICD scores than others, which, in other words, are easier to be mistaken as noisy during the process of NoiseGPT. We investigate such categories over different datasets, their average ICD scores are recorded as a indication of to what extent they tend to be mistaken. Note that for clean examples, lower score indicates easier to be mistaken. Figure 7 shows which classes are easy to be mistaken for noisy. Further exploration of detection biases are provided in Section 5 in our Appendix.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Our method is based on the zero-shot inference of backbone models. Following the experiment settings and hyperparameters mention in Section 4.1, all results of NoiseGPT can be reproduced. As for comparison baseline works, we follow the settings in their original codes and papers, and re-produce some of the results. 
conclusion: 
 Contribution In this work, we propose a novel label noise solution via leveraging MLLMs as experts to reduce and recycle noisy instances in datasets. Specifically, we investigate the probability curvature of MLLMs under input perturbation. Through a token-wise Miture-of-Feature technique, we can calculate ICD scores of input examples and divide them into clean and noisy. By conducting extensive quantitative and qualitative experiments on different datasets, our method is validated to sustain effective over varying noise conditions. Moreover, it surpasses previous LNL methods in noise detection and poses substantial potential to cope with other deep learning models to improve their performance. In the future, our method can be further explored for insights into more probelms like OOD detection, weakly-supervised learning, etc.Limitation Despite its adaptability to various datasets and noise levels, the performance of NoiseGPT is constrained by the capabilities of the underlying machine expert it relies on. Research [75] has highlighted the bottleneck effect in vision models within MLLMs. Furthermore, the instruction-following capability of MLLMs significantly influences the distributions of ICD scores, which are closely tied to the confidence of MLLM answers. Enhancing the proficiency of these machine experts can lead to improved performance of NoiseGPT in its tasks.  
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? 
 
 Appendix: In Section 4.3, we have compared the classification performance of "NoiseGPT+" and baselines on CIFAR-10/100 Sym./Asym. datasets. Here we expand this experiment on more datasets: CIFAR-10N Worst and CIFAR-100N Noisy. Table 8 shows the effects of NoiseGPT as a dataset cleansing method. And Table 9 shows the results of classification accuracy. In section 4.2 we introduce AUROC and Label Correction Accuracy as evaluating metrics for NoiseGPT noise detection and rectification performance. In order to better understand the improving impacts NoiseGPT poses to classification tasks, we further exhibit the evaluating results of NoiseGPT on CIFAR-10/100 Sym./Asym. datasets. Table 10 shows the noise detection and rectification results with AUROC score and correction accuracy. Figure 8 shows the ROC curves of noise identification. The experiments in Section 4.5 indicate that there are essentially biases in the noise detection stage of NoiseGPT, which will lead to unbalanced example quantities of different classes after rectification. Similar phenomenon also appears in the baselines. Thus, here we compare the biases in noise Figure 9 : Noisy categories that are easier to be mistaken as clean.detection between NoiseGPT and Proto-Mix, and calculate the proportions of 10 classes in selected clean data on CIFAR-10 sym. 90%. Table 11 shows the results. Our NoiseGPT exhibits significantly lower variance in the example distributions across different classes, indicating reduced bias in noise detection and rectification compared to Proto-Mix.NeurIPS Paper Checklist• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Justification: In Section 5 we discuss the positive impacts our work may have on the future researches in the fields of noisy labels and machine learning.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).Justification: There is no risk our released work could have posed.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We state the reference of all the models and data clearly in our paper, which can be seen as cites in the main content and detailed in Reference.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 


Decision: This manuscript received mixed reviews both pre- and post-rebuttal. After the rebuttal, the AC initiated and discussed with the reviewers. It seems two key concerns remain after the rebuttal, author-reviewer discussion, and post-rebuttal discussion.

1. Reviewer TkfP complains about the novelty, and considers the proposed method as "A+B". During post-rebuttal discussion, reviewer TkfP listed quite some references to support this claim. The AC read through the list and many posts by the authors and by reviewer TkfP. After careful evaluations, the AC feels that the list of references are talking about the theoretical aspects of noise rectification, which does not fall in the scope of this discussion or the purpose of NoiseGPT (this submission). Hence, the AC is with the authors in terms of this disagreement.

2. Reviewer 81A3 was still concerned about WebVision style experiments. The author(s) did provided some experimental results, but they were considered (partly) inappropriate by reviewer 81A3. After reading the materials, the AC is with reviewer 81A3 in this issue. The authors are suggested to conduct experiments following the adivce of reviewer 81A3, and incorprate these results into the paper.

Reviews: review: paper_summary: paper_summary: This paper employs Multimodal Large Language Models to detect noise samples. The In-Context Discrepancy is utilized to quantify the discrepancy between the original and perturbed samples. Additionally, the identified noise samples are integrated with the Pro-Mix and M-correction noise label learning framework. Furthermore, a comparison between the proposed method and existing approaches is conducted.
strength_weakness: strength_weakness: Weakness;
1) The novelty of the approach is limited. 
2) Some previous works are not properly cited, such as the lack of citation for In-Context Discrepancy. 
3) Several highly relevant works are missing, such as those measuring the In-Context Discrepancy between original and perturbed samples, including recent publications.

[1] Early-learning regularization for preventing memorization of noisy labels. 
[2] Strategies for preventing continuous damage caused by noise during model training.
 [3] A survey on learning from noisy labels with deep neural networks. 
[4] Learning to rectify for robust learning with noisy labels. 
[5] ...

4) The efficacy of the proposed method is questionable, as Table 3 shows only a limited improvement compared to Pro-Mix*.


Some typos:
"Figure ?? "

"50%„ "
questions: questions: The efficacy of the proposed method is questionable, as Table 3 shows only a limited improvement compared to Pro-Mix*.

The author is recommended to include a section on "Sample cleaning" in the related work to provide a comprehensive coverage of relevant studies.
limitations: limitations: YES.
paper_summary: paper_summary: The paper introduces NoiseGPT, a method that leverages Multimodal Large Language Models (MLLMs) for label noise detection and rectification in datasets. The approach exploits the probability curvature effect observed in MLLMs, where clean and noisy examples exhibit different curvature smoothness under perturbation. NoiseGPT uses a token-wise Mixture-of-Feature (MoF) technique to generate perturbations and an In-Context Discrepancy (ICD) measure to detect label noise and rectifies them. Experiments demonstrate the method's effectiveness in improving dataset quality and enhancing classification performance in various noisy datasets.
strength_weakness: strength_weakness: 1. Probability curvature has been previously explored in [1, 2] and [1] specifically for mislabeled detection in vision models, thus reducing the significance of the contribution in the paper, however, one should note [1,2] that these works are limited to vision classification models only. It might warrant discussion in the related work section.

2. It is not clear if the curvature properties leveraged in this paper are restricted to MLLMs, the paper does not provide any evidence for or against it. I.e. do smaller models behave differently or the same? I.e. is a `large' model needed?

3. The paper has missed some similar works, such as [1] Garg et al. (2023) and [2] Ravikumar et al. (2024), which also explore curvature in vision models and mislabelled sample detection. Thus the discussion of these works in the related section should be considered.

4. There needs to be a discussion on the scalability of the method to large-scale datasets, the results in the paper use small subsets of ImageNet and Webvision. Since multiple perturbations are needed on MLLM to detect and correct for label noise, can this method scale to millions and billons of images?



[1] Garg, et al. "Memorization through the lens of curvature of loss function around samples." arXiv preprint arXiv:2307.05831 (2023). \
[2] Ravikumar, et al. "Unveiling privacy, memorization, and input curvature links." arXiv preprint arXiv:2402.18726 (2024).
questions: questions: 1. How sensitive is the method to these hyperparameters? The paper could benefit from an analysis of the sensitivity to hyperparameters like the number of perturbations $n$, and the threshold $\tau$ and more details of how they were chosen.

2. The paper mentions the influence of prompt settings on the reliability of ICD scores. Can the authors elaborate on how different prompt designs impact the performance of NoiseGPT? 

3. Is curvature only relevant in MLLM models? I.e. can the baseline be a classification model and do curvature scores in that case perform similarly (see [1]) for mislabeled sample identification?

4. My understanding suggests that the method scales poorly to large datasets since multiple perturbations are needed on MLLM input for rectification. The authors should elaborate on this aspect, discussing potential strategies to mitigate the high computational cost and improve scalability. For instance, are there ways to reduce the number of perturbations required while maintaining detection accuracy? Can the method be parallelized more effectively? Thus the compute requirement and the number of perturbations $n$ and how it affects performance needs discussion.

[1] Garg, et al. "Memorization through the lens of curvature of loss function around samples." arXiv preprint arXiv:2307.05831 (2023).
limitations: limitations: 1. The paper adequately addresses some limitations, such as the reliance on MLLMs' vision capabilities and the effect of prompt settings on ICD scores. However, the computational cost associated with the method is not fully discussed. Since NoiseGPT requires generating multiple perturbations (n perturbations) for each example, this can lead to significant computational overhead, particularly when dealing with large datasets. This overhead may result in slow processing times, as the models involved (e.g., CLIP, large language models) are computationally intensive.

2. It is not clear if the curvature properties leveraged in this paper are restricted to MLLMs, the paper does not provide any evidence for or against it. I.e. do smaller models behave differently or the same?
paper_summary: paper_summary: This paper proposes to use large multimodal models (LMM) to detect noisy annotations in image datasets. The NoiseGPT method consists in observing whether a controlled perturbation in the latent representation of the LMM leads to a large modification of the LMM response. Experiments in the paper demonstrate that noisy samples are more sensitive to input perturbations than their clean counterparts. This discrepancy is used to detect noisy samples.

Experiments are proposed where NoiseGPT is used in conjunction with existing noise robust algorithms to correct noise on synthetically corrupted datasets as well as the real world CIFAR-N datasets. When NoiseGPT is used, the classification accuracy of datasets is improved.
strength_weakness: strength_weakness: Although the NoiseGPT idea is relevant to study, I find that the paper severely lacks in insights.
There are no comparison into the possible complementarity of the noise detection between NoiseGPT and ProMix/M-correction. The reader is left to wonder whether NoiseGPT is better in every case or if there exist complementarity in the detection.
There also lacks a baseline comparison/complementarity of the LMM detection with using CLIP directly (as studied in [1] for example) which would be much more efficient to run (see Table 5).

Secondly, because Webvision’s training set is naturally noisy, I believe it would have been more relevant to evaluate the detection capacities of NoiseGPT on these noisy labels instead of artificially corrupting the validation set. This experiment setting goes against previous label noise research evaluating on Webvision [2]. In general, the relevance of presenting ImageNet and Webvision results in this context is limited since both of their validation set are clean (before the proposed synthetic corruption) and classify the same classes.

Finally, because LMMs are trained mostly trained on (and possibly overfit to) noisy data themselves, I would expect that they might struggle to detect noisy samples in uncurrated datasets such as Webvision in its original setting. I believe studying this supposition would be a nice addition to a revised version.

[1] Liang, et al. "Combating Label Noise With A General Surrogate Model For Sample Selection." arXiv preprint arXiv:2310.10463 (2023).

[2] Ortego, et al. "Multi-objective interpolation training for robustness to label noise." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
questions: questions: Why are the experiment settings on Webvision different from previous research ?

Did the authors observe that NoiseGPT struggles to detect web-noisy samples more because they come from the same distribution as the data LMMs are trained on ? (see my last weakness point)
limitations: limitations: NoiseGPT requires a lot more compute than previous approaches to detect noisy samples.
This has been adequately evidenced in the paper (Table 6)
paper_summary: paper_summary: This paper proposes NoiseGPT, which utilizes a token-wise Mix-of-Feature (MoF) technique and an In-Context Discrepancy (ICD) measure to determine the noisy samples and find best candidate labels with CLIP and MLLM. The effectiveness of this approach is demonstrated through experiments, particularly on the ILSVRC12 dataset, where NoiseGPT achieves an AUROC of over 0.92.
strength_weakness: strength_weakness: 1.	This paper states that existing LNL methods either require the memorization effect to separate clean data from noisy data or rely on dataset assumptions that cannot extend to various scenarios. However, NoisyGPT seems similar to the former except that it uses LLM as an expert.

2.	This paper needs more ablastion study and baselines to demonstrating the performance of each component of NoiseGPT. For example, what's performance if we directly use CLIP as label corrector in Table 1,2? How does NoiseGPT perform alone in Table 3? What will happen if we use perturbation methods other than MoF?
questions: questions: •	The paper mentioned they introduce MLLMs as expert to cope with noisy labels for the first time,  is there really no other previous work or recent work？ Can you compare your job with them or some MLLM basic label correction method？
limitations: limitations: •	ICD score distribution does not remain unchanged among different categories. Some categories of clean examples tend to have relatively higher ICD scores than others.

•	Performance of NoiseGPT is constrained by the capabilities of the underlying machine expert it relies on.

Comments: title: Rebuttal Summary
Rebuttal recap
Further discussion
Remaining concerns
Remaining concerns
Further discussion
Reply
Response to the rebuttal
Further discussion
Further discussion
Dear reviewers -- please read the rebuttal and respond
comment: Dear Reviewers, AC, and SAC:

We deeply thank the hard work done by AC and SAC such as assigning reviewers, guiding the process, and further organizing the discussion. We also sincerely appreciate the reviewers for taking the time to read our paper, provide constructive opinions, and get involved in our discussion. Without your elaborative help, our paper could not have been improved.

Here we summarize our rebuttal to present a macro perspective which could hopefully help grasp our contribution and modification quickly.

Through interactive discussion, several consensuses have been achieved:
- This paper is clearly formulated and well-written.
- Most reviewers found the idea of our paper interesting and potential to be widely adopted.

Although several reviewers questioned the complementarity of performance comparison between NoiseGPT and other SoTA methods, we have carefully provided more results on the WebVision dataset, demonstrating the extendable effectiveness of NoiseGPT.
| Dataset\Method | ELR | Mix | DivideMix | NoiseGPT+DivideMix |
| ----------- | ----------- | ----------- | ----------- | ----------- |
| Webvision | 73.00 | 74.96 | 77.32 | 78.10 |

For a fair comparison, we evaluate the noise detection performance of NoiseGPT and Pro-Mix with precision, recall, and F1 score on the same dataset setting. NoiseGPT gains better results on this metrics.
| Method\Metrics | Precision | Recall | F1 |
| ----------- | ----------- | ----------- | ----------- | 
| DivideMix | 94.36% | 92.63% | 93.49% |
| Pro-Mix | 96.59% | 93.78% | 94.57% |
| NoiseGPT | 97.80% | 94.39% | 96.07% |

Although two reviewers remained silent during the discussion phase, here we stress our contribution for efficient evaluations:
- Investigate the properties of MLLM output possibilities and propose In-Context Discrepancy framework to combat label noise in datasets, which examines the different patterns of change in clean and noisy samples before and after perturbation. ICD goes further beyond the previous works because it:
  - Does not require the training of DNNs as previous LNL works do, which are based on memorization effects.
  - Addresses the varying output distributions of different categories when MLLMs are simply prompted to produce a “True” or “False” possibility. Thus, clean and noisy samples can be separated by a unique threshold.
- Carefully leverage these properties by designing a MoF method to introduce perturbations into query samples. 
  - Our MoF fuses information of query samples and perturbing samples on the latent representation level, which is essentially different from previous perturbation methods used in adversarial learning.
- Possesses superior capability of detecting and rectifying noisy labels compared with previous LNL works such as Pro-Mix, DivideMix, and M-correction. 
  - Since several reviewers mentioned the biases among different categories in detection and rectification. We conduct experiments on the same dataset with NoiseGPT and Pro-Mix, which can be seen in the table below. Our method shows a much lower variance, demonstrating a smaller bias.
| Method\Class index| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | Variance |
| ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |----------- |----------- |
| Pro-Mix | 9.79% | 11.02% | 9.52% | 7.84% | 10.05% | 8.23% | 10.61% | 11.01% | 10.80% | 11.12% | **1.24** |
| NoiseGPT | 10.43% | 10.67% | 8.95% | 10.63% | 10.91% | 9.04% | 10.75% | 8.46% | 9.98% | 10.18% | **0.68** |
  - Additional experiments are made with a different LLM backbone FLAN-T5-XL, and CLIP as a label corrector.
| Method\Dataset | C10sym20 | C10sym50 | C10sym80 | C10sym90 | C10asym40 | C100sym20 | C100sym50 | C100sym80 | C100sym90 |
| ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |  ----------- |  ----------- |  ----------- |  ----------- |  
| CLIP-ViT-L | 10.2% | 18.7% | 24.4% | 29.6% | 16.1% | 18.5% | 30.8% | 44.2% | 45.4% |
| FLAN-T5-XL | 9.8% | 18.1% | 25.7% | 31.2% | 11.6% | 18.0% | 31.4% | 47.9% | 50.3% |
| FLAN-T5-XXL| **7.4%** | **13.6%** | **19.3%** | **24.4%** | **9.4%** | **16.1%** | **28.0%** | **40.2%** | **44.6%** |
  - Ablation studies are conducted with different hyper-parameter settings. 

In the past several months, we have tried our best to improve the quality of this paper and address each concern from all reviewers. We sincerely hope our effort could be contributed to the community and advance the development of LNL. Thanks again for your kind help and constructive opinions, we will keep trying our best to polish this paper in the future.

Sincerely,

Authors.
Dear Reviewer yYJy:

Thank you for your valuable time to review our paper. It has been a while since you last discussed with us. Here we provide a recap to help you read our rebuttal without feeling unfamiliar with this paper.

Your initial concerns can be summarized into two points:
- The relationship between memorization effects studied in previous LNL works and the ICD in our NoiseGPT.
- Lack of discussion on previous LNL works using MLLMs as experts.
- Lack of ablation studies to demonstrate the performance of each component of NoiseGPT and different label corrector backbones.
- There is a bias in ICD scores among different categories.

Our rebuttal carefully addresses these concerns by:
- Elaborating on the essential differences between NoiseGPT and previous LNL works.
- Discussing prior utilization of MLLMs as experts in LNL demonstrating the further insights and superiority of our works, and including them in the related works.
- Conducting experiments to explore the performance of each component in NoiseGPT, including the influence of hyperparameters $n$ and $\tau$, performance comparison with different backbones (CLIP&FLAN-T5-XL), the results are shown in the PDF attachment of our official author rebuttal.
- Since other reviewers also put forward concerns about the influence of bias for different categories in label detection and rectification. We conducted another experiment to further compare the biases between NoiseGPT and Pro-Mix. We calculate the proportion each class takes up in selected clean data on CIFAR sym. 90%. The results are illustrated below. We can see that our method shows a much lower variance, which demonstrates the effectiveness and superiority of our method.
| Method\Class index| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | Variance |
| ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |----------- |----------- |
| Pro-Mix | 9.79% | 11.02% | 9.52% | 7.84% | 10.05% | 8.23% | 10.61% | 11.01% | 10.80% | 11.12% | **1.24** |
| NoiseGPT | 10.43% | 10.67% | 8.95% | 10.63% | 10.91% | 9.04% | 10.75% | 8.46% | 9.98% | 10.18% | **0.68** |

It is really important to us that you could kindly read our rebuttal and provide further questions if there are any. Thank you so much and hope you have a good day.

Best,
Authors.
Dear Reviewer 81A3:

We want to thank you again for taking the time to read our rebuttal. We have tried with our maximum effort to address your concerns, but it has been a while since your initial comments. If there are any concerns unresolved, we would be glad to have further discussions.

Thanks again for your time, looking forward to hearing from you soon.

Best,

Authors.
Dear Reviewer 81A3:

We really appreciate your constructive opinions that helped us improve this paper. We have carefully elaborated on the concerns about:
- Complementary of noise detection. We put forward our comparison experiment results of the noise detection biases existing in NoiseGPT and Pro-Mix, which demonstrates the effectiveness of our method.
- Webvision results. We made a classification performance comparison on Webvision between NoiseGPT and baseline methods.

If there are any concerns unresolved, we would be glad to have further discussions.

Thanks again for your time, looking forward to hearing from you soon.

Best,

Authors.
Dear Reviewer gxfG:

We thank you again for your valuable time to review this paper, your constructive advice is really helpful.

By carefully answering all your concerns, this paper has been improved on scalability to larger datasets, the sensitivity of hyperparameters, the prompt settings, and the invetigations in different backbone models. We hope to know whether our rebuttal solves your concerns. Since the NeurIPS conference supports interactive discussion, we wish we could have to chance to make further efforts to polish our work.

Thanks again for your previous help, we hope to hear from you soon!

Best,

Authors.
Dear Reviewer gxfG:

We want to express our appreciation for your valuable suggestions, which greatly helped us improve the quality of this paper. We are also glad that you agreed that our idea is novel and has the potential to be widely used. 

We have made our maximum effort to address your concerns on scalability to larger datasets, the sensitivity of hyperparameters, and the unique properties of ICD curvatures in MLLMs, and etc.

Your further opinions are very important for evaluating our revised paper and we are hoping to hear from you. Thank you so much.

Best,
Authors.
Dear Reviewer TkfP:

Thanks for your quick response. We will include the works you have mentioned in the later version of our paper.

Although the form and terminology of perturbations in NoiseGPT and robustness learning are similar, they are basically different. In robustness learning, perturbation methods are normally designed to attack models by subtly altering samples in ways that are imperceptible to the human eye, leading to incorrect classifications. 
- These methods typically introduce perturbations on the sample’s pixels by adding gradients opposite to the model’s predictions. 
- Existing research on model prediction distributions in robustness learning and robust training focuses on the distributional differences between samples before and after adding perturbations. 

In contrast, our study in NoiseGPT: 
- Our perturbations are applied at the latent feature level of the samples instead of a pixel level, which is a fusion of features from both query and perturbing samples. 
- In-Context-Discrepancy examines the different patterns of change in clean and noisy samples before and after perturbation: Clean samples exhibit prediction distribution changes similar to those caused by adversarial perturbations when information from noisy samples is incorporated, while noisy samples’ prediction scores are less susceptible to noisy perturbation. This differential response to perturbations between clean and noisy samples enables us to apply MLLMs to noise detection problems. 

Consequently, our approach significantly differs from previous studies on prediction distributions and introduces methods such as MoF to better address label noise problems in datasets. Ultimately, our method achieves superior performance compared to other state-of-the-art (SoTA) methods. All of these demonstrate the substantial novelty of the proposed method.

Thanks again for your discussion, we hope to hear your further opinions soon.

Best,
Authors.
Thanks to the authors for the rebuttal. The author mentioned that " In-Context Discrepancy is a novel". 

The author gives the experiment results as in Fig. 2, where the prediction probability distribution between the perturbed noise sample and clean samples is totally different. Therefore the author proposes the  In-Context Discrepancy to detect the noise samples.
However, the core idea that the prediction probability distribution between the perturbed noise sample and clean samples is totally different has been broadly studied in the robustness learning area. Many researchers study the prediction distribution between the noise samples and clean samples. However, the related works and analysis are missing. The author just claims that " In-Context Discrepancy is a novel". 

The author combines the GPT and noise label learning, which is an "A+B" work. The core idea is to find the difference between the noise sample and clean samples. However, the adopted core idea is similar to existing works. Therefore, I think the novelty of the approach is limited.
Dear Reviewer TkfP:

We really appreciate your efforts to help improve this paper. We have carefully addressed the mentioned concerns, such as citations of relevant works, and the limitation of improvement in classification performance compared to Pro-Mix. Experiments have also been added to elaborate on these problems.

Having further discussions really helps to achieve consensus and clarify misunderstandings, we are eager to know if there are any remaining problems. We will try our best to address them.

Best,
Authors.
Dear Reviewer 81A3:

We appreciate your prompt reply,

- Complementary of noise detection & Baseline of noise detection using CLIP:
  - We will include these comparison results in the later version of the paper. 
  - To further compare the bias in noise detection between NoiseGPT and Pro-Mix, we calculate the proportion each class takes up in selected clean data on CIFAR sym. 90%. The results are illustrated below. We can see that our method shows a much lower variance, which demonstrate less bias in noise detection and rectification compared to Pro-Mix.
| Method\Class index| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | Variance |
| ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |----------- |----------- |
| Pro-Mix | 9.79% | 11.02% | 9.52% | 7.84% | 10.05% | 8.23% | 10.61% | 11.01% | 10.80% | 11.12% | **1.24** |
| NoiseGPT | 10.43% | 10.67% | 8.95% | 10.63% | 10.91% | 9.04% | 10.75% | 8.46% | 9.98% | 10.18% | **0.68** |


- Webvision results
  - We make a classification experiment on Webvision using the same framework as Table 3. The “NoiseGPT+” denotes that the training set is first cleaned by NoiseGPT. Note that we train classification models on mini-Webvision and then test on the validation set of Webvision. The results below demonstrate that NoiseGPT remains effective on datasets like Webvision.
| Dataset\Method | ELR | Mix | DivideMix | NoiseGPT+DivideMix |
| ----------- | ----------- | ----------- | ----------- | ----------- |
| Webvision | 73.00 | 74.96 | 77.32 | 78.10 |

Thanks again for your discussion, we hope to hear your further opinions soon.

Kind regards,

Authors.
During August 7-13 there will be an open discussion between the reviewers and authors. I observed that the author responded to reviewer questions elaboratively, and Reviewer 81A3 have already responded to the rebuttal (thank you!).

Reviewer TkfP, gxfG, QCqA --- Please use this opportunity to further clarify remaining concerns and confusions.

Thank you!
AC
I thank the authors for their response.

- Complementary of noise detection & Baseline of noise detection using CLIP:

Thank you for providing the detection scores of a CLIP-based model and smaller LMMs. These baselines are enlightening and should be included in the paper.

My comments on the complementarity of NoiseGPT with existing approaches are more directed towards a study of the noise detection biases of LMMs when compared to non-pretrained algorithms such as ProMix or DivideMix. For example, are the difficult classes in Figure 6 and 8 also difficult for ProMix and DivideMix or is this a result of the training data used for the LMM ? 

- Webvision results

Although I agree that detection scores on Webvision are not directly computable, the test accuracy of a model trained with NoiseGPT on Webvision would give an estimation of the capacity NoiseGPT has to remove noisy data harmful to generalization. The separation observed in figure B is however encouraging.