CLIPCleaner: Cleaning Noisy Labels with CLIP

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-31 18:04:59

Abstract: Learning with Noisy labels (LNL) poses a significant challenge for the Machine Learning community. Some of the most widely used approaches that select as clean samples for which the model itself (the in-training model) has high confidence, e.g., 'small loss', can suffer from the so called 'self-confirmation' bias. This bias arises because the in-training model, is at least partially trained on the noisy labels. Furthermore, in the classification case, an additional challenge arises because some of the label noise is between classes that are visually very similar ('hard noise'). 
This paper addresses these challenges by proposing a method (*CLIPCleaner*) that leverages CLIP, a powerful Vision-Language (VL) model for constructing a zero-shot classifier for efficient, offline, clean sample selection. 
This has the advantage that the sample selection is decoupled from the in-training model and that the sample selection is aware of the semantic and visual similarities between the classes due to the way that CLIP is trained. We provide theoretical justifications and empirical evidence to demonstrate the advantages of CLIP for LNL compared to conventional pre-trained models.
Compared to current methods that combine iterative sample selection with various techniques, *CLIPCleaner* offers a simple, single-step approach that achieves competitive or superior performance on benchmark datasets. To the best of our knowledge, this is the first time a VL model has been used for sample selection to address the problem of Learning with Noisy Labels (LNL), highlighting their potential in the domain.

Main paper: 
 introduction: 
 Over the past two decades, deep neural networks have demonstrated exceptional success in various vision tasks, attributed to the existence of high-precision, large-scale datasets such as ImageNet-1K. However, collecting high-quality labels for such datasets is generally time-consuming and labor-intensive. To mitigate the cost, an alternative is automatic labeling (e.g. "webly-labeled" dataset by web-crawling the images and labels). While reducing the cost of manual labeling, it inevitably leads to low-quality noisy labels.To address the problem of label noise, a variety of methods have been proposed. Some methods, aim to develop robust loss functions [8, 11, 29, 41, 43, 54, 60, 64] or noise transition matrix [12, 15, 26, 34, 48, 53] . However, in practice, these methods are often sub-optimal dealing with high noise ratio and complicated noise.More recently, methods based on sample selection [19-21, 32, 33, 39, 42, 45, 58] to filter out samples with noisy labels become perhaps the dominant paradigm. For example, the most common sample selection strategy is the 'small-loss' mechanism motivated by the memorization effect, that is, the model tends to fit clean samples earlier than noisy samples in the training process thus resulting in relatively smaller losses for the clean ones. Following this, most of methods focus primarily on improving sample selection mechanisms, including different variants of 'small-loss' strategy [1, 23, 49] , and utilizing kNN [2, 7, 31] or graph models [46, 47] based on samples' feature space for sample selection. However, these methods are inherently affected by the label noise as they still rely on the current in-training model, leading to the infamous self-confirmation bias. Some methods [13, 56] attempt to alleviate self-confirmation bias through model co-training, but this approach noticeably introduces additional computational overhead. Moreover, these methods solely rely on the visual information within the images, which can readily lead to biased sample selection outcomes, as exemplified in cases of 'hard noise' -noisy sample exhibits a highly visual similarity with its incorrectly labeled class, as illustrated in fig. 1 .To address the aforementioned issues, this paper proposes utilizing popular vision-language model -CLIP [35] , for sample selection. Like any pre-trained models, CLIP is unaffected by the label noise in the collected dataset thus avoiding 'self-confirmation' bias. More importantly, CLIP's distinctive language modality and zero-shot classifier allow us to compensate for the biases that may arise from solely relying on visual information for sample selection. For instance, this allows us to identify 'hard noise' (fig. 1 ) that is difficult to distinguish using only the vision modality.To the best of our knowledge, we are the first to employ a large-scale vision-language model, particularly leveraging its language modality, for sample selection. Specifically, we simultaneously utilize CLIP's zero-shot classifier and an easilyinducible classifier based on noisy labels and CLIP's vision encoder. We name this method CLIPCleaner and theoretically and empirically demonstrate its effectiveness and unique advantages. To ensure the efficiency of the method and facilitate seamless comparison with existing methods, unlike common transfer learning techniques such as model fine-tuning [9] , knowledge distillation [44] , and prompt-based learning [3, 62] , CLIPCleaner does not involve training/fine-tuning the CLIP model. Furthermore, to evaluate the performance of CLIPCleaner on existing datasets, we introduce a minimal semi-supervised learning method tailored for noisy datasets, namely MixFix. In detail, we gradually introduce ignored clean samples and re-label noisy samples to expand the initial clean subset selected by CLIPCleaner.By leveraging CLIPCleaner and MixFix we establish a simple two-step framework for LNL: initiating with sample selection using CLIPCleaner and then perform semi-supervised learning solely using MixFix. Compared to existing methods involving iterations of sample selection and model training, our approach features a simpler structure and aligns better with end-toend training logic when the noise information in the dataset is unknown. Moreover, CLIPCleaner can serve as a plug-in module for existing methods, which further shows great potential of CLIP in learning with noisy labels. Despite its simplicity, our method achieves competitive and superior performance on various datasets, including CIFAR10/CIFAR100 with synthetic noise (symmetric, asymmetric, and instance-dependent), as well as real-world noisy datasets like Red Mini-ImageNet, WebVision, Clothing1M, and ANIMAL-10N. 
related works: 
 Sample selection for noisy dataset. Most of the recent sample selection methods do so, by relying on the in-training model, for example the per-sample losses [1, 13, 18, 23] or model predictions [30, 38, 55] . A few works focus on further improving the sample selection quality by modelling the loss with markov process [49] or dynamically select samples with multiple metrics [63] . In addition to selecting samples based on the model classifier, some works also try to utilize the feature representations for sample selection. Wu et al. [46] and Wu et al. [47] try to build a kNN graph and identify clean samples through connected sub-graphs, while Feng et al. [7] and Ortego et al. [31] propose to utilize a simpler kNN in feature space to alleviate the effect of noisy labels. Some recent methods involving contrastive learning also identify clean sample pairs based on neighborhood relationships in the feature space [25] or fit Gaussian distributions to model the clean distribution [16] . However, these methods remain unstable and prone to self-confirmation bias, especially in strong noise scenarios, due to its intrinsic reliance on the in-training model based on noisy dataset.Utilization of auxiliary model. To alleviate self-confirmation bias, the utilization of an auxiliary noise-free model is reasonable and straightforward. Related to us, some methods also try to use pretrained noise-free models for learning with noisy labels. Cheng et al. [6] , Zheltonozhskii et al. [61] propose to utilize self-supervised pretraining since it can learn good representations in the label-free case. Bahri et al. [2] utilize the pre-logit space of the pretrained model along with the kNN classifier for sample selection. Zhu et al. [65] follow the same idea and also involve CLIP, but they only utilize its vision encoder as a common pretrained encoder without utilizing the language encoder. We emphasize that language modality is critical as a supplementary modality. 
method: 
 In section 3.1, we cast the learning with noisy labels problem in a formulation that covers mainstream sample selection methods. We also provide essential details about the CLIP model. In section 3.2, we elaborate our sample selection method, namely CLIPCleaner. In section 3.3, we introduce our semi-supervised learning method, namely MixFix. In section 3.4 , we theoretically analyze the unique advantage of using CLIP for sample selection over common pretrained models. In section 3.5, we provide further discussions on the topics of sample selection and the use of the CLIP model for LNL.Sample selection with noisy labels. Given a dataset of training samples (𝒙 𝑖 , 𝑦 𝑖 ) 𝑁 𝑖=1 i.i.d sampled from a noisy joint distribution 𝑃 (𝒙, 𝑦) with support as sup(𝑃) = {𝒙 ∈ R 𝐶 ×𝐻 ×𝑊 , 𝑦 ∈ {1, ..., 𝐾 }} where 𝐾 denotes the number of semantic classes, the goal of our method is to learn a classifier 𝑓 that can accurately predict the true labels 𝑦 for new, unseen examples. Let us denote the clean joint distribution as 𝑃 𝑡𝑟𝑢𝑒 (𝒙, 𝑦). Most sample selection methods aim to approximate and optimize the unbiased empirical risk of 𝑓 on the clean joint distribution 𝑃 𝑡𝑟𝑢𝑒 (𝒙, 𝑦) with samples from noisy joint distribution 𝑃 (𝒙, 𝑦): R𝑡𝑟𝑢𝑒 (𝑓 ) = 1 𝑁 𝑁 𝑖=1 𝑤 𝑖 𝐿(𝒙 𝑖 , 𝑦 𝑖 ; 𝑓 ), where 𝑤 𝑖 are the sample weights.Particularly, with optimal weights (𝑤 𝑖 = 𝑃 𝑡𝑟𝑢𝑒 (𝑦 𝑖 |𝒙 𝑖 )/𝑃 (𝑦 𝑖 |𝒙 𝑖 )) we can achieve risk-consistent learning1 . However, since 𝑃 𝑡𝑟𝑢𝑒 (𝑦 𝑖 |𝒙 𝑖 ) and 𝑃 (𝑦 𝑖 |𝒙 𝑖 ) are typically both unknown for 𝒙 𝑖 , the objective of sample selection methods often revolves around estimating these two to subsequently estimate the optimal weights. In general, the noisy label 𝑦 𝑖 can serve as a confident proxy of the noisy distribution 𝑃 (𝑦 𝑖 |𝒙 𝑖 ), making our focus on utilizing an additional auxiliary classifier P (𝑦 𝑖 |𝒙 𝑖 ) to estimate 𝑃 𝑡𝑟𝑢𝑒 (𝑦 𝑖 |𝒙 𝑖 ). In addition, it is commonly accepted to restrict the weights as binary since for most classification datasets, 𝑃 𝑡𝑟𝑢𝑒 (𝑦 𝑖 |𝒙 𝑖 ) tends to be highly centered around only one class. As a result, the optimal weight 𝑤 𝑖 usually leans towards either 0 or 1 for most samples. Here, we propose a concise form sufficient to comprehensively represent most existing sample selection methods:EQUATIONwhere G denotes a specific sample selection mechanism, such as the 'small loss' strategy, to further refining the estimation.CLIP. We briefly introduce the CLIP model [35] , which is currently one of the most prevalent vision-language models. CLIP aims to learn from a dataset of image-text pairs, denoted as (𝒙 ′ 𝑖 , 𝒛 𝑖 ) 𝑀 𝑖=1 (we use 𝑥 ′ here for the CLIP training images to discriminate from above in-question dataset), which is i.i.d. sampled from a hidden joint distribution 𝑄 (𝒙, 𝒛) with support as sup(𝑄) = {𝒙 ∈ R 𝐶 ×𝐻 ×𝑊 , 𝒛 ∈ R 𝑑 }. We have below as CLIP training loss:𝐿(𝒙 ′ 𝑖 , 𝒛 𝑖 ; 𝑔, ℎ) = 1 2 ( -log exp(𝑔(𝒙 ′ 𝑖 ) 𝑇 ℎ(𝒛 𝑖 )) 𝑀 𝑗=1 exp(𝑔(𝒙 ′ 𝑖 ) 𝑇 ℎ(𝒛 𝑗 )) -log exp(𝑔(𝒙 ′ 𝑖 ) 𝑇 ℎ(𝒛 𝑖 )) 𝑀 𝑗=1 exp(𝑔(𝒙 ′ 𝑗 ) 𝑇 ℎ(𝒛 𝑖 ))).(EQUATIONHere, 𝑔 and ℎ denote the vision and language encoder, respectively.Intuitively, the CLIP model tries to maximize the correspondence between relatedrelated image-text pairs.In this section, we propose a new sample selection method based on CLIP, namely CLIPCleaner. According to eq. ( 1), our method (actually nearly all sample selection methods) is divided into two main steps: 1. estimate P (𝑦 𝑖 |𝒙 𝑖 ); 2. calculate weight w𝑖 with specific G. To enable the analysis between text 𝒛, image 𝒛 and label 𝑦, we consistent the notations for CLIP's training dataset and the in-question noisy dataset. Specifically, we extend the in-question noisy dataset to be i.i.d sampled from 𝑃 (𝒙, 𝑦, 𝒛) (actually from its marginalization), where sup(𝑃)= {𝒙 ∈ R 𝐶 ×𝐻 ×𝑊 , 𝑦 ∈ [0, 1, . . . , 𝐾], 𝒛 ∈ R 𝑑 };similarly, we extend the sampling distribution of CLIP's training dataset to 𝑄 (𝒙, 𝑦, 𝒛). Here we assume sup(𝑃) ⊂ sup(𝑄).). We consider two options for estimation: directly utilizing CLIP's zero-shot classifier, or, ignoring CLIP's language modality and treating its vision encoder as a regular pretrained model and training a new classifier atop it with in-question noisy dataset.Option 1: Estimate P (𝑦 𝑖 |𝒙 𝑖 ) with CLIP zero-shot classifier. Firstly, we assume the causal mechanism for 𝑃 and 𝑄 as: 𝒙 → 𝒛 → 𝑦 where 𝒛 denotes the description text and 𝑦 denotes the semantic label thus we have 𝑦 ⊥ 𝒙 | 𝒛. Roughly speaking, we assume that the semantic label 𝑦 𝑖 can be independently generated based on a decent image description 𝒛 𝑖 alone for each image 𝒙 𝑖 . We thus have: EQUATION𝑄 (𝑦 𝑖 |𝒛 = 'A photo of class name of 𝑦 𝑖 .') ≈ 1.Here, single prompt actually corresponds to sampling a single 𝑧.Obviously, it is plausible that with more high-quality samplings of 𝒛 𝑖 instead of only utilizing one single prompt the estimation would be better. In this work, we propose below template to generate multiple prompts {P 𝑗 } 𝐽 𝑗=1 using class-specific features2 :P 𝑗 ='A photo of {class name of 𝑦 𝑖 }, which is/has {class-specific feature 𝑗 of class 𝑦 𝑖 }.'Then we can simplify eq. ( 3) with above prompts as below:EQUATIONOption 2: Estimate P (𝑦 𝑖 |𝒙 𝑖 ) with CLIP vision encoder and noisy dataset. By treating the CLIP model as an ordinary large-scale pretrained model, we can also leverage its vision encoder 𝑔 solely along with the in-question noisy dataset (𝒙 𝑖 , 𝑦 𝑖 ) 𝑁 𝑖=1 to train a new classifier 𝑓 ′ (to discriminate it with the original classifier 𝑓 in section 3.1) for estimation. With the common cross-entropy loss, it is straightforward that the normalized prediction logits serve as an estimate of P (𝑦|𝒙): EQUATIONG 𝑙𝑜𝑠𝑠 = 1(P(-log P (𝑦 = 𝑦 𝑖 |𝒙 𝑖 ) ∈ GMM 𝑠𝑚𝑎𝑙𝑙 ) ≥ 𝜃 𝑙𝑜𝑠𝑠 ).For G 𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦 , we calculate a consistency measure (defined as the ratio of the probability of noisy label class to the highest class probability) and select samples with high consistency:G 𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦 = 1( P (𝑦 = 𝑦 𝑖 |𝒙 𝑖 )/max 𝑘 P (𝑦 = 𝑘 |𝒙 𝑖 ) ≥ 𝜃 𝑐𝑜𝑛𝑠 ).To evaluate our method on widely-acknowledged benchmarks, we propose a simple semi-supervised learning method for noisy dataset -namely MixFix. Please note, the notations employed in this section are defined independently. Specifically, we denote the selected subset and non-selected subset as (X 𝑐 , Y 𝑐 ) and (X 𝑛 , Y 𝑛 ). Motivated by pseudo-labelling [22] and FixMatch [37] , we then inspect each sample's current prediction 𝒑 𝑖 in non-selected subset with:EQUATIONHere we denote as 𝑝 𝑚 ≜ max 𝑙 𝒑 𝑖 (𝑙) and 𝑦 𝑚 ≜ arg max 𝑙 𝒑 𝑖 (𝑙).Intuitively, we 'absorb' ignored clean samples (𝑦 𝑖 = 𝑦 𝑚 ) and 'relabel' noisy samples (𝑦 𝑖 ≠ 𝑦 𝑚 ) with different thresholds in non-selected subset, and progressively append it to initial selected subset to form a dynamic larger training set. Different from existing semisupervised learning techniques, we typically set 𝜃 𝑟 ≤ 𝜃 ′ 𝑟 . This helps us make full use of noisy labels to differentiate the 'absorb' and 'relabel' process. To further counter the class imbalance in this new training set, the minority class is over-sampled. Then, we apply a common cross-entropy loss for training with this expanded and class-balanced training set, along with Mixup interpolation [57] . The detailed process is presented in Algorithm 1.Please note, with selected subset only, CLIPCleaner can also be utilized along with existing methods -see Supplementary C for more results. The rationale behind formulating the MixFix method, tailored explicitly for noisy datasets, stems from our belief that in scenarios where noise information remains unknown, an end-toend learning approach is not only more efficient but also stands out as an intuitive and primary choice. This is in contrast to the common style of iterative sample selection and model training. Considering above proposed two options for CLIPCleaner, an immediate question is: how does the zero-shot classifier (eq. ( 4)) compare to the trained classifier (eq. ( 5)) in estimating P (𝑦|𝑥). If the latter demonstrates comparable or even superior performance to the former, there may be little incentive to employ the CLIP model for sample selection. Rather, pursuing further enhancements to existing large-scale visual-only pre-trained models may yield greater potential. To this end, we conduct a theoretical analysis and compare the distances between the estimated P (𝑦 𝑖 |𝒙 𝑖 ) and true 𝑃 𝑡𝑟𝑢𝑒 (𝑦 𝑖 |𝒙 𝑖 ) of the two options. Specifically, following previous notations, we have below theorems: Theorem 3.1 (Estimation with zero-shot classifier). Let G, H be the hypothesis space of vision encoder 𝑔 and language encoder ℎ. Let us denote the rademacher complexity as ℜ(G • H ) of the combined CLIP model. Supposing the range of 𝐿 from eq. ( 2) as [0, 𝑙 𝑐𝑙𝑖𝑝 ∞ ] for all (𝒙, 𝒛) in sup(𝑄). Then, for any 𝛿 > 0, with probability at least 1 -𝛿 we have the following holds:𝑑 ( P𝑧𝑒𝑟𝑜𝑠ℎ𝑜𝑡 (𝑦 𝑖 |𝒙 𝑖 ), 𝑃 𝑡𝑟𝑢𝑒 (𝑦 𝑖 |𝒙 𝑖 )) ≤ 𝜀 𝑑𝑜𝑚𝑎𝑖𝑛 +Δ( 𝜆 1 ℜ(G • H ) + 𝜆 2 𝑙 𝑐𝑙𝑖𝑝 ∞ √︂ log 1/𝛿 𝑀 + 𝜆 3 𝜀 𝑛 ) with 𝜆 1 , 𝜆 2 , 𝜆 3 > 0.Here, 𝜀 𝑑𝑜𝑚𝑎𝑖𝑛 denotes the bias term induced by the domain gap between 𝑄 and 𝑃 𝑡𝑟𝑢𝑒 , and Δ ≥ 1 denotes the bias coefficient induced in designing prompts and sampling in eq. (3). ] for all (𝒙, 𝑦) in sup(𝑃). Then, for any 𝛿 > 0, with probability at least 1 -𝛿 we have the following holds:𝑑 ( P𝑡𝑟𝑎𝑖𝑛𝑒𝑑 (𝑦 𝑖 |𝒙 𝑖 ), 𝑃 𝑡𝑟𝑢𝑒 (𝑦 𝑖 |𝒙 𝑖 )) ≤ 𝜀 𝑛𝑜𝑖𝑠𝑒 + 𝜆 1 ℜ(F ) + 𝜆 2 𝑙 𝑛𝑜𝑖𝑠𝑦 ∞ √︂ log 1/𝛿 𝑁 with 𝜆 1 , 𝜆 2 > 0.Here, 𝜀 𝑛𝑜𝑖𝑠𝑒 denotes the difference term induced by the distribution difference between 𝑃 and 𝑃 𝑡𝑟𝑢𝑒 .Please refer to Supplementary F for full derivation. With theorem 3.1 and theorem 3.2, ignoring the uncontrollable and common optimization bound error terms (marked in gray), we confirm that the zero-shot classifier estimation is highly related to domain gap and prompts quality while the trained classifier estimation is affected by the noise of in-question dataset, which is intuitively consistent with our expectation. We also empirically verify that the higher the noise ratio, the greater the performance advantage of zero-shot classifier over the trained classifier (section 4.2). More importantly, 𝜀 𝑛𝑜𝑖𝑠𝑒 is always inevitable while Δ can be easily improved with better prompt engineering and 𝜀 𝑑𝑜𝑚𝑎𝑖𝑛 can be also reduced by training CLIP with more abundant dataset and thus minimizing the domain gap.To be greedy or conservative? For all sample selection methods, an inevitable challenge is how to balance the precision and recall of sample selection. In this paper, we introduce two different classifiers for estimation and two distinct sample selection strategies. Theoretical analysis and subsequent experiments indicate that these different classifiers and selection strategies exhibit their own preferences. In this study, we adopt a conservative sample selection strategy by taking the intersection of different sample selection outcomes, prioritizing the precision of sample selection. Compared to more greedy sample selection strategies, we lean towards relying on the semi-supervised learning strategy -MixFix to gradually introduce more samples into training. This can avoid magnifying the influence of noisy samples due to excessively greedy sample selection, but it also has obvious weaknesses, that some 'hard' clean samples will inevitably be missed. We leave the exploration of the optimal sample selection strategy to future work.To fully explore CLIP?. The utilization of the CLIP model for learning with noisy labels remains an area that requires further investigation. To ensure a fair comparison with existing work, we adopt standard sample selection paradigm, refraining from training or fine-tuning the CLIP model [3, 62] . In fact, the current prominent research directions related to CLIP involve fine-tuning the model, specifically through prompt-based learning. However, as expected, recent work (CoOp) has indicated that direct fine-tuning CLIP with noisy datasets can yield poorer performance compared to the initial zero-shot classifier. Therefore, in addition to sample selection, incorporating established techniques for LNL into prompt-based learning with CLIP may also offer promising directions. 
experiments 4.1 experiment details: 
 4.1.1 Dataset details. CIFAR10 and CIFAR100 datasets comprise 50,000 images. Following established conventions, we assess our method's performance with two types of artificial noise:"symmetric noise, " wherein labels are randomly flipped across all samples using a uniform distribution, and "asymmetric noise," wherein labels of visually similar categories, such as Horse ↔ Deer and Dog ↔ Cat, are randomly interchanged. Moreover, we conduct experiments with various noise levels: 20%, 50%, 80% and 90% symmetric noise, as well as 40% asymmetric noise, adhering to the settings in Di-videMix ( [23] ). For instance-dependent noise, we utilize the label noise file provided by [4] .Red Mini-ImageNet dataset [17] is a real-world dataset containing a total of 100 categories. It is an extension of the Mini-Imagenet dataset, where noise is introduced at varying ratios. Specifically, noisy images and their respective labels are obtained by crawling the internet, and these noisy images replace the original images in the Mini-ImageNet dataset, with different noise ratios. To ensure a fair comparison with previous studies [10, 52] , the images are resized from their original size of 84×84 pixels to 32×32 pixels. Moreover, in accordance with the existing literature [10, 52] , we utilize noise ratios of 20%, 40%, 60%, and 80%.WebVision [27] is an extensive dataset comprising 1,000 classes of images obtained through web crawling. In line with previous studies [18, 23, 31] , we evaluate our methods using the top 50 classes from the Google Subset of WebVision. The estimated noise ratio for this subset is approximately 20%. [38] is a recently introduced real-world noisy dataset comprises 10 classes of animals. The dataset has undergone manual labeling, with an estimated label noise ratio of around 8%. Similar to the CIFAR datasets, ANIMAL-10N consists of 50,000 training images and 10,000 test images.Clothing1M [50] is a large-scale dataset containing 14 classes of clothing images, obtained by crawling online shopping websites. It consists of a substantial collection of 1 million noisy images. The estimated noise ratio for this dataset is approximately 38.5%.4.1.2 Implementation details. We use CLIP model with VIT-B/32 backbone in all experiments except for specific ablations. In all experiments, our default approach is CLIPCleaner + MixFix (Ours). By default, we train the network with a SGD optimizer with a momentum of 0.9 in all experiments.For CIFAR10 and CIFAR100, we use a PresActResNet-18 [14] as the backbone in all experiments following previous works. For CIFAR10, we set 𝜃 𝑙𝑜𝑠𝑠 = 0.5, 𝜃 𝑐𝑜𝑛𝑠 = 0.8 for CLIPCleaner and 𝜃 𝑟 = 0.8, 𝜃 ′ 𝑟 = 0.9 for MixFix; For CIFAR10, we set 𝜃 𝑙𝑜𝑠𝑠 = 0.5, 𝜃 𝑐𝑜𝑛𝑠 = 0.8 for CLIPCleaner and 𝜃 𝑟 = 0.7, 𝜃 ′ 𝑟 = 0.8 for MixFix. We train both networks with for 300 epochs with a weight decay of 5e-4. The initial learning rate is 0.02 and is controlled by a cosine annealing scheduler. The batchsize is fixed as 128.For Red Mini-ImageNet, we also use a PresActResNet-18 [14] as the backbone following previous works [10, 52] . For CLIPCleaner, we set 𝜃 𝑙𝑜𝑠𝑠 = 0.5, 𝜃 𝑐𝑜𝑛𝑠 = 0.8. For MixFix, we set 𝜃 𝑟 = 0.8, 𝜃 ′ 𝑟 = 0.95. We train the network for 300 epochs with a weight decay of 5e-4. The initial learning rate is 0.02 and reduced by a factor of 10 after 200 and 250 epochs. The batchsize is fixed as 64.For WebVision, we use a InceptionResNetv2 as the backbone following [23] . For CLIPCleaner, we set 𝜃 𝑙𝑜𝑠𝑠 = 0.5, 𝜃 𝑐𝑜𝑛𝑠 = 1. For MixFix, we set 𝜃 𝑟 = 0.7, 𝜃 ′ 𝑟 = 1.0. We train the network for 150 epochs with a weight decay of 1e-4. The initial learning rate is 0.01 and reduced by a factor of 10 after 80 and 120 epochs. The batchsize is fixed as 32.For Clothing1M, we use a ResNet50 as the backbone following [23] with ImageNet pretrained weights. For CLIPCleaner, we set 𝜃 𝑙𝑜𝑠𝑠 = 0, 𝜃 𝑐𝑜𝑛𝑠 = 0.5. For MixFix, we set 𝜃 𝑟 = 0.7, 𝜃 ′ 𝑟 = 1.0. We train the network for 150 epochs with a weight decay of 1e-3. The initial learning rate is 0.002 and reduced by a factor of 10 after 50 and 100 epochs. The batchsize is fixed as 32.For ANIMAL-10N, we use a VGG-19 [36] as the backbone with batch-normalization following [38] . For CLIPCleaner, we set 𝜃 𝑙𝑜𝑠𝑠 = 0.5, 𝜃 𝑐𝑜𝑛𝑠 = 0.8. For MixFix, we set 𝜃 𝑟 = 0.7, 𝜃 ′ 𝑟 = 0.99. We train the network with SGD optimizer for 300 epochs with a momentum of 0.9 and weight decay of 5e-4. The initial learning rate is 0.02 and reduced by a factor of 10 after 150 and 250 epochs. The batchsize is fixed as 128.Hyper-parameters w.r.t MixFix. In this section, we ablate on the only two hyperparameters of our semi-supervised training strategy MixFix: the 'absorb' threshold 𝜃 𝑟 and the 'relabel' threshold 𝜃 ′ 𝑟 . Owing to the precision-recall dilemma when doing sample selection, here we also need to weigh the precision and recall when introducing additional training samples. In table 1 we demonstrate that under different noise ratios, a too high or too low threshold leads to performance degradation, and 𝜃 𝑟 < 𝜃 ′ 𝑟 leads to better performance than setting same value for both thresholds. In fig. 2 , we further reveal the inherent mechanism. Especially, after reducing the 'absorb' threshold 𝜃 ′ 𝑟 , the proportion of training samples increases and the accuracy of training samples decreases.Analyzing CLIP Zero-shot classification as a baseline. In this section, we consider utilizing CLIP's zero-shot classifier directly on the clean test set, following a procedure that we describe in Section 3.2. In table 2, we present the zero-shot classification results on six common benchmarks and compare them with current SOTA results as well as our own method. It's worth noting that CLIP is utilized with the VIT-B/32 architecture here, while our method and the SOTA methods adopt simpler structures, such as PreResNet-18 for the CIFAR dataset. Therefore, this comparison is indeed 'over stringent'. Even though, we observe that, when compared to directly utilizing CLIP's zero-shot classifier, our method delivers significantly improvements on most datasets and outperforms the SOTA LNL methods on all datasets. We also consider other vision-language models other than CLIP in Supplementary A.Analyzing sample selection w.r.t different classifiers and different mechanisms. In section 3.4, we theoretically conclude that the performance of the zero-shot classifier is influenced by the quality of utilized prompts and the domain gap between CLIP training dataset and the in-question noisy dataset, while the performance of the easily-inducible classifier trained based on CLIP's vision encoder and the in-question noisy dataset is influenced by the noise of the in-question dataset. To validate this, we empirically test with two datasets with controllable noise ratios, that is, the CIFAR10/100 dataset with synthetic noise and the Red Mini-ImageNet dataset with real-world noise.In fig. 3 , we show the sample selection performance and find that: i) As the noise ratio increases, regardless of the dataset, noise types, CLIP backbones or empirical variants of the trained classifier in option 2 (LogisticRegression VS kNN ), the zero-shot classifier (option 1) gradually outperforms the trained classifier. This further validates our theoretical findings in section 3.4; ii) Additionally, we notice that when comparing two different modes for obtaining the training classifier, the LogisticRegression classifier empirically exhibits superior performance to the kNN classifier. Therefore, we choose the LogisticRegression classifier as our default choice for trained classifier; iii) Furthermore, we find that different sample selection mechanisms (G 𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦 VS G 𝑙𝑜𝑠𝑠 ) show distinct advantages and disadvantages on different datasets. Given that noise information is typically unknown in real-world scenarios, as analyzed in section 3.5, we default to a conservative sample selection strategy, which involves utilizing both sample selection strategies and choosing their intersection as final selected subset. In this section, we firstly evaluate our method on the CIFAR datasets with synthetic symmetric/asymmetric noise. In table 3 , We can see that our method gets competitive and better performance in all experiment settings, especially when the noise ratio is high (63.11% testing accuracy with 90% symmetric noise on CIFAR100 dataset). Also, we would like to emphasize that we keep hyper-parameters fixed for all experiments here as we believe the method robustness in a noise agnostic scenario is critical.To further validate the performance of our method in handling the 'hard noise', we also conduct experiments on instance-dependent noise in table 5. Different from symmetric or asymmetric noise, instance-dependent noise assumes that semantic-similar samples are more prone to get mislabelled, aligning better with our earlier definition of 'hard noise'. Besides, here we here exclude MixFix and employ the selected samples for training with cross-entropy loss solely. This exclusion serves to provide an additional proof of the superior sample selection performance of CLIPCleaner. 8 we show results on the ANIMAL-10N, Red Mini-ImageNet and WebVision datasets, respectively. In summary, our proposed method demonstrates substantial improvements compared to the current state-of-the-art approaches on both large-scale web-crawled datasets and small-scale humanannotated noisy datasets. We note, that the proposed CLIPCleaner can also be used in combination with other schemes. In table 4 we show results on the Clothing1M dataset both with our default setting (CLIPCleaner + MixFix) and with it incorporated to two additional schemes: first incorporating our method with co-training, and second replacing MixFix with DivideMix [23] . We observe that we obtain results that are superior to the current state-of-the-art. Meanwhile, we would like to note that the majority of existing methods have small differences on the Clothing1M dataset despite the fact that they have large performance differences on other datasets. This suggests that additional training techniques may have a greater impact than [38] 81.8 PLC [59] 83.4 NCT [5] 84.1 InstanceGM [10] 84.6 SSR+ [7] 88.5Ours 88.85±0.61sample selection methods on this specific dataset, possibly due to the fact that the Clothing1M dataset is more fine-grained than other datasets. For such fine-grained noisy datasets, sample selection may not be the optimal strategy, as suggested in Section 3.1, where the basis of sample selection methods relies on highly concentrated conditional probabilities for the samples (eq. ( 1)). 
conclusion: 
 To mitigate the issues of 'self-confirmation bias' and compensate for visual-only modality in current mainstream sample selection methods, in this paper we propose a method utilizing the largescale vision-language model CLIP for sample selection, called CLIP-Cleaner. We substantiate its effectiveness through both theoretically and empirically. Furthermore, we introduce a straightforward semi-supervised learning method tailored for noisy datasets, called MixFix, without the need for intricate off-the-shelf techniques. We emphasize that the exploration of utilizing vision-language models for noisy datasets, such as the potential of existing prompt learning techniques, remains an open direction. Additionally, the possibility of a large domain gap between the CLIP model and the target dataset can influence results, indicating a need for more refined vision-language models. Lastly, our experiments suggest that sample selection methods may not be optimal for fine-grained noisy datasets, which presents itself also as one of our future research directions. 
 
 Appendix: Please refer to Supplementary E for details. We omit the variables for brevity, e.g, 𝑃 (𝑦 = 𝑦 𝑖 |𝒙 = 𝒙 𝑖 ) as 𝑃 (𝑦 𝑖 |𝒙 𝑖 ).Please refer to Supplementary B for more details about how to generate prompts. 


Decision: The manuscript received three acceptance ratings, indicating that all reviewers agreed the manuscript meets the bar of MM.

There is thus no basis to overturn the consensus. The AC recommends the submission to be accepted.

Please, however, do account for the comments in the final version. Congrats!

Reviews: review: paper_summary: paper_summary: The paper tackles the challenge of label noise learning by utilizing multimodal CLIP and introducing a novel strategy called MixFix. Traditional methods relying solely on visual information often result in biased sample selection, particularly with 'hard noise'. To address this, the authors propose using CLIP, a vision-language model, for more accurate sample selection. They present CLIPCleaner, which combines CLIP’s pre-trained zero-shot classifier and a classifier based on CLIP’s vision encoder with noisy labels, facilitating effective offline sample selection. MixFix enhances semi-supervised training by incorporating absorption and relabeling techniques. The authors support their approach with both theoretical and empirical evidence, demonstrating the advantages of their method over the others.
strength_weakness: strength_weakness: The paper is well written and filled with extensive experiments and theoretical results, showcasing the authors' rigorous approach. Each chapter is meticulously detailed, ensuring clarity and depth in the presentation of their work. The authors thoroughly explain the results for each table, emphasizing special cases to provide deeper insights and highlight the practical implications of their findings. This careful attention to detail and comprehensive analysis adds significant value to the study, making it a robust and insightful contribution to the field of label noise learning.
limitations: limitations: To compare with CLIP zero-shot in Table 2, it would be better to use the same architecture for consistency. In each table, instead of simply listing SOTA, it would be more informative to specify the methods that achieved those results. Additionally, there are typo errors in lines 541, 286, 275, and 267 that need correction. Another recent work, "Enhancing Noisy Label Learning Via Unsupervised Contrastive Loss with Label Correction Based on Prior Knowledge," also uses CLIP for label noise learning and appears to achieve better results. Comparing the current method with this recent work could provide a more comprehensive evaluation on CLIP-based LNL methods.
paper_summary: paper_summary: The authors leverage the large-scale vision-language model CLIP for sample selection, introducing CLIPCleaner. The proposed CLIPCleaner can serve as a plug-in module to existing methods, consistently delivering performance improvements.
strength_weakness: strength_weakness: - Employing pretrained vision-language models, such as CLIP,  to learn with noisy labels is reasonable and interesting.
- The experiments are conducted on various datasets, and the ablation study is comprehensive.
limitations: limitations: - The lack of description regarding the experimental setting. For instance, the choice of the sample selection method is not specified, as all six methods seem to have their advantages in certain situations as depicted in Figure 3. The paper does not clarify which sample selection method and what hyperparameters are used in the subsequent performance comparison.
- Missing references, such as CoOp on line 487, and SOTA in Table 2.
paper_summary: paper_summary: 1. This paper proposes addressing these LNL challenges by leveraging CLIP.
2. CLIPCleaner utilizes CLIP’s pre-trained zero-shot classifier along with a classifier based on CLIP’s vision encoder and the noisy labels themselves.
strength_weakness: strength_weakness: 1. The paper is the first to employ a large-scale vision-language model, particularly leveraging its language modality, for sample selection.
2. The method achieves competitive and superior performance on various datasets.
limitations: limitations: 1. The paper is not well-structured. More illustrative diagrams are needed to simplify the explanation of your method and pipeline.