Label-Retrieval-Augmented Diffusion Models for Learning from Noisy Labels
Authors: Jian Chen
Ruiyi Zhang
Tong Yu
Rohan Sharma
zhiqiang xu
Tong Sun
Changyou Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: diffusion model
label noise
retrieval augmented learning
TL;DR: We address learning from noisy labels by interpreting it from a generative model perspective and adopt the powerful diffusion model techniques for learning

Abstract: Learning from noisy labels is an important and long-standing problem in machine learning for real applications. One of the main research lines focuses on learning a label corrector to purify potential noisy labels. However, these methods typically rely on strict assumptions and are limited to certain types of label noise. In this paper, we reformulate the label-noise problem from a generative-model perspective, *i.e.*, labels are generated by gradually refining an initial random guess. This new perspective immediately enables existing powerful diffusion models to seamlessly learn the stochastic generative process. Once the generative uncertainty is modeled, we can perform classification inference using maximum likelihood estimation of labels. To mitigate the impact of noisy labels, we propose the **L**abel-**R**etrieval-**A**ugmented (LRA) diffusion model, which leverages neighbor consistency to effectively construct pseudo-clean labels for diffusion training. Our model is flexible and general, allowing easy incorporation of different types of conditional information, *e.g.*, use of pre-trained models, to further boost model performance. Extensive experiments are conducted for evaluation. Our model achieves new state-of-the-art (SOTA) results on all the standard real-world benchmark datasets. Remarkably, by incorporating conditional information from the powerful CLIP model, our method can boost the current SOTA accuracy by 10-20 absolute points in many cases. Code is available: https://anonymous.4open.science/r/LRA-diffusion-5F2F

Label-Retrieval-Augnnnetd Diffusion Models for Learning from Noisy Labels
Jian Chen'
Ruiyi Zhang² Tong Yu²
Rohan Sharma'
Zhiqiang Xu Tong Sun
Changyou Chen
University at Buffalo Adobe Research *MBZUAI {jchen378, rohan jg, changyouJ@buffalo. edu (ruizhang, tyu, tsun) @adobe. com zhiqiang xu@mbzuai. ac.
Abstract
Learning from noisy labels is long- oaggsstading problem in machine learning for real applications. One of the main research lines focuses on learning label corrector to purify potential noisy labels. However, these methods typically rely strict assumptions and are limited to certain types of label noise. In this paper, we reformulate the label-noise problem from generative-model perspective, i.e,, labels are generated by gradually refining an initial random guess. This new perspective immediately enables existing powerful diffusion models to seamlessly learn the stochastic generative process. Once the generative uncertainty is modeled, can perform classification inference using maximum likelihood estimation labels. To mitigate the impact of noisy labels, we propose -etrieev-ti Augmented (LRA) diffusion model which leverages neighbor consistency effectively construct pseudo-clean labels for diffusion training. Our model is flexible and general, allowing easy incorporation of different types of conditional information, e.g., use of pre-trained models, to further boost model performance. Extensive experiments state-of-the-art (SOTA) results on all standard real-world benchmark datasets. Remarkably, by incorporating conditional information from the powerful CLIP model, our method can boost the current SOTA accuracy by 10-20 absolute points in many cases.
conducted for evaluation. Our model achieves new
Introduction
Deep neural networks have achieved extraordinary accuracy various classification tasks. These models are typically trained through supervised learning using large amounts of labeled data. However, large-scale data labeling could cost huge amount of time and effort, and is prone enrors caused by human mistakes or automatic labeling algorithms 1 In addition, research has shown that the ability of deep neural network models to fit random labels can result in reduced generalization ability when learning with corrupted labels (2). Therefore, robust learning methods using noisy labels are essential for applying deep learning models cheap yet imperfect datasets for supervised learning tasks.
There are multiple types label noise investigated by previous research. More recent research has focused studying the more realistic ceature-dependerr label noise, where the probability of mislabeling a given instance depends on its characteristics. This type of noise is more consistent with label noise in real-world datasets 3
6 7). To address this type of noise, model is expected to be able to estimate the uncertainty of each training label. Many state-of-the-ee methods have primarily relied the observation that deep neural networks tend learn simple patterns before
Code is available at http::/phtbbbboommmuuuupaaaauu/a
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

noisy label YT
Yt
Yt-1
Yo ckan label
Label noising process
Label denoising prncess
Figure 1: Label denoising as a reverse noising process.
This means temporary phase exists in the learning process, where the model has learned useful features but has yet not started overfitting corrupt labels. At this stage, the model predictions can be used to modify the training labels, that they more consistent
memorizing random noise R 19.
with model predictions TO 12 13 14 [5]. By correctly modifying the labels, the number of clean training sample increases, which further benefit the training. These type of approaches, however, is inherently risky because the point at which the model starts to overfit varies with the network structure and dataset. Starting too early can corrupt the training data, while starting too late may not prevent overfitting [16]. Therefore, it is vital to carefully tune the hyper ppprmpprraeeees of the training strategy, such as the number of epochs for warm-up training, learning rate, and uncertainty threshold, to achieve successful training results.
Another class of methods adopts the assumption of label propagation in cemi-supervisee learning [8), where nearby data points in a feature space tend to have the same label. Therefore, they use neighbor consistency consistency"regulatiiin regularization to prevent overfitting of the model [9 20). The performance highly depends on the quality of the encoder that maps the data to the feature space, as retrieving neighbor that belongs to a different class could further mislead the training process. Encoders are therefore required to first learn high-level features of the data that can used for classification, which could be trained simultaneously with the classifier using noisy labels. However, the training can also lead to overfitting or underfitting.
In this paper, by contrast, we formulate the label noise problem from generative-model perspective, which naturally provides new insights into approaching the problem. Our intuition to view the noisy labeling process as a stochastic label generation process. Thus, we propose adopt the powerful diffusion model as the generative building block. [ process, we start with noisy estimation of the label, then gradually refine it to recover the clean label, which is equivalent to the reverse denoising process of the diffusion model.
our intuition. In the generative
Specifically, the diffusion model takes noisy label and some useful conditional information (to be specified) as inputs, and learns to recover/generate the ground-truth labels as outputs. One challenge in this setting is that only noisy labels are available in practice for training. To overcome this issue, we adopt the principle of neighhor consistency, and propose label-retrieval augmentation to construct pseudo clean labels for diffusion model training, where a pre-trained image encoder is used to define worth noting that the pre-trained image encoder would not be affected by the label noise, because they can be trained in self-supervised manner [21 22 or an additional clean dataset 223] 24] In fact, pre-training can tremendously improve the model's adversarial robustness (25) and has been used improve model robustness label corruption [26]. Another merit of our design is that is general enough to allow natural incorporation of powerful large pre-trained model such the CLIP model to further boost the performance.
the neighborhood of a sample.
In addition, the probability nature diffusion models can also be hetter equipped handle uncertainty in the data and label, thus providing more robust and accurate predictions. We call our model I.RA-
diffusion label-retrieval-augmeen diffusion).
Our main contributions are summarized as follows:
We formulate learning from noisy labels modeling a stochastic process of conditional label generation, and propose to adopt the powerful diffusion model to learn the conditional label distribution.
Nearby data points tend to have the same label.

---

We incorporate the neighbor consistency principle into the modeling, and design novel diffusion model learn effectively from noisy label data.
We further improve model by incorporating auxiliary conditional information from large pre-trained models such as CLIP.
Our model achieves the new state-of-the-art (SOTA) in various real-world noisy label benchmarks, e.g,, 20% accuracy improvement on noisy CIFAR-100 benchmark.
2
Preliminary
Diffusion models were initially designed for generative modeling. Recently, it has been extended for classification and regression problems. this section, we introduce the Classification and Regression Diffusion Models (CARD) [2]. which our model is based on.
The CARD model transforms deterministic classification into conditional label generation process, allowing for more flexible uncertainty modeling in the labeling process (27). Similar to the standard diffusion model, CARD consists of forward process and reverse process. In the forward process, n n-dimensional one-hot label yo is gradually corrupted to a series intermediate random vectors II:T. which converges random variable with multi-varian Gaussian distribution (N (x), I) (latent distribution) after 7 steps, where the mean is defined by pre-trained ne-dimensional image encoder fa- The transition steps between adjacent intermediate predictions is modeled Gaussian distributions, q 1. fq) fq) = (y: pc: 5,I), with mean values " HT and a variance schedule 31. ,3r, where v B.y--1 + (1 v fx) This admits a closed-form sampling distribution, q Yo: fg) N (yr M4: (I-ā;)I, with an arbitrary timestep and μe vāryo + (1 ff The mean term can be viewed as an interpolation between true data yo and the mean of the latent distribution fg(x) with weighting term ā, II, 3.).
In the reverse (generative) process, CARD reconstructs a label vector Yo from an n-dimensional Gaussian noise yr ~ (fg (x), I) by approximating the denoising transition steps conditioned on the data point x and another pre-trained image encoder fp in an arbitrary dimension. The transition
step is also Gaussian for an infinitesimal variance 3, 28 (define B, i-āt-1 1-a,
B.):
pe p fp) N 11 fpr t), 3.1)
The diffusion model is learned by optimizing the evidence lower bound with stochastic gradient descent:
CELBO = =Eg Cr
Σ Li-1 + Lu t>1
where
(2)
Lo = logpo(yoly1, fp), 4--1 DKL (/t.://-/.........blbblbb------
LT DRL. (g(yrlyo, [g)lp(yy((sss))
Following 22). the mean term is written as M4 (Ye; x, fp: vay (x
B. B. vT-ar colye, x, Sp: 1)) and
the objective can be simplified to Lsimple l co x, fp. t)??
3 LabelRetrrivval-Aggeened Diffusion Model
Inspired by CARD, noisy labels as a stochastic process of conditional label generation (i.e., label diffusion) process. In this section. we first provide an overview of the our model in Section 3.1 and then introduce the proposed rerbber-rrriilll-agmeettte component in Section 3.2 which can leverage label consistency in the training data. Next, we introduce an accelerated label diffusion process to significantly reduce classification model inference time in Section 3.3 Finally, y, a new conditional mechanism is proposed to enable the usage of pre-trained models in Section
(LRA) diffusion models reframe learning from

---

3.1 Model Overview
Our overall label-reerievallangneneee diffusion model illustrated in Figure 2 where a diffusion model is adopted for progressively label denoising, by leveraging both the retrieved labels and auxiliary information from pre-trained models. Our model employs two pre-trained networks, denoted as Jg and Jp encoders, to encode conditional information that facilitates the generation process. The fy encoder serves as a mean estimator for Yr. providing an initial label guess for a given image. This encoder could be a standard classifier trained noisy labels. On the other hand, the fp encoder operates as a high-dimensions feature extractor, assisting in guiding the reverse procedure. Yı and (x) are concatenated together before being processed. Details about neural-netword architecture design are provided in Supplementary C
During training, use labels retrieved from the neighborhood as the generation target yo. Then, in the forward process, the distribution of neighboring labels is progressively corrupted towards standard Gaussian distribution centered at the estimated mean (x). During testing, we employ generalized DDIM method to efficiently compute the maximum likelihood estimation of yo-
Intent distribution
Diffusion model
Soewand process
Laite aimiico arighhore
g'yee L.fe)
Polys-iby. x.
N--1
Yo
reverse process
pre-trained eacoder
Conditional information
pre-trained eacoder
Label retrieval
Figure Overview of the proposed framework for improving learning performance from noisy labels. The figure depicts the three main components, including (1) using diffusion models to imitate and inverse the label noising process; (2) using pre-trained encoders (i.e, fg and fp) within the diffusion model, and (3) the label-retrien www.erievalluggmeeneeeeee consistency image labels.
approach using the f, encoder to encourage neighbor
3.2 Label-retrieval Augmentation for Training
The noisy nature of labels excludes the availability clean labels for training. To mitigate the issue, propose a training strategy based the concept of retrieval augmented learning [30 31], such that it is more resistant to label noise. Our main assumption is that in a latent space, data points from different classes form distinctive clusters. Therefore, the majority of a data point's 's neighbors are expected to have the same label as the point itself. To this end, we used a pre-trained encoder, illustrated as fp in Figure 2 map the data into embedding space and retrieve label y/ from labels of the & nearest neighbors (1) .... y (k)) in the training set. The diffusion model then trained to learn the conditional distribution p(s/ (x) of labels within the neighborhood, rather than the distribution of labels p(y|x) for the data point itself. We empirically select the value of & based on the KNN accuracy obtained on the validation data, as detailed in Supplementary Section [
Label-retrieval augmentation enables the model to make use of the information from multiple and potentially more accurate labels to improve its prediction performance. Algorithm| T describes the training procedure. Additionally, diffusion models are known to be effective at modeling multimodal distributions. By training the model to generate different labels from neighbors based on the same data point, the model can produce stochastic predictions based the distribution capture the uncertainty inherent in the data labeling process. As a result, the trained model can be used not only classifier, but also sampler that simulates the actual labeling process.
3.3 Efficient Inference with generalized DDIM
The iterative generation nature of the classification diffusion model makes its inference efficiency not comparable traditional classifiers. To enhance the inference efficiency, we propose to incorporate the efficient sampling methods, Denoising Diffusion Implicit Model (DDIM) (32,, to accelerate the

---

Algorithm Training
Input: training {X, Y. image encoder fp fq.
I: while not converged do
2: Sample data (x,y) y) ~ (x, Y time slice t {1.... T) and noise (
3: Retrieve labels {y(") , ........ y of neighbors in the feature space defined by the encoder fp
4: Sample y ~ {y. ,(,,... .... ( ), and convert it one-hot vector yo
5: Take gradient descent step on the loss:
||c - colvaryo + (1 vārlf(()) + VI- a,c, x, f(x),,,((x))
6: end while
label diffusion process. However, the utilization of the mean estimator Jq makes DDIM incompatible with our setting, as our generation process begins with a non-zero mean Gaussian distribution N (x), I). Therefore, we adjust the DDIM method into more general form that fits our framework. Analogous DDIM. our sampling process maintains the same marginal distribution the original (o: f4) closed form sampling process. Detailed derivations can be found in Supplementary A
With DDIM, the trained model generate label vector in much less steps following pre-defined sampling trajectory of {T = Ts >, Ti = 1}, where S < T. Consequently, Yr can be computed as:
Yo + (1 v (ā,))f(()) + Vā,ā, yr. Vo
(3)
where .N(O,), Similar to CARD 27,, predict the denoised label ÿo. prediction of yo given Yr,, as:
yu = Var,
lyr, - (1 V (f.)5((x) āi- x, (o fx().
(4)
When Ta-1 > O, we can compute y given yr, from the non-Markovian forward process defined as:
Yr, Vär, So Var, (x()) /1- 0t, x (x),
As the dimension of label vectors is usually much lower than that of an image, the model can employ fewer steps in the reverse process without compromising generative quality. In our experiments, we use S 10 and T 1000, substantially reducing the time cost of the classification process. Supplementary Figure B gives an example of the label generation (classification) process on the CIFAR-10 dataset.
To further enhance the inference efficiency, we propose a simple and effective trick for computing the maximum likelihood estimation of labels. As the generative process is deterministic given yr, which is sampled from a uni-modal Gaussian distribution, we approximate the maximum likelihood estimation of labels by initiating from the mean, i.e. yo DDIM(yr f,(x), x). This trick circumvents the time-consuming majority voting approximation that demands repeated generation.
3.4 Flexible conditioning with pre-trained encoders
The original CARD model employs single model for both the fp and fg encoders. However, this limits their representation capacity [33] as the dimension of (x( is typical relatively small, i.e. cqualling the number of classes. To mitigate this and improve model performance, we abandon the assumption that f fqz enabling the use of more powerful pre-trained encoder (e.g., the CLIP image encoder 24 with arbitrary dimensions for fp-
Empirically, we find that the model can still achieve satisfactory performance when the magnitude of (x) is small, which means the latent representation yr fa (x) + e dominated by the noise term e ~ (O, In this case, the information provided by f (x) to the diffusion process is limited. As result, we simply set (x) = 0 to avoid handling an additional n-dimensional fg encoder. For the fp encoder, one can employ flexible pre-trained models as presented in Section 5 In this paper, we use the SimCLR model trained on the training images (without supervised information) and the pre-trained CLIP model.
5

---

4
Related work
Robust loss function and regularization techniques. Several noise-robust loss functions and regularization techniques have been proposed as alternatives to the commonly used cross-entropy loss (CE), which is not robust to label noise. Mean absolute error (MAE) 34 loss has been shown to be Cross-Entropy
robust against noisy labels. Generalized Cross- Entropy (GCE) [35 combines CE and MAE for faster
convergence and better accuracy. Symmetric pross-entropy Learning (SL) [6] couples CE with a noise-robust counterpart and has been found to have higher performance than GCE, particularly for high noise rates. Label Smoothing Regularization [37 alleviates overfitting by linearly combining labels with uniform distribution. Bootstrapping technique (38] combines the labels with the
current model prediction. Dynamic bootstrapping (39 40) uses the prediction confidence to control
the weighting in the combination. Neighbor Consistency Regularization (NCR) [19] encourages consistency of prediction based on learned similarity. Our method is also based on the principle of neighbor consistency, However, instead of encouraging consistent predictions among neighbors, our model directly learns from the labels of neighbors. This allows for estimating instance-level uncertainty by learning the label distribution among neighbors, rather than learning point estimation.
Data recalibration. Data recalibration techniques progressively remove or correct mislabeled data during training to improve the reliability of training data. Wang et al. [II] used the learned similarity and label consistency identify and discard data with noisy labels. TopoFilter [1 selects clean data by analyzing the topological structures of the training data in the learned feature space. Cheng et al.
4 defines Bayes optimal classifier to correcct labels. Zheng al. 114 proposed using a likelihood ratio test (LRT) to correct training labels based on predictions. Zhang et al. [15] used LRT to correct labels progressively and provides a theoretical proof for convergence to the Bayes optimal classifier. Dividemix 42), LongReMix [3]. and CC 44
treat the low confident data as unlabeled, and then
employ semi- eepi-super learning algorithms 45 for further analysis. C2D [46] combines Dividemix with seff-supereise pre- training to boost its performance by improving the quality of the extracted features. Our approach employs the same assumption as TopoFilter that data belonging to the same class should be clustered together with ideal feature representations. However, our technique isn't 't confined to learned features potentially distorted by label noises. Instead, similar to C2D, our method can effectively leverage the high-quality feature learned by pre-trained encoders to achieve superior accuracy.
Guided diffusion model and retrieval augmentation. Guided diffusion is technique applied to diffusion models for conditional generation. Classifier guidance [47] is an cost-effective method leveraging the gradient of a classifier to steer the generative process of trained diffusion model. On the other hand, Classifier-free guidance [48] learns the conditional distribution during training for improved generation quality. This approach also allows for the use of continuous guidance information, such as embedding vectors, rather than being limited to discrete labels. Classification and Regression Diffusion Models (CARD) (27 formulates classification and regression as conditional generation task that generates labels or target variables conditioned on images. Our approach follows the same paradigm, and leverages the multi-modal coverage ability of diffusion models to learn the label distribution within the neighborhood. SS-DDPM [4] proposes a diffusion model with arbitrary noising distributions defined on constrained manifold, e.g., the probabilistic simplex for label generation. Retrieval- augmented diffusion models (30] used retrieved neighbors from an external database as conditional information to train diffusion models for image synthesis. Retrieval Augmented Classification (31 used retrieval-augmentation
train classification model using class- imbalanced training data. Our approach differs from theirs by retrieving labels instead of data to reduce label noise in training rather than increasing the training data. In addition, our model does not require an external database.
5 Experiments
We first evaluate the performance of our method on datasets with various types synthetic noises. Then, we perform experiments on four real-world datasets. To better understand the performance gain sources, we conduct ablation studies to measure the impacts of conditional diffusion and different pseudo-label construction strategies. All experiments were done on four NVIDIA Titan V GPUs. Comprehensive implementation details and hyper- parameerrss are provided in the Supplementary C

---

Table 1: Classification accuracy (%) CIFAR-I0 10 and CIFAR-100 datasets under PMD noises and hybrid noises, combining PMD noise with Uniform (U) and Asymmetric (A) noise.
CIFAR-10
Methods
35% PMD
70% PMD
35%: PMD 30% U
35% PMD 60%
35% PMD 30%
Standard
78.11 0.774
41.98
196
75.26 0.32
64.25 0.78
75.21 0.64
Co-tenching+
79.97 0.15
40.69 1.99
78.72 0.53
5.149
75.43 2.96
GCE 35
80.65 a: 0.39
36.52
78.08 0.66
67.43 a: 1.43
76.91 a: 0.56
SL 36
7.77
=
77.79 0.46
67.63 1.36
77.14 0.70
LRT [4]
80,98 + 0.80
41.52
75,97 0.27
59.22 0.74
76.96 = 0.45
OC 44
81.23 + 0.78
1,56 42.43
79.6 0.44
70.71 0.34
78.66 = 0.66
PLC [5
82,80 a: 0.27
42.74 2.14
79.04 a 0.50
72.21 + 2.92
78.31 + 0.41
SimCLR KNN
83.71
29.45
78.25
54.82
75.37
C2D + SimCLE R (36]
83.84 0.13
34,23
0.45
85.61 0.29
81,39 0.68
83.06 + 0.57
LRA-diffusion (SimCLR)
88, 0.24
42.63
1.97
88.41 0.37
84.43 0.82
85 64 + 0.23
CLIP KNN
91.80
30,66
84,67
57.03
81,76
LRA--difusion (CLIP)
96.54 a 0.13
44.62 a: 0.18
95.71 0.17
87.21 a 0.71
93.65 a 0.40
CIFAR-100
Methods
35% PMD
70% PMD
35% PMD 30% U
35% PMD 60%
35% PMD + 30%
Standard
57.68 # 0.29
3
48.86 = 0.56
35.97 1.12
45.85 = 0.93
telchinnng
56.70 + 0.71
39.53 0.28
52.33 0.64
27.17 = 1.66
$1.21 = 0.31
GCE
58.37 + 0.18
40.01 0.71
$2,90 0.53
38.62 # 1.65
$2.69 = 1.14
0.33
40.02 0.85
51.34 0.64
37.57 0.43
50.18 + 0.97
+.34
45.29
45,66 1.60
23.37=0.72
$2.04 0.15
59.44
0.45
43.64
60.01 0.43
45.92 =
0.15 51.87
$6.40 +
SimCLR KNN
39.25
41.73
C2D SimCLR
69.28 0.331
51.63
59.87 0.66
64.45 =
65 + 0.93
LRA-diflusion (SiMCLR)
0.81
0.28
55.79
0.11
CLIP KNN
79.58
52.55
69.66
50.91
61.19
L.RA-diflusion (CLIP)
0.10
0.15
81.10
81.78 0.15
5.1 Results on Synthetic Noisy Datasets
We conduct simulation experiments on the CIFAR-10 and CIFAR-100 datasets [5]] to evaluate our method's performance under various noise types. Specifically, following [15,, we test with polynomial margin diminishing (PMD) noise, novel instance-dependee noise, at two noise levels and three hybrid noise types by adding independent and identically distributed (i.i.)) noises on top of instance-dependent noise.
For instance-dependent noise, we adopt the recently proposed polynomial margin diminishing (PMD) noise [15]. Following the original paper, we train a classifier (1) using clean labels to approximate the probability mass function the posterior distribution p(y[x)) Images are initially labeled as their most likely class ux according to the predictions of (2)). Then, randomly alter the labels to the
second most likely class Sx for each image with probability: Puz.es = =- [ (x) - 7sx (x)] s. where is constant noise factor that controls the final percentage of noisy labels. Since corrupting labels to the second most likely class can confuse the "clean" classifier the most, is expected to have the most negative impact the performance of models learned with noisy labels. For PMD noise, we simulate two noise levels where 35% and 70% of the labels are corrupted.
For i.i.d noise, following (52 15), we use transition probability matrix T to generate noisy labels. Specifically, we corrupt the label of the i-th class to the j-th class with probability Tıj. We adopt two types of i.i.d noise in this study: (1) Uniform noise, where samples are incorrectly labeled as one of the other (n - 1) classes with uniform probability T. = 7/(n 1) and Tú 1 т. with the pre-defined noise level; (2) Asymmetric noise: carefully design the transition probability matrix such that for each class i, the label can only be mislabeled as one specific class j or remain unchanged with probability Ti and Tü hybrid noise by adding 30%, 60% uniform, and 30% asymmetric noise on top of 35% PMD noise.
In our experiment, generated three types of
We test our proposed lebel- augmentte diffusion model using two pre-trained encoders: (1) SimCLR (21]: We trained two encoders using the ResNet50 (53] architecture on the CIFAR-10 and 100 datasets through contrastive learning; (2) CLIP 24 the model is pre-trained
large dataset comprising 400 million image-text pairs. Specifically, we used the vision transformer
(4) encoder (ViT-L/14) with pre-trained weights, the best-perf architecture in CLIP. For
7

---

simplification, we refer to these configurations as LRA-diffusion (SimCLR) and LRA-diffusia (CLIP). We also investigated the performance the KNN algorithm within the feature space defined by the SimCLR and CLIP encoders, denoted as SimCLR KNN and CLIP KNN respectively.
Table T lists the performance of the Standard method (train classifier using noisy labels), method, and baseline methods for learning from noisy labels. The results in white rows are borrowed directly from [15]. We can see that using the SimCLR encoder in the LRA A-diffussion method results in superior test accuracy on both CIFAR-IO and CIFAR-I00 datasets compared to other baselines, without the need for additional training data. This is because the SimCLR encoder is trained an unsupervised manner, making it immune to label noise, and it can effectively extract categorical features for accurate image retrieval. Therefore, when the correct labels dominate the label distribution in the neighborhood, training with the labels of the retrieved neighbor images allows the model to learn with more correct labels. C2D [46) also utilizes a pre-trained SimCLR encoder for initialization, but label noise may still affect the feature space during training. In contrast, our method freezes the feature encoder, shielding the pre-trained features from noise. Results on CIFAR- 10 demonstrate that when the pre-trained feature has high KNN accuracy, our method performs better. On CIFAR-I00, where the SimCLR feature has lower KNN accuracy, C2D is more effective, as can refine the feature space through training. Moreover, freezing the feature encoder allows for efficient integration of large pre-trained encoders like CLIP, saving us from the prohibitive computational cost of fine-tuning, Notably, incorporating the CLIP encoder into method significantly improves test accuracy over our LRA-diffusion (SimCLR) due to its excellent representation capabilities. fact, by performing KNN in the CLIP feature space alone was able to achieve accuracy surpassing all competing methods in most experiments. This allows for the use of more clean labels during training, thus result even higher accuracy.
5.2 Ablation Studies
To evaluate the contribution of diffusion and the pre-trained features, we conducted ablation experi- ments using CARD (27), SS-DDPM [49], and linear probing to incorporate pre-trained models. is worth noting that the SimCLR model was trained on the same training set without access to external data. Results are given in Table 2
Table 2: Classification accuracy (%) CIFAR-I0 and CIFAR-100 datasets with PMD noises using different combinations of model, pre-trained feature, and label.
CIFAR-10
CIFAR-I00
Methods
Feature space
Label
35% PMD
70% PMD
35% PMD
70% PMD
Linear probing
SimCLR
noisy
38.93
56.18
51.87
Linear probing
SimCLR
sample
63.8
35.84
53.34
$2
Linear probing
SimCLR
mean
86.27
39.94
55.95
52.61
ResNet+Linear
SimCLR
sample
86.55
38.06
56.57
51.36
CARD
SimCLR
sample
75.08
34.35
52.03
32.67
SS-DDPM (Dirichlet)
SimCLR
sample
88.13
40.11
59.3
$1.67
I.R.-diffusion (ours)
SimCLR
sample
88.96
42,63
61.38
53.57
Linear probing
CLIP
noisy
85.35
37.4
65.02
53.21
Linear probing
CLIP
sample
95.61
40.17
63.98
58.53
Linear probing
CLIP
96.19
35.19
69.05
62.76
ResNet+LLinea
CLIP
sample
88.72
43.26
59.78
51.47
CARD
CLIP
sample
79.72
33.57
47.1
23.45
SS-DDPM (Dirichlet)
CLIP
sample 96.03
42.03
80.72
72.24
LRA-diffusion (ours)
CLIP
sample
96.55
44,51
81.92
74.58
Linear probing with sampled labels yielded lower accuracy than using noisy labels or the mean of neighboring labels. This difference may be due to the linear layer's inability to yield stochastic outputs from a multimodal distribution. During training. conflicting gradient directions may arise if the model tries to predict different labels across gradient steps, which can impede learning. However, due to the mode coverage ability of the diffusion model, our method can effectively learn from retrieval- augmented labels to generate different labels with different probabilities. We also test a baseline using an additional ResNet encoder along with the linear layer mimic our model architecture shown in Figure C.I The results comparable with linear probing with sampled labels. Our model

---

significantly outperforms CARD, mainly due to the more informative fp encoder. We use the same model architecture for SS-DDPM, which uses Dirichlet distributions for noisy states. SS-DDPM performed slightly worse than our method, indicating that constraining noisy states within probability simplex may not benefit our task.
Additional ablation studies are included in the Supplementary D The results demonstrate that in the absence of pre-trained encoder, our model can leverage the features of noisy classifier to enhance its accuracy. We also include results of ablation experiments using different approaches for incorporating the CLIP model. The results indicate that the superior performance of our method is not solely attributable to the strength of the CLIP features. In conclusion, our LRA-difiiie RRA-iiffusion model provides an efficient approach for incorporating pre-traine pre-trained encoders for learning from noisy labels.
5.3 Results on Real-world Noisy Datasets
We further evaluate the performance of our proposed method on real-world label noise. Following
previous work 42 13 44 43], we conducted experiments on four image datasets, i.e., WebVision
[55]. ImageNet ILSVRC12 [66]. Food-101N [57], and Clothing IM [58] For experiments on Webvision, ILSVRCI2, and Food-101N datasets, we use the CLIP image encoder as the fp encoder to train LRA-diffusion models. Comprehensive dataset description and implementation details can be found in the Supplementary C We evaluated the performance of our method against group state-of-the-art (SOTA) methods. The results are presented in Table 3 and Table Our approach significantly outperforms all the previous methods in terms of classification accuracy. It important to highlight that EPL. [59] incorporates the most powerful CLIP and Xex-NeX [60] encoders and cooperates with other SOTA methods such ELR [61], DivideMix [42). and UNICON [62]. However, our method outperforms EPL by achieving ~6% higher accuracy on WebVision and ILSVRCI2 datasets. This improvement over EPL demonstrates that developing better ways to incorporate pre-trained models to facilitate learning from noisy labels is a non-trivial task, highlighting the valuable contribution of our approach.
Table 3: Classification accuracies (%) on Wisvis,on, ILSVRC2012 datasets.
Dataset
DivideMix
ELR
UNICON
EPL
LongReMix
C2D
CC
NCR
LRA-diffusion
Web Vision
77.32
77.78
77.60
78.77
78.92
79.42
79.36
80.5
84.16
ILSVRC2012
75.20
70.29
75.29
76.51
78.57
76.08
82.56
Table 4: Classification accuracies (%) on the 1ood-101N dataset.
Standard
CleanNet [57]
BARE (63
DeepSelf [64]
PLC
LongReMix
LRA-diffusion
81.67
83.95
84.12
85.10
85.28
87.39
93.42
For experiments on the Clothing IM dataset, we found that -if-diffusion conditioned on the CLIP image encoder did achieve the SOTA accuracy. potential explanation is that the CLIP feature is too general for this domain specific task for categorizing fashion styles. However, our method orthogonal to most traditional learning with noisy label approaches. additional ablation study in Supplementary D.I] our method can collaborate with trained classifier by conditioning on its feature encoder to achieve improved performance. We first use the CC [44] method to select clean samples and train ResNet50 classifier, which achieved 75.32% accuracy (refer to as CC*). Then, we condition on its feature before the classification head to train our LRA- diffusion model on the selected samples, which achieved 75.70% accuracy. As Table shows, our method achieved a 0.38% improvement based on CC* and beat all SOTA methods.
shown in the
Table 5: Classification accuracies (%) on Clothing IM
Standard
BARE
PLC
LongReMix
DeepSelf
C2D
NCR
CleanNet
68.94
72.28
74.02
74.38
74.45
74.58
74.60
74.69
DivideMix
ELR
UNICON
EPL
CC'
CC
SANM [65]
LRA-diffusion
74.76
74.81
74.98
75.21
75.32
75.40
75.63
75.70

---

5.4 Inference Efficiency Analysis
In order test the efficiency of our model, we perform experiments assessing the runtime on CIFAR- 10 dataset and compare our method with standard classifier that uses ResNet50, It's worth noting that our SimCLR encoder is also built on the ResNer50. Thus, the standard method's runtime also reflects the linear probing runtime on SimCLR. Table shows the results.
Table 6: Inference time (s) of standard classifier and LRA-diffusinn models on CIFAR-10 images.
LRA-diffusion
Number of images
Standard (ResNet50)
SimCLR (ResNer50)
CLIP (VIT-B/32)
CLIP (ViT-B/16)
10000
3.96
9.52
9.13
17.12
50000
20.77
41.31
39.82
92.34
We see, the computation bottleneck lies the large pre-trained encoder but not the diffusion model itself. In general, our method takes twice as long as a standard classifier (ResNet50) when using SimCLR (ResNet50) and CLIP (ViT-B/32) pre-trained encoders. Larger CLIP encoders can increase the time further. However, it can be further accelerated if the features can be pre-computed in advance or be computed in parallel (as they are only required to be computed once and can be reused later).
6 Limitations
Our method, while being effective in many scenarios, does have certain limitations that we acknowl- edge. Its performance enhancement can compromised if pre-trained fp feature encoder isn' available or is inadequately trained. Additionally, the diffusion model introduces Gaussian noise in forward process, leading to latent label vectors not being confined within the probability simplex, which could increase training time. Lastly, our method's s performance becomes less effective when label noise levels surpass 50%. However, supervised learning can not be the optimal choice such situations.
Conclusion
In this paper, by viewing the noisy labeling process conditional generative process, we leverage diffusion models to denoise the labels and accurately capture label uncertainty. label-retrieval- augmented diffusion model was proposed to effectively learn from noisy label data by incorporating the principle of neighbor consistency. Additionally, by incorporating auxiliary conditional information from large pre-trained models such CLIP, we are able to significantly boost the model performance. The proposed model is tested on several benchmark datasets, including CIFAR-10, CIFAR-100, Food- 101N, and Clothing MM, achieving state-of- the-arr results in most experiments. Future work could extend our model to multi-label settings, as it does not require one-hot representation for labels. It also promising to use semantic segmentation guide the generation, potentially enhancing our model's 's interpretability and performance.
This work is partially supported by NSF AI institute-2229777 NSF RI- 2223292, Amazon research award, and an Adobe gift fund. Any opinions, findings and conclusions or recommendations expressed this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation, the Institute of Education Sciences, the U.S. Department of Education.
Acknowledgement:
References
[1] Yu Yao, Tongliang Liu, Mingming Gong, Bo Han, Gang Niu, and Kun Zhang. Instance- dependent label-noise learning under structural causal model. Advances in Neural Information Processing Systems, 34:4409-4420 2021.
[2] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3): 115, 2021.
10

---

[3] Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang. Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. Advances in Neural Information Processing Systems, 33:7597-7610, 2020.
[4] Jiacheng Cheng, Tongliang Liu, Kotagiri Ramamohanarao, and Dacheng Tao. Learning with bounded instance and label-dependent label noise. In International Conference on Machine Learning, pages 1789-1799, PMLR, 2020.
[5] Antonin Berthon, Bo Han, Gang Niu, Tongliang Liu, and Masashi Sugiyama. Confidence scores make instance-dependent label-noise learning possible. In International Conference Machine Learning, pages 825-836. PMLR, 2021.
[6] De Cheng, Tongliang Liu, Yixiong Ning, Nannan Wang, Bo Han, Gang Niu, Xinbo Gao, and Masashi Sugiyama. Instance-dependent label-noise learning with manifold-regularizzed transition matrix estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16630-16639, 2022.
[7] Songhua Wu, Mingming Gong, Bo Han, Yang Liu, and Tongliang Liu. Fair classification with instance-dependent label noise. In Conference Causal Learning and Reasoning, pages 927-943, PMLR, 2022.
[8] Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj. Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning.
pages 233-242. PMLR, 2017.
[9] Mingehen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks. In International conference on artificial intelligence and statistics, pages 1313-4324. PMLR, 2020.
[10] Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3552-55601 2018.
[11] Yisen Wang, Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, and Shu-Tao Xia. Iterative learning with open-set noisy labels. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8688-808 08888866,,110 2018.
[12] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International conference on machine learning, pages 3304-2313. PMLR, 2018.
[13] Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S Kankanhalli. Learning to learn from noisy labeled data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5051-5051 2019.
[14] Songzhu Zheng, Pengxiang Wu, Aman Goswami, Mayank Goswami, Dimitris Metaxas, and Chao Chen. Error-bounded correction of noisy labels. In International Conference on Machine Learning, pages 1447-11457 PMLR, 2020.
[15] Yikai Zhang, Songzhu Zheng, Pengxiang Wu, Mayank Goswami, and Chao Chen. Learning with feature-dependent label noise: A progressive approach. arXiv preprint arXiv:2103.07756 2021.
[16] Lutz Prechelt. Early stopping-but when? In Neural Networks: Tricks of the trade, pages 55-69. Springer, 2002.
[17] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv: 169vv01609.022077 2016.
[18] Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi-supervised learning. In Proceedings of the IEFE/CVE Conference on Computer Vision and Pattern Recognition, pages 5070-5079, 2019.

---

[19] Ahmet Iscen, Jack Valmadre, Anurag Amab, and Cordelia Schmid. Learning with neighbor consistency for noisy labels. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages
2022.
[20] De Cheng, Haichun Tai, Nannan Wang, Zhen Wang, and Xinbo Gao. Neighbour consistency guided pseudo-label refinement for unsupervised person re-identification, arXiv preprint arXiv:221 16847, 2022.
[21] Ting Chen, Simon Kornllith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 1597-1607. PMLR, 2020.
[22] Yuanyi Zhong, Haoran Tang, Junkun Chen, Jian Peng, and Yu-Xiong Wang. Is self-supervieed learning more robust than supervised learning? arXiv preprint 2tii::2206.0225555 2022.
[23] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
[24] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748 8763. PMLR, 2021.
[25] Dan Hendryeks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In International Conference on Machine Learning, pages 2712- 2721. PMLR, 2019.
[26] Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. Training convolutional networks with noisy labels. arXiv preprint
2014.
[27] Xizewen Han, Huangjie Zheng, and Mingyuan Zhou. Card: Classification and regression diffusion models. arXiv preprint 2ixx:2206007275 2022.
[28] William Feller. On the theory of stochastic processes, with particular reference to applications. In Selected Pupers I, pages 769-798. Springer, 2015.
[29] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[30] Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Björn Ommer. Semi- parametric neural image synthesis. In Advances in Neural Information Processing Systems, 2022.
[31] Alexander Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Nguyen, Pulak Purkait, Ravi Garg, Alan Blair, Chunhua Shen, and Anton van den Hengel. Retrieval augmented classification for long-tail visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision 2022.
and Pattern Recognition, pages
[32] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[33] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems, 33:22243-22255, 2020.
[34] Aritra Ghosh, Himanshu Kumar, and P Shanti Sastry. Robust loss functions under label noise for deep neural networks. Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
[35] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
12

---

[36] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 322-330, 2019.
[37] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-28222 2826, 2016.
[38] Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy. Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks noisy labels with bootstrapping. arXiv preprint arXiv:1412.6596, 2014.
[39] Eric Arazo, Diego Ortego, Paul Albert, Noel O' D'Cnnor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In International conference machine learning, pages 312-321. PMLR, 2019.
[40] Jingkang Yang, Litong Feng, Weirong Chen, Xiaopeng Yan, Huabin Zheng, Ping Luo, and Wayne Zhang. Webly supervised image classification with self-contained confidence. In European Conference on Computer Vision, pages 779-795. Springer, 2020.
[41] Pengxiang Wu, Songzhu Zheng, Mayank Goswami, Dimitris Metaxas, and Chao Chen. A topological filter for learning with label noise. Advances in neural information processing systems, 33:21382-21333 2020.
[42] Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. arXiv preprint arXiv:2002.07394 2020.
[43] Filipe R Cordeiro, Ragav Sachdeva, Vasileios Belagiannis, Ian Reid, and Gustavo Carneiro. Longremix: Robust learning with high confidence samples in noisy label environment. Pattern Recognition, 133: 109013, 2023.
[44] Ganlong Zhao, Guanbin Li, Yipeng Qin, Feng Liu, and Yizhou Yu. Centrality and consistency: two-stage clean samples identification for learning with instance-dependen noisy labels. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXV, pages 21-37. Springer, 2022.
[45] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach semi-supervised learning. Advances neural information processing systems, 32, 2019.
[46] Evgenii Zheltonozhskii, Chaim Baskin, Avi Mendelson, Alex M Bronstein, and Or Litany. Contrast divide: Self-supervised pre-training for learning with noisy labels. In Proceedings of the IEEE/VVF Winter Conference on Applications of Computer Vision, pages 1657-1667,
2022.
[47] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. 2021.
Advances in Neural Information Processing Systems,
[48] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint vrXiv:2007.12598 2022.
[49] Andrey Okhotin, Dmitry Molchanov, Vladimir Arkhipkin. Grigory Bartosh, Aibek Alanov, and Dmitry Vetrov. Star-shaped denoising diffusion probabilistic models. arXiv preprint arXiv:2302.05259, 2023.
[50] Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In International Conference on Machine Learning, pages 7164-7173. PMLR, 2019.
[51] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[52] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: loss correction approach. Proceedings of the IEEE conference computer vision and pattern recognition, pages 1944-1952, 2017.
13

---

[53] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[54] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1192 2020.
[55] Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint arXiv: 17Xiv:1708.02862, 2017.
[56] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-252, 2015.
[57] Kuang-Huels Lee, Xiaodong He, Lei Zhang, and Linjun Yang. Cleannet: Transfer learning for scalable image classifier training with label noise. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5447-5456, 2018.
[58] Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2691-2699, 2015.
(59] Jongwoo Ko, Sumyeong Ahn, and Se-Young Young Yun. Efficient utilization of pre-trained model for learning with noisy labels. In ICLR 2023 Workshop on Pitfalls of limited data and computation for Trustworthy ML, 2023.
[60] Zhuang Liu, Hanzi Mao, Chao- Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 11976-1 1777--19966, 1986. 2022.
[61] Sheng Liu, Jonathan Niles Weed, Narges Razavian, and Carlos Fernandez-Grante Early- learning regularization prevents memorization of noisy labels. Advances in neural information processing systems, 33:20331 1-23322,- 2020.
[62] Nazmul Karim, Mamshad Nayeem Rizve, Nazanin Rahnavard, Ajmal Mian, and Mubarak Shah. Unicon: Combating label noise through uniform selection and contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2022.
[63] Deep Patel and PS Sastry. Adaptive sample selection for robust learning under label noise. In Proceedings of the IEEÉ/CVF Winter Conference on Applications of Computer Vision, pages 3932-3942, 2023.
[64] Jiangfan Han, Ping Luo, and Xiaogang Wang. Deep self-learning from noisy labels. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5138-5147, 2019.
[65] Yuanpeng Tu, Boshen Zhang, Yuxi Li, Liang Liu, Jian Li, Jiangning Zhang, Yabiao Wang, Chengjie Wang, and Cai Rong Zhao. Learning with noisy labels via self-supervised adversarial noisy masking, arXiv preprint arXiv:2302. 06805, 2023.
[66]
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning. volume 4. Springer, 2006.
[67] Shakir Mohamed and Balaji Lakshminarayanan. Learning in implicit generative models. arXiv preprint arXiv: 161.003480 2016.
[68] Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food- 11110011miii 101-mining discriminative components with random forests. In Computer Vision-ECCY 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446-661. Springer, 2014.
14

---

A Denoising Diffusion Implicit Model with non-zero mean latent space
The forward process of diffusion process with non-zero latent distribution yr [N) I)) has closed form representation:
yyy: f) Wyys a.yo (1 (1-
(A.1)
which could reprameterized as:
Yt Vāey + (x) vI aiE
(A.2)
Then, similar to the DDIM, we define Non-Markovian forward process with or 2 00tt=11
go a= f) II qu (v.-1lye.00.f)
(A.3)
+ Where 4-(yrlyo, = -) + - TT and for all
(v f) =N yt-18 1 ff v āe-1 of (,ootf (A.4)
(y: Varyo - (1 (1-vaa)ffl vāx)
We can prove that the sampling process defined Eq. and Eq. A.4 has the same marginal distribution as the closed-form sampling process in Eq. (A.I by the following Lemma:
Lemma A.1. For qo (yr.rlyo: f) defined in Eq.
A.3 and qo (yy--1ly f) defined Eq. A.4 we have:
To (y-loo,, f) Nye Va.yo + (x)), (1 a,)
(A.5)
Proof. Assume for any 8 < 7, Eq. A..5 is truc, following is also true:
go (y f) N (y: vy
+ (1 Va
(1 ā.-1)I),
(A.6)
then prove the statement with induction argument for from T since the base (L 7') already holds.
First, we have that:
go (o-y-ilyo) = f go
(A.7)
To (yelyo, N(y Vavii +1-
(A.8)
go (V = N Y1- Vi- vāx-1)((x) + 1 of z,oi).
(A.9)
VI- (y, Varyo - (1 vx))
According
66 Eq. (2.115), have that go (yr-1 Vt, Vo. f) is Gaussian, with mean M:-1 and co-variance
Σ,-1
P.11 = lyo - (o(x))
(A.10)
+ V1 ōx-1 of
Vauyo + vat) Vaiyo (1 v Vas [x))((x)
V1 az
=Vor-180 vor-1)
(A.11)
Σ = of - a. of (1 a. I (1-
(A.12)
oe Therefore, Eq. A.6 holds. Following the induction, the lemma is proved.
In our implementation, we follow DDIM 32] setting 0. The resulting model becomes an implicit probabilistic model 67 where generation process become deterministic given yr-
15

---

B
1-SNE visualization of the DDIM generation process
t=7
t=
t=5
t=3 t=2 t=1 t=0
Figure B.I: The t-SNE visualization of the CLIP feature space during the reverse generation process of conditional diffuusion model using an 7-step DDIM on the CIFAR-I0 dataset. The process begins at time / = 7. with the sampling the latent representation the label from the latent distribution N (x). I). Through series of multi-step reverse operations, latent distribution transformed into conditional distribution of labels. The data points are color-coded according the entry with the highest value intermediatc/fial label vectors, and the ground truth class labels are represented by distinct markers.
C Experimental setup and details
C.1 Real-world dataset details
Web Vision comprising 2.4 million images that erawled using Google and Flickr search engines, with the ILSVRCI2 taxonomy. Following prior studies, we trained our model on the initial 50 classes from the Google image subset of Webvision and tested the validation
both Webvision and ILSVRC12.
Food-101N consists 310k food images collected from the internet with the Food-10 101
68
taxonomy, and has
estimated label noise level of 20%, making it an ideal dataset to evaluate the robustness of our method under
real-world noisy labels. We assessed the classification accuracy on the curated label set of Food-101, which contains around 25k images.
ClothingIM contains million images of clothes obtained from shopping websites. Based on the keywords the surrounding text, the images automatically classified into 14 classes with ~40% estimated noise level. The dataset includes clean training set, validation set, and test set with manually refined labels, consisting of approximately 47.6k, 14.3k, and 10k pictures, respectively. We discarded the clean training set and only used the noisy label data for training.
C.2 Implementation details
To present the hyperparameter settings of our neural network, we first give a description our neural network design. As shown in Figure C.I the network consists of frozen encoder, ResNet encoder, and series of feed forward layers. Features encoded by the two encoders are combined with time embedding via hadamard product and passed through series of feed-forward networks, batch normalization, and softplus activation predict the noise term es.
In experiments,
ResNet34 for CIFAR 10 and CIFAR100, and use ResNet50 for real-world datascts
the trainable encoder (blue ResNet block Figure C. The dimensions feed-forward layers are set to 512 for CIFAR datasets and 1024 for real-world datasets, respectively. We train LRA difffsussion models for 200
16

---

Yx
Embedding fx(x)
Image
Feed Forward
ResNet Encoder
BatcliNorm Sottphus
BatchNorm
Feed Forward
Tiaee Einbedding
BatchDorm Sottphus
Forward
BatchhNomm Softplus
Concatenation
Feed Forward
Hadamard product
ealyi, x, fp(x).
Figure C.I: The network architecture for conditional diffusion models. The input to the network consists of four elements: ys, fo(x), x, and the time embedding for t, represented by pink blocks. The blue blocks in the figure represent the trainable network components.
epachs with Adam optimizer. The batch size is 256. We used learning rate schedule that included warmup phase followed by half-cyele cosine decay. The initial learning rate is set 0.001. Following [15. applied data augmentation the training, including resizing, random horizontal flip. and random cropping. To retrieve k=10 based on tests using range of k values from to 100 the validation sets. The KNN accuracy remained relatively stable for k between 10 and 50, and then starts to decline due to reduced label consistency among neighbors. Based these results, infer that LRA diffusion model is less sensitive to variations in l within this range. All experiments are conducted using four NVIDIA Titan GPUs.
the nearest neighbors,
D
Additional ablation study
D.1 Classifier feature conditioning for accuracy enhancement
To demonstrate how method can enhance trained classifier's performance by using its features conditional information, we conduct an ablation study examining the impact of conditional diffusion and KNN on the trained classifier. Specifically, train classifiers, denoted as v7()), using the standard method various noise levels. We then remove the classification head and utilize the remaining model fa, as the fp encoders in our conditional diffusion models.
The experimental results shown in Figure D.I indicate that these techniques can improve test accuracy. observe that when the noise level is below 55%. the conditional diffusion model (green) achieves ~ 1% improvement over the standard method. Moreover, when the LRA method applicd concurrently (purple), test accuracy can be further enhanced. This improvement occurs because learning from neighbors' labels reduces the noise level during training, as evidenced by the comparison between KNN results (blue) and clean label percentage (gray).
However, when noise level exceeds 55%, the use diffusion and LRA LRA-diffusion methods does not seem advantageous. This limitation arises because the distribution of labels in the neighborhood becomes too corrupted for KNN to effectively improve the proportion of clean labels during training, as illustrated by the interscction of the blue and gray curves in the figure. We argue this does not diminish the practical value of our method because a dataset with more than 50% label noise is not meaningful in practice.
D.2 Effects of pseudo-label construction strategies
We also conduct comparative experiments using another method that utilizes neighbor labels: replacing one-hot label vector with the mean vector of the neighbor's labels as the prediction target, which call Mean-diffusion. found that can achieve higher accuracy when the noise level is higher than 55%. This may be due to the increase in the diversity of neighbor labels. The sampling-basee RRA-diffusion will need
17

---

TRist acturry %6 on CIFAR10
70
Slandard
Diffusion
Mear-diffusion
40
LRA diffusion
SimCLR
SimCLR LRA-diffusion
30
Clean percentage
35%
40%
45%
50%
55%
60%
65%
70%
Noise levels
Figure D.I: Test accuracy of seven methods on the CIFAR-I0 dataset with different levels of PMD label noise. Other than the already introduced method names, here Diffusion a conditional diffusion model using the feature fn. The clean label percentage represented by the gray line.
to learn a more complex multi-modal distribution, but Mean-diffusion only needs to learn a point estimate. However, when the noise level lower than 55%, we found that IRA--iffusiinn is slightly more accurate than Mcan-diffusion. A possible explamation is that the distribution of yo in LRA-diffusion contains only n one-hot labels. contrast, Yo in Mcan-diffusion is more diverse (n possible mean neighbors). In conclusion, LRA ARA-diffusion has higher performance with less noisy labels. On the other hand, deaandiifusioon has faster and more stable convergence tend to perform similarly when the noise level is too high too low since neighbors' labels will become the same or too corrupted.
for classes and
more robust high noise level. However. they
D.3 Robustness of Sim CLR feature conditioning
Finally, we use the SimCLR model encoder fp in our conditional diffusion model (listed as SimCLR LRA-diffusion in Figure D.I. to showease the effectiveness of our proposed LRA-diffusion method in utilizing prior knowledge from pre-trained image representations to enhance test accuracy and robustness. The experimental results (red) show that its test accuracy significantly surpasses other settings until the noise level reaches 65%. Beyond this point, the labels in the neighborhood become too corrupted to provide additional supervision information.
D.4 Effects of CLIP feature conditioning strategies
Table D.1: Classification accuracy (%) of linear probing, KNN, and diffusion model using pre-trained CLIP feature real-world label noise datasets.
Method
Webvision
ILSVRC2012
Food-101N
Clothing!M
KNN
81.88%
82%
91.73%
65.70%
linear prob Mean label
68.56%
68.76%
90.68%
65.75%
lincar prob + sample label
54.84%
56.80%
89.56%
55.52%
diffusion (Mean label) 83.96%
82.24%
93.13%
71.79%
diffusion (sample label)
84.16
82.56 %
93.42 %
71.65%
carried out ablation study employing various strategies integrate the CLIP model classification on real-world datascts. The results demonstrate that the superior performance of our diffusion-based method does not simply reflect the strength CL.PP features. We show results linear probing and KNN using CLIP features in Table D. to further demonstrate the effectiveness of algorithm design. The results suggested that
18

---

utilizing linear probing with LRA and mean labels degrades the pre-trained feature space, leading to diminished performance in comparison to the unsupervised KNN approach. On the other hand, the diffusion process can effectively incorporate the CLIP feature space to achieve higher performance.
19


Reviews:
[{"review_id": "TSSjz7iHpm", "reviewer": "Reviewer_KNdw", "summary": "This paper considers image classification with noisy labels. Rather than consider a single label for each image, it considers the label distribution within the set of neighboring examples (determined using a pre-trained feature extractor), modelling this distribution by a learned diffusion process that is conditioned on image features, similar to CARD. The diffusion process maps a random label in the neighboring set to a normal distribution whose mean is either zero or the prediction of a baseline classifier, such that the reverse process should sample labels from the neighboring set. Inference is performed using DDIM to find the maximum likelihood label. The method assumes a pre-trained feature extractor (SimCLR trained on training data or CLIP trained on external data) is available for the purpose of finding neighbors and conditioning the diffusion process. The method is compared to strong baselines on several synthetic and real-world datasets and often provides a large improvement. Ablative studies demonstrate the importance of each component.", "questions": "Please address weaknesses.", "limitations": "The authors have identified limitations including:\n1. dependence on pre-trained feature extractors\n1. method is less effective beyond ~50% corrupted labels\n1. normal distribution not ideal for working on probability simplex\n\nI do not see any ethical issues.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The ablative study (Table 2; CIFAR10/100 with synthetic noise) demonstrates that the combination of CLIP/SimCLR features and LRA-diffusion is key to the effectiveness of the method: linear probing, LRA (label retrieval) and CARD (diffusion-based classification) alone do not achieve such an improvement.\n1. Idea is well motivated.\n1. Good coverage of related work.\n1. Well written.\n1. Synthetic experiments use strong type of noise based on second-highest confidence.\n1. Inference time included in empirical results.", "weaknesses": "1. Real-world evaluation (WebVision, ILSVRC, Food-101N) does not include simple baselines using CLIP features (KNN classifier and linear probe). This is important because it lets us verify that the results do not simply reflect the strength of the CLIP features.\n1. Real-world evaluation does not include SimCLR features. This is important because it shows the performance of the method without external data. In general, it would be good to highlight which methods use (which) external datasets and/or pre-trained weights in the SOTA comparison.\n1. (Sections 3.3 and 3.4) The modification of the DDIM procedure to allow non-zero mean seems to be redundant, since all experiments then used a zero-mean distribution by setting $f_q = 0$ (line 179)?\n1. Unclear what \"Linear probing + LRA\" is in Table 2. Does this entail training a linear model using the average of the neighboring labels as the softmax-cross-entropy target?\n1. It does seem inelegant to use a normal distribution to represent probability vectors (as acknowledged in text). Did the authors try working in real-valued logits instead?\n\nSuggestions:\n1. I'm not a big fan of naming the method \"retrieval\", since the term suggests to me that training data is retrieved from a larger, external source of training data (not the case).\n1. Discussion of impact of $k$ (number of neighbors) is deferred to the supp mat and I didn't see any references to this in the main text. It would be good to at least reference it and ideally add a plot.\n1. Could be good to include some discussion of singly-labelled vs multiply-labelled noisy labels, since the proposed method effectively constructs a multiply-labelled dataset from a singly-labelled dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: Real-world evaluation (WebVision, ILSVRC, Food-101N) does not include simple baselines using CLIP features (KNN classifier and linear probe). This is important because it lets us verify that the results do not simply reflect the strength of the CLIP features.\\\nA: We add the KNN and Linear Probing results to Table 3 in the attached PDF. Similar to our ablation study, linear probing results in decreased performance compared to KNN. On the other hand, diffusion methods are effective and robust in incorporating CLIP features. The test accuracy on ILSVRC2012 shows a performance drop due to supervised training bias on Webvision. This limitation does not apply to the unsupervised KNN.\n\nQ2: Real-world evaluation does not include SimCLR features. This is important because it shows the performance of the method without external data. In general, it would be good to highlight which methods use (which) external datasets and/or pre-trained weights in the SOTA comparison.\\\nA: We found that using SimCLR features is less effective compared to SOTA methods, partly due to the sub-optimality of the trained features. However, SimCLR+Diffusion (our method) still outperforms the simple SimCLR method. We also have shown improved results with CC, which isn't trained with external data, on the Clothing1M dataset, as presented in Table 5. This demonstrates that our method can achieve improved results when coupled with other feature-learning methods.\n\nQ3: (Sections 3.3 and 3.4) The modification of the DDIM procedure to allow non-zero mean seems to be redundant, since all experiments then used a zero-mean distribution by setting  (line 179)?\\\nA: We would like to argue that we show that allowing DDIM with non-zero mean is a more theoretically general framework of our method, but do not advocate the exact adoption of the method. In practice, we do need to adopt the most practically effective setting. We empirically found that utilizing a noisy classifier as the $f_q$ encoder does not consistently lead to improved results. For simplicity, we thus only use a zero mean to reduce the model complexity. \n\nQ4: Unclear what \"Linear probing + LRA\" is in Table 2. Does this entail training a linear model using the average of the neighboring labels as the softmax-cross-entropy target?\\\nA: An sample label is a one-hot label vector, sampled from neighboring labels as outlined in Algorithm 1, step 4, on page 4. We've included the result of the linear probing with mean label setting that you requested in Table 2 of the attached file. It is more effective than using sample labels for linear probing. Additionally, we have discussed the utilization of mean labels for diffusion in Supplementary D2, where we find that the performance is comparable to that of sample labels.\n\nQ5: It does seem inelegant to use a normal distribution to represent probability vectors (as acknowledged in text). Did the authors try working in real-valued logits instead?\\\nA: We agree that the generated label vector is not confined to the probability simplex, but it can be interpreted as the logit of the prediction head of a classifier. To further address your concerns, we additionally performed experiments using diffusion on simplex following the current Star-shaped DDPM (Okhotin 2023). As stated in their paper, the utilization of Dirichlet distribution in a diffusion model requires a different mechanism design than DDPM. We have added the results as Dirichlet diffusion in Table 2. Although it has a one-hot label interpretation, the performance is indeed slightly worse than using standard DDPM. We hope this response adequately addresses your concern.\n\n* Okhotin, Andrey, et al. \"Star-Shaped Denoising Diffusion Probabilistic Models.\" arXiv preprint arXiv:2302.05259 (2023).\n\nSuggestions:\n\nS1: I'm not a big fan of naming the method \"retrieval\", since the term suggests to me that training data is retrieved from a larger, external source of training data (not the case).\\\nA: We use the term 'retrieval' in the name because the label is augmented by the neighbor's label, and the neighbors are retrieved from an external feature space, which can be pre-trained by external data. However, we understand how the term 'retrieval' might cause misunderstandings. Accordingly, we will consider changing the name to “label-corrected diffusion” to better reflect the actual function.\n\nS2: Discussion of impact of  (number of neighbors) is deferred to the supp mat and I didn't see any references to this in the main text. It would be good to at least reference it and ideally add a plot.\\\nA: Thank you for your suggestion. We will add a reference to the impact of the number of neighbors in the main text and include a small plot in the supplementary material to address this concern.\n\nS3: Could be good to include some discussion of singly-labeled vs multiply-labeled noisy labels, since the proposed method effectively constructs a multiply-labeled dataset from a singly-labeled dataset.\\\nA: Thank you for your suggestion. We plan to include a discussion of applying our method to multi-label datasets in the conclusion section, which we hope to explore as interesting future work.\n\n"}, {"review_id": "9ow2I5i5KG", "reviewer": "Reviewer_21mt", "summary": "The paper proposes using a diffusion model to perform classification on a noisy-label dataset. The authors utilize diffusion to learn how to conditionally generate labels given an image. By combining recent work on classification diffusion models with label retrieval augmentations they demonstrate state-of-the-art results on real-world noisy datasets.", "questions": "- For a single sample, is the frequency of the neighbors seen in training the same regardless of their distances in the latent space? If so, how is a neighborhood defined?", "limitations": "The limitations are addressed. Broader societal impacts are not applicable.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The method provides a simple, yet seemingly effective way of performing classification on noisy data using generative models. The idea of using diffusion models for purpose this is novel and has the potential to significantly contribute to the \"Learning With Label Noise\" literature.\n- Some of the real-world dataset results showcased outperform the previous state-of-the-art by a significant margin.", "weaknesses": "- The novelty of the paper is limited. The authors combine the existing classification diffusion model with the retrieval augmentation technique to capture the noisy label distribution. The methodologies for both are inherited almost unchanged from their respective papers and there is no significant extension to either. Extending DDIM to a non-zero mean latent distribution is a minor contribution.\n\n- In lines 127 and 166, the authors claim that initializing with the predicted label mean and running the DDIM sampler approximates the maximum likelihood estimation of the labels. This is not justified anywhere, and there is no evidence to support the claim.\n\n- The experiments on the WebVision and ILSVRC2012 datasets should be examined further. The method presented in this paper relies heavily on having a high-quality pre-trained encoder network that can provide meaningful representations during training. For these two experiments, the authors utilized CLIP, whereas the next best-performing approaches, C2D and NCR, were only trained using samples from the smaller datasets. It is possible that the majority of the performance increases shown in this paper can be attributed to the discriminative ability of the CLIP encoder, since additionally on the Clothing1M dataset, where CLIP is not expected to perform well, the authors did not manage to greatly outperform the previous state-of-the-art results. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1: The novelty of the paper is limited. The authors combine the existing classification diffusion model with the retrieval augmentation technique to capture the noisy label distribution. The methodologies for both are inherited almost unchanged from their respective papers and there is no significant extension to either. Extending DDIM to a non-zero mean latent distribution is a minor contribution.\\\nA: Thanks for the comment. We wish to claim that there are at least two types of research papers: one focuses on theoretical innovation; the other focuses on improving practical performance of some particular problems. Our paper largely belongs to the second type. Our contribution lies in integrating and adapting diffusion models and the SOTA image encoder to a specific problem domain rather than proposing a new theoretical analysis. The proposed design allows us to achieve significant performance improvements, as demonstrated in our experimental results. On the other hand, extending DDPM to a non-zero mean latent distribution is essential to the previous CARD model. In our paper, we extended it to DDIM to provide a more general framework that includes CARD as a special case and potentially facilitates future work that uses a more refined $f_q$ encoder.\n\nW2: In lines 127 and 166, the authors claim that initializing with the predicted label mean and running the DDIM sampler approximates the maximum likelihood estimation of the labels. This is not justified anywhere, and there is no evidence to support the claim.\\\nA: We empirically find that initializing DDIM with the predicted label mean results in slightly improved testing accuracy compared to the stochastic version. Since the initial Gaussian distribution is unimodal, and it serves as the sole source of randomness in the generation process, using the mode to obtain a deterministic and most probable prediction emerges as an intuitive choice.\n\nW3: The experiments on the WebVision and ILSVRC2012 datasets should be examined further. The method presented in this paper relies heavily on having a high-quality pre-trained encoder network that can provide meaningful representations during training. For these two experiments, the authors utilized CLIP, whereas the next best-performing approaches, C2D and NCR, were only trained using samples from the smaller datasets. It is possible that the majority of the performance increases shown in this paper can be attributed to the discriminative ability of the CLIP encoder, since additionally on the Clothing1M dataset, where CLIP is not expected to perform well, the authors did not manage to greatly outperform the previous state-of-the-art results.\\\nA: As discussed in our global response, the KNN and linear probing results added to Table 3 illustrate that our method effectively incorporates CLIP features in supervised learning from noisy labels. Utilizing naive linear probing with neighboring labels degrades the pre-trained feature space, leading to diminished performance in comparison to the unsupervised KNN approach. Moreover, our method has the flexibility to incorporate other techniques to further enhance accuracy.\n\nQ1: For a single sample, is the frequency of the neighbors seen in training the same regardless of their distances in the latent space? If so, how is a neighborhood defined?\\\nA: Neighbor is defined the same as in the K-nearest neighbor algorithm, which is by the rank of distances. We choose K based on KNN performance on the validation set as discussed in supplementary C2.\n"}, {"review_id": "VJFNZkNtbb", "reviewer": "Reviewer_3Gk3", "summary": "The paper proposes the application of denoising diffusion probabilistic models for modeling the true class probability distribution when training with noisy labels. In particular, the paper extends Classification and Regression Diffusion Models for this problem and uses pretrained self-supervised representation models for Label Retrieval Augmentation. At test time, they approximate the true label by starting with the reverse diffusion at the mean of the final gaussian distribution. The empirical results on CIFAR10, CIFAR100 and other real-world noisy datasets show improvements over considered baselines. ", "questions": "1. In line 179, you write that you simply set $f_q({\\bf x}) = {\\bf 0}$ whereas you write in line 151 that the DDIM generation begins with a non-zero centered gaussian. So, the generalized DDIM is essentially used for the CARD+LRA Diffusion experiments in Table 2? And, you have a zero-centered gaussian distribution for the LRA-diffusion that you actually implement? \n2. Do you have any intuition about the performance of the baseline proposed in Weakness (3)?\n3. Do the baselines considered in Tables (3), (4) and (5) have access to extra training data? \n4. Will you release all the trained model checkpoints used for reporting the results in tables?\n", "limitations": "Yes, the limitations are adequately addressed. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The main strength of this work is that it demonstrates successful application of diffusion models for learning the probability distribution of labels in noisy label settings. The paper considers a variety of datasets and includes the relevant ablation studies and inference times. ", "weaknesses": "I believe the key weaknesses are:\n1) The requirement of a pre-trained self-supervised image embedding network. \n2) The diffusion model does not sample discrete labels and does not even model the categorical distribution. \n3) I think the evaluation should also consider the following baseline: train a classifier network exactly as in Algorithm 1 except in step 5 where the network is trained with an MSE loss to predict one-hot vector obtained in step 4. More specifically, it would be good to check the performance when the classifier network can take in the image and the simclr/clip features and have a linear prediction head. This baseline would be more closer to the LRA-diffusion model than the other baselines considered in Table 2 in terms of the model input and output. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1: The requirement of a pre-trained self-supervised image embedding network.\\\nA: We believe the requirement of a self-supervised embedding model like SimCLR is not necessarily a limitation, as it can be obtained for free by training on the same training dataset. Furthermore, the utilization of SimCLR is a common practice in the field. It has been leveraged in previous state-of-the-art methods such as C2D and the methods proposed by (Smart\n2023 and Cordeiro 2021). \n* Smart, Brandon, and Gustavo Carneiro. \"Bootstrapping the Relationship Between Images and Their Clean and Noisy Labels.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2023.\n* Cordeiro, Filipe R., et al. \"Propmix: Hard sample filtering and proportional mixup for learning with noisy labels.\" arXiv preprint arXiv:2110.11809 (2021).\n\nW2: The diffusion model does not sample discrete labels and does not even model the categorical distribution. \\\nA: We would like to argue that this is not a fundamental issue, as the generated label vector can be interpreted as the logit of a prediction head of a traditional classifier. To further address your issue, we also performed experiments using diffusion on the probability simplex following the Star-shaped DDPM (Okhotin 2023). As stated in their paper, the utilization of Dirichlet distribution (the conjugate prior of the categorical distribution) in a diffusion mode requires a different mechanism design than DDPM. We have added the results as Dirichlet diffusion in Table 2. Although it has a one-hot label interpretation, the performance is indeed slightly worse than using standard DDPM. We hope this response adequately addresses your concern.\n\n* Okhotin, Andrey, et al. \"Star-Shaped Denoising Diffusion Probabilistic Models.\" arXiv preprint arXiv:2302.05259 (2023).\n\nW3 and Q2: I think the evaluation should also consider the following baseline: train a classifier network exactly as in Algorithm 1 except in step 5 where the network is trained with an MSE loss to predict one-hot vector obtained in step 4. More specifically, it would be good to check the performance when the classifier network can take in the image and the simclr/clip features and have a linear prediction head. This baseline would be more closer to the LRA-diffusion model than the other baselines considered in Table 2 in terms of the model input and output.\\\nA: We have included the results of this baseline, denoted as \"ResNet + linear,\" in Table 2 of the attached PDF file, and plan to incorporate it into the final version of the manuscript. We used a similar network architecture and the same pre-trained features and training labels as those used for our diffusion model.  In certain settings, its accuracy is found to be lower than that of linear probing without using images as input. This difference may be attributed to the additional learning capacity introduced by the ResNet component being used to overfit the label noise. Moreover, the performance gap between our method and this baseline demonstrates the significance of the diffusion process.\n\nQ1: In line 179, you write that you simply set whereas you write in line 151 that the DDIM generation begins with a non-zero centered gaussian. So, the generalized DDIM is essentially used for the CARD+LRA Diffusion experiments in Table 2? And, you have a zero-centered gaussian distribution for the LRA-diffusion that you actually implement? \\\nA: You are correct. The non-zero centered Gaussian for the start of the generation process is an essential component introduced in CARD. We also include this design in our model to present a more theoretically general framework of our method. However, in practice, we must adopt the setting that is most effective. Through our empirical analysis, we found that utilizing a noisy classifier as the $f_q$ encoder does not consistently lead to improved results. For the sake of simplicity and to reduce model complexity, we thus chose to use only a zero mean.\n\nQ3: Do the baselines considered in Tables (3), (4) and (5) have access to extra training data? \\\nA: The baseline methods do not have access to extra training data. In the global response, we discussed that the performance of our method does not simply reflect the strength of the CLIP features by comparing our method to the KNN and linear probing results using CLIP features on the real-world datasets presented in Tables 3, 4, and 5.\n\nQ4: Will you release all the trained model checkpoints used for reporting the results in tables?\\\nA: Yes, we will release all checkpoints and the code to recreate the results of our experiments.\n"}, {"review_id": "iTTDY5MP0Q", "reviewer": "Reviewer_ZUT9", "summary": "The paper focuses on learning from noisy labels using diffusion models. They reformulate the noisy label problem from a generative perspective. Specifically, the clean label is our target variable ($y_0$) and the noisy label is the diffused variable ($y_T$), which they call Label-Retrieval-Augmented (LRA) diffusion model. Actually, we don't know the clean label, they extract the pseudo clean labels using the neighborhood consistency. ", "questions": "Please see Weaknesses part.", "limitations": "They discussed in Limitations section.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper is well-written.\n* The proposed method is a well-extended approach that builds upon the utilization of diffusion models in classification tasks and applies it effectively to tackle the noisy label problem.", "weaknesses": "* Motivation of LRA diffusion model\n  * It is important to explain why the noisy label process should be formulated as a diffusion process and to discuss in detail the advantages it offers.\n  * Since I think it is a novel approach, it is crucial to provide detailed explanations of the motivation behind it.\n  * When using the neighbor consistency principle, it is important to address whether the same problems mentioned by the authors in lines 44-51 would arise. If not, it is necessary to explain how the proposed model overcomes these challenges.\n\n* Comparison with CARD\n  * It appears that the original CARD paper introduced two pre-trained encoders, while the current paper seems to have used only one. This distinction needs to be clarified, and the roles of the two encoders should be explained separately in the description.\n  * Equation 4 seems to be adapted from the formulation presented in CARD, and proper citation should be provided.\n\n* Experiments\n  * The utilization of CLIP in the performance evaluation raises concerns because it involves the incorporation of additional data. It is important to conduct a fair comparison by addressing this aspect.\n  * An explanation is needed to clarify why the diffusion model is not a computational bottleneck. It is hypothesized that this may be due to the smaller dimensions, but this should be verified.\n\n* Minor comments\n  * The appearance of $\\tilde{y}$ in line 136 without prior definition should be addressed.\n  * \"Require: Input:\" in Algorithm 1 is typo.\n  * The inconsistent use of $\\mu_{\\theta}$ and $\\epsilon_{\\theta}$ should be corrected.\n  * The subscript in line 153 needs to be written correctly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1.1: It is important to explain why the noisy label process should be formulated as a diffusion process and to discuss in detail the advantages it offers. Since I think it is a novel approach, it is crucial to provide detailed explanations of the motivation behind it.\\\nA: Thank you for highlighting the need to explain our approach. The intrinsic ambiguity of data introduces uncertainty to the labeling process, resulting in controversial labels. Modeling this process with a stochastic conditional generative model is intuitive, and using the most probable generation is expected to improve the annotation (classification) accuracy. We chose the diffusion model because it has demonstrated the ability to model prediction uncertainty in the CARD model for classification. In addition, the mode coverage ability of the diffusion model (generating diverse samples given the same conditional information) is intuitively suitable for training with neighboring labels. We'll provide more detailed explanations in the revised manuscript. \n\nQ1.2: When using the neighbor consistency principle, it is important to address whether the same problems mentioned by the authors in lines 44-51 would arise. If not, it is necessary to explain how the proposed model overcomes these challenges.\\\nA: In our method, the feature encoder is fixed and does not change during the training process. As a result, the feature space is determined prior to the introduction of noisy labels and is not subject to distortion by label noise. \n\nQ2.1: It appears that the original CARD paper introduced two pre-trained encoders, while the current paper seems to have used only one. This distinction needs to be clarified, and the roles of the two encoders should be explained separately in the description.\\\nA: The original CARD paper only introduced one pre-trained encoder, which is used as the mean estimator of the diffusion start at $t=T$. This encoder's features were also used as input to CARD's neural network. However, due to its role as the mean estimator, the encoder was restricted to a low dimension (the number of classes), thereby limiting its representation capacity. Intuitively, one might expect that feeding higher-dimension, more powerful features to the network could lead to performance improvements.\\\nTherefore, in our method, we introduced a separate feature encoder specifically to feed conditional information to the network. We denote the mean estimator as $f_q$ (the same dimension as the labels), and the feature encoder (higher dimension) as $f_p$. As a result, the CARD model can be seen as a special case of our model, where $f_p = f_q$. In Table 2, we show CARD+LRA results in poor performance in our ablation study. This design is explained in section 3.4, lines 170-175 of our manuscript. We apologize if this was not clear in our initial presentation, and we will certainly make efforts to clarify this in the revised manuscript. \n\nQ2.2: Equation 4 seems to be adapted from the formulation presented in CARD, and proper citation should be provided.\\\nA: Thanks for pointing out this. We have updated our manuscript with the appropriate reference to the CARD work.\n\nQ3.1: The utilization of CLIP in the performance evaluation raises concerns because it involves the incorporation of additional data. It is important to conduct a fair comparison by addressing this aspect.\\\nA: Please refer to the global response. We have added additional results in Table 3 of the attached PDF file, where simply applying the CLIP features leads to poor performance. We hope this response addresses your concern.\n\nQ3.2: An explanation is needed to clarify why the diffusion model is not a computational bottleneck. It is hypothesized that this may be due to the smaller dimensions, but this should be verified.\\\nA: As depicted in the supplementary figure C.1, all the time embeddings are inserted into the feed-forward layers of the neural network. Thus, the computation of the ResNet encoder is required only once and is the same as that for a standard ResNet classifier. The additional computation cost involves the computation of the $f_p$ encoder and a label-dimension (e.g., 10 in CIFAR10) diffusion process, which is quite efficient. A qualitative comparison in Table 6 shows that the time cost of our method using a ResNet50 as $f_p$ encoder is comparable to a standard ResNet50 classifier, while the performance is much better. \n\nMinor Comments\\\nA: Thank you for identifying the writing errors in our manuscript. We appreciate your attention to detail. We will correct these mistakes and do proofreading to ensure that no further errors remain. \n"}, {"review_id": "ORc9MrQSSw", "reviewer": "Reviewer_poWF", "summary": "This paper proposes a Label-Retrieval-Augmented (LRA) diffusion model for learning from noisy labels. The model leverages the neighbor consistency principle and incorporates pre-trained models to improve performance. The paper introduces the label-retrieval-augmented component, an accelerated label diffusion process, and a new conditional mechanism for incorporating pre-trained models. The proposed model achieves good performance both on synthetic and real-world benchmark datasets and the experimental results show that it can boost accuracy by 10-20 absolute points in many cases.", "questions": "1. The paper mentions that the proposed model is flexible and general, but it does not provide a detailed discussion of the types of label noise that the model can effectively handle. For example, feature-dependent label noise. Since the comparison method [15] is a baseline that focuses on feature-dependent label noise.\n\n2. It seems the pre-trained model is important for the proposed method. It would be better to discuss the effect of different backbone networks as well as pre-trained model on model performance.\n\n3. The description of the comparison algorithms is lacking in the paper.\n\n4. In Table 1, the comparison methods are outdated. It would be better to add some state-of-art methods published in 2022 or 2023.", "limitations": "Please refer to the weakness and limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The idea of using the diffusion model to address the noisy label learning problem is interesting.\n\n2.Extensive experimental results demonstrate the proposed method can achieve the state-of-the-art method compared with multiple peer methods.", "weaknesses": "The experiment part needs to be improved: add state-of-the-art method, discuss the influence of the label noise type, and the influence of different pre-trained models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: The paper mentions that the proposed model is flexible and general, but it does not provide a detailed discussion of the types of label noise that the model can effectively handle. For example, feature-dependent label noise. Since the comparison method [15] is a baseline that focuses on feature-dependent label noise.\\\nA: Following PLC [15], we conduct experiments using their settings on polynomial margin diminishing (PMD) noise, a novel class of synthetic feature-dependent noise (synonyms of instance-dependent noise). This noise is more realistic to real-world label noise and more difficult to handle than i.i.d noise (instance-independent). However, our method can handle instance-dependent noise, since our model approximates the distribution of neighboring labels of each instance, allowing it to make instance-dependent predictions. Results shown in Table 1 demonstrated that our method with the SimCLR feature achieved state-of-the-art results on the PMD noise. We will provide a clearer explanation in the revised manuscript.\n\nQ2: It seems the pre-trained model is important for the proposed method. It would be better to discuss the effect of different backbone networks as well as pre-trained models on model performance.\\\nA: We agree that the network design of diffusion models is an interesting topic. However, the backbone we adopted from the CARD model for 1D diffusion (supplementary figure C.1) is already very efficient and effective, and there are no well-known alternative designs. Thus, the impact of different backbone networks is hard to evaluate. In addition, we believe the quality of the feature has a positive relationship with the classification accuracy.\n\nQ3: The description of the comparison algorithms is lacking in the paper.\\\nA: Thank you for noting the lack of detail in our comparison algorithms. We will add the required information to the supplementary materials in the revised manuscript. \n\nQ4: In Table 1, the comparison methods are outdated. It would be better to add some state-of-art methods published in 2022 or 2023.\\\nA: Table 1 reports the results on synthetic label noise, specifically focusing on the polynomial margin diminishing (PMD)  noise, which is a new class of synthetic instance-dependent label noise. Recent state-of-the-art (SOTA) methods have not performed evaluations on PMD noise in their original papers. To address the problem, we additionally perform experiments using the source code of two SOTA methods, C2D (2022) and CC (2022) and show the results in Table 1 of the attached PDF. C2D also utilizes a pre-trained SimCLR encoder for initialization, but label noise may still affect the feature space during training. In contrast, our method freezes the feature encoder, shielding the pre-trained features from noise. Results on CIFAR-10 demonstrate that when the pre-trained feature is of high quality, our method achieves superior accuracy. On CIFAR-100, where the SimCLR feature has lower KNN accuracy, C2D is more effective, as it can refine the feature space through training. Freezing the feature encoder has another advantage: it enables us to efficiently incorporate more powerful pre-trained encoders, such as CLIP. This approach spares us from the prohibitive computational burden of fine-tuning the feature encoder, allowing for more effective integration.\\\nOn the other hand, we have included more recent SOTA baselines in experiments on real-world datasets. This inclusion is of greater importance, as it reflects performance in real-world applications, providing a more practical and relevant assessment of our method. \n"}]

Author rebuttal:
We thank the reviewers for their valuable comments. We are happy that the reviewers find our manuscript well-written, and demonstrate effective application of our method. A common issue raised by the reviewers is the incorporation of the CLIP feature which uses external data to train. We will address this issue below. For other specific questions raised by each reviewer, we will post our responses separately. We also have added requested experiment results in the reviews in the attached one page pdf file. We will incorporate more detailed revisions into the camera-ready version according to our responses to the reviews.

Q: The utilization of CLIP in the performance evaluation raises concerns because it involves the incorporation of additional data. It is important to conduct a fair comparison by addressing this aspect.\
A: We thank the reviewers for this general comment. We argue that the results do not simply reflect the strength of the CLIP features, for the following reasons. We will incorporate these arguments into the final revision for clarification.

1. We have added multiple baselines in the ablation study, the results are shown in Table 2 of the attached PDF. All the baseline methods use the pre-trained SimCLR and CLIP features. The SimCLR feature is trained on the same training data through contrastive learning without access to external dataset. The results demonstrate that our method is the key to the successful application of pre-trained features in learning from noisy labels. In addition, we want to emphasize that we are the first to develop a method that effectively uses the CLIP feature in learning with noisy labels. This innovation is a significant part of our contribution.

2. We show results of linear probing and KNN using CLIP features in Table 3 of the attached PDF to further demonstrate the effectiveness of our design. The results suggested that utilizing linear probing with LRA and mean labels degrades the pre-trained feature space, leading to diminished performance in comparison to the unsupervised KNN approach. On the other hand, the diffusion process can effectively incorporate the feature space to achieve higher performance.
In addition, we included results of linear probing using average of neighboring labels (denoted as mean) instead of sampling from neighboring labels (denoted as sample) in Table 2 in response to reviewer KNdw’s inquiry. We found that mean labels are more robust than sample labels for linear probing. However, we show both sample and mean label works with the diffusion model and result in similar results. We also included the diffusion result with mean labels to show it is slightly worse than sample labels in most cases on real-world dataset. We have also discussed the use of mean label in Supplementary D2.

3. In Table 5 of the manuscript, we have shown improved results with CC, which isn't trained with external data, on the Clothing1M dataset. This demonstrates that our method can achieve improved results when coupled with other feature-learning methods. In fact, we would like to emphasize that a common practice adopted by most existing methods to achieve state of the arts is by using pretrained models (why wouldn’t if the pretrained models are available?). In our setting, for example, NCR achieves SOTA by the combination of NCR+Mixup+DA, C2D also has to collaborate with ELR+, DivideMix, and SimCLR to achieve SOTA results. We will highlight these hybrid methods in our comparison. Since the LRA diffusion method is orthogonal to many existing techniques, we believe that by combining it with other learning techniques, such as Mixup augmentation and Co-teaching, our method will facilitate the development of new methods for learning from noisy labels in the future.

Meta-review: This paper takes a (deep) generative modeling approach to learning from noisy labels. The proposed approach leverages a diffusion model which iteratively refines a noisy label from an initial random guess. It is based on the recently proposed Classification and Regression Diffusion Models (CARD).

The paper largely received mostly positive reviews. However, one of the reviewers (Reviewer 21mt) expressed some serious concerns regarding novelty. The reviewer felt that the paper is a simple combination of classification diffusion model with retrieval augmentation. The reviewer also expressed concern about the fact that the experiments use a pre-trained CLIP encoder for feature extraction. The latter concern was shared by some of the other reviewers as well.

The authors submitted a detailed rebuttal arguing about the novelty aspect and also reported some additional ablation results showing that it is not just the CLIP encoder but the other aspects of the proposed method that contribute to the performance gain. One of the other reviewers also acknowledged this post-rebuttal.

Regarding the novelty aspects, my own assessment (which is shared by some of the other reviewers as well) is that even though the proposed method is a combination of existing ideas (classification diffusion model and retrieval augmentation), the paper is applying them in an important setting (learning from noisy labels) and that the method works well in practice.

In view of the above points, looking at the reviewers, the rebuttal, and the discussion, and my own reading of the paper, I recommend the paper for acceptance. The authors should incorporate the reviewers' suggestions and address the other points of criticism when preparing the final version.
Decision: Accept (poster)