Title: Label-Retrieval-Augmented Diffusion Models for Learning from Noisy Labels
Venue: NeurIPS 2023
Accepted: yes
Keywords: diffusion model

label noise

retrieval augmented learning
Abstract: Learning from noisy labels is an important and long-standing problem in machine learning for real applications. One of the main research lines focuses on learning a label corrector to purify potential noisy labels. However, these methods typically rely on strict assumptions and are limited to certain types of label noise. In this paper, we reformulate the label-noise problem from a generative-model perspective, *i.e.*, labels are generated by gradually refining an initial random guess. This new perspective immediately enables existing powerful diffusion models to seamlessly learn the stochastic generative process. Once the generative uncertainty is modeled, we can perform classification inference using maximum likelihood estimation of labels. To mitigate the impact of noisy labels, we propose the **L**abel-**R**etrieval-**A**ugmented (LRA) diffusion model, which leverages neighbor consistency to effectively construct pseudo-clean labels for diffusion training. Our model is flexible and general, allowing easy incorporation of different types of conditional information, *e.g.*, use of pre-trained models, to further boost model performance. Extensive experiments are conducted for evaluation. Our model achieves new state-of-the-art (SOTA) results on all the standard real-world benchmark datasets. Remarkably, by incorporating conditional information from the powerful CLIP model, our method can boost the current SOTA accuracy by 10-20 absolute points in many cases. Code is available: https://anonymous.4open.science/r/LRA-diffusion-5F2F

Label-Retrieval-Augmented Diffusion Models for Learning from Noisy Labels

Abstract

Learning from noisy labels is a long-standing problem in machine learning for real applications. One of the main research lines focuses on learning a label corrector to purify potential noisy labels. However, these methods typically rely on strict assumptions and are limited to certain types of label noise. In this paper, we reformulate the label-noise problem from a generative-model perspective, i.e. , labels are generated by gradually refining an initial random guess. This new perspective immediately enables existing powerful diffusion models to seamlessly learn the stochastic generative process. Once the generative uncertainty is modeled, we can perform classification inference using maximum likelihood estimation of labels. To mitigate the impact of noisy labels, we propose L abelR etrievalA ugmented (LRA) diffusion model 1 , which leverages neighbor consistency to effectively construct pseudo-clean labels for diffusion training. Our model is flexible and general, allowing easy incorporation of different types of conditional information, e.g. , use of pre-trained models, to further boost model performance. Extensive experiments are conducted for evaluation. Our model achieves new state-of-the-art (SOTA) results on all standard real-world benchmark datasets. Remarkably, by incorporating conditional information from the powerful CLIP model, our method can boost the current SOTA accuracy by 10-20 absolute points in many cases.

1 Introduction

Deep neural networks have achieved extraordinary accuracy on various classification tasks. These models are typically trained through supervised learning using large amounts of labeled data. However, large-scale data labeling could cost huge amount of time and effort, and is prone to errors caused by human mistakes or automatic labeling algorithms [1]. In addition, research has shown that the ability of deep neural network models to fit random labels can result in reduced generalization ability when learning with corrupted labels [2]. Therefore, robust learning methods using noisy labels are essential for applying deep learning models to cheap yet imperfect datasets for supervised learning tasks.

There are multiple types of label noise investigated by previous research. More recent research has focused on studying the more realistic feature-dependent label noise, where the probability of mislabeling a given instance depends on its characteristics. This type of noise is more consistent with label noise in real-world datasets [3, 4, 1, 5, 6, 7]. To address this type of noise, a model is expected to be able to estimate the uncertainty of each training label. Many state-of-the-art methods have primarily relied on the observation that deep neural networks tend to learn simple patterns before

<latexi sh

1_b

64="

7

V8mRW

+

p

qO

>A

B

c

DLS

N

F

2

r

f

Zdu ovg

T

0

J

X9

EM

G

H

y

U

P

wY

5

z

I

K

n

/

k

Q

j

<latexi sh

1_b

64="O

NWM

8g o5nPu

+0

zUj

I

>A

B7X

c

VDLS

E

y

r

f

Y9

F

K

2

Q

HJ

mZ

p

/

wd

v

3

k

G

q

T

C

R

<latexi sh

1_b

64="O

H

9

/p

VPLfUj

ZG

zg

FnE

>A

B

X

c

D

S

N

3

W

ur

J

K+

k

vY

7

T

d

m

y

25

w

q

R

Q

I

o

M

C

8

0

<latexi sh

1_b

64="Y

3UnQd

ZDF80f

C

IN

H2X

M

>A

c

V

LS

J

rP

m

u

W

K

g

9

G

O

q

T

5

j

/ok7

v

w

E

R

y

p

z

<latexi sh

1_b

64="O

NWM

8g o5nPu

+0

zUj

I

>A

B7X

c

VDLS

E

y

r

f

Y9

F

K

2

Q

HJ

mZ

p

/

wd

v

3

k

G

q

T

C

R

<latexi sh

1_b

64="XG+

qI

P

p

98d

7W

5FcgHwJEY

>A

B

3

VDLS

M

T

v

y

r2

Rj

K

Q

mk

0

/o

f

Z

u

n

z

O

U

N

C

y

T

planes

cars

birds

cats

deer

dogs

frogs

horses

ships

trucks

y

0

planes

cars

birds

cats

deer

dogs

frogs

horses

ships

trucks

Label denoising process

Label noising process

· · ·

y

t

planes

cars

birds

cats

deer

dogs

frogs

horses

ships

trucks

y

t

-

1

planes

cars

birds

cats

deer

dogs

frogs

horses

ships

trucks

· · ·

clean label

memorizing random noise [8, 9]. This means a temporary phase exists in the learning process, where the model has learned useful features but has yet not started overfitting corrupt labels. At this stage, the model predictions can be used to modify the training labels, so that they are more consistent with model predictions [10, 11, 12, 13, 14, 15]. By correctly modifying the labels, the number of clean training sample increases, which can further benefit the training. These type of approaches, however, is inherently risky because the point at which the model starts to overfit varies with the network structure and dataset. Starting too early can corrupt the training data, while starting too late may not prevent overfitting [16]. Therefore, it is vital to carefully tune the hyper-parameters of the training strategy, such as the number of epochs for warm-up training, learning rate, and uncertainty threshold, to achieve successful training results.

Another class of methods adopts the assumption of label propagation in semi-supervised learning [17, 18], where nearby data points in a feature space tend to have the same label. Therefore, they use neighbor consistency 2 regularization to prevent overfitting of the model [19, 20]. The performance highly depends on the quality of the encoder that maps the data to the feature space, as retrieving a neighbor that belongs to a different class could further mislead the training process. Encoders are therefore required to first learn high-level features of the data that can be used for classification, which could be trained simultaneously with the classifier using noisy labels. However, the training can also lead to overfitting or underfitting.

In this paper, by contrast, we formulate the label noise problem from a generative-model perspective, which naturally provides new insights into approaching the problem. Our intuition is to view the noisy labeling process as a stochastic label generation process. Thus, we propose to adopt the powerful diffusion model as the generative building block. Figure 1 illustrates our intuition. In the generative process, we start with a noisy estimation of the label, then gradually refine it to recover the clean label, which is equivalent to the reverse denoising process of the diffusion model.

Specifically, the diffusion model takes a noisy label and some useful conditional information (to be specified) as inputs, and learns to recover/generate the ground-truth labels as outputs. One challenge in this setting is that only noisy labels are available in practice for training. To overcome this issue, we adopt the principle of neighbor consistency , and propose label-retrieval augmentation to construct pseudo clean labels for diffusion model training, where a pre-trained image encoder is used to define the neighborhood of a sample. It is worth noting that the pre-trained image encoder would not be affected by the label noise, because they can be trained in a self-supervised manner [21, 22] or on an additional clean dataset [23, 24]. In fact, pre-training can tremendously improve the model's adversarial robustness [25] and has been used to improve model robustness to label corruption [26]. Another merit of our design is that it is general enough to allow natural incorporation of powerful large pre-trained model such as the CLIP model to further boost the performance.

In addition, the probability nature of diffusion models can also be better equipped to handle uncertainty in the data and label, thus providing more robust and accurate predictions. We call our model LRAdiffusion (label-retrieval-augmented diffusion).

Our main contributions are summarized as follows:

We formulate learning from noisy labels as modeling a stochastic process of conditional label generation, and propose to adopt the powerful diffusion model to learn the conditional label distribution.

We incorporate the neighbor consistency principle into the modeling, and design a novel label-retrieval-augmented diffusion model to learn effectively from noisy label data.

We further improve our model by incorporating auxiliary conditional information from large pre-trained models such as CLIP.

Our model achieves the new state-of-the-art (SOTA) in various real-world noisy label benchmarks, e.g. , 20% accuracy improvement on noisy CIFAR-100 benchmark.

2 Preliminary

Diffusion models were initially designed for generative modeling. Recently, it has been extended for classification and regression problems. In this section, we introduce the Classification and Regression Diffusion Models (CARD) [27], which our model is based on.

The CARD model transforms deterministic classification into a conditional label generation process, allowing for more flexible uncertainty modeling in the labeling process [27]. Similar to the standard diffusion model, CARD consists of a forward process and a reverse process. In the forward process, an n -dimensional one-hot label y 0 is gradually corrupted to a series of intermediate random vectors y 1: T , which converges to a random variable with a multi-variant Gaussian distribution N ( f q ( x ) , I ) (latent distribution) after T steps, where the mean is defined by a pre-trained n -dimensional image encoder f q . The transition steps between adjacent intermediate predictions is modeled as Gaussian distributions, q ( y t | y t -1 , f q ) = N ( y t ; µ t , β t I ) , with mean values µ 1 , · · · , µ T and a variance schedule β 1 , · · · , β T , where µ t = √ 1 -β t y t -1 +(1 - √ 1 -β t ) f q ( x ) . This admits a closed-form sampling distribution, q ( y t | y 0 , f q ) = N ( y t ; µ t , (1 -¯ α t ) I , with an arbitrary timestep t and µ t = √ ¯ α t y 0 +(1 - √ ¯ α t ) f q ( x ) . The mean term can be viewed as an interpolation between true data y 0 and the mean of the latent distribution f q ( x ) with a weighting term ¯ α t = ∏ t (1 -β t ) .

In the reverse (generative) process, CARD reconstructs a label vector y 0 from an n -dimensional Gaussian noise y T ∼ N ( f q ( x ) , I ) by approximating the denoising transition steps conditioned on the data point x and another pre-trained image encoder f p in an arbitrary dimension. The transition step is also Gaussian for an infinitesimal variance β t [28] (define ˜ β t = 1 -¯ α t -1 1 -¯ α t β t ):

The diffusion model is learned by optimizing the evidence lower bound with stochastic gradient descent:

Following [29], the mean term is written as µ θ ( y t , x , f p , t ) = 1 √ α t ( x t -β t √ 1 -¯ α t ϵ θ ( y t , x , f p , t )) and the objective can be simplified to L simple = ∥ ϵ -ϵ θ ( y t , x , f p , t ) ∥ 2 .

3 Label-Retrieval-Augmented Diffusion Model

Inspired by CARD, Label-Retrieval-Augmented (LRA) diffusion models reframe learning from noisy labels as a stochastic process of conditional label generation ( i.e. , label diffusion) process. In this section, we first provide an overview of the our model in Section 3.1 and then introduce the proposed label-retrieval-augmented component in Section 3.2, which can leverage label consistency in the training data. Next, we introduce an accelerated label diffusion process to significantly reduce classification model inference time in Section 3.3. Finally, a new conditional mechanism is proposed to enable the usage of pre-trained models in Section 3.4.

3.1 Model Overview

Our overall label-retrieval-augmented diffusion model is illustrated in Figure 2, where a diffusion model is adopted for progressively label denoising, by leveraging both the retrieved labels and auxiliary information from pre-trained models. Our model employs two pre-trained networks, denoted as f q and f p encoders, to encode conditional information that facilitates the generation process. The f q encoder serves as a mean estimator for y T , providing an initial label guess for a given image. This encoder could be a standard classifier trained on noisy labels. On the other hand, the f p encoder operates as a high-dimensional feature extractor, assisting in guiding the reverse procedure. y t and f p ( x ) are concatenated together before being processed. Details about our neural-network architecture design are provided in Supplementary C.

During training, we use labels retrieved from the neighborhood as the generation target y 0 . Then, in the forward process, the distribution of neighboring labels is progressively corrupted towards a standard Gaussian distribution centered at the estimated mean f q ( x ) . During testing, we employ a generalized DDIM method to efficiently compute the maximum likelihood estimation of y 0 .

y

T

y

t

y

t

-

1

· · ·

N

(

f

q

(

x

)

,

I

)

pre-trained encoder

f

q

· · ·

y

0

pre-trained encoder

f

p

{

y

(1)

,

· · ·

,

y

(

k

)

}

Label distribution

of neighbors

latent

distribution

reverse process

forward process

q

(

y

t

|

y

t

-

1

, f

q

)

p

θ

(

y

t

-

1

|

y

t

,

x

, f

p

)

Diffusion model

Conditional information

Label retrieval

3.2 Label-retrieval Augmentation for Training

The noisy nature of labels excludes the availability of clean labels for training. To mitigate the issue, we propose a training strategy based on the concept of retrieval augmented learning [30, 31], such that it is more resistant to label noise. Our main assumption is that in a latent space, data points from different classes form distinctive clusters. Therefore, the majority of a data point's neighbors are expected to have the same label as the point itself. To this end, we used a pre-trained encoder, illustrated as f p in Figure 2, to map the data into an embedding space and retrieve a label y ′ from labels of the k nearest neighbors { y (1) , · · · , y ( k ) } in the training set. The diffusion model was then trained to learn the conditional distribution p ( y ′ | x ) of labels within the neighborhood, rather than the distribution of labels p ( y | x ) for the data point itself. We empirically select the value of k based on the KNN accuracy obtained on the validation data, as detailed in Supplementary Section C.

Label-retrieval augmentation enables the model to make use of the information from multiple and potentially more accurate labels to improve its prediction performance. Algorithm 1 describes the training procedure. Additionally, diffusion models are known to be effective at modeling multimodal distributions. By training the model to generate different labels from neighbors based on the same data point, the model can produce stochastic predictions based on the distribution to capture the uncertainty inherent in the data labeling process. As a result, the trained model can be used not only as a classifier, but also as a sampler that simulates the actual labeling process.

3.3 Efficient Inference with generalized DDIM

The iterative generation nature of the classification diffusion model makes its inference efficiency not comparable to traditional classifiers. To enhance the inference efficiency, we propose to incorporate the efficient sampling methods, Denoising Diffusion Implicit Model (DDIM) [32], to accelerate the

<latexi sh

1_b

64="M

28

WVGJmP

I

7

FfH

w

>A

CEX

c

DLS

N

3U

oQ

pZ

R

YB

T

dO

n

j

/

5

+

9

u

y

zr

K

k

<latexi sh

1_b

64="gM

zdY0

ZL

vu79XSqf

y

n

GQc

>A

B/H

VD

N

FJ3U

2

o

T

KC

m

O

R

P

+

w

8

5

k

r

E

p

W

I

j

<latexi sh

1_b

64="B

U

9q

P

2J

p

m

Y/v

WG

HI

>A

C

c

V

5TgM

F

Ry

n

ES

O0

rjK

3

Z

f

k

D

Q

+

7

z8wuL

Xd

o

N

<latexi sh

1_b

64="Uw

NZ9

7

8S

/kPQ+2

y

nY

>A

B

H

c

VDJ

g

F

T

3qE

vj

G

R

0

CW

fI

X

p

O

d

K

ou

5

r

L

m

M

z

<latexi sh

1_b

64="

7

3C

V8mRW

+

p

qO

>A

B

c

DLS

N

F

2

r

f

Zdu ovg

T

0

J

X9

EM

G

H

y

U

P

wY

5

z

I

K

n

/

k

Q

j

<latexi sh

1_b

64="O

NWM

8g o5nPu

+0

zUj

I

>A

B7X

c

VDLS

E

y

r

f

Y9

F

K

2

Q

HJ

mZ

p

/

wd

v

3

k

G

q

T

C

R

<latexi sh

1_b

64="O

H

9

/p

VPLfUj

ZG

zg

FnE

>A

B

X

c

D

S

N

3

W

ur

J

K+

k

vY

7

T

d

m

y

25

w

q

R

Q

I

o

M

C

8

0

<latexi sh

1_b

64="

0U

3

kwJ9S

7Y

>A

C

c

VDL

gM

F

2vq

I

dO

y

W

+

5

/

B

XR

8E

G

o

N

H

f

z

P

K

<latexi sh

1_b

64="/

LI

JG7

u3qXAo rDK

c

>

CEn

V

S

N

F

U

2v g0

Q

pZ

y

mk

5

RB

+

z

Wdj

O

9

f

P

MH

w

Y

T

8

<latexi sh

1_b

64="Km q2L

NZJR9

FB

kguIYp

>A

/3

c

VD

S

Oy

W

P

TE

n

M

Xj

G

0

of

d

H

z

78

v

5

U

r

w

C

<latexi sh

1_b

64="Gyz2

HX8W

dw5IYPD

Q

f

+

>A

B/3

c

VC7SgN

FJ

r

jK

U

n

T

k9

M

0

O

L

Z

p

m

E

R

o

v

u

q

<latexi sh

1_b

64="oY

qW78

ZJEd

2

39TB+yQ

c

>A

C

H

V

SgN

F

r

fU

L

X

jK

I

O

k

0

M

5

p

m

v

/

R

D

G

P

n

<latexi sh

1_b

64="Y

3UnQd

ZDF80f

C

IN

H2X

M

>A

c

V

LS

J

rP

m

u

W

K

g

9

G

O

q

T

5

j

/ok7

v

w

E

R

y

p

z

<latexi sh

1_b

64="O

NWM

8g o5nPu

+0

zUj

I

>A

B7X

c

VDLS

E

y

r

f

Y9

F

K

2

Q

HJ

mZ

p

/

wd

v

3

k

G

q

T

C

R

<latexi sh

1_b

64="XG+

qI

P

p

98d

7W

5FcgHwJEY

>A

B

3

VDLS

M

T

v

y

r2

Rj

K

Q

mk

0

/o

f

Z

u

n

z

O

U

N

C

<latexi sh

1_b

64="jp

G87BCrIqXV

D

mng

T

z3EU

>A

F

c

LS

N

J

P

d

o

+

k0

Z

H

9

K25

O

W

/

M

v

u

f

R

Q

y

w

Y

<latexi sh

1_b

64="

c0Hwq5

B

TvSg7WEmY8

D

>A

C

n

V

N

FJ2

r

f

Z

j

PO

zk

M3

Q

XG

L

UI

+d

K

9

R

o

/

p

u

y

<latexi sh

1_b

64="

QByj k9

VL

gN

>A

H

D

S

EJz

G

r

U

v

0

wF

o

WM

I

u

m

n

Y

X

R

3

P

Z

5

O

p

T

+

2

C

8d

q

f

Algorithm 1 Training

Input:

1: while not converged do

training set { X , Y } , image encoder f p , f q .

2: Sample data ( x , y ) ∼ { X , Y } ; time slice t ∼ { 1 , · · · , T } ; and noise ϵ ∼ N (0 , I )

4: Sample y ′ ∼ { y, y (1) , · · · , y ( k ) } , and convert it to a one-hot vector y 0

3: Retrieve labels { y (1) , · · · , y ( k ) } of neighbors in the feature space defined by the encoder f p

5: Take gradient descent step on the loss:

6: end while

label diffusion process. However, the utilization of the mean estimator f q makes DDIM incompatible with our setting, as our generation process begins with a non-zero mean Gaussian distribution N ( f q ( x ) , I ) . Therefore, we adjust the DDIM method into a more general form that fits our framework. Analogous to DDIM, our sampling process maintains the same marginal distribution as the original q ( y t | y 0 , f q ) closed-form sampling process. Detailed derivations can be found in Supplementary A.

With DDIM, the trained model can generate a label vector in much less steps following a pre-defined sampling trajectory of { T = τ S >, · · · , > τ 1 = 1 } , where S < T . Consequently, y t can be computed as:

where ϵ ∼ N ( 0 , I ) . Similar to CARD [27], we predict the denoised label ˜ y 0 , a prediction of y 0 given y τ s , as:

When τ s -1 > 0 , we can compute y τ s -1 given y τ s from the non-Markovian forward process defined as:

As the dimension of label vectors is usually much lower than that of an image, the model can employ fewer steps in the reverse process without compromising generative quality. In our experiments, we use S = 10 and T = 1000 , substantially reducing the time cost of the classification process. Supplementary Figure B gives an example of the label generation (classification) process on the CIFAR-10 dataset.

To further enhance the inference efficiency, we propose a simple and effective trick for computing the maximum likelihood estimation of labels. As the generative process is deterministic given y T , which is sampled from a uni-modal Gaussian distribution, we approximate the maximum likelihood estimation of labels by initiating from the mean, i.e. , y 0 = DDIM ( y T = f q ( x ) , x ) . This trick circumvents the time-consuming majority voting approximation that demands repeated generation.

3.4 Flexible conditioning with pre-trained encoders

The original CARD model employs a single model for both the f p and f q encoders. However, this limits their representation capacity [33] as the dimension of f q ( x ) is typical relatively small, i.e. , equalling the number of classes. To mitigate this and improve model performance, we abandon the assumption that f p = f q , enabling the use of a more powerful pre-trained encoder ( e.g. , the CLIP image encoder [24]) with arbitrary dimensions for f p .

Empirically, we find that the model can still achieve satisfactory performance when the magnitude of f q ( x ) is small, which means the latent representation y T = f q ( x ) + ϵ is dominated by the noise term ϵ ∼ N ( 0 , I ) . In this case, the information provided by f q ( x ) to the diffusion process is limited. As a result, we simply set f q ( x ) = 0 to avoid handling an additional n-dimensional f q encoder. For the f p encoder, one can employ flexible pre-trained models as presented in Section 5. In this paper, we use the SimCLR model trained on the training images (without supervised information) and the pre-trained CLIP model.

4 Related work

Robust loss function and regularization techniques. Several noise-robust loss functions and regularization techniques have been proposed as alternatives to the commonly used cross-entropy loss (CE), which is not robust to label noise. Mean absolute error (MAE) [34] loss has been shown to be robust against noisy labels. Generalized Cross-Entropy (GCE) [35] combines CE and MAE for faster convergence and better accuracy. Symmetric cross-entropy Learning (SL) [36] couples CE with a noise-robust counterpart and has been found to have higher performance than GCE, particularly for high noise rates. Label Smoothing Regularization [37] alleviates overfitting by linearly combining labels with a uniform distribution. Bootstrapping technique [38] combines the labels with the current model prediction. Dynamic bootstrapping [39, 40] uses the prediction confidence to control the weighting in the combination. Neighbor Consistency Regularization (NCR) [19] encourages consistency of prediction based on learned similarity. Our method is also based on the principle of neighbor consistency. However, instead of encouraging consistent predictions among neighbors, our model directly learns from the labels of neighbors. This allows for estimating instance-level uncertainty by learning the label distribution among neighbors, rather than learning a point estimation.

Data recalibration. Data recalibration techniques progressively remove or correct mislabeled data during training to improve the reliability of training data. Wang et al. [11] used the learned similarity and label consistency to identify and discard data with noisy labels. TopoFilter [41] selects clean data by analyzing the topological structures of the training data in the learned feature space. Cheng et al. [4] defines a Bayes optimal classifier to correcct labels. Zheng et al. [14] proposed using a likelihood ratio test (LRT) to correct training labels based on predictions. Zhang et al. [15] used LRT to correct labels progressively and provides a theoretical proof for convergence to the Bayes optimal classifier. Dividemix [42], LongReMix [43], and CC [44] treat the low confident data as unlabeled, and then employ semi-supervised learning algorithms [45] for further analysis. C2D [46] combines Dividemix with self-supervised pre-training to boost its performance by improving the quality of the extracted features. Our approach employs the same assumption as TopoFilter that data belonging to the same class should be clustered together with ideal feature representations. However, our technique isn't confined to learned features potentially distorted by label noises. Instead, similar to C2D, our method can effectively leverage the high-quality feature learned by pre-trained encoders to achieve superior accuracy.

Guided diffusion model and retrieval augmentation. Guided diffusion is a technique applied to diffusion models for conditional generation. Classifier guidance [47] is an cost-effective method leveraging the gradient of a classifier to steer the generative process of a trained diffusion model. On the other hand, Classifier-free guidance [48] learns the conditional distribution during training for improved generation quality. This approach also allows for the use of continuous guidance information, such as embedding vectors, rather than being limited to discrete labels. Classification and Regression Diffusion Models (CARD) [27] formulates classification and regression as a conditional generation task that generates labels or target variables conditioned on images. Our approach follows the same paradigm, and leverages the multi-modal coverage ability of diffusion models to learn the label distribution within the neighborhood. SS-DDPM [49] proposes a diffusion model with arbitrary noising distributions defined on constrained manifold, e.g., the probabilistic simplex for label generation. Retrieval-augmented diffusion models [30] used retrieved neighbors from an external database as conditional information to train diffusion models for image synthesis. Retrieval Augmented Classification [31] used retrieval-augmentation to train classification model using classimbalanced training data. Our approach differs from theirs by retrieving labels instead of data to reduce label noise in training rather than increasing the training data. In addition, our model does not require an external database.

5 Experiments

We first evaluate the performance of our method on datasets with various types synthetic noises. Then, we perform experiments on four real-world datasets. To better understand the performance gain sources, we conduct ablation studies to measure the impacts of conditional diffusion and different pseudo-label construction strategies. All experiments were done on four NVIDIA Titan V GPUs. Comprehensive implementation details and hyper-parameters are provided in the Supplementary C.

5.1 Results on Synthetic Noisy Datasets

We conduct simulation experiments on the CIFAR-10 and CIFAR-100 datasets [51] to evaluate our method's performance under various noise types. Specifically, following [15], we test with polynomial margin diminishing (PMD) noise, a novel instance-dependent noise, at two noise levels and three hybrid noise types by adding independent and identically distributed (i.i.d) noises on top of instance-dependent noise.

For instance-dependent noise, we adopt the recently proposed polynomial margin diminishing (PMD) noise [15]. Following the original paper, we train a classifier η ( x ) using clean labels to approximate the probability mass function of the posterior distribution p ( y | x ) . Images are initially labeled as their most likely class u x according to the predictions of η ( x ) . Then, we randomly alter the labels to the second most likely class s x for each image with probability: p u x ,s x = -c 2 [ η u x ( x ) -η s x ( x ) ] 2 + c 2 , where c is a constant noise factor that controls the final percentage of noisy labels. Since corrupting labels to the second most likely class can confuse the "clean" classifier the most, it is expected to have the most negative impact on the performance of models learned with noisy labels. For PMD noise, we simulate two noise levels where 35% and 70% of the labels are corrupted.

For i.i.d noise, following [52, 15], we use a transition probability matrix T to generate noisy labels. Specifically, we corrupt the label of the i -th class to the j -th class with probability T ij . We adopt two types of i.i.d noise in this study: (1) Uniform noise, where samples are incorrectly labeled as one of the other ( n -1) classes with a uniform probability T ij = τ/ ( n -1) and T ii = 1 -τ , with τ the pre-defined noise level; (2) Asymmetric noise: we carefully design the transition probability matrix such that for each class i , the label can only be mislabeled as one specific class j or remain unchanged with probability T ij = τ and T ii = 1 -τ . In our experiment, we generated three types of hybrid noise by adding 30%, 60% uniform, and 30% asymmetric noise on top of 35% PMD noise.

We test our proposed label-retrieval-augmented diffusion model using two pre-trained encoders: (1) SimCLR [21]: We trained two encoders using the ResNet50 [53] architecture on the CIFAR-10 and CIFAR-100 datasets through contrastive learning; (2) CLIP [24]: the model is pre-trained on a large dataset comprising 400 million image-text pairs. Specifically, we used the vision transformer [54] encoder (ViT-L/14) with pre-trained weights, the best-performing architecture in CLIP. For

simplification, we refer to these configurations as LRA-diffusion (SimCLR) and LRA-diffusion (CLIP). We also investigated the performance of the KNN algorithm within the feature space defined by the SimCLR and CLIP encoders, denoted as SimCLR KNN and CLIP KNN respectively.

Table 1 lists the performance of the Standard method (train classifier using noisy labels), our method, and baseline methods for learning from noisy labels. The results in white rows are borrowed directly from [15]. We can see that using the SimCLR encoder in the LRA-diffusion method results in superior test accuracy on both CIFAR-10 and CIFAR-100 datasets compared to other baselines, without the need for additional training data. This is because the SimCLR encoder is trained in an unsupervised manner, making it immune to label noise, and it can effectively extract categorical features for accurate image retrieval. Therefore, when the correct labels dominate the label distribution in the neighborhood, training with the labels of the retrieved neighbor images allows the model to learn with more correct labels. C2D [46] also utilizes a pre-trained SimCLR encoder for initialization, but label noise may still affect the feature space during training. In contrast, our method freezes the feature encoder, shielding the pre-trained features from noise. Results on CIFAR-10 demonstrate that when the pre-trained feature has high KNN accuracy, our method performs better. On CIFAR-100, where the SimCLR feature has lower KNN accuracy, C2D is more effective, as it can refine the feature space through training. Moreover, freezing the feature encoder allows for efficient integration of large pre-trained encoders like CLIP, saving us from the prohibitive computational cost of fine-tuning. Notably, incorporating the CLIP encoder into our method significantly improves test accuracy over our LRA-diffusion (SimCLR) due to its excellent representation capabilities. In fact, by performing KNN in the CLIP feature space alone was able to achieve accuracy surpassing all competing methods in most experiments. This allows for the use of more clean labels during training, thus result in even higher accuracy.

5.2 Ablation Studies

To evaluate the contribution of diffusion and the pre-trained features, we conducted ablation experiments using CARD [27], SS-DDPM [49], and linear probing to incorporate pre-trained models. It is worth noting that the SimCLR model was trained on the same training set without access to external data. Results are given in Table 2.

Linear probing with sampled labels yielded lower accuracy than using noisy labels or the mean of neighboring labels. This difference may be due to the linear layer's inability to yield stochastic outputs from a multimodal distribution. During training, conflicting gradient directions may arise if the model tries to predict different labels across gradient steps, which can impede learning. However, due to the mode coverage ability of the diffusion model, our method can effectively learn from retrievalaugmented labels to generate different labels with different probabilities. We also test a baseline using an additional ResNet encoder along with the linear layer to mimic our model architecture shown in Figure C.1. The results are comparable with linear probing with sampled labels. Our model

significantly outperforms CARD, mainly due to the more informative f p encoder. We use the same model architecture for SS-DDPM, which uses Dirichlet distributions for noisy states. SS-DDPM performed slightly worse than our method, indicating that constraining noisy states within probability simplex may not benefit our task.

Additional ablation studies are included in the Supplementary D. The results demonstrate that in the absence of a pre-trained encoder, our model can leverage the features of a noisy classifier to enhance its accuracy. We also include results of ablation experiments using different approaches for incorporating the CLIP model. The results indicate that the superior performance of our method is not solely attributable to the strength of the CLIP features. In conclusion, our LRA-diffusion model provides an efficient approach for incorporating pre-trained encoders for learning from noisy labels.

5.3 Results on Real-world Noisy Datasets

We further evaluate the performance of our proposed method on real-world label noise. Following previous work [42, 15, 44, 43], we conducted experiments on four image datasets, i.e. , WebVision [55], ImageNet ILSVRC12 [56], Food-101N [57], and Clothing1M [58]. For experiments on Webvision, ILSVRC12, and Food-101N datasets, we use the CLIP image encoder as the f p encoder to train LRA-diffusion models. Comprehensive dataset description and implementation details can be found in the Supplementary C. We evaluated the performance of our method against a group of state-of-the-art (SOTA) methods. The results are presented in Table 3 and Table 4. Our approach significantly outperforms all the previous methods in terms of classification accuracy. It is important to highlight that EPL [59] incorporates the most powerful CLIP and ConvNext-XL [60] encoders and cooperates with other SOTA methods such as ELR [61], DivideMix [42], and UNICON [62]. However, our method outperforms EPL by achieving ∼ 6% higher accuracy on WebVision and ILSVRC12 datasets. This improvement over EPL demonstrates that developing better ways to incorporate pre-trained models to facilitate learning from noisy labels is a non-trivial task, highlighting the valuable contribution of our approach.

For experiments on the Clothing1M dataset, we found that LRA-diffusion conditioned on the CLIP image encoder did not achieve the SOTA accuracy. A potential explanation is that the CLIP feature is too general for this domain specific task for categorizing fashion styles. However, our method is orthogonal to most traditional learning with noisy label approaches. As shown in the additional ablation study in Supplementary D.1, our method can collaborate with a trained classifier by conditioning on its feature encoder to achieve improved performance. We first use the CC [44] method to select clean samples and train a ResNet50 classifier, which achieved 75 . 32% accuracy (refer to as CC ∗ ). Then, we condition on its feature before the classification head to train our LRAdiffusion model on the selected samples, which achieved 75 . 70% accuracy. As Table 5 shows, our method achieved a 0.38% improvement based on CC ∗ and beat all SOTA methods.

5.4 Inference Efficiency Analysis

In order to test the efficiency of our model, we perform experiments assessing the runtime on CIFAR10 dataset and compare our method with a standard classifier that uses ResNet50. It's worth noting that our SimCLR encoder is also built on the ResNet50. Thus, the standard method's runtime also reflects the linear probing runtime on SimCLR. Table 6 shows the results.

We can see, the computation bottleneck lies on the large pre-trained encoder but not the diffusion model itself. In general, our method takes twice as long as a standard classifier (ResNet50) when using SimCLR (ResNet50) and CLIP (ViT-B/32) pre-trained encoders. Larger CLIP encoders can increase the time further. However, it can be further accelerated if the features can be pre-computed in advance or be computed in parallel (as they are only required to be computed once and can be reused later).

6 Limitations

Our method, while being effective in many scenarios, does have certain limitations that we acknowledge. Its performance enhancement can be compromised if a pre-trained f p feature encoder isn't available or is inadequately trained. Additionally, the diffusion model introduces Gaussian noise in the forward process, leading to latent label vectors not being confined within the probability simplex, which could increase training time. Lastly, our method's performance becomes less effective when label noise levels surpass 50%. However, supervised learning can not be the optimal choice in such situations.

7 Conclusion

In this paper, by viewing the noisy labeling process as a conditional generative process, we leverage diffusion models to denoise the labels and accurately capture label uncertainty. A label-retrievalaugmented diffusion model was proposed to effectively learn from noisy label data by incorporating the principle of neighbor consistency. Additionally, by incorporating auxiliary conditional information from large pre-trained models such as CLIP, we are able to significantly boost the model performance. The proposed model is tested on several benchmark datasets, including CIFAR-10, CIFAR-100, Food101N, and Clothing1M, achieving state-of-the-art results in most experiments. Future work could extend our model to multi-label settings, as it does not require a one-hot representation for labels. It is also promising to use semantic segmentation to guide the generation, potentially enhancing our model's interpretability and performance.

Acknowledgement: This work is partially supported by NSF AI Institute-2229873, NSF RI2223292, an Amazon research award, and an Adobe gift fund. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation, the Institute of Education Sciences, or the U.S. Department of Education.

References

Yu Yao, Tongliang Liu, Mingming Gong, Bo Han, Gang Niu, and Kun Zhang. Instancedependent label-noise learning under a structural causal model. Advances in Neural Information Processing Systems , 34:4409-4420, 2021.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107115, 2021.

Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. Advances in Neural Information Processing Systems , 33:7597-7610, 2020.

Jiacheng Cheng, Tongliang Liu, Kotagiri Ramamohanarao, and Dacheng Tao. Learning with bounded instance and label-dependent label noise. In International Conference on Machine Learning , pages 1789-1799. PMLR, 2020.

Antonin Berthon, Bo Han, Gang Niu, Tongliang Liu, and Masashi Sugiyama. Confidence scores make instance-dependent label-noise learning possible. In International Conference on Machine Learning , pages 825-836. PMLR, 2021.

De Cheng, Tongliang Liu, Yixiong Ning, Nannan Wang, Bo Han, Gang Niu, Xinbo Gao, and Masashi Sugiyama. Instance-dependent label-noise learning with manifold-regularized transition matrix estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16630-16639, 2022.

Songhua Wu, Mingming Gong, Bo Han, Yang Liu, and Tongliang Liu. Fair classification with instance-dependent label noise. In Conference on Causal Learning and Reasoning , pages 927-943. PMLR, 2022.

Devansh Arpit, Stanisław Jastrz˛ ebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning , pages 233-242. PMLR, 2017.

Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks. In International conference on artificial intelligence and statistics , pages 4313-4324. PMLR, 2020.

Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5552-5560, 2018.

Yisen Wang, Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, and Shu-Tao Xia. Iterative learning with open-set noisy labels. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 8688-8696, 2018.

Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International conference on machine learning , pages 2304-2313. PMLR, 2018.

Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S Kankanhalli. Learning to learn from noisy labeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5051-5059, 2019.

Songzhu Zheng, Pengxiang Wu, Aman Goswami, Mayank Goswami, Dimitris Metaxas, and Chao Chen. Error-bounded correction of noisy labels. In International Conference on Machine Learning , pages 11447-11457. PMLR, 2020.

Yikai Zhang, Songzhu Zheng, Pengxiang Wu, Mayank Goswami, and Chao Chen. Learning with feature-dependent label noise: A progressive approach. arXiv preprint arXiv:2103.07756 , 2021.

Lutz Prechelt. Early stopping-but when? In Neural Networks: Tricks of the trade , pages 55-69. Springer, 2002.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5070-5079, 2019.

Ahmet Iscen, Jack Valmadre, Anurag Arnab, and Cordelia Schmid. Learning with neighbor consistency for noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4672-4681, 2022.

De Cheng, Haichun Tai, Nannan Wang, Zhen Wang, and Xinbo Gao. Neighbour consistency guided pseudo-label refinement for unsupervised person re-identification. arXiv preprint arXiv:2211.16847 , 2022.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Yuanyi Zhong, Haoran Tang, Junkun Chen, Jian Peng, and Yu-Xiong Wang. Is self-supervised learning more robust than supervised learning? arXiv preprint arXiv:2206.05259 , 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026-1034, 2015.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021.

Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In International Conference on Machine Learning , pages 27122721. PMLR, 2019.

Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. Training convolutional networks with noisy labels. arXiv preprint arXiv:1406.2080 , 2014.

Xizewen Han, Huangjie Zheng, and Mingyuan Zhou. Card: Classification and regression diffusion models. arXiv preprint arXiv:2206.07275 , 2022.

William Feller. On the theory of stochastic processes, with particular reference to applications. In Selected Papers I , pages 769-798. Springer, 2015.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Björn Ommer. Semiparametric neural image synthesis. In Advances in Neural Information Processing Systems , 2022.

Alexander Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Nguyen, Pulak Purkait, Ravi Garg, Alan Blair, Chunhua Shen, and Anton van den Hengel. Retrieval augmented classification for long-tail visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6959-6969, 2022.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems , 33:22243-22255, 2020.

Aritra Ghosh, Himanshu Kumar, and P Shanti Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI conference on artificial intelligence , volume 31, 2017.

Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems , 31, 2018.

Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 322-330, 2019.

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2818-2826, 2016.

Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.6596 , 2014.

Eric Arazo, Diego Ortego, Paul Albert, Noel O'Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In International conference on machine learning , pages 312-321. PMLR, 2019.

Jingkang Yang, Litong Feng, Weirong Chen, Xiaopeng Yan, Huabin Zheng, Ping Luo, and Wayne Zhang. Webly supervised image classification with self-contained confidence. In European Conference on Computer Vision , pages 779-795. Springer, 2020.

Pengxiang Wu, Songzhu Zheng, Mayank Goswami, Dimitris Metaxas, and Chao Chen. A topological filter for learning with label noise. Advances in neural information processing systems , 33:21382-21393, 2020.

Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. arXiv preprint arXiv:2002.07394 , 2020.

Filipe R Cordeiro, Ragav Sachdeva, Vasileios Belagiannis, Ian Reid, and Gustavo Carneiro. Longremix: Robust learning with high confidence samples in a noisy label environment. Pattern Recognition , 133:109013, 2023.

Ganlong Zhao, Guanbin Li, Yipeng Qin, Feng Liu, and Yizhou Yu. Centrality and consistency: two-stage clean samples identification for learning with instance-dependent noisy labels. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXV , pages 21-37. Springer, 2022.

David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in neural information processing systems , 32, 2019.

Evgenii Zheltonozhskii, Chaim Baskin, Avi Mendelson, Alex M Bronstein, and Or Litany. Contrast to divide: Self-supervised pre-training for learning with noisy labels. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1657-1667, 2022.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

Andrey Okhotin, Dmitry Molchanov, Vladimir Arkhipkin, Grigory Bartosh, Aibek Alanov, and Dmitry Vetrov. Star-shaped denoising diffusion probabilistic models. arXiv preprint arXiv:2302.05259 , 2023.

Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In International Conference on Machine Learning , pages 7164-7173. PMLR, 2019.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1944-1952, 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint arXiv:1708.02862 , 2017.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115:211-252, 2015.

Kuang-Huei Lee, Xiaodong He, Lei Zhang, and Linjun Yang. Cleannet: Transfer learning for scalable image classifier training with label noise. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5447-5456, 2018.

Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2691-2699, 2015.

Jongwoo Ko, Sumyeong Ahn, and Se-Young Yun. Efficient utilization of pre-trained model for learning with noisy labels. In ICLR 2023 Workshop on Pitfalls of limited data and computation for Trustworthy ML , 2023.

Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11976-11986, 2022.

Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda. Earlylearning regularization prevents memorization of noisy labels. Advances in neural information processing systems , 33:20331-20342, 2020.

Nazmul Karim, Mamshad Nayeem Rizve, Nazanin Rahnavard, Ajmal Mian, and Mubarak Shah. Unicon: Combating label noise through uniform selection and contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9676-9686, 2022.

Deep Patel and PS Sastry. Adaptive sample selection for robust learning under label noise. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 3932-3942, 2023.

Jiangfan Han, Ping Luo, and Xiaogang Wang. Deep self-learning from noisy labels. In Proceedings of the IEEE/CVF international conference on computer vision , pages 5138-5147, 2019.

Yuanpeng Tu, Boshen Zhang, Yuxi Li, Liang Liu, Jian Li, Jiangning Zhang, Yabiao Wang, Chengjie Wang, and Cai Rong Zhao. Learning with noisy labels via self-supervised adversarial noisy masking. arXiv preprint arXiv:2302.06805 , 2023.

Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning , volume 4. Springer, 2006.

Shakir Mohamed and Balaji Lakshminarayanan. Learning in implicit generative models. arXiv preprint arXiv:1610.03483 , 2016.

Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13 , pages 446-461. Springer, 2014.

A Denoising Diffusion Implicit Model with non-zero mean latent space

The forward process of a diffusion process with non-zero mean latent distribution y T ∼ N ( f ( x ) , I )) has a closed form representation:

which could be reprameterized as:

Then, similar to the DDIM, we define a Non-Markovian forward process with σ t ≥ 0 , t = 1 : T .

Where q σ ( y T | y 0 , f ) = N ( y T ; √ ¯ α T y 0 +(1 - √ ¯ α T ) f ( x ) , (1 -¯ α T ) I ) and for all t > 1 .

We can prove that the sampling process defined by Eq. (A.3) and Eq. (A.4) has the same marginal distribution as the closed-form sampling process in Eq. (A.1) by the following Lemma:

Lemma A.1. For q σ ( y 1: T | y 0 , f ) defined in Eq. (A.3) and q σ ( y t -1 | y t , y 0 , f ) defined in Eq. (A.4), we have:

Proof. Assume for any t ≤ T , if Eq. (A.5) is true, the following is also true:

then we can prove the statement with an induction argument for t from T to 1 , since the base case ( t = T ) already holds.

First, we have that:

and

According to [66] Eq. (2.115), we have that q σ ( y t -1 | y t , y 0 , f ) is Gaussian, with mean µ t -1 and co-variance Σ t -1 :

Therefore, Eq. (A.6) holds. Following the induction, the lemma is proved.

In our implementation, we follow DDIM [32] setting σ t = 0 . The resulting model becomes an implicit probabilistic model [67], where the generation process become deterministic given y T .

B t-SNE visualization of the DDIM generation process

t=7

t=5

t=4

t=3

t=2

t=1

t=0

C Experimental setup and details

C.1 Real-world dataset details

WebVision comprising 2.4 million images that were crawled using Google and Flickr search engines, with the ILSVRC12 taxonomy. Following prior studies, we trained our model on the initial 50 classes from the Google image subset of Webvision and tested it on the validation sets of both Webvision and ILSVRC12.

Food-101N consists of 310k food images collected from the internet with the Food-101 [68] taxonomy, and has an estimated label noise level of 20%, making it an ideal dataset to evaluate the robustness of our method under real-world noisy labels. We assessed the classification accuracy on the curated label set of Food-101, which contains around 25k images.

Clothing1M contains 1 million images of clothes obtained from shopping websites. Based on the keywords in the surrounding text, the images are automatically classified into 14 classes with ∼ 40% estimated noise level. The dataset includes a clean training set, validation set, and test set with manually refined labels, consisting of approximately 47.6k, 14.3k, and 10k pictures, respectively. We discarded the clean training set and only used the noisy label data for training.

C.2 Implementation details

To present the hyperparameter settings of our neural network, we first give a description of our neural network design. As shown in Figure C.1, the network consists of a frozen f p encoder, a ResNet encoder, and a series of feed forward layers. Features encoded by the two encoders are combined with time embedding via hadamard product and passed through a series of feed-forward networks, batch normalization, and softplus activation to predict the noise term ϵ θ .

In our experiments, we use ResNet34 for CIFAR10 and CIFAR100, and use ResNet50 for real-world datasets as the trainable encoder (blue ResNet block in Figure C.1). The dimensions of all feed-forward layers are set to 512 for CIFAR datasets and 1024 for real-world datasets, respectively. We train LRA-diffusion models for 200

<latexi sh

1_b

64="oTz2

S

DI

NmVkZPv

c

0

w

>A

B8X

L

gM

F

7W+q

d

y

r

Rj

K9

H

U

Jp

3

C

G

Y

/

f

n

u

O

Q

5

E

<latexi sh

1_b

64="un59K

qDR

M+/

L

WmN

A

Uo

>

B7X

c

V

EOy

r

f

Y

P

2

Q

8

I

HJ

Z

p

wdFv

3

k

G

j

T

0

C

z

<latexi sh

1_b

64="oDm/PwJ

yr05

Tf

GLFqM

I

>A

B83

c

V

S

N

2p

Zdu vg

7KC

YU

E

+Okz

X

RQ

W9

H

n

j

<latexi sh

1_b

64="

pRouzTOc3

XPy

NIJ

/2Z

Q

>A

C

H

VD

BE

j

GL

v

n

KM

Y

q

Um

9

0

W

7

+

k

r

d

5

w

F

8

<latexi sh

1_b

64="

5qN

OkKV2

PIyGW

j

H

SYgw

>A

B/X

c

ZDL

F

3

7z

o

m

M

u

pTJd

n

E

Q

v

0

8

+

C

U

R

f

9

r

<latexi sh

1_b

64="AgBME

TcS

O38u z/J

0

>

n

VD

N

25R

Q

woy

HY

F

I

mp dPU

j

G

L

9q fW

7

Z

K

+

C

r

k

<latexi sh

1_b

64="oTz2

S

DI

NmVkZPv

c

0

w

>A

B8X

L

gM

F

7W+q

d

y

r

Rj

K9

H

U

Jp

3

C

G

Y

/

f

n

u

O

Q

5

E

<latexi sh

1_b

64="F

0HfU

ukWR+G

w

gI

>A

CBX

c

V

5T

N

E

z

n

ZK

Y

o

q

Q

J

p

O

/

D

m9

M

j

L

y

2

r

S

37

d

8

<latexi sh

1_b

64="z

HB9G

kK0d2

5V

MRF

O

g

n

o

>A

/

c

DJS

N

E

jX

L

u

Q

w

y

Y

C8

m

p

U

+T

P

v

rq

3

7

IW

f

Z

<latexi sh

1_b

64="PpC

BU3RfHk

9u0

Dn

+

5LzSI

>A

c

V

N

8

E

2q

v

yWg

T

Oo

F

wKj

d

m

Y

J

/

G7

M

Z

X

Q

r

<latexi sh

1_b

64="un59K

qDR

M+/

L

WmN

A

Uo

>

B7X

c

V

EOy

r

f

Y

P

2

Q

8

I

HJ

Z

p

wdFv

3

k

G

j

T

0

C

z

<latexi sh

1_b

64="un59K

qDR

M+/

L

WmN

A

Uo

>

B7X

c

V

EOy

r

f

Y

P

2

Q

8

I

HJ

Z

p

wdFv

3

k

G

j

T

0

C

z

<latexi sh

1_b

64="un59K

qDR

M+/

L

WmN

A

Uo

>

B7X

c

V

EOy

r

f

Y

P

2

Q

8

I

HJ

Z

p

wdFv

3

k

G

j

T

0

C

z

<latexi sh

1_b

64="F

0HfU

ukWR+G

w

gI

>A

CBX

c

V

5T

N

E

z

n

ZK

Y

o

q

Q

J

p

O

/

D

m9

M

j

L

y

2

r

S

37

d

8

<latexi sh

1_b

64="F

0HfU

ukWR+G

w

gI

>A

CBX

c

V

5T

N

E

z

n

ZK

Y

o

q

Q

J

p

O

/

D

m9

M

j

L

y

2

r

S

37

d

8

<latexi sh

1_b

64="un59K

qDR

M+/

L

WmN

A

Uo

>

B7X

c

V

EOy

r

f

Y

P

2

Q

8

I

HJ

Z

p

wdFv

3

k

G

j

T

0

C

z

<latexi sh

1_b

64="

7SV

z0QN/K

IX8

gf

GM

c

>A

C

n

DJ

BEO

25Rj

P

d

9

UT

yY

p

v

W

qu

+

3

H

o

r

F

Z

w

m

L

k

<latexi sh

1_b

64="z

HB9G

kK0d2

5V

MRF

O

g

n

o

>A

/

c

DJS

N

E

jX

L

u

Q

w

y

Y

C8

m

p

U

+T

P

v

rq

3

7

IW

f

Z

<latexi sh

1_b

64="z

HB9G

kK0d2

5V

MRF

O

g

n

o

>A

/

c

DJS

N

E

jX

L

u

Q

w

y

Y

C8

m

p

U

+T

P

v

rq

3

7

IW

f

Z

<latexi sh

1_b

64="z

HB9G

kK0d2

5V

MRF

O

g

n

o

>A

/

c

DJS

N

E

jX

L

u

Q

w

y

Y

C8

m

p

U

+T

P

v

rq

3

7

IW

f

Z

<latexi sh

1_b

64="SYARK

9

GV

u3y qrv

Q

+

Fo

>

B

X

c

DL

gN

EJ

f

U

8

5

k

n

TZ

Ww

2m

/

I

7

j

C

P

d

p

M

H

0

z

O

<latexi sh

1_b

64="PWgyQ

R

9BNdF

r

+

MfK

J

>A

X

c

V

S8

E

3

O

Y

UT

Z

j

k

2

z

u

m

op

/

H

D

0

7

5

L

w

I

n

C

q

G

v

<latexi sh

1_b

64="YC2k3

IKzN

FV

DU

>A

X

c

L

J

vqB

R

n

wT

g

y

d

9

Z

+

/

m

uH

5

7

G

p

j

O

M

0

P

<latexi sh

1_b

64="X

Yw

v

E

5KCMAj fL

Vkc7Bzo

>

9H

NS8

3Ur q/

I

g

T

Z

R

Q

m0

n

O

2

p

P

D

F

W

d

u

J

G

+

y

<latexi sh

1_b

64="Y

3UnQd

ZDF80f

C

IN

H2X

M

>A

c

V

LS

J

rP

m

u

W

K

g

9

G

O

q

T

5

j

/ok7

v

w

E

R

y

p

z

/circledot

/circledot

/circledot

/circledot

/epsilon1

θ

(

y

t

,

x

, f

p

(

x

)

, t

)

Embedding

f

p

(

x

)

Image

x

Feed Forward

Feed Forward

Feed Forward

Feed Forward

BatchNorm

Time Embedding

y

t

ResNet

Encoder

BatchNorm & Softplus

BatchNorm & Softplus

BatchNorm & Softplus

⊕

/circledot

Hadamard product

Concatenation

⊕

y

t

-

1

epochs with Adam optimizer. The batch size is 256. We used a learning rate schedule that included a warmup phase followed by a half-cycle cosine decay. The initial learning rate is set to 0.001. Following [15], we applied data augmentation in the training, including resizing, random horizontal flip, and random cropping. To retrieve the nearest neighbors, we set k=10 based on our tests using a range of k values from 1 to 100 on the validation sets. The KNN accuracy remained relatively stable for k between 10 and 50, and then starts to decline due to reduced label consistency among neighbors. Based on these results, we infer that our LRA diffusion model is less sensitive to variations in k within this range. All experiments are conducted using four NVIDIA Titan V GPUs.

D Additional ablation study

D.1 Classifier feature conditioning for accuracy enhancement

To demonstrate how our method can enhance a trained classifier's performance by using its features as conditional information, we conduct an ablation study examining the impact of conditional diffusion and KNN on the trained classifier. Specifically, we train classifiers, denoted as η ( x ) , using the standard method at various noise levels. We then remove the classification head and utilize the remaining model f η as the f p encoders in our conditional diffusion models.

The experimental results shown in Figure D.1 indicate that these techniques can improve test accuracy. We observe that when the noise level is below 55% , the conditional diffusion model (green) achieves a ∼ 1% improvement over the standard method. Moreover, when the LRA method is applied concurrently (purple), test accuracy can be further enhanced. This improvement occurs because learning from neighbors' labels reduces the noise level during training, as evidenced by the comparison between KNN results (blue) and clean label percentage (gray).

However, when the noise level exceeds 55% , the use of diffusion and LRA-diffusion methods does not seem advantageous. This limitation arises because the distribution of labels in the neighborhood becomes too corrupted for KNN to effectively improve the proportion of clean labels during training, as illustrated by the intersection of the blue and gray curves in the figure. We argue this does not diminish the practical value of our method because a dataset with more than 50% label noise is not meaningful in practice.

D.2 Effects of pseudo-label construction strategies

We also conduct comparative experiments using another method that utilizes neighbor labels: replacing the one-hot label vector with the mean vector of the neighbor's labels as the prediction target, which we call Mean-diffusion. We found that it can achieve higher accuracy when the noise level is higher than 55% . This may be due to the increase in the diversity of neighbor labels. The sampling-based LRA-diffusion will need

35%

40%

45%

50%

55%

60%

65%

70%

Noise levels

30

40

50

60

70

80

90

Test accuracy % on CIFAR10

Standard

Diffusion

KNN

Mean-diffusion

LRA-diffusion

SimCLR KNN

SimCLR LRA-diffusion

Clean label percentage

to learn a more complex multi-modal distribution, but Mean-diffusion only needs to learn a point estimate. However, when the noise level is lower than 55% , we found that LRA-diffusion is slightly more accurate than Mean-diffusion. A possible explanation is that the distribution of y 0 in LRA-diffusion contains only n one-hot labels. In contrast, y 0 in Mean-diffusion is more diverse ( n k /k ! possible mean vectors for n classes and k neighbors). In conclusion, LRA-diffusion has higher performance with less noisy labels. On the other hand, Mean-diffusion has faster and more stable convergence and is more robust for high noise level. However, they tend to perform similarly when the noise level is too high or too low since neighbors' labels will become the same or too corrupted.

D.3 Robustness of SimCLR feature conditioning

Finally, we use the SimCLR model as the encoder f p in our conditional diffusion model (listed as SimCLR LRA-diffusion in Figure D.1), to showcase the effectiveness of our proposed LRA-diffusion method in utilizing prior knowledge from pre-trained image representations to enhance the test accuracy and robustness. The experimental results (red) show that its test accuracy significantly surpasses other settings until the noise level reaches 65% . Beyond this point, the labels in the neighborhood become too corrupted to provide additional supervision information.

D.4 Effects of CLIP feature conditioning strategies

We carried out an ablation study employing various strategies to integrate the CLIP model for classification on real-world datasets. The results demonstrate that the superior performance of our diffusion-based method does not simply reflect the strength of the CLIP features. We show results of linear probing and KNN using CLIP features in Table D.1 to further demonstrate the effectiveness of our algorithm design. The results suggested that

utilizing linear probing with LRA and mean labels degrades the pre-trained feature space, leading to diminished performance in comparison to the unsupervised KNN approach. On the other hand, the diffusion process can effectively incorporate the CLIP feature space to achieve higher performance.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Label-Retrieval-Augmented (LRA) diffusion model for learning from noisy labels. The model leverages the neighbor consistency principle and incorporates pre-trained models to improve performance. The paper introduces the label-retrieval-augmented component, an accelerated label diffusion process, and a new conditional mechanism for incorporating pre-trained models. The proposed model achieves good performance both on synthetic and real-world benchmark datasets and the experimental results show that it can boost accuracy by 10-20 absolute points in many cases.\n\nSTRENGTHS:\n1.The idea of using the diffusion model to address the noisy label learning problem is interesting.\n\n2.Extensive experimental results demonstrate the proposed method can achieve the state-of-the-art method compared with multiple peer methods.\n\nWEAKNESSES:\nThe experiment part needs to be improved: add state-of-the-art method, discuss the influence of the label noise type, and the influence of different pre-trained models.\n\nQUESTIONS:\n1. The paper mentions that the proposed model is flexible and general, but it does not provide a detailed discussion of the types of label noise that the model can effectively handle. For example, feature-dependent label noise. Since the comparison method [15] is a baseline that focuses on feature-dependent label noise.\n\n2. It seems the pre-trained model is important for the proposed method. It would be better to discuss the effect of different backbone networks as well as pre-trained model on model performance.\n\n3. The description of the comparison algorithms is lacking in the paper.\n\n4. In Table 1, the comparison methods are outdated. It would be better to add some state-of-art methods published in 2022 or 2023.\n\nLIMITATIONS:\nPlease refer to the weakness and limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on learning from noisy labels using diffusion models. They reformulate the noisy label problem from a generative perspective. Specifically, the clean label is our target variable ($y_0$) and the noisy label is the diffused variable ($y_T$), which they call Label-Retrieval-Augmented (LRA) diffusion model. Actually, we don't know the clean label, they extract the pseudo clean labels using the neighborhood consistency.\n\nSTRENGTHS:\n* The paper is well-written.\n* The proposed method is a well-extended approach that builds upon the utilization of diffusion models in classification tasks and applies it effectively to tackle the noisy label problem.\n\nWEAKNESSES:\n* Motivation of LRA diffusion model\n  * It is important to explain why the noisy label process should be formulated as a diffusion process and to discuss in detail the advantages it offers.\n  * Since I think it is a novel approach, it is crucial to provide detailed explanations of the motivation behind it.\n  * When using the neighbor consistency principle, it is important to address whether the same problems mentioned by the authors in lines 44-51 would arise. If not, it is necessary to explain how the proposed model overcomes these challenges.\n\n* Comparison with CARD\n  * It appears that the original CARD paper introduced two pre-trained encoders, while the current paper seems to have used only one. This distinction needs to be clarified, and the roles of the two encoders should be explained separately in the description.\n  * Equation 4 seems to be adapted from the formulation presented in CARD, and proper citation should be provided.\n\n* Experiments\n  * The utilization of CLIP in the performance evaluation raises concerns because it involves the incorporation of additional data. It is important to conduct a fair comparison by addressing this aspect.\n  * An explanation is needed to clarify why the diffusion model is not a computational bottleneck. It is hypothesized that this may be due to the smaller dimensions, but this should be verified.\n\n* Minor comments\n  * The appearance of $\\tilde{y}$ in line 136 without prior definition should be addressed.\n  * \"Require: Input:\" in Algorithm 1 is typo.\n  * The inconsistent use of $\\mu_{\\theta}$ and $\\epsilon_{\\theta}$ should be corrected.\n  * The subscript in line 153 needs to be written correctly.\n\nQUESTIONS:\nPlease see Weaknesses part.\n\nLIMITATIONS:\nThey discussed in Limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes the application of denoising diffusion probabilistic models for modeling the true class probability distribution when training with noisy labels. In particular, the paper extends Classification and Regression Diffusion Models for this problem and uses pretrained self-supervised representation models for Label Retrieval Augmentation. At test time, they approximate the true label by starting with the reverse diffusion at the mean of the final gaussian distribution. The empirical results on CIFAR10, CIFAR100 and other real-world noisy datasets show improvements over considered baselines.\n\nSTRENGTHS:\nThe main strength of this work is that it demonstrates successful application of diffusion models for learning the probability distribution of labels in noisy label settings. The paper considers a variety of datasets and includes the relevant ablation studies and inference times.\n\nWEAKNESSES:\nI believe the key weaknesses are:\n1) The requirement of a pre-trained self-supervised image embedding network. \n2) The diffusion model does not sample discrete labels and does not even model the categorical distribution. \n3) I think the evaluation should also consider the following baseline: train a classifier network exactly as in Algorithm 1 except in step 5 where the network is trained with an MSE loss to predict one-hot vector obtained in step 4. More specifically, it would be good to check the performance when the classifier network can take in the image and the simclr/clip features and have a linear prediction head. This baseline would be more closer to the LRA-diffusion model than the other baselines considered in Table 2 in terms of the model input and output.\n\nQUESTIONS:\n1. In line 179, you write that you simply set $f_q({\\bf x}) = {\\bf 0}$ whereas you write in line 151 that the DDIM generation begins with a non-zero centered gaussian. So, the generalized DDIM is essentially used for the CARD+LRA Diffusion experiments in Table 2? And, you have a zero-centered gaussian distribution for the LRA-diffusion that you actually implement? \n2. Do you have any intuition about the performance of the baseline proposed in Weakness (3)?\n3. Do the baselines considered in Tables (3), (4) and (5) have access to extra training data? \n4. Will you release all the trained model checkpoints used for reporting the results in tables?\n\nLIMITATIONS:\nYes, the limitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes using a diffusion model to perform classification on a noisy-label dataset. The authors utilize diffusion to learn how to conditionally generate labels given an image. By combining recent work on classification diffusion models with label retrieval augmentations they demonstrate state-of-the-art results on real-world noisy datasets.\n\nSTRENGTHS:\n- The method provides a simple, yet seemingly effective way of performing classification on noisy data using generative models. The idea of using diffusion models for purpose this is novel and has the potential to significantly contribute to the \"Learning With Label Noise\" literature.\n- Some of the real-world dataset results showcased outperform the previous state-of-the-art by a significant margin.\n\nWEAKNESSES:\n- The novelty of the paper is limited. The authors combine the existing classification diffusion model with the retrieval augmentation technique to capture the noisy label distribution. The methodologies for both are inherited almost unchanged from their respective papers and there is no significant extension to either. Extending DDIM to a non-zero mean latent distribution is a minor contribution.\n\n- In lines 127 and 166, the authors claim that initializing with the predicted label mean and running the DDIM sampler approximates the maximum likelihood estimation of the labels. This is not justified anywhere, and there is no evidence to support the claim.\n\n- The experiments on the WebVision and ILSVRC2012 datasets should be examined further. The method presented in this paper relies heavily on having a high-quality pre-trained encoder network that can provide meaningful representations during training. For these two experiments, the authors utilized CLIP, whereas the next best-performing approaches, C2D and NCR, were only trained using samples from the smaller datasets. It is possible that the majority of the performance increases shown in this paper can be attributed to the discriminative ability of the CLIP encoder, since additionally on the Clothing1M dataset, where CLIP is not expected to perform well, the authors did not manage to greatly outperform the previous state-of-the-art results.\n\nQUESTIONS:\n- For a single sample, is the frequency of the neighbors seen in training the same regardless of their distances in the latent space? If so, how is a neighborhood defined?\n\nLIMITATIONS:\nThe limitations are addressed. Broader societal impacts are not applicable.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers image classification with noisy labels. Rather than consider a single label for each image, it considers the label distribution within the set of neighboring examples (determined using a pre-trained feature extractor), modelling this distribution by a learned diffusion process that is conditioned on image features, similar to CARD. The diffusion process maps a random label in the neighboring set to a normal distribution whose mean is either zero or the prediction of a baseline classifier, such that the reverse process should sample labels from the neighboring set. Inference is performed using DDIM to find the maximum likelihood label. The method assumes a pre-trained feature extractor (SimCLR trained on training data or CLIP trained on external data) is available for the purpose of finding neighbors and conditioning the diffusion process. The method is compared to strong baselines on several synthetic and real-world datasets and often provides a large improvement. Ablative studies demonstrate the importance of each component.\n\nSTRENGTHS:\n1. The ablative study (Table 2; CIFAR10/100 with synthetic noise) demonstrates that the combination of CLIP/SimCLR features and LRA-diffusion is key to the effectiveness of the method: linear probing, LRA (label retrieval) and CARD (diffusion-based classification) alone do not achieve such an improvement.\n1. Idea is well motivated.\n1. Good coverage of related work.\n1. Well written.\n1. Synthetic experiments use strong type of noise based on second-highest confidence.\n1. Inference time included in empirical results.\n\nWEAKNESSES:\n1. Real-world evaluation (WebVision, ILSVRC, Food-101N) does not include simple baselines using CLIP features (KNN classifier and linear probe). This is important because it lets us verify that the results do not simply reflect the strength of the CLIP features.\n1. Real-world evaluation does not include SimCLR features. This is important because it shows the performance of the method without external data. In general, it would be good to highlight which methods use (which) external datasets and/or pre-trained weights in the SOTA comparison.\n1. (Sections 3.3 and 3.4) The modification of the DDIM procedure to allow non-zero mean seems to be redundant, since all experiments then used a zero-mean distribution by setting $f_q = 0$ (line 179)?\n1. Unclear what \"Linear probing + LRA\" is in Table 2. Does this entail training a linear model using the average of the neighboring labels as the softmax-cross-entropy target?\n1. It does seem inelegant to use a normal distribution to represent probability vectors (as acknowledged in text). Did the authors try working in real-valued logits instead?\n\nSuggestions:\n1. I'm not a big fan of naming the method \"retrieval\", since the term suggests to me that training data is retrieved from a larger, external source of training data (not the case).\n1. Discussion of impact of $k$ (number of neighbors) is deferred to the supp mat and I didn't see any references to this in the main text. It would be good to at least reference it and ideally add a plot.\n1. Could be good to include some discussion of singly-labelled vs multiply-labelled noisy labels, since the proposed method effectively constructs a multiply-labelled dataset from a singly-labelled dataset.\n\nQUESTIONS:\nPlease address weaknesses.\n\nLIMITATIONS:\nThe authors have identified limitations including:\n1. dependence on pre-trained feature extractors\n1. method is less effective beyond ~50% corrupted labels\n1. normal distribution not ideal for working on probability simplex\n\nI do not see any ethical issues.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}