Label-Noise Robust Diffusion Models
Authors: Byeonghu Na
Yeongmin Kim
HeeSun Bae
Jung Hyun Lee
Se Jung Kwon
Wanmo Kang
Il-chul Moon
Venue: ICLR 2024 poster (2024)
Keywords: diffusion model
noisy label
robustness
Primary area: generative models
TL;DR: We propose an approach to training diffusion models with noisy labels.

Abstract: Conditional diffusion models have shown remarkable performance in various generative tasks, but training them requires large-scale datasets that often contain noise in conditional inputs, a.k.a. noisy labels. This noise leads to condition mismatch and quality degradation of generated data. This paper proposes Transition-aware weighted Denoising Score Matching (TDSM) for training conditional diffusion models with noisy labels, which is the first study in the line of diffusion models. The TDSM objective contains a weighted sum of score networks, incorporating instance-wise and time-dependent label transition probabilities. We introduce a transition-aware weight estimator, which leverages a time-dependent noisy-label classifier distinctively customized to the diffusion process. Through experiments across various datasets and noisy label settings, TDSM improves the quality of generated samples aligned with given conditions. Furthermore, our method improves generation performance even on prevalent benchmark datasets, which implies the potential noisy labels and their risk of generative model learning. Finally, we show the improved performance of TDSM on top of conventional noisy label corrections, which empirically proving its contribution as a part of label-noise robust generative models. Our code is available at: https://github.com/byeonghu-na/tdsm.

Published as a conference paper at ICLR 2024
LABEL-NOISE ROBUST DIFFUSION MODELS
Byeonghu Na', Yeongmin Kim', HeeSun Bae', Jung Hyun Lee", Se Jung Kwon', Wanmo Kang' & Il-huul Moon KAIST, NAVER Cloud, 'summary.ai
(wp03052, alsdudrial, cat2507, wanmo kang, @kaist................ .astt.ac. acc. kr, (junghyun122 23. lee, sejung. kwon) ( @navercorp.
ABSTRACT
Conditional diffusion models have shown remarkable performance in various gen- erative tasks, but training them requires large-scale datasets that often contain noise in conditional inputs, a.k.a. noisy labels. This noise leads to condition mismatch and quality degradation of generated data. This paper proposes Transition-aware weighted Denoising Score Matching (TDSM) for training conditional diffusion models with noisy labels, which the first study in the line of diffusion models. The TDSM objective contains weighted sum of score networks, incorporating instance- wise and dependenn label transition probabilities. We introduce transition- aware weight estimator, which leverages time-dependen noisy-label classifier dis- tinctively customized to the diffusion process. Through experiments across various datasets and noisy label settings, TDSM improves the quality of generated samples aligned with given conditions. Furthermore, our method improves generation per- formance even on prevalent benchmark datasets, which implies the potential noisy labels and their risk of generative model learning. Finally, we show the improved performance of TDSM on top of conventional noisy label corrections, which em- pirically proving its contribution as a part of label-noise robust generative models. Our code is available at: /ettps://githbbe combb/ooooooo///ggggooococo na/tdsm
INTRODUCTION
Diffusion models have gained significant interest in various fields, such as image (Song et al. 2020 Dhariwal & Nichol 2021 Karras et al., 2022 Kim et al. 2023) and video generation (Ho al.
2022; Voleti et al.) 2022). for their high-quality sample generation. Moreover, they have shown
impressive results conditional generation problems (Kong et al., 202T Meng et al. 2022a b
Rombach et al. 2022 Saharia et al. (022). However, training diffusion models requires large-scale datasets, which often contain data instances with noisy labels due to expensive labeling costs and a lack of experts (Xiao al., 2015 Song et al. (022). These noisy labels cause problems when learning
conditional generative models, as shown in Figure Kancko et al. (2019) and Figure [b] Although
the problem of learning with noisy labels has been extensively studied in supervised learning (Patrini
eet al. 2017 Han et al. 2018 Li et al., 2202] Nordstrom et al. 2022 Bae et al.) (024), there are only
few studies on generative models TThkkumprammpil al. 2018 Kaneko et al. [2019). and as far as we know, there has been no theoretical discussion and remedies for diffusion models.
This paper proposes method for training conditional diffusion models with noisy labels. To achieve this, we show that the noisy-label conditional score can be interpreted as a linear combination of clean-label conditional scores based on instance-wise and time-dependent label transition information. Using this relationship, we propose an objective of Transition-awar weighted Denoising Score Matching (TDSM). To train a score network with the proposed objective, we introduce an estimator of the transition- -warre weights, and we explain the practical implementation that is not trivial because of multiple score network evaluations from class transitions. We conduct experiments on various datasets and label noise settings, and our diffusion models, which are trained by the TDSM objective, generate samples that match the given condition well (see Figure Ic), and our models outperform well most unconditional metrics. Furthermore,
baseline models conditional metrics empirically demonstrate that TDSM is a key component of label- noise robust diffusion models by showing improved performance when combined with conventional noisy label corrections.

---

Published as a conference paper at ICLR 2024
0
0
1
9
4
4
9
8
6 2
2
0
3
0
0
0 1
0
0
6 2 9
0 2 9
C 2
0
2
2
2
1
2
8
2
2
0
1
2
9
4
7 9
7 9
9
5
9
3
4
9
9
6
3
1
7
8
9
9
9
Auto.
Frog
Ship
(a) Dataset with noisy labels
(b) Baseline
(c) Ours
Figure 1: (a) Examples of noisy labeled datasets of MNIST (top) and CIFAR-10 10 (bottom), and (b-c)
the randomly generated images of baseline and our models, trained with the noisy labeled datasets.
In summary, our contributions are as follows:
We propose Transition-awarr weighted Denoising Score Matching for training conditional diffusion models with noisy labels, which is the first time in the diffusion model.
We prove that there is linear relationship between noisy-label and clean-labe conditional scores based on the instance- wise and dimeddepeneett label transition probability.
We suggest an estimator structure with time- deme-deeeddent noisy-label classifier and transition matrix to evaluate pairwise class transitions.
We empirically investigate the negative impact of noisy labeled datasets on diffusion models, and the proposed method achieves superior performance in the presence of noisy labels.
PROBLEM FORMULATION
This section formulates the problem of learning diffusion models from noisy labels. We define the data space as X' E TR' and the label space as ) = {1,..., c), where d is the data dimension and c the number of classes, A data instance is denoted by x E X, and its corresponding clean label is denoted by y E V. We assume that only noisy labeled training dataset D {[x(1, o()) )= available, sampled from noisy-label data distribution Peauu (X, Y), where y E y represents the noisy label corresponding to X, and is the number of data instances.
Label Noise We focus the class-conditioaal label noise setting, where the noisy label Y is
assumed to be independent of the instance X given the clean label Y (Natarajan et al., 2013
Patrini
et al. 2017). This setting is commonly used in the label noise literature (Kaneko et al. 2019 Li et al.
2021
Zhang et al. 20216). From a generative perspective, it can be expressed as follows:
p(x = ū) = Σ p(Y = u/ -ip(x = y. Ý = 0) Σ D( = v -)p(x( = v), (1)
where the second equation is derived by the assumption from the class-condition label noise setting. Eq. (I) means that each noisy-label conditional distribution is a mixture of clean-label conditional distributions. We define reverse transition matrix as S = [Stg] E [0, 1jxe where
Si, p(Y ý =i) .) By the definition, we have Ei=1S = 1. We provide detailed review of learning with noisy labels in Appendix B.3
Diffusion Model Now, we briefly introduce the diffusion model. We describe it through stochastic
differential equations. Specifically, Eqs. (2) and (3) represent the forward diffusion process and its
corresponding reverse process, respectively (Anderson 1982 Song et al. 2020 Kim et al. 2022a).
dx; f(x,, t) + gt)
dx;- = [f(x,,) ()
- 1000-g2210) (1) logp( + g(()dw:,
where f is the drift function; y is the volatility function; Ps (xx) is the probability density of xx; and w, denote the forward and reverse standard Brownian motion, respectively; and t [0, 7). The
We called S the reverse transition matrix because in most of the literature on learning with noisy labels, the transition matrix usually contains the information of p o y). We omit reverse if it is not confusing.
2

---

Published as a conference paper at ICLR 2024
forward process gradually perturbs the data instance Xo to xy, and the reverse process gradually denoises from Xy to Xg. We can sample data instances through the reverse process.
In spite that the generation task will require the reverse diffusion process, this process is intractable because a data score Vx, log Pr. (x) is generally not accessible. Therefore, the diffusion model aims train the score network to approximate Xx, log Pa (xx) through the score matching objective function with L.2 Ioss (Song et al. 2020). Specifically, the conditional diffusion model trains a score
network Sg. where 0 is trainable parameters, estimate the gradient of conditional log-likelihood.
Vx, log Pt (x,|Y) (Chao et al., 2022). Denoising score matching (DSM) (Vincent 2011
Song et al.
2020 is commonly used to train a score network:
CDSM((; Panta ())
(4)
=Ee (x) (X.) (X.))))) o(X, (x., yo Vx, log Peoo(x xx, Y w)|
where JAanta is clean-label data distribution, Pejo X.|X,, x, Y y) is perturbation kernel, / is sampled over [0, 2". and X(t) is temporal weight function The perturbation kernel Pajo (X, |x, y) is generally Gaussian distributed and independent of the labeT, so can be computed in closed form. In this paper, we mainly focus on directly learning conditional score network, and discuss the application of guidance methods in Appendix C
More discussion is in Appendices B. and B.2
Label-Noise Robust Generation of Diffusion Models Our goal is, given training dataset with noisy labels, to train a conditional diffusion model that can generate samples conditioned on specified clean label. This task poses a significant challenge due to the presence of noisy labels. When we train a diffusion model using Eq. 44) with a noisy labeled training dataset, the model eventually learns the noisy-label conditional score, Xx. (xxx As a result, the generative distribution from the model follows the mixture of the conditional distributions (see Eq. []. leading to the generation of samples from different classes. Furthermore, this problem cannot be solved by directly applying methods from generative adversarial networks (GANs) (Kaneko et al. 2019) Thekumparampil et al. 2018), as we dealing with dimeddepeneent vector that has a probabilistic interpretation, namely the data score, and the optimization problem has a different structure.
LABEL-NOISE ROBUST DIFFUSION MODELS
This section presents our approach training diffusion models with noisy labels. We begin by formulating an objective function that leverages the linear relationship between the clean-label and noisy-label conditional scores. Our proposed objective function involves weighted sum of conditional score networks, where the weights are determined by time-dependent label transition information at the instance level. To estimate these weights, we introduce an estimator that utilizes transition matrix and a time- dependennn noisy-label classifier. Lastly, we present some useful techniques for practical implementations.
3.1 LINEAR RELATIONSHIP BETWEEN CLEAN- AND NOISY-LABEL CONDITIONAL SCORES
As discussed in Section 2 if the score network is optimized by the original DSM objective, Eq. 4. with noisy labeled dataset, assuming Y = Y; then the score network is expected to converge on the noisy-label conditional score. See Appendix A. for further discussion.
Remark. Let 0jSH == mrng LDSM (0: Pdlata [X, V)) be the optimal parameters obtained by minimizing the DSM objective. Then, Seyzy (X1, y, t) = Vx, log = for all x4, y,
Therefore, to train the score network in the alignment of the clean-label conditional score, we modify the objective function to adjust the gradient signal from the score matching. We start this adjustment by establishing the relationship between clean-label and noisy-label conditional scores.
Theorem 1. Under class conss-condiionall label noise setting, for all Xx, U, t,
Xx, log (x Y = ( =
E== w(x,, U. V, t) V x1 log Pa ( = v):
(5)
Px y) where w(x., = p(Y y px(x W)
We add the distribution Paat [X, Y) in the notation of L to emphasize the input distribution since we are considering the distribution change due to label noise.
3

---

Published as a conference paper at ICLR 2024
t-5
20
20
20
20
20
1.00
0-
0-
0
0.50
-20 -20
-70
-70
-20
0.00
Figure 2: Contour maps of w(x,, Y = 1,Y 1,t) () the 2-D Gaussian mixture model at different diffusion timesteps. The label transition probability is set to p(Yl = 1) = (0.8, 0.2). The dots represent samples from each clean label (orange for class I, green for class 2), and the dashed lines represent contours with annotated values.
The proof of Theorem is provided in Appendix A.2
Since w(x,, V, > 0 and Eyn1 w(x,, g. Theorem implies that the noisy-label conditional score can be expressed as convex combination of the clean-label conditional scores with coefficients y...)......
We call w(x,, U. t) the transition aware weight function. This function represents instance-wise and time-dependent (reverse) label transitions by Proposition See Appendix A.3 for the proof.
Proposition 2. Under class-conditional label noise setting, w t, Ø. y, () pc (Y y y. x2).
This probability has been used for evaluating a confidence score p(Y y y. in some supervised classification frameworks for instance-dependent label noise learning (Berthon et al. 202T) whereas this paper is its first application to the deep generative model community. There are similar approaches to overcome noisy labels in GANs under the glass-conditional label noise setting (Kaneko et al. 2019; Thekumparampil et al. 2018), but these approaches can simply utilize p(Y==) = y = 57) without either instance-based estimation or diffusion time-dependent estimation. However, training diffusion models with noisy labels poses a significant challenge because need destnce-dependent label noise information, even under the class conditiool label noise, and this distinction is the contribution of Theorem
To visualize our transition-aware weight function, we create an analogy of a 2-D Gaussian mixture model. The detailed setup is explained in Appendix E.4 Figure plots contour maps of w (xx, xx, y L = 1, with different diffusion timesteps. Note that Zy=1 w(x,, = Y = y, t) = 1.
In the small timesteps, w(x,, Y = 1, Y)t) tends to become hard labels with respect to the clean labels. This means that the noisy-label conditional score, computed by the weighted sum of the scores, becomes equal to the conditional score of the clean label of x,. On the other hand, in the large timesteps, w (x,, y = 1, Y, vectors for most x; are similar to the prior probability of label transition, pY = 1) = 0..). Consequently, the noisy-label conditional scores for most x influenced by the clean conditional scores of the other class; and this mis-matched score is being learned. This mis-matched learning leads to both improper generation with intended conditions and lower generation performance in its quality.
3.2
TRANSITION-WAARE WEIGHTED DENOISING SCORE MATCHING
From the result of Theorem I we propose an objective of transition-awan weighted denoising score matching (TDSM), which is designed minimize the distance between the transition-aware weighted sum of conditional score network outputs and the perturbed data score:
CIDSMa Psaaa (X, Y))
(6)
= =IE (()) grm E Expepeo
Σ y=1
x,, w( w(x,, G. W, 1),.),))( (), Vxi 1xg |
The intuition behind the TDSM objective is as follows. According to Remark the denoising score matching objective optimizes the noisy-label conditional score. However, Theorem this score can be expressed as a convex combination of the clean-label conditional scores using the transition-awanl weight function.. Therefore. by using the transition-awar weighted sum of the conditional score model outputs as the target of L2 loss, our score network will eventually converge the clean-label conditional score. This intuition is theoretically guaranteed by Theorem
[ shows that
It should be noted that Pe y|6 = xx) # PY vIV = 0)
4

---

Published as a conference paper at ICLR 2024
Let @TDSM := argming LrDss Paato (X,Y)) be the optimal parameters obtained by label noise setting with an invertible transition matrix, S0 (x,, y, Xx, log (x = y) for all xx, v,
Theorem 3.
minimizing the TDSM objective. Then, under a slass-condition
See Appendix A.4 for the proof. We discuss the other forms of the objective function, such noise estimation (Ho et al. 2020) and data reconstruction (Kingma et al. 202T). in Appendix A.6
It should be noted that the invertibility assumption of the transition matrix has been widely used in the noisy label community (Li et al. 2021: Zhu et al. 2021). Here, we assume that the given noisy label is sufficiently likely to be clean label, and without this assumption, there is no empirical evidence to claim a certain class to be a clean label. In other words, the transition matrix would be a diagonally dominant matrix, which is invertible. Additionally, possible approach to ensure the invertibility of the transition matrix is to mix the transition matrix with the identity matrix (Patrini et al.) 2017).
The alternative approach to consider is the S-weighted DSM objective, denoted LSDSM- In this objective, the weights in the TDSM objective are replaced by time- and instance-independeett weights Si.9 = p( = y y = 7)) from the transition matrix S, which is inspired by the GAN-based methods:
LSDSM(O: Paata (x, Y)) = E. ,,) ||| E| So v. Vx, (x, Y = 2) 10112) (7)
However, Proposition 4 proves that the score network trained by the S-weighted DSM objective cannot converge to a clean-label conditional score. The proof is provided in Appendix A.5
Let SDSM arg ming LO Paaso Panno (X, Y)) be the optimal parameters obtained by minimizing the S-weighted DSM objective. Then, under a class-conditional label noise setting
Proposition 4.
with invertible transition matrix, S0 (x,, V. t) differs from Vx, log px = =y)-
Figure 8 in Appendix
F.1 visualizes the optimal score networks of DSM variants in 2-D toy case.
3.3 ESTIMATION OF TRANSITION-AWARE WEIGHTS
To implement the TDSM objective, we need to estimate the transition-aware weight function wr(x, (x, Ú; y, t). First, we consider the case that have the transition matrix S, so we evalu- p(Y Y = Y y). which has been a common goal in the community of noisy labels (Li al. 2021: Zhang et al. 20216). Then, we can compute the transition- aware weight function w using the
noisy label prediction probability Pt (Y()))) (x,) and the transition matrix S. The below introduces the estimator of u using the noisy labeled dataset (see Appendix A.7 for detailed derivations):
SQ.mp(E=)) w(Xx, G. t, (()x)
S (Y=ilx) p(Y=()
(x U. 2)
Sg.pn (x(x)
Σ
5 (ha
(8)
where his (x,,)) t) is the time-dependent noisy-label classifier; & is the parameters of the classifier; and n, is the number of data instances for i-th class. We obtain the estimator of the noisy label prediction estimate the noisy label prior p(Y) using the
t). probability p. (x,, as h& t). Additionally, statistics of the dataset, i.e., ns/n. Note that the transition matrix S is fixed and independent of x,, so the inverse matrix operation is required only once during the entire training process.
In practice, the reverse transition matrix, S, may not be known. However, there exist methods in the learning with the noisy label community that can estimate the transition matrix T, where Tig ,( -|Y i). from the noisy labeled dataset (Li et al. 2021| Zhang et al. 2021b). Using the estimated transition matrix and the noisy label prior, we can obtain the reverse transition matrix by applying Bayes' theorem. (See Appendix A..)) ) We applied the transition matrix estimation method from VolMinNet (Li et al. 2021 to our dependent noisy-label classifier, hg. and we analyzed the effect of this estimation in Section 4.5
3.4 PRACTICAL IMPLEMENTATION
Training Procedure Figure 3 illustrates the overall training procedure with the TDSM objective. First, we prepare the úime-dependent noisy-label classifier and the transition matrix. The nabsy-label classifier h3- is obtained by training with combination of cross-entropy losses over different time steps (Song et al. [2200] Chao et al. 2022) on the noisy labeled dataset. We estimate the transition matrix S using prior knowledge or the existing methods discussed in Section 3.3 During the training
5

---

Published as a conference paper at ICLR 2024
Noiey labeled dottedd
Trunsition-awed weight evaluation
Pel Xz)
Transition natris
Noisy-label classiffer hg (R.
Transition-awwre weight fimetion
(L....e
Pajel
10 Xe, (.
Transition-wwar deseising score matching
K
E R
Cond. model no (Ne ()
CDSsa
So(Ner (.8)
Vxc. logge(Xe x)
The training procedure of the proposed approach. The solid black arrows indicate the forward propagation, and the dashed red arrows represent the gradient signal flow. The filled circle operation denotes the dot product operation, and the dashed operation represents the L2 loss. The noisy-label classifier hą- can be obtained by the cross-entropy loss the noisy labeled dataset D.
Figure 3:
iterations of the score network, we evaluate the transition aware weights using Eq. Next, we obtain the score network outputs for all classes. Finally, we compute the TDSM objective value using the weights and score outputs via Eq. (6). and optimize the model using the gradient descent method.
Reduction on Time and Memory Usage The TDSM objective involves network eval- uations for all classes, which can cause memory shortages during backpropagation. To mitigate this problem. backpropagation for the network output cor- responding to the given noisy label (lines 7-8 of Algorithm [ We choose this out-
Algorithm 1: Training algorithm with TDSM
Input: Noisy labeled dataset D, transition matrix S, noisy-label classifier h perturbation kernel
only perform
Pro: temporal weights X, skip threshold r
Output: Conditional score model 5o
while not converged do
Sample (x, ÿ) from D, and time from [0,
put corresponding the given label it is
Sample x; from the transition kernel pıjo
Evaluate w(y) ii( U. ys using Eq. 8
the dominant term of the transition-aware
weight function for most instances.
s = for E {1,.... do if then s wo v. clse w(xi, D. y. 1) then s + w(y)s v. 9. detacch)
Furthermore, we observed that the transition-awan weights w(x,, y, y, t) have negligible values for most (ū, y) pairs. Specifically, on average, only 1.3 elements of the weight vector w (x(x,, (.) have values greater than 0.01 our experiments on the CIFAR datasets. To reduce the training time of forward propagation, we skip network evaluations for instances, end where the corresponding transition-aware weight function value was below threshold r (line of Algorithm Tb. We set to 0.01 for all experiments, and Appendix F.9 provides the ablation study 7. We discuss the training time further in Appendix F.4 Algorithm specifies training procedure for our method.
11 end
end
13 1-- X(D Vxx Pr x. Ý M
14
Update by using the gradient descent method
EXPERIMENTS
We evaluate method on three benchmark datasets commonly used for both image generation and label noise learning: MNIST (LeCun et al., 2010), CIFAR-10, and CIFAR-100 (Krizhessky real-world noisy labeled dataset, Clothing 1M (Xiao et al. 2015). For the benchmark datasets, we create two types of label noise: symmetric and asymmetric noise (Kaneko et al. 2019). For symmetric noise, randomly flip the ground-truth label to another
2009). We also perform experiments

---

Published as a conference paper at ICLR 2024
Table Experimental results on the MNIST, CIFAR-IO, and CIFAR-100 datasets with various noise settings. The percentages in headers represent the noise rate. 'un' and 'cond' indicate whether metric unconditional or conditional. Bold numbers indicate better performance.
Symmetric
Asymmetric
Clean
Metric
20%
40%
20%
40%
0%
DSM
TDSM
DSM
TDSM
DSM
TDSM
DSM
TDSM
DSM
Density
(
81.11
84.83
81.93
84.55
84.23
85.27
84.47
84.71
86.20
cond un
MNIST
Coverage
(1)
81.23
82.16
81.65
81.31
82.30
82.45
81.97
82.27
82,90
CAS
(
94.31
98.22
72.52
96.49
95.25
98.22
89.29
96.54
98.55
CW-Deensty CW-Coverage
69.78
82.99
55.70
80.09
78.58
83.74
73.54
81.65
85.79
76.77
80.93
70.45
79.21
79.97
81.35
77.50
80.57
82.09
FID IS
(
2.00
2.06
2.07
2.43
2.02
1,95
2.23
2.06
1.92
(
9.91
9.97
9.83
9.96
10.06
10.04
10.09
10.02
10.03
un
CIFAR-IR 10
Density
100.03
106.13
100.94
111.63
100.66
104.15
101.25
105.19
103.08
Coverage
81.13
81.89
80.93
82.03
81.36
81.81
81.10
81.90
81.90
CW-FFD
(L)
16.21
12.16
30.45
15.92
11.97
10.89
15.18
12.54
10.23
cond
CAS
(
66,80
70.92
47.21
62.28
72.66
74.28
68.98
71.51
77.74
CW- Ow--Desity
(
88,45
99.52
73.02
97.80
96.10
101.77
92.13
99.21
102.63
CW-Coverage
(
77.80
80.29
71.63
78.65
79.95
80.99
78. 12
79.98
81,57
FID
(
2.96
4.26
3.36
6.85
2.76
2.64
2.73
2.81
2.51
IS
(T)
12.28
12.29
11.86
12.07
12.49
12.79
12.51
12.57
12.80
CIFAR- 100
Density
(T) 83.01
85.66
81.70
88.45
87.36
88.41
87.06
87.01
87.98
Coverage
(f)
75.02
74.90
73.92
72.12
77.04
77.46
76.56
76.27
77.63
CW-FID
(L) 79,91
78.71
100.04
93.24
75.39
69.83
89.13
73.13
66.97
cond
CAS
(f) 25.49
28.54
15.41
21.17
33.31
37.33
23.50
34.47
39.50
CW-Density
(7) 66.47
70.62
60.60
72.14
78.92
60.27
74.30
82.58
CW-Coverage
(T) 70.11
70.77
60.64
63.89
71.08
74.01
64.19
71.48
75.78
class, while for asymmetric noise, we flip the ground-truth label to predefined similar class. We refer to flipping probability as noise rate. Throughout the experiments, we mainly use EDM (Karras) et al. 2022) as the backbone of the diffusion models. We provide the results of other backbones in
Appendix F.6 Additional experimental settings are provided Appendices E. and E.2
We evaluate conditional generative models from multiple perspectives using four unconditional m including Fréchet Inception Distance (FID) (Heusel et al. 2017). Inception Score (IS) (Sal imans al.] 2016). Density, and Coverage (Naeem al.) 2020). and four conditional metrics, namely CW-FID, CW-Density, CW-Coverage (Chao et al. (022)). and Classification Accuracy Score (CAS) (Ravuri & Vinyals 2019). The Class-Wist (CW) metric evaluates the metric separately for
each class and averages them. A detailed description of the metrics is in Appendix E.3
4.1 ANALYSIS ON BENCIIMARK DATASET WITH SYNTHETIC LABEL NOISE
Table I presents the performance the baseline models trained with the original DSM objective and our models trained with the TDSM objective the benchmark datasets with various noise settings. First, the performance of the baseline models decreases on both unconditional and conditional metrics when trained on noisy datasets compared the clean datasets. This indicates that the label noise in the diffusion model training degrades the sample quality and causes a class mismatch problem.
Second, our models outperform the baseline models in all cases with respect the conditional metrics. As the noise increases, the performance differences become more significant. We compare the CW metrics evaluated separately for each class in Appendix [.7] which shows that our models perform better than the baselines in most classes. Furthermore, for the unconditional metrics, our models beat the baseline models most cases. Additionally, Figure 4 provides conditionally generated images from fixed xy for each model. The images generated by our model have better quality with an accurate class representation of the intended class than those generated by the bascline model.
We also compare the generation performances with the label-noise robust GAN model (Kaneko et al. 2019) in Table of Appendix [.5] The diffusion models generate images much better than the GAN models even in the noisy label dataset, and our models boost the performances by increasing the robustness to label noise.
*The unconditional metric calculated from the generated images only, accounting their labels.
7

---

Published as a conference paper at ICLR 2024
Arghac Auto.
Bird
Cat
Deer
Dog
Frog
Horse
Ship
Truck
Akghac Auto.
Bird
Cat
Deer
Dog
Frog
Horse
Ship
Truck
DSM
(b) TDSM (ours)
Figure 4: Generated images from (a) baseline and (b) our models, trained on the CIFAR-10 10 datasets under 40% symmetric noise. Each row contains the samples generated by each class for a fixed Xy.
3 9
Table 2: Experimental results on the clean MNIST, CIFAR-10, 10, and CIFAR-100 dataset. The results of conditional metrics are in Table T2
5-3
4-9
4-7
3-5
5-
1 Φ
d
4 q
MNIST
CIFAR-I0
CIFAR-100
2-+1
2-9
6-0
9--7
Metric
DSM
TDSM
DSM
TDSM
DSM
TDSM
1 3
1
9
FID
1.91
2.51
2.67
9-7
7-2
10.03
10.10
12.80
12.85
Figure 5: Noisy labels
Density
86.20
88.08
103.08
104.35
87.98
90.04
of MNIST, captured by á.
Coverage
82.90
83.69
81.90
82.07
77.63
78.28
Marks below images denote 'label prediction'.
4.2 ANALYSIS ON BENCHMARK DATASET WITH ANNOTATED LABEL
We evaluate the performance of our model on the benchmark datasets with original annotated labels to explore potentially noisy or ambiguous labels in these datasets. We apply our proposed method using the transition matrix with 5% symmetric noise. Table shows that our label-noise robust models consistently outperform the baseline models. This indicates that existing benchmark datasets may suffer from noisy labels. This phenomenon
well-known in the noisy-label supervised tasks (Bae) et al. 2022). To further investigate this observation, we compute the transition aware weights Vi (x,,, V. y, te) 2 Pt, (Y = y|ا² y. xt,) ) at small time fe = 0.01 for each MNIST train image and label pair (x, 8). Then, we identify 15 images with the smallest values of u (Xt,, V. D, tz) and check the predicted label from ours, which is arg max, ú (xx,, ; y, f). Figure 5 shows that the benchmark dataset also contains examples with noisy or ambiguous labels, and this could be the reason why our models improve the performance even when trained on the clean labeled datasets.
4.3 ANALYSIS ON REAL- WORLD DATASET
We apply our model to the Clothing- IM dataset, which contains of I million clothing images for 14 classes, evaluate its performance under real-world label noise. This dataset is obtained by crawling images from websites, which introduces label noise, and its label accuracy is 61.54% (Xiao et al. 2015). We also have 25K images containing both noisy and clean labels. We followed the experimen- tal setup from Kaneko et al. (211)) Specifically, we estimate the transition matrix using the statistics of the 25K clean training dataset, and we resize the images to a resolution of 64x64. Table shows that our model improves both FID
Table 3: Experimental results on the Clothing- 1 M dataset.
Metric
DSM
TDSM
FID (L) CAS (t)
6.67
4.94
46.52
47.79
and CAS metrics compared to the baseline model. We did not use other metrics, such as CW-FID, due to the limited amount of clean labeled data for each class (Kaneko al.] 2019). We attach the generated images the Clothing- 1M dataset in Appendix F.16 These results indicate that our model performs well even with rerl-world label noise.
8

---

Published as a conference paper at ICLR 2024
Table 4: Experimental results of combining with the noisy label corrector, DISC (Li et al. 2023), on the CIFAR- 100 under 40% noise.
Table 5: Ablation study of weight functions on the
CIFAR-10 10 dataset under 40% symmetric noise.
Symmetric
Asymmetric
TDSM
Metric
Metric
DSM
S-DSM
DSM
TDSM
DSM
TDSM
Was
HD
(1) 2,54
2.84
4,00
3.41
FID
(1) 2.07
3.20
2.43
2.32
(t)
12.83
IS
9.92
9.96
9.97
Density
(7) 87.28
90.20
83.65
88.10
Density
(1)
100.94
118.85
111.63
110.67
Coverage
(t)
77.63
Coverage
(t)
82.03
CW-FII FID
(1) 67.52
67.33
78.93
76.62
CW-FID
(1) 30.45
16,26
15,92
15.83
cond
(t)
42.39
39.72
CAS
(t) 47.21
63.46
62.28
61.37
CW Density
(7)
82,04
85.44
76.04
81.69
CW WW-Density
(t) 73.02
107.24
97.80
97.59
CW-Coverage
(1)
75.61
71.62
CW Coverage
(1) 71.63
78.32
78.65
78.68
4.4 COMBINING WITH THE EXISTING NOISY LABEL CORRECTOR
The existing classifiers to mitigate the noisy label can be considered as finding the true label after noise filtering. By pipelining this noisy label corrector and our TDSM approach, noise filtering in terms of generation performance. From this perspective, our approach is orthogonal existing supervised learning methods for noisy labels, offering integration of both paradigms. To validate this premise, we conducted experiments by 1) obtaining the corrected labels from the existing classifier learning methods with noisy labels, VolMinNet (Li al.) 202T) and DISC (Li et al.) 2023). and 2) training the diffusion model with the corrected labels. The results are summarized in Table and Tables T0 and IT of of Appendix F.8 for the full of results. The experimental results indicate that applying our TDSM objective consistently improves the performance even with the corrected labels. Therefore, we believe that our approach tackles the noisy label problem from diffusion model learning perspective, providing an orthogonal direction compared to conventional noisy label methods.
find better
4.5 ABLATION STUDY OF TRANSITION-AWAR WEIGHTS
Estimating the Transition Matrix S So far, we have assumed that the transition matrix S is given. To demonstrate the practical usefulness our model, we combine our model with an existing method of classifier learning with noisy label, VolMinNet (Li al. 2021), estimate the nobsy-label classifier and the transition matrix, simultaneously. (See Appendix E.5 for a detailed procedure.) Subsequently, we obtain the estimated transition matrix Š and we evaluate the weight function from S, denoted as wg. As shown in the W's column of Table we observe that the model outperforms the baseline model, even when utilizing the learned transition matrix.
Instance- and Time-Dependent Trawsition-Aware Weights To investigate the effect of instance- and demendennndent transition aware weights, we train model with the S-weighted DSM objective, where the weights contain instance- and time-independent label transition information. As shown in S-DSM column of Table 5 we observe that the model is still robust to noisy labels compared to the baseline model as it still reflects the label transition information. However, the model performs slightly worse on some metrics than the one trained with instance- and time-dependent weight. Interestingly, the S-weighted model performs well density-related metries. However, since density metrics are insensitive to mode dropping (Naeem et al., 2020). it is possible that the model ignores some areas that relatively more affected by label noise due to eestanceiindeeendee this case, the performance of other metrics will suffer, but the density metrics will be unaffected.
weights. In
5 CONCLUSION
This paper addresses the problem of noisy labels in conditional diffusion models and proposes a new objective called Transition-awarr weighted Denoising Score Matching. This objective utilizes transition-awe aware weight function that leverages an instance-wise and time-dependent label transition information specifically tailored to the diffusion model. Experimental results various datasets and noisy label settings demonstrate that the proposed methods outperform baseline models both conditional and unconditional performance.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the National Research Foundation Korea (NRF) grant funded by the Korea government (MSIT) (NRF-2019R A 1028324).
REFERENCES
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-326, 1982.
HeeSun Bae, Seungjae Shin, Byeonghu Na, JoonHo Jang, Kyungwoo Song, and Il-Chul Moon. From noisy prediction true label: Noisy prediction calibration via generative model. In International Conference on Machine Learning, pp. 1277-1297. PMLR, 2022.
HeeSun Bae, Seungjae Shin, Byeonghu Na, and II chul Moon. Dirichlet-based per-sample weighting by transition matrix for noisy label learning. In The Twelfth International Conference on Learning Representations, 2024.
Antonin Berthon, Bo Han, Gang Niu, Tongliang I Liu, and Masashi Sugiyama. Confidence scores make instance-depent -eeeedee label-noise learning possible. In Marina Meila and Tong Zhang (eds.), Pro- ceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 825-836. PMLR, 18-24 Jul 2021.
Chen-Hao Chao, Wei-Fang Sun, Bo- Wun Cheng, Yi-Chen Lo, Chia-Che Chang, Yu-Lun Liu, Yu-Lin Chang, Chia-Ping Chen, and Chun-Yi Yi Lee. Denoising likelihood score matching for conditional score-based data generation. In International Conference on Learning Representations, 2022.
De Cheng, Yixiong Ning, Nannan Wang, Xinbo Gao, Heng Yang. Yuxuan Du, Bo Han, and Tongliang Liu. Class-dependent label-noise learning with cycle-consistency regularization, Advances in Neural Information Processing Systems, 35: 19111044 101001111166 16, 2022.
Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance- dependent label noise: sample sieve approach. In 9th International Conference on Learning Reprexentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffuuson guidance. In NewrIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Jonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan, Mohammad Norouzi, and David J. Flect. Video diffusion models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.). Advances in Neural Information Processing Systems, 2022.
10

---

Published as a conference paper at ICLR 2024
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentomet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In International Conference Machine Learning, pp. 2304 2304-2313. PMLR, 2018.
Takuhiro Kaneko, Yoshitaka Ushiku, and Tatsuya Harada. Label-noise robust generative adver- sarial networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2467-2476, 2019.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Dongjun Kim, Byconghu Na, Se Jung Kwon, Dongsoo Lee, Wanmo Kang, and Il-chul Moon. Maximum likelihood training of implicit nonlinear diffusion model. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), ) Advances in Neural Information Processing Systems, volume 35, pp. 32270-322222 -22844 Curran Associates, Inc., 2022a.
Dongjun Kim, Yeongmin Kim, Se Jung Kwon, Wanmo Kang, and Il-Chul Moon. Refining generative process with discriminator guidance in score-based diffusion models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Pro- ceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 16567-16598, PMLR, 23-29 Jul 2023.
Yeongmin Kim, Dongjun Kim, HyeonMin Lee, and Il chul Moon. Unsupervised controllable generation with score-based diffusion models: Disentangled latent code guidance. In NeurIPS 2022 Workshop on Score-Based Methods, 2022b.
Yeongmin Kim, Byeonghu Na, JoonHo Jang, Minsang Park, Dongjun Kim, Wanmo Kang, and Il chul Moon. Training unbiased diffusion models from biased dataset. In The Twelfth International Conference on Learning Representations, 2024.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models, Advances in neural information processing systems, 34:21696-21707,, 2021.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021.
A Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: htlp://yann.. www.l..ccooooc.ot...... www.n.ccom//xd//miis.... 2, 2010.
Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label-noise learning without anchor points. In International Conference on Machine Learning, pp. 6403-6413. PMLR, 2021.
Yifan Li, Hu Han, Shiguang Shan, and Xilin Chen. Disc: Learning from noisy labels via dynamic instance-specific selection and correction. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24070-24079, 2023.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on paltern analysis and machine intelligence, 38(3)447-461, 2015.
Chenlin Meng. Ruiqi Gao, Diederik P Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In NeurlPS 2022 Workshop on Score-Based Methods, 2022a.
11

---

Published as a conference paper at ICLR 2024
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference Learning Representations, 2022b.
Takeru Miyato and Masanori Koyama. cgans with projection discriminator. In International Conference on Learning Representations, 2018.
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metries for generative models. In International Conference on Machine Learning, pp. 7176-7185, PMLR, 2020.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances in neural information processing systems, 26, 2013.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text- text-guided diffusion models. In International Conference on Machine Learning, pp. 16784-16804. PMLR, 2022.
Marcus Nordstrom, Henrik Hult, Fredrik Löfman, and Jonas Söderberg. On image segmentation with noisy labels: Characterization and volume properties of the optimal solutions to accuracy and dice. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 34321-34333. Curran Associates, Inc., 2022.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1944-1952. 2017.
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Dif- fusion autoencoders: Toward meaningful and decodable representation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10619-10629, 2022.
Yiming Qin, Huangjie Zheng, Jiangchao Yao, Mingyuan Zhou, and Ya Zhang. Class-balancing diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18434-18443, 2023.
Harish Ramaswamy, Clayton Scott, and Ambuj Tewari. Mixture proportion estimation via kernel embeddings of distributions. In International conference on machine learning, pp. 2062-2060.
PMLR, 2016.
Harsh Rangwani, Konda Reddy Mopuri, and R Venkatesh Babu, Class balancing gan with a classifier in the loop. In Uncertainty Artificial Intelligence, pp. 1618-1627. PMLR, 2021.
Harsh Rangwani, Naman Jaswani, Tejan Karmali, Varun Jampani, and R Venkatesh Babu. Improving gans for long-tailed data through group spectral regularization. In European Conference on Computer Vision, pp. 426-442. Springer, 2022.
Suman Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative models. Advances in neural information processing systems, 32, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp.
2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image imaee diffusion models with deep lan- guage understanding. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 06479-36494, Curran Associates, Inc., 2022.
12

---

Published as a conference paper at ICLR 2024
Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. Advances in neural information processing systems, 31, 2018.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Clayton Scott. A rate of convergence for mixture proportion estimation, with application to learning from noisy labels. In Artificial Intelligence and Statisticx, pp. 838-846. PMLR, 2015.
Clayton Scott, Gilles Blanchard, and Gregory Handy. Classification with asymmetric label noise: Consistency and maximal denoising. In Conference on learning theory, pp. 489-511. PMLR, 2013.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, 2015.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference machine learning, pp. 2265. PMLR, 2015.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 11918 19918-11930, 11930, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishck Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances Neural Information Processing Systems, 34:1415-1428,
2021.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Kiran K Thekumparampil, Ashish Khetan, Zinan Lin, and Sewoong Oh. Robustness of conditional gans to noisy labels. Advances in neural information processing systems, 31, 2018.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computa- fion, 23(7):1661-1674 2011.
Vikram Voleti, Alexia Jolicoeur-Martin and Christopher Pal. MCVD masked conditional vidco diffusion for prediction, generation, and interpolation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Xinshao Wang, Yang Hua, Elyor Kodirov, David A Clifton, and Neil M Robertson. Proselfic: Progressive self label correction for training robust deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 752-761, 2021.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey, Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 322-330, 2019.
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? Advances in Neural Information Processing Systems, 32:6838 68499,6666088888 2019.
13

---

Published as a conference paper at ICLR 2024
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2691-26999 2015.
Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual t: Reducing estimation error for transition matrix in label-noise learning. Advances in neural information processing systems, 33:7260-7277 2020.
Xiyu Yu, Tongliang Liu, Mingming Gong, Kayhan Batmanghelich, and Dacheng Tao. An efficient and provable approach for mixture proportion estimation using linear independence assumption. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4480-4489, 2018.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021a.
Yivan Zhang, Gang Niu, and Masashi Sugiyama. Learning noise transition matrix from only noisy labels via total variation regularization. In International Conference on Machine Learning, pp. 12501-12512. PMLR, 2021b.
Zhilu Zhang and Mert R Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In 32nd Conference on Neural Information Processing Systems (NeurIPS), 2018.
Guoqing Zheng, Ahmed Hassan Awadallah, and Susan Dumais. Meta label correction for noisy label learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11053-11061, 2021.
Songzhu Zheng, Pengxiang Wu, Aman Goswami, Mayank Goswami, Dimitris Metaxas, and Chao Chen. Error-bounded correction of noisy labels. In International Conference Machine Learning, pp. 11447-11457. PMLR, 2020.
Yilun Zhu, Aaron Fjeldsted, Darren Holland, George Landon, Azaree Lintereur, and Clayton Scott. Mixture proportion estimation beyond irredueibility. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Searlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 42962-42982. PMLR, 23-29 Jul 2023.
Zhaowei Zhu, Yiwen Song, and Yang Liu. Clusterability as an alternative to anchor points when learning with noisy labels. Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 1291 12-223,,,, PMLR, 18-24 Jul 2021.
14

---

Published as a conference paper at ICLR 2024
CONTENTS
Introduction
2
Problem Formulation
3
Label-Noise Robust Diffusion Models
3.1
Linear Relationship between Clean- and Noisy-Label Conditional Scores
3.2
Transition-Aware Weighted Denoising Score Matching
3.3
Estimation of Transition-Awarn Weights
3.4
Practical Implementation
Experiments
Analysis on Benchmark Dataset with Synthetic Label Noise
Analysis Benchmark Dataset with Annotated Label
8
4.3
Analysis on Real-World Dataset
Combining with the Existing Noisy Label Corrector
4.5
Ablation Study of FransitionAwaal Weights
5
Conclusion
A
Proofs and derivations
17
A.1
Discussion of Remark
17
A.2 Proof of Theorem
17
A.3
Proof of Proposition
18
A.4
Proof of Theorem 3
18
A.5
Proof of Proposition
18
A.6
Derivation of the TDSM with other formats of objective function
19
Derivation the Eq. 8
20
A.8
Computing the reverse transition matrix from the transition matni
20
Related works
20
Diffusion models
20
B.2
2
Conditional diffusion models
................................
21
B.3
Learning with noisy labels
22
B.4
Learning generative models with uncurated label distribution
22
Guidance methods with transition-aware weights
23
Affine score
24
Additional experimental settings
25
15

---

Published as a conference paper at ICLR 2024
Datasets
25
Model configurations
25
Evaluation metrics
26
Experimental setting for 2-D toy case
26
E.S
Estimating the transition matrix
26
Additional experiment results
27
FT
Analysis 2-1) toy case
27
Repeated Experiments
27
Learning trajectory
27
................................
Training and inference time
28
F.5
Comparison with GAN-based models
28
F6
Experimental results with other diffusion framework
29
F7
Performance differences for the class-wise metrics
29
F8
Additional results for combining with the existing noisy label corrector
29
F9
Ablation studies on skip threshold
30
Additional results for the benchmark dataset with annotated label
30
F
Experimental results on severe noise rate
32
12
Further analysis of the transitio-awwar aware weighted function
32
13
Analysis of the noisy label classifier
34
F
Analysis of the potential label noise in the benchmark datasel
36
15
Additional generated images
36
1.
Generated images Clothing- 1M
36
16

---

Published as a conference paper at ICLR 2024
A PROOFS AND DERIVATIONS
A.1 DISCUSSION OF REMARK
We provide detailed derivation for the statement the Remark.
Remark. Let DSM == arg ming D (0; Paatta (X, Y)) be the optimal parameters obtained by minimizing the DSM objective. Then, Sp (xe: y, 1) = Xx, logge - y) for all Xx y,
Although it is direct consequence of previous papers (Vincent 2011 I |Song & Ermon 2019), start by presenting the theoretical derivation from them. We denote the explicit score matching objective LESM (0; Pdata (X, Y)), i.e..
LESM Pdata
Pdata (X,Y), Y))
=E ((t) {(tt) Px (x)
||see (xe, U, t) Vx, log (x, = v) 3 (9)
Then, by the previous work (Vincent 2011; Song & Ermon 2019), the denoising score matching objective is equivalent to the explicit score matching objective, i.e., for all pama (X, Y).
LESM (0; Pdata X, Y)) LOs Pdata (X, Y)) +C
(10)
where C' is constant that does not depend on 0. Therefore, the optimal points of the objective functions are the same.
By assuming Y Y. we can apply LESM and LDMM to noisy labeled dataset. Let 0EsM arg ming LESM (0; Pdmaa(X, Y)). Then, 80 Vx, Vx, 1x. = y).
y). Thus, 80
80 iss
A.2 PROOF OF THEOREM 1
Theorem 1. Under a class-conditional label noise setting, for all xx. D, t,
Vx, log (x, Y 5 E w(xx,
v. y, t) Vx, log log (x, y):
(5)
Y=vl where wxx, t) p(Y y Y 5)) p. Y=
Proof. First, for all f, the perturbed distribution satisfies:
(x (7) = Σ p(V ~ = (). = y)
Vx,,
(11)
under the class-conditional label noise setting (see Eq. Iin the main paper). It means that the noise label transition does not depend on the timesteps. This is because the label determined by the data distribution Pasta- Then, Eq. (5) can be derived as follows.
Xx, Px ū) Vx.Pe(Xi Y = 1)
(12)
(x Y 15)
E =vlē () Vx = y)
(: Eq... 11
(13)
px(x, 8 ===1 =-=))
P(V = v/ = (xp((x Vx,Pr(XelY = v)
(14)
(x, 1 = (1)
Pi(xo|Y = y)
p(Y 3/o T) =v) Xx, log Y v)
(15)
(x Y = 1
wxx,, G. y. Vx. log = v)
(16)
17

---

Published as a conference paper at ICLR 2024
A.3 PROOF OF PROPOSITION 2
Proposition 2. Under class-conditioaal label noise setting, w(xt, ((,,, - Pr - y = D. X.).
Proof. We derive the property that the transition-awes aware weight function represents the instance-wise label transition under the class-conditional label noise setting, i.e.,
(xx, 9. y. t) Pr = y| = xt).
(17)
The derivation is based on Bayes' theorem and the class-conditional label noise assumption.
w (xe, U. p(Y -yY ÿ) Pr (x+ Y y)
Pe xx1 1 = =-)
-(YY y 8 5) p(Y = ū) Pr = (xx))
p(Y v) Pi = (xxx))p((x))
p(8 = B = v)P(Y = vix{)
pt ( -0/x,)
P( = B = y. x)) y|xi)
(:: conditional indep. of ÿ and X, given Y)
PY (1x1)
p(γ ÿ. Y x/xx)
PY x//x;)
Pc y) 30 Xt).
(18)
A.4 PROOF OF THEOREM 3
Theorem 3. Let OTnSu :- arg ming LTDSM Paona (X, be the optimal parameters obtained by minimizing the TDSM objective. Then, under a class- conditional label noise setting with an invertible transition matrix, S (Xt, v. Vx, log (x y) for all xx, y, t.
Proof. First, by Remark and Theorem
I we have the following equation for OTDSM
w(xx V. U, to (Xe, (Xe, 1.t) Vx, log logpe = wo (..... Vxx, logp: = v).
(19)
for all xx. t. Let (()) (xx) ((x,, -. t) is the x c matrix, where (x,) = w(xi: U. for Ú. y 1,..., c. Then, Eq. (19) can be written in matrix form:
w( (x+)
So (xt, U,
(10 Vx, log (x, = 4) y
(20)
y=1...
for all Xx, f. Then, it is enough to show that W (9) (xx) is invertible. This is because, by multiplying the inverse of W( (x.) for both sides in Eq. 10 we obtain Sojpsu (xr, y, () = Vx, log (ogg.((x/)) (x+lY = v)-
We can express ((1)
5) and (1) (x₂) = diag(pt Y = y))- Then, (xx)) and L(x;) is invertible since the support of p is the whole space. Also, S is invertible due to the assumption of the statement. Thus, (!!) (x,) is invertible.
( (x,)SL (x:) where (
diag (1/p diag(1/pr
A.5 PROOF OF PROPOSITION
Proposition 4. Let 0 SDSs arg ming Ls Paaso (X. Y)) be the optimal parameters obtained by minimizing the S S-weighted DSM objective. Then, under class-conditional label noise setting with invertible transition matrix, So, (x,, 9. differs from Vx, log lo,p((x.|y y)-
18

---

Published as a conference paper at ICLR 2024
Proof.
Similar to the proof of Theorem B in Appendix A.4
have the following equation for
B--DSM-
Σ y=1
p(Y y/8
(x, y.
Vx, logpi(x, D)
(21)
w Xt, (xx,
U. V. logp:(x; y),
(22)
for all Xx, U, t. With the same notation in Appendix
A.4
Eq. 21 can be written in matrix form:
S usss (xe, t)
= W
(xx)
V
log Pi (x, Y = y) y=1....e
(23)
y=1..c
+ s0n (xe, y, t)
= =S-1W(00 (xx)
Xx logp: = v) y=
(24)
y=l..c
for all X,, Since (x,, V. 1, + p(Y - y/ر 3)) for label noise setting, S S (x)
#
Thus, S0% (Xe, y, t) differs from the clean clean-label conditional score, Vx. log Pr (x y).
A.6 DERIVATION OF THE TDSM WITH OTHER FORMATS OF OBJECTIVE FUNCTION
Assuming a Gaussian perturbation kernel, noise or data samples from the perturbation kernel can be expressed as linear transformation on the score vector. Since our proposed TDSM is also formed as convex combination of score networks, it can be applied by replacing single network output with weighted sum network outputs.
We provide an example for the derivation of the TDSM with EDM framework we used primarily in our experiments. In EDM (Krrras et al., 2022), the perturbed data sample x, from pt is obtained by adding i.idd Gaussian noise of standard deviation to the data instance They introduce denoiser network D that is trained by minimizing the data reconstruction loss:
LDSM-RC (0; Pdato Y)) = EE ((t)lxx. (.Y) (nn))nnnn
x|12] (25)
The optimal denoiser network DMsM trained by Eq. 25 satisfy:
Vxi lo (x y) = (Disu (X1, v,
xx)/1²?
(26)
Then, we can formulate the TDSM objective function for applying Eq. called TDSM-RC objective.
25 to a noisy labled dataset,
LTDSM-RC Paata (X, Y))
== =Be{X(1)B (((() Ax ( En
Σ w(x n, n. ()D( -, -x1 (27)
By Theorem and Eq. (26), the optimal denoiser network DinsM trained by Eq. (27 on a noisy-labed data distribution satisfy:
(xx, T. t) (x: =y) = Vx, log logpp(xxl(* = 1))
Σ 9=1
we xt, T. U.1. DiDSM) v. - xx x1) (28)
Because Eg (...,..... for all xx, G, we can further derive follows:
D DiTDSM Xt. H, X1 x1).
w (oxx....V.))V. (x =) w woxe. U. v,
(29)
t2
By following the same derivation as in the proof of Theorem 3 the followings are satisfied:
Vx. log Ix = y) = (Di (x, t) - x/)/t²
(30)
Therefore, we can verify that the denoiser network which is trained by the TDSM-RC objective function the nobsy-label data distribution converges the clean denoiser network.
19

---

Published as a conference paper at ICLR 2024
A.7 DERIVATION OF THE EQ.
We can compute the transition-aware weight function w using the noisy label prediction probability (Y' xx) and the transition matrix S. To achieve this, we use Bayes' theorem on both the numerator and the denominator:
Pr Y v) Pc = y)xx) = 1) p( 5) Pc y/xx)
(31)
Px Y D) Pc(Y = (p) = y) Pt = ū|x,) p(Y = y)
Furthermore, because of the class-conditional label noise assumption, we know that:
PY x|x)) p(Y =
EplY=
j(xx)) for i I,,...
- = p(Y j)
(32)
j=1
This equation can be written in matrix form using the invertible S follows:
p((Y=)
px())) p(Y=)) jil..
=S1 Pn(Y=ilxx)
iill..c
P(Y=))
j=1..c
m(Y=i)
vi=l..
(33)
Then, we can compute the transition-aware weight function w using the noisy label prediction probability pr( (x|() and the transition matrix S:
Sg 1 SpiPo = x1xx)
wr(x;, 0. P:(Y = 17 X1)
(34)
p(Y= =)
A.8 COMPUTING THE REVERSE TRANSITION MATRIX S FROM THE TRANSITION MATRIX T
We would like to clarify that have defined the transition matrix T where Tu p(Ý = =|l i) and the reverse transition matrix S where St, = p(Y = jă = i). The elements of the reverse transition matrix can be computed follows:
St = p(Y = =) = p(Ý = =| = j)p(Y = j)/p(Ý = = TT,ppY = j)/p(( = =). (35)
=
we have the transition matrix T and the noisy label prior p(Y), we can compute the clean label prior pYY) using the following relationship.
diag (Lo 1
T p(Y = =i) j=1..c i) im
(36)
This equation is based on Eq. 35 and E₁ Si, = Thus, we can compute the reverse transition
matrix S from the transition matnx T.
B RELATED WORKS
B.1 DIFFUSION MODELS
The goal of generative models is to approximate the data distribution Paau(Xp) to the model distri- bution Pa (x.). Specifically, Rkelihood-basse generative models define explicit specification of the distribution, and the objectives of these models are to maximize the likelihood. These models typically define the easy-to-sample prior distribution and generate samples from this prior distribution.
Denoising Diffusion Probabilistic Model (DDPM) defines fixed noise process, characterized as Markov chain, that perturbs the data instance by adding Gaussian noise (Sohl-Dickstein al. 2015 Ho et al. 2020):
((x1.1 (x0) = II gxx, xx-1) where ((-x)x == N(x /1 S,x--1,,111
(37)
1=1
20

---

Published as a conference paper at ICLR 2024
where X1:7 are latent variables of the same dimensionality as x and { be the varaince schedule parameters. Then, the goal of DDPM is to approximate this fixed noise process by a trainable Markov chain with Gaussian transitions, which is called denoising process:
T Pt 1|c (x xx) where (x = 1i t), (x,, t)),
(o(x)))) = Pr (xT) II
(38)
where (x(x)) is the prior distribution, e.g., the standard Gaussian distribution. The training objective of DDPM is to minimize the upper bound of the negative log- logllikeihooo::
E| log pe (xx)) - B. log p8 (xo/x1) +
Σ t=2
DKL (9(xx-111 xo) pp (x 1xx))
+ +DKI. (g(x7/2(())) pr((x)))
(39)
For the continuous time spaces, this DDPM process can be described through Stochastic Differential Equations (SDEs) (Song et al.] [02T]] 2020). Specifically, Eq. 2 represents the forward diffusion process, which is defined by the drift function f and the volatility function g.
dx, f(x,, t)dt + g(t)dw,,
(40)
where w, denotes the standard Brownian motion and / E [0. The forward process gradually perturbs the data instance x X to Xx.
The SDE community also elucidated the existence of a corresponding reverse diffusion process (An- derson 1982):
dx,
V1) log):(x;)Jdī g())dw,,
(41)
where (x,) is the probability density of xx, defined by the forward process, and w, is the reverse- time standard Brownian motion. The reverse process gradually denoises from XT to Xo. We can sample data instances through the reverse process.
In spite that the generation task will require the reverse diffusion process, this process is intractable because data score Vx, log Pa (x)) is generally not accessible. Therefore, the diffusion model aims to train the score network to approximate Vx, log Pa (x+) through the score matching objective function with I.2 loss (Song al. 2020). Like the training objective of DDPM, it is known that the score matching objective is known to be an upper bound on the negative log-likelihood for specific temporal weight functions. This score matching objective is equivalently formulated the noise
prediction (Ho et al. 2020; Dhariwal & Nichol 2021) or the data reconstruction (Kingma et al. 202T
Karras et al. 2022 objectives.
B.2 CONDITIONAL DIFFUSION MODELS
Conditional diffusion models have also been developed in order generate samples that match a desired condition y. For the conditional models, we need to approximate the conditional score (y). so additionally take condition input score network (Dhariwal & Nichol 2021 Karras et al.) 2022). Additionally, there are some approaches to guide conditional generation.
Xx, log (x.xxxx())) (xely)
One approach decomposes the conditional score using Bayes' theorem, and estimates the gradient of the log posterior probability of label using an auxiliary classifier:
Vx, lx, = y) Vx, logp.((x) + Xx, log = y x))
(42)
This is known as the classifier guidance method (Dhariwal & Nichol 2021 Chao et al. 2022 Kim al. 2023). To amplify the effect of the condition, positive scaling factor can be multiplied to the classifier gradient term.
On the other hand, the classifier-free guidance method (Ho & Salimans 2021 is also proposed, which is a way to achieve the same effect as the classifier guidance without using a classifier. This method utilizes the score network, which produces both unconditional and conditional scores by
21

---

Published as a conference paper at ICLR 2024
introducing auxiliary class for the unconditional score. Then, they use linear combination of the unconditional score (x(x, t) and the conditional score so (xe, y, t) with a > 0 for sampling.
SCIG (Xe, y. (1 + a: 3=)80 Xx, y,
ase aso(xe;;)
(43)
Conditional diffusion models have been used in various applications by using conditions as la- bels (Kong et al. 2021; Meng et al.
2022a), text (Nichol et al. 2022 Rombach al. 2022), or latent
representations (Kim et al. 2022b Preechakul et al. 2022).
B.3 LEARNING WITH NOISY LABELS
Deep neural networks are known be vulnerable to overfitting, even when trained on dataset with
random labels (Zhang et al., 2021a). Consequently, the presence of noisy labels in dataset poses
a significant challenge. To mitigate this problem, numerous approaches to classifier learning from
noisy labels have been introduced, including sample selection (Cheng et al. [02T]] Han et al. [0188]
Jiang ct al.
2018), label correction (Wang et al. [022 Zheng et al. 2021 2020), robust loss (hhosh
ct al.
2017
Wang et al. 2019 Zhang & Sabuncu 2018), and transition matrix estimation (Patrini
et al.
2017
Xia et al.) 2019 Zhang et al. (021b).
Specifically, transition matrix estimation methods model the relationship between clean and noisy labels when estimating the transition matrix, which represents the probability that a clean label of a sample is corrupted another label. Patrini al. (2017) proposed two loss correction structures: forward and backward correction. They also provided the theoretical analysis of the statistical consistency. To estimate the transition matrix, they use the anchor points that belong to a specific class with probability one. Xia et al. (2019) argued that anchor points are not available in the datasets. To address this problem, they proposed T-revision, which revises the learned transition matrix and validates from noisy validation dataset. Yao et al. (2020) proposed the dual-T estimator by factorizing the product of two easily estimated transition matrices. Zhang et al. (202Tb) pointed out the overconfident predictions of neural networks and proposed the new regularization terms which based on the total variation distance. [i et al. (202T) proposed another regularization term which minimizes the volume of the simplex formed by the transition matrix. Zhu et al. (2021) suggested the new transition matrix estimator based on the clusterability, which implics that nearby instances would have the same label. Cheng al. (022)) estimated the transition matrix using karward-bacwward cycle-consistency regularization.
Mixture proportion estimation has been used solve weakly supervised learning problems, it is an area that is closely related to the transition matrix estimation (Scott 2015) [l For example, Scott al.] (2013) proposed consistent estimator from the results of mixture proportion estimation to train the classifier under asymmetric label noise. There have been a number of studies based on mixture fraction estimation, each with assumptions such as anchor set (Liu & Tao 2015 [Sott] 2015). separability (Ramaswamy et al.) (2066), linear independence (Yu et al. 2018), and tight posterior bound (Zhu et al.] [022)). Our estimation of transition-awar weights differs from the traditional mixture proportion methods in that our weights depend on the diffusion timestep, which requires training a time-dependent noisy label classifier. Additionally, it is important to note that our weight function represents the weights related to the data score relationship, and these values do not directly indicate the proportion of the distribution. Therefore, conventional methods for estimating mixture proportions based on the samples may not be directly applicable.
B.4 LEARNING GENERATIVE MODELS WITH UNCURATED LABEL DISTRIBUTION
The labeled datasets required for training conditional generative models may suffer from the uncurated label distribution, such noisy labels and imbalanced labels. Some previous studies have pointed out these problems.
There are a few studies for generative model learning from noisy labels. Thekumparampil al. (2018); Kaneko et al. (2019) proposed algorithms to overcome the difficulties of training GANs with noisy labels by matching the nobsy-label conditional probability with the transition matrix. Specifically, the structure of a generator is similar to the traditional GAN, but they modify the
*This phenomenon can also be found in Fq. (1) in the main paper, where each noisy-label conditional distribution he expressed by a mixture of clean-lahel conditional distributions.
22

---

Published as a conference paper at ICLR 2024
Table 6: Comparison between the label-noise robust GAN (rGAN) Kaneko et al. 2019) and diffusion models (ours).
Thekumparampil al. 2018
Thekumparampil et al.
2018
Kaneko al. 2019
Ours
Base model
GAN
Diffusion model
Objective
Noisy- label conditional distribution matching
Nabey conditional score matching Pc(y ū.m)
Label transition information
r(
Time- dependent label transition
Instance-wise label transition
discriminator structure by incorporating the label transition matrix. When the discriminator is fed the samples generated by the generator and the corresponding labels, the labels are corrupted by the label transition matrix to become noisy labels. Since the given dataset is also a noisy labeled dataset, training the discriminator in this way makes a match between the noisy labeled datasets. Then, the generator can have clean label conditional distribution. However, the diffusion model cannot completely mitigate the impact of noisy labels with the transition matrix alone because targets the gradient of the -illeeelioood, which is explained in Section Therefore, propose adapt the transition matrix framework to fit the diffusion models. We summarize the differences between the GAN-based models and our diffusion model in Table
In addition to the problem of noisy labels, various efforts have been made to tackle the issue of uncurated label distribution in generative models. For example, (Rangwani et al., [021]] [022] Qin
ct al. 2023 Kim et al. 2024) addressed the problem of class imbalance or long- tailed distribution when training generative models. Rangwani al. (2021) highlighted the problem of long-tail distributions in conditional GAN training, and they proposed the class balancing regularizer with the theoretical evidence. |Ranwwani al. (2022) identified class-specific mode collapse in conditional GANs trained on long-tailed distributions, and they introduced a group spectral regularization to reduce the spectral norms of grouped parameters, as motivated by their analysis. More recently. Qin et al. 2023) aimed to alleviate the class imbalance problem in diffusion models. They proposed a new class-balancing diffusion objective based on theoretical analysis. Also, Kim et al. (2024) proposed time-dependent importance reweighted denoising score matching to address the problem of dataset bias. Similar to these works, we identify challenges of diffusion models under noisy labeled datasets and propose novel label-noise robust objective based on theoretical analysis.
C GUIDANCE METHODS WITH TRANSITION-AWARE WEIGHTS
In addition to generating conditional samples directly from a conditional score network, there are two methods for guiding conditional generation in diffusion models: classifier guidance (Song et al.) 2020 Dhariwal & Nichol 202T) and classifier-free guidance (Ho & Salimans| 2021). However, these guidance methods also suffer from label noise, as shown the below experiment. To address this issue, we propose strategies to apply our transition-ane weight function in these methods, which can mitigate the effects of noisy labels.
The classifier guidance method (Song et al., 2020 Dhariwal & Nichol 2021) utilizes an unconditional diffusion model and a classifier to generate samples that satisfy a certain condition by estimating a conditional score using Eq.
Vx, log
Pa (x, V -y Vx (x.) Xx, lo = y/x.).
(44)
However, when dealing with noisy labels, the classifier provides misleading guidance even though the unconditional diffusion model is not affected by noisy labels. In this case, our proposed transition- aware weight function can provide correct guidance using an unconditional diffusion model and a nabsy-label classifier. We can use Theorem ] and Eq. (44) to derive the relationship between the gradient of the log probability of clean and noisy labels:
Vx, logp = -0)
w(x,, G. U. Vx, logpa(X. = v).
(45)
Vx, log + Vx, logpa = ū(xx))= = Σ wxx, G. v. (V lxg) + Vx, logpe = n|x:)]
(46)
23

---

Published as a conference paper at ICLR 2024
Because Σy=1 w(x,, y. for all Xx, U, we have:
x, log = =]/x,)= = w (xx, 1, logpi = 5/xx)) for D = 1, (47)
Xx, pY = y/xi)= = Σ w (x,, D. t) V lo = }(ز{) for = 1,..,, (48)
where w' (X,, y. y. is the (y. 37-th element of the matrix (w(x,, 's (t) and w" (x,, V, v.y) is well-defined the transition matrix S is invertible. Using this derivation, we propose a transition- aware weighted classifier guidance method that can be sampled by noisy-label classifier:
Xx, log (xx = Vx, log (x., +
w" (x,, v, v) Vx, logp:(Y 5(xx))
(49)
We can evaluate the weighted sum of gradients by Vx, EE=1 (x,, (X,, y. lo = r)*)) where w" (x4, J. y, t) a detached version of w" (x,, D. y, t). This method can be applied without additional training by utilizing the already trained noisy classifier.
On the other hand, the classifier-free guidance method (Ho & Salimans 2021 uses unconditional and conditional scores, without classifier, achieve the same effect the classifier guidance method. Therefore, when training dataset contains noisy labels, we only need to change the training objective of the conditional score network from DSM to TDSM.
We experiment with applying guidance meth- ods on the noisy labeled dataset. We report the performance by changing the hyperparameter that balances the effects of the unconditional score network and the guidance signal. This hy- perparameter is known to trade off between un- conditional and conditional performance (Chao) ct al. 2022). Figure 6 observes that our ap- proach generates better conditioned images for the same image quality compared to the base- line.
CW-31D
CW-3ID-L
FID
FID
(a) Classifier guidance
(b) Classifier-free guidance
Figure 6: Trade-off between FID and CW-FID on CIFAR-10 dataset under 40% symmetric noise rate.
D AFFINE SCORE
Although our model tends to produce more samples that match the given condition, it still produces some mislabeled samples. To address this issue, we propose an affine combination method of the noisy-label conditional score and the clean-label conditional score by leveraging their linear relationship, which is proved by Theorem I obtain equation follows.
For each condition E y and non-negative X, we can
(1 + x) logp = v) - AV 1og = y)
(1 X)V (xg = y)
wl (x.J. (x =
(50)
= +1+ 11+AA1 X(1
wxx, v. U. V log (x, = y) Σ (-xw(xx y, V log pe (x: = =) juy
should be noted that, for non nog-tvgative X, the followings are satisfied: {1 + X(1 (x, y. y. t))} Ejty( (-xo(x> >w((x,, y, 1)) = 1. Therefore, Eq. (50), called affine score, is affine combination of the clean-label conditional scores.
Specifically, this affine score gives more weight to the target label y. while exerting the opposite force on the non-target labels. Theoretically, satisfies 1 w(x, y, y. I 0, and - -Xw(X,, y. t) < 0 for # y. We experimentally confirmed that the affine score with the proper maintained the sample quality and produced almost conditionally mismatched outliers.
24

---

Published as a conference paper at ICLR 2024
Base Alline
Density (1)
120
Coverage (t)
85
FDD
10.0
110
80
100
9.5
75
A
70
DWr-Denity (1)
120
85
CW-FID (1)
30
CAS
WW-Covenge
80
20
75
50
80
70
A
A
A
Figure 7: Ablation study of the hyperparameter À in the affine scores using the model trained on the CIFAR-10 datasets under 40% symmetric noise rate.
Figure illustrates the performance of images generated from an affine score by varying the hyperpa- rameter X. The results indicate that using an affine score yields better samples across all performance aspects, except for FID. The generated images presented in this paper employ the affine score with
E ADDITIONAL EXPERIMENTAL SETTINGS
E.I DATASETS
We describe the generation of noisy labeled datasets in more detail. The overall setup follows Kaneko et al. (2019). Specifically, for symmetric noise, labels are randomly perturbed with the probability of the noise rate. For asymmetric noise, we perturbed labels with the probability of the noise rate only for similar classes. In this case, for MNIST, labels are flipped by 2 - 7, 3 -> 8, and ++ 6, and for CIFAR-10, labels are flipped by truck + aut automobi i 1e, bird - ai airplans deer + horse, cat + dog. For -00, we have circularly flipped classes in the same superclass.
E.2 MODEL CONFIGURATIONS
We utilized 8 NVIDIA Tesla P40 GPUs and employed CUDA 1 11.4 and PyTorch 1.12 versions in our experiments. Our model framework and code are based on EDM (Karras et al. 2022). 6 For all experiments, used DDPM++ network architecture with U-net backbone, which is originally proposed by Song et al. (2020) and modified by Karras et al. (2022). The experimental settings align with CIFAR-10 experiments in Karras et al. (022). The score network was trained with batch size of 512, and the training iterations were set to 400,000 for MNIST and CIFAR-10, 200,000 for CIFAR-100. For Clothing MM, the score network trained with a batch size 256 for 200,000 training iterations.
For the lobey-label classifier, we use shallow convolutional neural network backbones for MNIST, ResNet-18 (He et al. (016)) backbones for CIFAR-10 10, and ResNer-34 backbones for CIFAR-100 and Clothing- When computing the noisy-label probability, we multiply the softmax output of the classifier and the transition matrix ensure that the output belongs the simplex formed by the transition matrix. The classifier was trained by the cross-entropy loss with Adam optimizer using learning rate of 0.001 and a batch size of 1, 1,024 for 200,00 0000 training iterations. We the EDM deterministic sampler (Karras et al. 2022) with 18 steps (NFE 35), which is the default setting of CIFAR-10 in Karras al. (2022).
25

---

Published as a conference paper at ICLR 2024
E.3 EVALUATION METRICS
We evaluate conditional generative models from multiple perspectives using four unconditional generation metries, including Fréchet Inception Distance (FID) (Heusel et al. (017). Inception Score (IS) (Salimans et al. 2016). Density, and Coverage (Nacem al. 2020). and four conditional genera- tion metrics, namely CW-FID, CW-Density, CW-Coverage (Chao et al. 2022), ) and Classification Accuracy Score (CAS) (Ravuri & Vinyals 2019). We use DLSM (Chao et al. 2022) code _ for evaluations.
The term unconditional metric refers to metric that can measure samples regardless of their labels. We generate samples by performing conditional generation on each class. We use these samples to evaluate both unconditional and conditional metrics on each model. The unconditional metric is calculated from the generated images only, not accounting their labels. The unconditional metric is evaluated in the same way in other papers addressing the conditional generation (Kancko et al. 2019 Chao et al. 2022).
FID measures the distance between real and generated images in the pre-trained feature space (Szegedy et al. 2016), indicating the fidelity and diversity of generated images. IS quan- tifies how well the generated images capture diverse classes and whether each image looks like a single class, reflecting image fidelity. The Density and Coverage are reliable versions (Naeem et al., 2020 of Precision and Recall (Sajjadi et al., 2018 Kynkäänniemi et al.) 2019), respectively. These two metrics examine how well one image distribution covers the other in the feature space (Szegedy) cct al., 2016). Higher Density and Coverage values indicate better image quality and diversity, respec- lively. We do not use Precision and Recall metrics. This is because the nearest neighbor distribution produces an overestimated distribution around the outliers (Naeem et al. 2020), so these metrics are not appropriate in the presence of noisy labels.
To measure how well the generated images match the given conditions, we evaluate metrics on Class-Wise (CW) images. The CW-metric evaluates the metric separately for each class and averages them (Chao et al. 2022). Note that CW-FID, also called intra-FID, is a widely used metric in the field of conditional generative models (Miyato & Koyama 2018: Kaneko et al., 2019). Previous label-noise robust GAN model (Kaneko et al.) 2019) mainly used this metric, and they mentioned that this metric measure of the quality of conditional generative distribution. Additionally, we evaluate CAS, which is the classification accuracy of the test dataset, where the classifier is trained on generated samples. This metric demonstrates that the generated images contain representative class information. The classifier, which is to evaluate CAS metric, is trained by the cross-entropy loss with Adam optimizer using learning rate of 0.01 with batch size 128 for 200 epochs. The classifiers shallow convolutional neural networks for MNIST, ResNet-50 for CIFAR- 10 and CIFAR-100, and pre-trained ResNet-50 for Clothing- 1M.
It should be noted that the MNIST dataset is not suitable for evaluation using these metrics except for CAS, so instead we use density and coverage metrics based on the features of a randomly initialized VGG-16 (Simonyan & Zisserman 2015), as suggested by Naeem et al. (2020)
E.4 EXPERIMENTAL SETTING FOR 2-D TOY CASE
We consider a 2-class Gaussian mixture model with p(xly 1) N (3. I), p(xly= A((x; (-3, -3)" I), and p(y = 1) ply 2) 0.5. Also, set the transition matrix as S = 0.8 0.5 ). In this example, we use the VE SDE for the diffusion process, e.g., the conditional distributions of class 1 on time / are: Po(xxly ly 1) = (x; (3,3)T, (1+12)I), Then, can evaluate w(x w(x, U. y. y. from definition. Note that (x,, 0. y = 1, t) + u (x. N.V = 2, for all xx, ū. t.
E.5 ESTIMATING THE TRANSITION MATRIX
We utilized the VolMinNet Li et al. 2021) framework to obtain the noisy label classifier and the transition matrix in an end-to-end manner. The VolMinNet loss function consists of the sum of the cross-entropy loss for the noisy label using the transition matrix and the loss that minimizes the volume of the simplex formed by the transition matrix. We utilize this loss to train the classifier.
26

---

Published as a conference paper at ICLR 2024
Original DSM
S-weighted DSM
TDSM (ours)
True score
1
Figure 8: Comparison of the optimal score networks in 2-dimensional toy case. The dots depict generated samples based on each score network. The arrows represent the score vectors, with the color representing the norm of the vectors.
In this case, we get the estimated transition matrix, and the noisy-label probability is obtained by applying the transition matrix the classifier output.
To train the transition matrix using the VolMinNet loss, we make the elements the transition matrix trainable parameters. The initialization and the optimizer setting of the transition matrix is followed by Li et al. (Li et al.) 2022I. In particular, the transition matrix was trained for about 7,500 iterations, which different from the number of iterations used for training the classifier. Therefore, we fix the transition matrix after the matrix training iterations while only the classifier was further trained. Since our noisy-label classifier needs learn the noisy labels, it requires more iterations. We observed that continuously training of the transition matrix causes it to converge towards the identity matrix.
ADDITIONAL EXPERIMENT RESULTS
ANALYSIS ON 2-D TOY CASE
Figure g visualizes the optimal score networks of DSM variants in 2- 2-dimensional toy case. We consider a 2-class Gaussian mixture model with p(xly = 1) = (xx (3, 3)" p(xly = 2) N(x: (-3, -3)T. I). and we the transition matrix as S 0.8 0.2 S S-eeggtted DSM means that the weight function is determined solely by the transition matrix S, i.e., w w'(x, J. v,() p(Y v 5)) This analysis reveals that the instance- indeeeedett transition-aware weights cannot produce clean-label conditional score in the diffusion models, which is different from the GAN models.
F.2 REPEATED EXPERIMENTS
repeated experiments the CIFAR-10 dataset with 40% symmetric and asymmetric noise. Figure / shows the average values of each metric along the training iterations, with min-max values. We also perform t-tests for all cases, and in most cases observe statistically significant improvement over the baselines.
F.3 LEARNING TRAJECTORY
Figure presentss the trajectory of the metrics during the training process. For the conditional metrics, we find that the baseline model does not reach the performance of our early-stage model, even with continued training. Furthermore, for all metrics except FID, our model consistently outperforms the baseline across all snapshots. This observation highlights the persistent efforts of our model in estimating the clean-label conditional distribution and the difficulty existing models in overcoming label noise.
27

---

Published as a conference paper at ICLR 2024
3.0
10.0
Density (
110
Coverage (f)
FID (
80
2.5
9.5
100
90
75
2.0
Training iterutions
Training iterations
Training iterations
Training iterations
CW-FID ())
60
Aoppsace (1)
100
CW- CyV-Coeragge
75
CAS
70
50
65
50
100
150
200
50
100
150
200
100
150
200
150 200 100 50
Training iterutions
Training iterations
Training iterations
Training iterations
Base
Our
40% symmetric noise
3.0
10.0
Doassa (1)
18 (1)
100
Coverage
F1D
90
50
100
150
200
50
100
150
200
50
100
150
200
100 50
150
200
Training iterations
Training itcrations
Training iterations
Training itcrations
CW-FID ())
CW- Dii-Denityy (1)
100
CW Cer-CCoerggge (t)
80
CAS (1)
90
-5
14
68
100
150
200
50
100
150
200
50
100
150
200
50
150 100
200
Training iterations
Training iterations
Training iterations
Training iterations
Base
Ours
(b) 40% asymmetric noise
Figure 9: The results of repeated experiments on CIFAR-10 10 dataset for each training iteration (every 25 million images). We ran the experiment three times. The solid lines represent the mean of the metrics, and the shaded regions illustrate the minimum and maximum values.
F.4 TRAINING AND INFERENCE TIME
First, the inference (or sampling) time remains unchanged compared to the baseline model. This because we only change the training objective ensure that the score network becomes the clean-label conditional score given noisy labeled dataset. During inference, the existing sampler is used for sampling, which unaffected by the introduced weights, thus preserving the inference time.
In the process of evaluating the training objective, it is necessary to compute the weighted sum of the score networks. However, we introduce the practical techniques for reducing on time and memory usage in Section 3.4 In our experiments, Table 7 shows that the training time for our models on CIFAR-10 and CIFAR- only about 1.5 times, not the number of classes times, as long.
In addition, Figure TO provides comparison of performance based training iterations. For the conditional metrics, our models outperform the best baseline model for all training iterations, and for the unconditional metric, observe that most metrics similar to or better than the best baseline model's performance around the halfway point our model's training. Consequently, we can achieve better performance noisy labeled datasets within the same training time.
F.5 COMPARISON WITH GAN-BASED MODELS
We compare the generation performances with the label-noise robust GAN model (Kaneko et al.. 2019) in Table g The diffusion models generate images much better than the GAN model even in
28

---

Published as a conference paper at ICLR 2024
82
base
110
2.75
9.8
80
2.50
5
9.6
Density
100
Coverage
78
9.4
basc
basc ouzs
76
basc
2.25
90
ou15
outs
9.2
74
200
100
200
100
200
100
200
Truining iterations
Training iterations
Training iterations
Training iterations
CW- De--Density (t)
CW-Covcrage (t)
CW-FID (1)
90
75
25
ourS
CAS
55
base
ours
80
base ours
70
base
50
65
200
Training iterations
Training iterations
Training iterations
Training iterations
Figure 10:
The experiment results on the CIFAR-I0 datasets under 40% symmetric noise rate with respect to the training iterations (in every 25 millions of images).
Table Training time on the CIFAR-10 and CIFAR-101 100 datasets. The values represent the training time (second) per 1K images.
Symmetric
Asymmetric
Dataset
Model
20%
40%
20%
40%
DSM TDSM
1.77
CIFAR-10
2.21
2.43
1.85
1.90
DSM
1.78
CIFAR-100
TDSM
2.35
3.06
2.01
2.27
the noisy label dataset, and our models boost the performances by increasing the robustness to label noise. Overall, the results demonstrate that our proposed approach is effective in handling label noise in diffusion models, leading improved image generation performance under noisy label settings.
F.6 EXPERIMENTAL RESULTS WITH OTHER DIFFUSION FRAMEWORK
Table 9 shows the experimental results with the +DPM+ (Song al., 2020) framework. Consistent with previous results Table D we observe that our model outperforms the baseline model on all conditional metrics and most unconditional metrics.
F.7 PERFORMANCE DIFFERENCES FOR THE CLASS-WISE METRICS
Figure IT shows the performance difference between our models and the baseline models for each class. As shown in the figure, our models outperform in most classes. In particular, we see an improvement in performance even for classes that are not affected by label noise for asymmetric noise cases.
F.8 ADDITIONAL RESULTS FOR COMBINING WITH THE EXISTING NOISY LABEL CORRECTOR
We conducted experiments by 1) obtaining the corrected labels from the existing classifier learning methods with noisy labels, VolMinNet (Li et al. 2021) and DISC Li et al. 2023), and 2) training the diffusion model with the corrected labels. We the noise ratio for training the diffusion model. The results are summarized Table T fo VolMinNet and Table 11 for DISC. The experimental results indicate that applying TDSM objective consistently
transition matrix structure with revised
29

---

Published as a conference paper at ICLR 2024
Table 8: Experimental results on the CIFAR-10 and CIFAR-100 datasets under 40% noise rate, compared with the GAN-based model. 'GAN' is the baseline GAN model and 'GANN' is the label- noise robust GAN model.
Symmetrie
Asymmetric
Metric
GAN
rGAN
DSM
TDSM
GAN
IGAN
DSM
TDSM
FID
(4) 14.93
11.08
2.07
2.43
11.04
10.46
2.23
2.06
IS
(1)
8.21
8.51
9.83
9.96
8.54
8.56
10.09
10.02
CIFAR-I 10
Density
(1)
70.02
78.90
100.94
111.63
78.06
77.47
101.25
105.19
Coverage
(1)
52.51
58.92
80.93
82.03
59.19
58.45
81.10
81.90
CW-FID
(1) 53.26
27.03
30.45
15.92
29.10
26.29
15.18
12.54
cond
CAS
(
33.26
53.27
47.21
62.28
45.07
49.34
68.98
71.51
CW-Density
(
49.46
71.63
73.02
97.80
68.40
70.71
92.13
99.21
CW-Coverage
(1)
44.02
56.51
71.63
78.65
55.48
55.91
78.12
79.98
FID
()) 19.46
15.09
3.36
6.85
13.81
13.96
2.73
2.81
IS
( 8.05
9.11
11.86
12.07
9.12
9.43
12.51
12.57
CIFAR- 100
Density
( 69.85
75.16
81.70
88.45
79.13
78.16
87.06
87.01
Coverage
(1) 45.14
50.42
73.92
72.12
53.25
52.40
76.56
76.27
CW-FID
(3) 137.35
109.64
100.04
93.24
114.71
105.93
89.13
73.13
cond
CAS
(1) 9.89
19.84
15.41
21.17
13,84
18.08
23,50
34.47
"W-Density
(1) 33.37
53.21
49.77
60.60
47.92
60.29
60.27
74.30
CW- WW-Coveruge
(1) 33.50
42.31
60.64
63.89
41.57
45.09
64.19
71.48
Table 9: Experimental results for DDPM++ (Song et al. 2020) backbone on the CIFAR-10 10 dataset
with various noise settings. The percentages headers represent the noise rate. 'un" and 'cond" indicate whether a metric is related to unconditional or conditional generation, respectively. Bold numbers indicate better performance.
Symmetric
Asymmetric
Metric
20%
40%
20%
40%
DSM
TDSM
DSM
TDSM
DSM
TDSM
DSM
TDSM
FID
(L)
2.11
2.24
2.27
2.61
2.20
2.07
2.44
2.24
IS
(1)
9.85
10.04
9.79
9.93
9.89
9.98
9.95
9.95
S
Density
(t)
103.17
110.15
101.67
110.72
104.01
106.22
104.30
106.04
Coverage
(1)
81.21
82.14
80.69
81.28
81.58
81.87
81.48
81.73
CW-FID
(L) 16.11
11.95
29.31
15.87
12.06
10.99
15.32
12.92
cond
CAS
(1) 66.86
71.18
52.70
61.35
72.75
73.80
68.58
71.22
CW-Density
(t) 91.83
103.94
74.57
98.02
99.30
103.30
95.12
100.30
CW-Coverage
(1) 78.47
80.72
71.76
78.27
80.50
81.14
78.67
79.77
improves the performance with the corrected labels. Therefore, we believe that approach tackles the noisy label problem from diffusion model learning perspective, providing an orthogonal direction compared conventional noisy label methods.
F.9 ABLATION STUDIES ON SKIP THRESHOLD
Figure 12 demonstrates the trade-off between CW-FID and the training time 1K images depending on the skip threshold т, as explained in Section 3.4 Using a small threshold leads long training time due more network evaluations, and using a high threshold leads to decrease in performance due to inaccurate targets. From this result, we set 0.01 for all experiments.
F. ADDITIONAL RESULTS FOR BENCHMARK DATASET WITH ANNOTATED LABEL
The experiments for Table 2 in Section 4.2 applied models to benchmark dataset with the conjecture that some annotated labels may be noisy. For the conditional metrics, we need to have true labels, but due the conjecture, it may not appropriate these metrics with the annotated labels.
30

---

Published as a conference paper at ICLR 2024
CW-FID
CW-Density
CW-Coverage
Differences
0.05
0.1
0.00
6
8
4 5
9
2 4
Sorted classes
Sorted classes
Sorted classes
(a) CIFAR-10 with 20% symmetric noise
CW-FID
CW-Density
CW-Coverage
Differences
2.0
0.1
0.25
6
2 3 5 8
0
2 4 5
Sorted classes
Sorted classes
Sorted classes
(b) CIFAR-10 with 40% symmetric noise
CW-FID
CW-Density
CW-Coverage
Difference
0.02
2.5
0.00
0.0
0.0
5
4 Sorted classes
2 4 6
Sorted classes
Sorted classes
(c) CIFAR- 10 with 20% asymmetric noise
CW-FID
CW-Density
CW-Coverage
Differencus
0.025
0.0
0.000
5 7
2
5
6
Sorted classes
Sorted classes
Sorted classes
(d) CIFAR- 10 with 40% asymmetric noise
Figure 11: The sorted differences between the DSM (baseline) and TDSM (ours) on the CIFAR-10 dataset with various noise settings.
2.6
30
CW-FID
Time
2.5
CW-FID
25
Time
244
2.3
0.001
0.005
0.01
0.05
0.1
Figure 12: Ablation study of the skip threshold 7.
Nevertheless, is possible evaluate the conditional metrics with the annotated labels. The results are shown Table 12 and we find that our model further improves the intra-class sample quality.
To see the impact of the noise rate of TDSM on the clean dataset, train TDSM models the clean CIFAR-10 10 dataset assuming different symmetric noise rates. As shown in Table the model performs best noise rate of 5%, In addition, most TDSM models show improvements compared to
31

---

Published as a conference paper at ICLR 2024
Table 10: Experimental results of combining with the noisy label corrector, VolMinNet (Li al. 2021 on the CIFAR-10 and CIFAR- 100 datasets under various noise settings.
Symmetric
Asymmetric
Clean
Metric
20%
20%
40%
DSM
TDSM
DSM
TDSM
DSM
TDSM
DSM
TDSM
DSM
FID
(L)
1.94
2.04
1.97
2.18
1.92
2.03
1.95
1.98
1.92
IS
(1)
9.90
10.02
9.90
10.09
9.98
9.98
9.96
9,99
10.03
CIFAR-I1 10
Density
(f)
101.32
104.83
103.26
110.09
102.15
103.78
102.33
102.98
103.08
Coverage
(f)
81.25
82.04
81.45
82.30
81.65
81.79
81.67
81.98
81.90
CW-FID
(
10.57
10.56
11.42
10.88
10.33
10.26
10.62
10.61
10.23
cond
(
75.85
75.96
73.32
73.38
75.63
76.27
76.53
CW-Density
99.92
103.61
99.03
107.22
100.86
102.62
100.75
101.37
102.63
CW-Coverage
80.65
81.48
80.25
81.46
81.09
81.31
80.94
81.10
81.57
FID
(
2.62
3.79
2.85
3.76
2.51
3.74
3.24
IS
12.53
12.63
12,46
12.70
12.81
12.90
12.52
12.86
12.80
CIFAR- 100
un
Density
87.20
87.68
85.01
91.76
86.43
88.63
84.77
87.97
87.98
Coverage
( 77.27
77.29
76.40
76.55
77.58
78.31
76.67
77.76
77.63
CW-FFID
(L) 68.69
67.96
72.17
71.18
67.43
66.96
75.05
73.06
66.97
cond
CAS
(t) 36.08
38.38
35.32
36.47
39.64
40.46
38.84
40.33
39.50
CW-Density
(T) 79.73
80.11
74.58
82.39
80.72
82.74
77.50
80.65
82.58
CW-Coverage
(1) 75.04
75.51
72.65
73.24
75.34
76.12
72.68
73.42
75.78
Table 11: Experimental results of combining with the noisy label corrector, DISC (Li et al. the CIFAR-100 datasets under various noise settings.
2023 on
Symmetric
Asymmetric
Clean
Metric
20%%
40%
20%
40%
0%
DSM
TDSM
DSM
TDSM
DSM
TDSM
DSM
TDSM
DSM
FID
(1)
2.47
2.65
2.54
2.84
2.51
2.45
4.00
3.41
2.51
IS
(f)
12.69
12.83
12.80
12.94
12.75
12.96
12.51
12.83
12.80
CIFAR-11 100
Density
(1)
87.89
89.91
87.28
90.20
87.79
90.52
83.65
88.10
87.98
Coverage
(1)
77.61
78.08
77.44
77.63
77.62
78.42
75.94
77.57
77.63
CW-FID
(1) 67.10
66.82
67.52
67.33
67.32
66.68
78.93
66.97
CAS
(1) 39.19
41.83
42.15
42.39
41.40
41.90
39.60
39,50
CW-Density
(1) 82.62
84.94
82.04
85.44
82.58
85.52
76.04
81.69
82.58
CW-Coverage
(1) 75.70
76.54
75.20
75.61
75.44
76.00
70.39
71.62
75.78
DSM. This suggests that the consideration of label transitions over timesteps contributes to improved performance even on clean datasets. However, it is important to note that additional research needed to account for the actual noise labels present in the benchmark dataset because experiments assume symmetric noise.
F.11 EXPERIMENTAL RESULTS ON SEVERE NOISE RATE
To verify the robustness of the proposed model under extreme noise, perform the experiments the CIFAR-10 dataset with severe symmetric noise rates 60% and 80%. The results are presented in Table [4 These results show that TDSM consistently improves performance even at severe noise rates.
F.12 FURTHER ANALYSIS OF THE TRANSITION-AWAEE WEIGHTED FUNCTION
In this subsection, we clarify the meaning of the transition-awaar weighted function Label recovery an intuitive way to overcome the problem noisy labels in a dataset. Our method does not specifically focus robustness to noisy labels through label recovery; instead, it focuses robustness to noisy labels from a diffusion model training perspective. Consequently. our approach be synergistically combined with existing label recovery methods to further improve performance.
32

---

Published as a conference paper at ICLR 2024
Table 12: Full experimental results on the clean MNIST, CIFAR-IO, and CIFAR-100 dataset.
MNIST
CIFAR-10
CIFAR-100
Metric
Base
Ours
Base
Ours
Base
Ours
FID
(L)
1.92
1.91
2.51
2.67
IS
(f)
10.03
10.10
12.80
12.85
Density
(f)
86.20
88.08
103.08
104.35
87.98
90.04
Coverage
(f) 82.90
83.69
81.90
82.07
77.63
78.28
CW-FID
(L)
10.23
10.18
66.97
66.68
cond
CAS
(1) 98.55
98.50
77.74
77.07
39.50
39.10
CW-Density
(1) 85.79
87.96
102.63
103.69
82.58
84.96
EW-Coverage
(1) 82.09
82.98
81.57
81.88
75.78
76.53
Table 13: Experimental results on the clean CIFAR-10 dataset varying the noise rate parameters in TDSM. The percentage in the header indicates the assumed noise rate.
TDSM
Metric
DSM
1%
2.5%
5%
7.5%
10%
FID
1.92
1.94
1.93
1.91
2.02
2.00
IS
(t)
10.03
10.08
10.04
10.10
10.04
10.06
Density
(1)
103.08
103.83
104.07
104.35
105.44
105.15
Coverage
(1)
81.90
81.79
81.90
82.07
82.04
81.95
is important to analyze how our method overcomes the noisy label robustness in diffusion mod- els. Our analysis indicates that proposed TDSM objective provides the diffusion model with information about the diffused label noise in the dataset. This information is represented by the transition-awanl weight function w, depending upon the diffusion time. Therefore, estimate the instance- and time-dependent label transition probability with the transition-awee
weight function.
We want emphasize that the transition-awann weight function w plays a different role from typical classifier. This probability represents the relationship between the noisy and clean label conditional scores (which are also imeedeeendent) and is an important element the proposed training objective of diffusion model. Therefore, we need to estimate this label transition probabilities over diffusion timesteps.
Specifically, our transition-awart weight function can be reformulated as follows:
wi(x,, D. v, Pr(Y y/x, ) ( - 8xx,)
v)
(51)
As seen in the expression, the weight function is composed of the clean label classifier Py y/xx), label transition prior p(Ý y y). and the noisy label classifier p.(Ý z ()))) Given the triplet of perturbed data instance and its corresponding noisy labels and timestep, i.c., (x,, f). each component of has the following characteristics with respect to y: 1) p(Y 3 xxx) is maximized when y the clean label of the clean data Xg of x,; 2) p(Y = r y) maximized when y the noisy label ý in general. The reason for 2) is that in general, a given noisy label is sufficiently likely be clean label. These two trade-offs imply that the w function does not behave like a clean label classifier.
Furthermore, for large enough t, the distribution of x, converges to a abbel-independent prior distribution by the design of the diffusion process, so that Px (Y |x:) converges to uniform distribution. Therefore, for sufficiently large f, the w function converges to a transition matrix S. This phenomenon also be demonstrated in Figure 2 of the 2-D Gaussian mixture model example. In summary, the transition aware weight function needed to overcome noisy labels a diffusion model is not represented clean label recovery information only, and this function has time-varying information.
33

---

Published as a conference paper at ICLR 2024
Table 14: Experimental results on the CIFAR-I0 datasets with symmetric noise containing severe noise rates. The percentages in headers represent the noise rate. "un' and 'cond' indicate whether metric unconditional or conditional. Bold numbers indicate better performance.
20%
40%
60%
80%
Metric
DSM
TDSM
DSM
TDSM
DSM
TDSM
DSM
TDSM
FID
(L)
2.00
2.06
2.07
2.43
2.17
2.55
2.22
2.44
IS
(t)
9.91
9.97
9.83
9.96
9.75
10.09
9.73
9.79
Density
(1) 100.03
106.13
100.94
111.63
102.17
110.78
102.09
103.69
Coverage
(
81.13
81.89
80.93
82.03
81.00
81.65
80.81
81.15
CW-FID
(L) 16.21
12.16
30.45
15.92
48.94
23.36
72.51
53.57
cond
CAS
(1) 66.80
70.92
47.21
62.28
29.72
52.38
9.93
24.68
CW-Density
88.45
99.52
73.02
97.80
58.42
86.58
43.97
54.30
CW- Cw-CCverage
77.80
80.29
71.63
78.65
61.88
75.08
44.53
56.47
1.0
0.8
Clean label uccuracy
0.6
Noisy labeled
0.4
VolMinNet
Base (DSM)
[CW-FID- FI.-F003444)
0.2
Ours (TDSM)
(CW-FID-15.92)
Base (VolMinNer DSM)
[CW-FID-11.4
(VolMinNet TDSM) [CW-FID-10.8])
0.0
10"
10
10
10
Diffusion timesteps (log scale)
Figure 13: The clean label accuracy of the transition-awal weight function diffusion timesteps on the CIFAR-10 training datasets under 40% symmetric noise.
It is possible check the accuracy of the transition-aware weight function the training dataset since the function provides the predicted probability of a clean label given a noisy label and an instance. Figure 13 shows the clean label accuracy of the transition-aware weight function over diffusion timesteps. Since the noisy labels in the dataset are used as the input conditions of the baseline model for the entire diffusion timesteps, we plot the percentages of clean labels in the datasets as dots at / 0 and plot a dashed horizontal line for comparison.
Focusing on 'Ours (VolMinNet + TDSM)', which provides the best generation performance, we found that the behavior of the w. function varies with diffusion timesteps, as mentioned above. In particular, as t increases, the clean label accuracy converges to the clean rate of the given dataset, which is 0.6. This because the w function converges to the label transition prior p(Ý = = = y)- 'Base (VolMinNet DSM)', which does not take this into account, learns the diffusion model only with corrected labels, leading to the performance difference. Therefore, while existing noisy label methods that focus on clean label recovery certainly contribute significantly to generative perfor- mance, considering the TDSM objective in diffusion model training enables additional performance improvement independent of clean label recovery.
F.13 ANALYSIS OF THE NOISY LABEL CLASSIFIER
To investigate the effect of the noisy label classifier, train the diffusion models with different training levels of noisy label classifiers. To analyze the noisy label classifiers, we measured 1) the noisy label classification performance of the noisy label classifier over diffusion timesteps; and the
34

---

Published as a conference paper at ICLR 2024
Noisy label accuracy of label classifier
1.0
1.0
Epochs
Clean label accuracy ( function
75
0.8
125
0.6
0.6
4000
Epochs
25
0.4
0.4
75
0.2
0.2
125
4000
0.0
0.0
10
10
10"
10'
10a
10
10
10"
10
10a
Diffusion timesteps (log scale)
Diffusion timesteps (log scale)
(a) Noisy label accuracy of noisy label classifier
(b) Clean label accuracy of w function
Figure 14: Classification performance of the noisy label classifier with varying training levels over diffusion timesteps on the CIFAR-10 dataset with 40% symmetric noise. The numbers in the legend represent the training epochs of the noisy label classifier.
10
DSM
10.2
Density (1)
120
Coverage (f)
83
10.0
82
FID
5
TDSM
110
9.8
81
9.6
80
125
Classifier training epochs
Classifier training epochs
Classifier training epochs
Classifier training epochs
70
CW -D--Dittty (t)
120
WW-Covenge (?)
85
CW-FID (
CAS
80
60
100
20
75
50
10
70
75 100 125 200 4000
75 100 125 200 4000 Classifier training epochs
100 125 200 4000
75
100 125 200 4000
Classifier training epochs
Classifier training epochs
Classifier training epochs
Figure 15: Generation performance of TDSM models on the CIFAR-10 dataset with 40% symmetric noise with respect to the training epochs of the noisy label classifier. DSM is independent of the classifier, but is shown horizontal line for comparison.
clean label classification performance of the w function, evaluated with the classifier, over diffusion timesteps, in Figure 14] We also compare the generation performance for the diffusion model trained with each classifier in Figure 15 and Table 15
The performance of the noisy label classifier in predicting noisy labels increased with the number of epochs trained on the classifier, but its ability predict clean labels in clean data space decreased. Meanwhile, in terms of the generation performance of the diffusion model, the less trained classifier performed better with respect to the conditional metrics, and the more trained classifier performed better with respect to the unconditional metrics. Note that all TDSM models outperform the baseline. DSM, on most metrics.
We interpret this result follows. 1) Due to the iterative sampling procedure of the diffusion model, the noisy label classifier needs to learn enough all the perturbed samples the diffusion timesteps to improve the quality of the generated samples. 2) At this time, excessive training of the noisy label classifier leads to an overconfidence problem, particularly in the data space at t = 0. This overconfidence reduces the information of the clean label and affects the information of the condition.
35

---

Published as a conference paper at ICLR 2024
Table 15: Generation performance of TDSM models with different training levels of noisy label classifiers on the CIFAR-10 10 dataset with 40% symmetric noise. The epochs represent the training epochs of the noisy label classifier.
TDSM
Metric
DSM
75 epochs
100 epochs
125 epochs
200 epochs
4000 epochs
FID
2.07
2.84
2.55
2.57
2.51
2.43
IS
9.83
9.72
9.90
9,93
9.95
9.96
5
Density
100.94
110,92
111.40
111.43
111.48
111.63
Coverage
())
80.93
81.23
81.84
81.98
81,95
82.03
CW-FID
())
30.45
14.63
14.06
14.70
14.58
15.92
cond
CAS
47.21
65.70
65.57
64.95
63.74
62.28
CW-Density
73.02
102,43
104.54
103.65
102,03
97.80
CW-Coverage
71.63
79.07
80.26
79.61
79.57
78.65
Table 16:
Generation performance of TDSM models for different assumed noise rates the CIFAR 10 dataset with 40% symmetric noise. The percentage in parentheses refers to the noise rate assumed by the TDSM models.
Metric
DSM
TDSM (40%)
TDSM (43%)
FID
(L) 2.07
2.43
2.53
IS
(f) 9.83
9.96
9.96
5
Density
(f) 100.94
111.63
113.21
Coverage
(t)
80.93
82.03
82.18
CW-FID
(L) 30.45
15.92
15.76
cond
CAS
(f) 47.21
62.28
62.10
CW-Density
(f) 73.02
97.80
100.06
CW-Coverage
(T) 71.63
78.65
79.11
Therefore, it is possible to improve the generation performance by training the noisy label classifier more effectively, e.g., by adjusting the temporal weight function A(t), and further work is needed.
F.14 ANALYSIS OF THE POTENTIAL LABEL NOISE THE BENCHMARK DATASET
we discuss the presence of noise in the annotated labels of existing benchmark datasets. This could potentially affect the results in our experiments on benchamrk dataset with synthetic label noise. To assess this influence, we apply TDSM to the CIFAR- 10 dataset with 40% symmetric noise, assuming a 43% noise rate. In this case, we assume that the 60% of unaffected data our noisy label generation process has the 5% potential noise predicted by the previous experiment, resulting in an additional 3% total data.
In Section 4.2
Table 16 shows the results of the TDSM model assuming 43% noise rate on the CIFAR-10 dataset with 40% symmetric noise. Interestingly, we find that assuming potential label noise yields additional performance improvements for most metrics. This result further supports our conjecture that the benchmark dataset contains examples with noisy or ambiguous labels.
F.15 ADDITIONAL GENERATED IMAGES
Figures 16 to 21 provides the uncurated generated images of the baseline and our models Table As with the quantitative results, models demonstrate the ability generate images that closely match the given conditions.
16 GENERATED IMAGES ON CLOTHING- -M
Figure 22 contains the uncurated generated images of the baseline and our models, trained on the Clothing- MM dataset. Although model generates images that exhibit some alignment with
36

---

Published as a conference paper at ICLR 2024
the specified conditions, the disparity is not significant. This observation can be attributed to the underlying assumption of class-conditional label noise. Consequently, it is evident that further research is necessary in the area of generative model learning from instance-dependent noisy labels, similar to the research in supervised learning dealing with such noisy labels.
37

---

Published as a conference paper at ICLR 2024
0
2
0
0
0
0
0
3
8
O
2
3
/
O
3
2
1
/
9
1
6
2
2
/
O
A
2
2
2
8
2
2
2
2
2
2
3
2
2
8
θ
2
2
>
2
2
2
2
3
2
2
2
2
2
2
2
2
2
3
O
>
3
3
3
3
3
8
3
3
3
3
3
3
3
3
3
3
3
O
Y
8
3
3
5
3
3
5
3
3
3
3
3
3
3
3
3
3
0
4
L
Q
4
4
4
3
4
4
4
4
4
4
4
4
4
4
4
N
5
4
4
4
5
4
4
4
4
H
4
4
4
4
4
4
4
4
7
5
S
5
5
5
5 5
5
5
5
5
5
5
5
5
5 5
6
5
7
5
S
J
5
5
O
6
5
5
5
S
5
5
5
S
6
6
7
9
6
6
O
6
6
5 6
6
b
6
6
6
6
6 6 6
3
O
6
4
6
6
9
6
6
6
6
6
6
6
6
6
6
6
7
O
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
1
7
6
O
7
7
7
7
9
7
7
7
7
7
7
7
7
7
9
8
8
0
8
9
8
2
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
]
&
7
8
4
8
8
8
8
8
8
8
8
8
8
9
9
9
9
/
9
4
9
9
9
9
9
9
9
4
9
9
9
5
9
0
9
9
9
1
9
9
9
9
7
9
9
(a) Base
(b) Ours
Figure 16: The uncurated generated images of (a) baseline and (b) models the MNIST dataset
with 40% symmetric noise. Each block has images of the same class. The class labels are 0, 1. 2, 3, 4, 5, 6, 8, and 9, from top bottom.
38

---

Published as a conference paper at ICLR 2024
2
2
2
2
2
2
P
2
2
2
2
2
1
2
22
2
2
2
2
2
2
2
2
2.
2
2
2
2
2
2
2
2
2
2
2
7
2
2
2
2
3
3
3
3
3
3
3
3
3
3
3
9
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
4
4
4
4
4
4
4
4
4
4
4 4
4
4
4
4
4
4
4
4 4
4
4
4
4
4
4
4
4
4.
Y
4
4
4
4
4
4
4
4
4
6
6
6
5
C
5
5
6
6
5 6
S
5
5
S
5
5
S
5 5 5 5
5
5
6
6
5
J
5
5
5
5
S
S
5
5
5
G
6
5
5
6
5
6
6
6 6
5
6
6
6
6
6
6
6
6
6 6
6 6
5
5
6
5
6
6
5
6
6
6
6
6
5
6
7
2
7
7
1
7
7
7
7
7
7
7
7
7
7
7
7 7
7
7
7
7
2
7
7
2
7
7
7
7
7
7
7
7
6
8
8
8
8
3
8
8
3
3
8
8
8
3
8
8
8
8
8
8
8
8
9
8
7
8
8
8
8
8
&
8
8
8
8
8
8
9
a
9
9
9
9
9
9
9
G
9
4
9
9 9
9
9
9
9
9
9
9
5
9
9
9
9
9
9
(a) Base
(b) Ours
Figure 17: The uncurated generated images of (a) baseline and (b) models the MNIST dataset
with 40% asymmetric noise. Each block has images of the same class. The class labels are 0, 1, 2, 3, 4, 5, 6, 8, and 9, from top bottom. The labels flipped by 7, and 6.
39

---

Published as a conference paper at ICLR 2024
(a) Base
(b) Ours
Figure 18: The uncurated generated images of (a) baseline and (b) our models on the CIFAR-10 10 dataset with 40% symmetric noise. Each block has images of the same class. The class labels are ai airplane, aut omobi le, bird, cat, deer, dog, frog, horse, ship, and ruck,, from top to bottom.
40

---

Published as a conference paper at ICLR 2024
Base
(b) Ours
Figure 19: The uncurated generated images of (a) baseline and (b) our models on the CIFAR-10 dataset with 40% asymmetric noise. Each block has images of the same class. The class labels are airplane, automobile bird, cat, deer, dog, frog, horse, ship, and truck, from top bottom. The labels are flipped by truck + automobile, bird airplane, deer + horse, cat t> dog.
41

---

Published as a conference paper at ICLR 2024
COOL
(a) Base
(b) Ours
Figure 20: The uncurated generated images of (a) baseline and (b) our models on the CIFAR-100 dataset with 40% symmetric noise. Each block has images of the same class. The class labels are apple, aquarium fish, baby, bear, beaver, bed, bee, beetle, bicycle, and bottle, from top to bottom.
42

---

Published as a conference paper at ICLR 2024
Base
(b) Ours
The uncurated generated images of (a) baseline and (b) our models on the CIFAR-100 dataset with 40% asymmetric noise. Each block has images of the same class. The class labels are apple, aquariun im fish, baby, bear, beaver, bed, bee, beetle, le, bicycle, and bottle, from top bottom. For these labels, the labels are flipped to mushroom. flatfish, boy, leopard, dolphin, chair, beet le, butterfly bus, and bowl, respectively.
Figure 21:
43

---

Published as a conference paper at ICLR 2024
(a)
(b) Ours
Figure 22: The uncurated generated images of (a) baseline and (b) our models on the Clothing- 1M dataset. Each block has images of the same class. The class labels are T-shi rt, Shirt, Knitwear, Chi iniffon, Sweater, Hoodie, Windbreaker, Jacket, Down Coat, Suit, Shawl, Dress, Vest, and Underwear, from top to bottom.
44


Reviews:
[{"review_id": "ELc61olWVH", "reviewer": "Reviewer_BRfo", "summary": "the authors find the noisy-label conditional score can be expressed as a convex combination of the clean- label conditional scores with some coefficients\n, accordingly they propose a weighted loss function to address the problem of noisy labels in class-conditional diffusion models.", "questions": "Please refer to my comments.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper is well organized and the proofs are detailed.\n2.\tThis paper is the first work to consider the influence of noisy label condition to the generation performance in diffusion models", "weaknesses": "The meaning of this work is limited. The proposed method is tailored for diffusion models which are conditioned on class, but most existing diffusion models are conditioned on text or other modalities, and the class label also can be expressed by language. In addition, noisy label datasets are not common in diffusion model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bw59POGCWn", "reviewer": "Reviewer_Zgz7", "summary": "This paper introduces a new label-noise robust method for training conditional diffusion models. This is achieved by making use of an estimated transition relation from noisy labels to clean labels. Some theoretical analyses have also been proposed under the class-dependent label-noise setting. Experiments across various datasets demonstrate the effectiveness of the proposed method.", "questions": "1. Could the authors elaborate on how their method of learning the transition relation from noisy labels to clean labels differs from or aligns with the existing literature in mixture proportion estimation?\n2. Could the authors provide further details on how their method more effectively recovers clean labels compared to existing state-of-the-art methods VolMinNet (ICML 21), InstanceGM (WACV23), and DivideMix (ICLR20)?\n3. Would the authors consider adding additional experiments to demonstrate the performance when combining the existing state-of-the-art methods for learning with noisy labels with a conditional diffusion model? Additionally, could they compare the accuracy of clean label recovery with these methods?\n4. Could the authors explain which specific mechanisms or features in their approach contribute to improved label recovery?\n5. How do the authors verify the presence of class-dependent noise in practical applications?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **Targeting an Important Problem:** The paper aims to handle noisy labels in large-scale datasets used for training diffusion models. Addressing this problem is crucial as it is a common and practical challenge in the deployment of these models in real-world scenarios.\n2. **A New Approach to Noisy Labels in Diffusion Models:** The paper introduces a new methodology to address noisy labels in conditional diffusion models, a topic not extensively covered in existing literature.\n3. **Clarity and Accessibility:** The introduction and overall presentation of the paper are clear and easy to follow.", "weaknesses": "1. **Missing Citations:** The paper could be significantly enhanced by including additional relevant literature. Learning the transition relation from noisy labels to clean labels has been previously explored under the umbrella of mixture proportion estimation. Moreover, methods for estimating the transition matrix in class-dependent settings are also very related. Acknowledging popular works could provide a richer theoretical foundation.\n2. **Missing Baselines:** There are many methods for learning with noisy labels. It would be beneficial to combine existing state-of-the-art (SOTA) methods for learning with noisy labels to obtain estimated clean labels. Then, utilizing conditional diffusion models on these cleaned labels and comparing the performance with the author's method could offer a more comprehensive evaluation of the proposed method’s efficacy compared to current alternatives.\n3. **Unclear Advantage:** There is a need for a more detailed explanation regarding the advantage of the proposed method over existing methods, especially in recovering clean labels. Since the performance of diffusion models in noisy label scenarios hinges significantly on the accuracy of label recovery, explaining how the method enhances this aspect compared to others would greatly benefit readers in understanding the true potential and innovation of your approach.\n4. **Limited Application:** The focus on class-dependent noise settings might limit the broader applicability of the proposed method. The assumption of class-dependent noise could be strong and not verifiable in practical scenarios. It would be insightful if the paper could discuss the potential implications and limitations of this assumption, including how it might affect the generalizability of the proposed method to other noise settings or real-world applications where such assumptions may not be easily verified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jEXKzGdvwi", "reviewer": "Reviewer_TMN7", "summary": "This paper highlights the challenges associated with training on extensive datasets, which often contain noise in their condition to make noisy labels. Such noise introduces the risk of condition mismatches, which can degrade the quality of the generated data. To tackle this issue, the paper presents the Transition-aware weighted Denoising Score Matching (TDSM) method. This approach is specifically designed to robustly train conditional diffusion models with noisy labels.\n\nThe TDSM framework incorporates a label-transition weight for the score networks. These weights are derived from the relationship between conditional scores for both noisy and genuine labels, and can be estimated with a pre-trained noisy-classifier. Empirical evaluations, on multiple datasets and a range of noisy label configurations, demonstrate the efficiency of the TDSM approach.", "questions": "Please see the Weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and polished, facilitating a smooth reading experience. The mathematical presentations are articulated clearly and the theoretical results are complete and sound. Additionally, the inclusion of model overviews and illustrative figures for the components simplifies the understanding of the proposed method. \n\n- The experimental results well validate the approach. Notably, the paper comprehensively study both the effects of the conditional models and the impact of conditional generation with label guidance. Comprehensive results are provided in both the main paper and supplementary material.\n\n- The practical side of the research is solid. The authors have been very detailed in their implementation and provided their experiment code, which ensures reproducibility.", "weaknesses": "- Given that the estimation of the transition-aware weight relies on a noisy-classifier, it would be advantageous for the authors to present studies evaluating how the performance of the noisy-classifier affects the model's overall performance.\n\n- In Table 2, the authors seem to only compare with DSM with non-class-aware evaluation metrics. What's the reason for this comparison with the specific metrics?\n\n- On \"clean\" datasets, TDSM demonstrates notably superior performance, suggesting the potential presence of noisy labels. It would be insightful to know the threshold or proportion of noisy labels at which a significant performance difference emerges between DSM and TDSM. Furthermore, if the datasets used to train the noisy-classifier (for estimating class transitions) contain noisy labels, would this introduce additional inaccuracies in label correction? It would be beneficial for the authors to conduct a thorough analysis of these concerns.\n\n- In the review of diffusion models, the authors seem to only review from the score matching networks, while omit the diffusion models derivated from optmizing the ELBO.\n\n\n- Previous works are not correctly reviewed or cited. For example, the reference of denoising diffusion probabilistic model (Ho et al., 2020) is classified into video generation, in the introduction, while this is a fundamental work in diffusion models, and the authors may wanted to put video diffusion models (Ho et al., 2022) there. Moreover, some prior works that tackle the uncurated label distributions are not discussed and compared in the paper. \n\n- It would be beneficial to include a color bar in Fig 8 to interprete the meaning of the colors presented.\n\n\n*Reference*:\n\n*Ho, Jonathan, Ajay Jain, and Pieter Abbeel. \"Denoising diffusion probabilistic models.\" Advances in neural information processing systems 33 (2020): 6840-6851.*\n\n*Rangwani, Harsh, Konda Reddy Mopuri, and R. Venkatesh Babu. \"Class balancing gan with a classifier in the loop.\" Uncertainty in Artificial Intelligence. PMLR, 2021.*\n\n*Rangwani, Harsh, et al. \"Improving GANs for Long-Tailed Data Through Group Spectral Regularization.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.*\n\n*Ho, Jonathan et al. “Video Diffusion Models.” ArXiv abs/2204.03458 (2022).*\n\n*Qin, Yiming, et al. \"Class-Balancing Diffusion Models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.*", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PV87pIXz8w", "reviewer": "Reviewer_LRz9", "summary": "This paper proposes a novel method, namely Transition-aware weighted Denoising Score Matching (TDSM), to train conditional diffusion models with noisy labels. The TDSM objective contains a weighted sum of score networks. Additionally, it also introduces a transition-aware weight estimator to leverage a time-dependent noisy-label classifier distinctively customized to the diffusion process. Experimental results on multiple popular datasets demonstrate the effectiveness of the proposed method.", "questions": "1. Why boosting the robustness of diffusion models on noisy labels is very significant?\n2. Can the proposed method boost the model classification performance on noisy labels? \n3. What the limitations of the proposed method and please point out the future work.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written with clear method procedure.\n2. The idea is clear and effective.\n3. This paper have good experimental results.", "weaknesses": "1. It is not clear what is the major difference from the methods that boosting the robustness of generative models on noisy labels.\n2. The significance of this research topic is not clear, please explain it. Specifically, to the best of my knowledge, generative models are usually unsupervised, and thus there is only a few methods on boosting the model robustness against on noisy labels.\n3. It is not clear the model performance on severe noisy labels, like 60%, 80%.\n4. It is not clear whether the proposed method can boost the model classification performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)