Slight Corruption in Pre-training Data Makes Better Diffusion Models
Authors: Hao Chen
Yujin Han
Diganta Misra
Xiang Li
Kai Hu
Difan Zou
Masashi Sugiyama
Jindong Wang
Bhiksha Raj
Venue: NeurIPS 2024 spotlight (2024)
Keywords: pre-training noise
diffusion models
latent diffusion models
diffusion transformers
latent consistency models
Primary area: diffusion_based_models

Abstract: Diffusion models (DMs) have shown remarkable capabilities in generating realistic high-quality images, audios, and videos. 
They benefit significantly from extensive pre-training on large-scale datasets, including web-crawled data with paired data and conditions, such as image-text and image-class pairs.
Despite rigorous filtering, these pre-training datasets often inevitably contain corrupted pairs where conditions do not accurately describe the data. 
This paper presents the first comprehensive study on the impact of such corruption in pre-training data of DMs.
We synthetically corrupt ImageNet-1K and CC3M to pre-train and evaluate over $50$ conditional DMs. 
Our empirical findings reveal that various types of slight corruption in pre-training can significantly enhance the quality, diversity, and fidelity of the generated images across different DMs, both during pre-training and downstream adaptation stages. 
Theoretically, we consider a Gaussian mixture model and prove that slight corruption in the condition leads to higher entropy and a reduced 2-Wasserstein distance to the ground truth of the data distribution generated by the corruptly trained DMs.
Inspired by our analysis, we propose a simple method to improve the training of DMs on practical datasets by adding condition embedding perturbations (CEP).
CEP significantly improves the performance of various DMs in both pre-training and downstream tasks.
We hope that our study provides new insights into understanding the data and pre-training processes of DMs.

Slight Corruption in Pre-training Data Makes Better Diffusion Models
Hao Chen'; Yujin Han Diganta Misra Xiang Li Kai Hu? Difan Zou?, Masashi Sugiyama 4,5 Jindong Wang Bhiksha Raj
CCarnegie Mellon University. "The University of Hong Kong, Miila Quebee Al Institute, RIKEN AIP, Thhe University of Tokyo, "William & Mary, 'BBZUAI
Abstract
Diffusion models (DMs) have shown remarkable capabilities in generating realistic high-quality images, audios, and videos. They benefit significantly from extensive pre-training on large-scale datasets, including web-crawled data with paired data and conditions, such as image-text and image-class pairs. Despite rigorous filtering, these pre-training datasets often inevitably contain corrupted pairs where conditions do not accurately describe data. This paper presents the first comprehensive study on the impact of such condition corruption in pre-training data of DMs. We synthetically corrupt ImageNet-IK and CC3M to pre-train and evaluate over 50 conditional DMs. Our empirical findings reveal that various types of slight corruption in pre-training can significantly enhance the quality, diversity, and fidelity of the generated images across different DMs, both during pre-training and downstream adaptation stages. Theoretically, we consider a Gaussian mixture model and prove that slight corruption in the condition leads to higher entropy and reduced 2- W-Wassesstein distance to the ground truth of the data distribution generated by the corruptly trained DMs. Inspired by our analysis, we propose simple method to improve the training of DMs on practical datasets by adding condition embedding perturbations (CEP). CEP significantly improves the performance of various DMs in both pre-training and downstream tasks. We hope that our study provides new insights into understanding the data and pre-training processes of DMs and all models are released at https: /httts::///ugee /taps::://nggngaace... wtt.://aaaco/iifffssiiooooo.ooo.
1 Introduction
Recently, diffusion models (DMs) have been demonstrating unprecedented capabilities in generating high-quality, realistic, and faithful images (155), audios [6] 77] and videos (8). In addition, they exhibit impressive conditional generation results II when trained with classifier-free guidance
[12. The successes of DMs are often attributed to the massive pre-training on large-scale datasets consisting of paired data and conditions [13+17,, which also empowered and facilitated numerous downstream applications and personalization of pre-trained models, such as subject-driven generation
18 [9] controllable conditional generation (20 and synthetic data training (23/25).
The large-scale pre-training datasets of paired data and conditions are usually web-crawled. For example, Stable Diffusion [26] was pre-trained LAION-2B [17), which contains billion-scale image-text pairs collected from Common Crawl (27). Despite the heavy filtering mechanisms used in collecting pre-training datasets [1728] they still inevitably contain corrupted pairs where conditions
do not correctly describe or match the data, such as corrupted labels and texts [2]] While
large-scale datasets are necessary for DMs to perform well, the corruption may lead unexpected
haooc3 (eandrew.cmu.cd. Correspondence to: @wang&0@wm.edu
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

"A burcle bue yolow
"Pariing of yourg basper delicailyy
"An estronad cding "Allmma westro
Macaw
Samoyed
Catamaran
Valley
the forss surplasses Aug.
Aug.
14.91
96.64
Siigt
14.23 98.07
Sever
LDW-4 WM-IK
DI-XL22 NN-KK
LDM- CCBM
LCM.VL5
Figure 1: Visualization from class and text-conditional DMs pre-trained with clean, slight, and severe condition corruption. Slight corruption in pre-training improves the quality and diversity of images.
behavior or generalization performance of models 33 during both pre-training and adaptation stages, especially for safety-critical domains such as healthcare 36 and autonomous driving 377 38
Conventional wisdom may suggest that training under corrupted conditions could lead to deterio- ration in performance. For example, Noisy Label Learning 42
aims to improve 29 43
the generalization of models when training with corrupted labels. Label-noise robust conditional and DMs 4 have also been studied. However, these works primarily concerned with supervised learning in downstream scenarios with assumptions high noise ratios and the same training and testing data distributions. Due to the misalignment with large-scale self-supervised pre-training in practice on filtered datasets with relatively smaller noise ratios, the effects of corruption in pre-training can also differ from those in downstream the effects of pre-traie pre-training with such corruption is challenging and still remains largely unexplored.
generative adversarial nets 45 46
48 49) Understanding
In this paper, we provide the first comprehen- sive and practical study on condition corrup- tion in the pre-trane pre-training of DMs. Through in- depth analysis, we empirically, theoretically, and methodologically verify that slight condition corruption in pre-training makes better DMs. We pre-train over 50 class-conditional and text- conditional DMs using classifier-fres guidance (CFG) 122
-4-17M I XT. IE
-nd LTM a IN IX (Aayen) L.M CCCM CULAE
LU06- 02000
-H-L06 1 DEIM
150
52
100
on ImageNet- IK (IN-1K) [50] and CC3M with synthetically corrupted condi- tions, i.c., classes and texts, of various levels. Our study covers a wide range of DM families, including Latent Diffusion Model (LDM) (9) Diffusion Transformer (DiT) [11. and Latent Consistency Model (LCM) [ST
Corruption Ratio (96)
Conruption Ratio (%)
(a) FID
(b) IS (T)
Figure (a) FID and (b) IS of DMs pre pre-trained on IN-1K and CC3M with various corruption. Slight corruption of various types helps DMs achieve better performance, compared to the clean ones.
52) Due the known obstacles of evaluating generative mod- els 55. we conduct both pre-training and downstream evaluation from the perspectives of image quality, fidelity, diversity, complexity, and memorization, to comprehensively understand the effects of pre- pre-training corruption of DMs. More specifically, for pre-training, we directly evaluate the images generated from the pre-trained models, and for downstream adaptation, we evaluate the images generated using personalized models with ControlNet [20] and T2I-Adapter [2 from the pre-trained ones. In addition, theoretically investigate how slight corruption in conditional embeddings benefits the training and generative processes of DMs. Our key findings include:
Empirically, slight corruption in pre- pre-training facilitates the DMs to generate images with higher quality and more diversity, both qualitatively (in Fig. 1 and quantitatively (in Fig. 2
Theoretically, we employ Gaussian mixture model to show slight condition corruption improves the diversity and quality of generation by increasing entropy over clean condition generation and reducing the quadratic 2- Wasserstein distance to the true data distribution (in Section

---

Methodologically. based on our analysis, we propose a simple method to improve the pre- vre-training of DMs by adding conditional embedding perturbations (CEP). We show that CEP can significantly boost the performance of various DMs in both pre-training and downstream tasks (in Section Sp
Going beyond images, we do the potential of this study in other modalities. Our efforts may also inspire future investigation on other types of corruption and bias inside pre-training datasets. We hope that our work can shed light on the future research of diffusion models and responsible Al.
2 Preliminary
Denoising Diffusion Models. DMs are probabilistic models that learn the data distribution Pxx). with x denoting the observed data over a set of latent variables Z1 21..... Zr with length T [ 57
assumes forward diffusion process, gradually adding Gaussian noise to the data with a fixed Markov chain: g ((z N (Vax, (1 TI) which can be re- re-parameterized as z, V vI ā with e ~ (0, IJ and 0e as constants produced by noise scheduler. DMs are trained via the reverse process, inverting the forward process as: po po(2: (t)) = N (Jug (z+). En (2t)). with a network that predicts the statistics of ps- Setting Eg (z+) (1 or I to untrained constants, the reverse process simplified as training equally weighted denoising autoencoders (Z, 1) with uniformly sampled
LDM = Ex.e= Ex..c-N(0..),.000... [||e Eq (z:, 1)]3]
After training, new images be generated by sampling 2;-1 (Ze-1()zz) starting with (V I).
Classifier-free Guidance (CFG). Extra condition information y. such as class labels and text prompts, can be injected into DMs with conditional embeddings c=(y) from modality-specifie
encoders [9] for conditional generation: ps(z,-1 |e, co (y)). CFG [12] jointly learns unconditional model Eg (ze, L, (0)) with empty condition y 0 and conditional model 6ø(2,, (, cs cv)),, and combines them linearly to control the trade-off of sample quality and diversity in generation:
Eo(Ze, t, co (v)) Es (z, t, (0)) (eo(Ze, t, ce(v)) Es Ez(z, t, (0)))
(2)
where $ > -1 denotes the guidance scale. We adopt CFG by default with the training objective:
[Ilc- cg (ze, tz (o))))
LDM Ex.,
Condition Corruption. Ideally, each should accurately describe and match x. However, in practice, due to errors from the collection of web-crawled datasets, conditions yf may un-match We define (x, yf) as pairs with condition corruption, and assume that (o(")) ce (y: E), where denotes certain noise and 17 denotes corruption ratio that implicitly controls the noise magnitude.
3 Understanding the Pre-training Corruption in Diffusion Models
In this section, we conduct the first comprehensive and practical study on pre-training DMs with condition corruption. Through holistic exploration with synthetically corrupted datasets, we reveal a surprising observation that slight pre-training corruption can be beneficial for DMs.
3.1 Pre-training Evaluation
Pre-training Setup. Here, we adopt Latent Diffusion Models (LDMs) with the pre-trained
VQ-VAE [58,55 (6) and a down-sampling factor of 4 for the latent space of observed data x, denoted as LDM-4. More specifically, we train class-conditional and text-conditional LDM-4 from scratch on synthetically corrupted IN-IK [50] and CC3M [1], respectively. with resolution of 256 x 256 We use class embedding layer and a learnable pre-trained BERT [59] to compute the conditional embeddings of the IN-IK class labels and the CC3M text prompts. To introduce synthetic corruption
into the conditions, we randomly flip the class label into random class for IN-IK, and randomly swap the text of two sampled image-text pairs for CC3M, following 48 49) (other corruption types studied in Section 3.3). We train models with different corruption ratios 1) E (0, 2.5, 5, 7.5, 10, 15, 20 20%% More details on synthetic corruption and pre-training recipes are shown in Appendix B. and B.3
We use for images in both the pixel space and the latent space of VQ-VAE (56
3

---

Pre
07
F1D
12
05-
100
200
82
B4
15
Recal
Score
Recall
(a) FID IS, IN-IK (b) Prec. Recall, IN-IK
(c) FID- CS, CC3M
(d) Prec. Recall, CC3M
Figure 3: Quantitative evaluation of generated images from class and text-conditional LDMs
pre-traine pre-trained with condition corruption. All metrics are computed over 50K generated images and validation images of IN- and MS-COCO. We plot FID IS or CS ((a) and (c)), and Precision vs. Recall ((b) and (d)), where each point indicates the results computed from using a guidance scale. Models pre-trained with slight condition corruption achieve better FID, IS or CS, and PR trade-off.
Sauple Entrpyy
Deautty C.
Sumpl Entrupy
4.23
Deaatt
700
4.1
1.00
Top 19 TMD
Gridinee Stale
Top 1 RMD
Guidazoe Scale
(a) RMD, IN-IK
(b) Entropy, IN-IK
(c) RMD, CC3M
(d) Entropy, CC3M
Figure 4: Quantitative evaluation of complexity and diversity of class and text-conditional LDMs. We plot the top-11 1% RMD score ((a) and (c)) which measures the complexity and diversity of samples (with s 2.0 and 3.0 for IN-IK and CC3M LDMs), and the sample entropy ((b) and (d) as a proxy measure of diversity, where each point indicates the result of a guidance scale. Models pre-trained with slight condition corruption generate samples of higher complexity and diversity.
Evaluation of Pre-trained Models. We directly use the pre-trained L.MMs to generate images to study the effects of condition corruption in the pre-training stage. We use IN-IK class labels for class-conditional LDMs and MS-COCO text prompts [60] for text-conditional LDMs to generate 50K images and compare with the real validation images. The images are generated using set of guidance scales 8 E (1.5, ..,, 10.0} and DPM (61] scheduler with 50 steps for faster inference speed"] We adopt Fréchet Inception Distance (FID) [63]. Inception Score (IS) (64), Precision, and Recall [65] to evaluate the quality, fidelity, and coverage of the generated images. For CC3M models, use the CLIP score (CS) [66] to measure the similarity of the generated images and conditional text prompts. From the perspectives of sample complexity and diversity, we compute the top- Relative Mahalanobis Distance (RMD) 67 68), calculated from the estimated class-specifie and class-agnostic distributions of generated data, and the sample entropy [69] [70,, calculated from the VQ-VAE codebook. We also adopt other metrics, including sFID (71). TopPR Fl [72], average L2 distance, and memorization ratio 73 More details of the metrics used are shown in Appendix (B7
Results. We present the main quantitative results of pre-training in Fig. 3 and and the qualitative results in Fig. 55) More results are shown in Appendix C In summary, we found that slight pre- training corruption can facilitate the quality, fidelity, and diversity of generated images:
Class and text-conditional models pre-trained with slight corruption achieve significantly lower FID and higher IS and CLIP score (Fig. ((a) and ((c)). They also present comparable and better Precision- Recall curves (Fig, 3(b) and 3()),, compared to clean pre-trained models.
Models pre-trained with slight corruption generate images with higher complexity and diversity, with a sight-shifted density of RMD (Fig. 4(aa and ([)). and larger entropy (Fig. H[b)annd (d))
Qualitatively, models with slight corruption learn more diverse distribution, Generated images present better variability in the circular walk around the latent space (Fig. [(a) and 5(b).
'In Appendix we also present the results IN-IK LDM-4 using the DDIM 62 scheduler with 250 steps. 'llghh corruption corresponds to 1) 7.5%, which might be common in practical large-scale datasets.

---

Walk Learned Latert
Circuar Walk
Incressing Compution Retio
Incressing Computoo
Goldfish
with
Lorkeet
Aphotograph
applaaces"
(a) IN-IK
(b) CC3M
Figure 5: Qualitative evaluation of images generated from circular walk around the learned latent space using (a) classcconditional IN-IK LDMs and (b) text-conditional CC3M LDMs. Models pre-trained with slight condition corruption present more diversity in the learned distribution.
More corruption pre-training can potentially lead quality and diversity degradation. As increases, almost all metrics first improve and then degrade. However, the degraded measure with more corruption sometimes is still better than the clean ones (e.g. IS and Entropy).
3.2 Downstream Personalization Evaluation
common scenario of DMs pre-trained large-scale datasets is that they can be personalized and customized for more controllable generation [74 75) after tuning on smaller datasets. Here, we also examine the effects of pre-training condition corruption of DMs downstream personalization tasks.
Downstream Personalization Setup. We personalize the pre-trained LDMs with two common methods: ControlNet [20] and T21-Adapter (2I] Both methods enable the pre-trained DMs to generate more controllable images according to input spatial conditioning. For fair comparison, we automatically annotate the ImageNet-100 (IN-100) dataset canny edges using the OpenCV canny detector (76] and segmentation masks using SegmentAnythin
(SAM) (77), similar to Zhang et al. text prompts for IN0. We then fine-tune the LDMs on the annotated training set of IN-100. More details the annotation and personalization setup are shown in Appendix B.2 and B.6 respectively.
(20] For text-conditional LDMs, additionally use BLIP 78
generate MS-COCO-sIE
Evaluation Personalized Models. After tuning, we evaluate the personalized LDMs in annotated validation set of IN-100. We mainly compute FID, IS, Precision, and Recall to compare the models. We similarly use a set of guidance scales to generate images, but only report results of the best guidance scale in this part due to space limit. The complete results are shown in Appendix D
we found that slight pre-training corruption can also benefit the quality of generated images at downstream personalization:
Results. Similarly, from the main results in Fig. and Fig. 7
Slight pre-training corruption helps the personalized model generate images with lower FID (Fig. 6(a) and ((c) and higher score (Fig. 6(b) and 6(d)), whereas more corruption deteriorates.
Qualitatively, the personalization images from slight corruption pre-trained models also present more diversity, better fidelity with the input spatial controls, and higher quality.
3.3 Discussion: Other Types of Pre-training Corruption and Diffusion Models
Our previous studies mainly involve LDMs and symmetric random corruption. Here, we additionally study other types of corruption and DMs to verify that the above observations universally hold.
Condition Corruption. We consider asymmetric (Asym.) label corruption for IN-1K and Large Language Model (LLM) corruption for CC3M. For IN-IK, we introduce corruption only within the
5

---

Exet SHISS
Recs alas
Seer las
Aa
ahea
(a) FID, IN-IK
(b) iS, IN-IK
(c) FID, CC3M
(d) IS, CC3M
Figure 6: Quantitative evaluation of ControlNet and T2I-Adapter personalized class and text- conditional LDMs. FID ((a) and (c)) and IS ((b) and (d) are computed using the 5K generated images.
Slightly corrupted pre-trained models also present better performance in downstream personalization.
Increasing Compten Rado
Ircressing Compios Raio
ComtoNo
CormoRes
3AR
oll by on boanh in goos.
T3 MUSEE
up oiu
relan
SAH
(a) IN-IK
(b) CC3M
Figure 7: Qualitative evaluation of ControlNet and T21-Adapter (a) IN-1K and (b) CC3M LDMs.
overlapped classes with CIFAR-100 79,, while maintaining others as clean. For CC3M, we prompt GPT-4 80 to corrupt the texts. More details of the corruption are shown in Appendix B.T
Diffusion Models. LDMs utilize U-Net [8] as backbone and Cross- Attention for adding conditional
information (9] We pre-train class-conditional diffusion transformers on IN-1K for extra assessment, termed DiT-XL/2 [ with Transformer
[82] backbone and adaptive LayerNorm |6 for
conditional information. We also pre-train the recent text-conditional Latent Consistency Models (LCMs) 52 51 on CC3M, which distill Stable Diffusion v1.5 (6] models enable swift inference
with minimal steps, noted as LCM-v1.5. Detailed setup is shown in Appendices B.4 and B.5
Results. We present the main results in Fig. due to the space limit. Full results are shown in We find that slight condition corruption of various types universally facilitates the performance of different DMs and consistently makes them outperform the clean pre-trained ones.
Appendix C
4 Theoretical Analysis
In this section, we theoretically amalyze condition corruption and find that slight corruption prevents the generated distribution from collapsing to the empirical distribution of the training data and encourages coverage of the entire data space, thereby enhancing diversity and alignment with the ground truth. We present concise overview here and provide comprehensive analysis in Appendix A
Data Distribution. We concentrate on the prototypical problem of sampling from Gaussian mixture models (GMMs). Specifically, we consider the distribution of data IR that satisfies:
P(x) := Σ WoNN I).
(4)
DEY
where y denote class labels finite set y E (1...., D') Given any class, xly follows a Gaussian NH 1), and wy represents the weight of the Gaussian components which satisfies Eyey wy 1.

---

Denoising Networks and Condition Corruption. Inspired by recent works that also target on GMMs 87 88 of DMs, we parameterize the denoising network as piece-wise linear function:
DJ co(X,, 3 - E + V)c(1r))
(5)
where c(y/") the one-hot encoding of corrupted label y" and {W V are trainable parameters. Specifically, following a line of existing work [89|9]], we adopt a simpler label-noise model by adding Gaussian perturbation to the label embedding, perturbing the clean condition c(y) with standard Gaussian noise & to obtain c(v²) c(v) > Here, the corruption control parameter corresponds to the corruption ratio yj for more direct noise magnitude control. While our theoretical framework focuses on Gaussian noise, it can also be extended to distributions such as uniform.
4.1 Generation Diversity: Clean vs. Corrupted Conditions
We employ entropy to evaluate the diversity of generated images, following Wu et al. [70]. Higher entropy suggests wider spread of data, yielding greater diversity in generated images, while lower entropy implies the difference in entropy between generations with corrupted and clean conditions:
concentrated distribution with less diversity. We present Theorem showing
Theorem 1. For any class k E y and sufficiently large length T, assuming the norm of corresponding
expectation || is a constant and the empirical covariance of training data is full rank, let zr and 24 be the generation with clean and corrupted conditions respectively, then it holds that
Hizryy = k) - Hzrly = k) k)?Pd??
(6)
where y is the corruption control parameter and is the data dimension.
The proof is provided in Appendix A.4.1 Theorem | indicates that for any class k, corrupted
conditions enhance image diversity by increasing generation entropy. Moreover, with suitable values, image diversity can grow with noise, aligning with observations in Fig.
4.2 Generation Quality: Clean vs. Corrupted Conditions
We then analyze why corrupted conditions benefit the quality of generated images, as also observed in Section We employ the 2- 2-Wasserstein distance as metric to evaluate the sampling error between the true and the generated distributions, with clean and corrupted conditions. distributed generated closer to the real data distribution indicates better image quality (63). In Theorem 2 we analyze the difference in the quality of data generated by corrupted DMs and clean ones:
Theorem 2. For any k E y and sufficiently large length T, assuming the norm of corresponding expectation | 112 is constant, let IP, Qx and Qx be the ground truth, clean, and corrupted condition distributions over training data X. Then 7 = =O1// mmax, ngg), it holds that
W2 - @x)ly = = -52 -?(
Ex W (TP (x)
Th
where W2(+,)) denotes the 2- Wasserstein distance between two distributions, ng is the sample size of k-labeled dataset, and d is the data dimension.
Here the expectation is taken over the random sample of the training dataset from the data distribution. Detailed proof is shown in Appendix A.4.2 Theorem 2 reveals that for any class k, small corruption yields generation distributions closer to the true distribution than clean ones. This partially verifies that the generation quality of the uncorrputly trained diffusion model can be improved by adding slight corruption to the training data. This is also well consistent with our empirical observation in Section 3.1, where the noise we used approximately 0.04c, close the theoretical noise level of 0.03c, showing that the FID of the generated images can be improved with a small corruption.
5
Improving Diffusion Models with Conditional Embedding Perturbation
5.1
Method
Our previous analysis demonstrates that slight condition corruption in the pre-training could po- tentially benefit both the image quality and diversity of DMs, which inspires us to improve the
7

---

Table 1: Pre-trate Pre-training results of IN-IK and MS- Table 2: ControlNet personalization results of COCO using diffusion models pre-trained with per-
IN-100 using LDMs pre-trained with perturba- tion. CEP achieves the best results (in bold).
turbation. CEP achieves the best results (in bold).
Model
Perturt
FID (
18 (1)
Precision (1)
Recall (t)
Control
Perturb.
FID ())
IS (1)
Precision (f)
Recall (t)
LDM-4
IN-IK (s 2.0)
IP
9.18
9.44
141.77
138.46
0.71
0.67
0.43
0.43
IN-IK Canny
IP
12.31
11.59
57.39
57.01
0.82
0.77
0.59
0.61
CBP-U
7:00
170.73
0.73
0.45
2.25)
CEP-U
11.46
59.29
0.84
0:58
CEP-G
6.91
180.77
0.76
11.53
57.59
0.83
DIT-X/22 11
IN-IK (e 1.76)
CER-U
IP
6.76
5.51
6.75
179.67
182.28
189.94
0.74
0.75
0.77
0.46
0.45
IN-IK
SAM
CEP-U
12.95
13.74
13.61
54.52
55.13
56.68
0.79
0.75
0.79
0.50
0.49
0.48
CEP G
5.92
185.21
0.75
0.45
(a 2.20)
CEP-G
13.44
56.81
0.80
0.49
LDM-4 9
CC3M (8
CEP-U
IP
19.48
17.93
30.09
30.77
30:17
0.61
0.59
0.65
0.42
0.41
CC3M
Canny
CEP-U
IP
40. 12
40.65
35.91
33.86
32.43
0.71
0.62
0.51
0.52
CEP-G
18.59
30.50
34.57
36.59
0.68
LCM.vL5 52
23.59
39.15
0.67
0.35
CC3M
42.64
32,00
0.63
0.51
CCSM
1.5)
CEP-U CFP-G
23.40
23.63
22.91
40.07
40.31
40:12
0.65
0.68
0.67
0.36
0.35
(e 1.0)
SAM
CEP-U CEP-G
38.00
35,02
43.79
35.77
32,98
0.64
0.67
0.67
0.53
0.51
pre-training of DMs using this conclusion. In practice, it is usually infeasible to directly corrupt the conditions in the pre-training datasets either due to their large-scale nature or difficulties to select which conditions to corrupt. Instead, we propose to add the perturbation directly to the conditional embeddings of DMs, which is termed conditional embedding perturbation (CEP). Compared the fixed proportion of condition corruption in datasets we studied before, CEP adds perturbation to every data instance during training on the fly. Specifically, CEP slightly modifies the DM objective:
LDM Ex,..e-W((o., ||ee eg (x+, L, co(v) + 0)12]]
(8)
where 8 denotes the perturbation added to conditional embeddings co (v). We simply set the perturba- tion to Uniform, i.e., 8 ~ L (- ) to Gaussian, i.e., 8 ~~ N (o, 3a where the design d denotes the dimension of co (3), and Y controls the perturbation magnitude, mimicing the corruption ratio 1). The main purpose of CEP is to learn better DMs with perturbation on relatively clean and heavily filtered datasets, such as CC3M and IN- 1N studied in this paper, but it is also applicable to slightly corrupted datasets. Recently, Ning et al. 196) found that adding input perturbations (IP) to latent variables z, during the forward process also helps diffusion training by mitigating exposure bias (97]. Compared to IP, CEP does not alter the marginal data distribution, but encourages the learned joint distribution to be more diverse.
of the factor V2 mainly follows previous works 82
92 951,
5.2 Experiments
Setup. We pre-trained previous class-conditional LDM-4, text-conditional LDM-4, class-conditional DiT-XL/2, and text-conditional LCM-v1.5 with CEP, and compare with IP and clean pre-trained ones. We use both Uniform and Gaussian perturbation, denoted as CEP-U and CEP-G, respectively, We set 7 for all models, with an ablation study with class-conditional LDM-4 with different ys. We evaluated the pre-trained class-conditions models on IN-IK and and centiconiiional models on MS-COCO with FID, IS, Precision, and Re- call. Additionally, we personalize the pre-trained LDMs with ControlNet on IN-100 to validate the effectiveness of CEP pre-training at downstream.
FandCare CAPE maN0
Results. We present the pre-training results of CEP in Table [ CEP significantly and universally improves the performance for different class and text-conditional DMs, e.g., 2.53 and 1.25 FID im- provement, and 42.31 and 10.27 IS improvement of LDM-4 and DIT-XL/2. CEP also improves precision and recall of DMs. In contrast, IP only achieves marginal improvement and yields slightly worse precision. Adopting CEP in pre-training also benefits the personalization tasks, especially for text-conditional LDMs, with FID improvement
Lg Duf.
Coeupttac (9S)
(a) Ablation o
(b) Corrupted IN-1K
Figure 8: Ablation with LDM-4 IN-IK. (a) FID and average L2 distance of conditional embeddings against clean ones with 7 = 10.1, 0.5, 1.0, 5.0, 10.0), indicated by square points (left to right). We compare with fixed syn- thetic corruption by circle points. (b) CEP on corrupted IN-IK.
(2.5, 5, 10, 15) %, shown
8

---

Puder resatti Yipenii un fughl brsie"
Cernil
Coa
CCPUU
CEP
Teepet
(a) Pre-training
(b) Personalization
Figure 9: Comparison of DMs pre-trained with CEP against IP and without perturbation.
of 6.08 and 7.02 for Canny and SAM spatial control, shown in Table Qualitatively, shown in Fig. 9 images generated from DMs with CEP also look more visually appealing and realistic.
The ablation results of are shown in Fig. 8(a) We also compare the average L2 distance of CEP and fixed corruption against the clean condition embeddings. Interestingly, one can observe that CEP achieves a lower FID with more corruption in the embedding space (larger L.2 from the clean ones), demonstrating its effectiveness. CEP is applicable to slightly corrupted datasets that we may often encounter in practice, shown in Fig. 8(b] where also facilitates the performance significantly.
In addition, we also compare the proposed CEP with traditional regularization methods, such as Dropout (98) and Label Smoothing [99], and study the effects of fixed and random perturbation during training in Appendix E.3 and Appendix E.4 The results show that CEP more effective.
6 Related Work
Diffusion Models.
Inspired by thermodynamics, DMs were first proposed by Sohl-Dickstein et al. 100]. DMs have soon been developed into image generation with fixed Gaussian noise diffusion process
101 Various techniques have then been proposed for more effective and efficient DMs
102
4 5.
One of the most well well-known is modeling the diffusion process at the latent space of
pre-trained image encoders as a strong prior (58 [56), instead raw pixels spaces g which
allows for high-quality image generation with affordable inference speed. Numerous foundational DMs that generate photorealistic images have thus been built [103 TO,
T08
109]. These
powerful models generally pre-trained on web-crawled billion- data with conditions (usually text), which may inevitably contain corruption [31 110 III 32 35). Recently, consistency models
112 52 were also developed from DMs, allowing generation with much fewer inference steps.
These foundational DMs also enabled many downstream applications [20] 21 [13] However,
the effects of the pre-training corruption downstream applications remain unknown.
Learning with Noise. Learning with noise is a mangsstandngg challenge [23 126
Noisy label 41
learning has been widely studied in classification, from noise correction [17
40
128
131
132
133 134
44
[35 and noise-robust loss functions [39 136
29)
142 Learning with noise
has also been studied in the context of generative models 143]
Robust GANs and DMs 44
alleviated the quality degradation and condition misalignment of training generative models with label noise. In contrast, study more practical scenario. where the models trained on corrupted pre-training data with a low noise ratio, and then adapted to downstream tasks.
In fact, more aligned with our work, there are several recent studies on exploring and exploiting the pre-training noise. Chen al. [48] 49] found that slight label noise in supervised pre-training can be beneficial for in-domain downstream tasks, whereas detrimental for out-of-domail tasks. NoisyTune 147 NEFTune (95), and SymNoise [148] found that introducing noise to the weights and embedding of pre-trained language models can facilitate downstream performance. Ning et al. (96) also found that adding perturbation in the forward diffusion process helps reduce the exposure bias of DMs [97]. Similarly, Naderi al. [149] introduced noise into the input of image translation

---

networks for better learning with limited data. Synthetic data (potentially with corruption) have also been found be useful in pre-training [24 [50]. We demonstrate that slight corruption in conditions of the pre-training DMs can also be beneficial at both the pre-training and downstream.
7 Conclusion and Limitation
We presented the first comprchensive study on condition corruption in pre- pre-training of DMs. Our empirical and theoretical analysis surprisingly demonstrate that slight condition corruption benefits DMs in both the pre-training and downstream adaptation, based on which we proposed CEP as simple yet general technique that significantly improves the performance of DMs. We hope findings could inspire more future work on understanding the pre-training data of foundation models.
This work has the following limitations. First, due to a lack of computing resources, cannot study all types of DMs on larger datasets. Second, the theoretical analysis is based on several assumptions that might be further explored in the future. Third, the evaluation of image generation remains an open question, and we used most of the existing criteria for fair comparison.
Disclaimer
While we study DMs for image generation in this paper, it is important to note that all generations have been selected and verified by human experts to ensure that they are responsible. Although we release all the pre-trained models under different corruption settings, it is possible that these models will generate inappropriate content due to the scale of pre-training and without alignment with human preferences. The main purpose of this research cleaning and corruption in the research of diffusion models.
raise the awareness of the community data
Acknowledge
MS was supported by the Institute for Al and Beyond, UTokyo. DZ was supported by NSFC 62306252, Guangdong NSF 1144A1515012444 and Hong Kong ECS awards 27309624.
10

---

References
[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[2] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[3] Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 34:1 1287-18811111 118877-13022 2021.
[4] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[5] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 35:26565-26577, 2022.
[6] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. AudioLDM: Text-to-audio generation with latent diffusion models. Proceedings of the International Conference Machine Learning, 2023.
[7] Muqiao Yang, Chunlei Zhang, Yong Xu, Zhongweiyang Xu, Heming Wang. Bhiksha Raj. and Dong Yu. usee: Unified speech enhancement and editing with conditional diffusion models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7125-7129. IERE, 2024.
[8] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8646, 2022.
[9] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10684-10699 2022.
[10] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36494, 2022.
[11] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4205, 2023.
[12] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
[13] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Y0cc100m: The new data in multimedia research. Communications of the ACM, 59(2):64-73,20 2016.
[14] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL, 2018.
[15] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail tail visual concepts. In CVPR, 2021.
[16] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. Advances Neural Information Processing Systems, 2021.

---

[17] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in 2022.
Neural Information Processing Systems,
[18] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint
Stt.::20080161.38 2022.
[19] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aber- man. Dreambooth: Fine tuning text-to-iman tomag diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500-22510, 10, 2023.
[20] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[21] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models, arXiv preprint wxXiv23322.08453 2023.
[22] Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan-Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[23] Ruifci He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenging Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition? arXiv preprint arXiv:2210.07574 2022.
[24] Lijie Fan, Kaifeng Chen, Dilip Krishnan, Dina Katabi, Phillip Isola, and Yonglong Tian. Scal- ing laws of synthetic images for model training... for now. arXiv preprint 12X1v:2312.04567, 2023.
[25] Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. Stablerep: Syn- thetic images from text-to-image models make strong visual representation learners. Advances in Neural Information Processing Systems, 36, 2024.
[26] Stability AI. Stable diffusion. Software available from Stability AI, 2022.
[27] Url https://oommoncrawl.org.
[28] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36, 2024.
[29] Curtis G. Northcutt, Lu Jiang, and Isaac L. Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research,
1411, 2021.
[30] Vijay Vasudevan, Benjamin Caine, Raphael Gontijo Lopes, Sara Fridovich- Keil, and Rebecca Roelofs. When does dough become a bagel? analyzing the remaining mistakes on imagenet. Advances in Neural Information Processing Systems, 35:6720-6734, 2022.
[31] Brian Gordon, Yonatan Bitton, Yonatan Shafir, Roopal Garg, Xi Chen, Dani Lischinski, Daniel Cohen-Or, and Idan Szpektor. Mismatch quest: Visual and textual feedback for image-text misalignment. arXiv preprint urXiv:2312.03766 2023.
[32] Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, et al. What's in my big data? arXiv preprint arXiv:2310.20707, 2023.
[33] Eric Frankel and Edward Vendrow. Fair generation through prior modification. In 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), 2020.
12

---

[34] Melissa Hall, Laurens van der Maaten, Laura Gustafson, Maxwell Jones, and Aaron Adcock.
A systematic study of bias amplification. arXiv preprint arXiv:2201.11706, 2022.
[35] Hao Chen, Bhiksha Raj. Xing Xie, and Jindong Wang. On catastrophic inheritance of large foundation models. arXiv preprint urXiv:2402.01909 2024.
[36] Amirhossein Kazerouni, Ehsan Khodapanah Aghdam, Moein Heidari, Reza Azad, Mohsen Fayyaz, llker Hacihaliloglu, and Dorit Merhof. Diffusion models in medical imaging: A comprehensive survey, Medical Image Analysis, page 102846, 2023.
[37] Xiaofan Li, Yifu Zhang, and Xiaoqing Ye. Drivingdiffusion: Layout-guided multi-view driving scene video generation with latent diffusion model. arXiv preprint arXiv:2310.07771, 2023.
[38] Chiyu Jiang, Andre Cornman, Cheolho Park, Benjamin Sapp. Yin Zhou, Dragomir Anguelov, et al. Motiondiffuser: Controllable multi-agent motion prediction using diffusion. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9644 9653, 2023.
[39] Aritra Ghosh, Himanshu Kumar, and P. Shanti Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, 2017.
[40] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Wai-Hung Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in Neural Information Processing Systems, 2018.
[41] Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In International Conference on Learning Representations, 2020.
[42] Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label-noise learning without anchor points. In International Conference on Machine Learning. pages 6403 6413. PMLR, 2021.
[43] Sheng Liu, Zhihui Zhu, Qing Qu, and Chong You. Robust training under label noise by werrppraaeeterization In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the International Conference on Machine Learning, volume 162, pages 14153-14172. PMLR, 17-23 Jul 2022.
[44] Hao Chen, Ankit Shah, Jindong Wang, Ran Tao, Yidong Wang, Xing Xie, Masashi Sugiyama, Rita Singh, and Bhiksha Raj. Imprecise label learning: A unified framework for learning with various imprecise label configurations. arXiv preprint 32571:2305.12715, 2023.
[45] Kiran K Thekumparampil, Ashish Khetan, Zinan Lin, and Sewoong Oh. Robustness of conditional gans to noisy labels. Advances in neural information processing systems, 31, 2018.
[46] Takuhiro Kaneko, Yoshitaka Ushiku, and Tatsuya Harada. Label-noise robust generative adversarial networks. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 2467-2476, 2019.
[47] Byconghu Na Ycongmin Kim, HeeSun Bac, Jung Hyun Lee, Se Jung Kwon, Wanmo Kang, and 1l-Chul Moon. Label-noise robust diffusion models. arXiv preprint 17X55 2024.
[48] Hao Chen, Jindong Wang, Ankit Shah, Ran Tao, Hongxin Wei, Xing Xie, Masashi Sugiyama, and Bhiksha Raj. Understanding and mitigating the label noise in pre-training on downstream tasks. In International Conference on Learning Representations (ICLR), 2024.
[49] Hao Chen, Jindong Wang, Zihan Wang, Ran Tao, Hongxin Wei, Xing Xie, Masashi Sugiyama, and Bhiksha Raj. Learning with noisy foundation models. arXiv preprint arXiv:2403. 06869, 2024.
[50] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25, 2012.
13

---

[51] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01400 2023.
[52] Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consisteney models: Synthesizing high-resolution images with fte- inference. arXiv preprint arXiv:2310.04378, 2023.
[53] Ali Borji. Pros and cons of gan evaluation measures: New developments. Computer Vision and Image Understanding, 215:103329, 2022.
[54] Eyal Betzalel, Coby Penso, Aviv Navon, and Ethan Fetaya. A study on the evaluation of generative models. arXiv preprini arXiv:2206. 10935, 2022.
[55] Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Teufel, Marco Bellagente, et al. Holistic evaluation of text-to-image models. Advances in Neural Information Processing Systems, 36, 2024.
[56] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873 11773-1883. 12883, 2021.
[57] Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data augmentation. Advances in Neural Information Processing Systems, 36, 2024.
[58] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in Neural Information Processing Systems, 30, 2017.
[59] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 0rxiv:18/0.04805, 2018.
[60] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision-ECCV 20/4: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pages 740-755. Springer, 2014.
[61] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver. fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
[62] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint 20Xiv:2010.02500 2020.
[63] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in Neural Information Processing Systems, 30, 2017.
[64] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in Neural Information Processing Systems, 29, 2016.
[65] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019.
[66] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A aeference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.081 2021.
[67] Peng Cui, Dan Zhang, Zhijic Deng, Yinpeng Dong, and Jun Zhu. Learning sample difficulty from pre-trained models for reliable prediction. Advances in Neural Information Processing Systems, 36, 2024.
[68] Minhyuk Seo, Diganta Misra, Seongwon Cho, Minjae Lee, and Jonghyun Choi. Just say the name: Online continual learning with category names only via data generation. arXiv preprint arXiv:2403. 10853, 2024.
14

---

[69] Claude Elwood Shannon. A mathematical theory of communication. ACM SIGMOBILE mobile computing and communications review, 5(1):3-55, 2001.
[70] Yuchen Wu, Minshuo Chen, Zihao Li, Mengdi Wang, and Yuting Wei. Theoretical in- sights for diffusion guidance: A case study for gaussian mixture models. arXiv preprint 2XXi:22403001633 2024.
[71] Charlic Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. In International Conference on Machine Learning, 2021.
[72] Pum Jun Kim, Yoojin Jang, Jisu Kim, and Jaejun Yoo. Topp&r: Robust support estimation approach for evaluating fidelity and diversity in generative models. Advances in Neural Information Processing Systems, 36, 2024.
[73] Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito. and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pages 5253-5270, 2023.
[74] Pu Cao, Feng Zhou, Qing Song, and Lu Yang. Controllable generation with toxt-to-image diffusion models: A survey. arXiv preprint arXiv:2403.04279, 2024.
[75] Xulu Zhang, Xiao Yong Wei, Wengyu Zhang, Jinlin Wu, Zhaoxiang Zhang, Zhen Lei, and Qing Li. A survey on personalized content synthesis with diffusion models. arXiv preprint arXiv:2405.052: 00558858, 2024.
[76] Itseez. Open source computer vision library. htttps//gghhubb.cmm//eeeee/
2015.
[77] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015 4026, 2023.
[78] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-
image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 11888-19000. PMLR, 2022.
[79] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[80] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint WrXiv:2303008774 2023.
[81] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCAA 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[82] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[83] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 16Xivv1607.06450 2016.
[84] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018
[85] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXivv1809.11099 2018.
[86] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEFF/CVE conference on computer vision and pattern recognition, pages 4401-4410, 2019.
15

---

[87] Sitan Chen, Vasilis Kontonis, and Kulin Shah. Learning general gaussian mixtures with efficient score matching. arXiv preprint arXiv:2404. 18893, 2024.
[88] Khashayar Gatmiry, Jonathan Kelner, and Holden Lee. Learning mixtures of gaussians using diffusion models. arXiv preprini arXiv:2404. 18869, 2024.
[89] Wei Hu, Zhiyuan Li, and Dingli Yu. Simple and effective regularization methods for training on noisily labeled data with generalization guarantee. arXiv preprint arXiv: 2019.
[90] Yu-Hang Tang, Yuanran Zhu, and Wibe A de Jong. Detecting label noise via leave-one-out cross-validation. arXiv preprint arXiv:2103. 11352, 2021.
[91] Jeremy Speth and Emily M Hand. Automated label noise identification for facial attribute recognition. In CVPR Workshops, pages 25-28, 2019.
[92] Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. Freelb: Enhanced adversarial training for natural language understanding. arXiv preprint arXiv:1909, 1764, 2019.
[93] Kezhi Kong, Guohao Li, Mucong Ding, Zuxuan Wu, Chen Zhu, Bernard Ghanem, Gavin Taylor, and Tom Goldstein. Robust optimization as data augmentation for large-scale graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 60-69, 2022.
[94] David Nukrai, Ron Mokady, and Amir Globerson. Text-only training for image captioning using noise-injected clip. arXiv preprint arXiv:2211.00575 2022.
[95] Neel Jain, Ping-ych Chiang, Yuxin Wen, John Kirchenbauer, Hong-Min Chu, Gowthami Somepalli, Brian R Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Aniruddha Saha, et al. Neftune: Noisy embeddings improve instruction finetuning. arXiv preprint 4rXiv:2310.05914, 2023.
[96] Mang Ning, Enver Sangineto, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Input perturbation reduces exposure bias in diffusion models. arXiv preprint arXi::2301.11006 2023.
[97] Mang Ning, Mingxiao Li, Jianlin Su, Albert Ali Salah, and Itir Onal Ertugrul. Elucidating the exposure bias in diffusion models. arXiv preprint arXiv:2308. 15321, 2023.
[98] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199 1958, 2014.
[99] Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
[100] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep un- supervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265 PMLR, 2015.
[101] Yang Song, Jascha Dohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprini arXiv:2011.13445: 2020.
[102] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8 1672-177. PMLR, 2021.
[103] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mc- Grew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112. 2021.
[104] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, 34:19822-19835. 2021.
16

---

[105] Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. Cogview2: Faster and better text-to- image generation via hierarchical transformers. Advances in Neural Information Processing Systems, 35: 18910-1699992, 2022.
[106] Wendi Zheng, Jiayan Teng. Zhuoyi Yang, Weihan Wang, Jidong Chen, Xiaotao Gu, Yuxiao Dong, Ming Ding. and Jie Tang. Cogview3: Finer and faster text-to-image generation via relay diffusion. arXiv preprint arXiv:2403.05121, 2024.
[107] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based lext-to-image generation with human priors. In European Confer- ence on Computer Vision, pages 89-106. Springer, 2022.
[108] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pages 8821-888 Pmir, 2021.
[109] MidJoumey Inc. Midjoumny. Software available from MidJourny Inc., 2022.
[110] Ryan Webster, Julien Rabin, Loic Simon, and Frederic Jurie. On the de-duplication of laion-2b. arXiv preprint arXiv:2303. 12733, 2023.
[111] Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Understanding and mitigating copying in diffusion models. Advances in Neural Information Processing Systems, 36:47783-478033 2023.
[112] Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXiv preprint arXiv:2310. 14189, 2023.
[113] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv: 1nt:22008.0007. 2021.
[114] Tim Brooks, Aleksander Holynski. and Alexei Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 18392-18402, 2023.
[115] Nisha Huang, Fan Tang, Weiming Dong, Tong Yee Lee, and Changsheng Xu. Region-aware aware diffusion for sero-shot text-driven image editing. arXiv preprint arXiv:2302.1179)) 2023.
[116] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1921-1930, 2023.
[117] Andrey Voynov, Kfrr Aberman, and Daniel Cohen-Or. Sketch-guided text-to-image diffusion models. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1-11, 2023.
[118] Nisha Huang, Yuxin Zhang, Fan Tang, Chongyang Ma, Haibin Huang. Weiming Dong, and Changsheng Xu. Diffstyler: Controllable dual diffusion for text-driven image stylization. IEEE Transactions on Neural Networks and Learning Systems, 2024.
[119] Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Com- poser: Creative and controllable image synthesis with composable conditions. arXiv preprint TrXiv:2302.09778 2023.
[120] Dina Bashkirova, José Lezama, Kihyuk Sohn, Kate Saenko, and Irfan Essa Masksketch: Un- paired structure-guided masked image generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1879-1889 2023.
[121] Omer Bar- Bal-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. 2023.
[122] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to text-to-image generation. In Proceedings of the IEFF/CV Conference on Computer Vision and Puttern Recognition, pages 22511-2255 1-2221,,, 2023.
17

---

[123] Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances in Neural Information Processing Systems, 26, 2013.
[124] Bo Han, Quanming Yao, Tongliang Liu, Gang Niu, Ivor W Tsang, James T Kwok, and Masashi Sugiyama. A survey of label-noise representation learning: Past, present and future. arXiv preprint arXiv:201 0,4406, 2020.
[125] Görkem Algan and Ilkay Ulusoy, Image classification with deep learning in the presence of noisy labels: A survey. Knowledge Baasdd Systems, 215:106771, 2021.
[126) Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE transactions on neural networks and learning systems, 2022.
[127] Bo Han, Quamming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyara. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in Neural Information Processing Systems, 31, 2018.
[128] Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In International Conference on Machine Learning, pages 7 7164-7173. PMLR, 2019.
[129] Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? Advances in Neural Information Processing Systems, 32, 2019.
[130] Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang. Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Pert-dependent label noise: Towards instance-dependent label noise. Advances in Neural Information Processing Systems, 33:7597-7610, 2020.
[131] Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19726-13735 2020.
[132] Lei Feng, Senlin Shu, Zhuoyi Lin, Fengmao Lv, Li Li, and Bo An. Can cross entropy loss be robust label noise? In Proceedings of the twenty-ninth international conference on international joint conferences on artificial intelligence, pages 2206-2212, 2021.
[133] Yivan Zhang, Gang Niu, and Masashi Sugiyama. Learning noise transition matrix from only noisy labels via total variation regularization. In International Conference on Machine Learning, pages 12501-12512. PMLR, 2021.
[134] Shuo Yang, Erkun Yang, Bo Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu. Estimating instance-dependent bayes-label transition matrix using a deep neural network. In International Conference on Machine Learning, pages 25302-25312. PMLR, 2022.
[135] Hongxin Wei, Huiping Zhuang, Renchunzi Xie, Lei Feng, Gang Niu, Bo An, and Yixuan Li. Mitigating memorization of noisy labels by clipping the model prediction. In International Conference on Machine Learning, pages 36868 36886. PMLR, 2023.
[136] Zhilu Zhang and Mert Sabuneu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in Neural Information Processing Systems, 31, 2018.
[137] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 322-330, 2019.
[138] Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Monazam Erfani, and James Bailey. Normalized loss functions for deep learning with noisy labels. In Proceedings of the International Conference on Machine Learning, 2020.
[139] Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition, pages 2691-2699, 2015.
18

---

[140] Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using noise adaptation layer. In International Conference on Learning Representations, 2016.
[141] Sheng Liu, Jonathan Weed, Narges Razavian, and Carlos Fernandez- Grmanddz-Granda. Early- learning regularization prevents memorization of noisy labels. Advances in Neural Information Processing Systems, 33, 2020.
[142] X. Li, T. Liu, B. Han, G. Niu, and M. Sugiyama. Provably end-to-end label-noise learning without anchor points. In Proceedings of 38th International Conference on Machine Learning. pages 6403 6413, 2021.
[143] Kiran Koshy Thekumparampil, Sewoong Oh, and Ashish Khetan. Robust conditional gans under missing uncertain labels. arXiv preprint arXiv:196600333 03579, 2019.
[144] Sandhya Tripathi and N Hemachandra, Gans for learning from very high class conditional noisy labels. arXiv preprint arXiv:2010.09577 2020.
[145] Takuhiro Kaneko and Tatsuya Harada. Blur, noise, and compression robust generative adver- sarial networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13579- 13589, 2021.
[146] Lizhen Deng, Chunming He, Guoxia Xu, Hu Zhu, and Hao Wang. Pegan: A noise robust conditional generative adversarial network for one shot learning. IEEE Transactions on Infelligent Transportation Systems, (3(12):25249-25588 2022.
[147] Chuhan Wu, Fangzhao Wu, Tao Qi, Yongfeng Huang, and Xing Xie. Noisytune: A little noise help you finetune pretrained language models better. arXiv preprint arXiv:2202. 12024, 2022.
[148] Arjun Singh and Abhay Kumar Yadav. Symnoise: Advancing language model fine-tuning with symmetric noise. arXiv preprint arXiv:2312.01523, 2023.
[149] Mohammadrezz Naderi, Nader Karimi, Ali Emami, Shahram Shirani, and Shadrokh Samavi. Dynamic-pix Medical image segmentation by injecting noise egan for modeling input and target domain joint distributions with limited training data. Biomedical Signal Processing and Control, 15:104 2023.
[150] Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, and Oncel Tuzel. Clip with quality captions: strong pretraining for vision tasks. 2024.
[151] Hongrui Chen, Holden Lee, and Jianfeng Lu. Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions. In International Conference on Machine Learning, pages 4735-4763 PMLR, 2023.
[152] Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint arXiv:2209.1 11215, 2022.
[153] Xuefeng Gao, Hoang M Nguyen, and Lingjiong Zhu. Wasserstein convergence guarantees for general class of score-based generative models. arXiv preprint arXiv:2311.11003, 2023,
[154]
Christiane Fellbaum. WordNet: An electronic lexical database. MIT press, 1998.
[155] Patrick von Platen. Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj. Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-arr diffusion models. https: thub. www.hoggiigeeeeeeeeeoo diffusers 2022.
[156] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEFE conference computer vision and pattern recognition, pages 2818-2826, 2016.
19

---

[157] Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
[158] Xiangming Gu, Chao Du, Tianyu Pang, Chongxuan Li, Min Lin, and Ye Wang. On memoriza- tion in diffusion models. arXiv preprint arXtv:2310.02664, 2023.
[159] TaeHo Yoon, Joo Young Choi, Sehyun Kwon, and Ernest K Ryu. Diffusion probabilistic models generalize when they fail to memorize. In ICML 2023 Workshop on Structured Probabilistic Inference Generative Modeling, 2023.
20

---

Appendix
Contents
Derivations and Proofs
22
Preliminaries
22
A.2
2
The Estimation of Conditional Score
23
A.3
The Distribution of Generation
26
4
Diversity and Quality: Clean vs. Corrupted Conditions
30
B
Details of Condition Corruption, Model Training and Evaluation
33
B.
Synthetic Condition Corruption
33
Automatic ImageNet- 100 Annotation
33
B.3
LDM Pre-training Setup
34
B.4
DIT Pre-training Setup
34
B.5
LCM Pre-training Sctup
35
B.6
ControlNet and 121-Adapter Personalization Setup
35
B. 7
Evaluation Metrics
35
Full Results of Pre-training Evaluation
36
Quantitative Results
36
Qualitative Results
38
Full Results of Downstream Personalization Evaluation
D.I
I Quantitative Results
39
D.2
Qualitative Results
39
Full Results of Conditional Embedding Perturbation
40
E.
Qualitative Results
40
E.2
Ablation Study
40
Comparison with Dropout and Label Smoothing
40
Comparison with Fixed and Random Corruption
40
21

---

A Derivations and Proofs
In this section, we theoretically investigate the behavior of condition corruption on DMs. Our investigation encompasses the DDIM samplers with continuous-tiin time processes. We focus on how slight conditional embedding corruption can affect the training process of DMs, as well as the consequences on the generative process. As our theoretical analysis indicates, slight conditional embedding corruption will benefit both the generation quality and diversity, which aligns with experimental conclusions in Section B
A.1 Preliminaries
We start by giving concise overview of the problem setup.
Data Distribution. For precise theoretical characterizations, we concentrate on the prototypical problem of sampling from Gaussian mixture models (GMMs). Specifically, consider that the distribution of the data x E IRd satisfies
P(x) = Σ (p
yEV
Here, y is denoted as the class labels with finite set of values y E (1, | Given any class label E y, the data distribution x a Gaussian with the center and covariance as (fyy I). And the positive Wy represents the weights the Gaussian components which satisfies Σyey Wy =
The diffusion model two processes framework: a forward process that transforms the target distribution into Gaussian noise, and reverse process that progressively denoises in order to reconstitute the original target distribution. this paper, we consider the continuous-time processes and define the forward process as an Omstein-
(OU) process:
Forward Process. At time step E [0, T), the forward process is
dx+ = xx, + 2dw,, X = P(-l)),
(10)
where w, d-dimensional standard Brownian motion.
The advantage of considering the forward process an OU process is that it enables us to directly derive the closed-form expression for the conditional sample distribution at any given time
x,|x N(rrx,ooI)
(11)
where 7, = e et and 0N = =
By the reparameterization trick, can be represented as
X = r:X + TrE.
(12)
We explore the widely adopted sampling method, DDIM, augmented with classifier-fres guidance. And the associated reverse process is stated the following ODE implementation:
Reverse Process.
We write the reverse process forward version by switching time direction
t T as
dz: = Z; (1 Vz, F(z w Ve. log loz)((z))e dt, 2o ~ N(O,I),
(13)
where w is hyperparameter that controls the strength of the classifier guidance.
Given our primary focus on the impact of corrupted conditional embedding on generation, we simplify the reverse process by setting w and concentrate solely on the conditional score network, i.c..
dz:
Ze + V log [((, (w) dt.
(14)
The remaining task is estimate the unknown conditional score function Vz, log (P(z)))) lv) and unconditional score function Vz, log PP(, (v) via training. In the subsequent analysis, we will indicate that by minimizing Equation (3) and optimizing the denoising network e, we can achieve an estimate of the conditional score.
22

---

Denoising Networks. Inspired by recent work that also target on GMMs 87 88 we parameterize the denoising networks as the following piecewise linear function:
D 3 1 (Wixe vyc(y)).
EB (x y)
(15)
where c(y) is the one-hot encoding of label y. 1y is an indicator function and {W?, trainable parameters.
V
k== are
Conditional Embedding Corruption. We examine the scenario of conditional embedding corruption, wherein the conditional embedding c(y) is no longer matched with the data x, but instead is perturbed by Gaussian noise. Consequently, the corrupted conditional embedding causes the denoising networks to be as follows:
(x),
Σ 1
Wix +E)))
(16)
where the d-dimensional corrupted noise E ~ (O, and y 2 0 serves as the corruption control parameter, mirroring the noise ratio 7) for direct control of noise magnitude.
Based on the aforementioned setup, our ultimate goal to obtain the closed form of the final generated data distribution by considering the impact of corrupted noise on the training and generative processes. By comparing the differences data distribution before and after the addition of corrupted noise, aim to accurately characterize the impact of embedding corruption on image generation and explain the phenomena observed experiments.
A.2
The Estimation of Conditional Score
A.2.1
Clean Conditions
We first analyze the case of clean conditional embedding. Note that the denoising networks is piecewise linear function and the training objective can be represented as
E. lllcol (xe,; v) c
Wike
v))
(17)
T&k
wi=k
where the class sample size n2 := naKk-
We observed that the optimization objective in Equation (77 can be divided into |)]] independent bob-problems based on the label y. This inspires us to analyze the training and generation processes according to different classes.
Given any class V, we present the following lemma for determining the optimal parameters of the corresponding denoising network and the associated conditional score.
Lemma (Clean Conditional Embedding). Given any class le E Y, the optimal linear denoising network is
(x,x,,y (.) x; - rior 00 ,イ iss-
(18)
And the corresponding optimal linear estimation of conditional score Vx, log P(xely- = k)) is
Vx, log Pxx = k) cx k)
(19)
where Σg nh. Li=1 X
1 Σm X is the empirical covariance of k-labeled
dataset and f4: my En x; is the empirical mean of k-labeled dataset, ng is the sample size of k-labeled dataset. And r2 -t, = v1 -
Proof. Given any class k. the training objective
16k
Bellc((x,...0) v) <||2
Wi
Σ
Eell Whole+ V:o(y) - -[1/)
(20)
n2 m
S/
23

---

Since the weights wi is fixed in our analysis, we omit the notation without ambiguity in the following contents and for enhanced clarity, we initially omit the subserip/suppeseripp in ag. W.. VE, My + My and Eg. Then Equation (20) can restated as Equation (21 for simplicity.
Well Woote + Vic(y) =112),
(21)
where x, is labeled as k. This training loss can be further simplified as
Well Wiote V,c(y) =E
1
Wirx
E. - I)ll
W. + 2r (y) W, Σ x + te (e +
n
For simplicity, we denote b, = Vec(y) and we get
Bel Bell W + Wioie+ Vic(y) el
n
w. Wxx + 2d w. Σ
x, b, I
1=1
Then, define the loss function
J(W, b,) = r Σ x
n
W.w b, W. x, + b,,b, + Tr - -1))
n 1=1
The optimal Wi and by can be obtained by taking gradient to JW,, b,) such that,
2r W.
+ 2r: b, Σ
"
Vw, J(W, by) =
x,
T 2 W. -,(1,
T2k
i=1
2r W.
J(W, b;)
x; 2b.
n
And the optimal Wi and bj is,
W; oe 0f -
b; Vic(y)= -ro (o² (o, + +イン
j..
where Σ = 1 Dn x
में Σ x; ET x is the empirical covariance of k-labeled dataset
and fa = Σt X; is the empirical mean of k-labeled dataset.
Based on above analyze, for class k. we then have the following optimal denoising network as the linear estimator of noise E,
E Ex y vx (0
(7 f..
Given any class k. derive
(x, rxx)
Vx. log (xxe/xx k)
0+
where data x and x; are labeled with A.
24

---

Therefore, the linear estimator of Vx, log P(x, x, y k) given k-labeled data x, is
- +
X, + rt 0f 1 +ラン) μ.
Since the estimation of V log Px x, y k) and Vx log [x k) equivalent in optimiza- tion, the optimal linear estimator Xx, log (x / = k) also can be
V log P(x:ly= k) E0(x1, y k)
dt
- (0,I+ x + tr
(0 + jz.
The proof is completed.
Note that the ground truth functions of conditional score given the label is
Xx, log P(x =k Xx + ref4g-
(22)
We note the similarities in form and coefficients between the optimal linear estimator (19 and ground truth (2 they both are linear combinations of /4/ and x,. Specifically, when the empirical covariance Σg equals the population covariance I and the empirical mean fi matches the expected value p4, the estimated conditional score Equation (19) coincides with the true conditional score
in Equation 22
A.2.2 Corrupted Conditions
The same analytical approach as in Section A.2. will be applied to the scenario where the conditional embedding is perturbed. Given the class k, we propose the following Lemma 2 to describe optimal linear denoising network and the conditional score estimator with the corrupted conditional embedding (c(y) + +E) where & is the standard Gaussian noise.
Lemma 2. (Corrupted Conditional Embedding). Given any class ls E V the optimal linear denois-
ing network
E|x,, k) ae (o(2f1 2 14g2
X.
Tt I4 (of1
12
1+
rfa2"
(23)
And the corresponding optimal linear estimation of conditional score Vx. log P*(x = k) is
Vx, log (P (x+l1) =k) (x y k)
(24)
Ot
Σ Eie x! is the empirical covariance of k-labeled dataset and fby. Th Li=1 X; is the empirical mean of k-labeled dataset, na the sample size of k-labeled dataset, and = e e-
where Eg Thg =i= x
Proof. For enhanced clarity, we initially omit the subscript/superseriptt in ng. W V A M%
and Σ,.
For any class io and standard Gaussian noise g, consider the following training loss
BeEa:///rxx + Wio.e + V.(c(w) + > -< -E1
(25)
And further optimize the training loss
x
W Wigc+ V.((g) + c|@l
7
1
Wirx + Vic(y) + + E I)
2 T.T.W.t+ +
2r. Th e. Σ
x; T
==1
TTV,Vv + Tr (W I
25

---

Similarly, we get the optimal Wi and b; is,
W = a: ( r E + r?
1 72 11 -1
r??? 1+
b = Vic(y) - nt
0 + +
| j..
1+
where Σ = 4 Σ x
+ x x is the empirical covariance and ja
"
- E= X is the empirical mean of k-labeled dataset.
So for any class k. we get the following optimal denoising network as the linear estimator of noise e,
<xx.,y a. (o E
I+
1t
P Pullal)
Dx.
The corresponding optimal linear estimator Vx, log (x = k:) is
co(xe.y= k) Vx, log (x,lV k)
de
=- (f +
+ 1
1272
||
-1
1
7t
0f
1+73
-1 [1:113))
a.
A.3
The Distribution of Generation
Given the class after getting the optimal linear estimator of conditional score Vx, log (x
k), accurately characterize the reverse process and derive the closed form of generation distribution.
A.3.1 Clean Conditions
According to Lemma we first replace the conditional score Vx, log (egpP(x,12 /yy k)) in Equation (14) with the optimal linear estimator. Given the class k; we get
dz,
(z: (0.1+ ze re (o
is dt.
(26)
Assume the empirical covariance Σg is full rank, since the empirical covariance is diagonalizable and semi-positive, we decompose Σg as
Σ UAUT,
(27)
where U = jui ... [ is an orthogonal matrix whose columns are the real, orthonormal eigenvectors of Σx and diag (AI,..., Aa) is diagonal matrix whose entries are the eigenvalues arranged in descending order of Σg, i.e., 1 2 A
Aa > O.
Therefore, problem 26 can be decomposed into d independent sub-problem
n dzz (1 of ,? (
1
u. 2,
Ft. u. dt
(28)
of+
where is i-th eigenvector and A₃ is its corresponding eigenvalue.
By analyzing Equation 28
obtain the expectation and variance of the final generation distribution separately, thereby inferring the distribution the ultimately generated data as outlined in the subsequent lemma.
Lemma 3. (Clean Conditional Embedding). For any class k. the distribution of the generated data zr satisfies
lim ZT ~ N(fa Ek), T
(29)
where Σg. = Li= Xix!
Σ E xJ is the empirical covariance of k-labeled is the empirical mean of k-labeled dataset, is the sample size of
dataset and jax k k-labeled dataset.
Ti D
my
26

---

Proof. For enhanced clarity, we initially remove the subscript k in Pu Mg and Σx. In order to obtain the solution to Equation (28) and achieve the closed form the generation distribution, we first examine the discrete solution of Equation (28).
Given any class k. we consider the discretization Fuler- Maruyama scheme which is widely used in existing work [151H153] and discretize the interval [o, 2] into N discretization points,
u 1j+ = u 2sh + (1 oi, + (t,))
u zyh + oi (r
r(tz) u ih,
(30)
where h : N is the step size and f2 jh.
According to Equation (30), u, 1it can be regarded as a linear transformation of the initial distribution, which is standard Gaussian distribution. Therefore, u. (] 1)a still satisfies Gaussian distribution. The remaining task is analyze the mean and variance of the uf Zú+1jA separately.
Expectation. By Equation (3), we have
E(u z)+1)))= = 1+
of (1, ())))))))))))))))))))))))))))) B(u zjn) + r(t,)
o7, + (t,)) uf
By telescoping, we get the discretization solution
N-1
E(uZZNn) ZNN)
II
1 (1
h EE(u, zo)
o + +r 2
k=0
N 1
r(tx)
u juh
N-1
II
1 (1
+())))))))))
1 1
+
rr²(tx) )x
ot
k=
j=k+1
Since the initial distribution of reverse process is Zo ~ NN I), thus E(u zo) O. We further have
N-1
r(ta) K= of (tt(t)
N-1
(1 1+ (1+(1- of +P()))))))
Bu 2NE)
Σ
u ju II
(31)
By limiting + O, we can use the identity + ha exp(hx). Then we get the continous version of Equation 31 that when h + O, we have
E Eu T ZT)
oi
F2
X
exp
1.
1 1-
o7 + (()))
1
dr und isdt
L-T
1 ( T), -1) dr uu judt
exp
+ 2(1 (1: 1)
T
e Vi
judt
1 2( T) (d: 1)]
eT
T
u i
1 221 ()
1)
=(1 v1+ (2, 1)e-2T 2T
Vype-T
11 jt.
Hence, we have
E(zr) Udiag (c1, E2 ez, U p.
Vize-
where & =1 1 VI+ 11)
When T + 0o, we get
Vdse-1
6 -=1-
I (A: 1)e-2T
27

---

And the expectation of generation distribution is the empirical mean, i.e.,
E(zr) = p.
Variance. Similarly, by Equation 30 we have the variance
Var(u] 3141)))) = + (1 oi - (()))) Var(u, 2,n).
By telescoping, we get the discretization solution
of + +r2(1e) (42)) )r) Var(u
Var(u,
II
1
k=0
Since Var(u, zo) = u Var(zo)u, = 1, the variance at time T is
N-1
1+ (1 Ti + 1 (((a)t))))
Var(u.ZNN)
II
(1x))
k=
When - and use the identity (1 + hx)² exp(2hxz). we get
1
Var(u.27 2T) exp
2
de
σ + rrt
0
T
2(1
= exp
0
1+ (A: 1)e2(1-T))
A
1+ (A: 2T
Hence, we have
Var(zr) Udiag(E1, U2 ..,..., ,,) U².
where Us 1+( 27
When T' + 00, we get
A 1/ 1 (Ai 1)e-22 - Ai-
And the expectation of generation distribution the empirical mean, i.e.
Var(zr) = E.
We complete the proof.
A.3.2
Corrupted Conditions
We then discuss the distribution of generated data when the conditional embedding is perturbed by noise. Using the same method derive the data distribution under the corrupted conditional embedding setting, we conclude the following Lemma
Lemma 4. (Corrupted Conditional Embedding). For any class E V, the distribution of generation 24 is
fur 1 I+a,²
lim zÿp ~ N T0c
Ex + 1+ P (11
(32)
where Σg. dataset and jun := ny k k-labeled dataset. And 0 is the corruption control parameter.
Th Li=1
Σnh x: Eis x3 is the empirical covariance of k-labeled L is the empirical mean of k-labeled dataset, is the sample size of
X
n
28

---

Proof. To improve clarity, we initially disregard the subscript k in Pe My and Σg. Building upon the proof details presented in Lemma B and the optimal conditional score estimation outlined in we arrive at the subsequent findings:
Lemma 2
Expectation. Similarly, we derive
14 L of (2 17 ( K 25)
exp
dr u! pdt
of
(T)(AA) 1 17-0 (111))))))
1+ L (")
exp
1
dr ot patt
1+ 2a1- (T - 1 + (
1 - |1 12)
c
アサ 1
= 11 Jo [1 1++ (1+ (A, + 1+ |11
i pult
72
+ 1 1
u j
V caIe (A 11 11
A
(Aa 1 1477 e of
Hence, we have
E[ = Udiag c U'p
1 x 14 ||l/| 3 T
where of = 1470
(2, - 1+ ببب |1
When T + 00, we get
v Ax + 72 |101
1
1+
(A; + - ja 12)e-
27
1+
And the expectation of generation distribution is
B(2T)=
1 + 77
Variance. We get
Var(u,zZ) z4)) = exp
2
1
dt
o
0, + r?, +
72
1+
1
exp 0
2
dt
1 + (Ax - 1+ T
l1
e
A 1++= A 1
7²
1 (Ax - 1 + 1ि | 13)--22 2T
Hence, we have
Var 2 Udiag (of, (5)...,,)))"
A 1A11
where of 1+(A,-1 1752 [1lB)c 25'
When T + 00, we get
A +
32
of
A +
1/..
1+ 11+ 11 27
1+22
And the expectation of generation distribution is the empirical mean, i.c.,
72
Var(z)) = Σ + 1 + 1 A1 1
29

---

A.4 Diversity and Quality: Clean vs. Corrupted Conditions
A.4.1 Generation Diversity
Given any class k, building on previous work [70,, we consider entropy to measure the diversity of generated images. In particular, let IP and P be the probability densities for the generated data using clean and corrupted conditional embeddings respectively, we have the following for z₁ and z;
H ( y = k) / P(zly - k) log P(zly k)dz,
(33)
H = k) := /
(zly = k:) log (zly = k)dz.
(34)
We propose the following theorem to describe the difference between these two conditional differential entropy.
Theorem 3. (Restatement of Theorem ! For any class k E V, assuming the norm of corresponding expectation || is a constant and the empirical covariance of training data is full rank, let 2y and zÿ be the generation featuring clean and corrupted conditions respectively, then it holds that
H(zrryy- = k) H k) = O
(35)
where y is the corruption control parameter and is the data dimension.
Proof. For the sake of clarity, we begin by omitting the subscript & from üu M4 and Σg. Given any class k, since both the clean generation žт and the cornupted generation zy follow multivariate Gaussian distributions, we can derive the closed-form expression for the difference in their differential entropy by Lemma 3 and Lemma 4 as follows
72
H(zgly- = k) H(Zrly = k) = log Σ
log Σ
1 72
72 1B)
log I
(1 71 i
where Σ = 1 Ei= x + Ei= Ei=1 x is the empirical covariance of k-labeled dataset and j := Σ X is the empirical mean of k-labeled dataset.
When the noise ratio y small and A w()),
log (1 (1 21+ [1112) = 72 A; IAAl +
72
Therefore,
1 d
22
Hz k) H2Tlly k) - log (1+
(1 + ) [1 -
O()2d).
A.4.2 Generation Quality
Before starting proving that slight noise is beneficial to the quality of generation, we first introduce the lemmas required for the proof.
Lemma 5. Given "","", Xn independent and all distributed as Gaussian Nu, I). Then,
E(Var(x;(x1, + +... X, = 2)) n1 I.
(36)
n
Proof. The expectation is
E x;
X + ... + X, =2)
E(z - X
Xi-1 Xi+1 - _..., ....,,,) xx,xxx [x] + .... Xn = z)
z (n 1)E(x,/x1 + ... Xn, = =).
30

---

Therefore,
E(x: X1 4... Xn = 2) 2
By the law of total variance
[[Var(x;|>) + + Xx = =2) VVr(x;) Var Var((E(x;/21) X, = =2))
I.
We consider the Wasserstein distance to measure the distance between the generation distribution and the true data distribution. A smaller Wasserstein distance implies closer proximity between the generated data distribution and the true data distribution, thereby indicating better generation quality
Given class k, define 2- Wasserstein distance between the true data distribution xly k and the clean generation as d := W2 (M I), Ex)) Similarly, the Wasserstein distance between the true data distribution x|y k and the corrupted generation zi is denoted as de :- (M
W2 [ (1 N), (T En +ス I (
Theorem 4. (Restatement of Theorem 2 For any class k Y, assume the norm of corresponding expectation 1 is constant, let P. Qx and Qx be the ground truth, clean, and corrupted condition distributions, respectively, where X represents the collection of training data points. O(1/ 101///maaa /max; TK),, it holds that
2d 13
Ex -
1xl k
(37)
where (..) denotes the quadratic 2- Wasserstein distance between two distributions, is the sample size of k-labeled dataset and d is the data dimension.
Proof. To express more clearly, we first omit the subscript i of nı- P&g H4 and Σg. According to Lemma 3 and Lemma we then can directly derive the closed form of Wasserstein distance between two Gaussian as
The squared Wasserstein distance between true data distribution and clean generation distribution
de 1la 111a-+112+ Tr(I) + Tr(E) - 2Tr(E*)
| 21 A Σ
i=1
The squared Wasserstein distance between true data distribution and corrupted generation distribution
df
2
Tr(II + +TH
1 1 1+ (
72
2Tr (
1 + |1 11)))
22
1+
11
jt
2
Σ > 2
22
+
1 1+ [1a013))
-2 2
A +
1 1+
1 1 22
1=1
The difference in squared Wasserstein distance is
11 + f
Σ
d
(v
x
-
x2
111 1 vxx)
We consider the expectation error reduce the randomness of jı as
E
-0 = E
+ E
Au 1 All xx
1A
1+
emors by variance
31

---

We notice that the expectation error can be decomposed into two parts. The error of the first part being caused by the difference in means between the generated distribution and the true distribution. Since the distribution of empirical mean is ja ~ NN n I) where ne is the sample size of k-labeled dataset, we get
E
P 72
1
+
E
|l
1112
n11 +
d
72y2
2 1 + 72
1
d
2 d
2d72
o(72)) +
n1 n
2d7²
o(771)
n
The second part is attributable to the difference in covariance.
72
22
10
2
A
VA.)
|
I+
1 + 7
d
E
2
111111 - o(7)))
A
E
1]
o(7Ad).
By the law of total expectation
1
E
|ABE
d
1
E
d
j 1
TE
d.
E(MAA))
(a) and (b) achieve by the Jensen's 's inequality given vx, is a convex function
By Lemma 5 we derive
Tr 12 Var(x((2)) = Tr E
(1(2)(0)
Tr (UE [UE[AIA]UT)
Σ B(M n-
d.
n2
Hence, we get
1
E
1)112111
> E
d
X
Σ 1 B(Al())))
n
1 dlE |I
n
(v n
n
-1) ( + dd1
The second part is
d
22
72
2
A +
|
A11)
1+13 1121/3
1 1
1=1
d²
2
(v
n 1
n
- 1)
+
d|
(2 -(7
n
32

---

Therefore, with small noise ratio we then can conclude that
Eld
+ [I1a-
レ 1 2( 20
72 1
1lal
vA)
x2
2dy² I-I
2 +>²
1)
(
d
+ dl B))
ol7 d).
Noting that (n/(n 1) - ((L/n)) when n is large, then if the cortuption level y satisfies O(1/ nn), for any class k. we have
Eld d = =Ω (dd
This completes the proof.
B Details of Condition Corruption, Model Training and Evaluation
In this section, provide detailed training setup of each diffusion models studied in the main paper, the synthetic corruption for IN-1K and CC3M, the annotation process of IN-100, and the evaluation metrics we adopted.
B.1 Synthetic Condition Corruption
We mainly studied four types of condition corruption in this paper. with two datasets. For IN-IK, we used random symmetric and asymmetric condition corruption. For CC3M, we adopted text swapping and LLM re-writing corruption. We used several levels of corruption
{0, 2.5, 7.5, 10, 15, 20 %.
Symmetric Condition Corruption for IN-1K. To introduce symmetric condition corrpption in IN-IK according to a corruption ratio we randomly sample a (x. y) pair from the dataset, and flip y another class according the class prior in IN-1K to obtain y", until the ratio of y satisfies 1)-
Asymmetric Condition Corruption for IN-1K. For asymmetric condition corruption of IN-1K, we first find the class overlap between IN-1K and CIFAR- using WordNet [54], denoted as C-0100 --0C-10 Then, we randomly sample (x, 3) from the data subset whose y satisfies E IN-IK and flip into the remaining classes of the overlapped set C/C- IN-1K/ /y-
Text Swapping Condition Corruption for CC3M. For CC3M, where y is text captions for the images, we randomly sample two pairs and swap the text of these two pairs to introduce condition corruptions. This mainly follows Chen et al. [48]. where very disruptive corruption is introduced.
LLM Text Condition Corruption for CC3M. Text swapping corruption may not be common in practice for image-text datasets. Instead, we may encounter captions that have unmatched entities or partially unmatched sentences with the images. To study the text corruption in more realistic scenarios, we use GPT-4 and prompt it to re- we-ir the captions to introduce corruptions. We pre-define 5 levels of corruption in the prompt, and randomly sample a level as input to GPT-4.
B.2 Automatic ImageNet-100 Annotation
Here, we present the details of annotate ImageNet- 100 for personalization of LDMs using ControlNet and T2I-Adapters. A few examples of the annotated images and captions are shown in Fig. I0
100
Canny Edge. For canny edge, we directly use the Canny detector from OpenCV to annotate the images. We set the low threshold and high threshold of canny detector to 100 and 200 respectively.
Segmentation Mask from SAM. We use SAM to annotate segmentation masks from IN-100 images. We directly use the colormap of the segmentation masks as input control to ControlNet and T21-Adapters.
Captions from BLIP. We use BLIP captioning model to generate captions for IN-100 for adapting text-conditional LDMs.
33

---

Graun
SAM
There ia
"Thers x grosing ground the wrods
dew bushvs"
There kasters Iodae"
standing
srail bad
laynng the ground in the grass
BLIP
edge of
Figure 10: Annotation examples of IN-100.
B.3 LDM Pre-training Setup
The pre-training setup of LDM-4 mainly follows Rombach al. 19. shown in Table 3 For LDM-4 models, we use a VQ-VAE [56] with down-sampling factor and a latent space with shape 64 64 3. It also has vocabulary size of 8196.
Table Hyper-parameters of IN-IK class-conditional and CC3M text-conditional LDMs.
IN-IK 256
256
CC3M 256 256
Down-sampling Factor
4
4
Latent Shape
64 64 x
64 64 x
Vocabulary Size
8192
8192
Diffusion Steps
1000
1000
Noise Schedule
I Lincar
Linear 400M
U-Net Param. Size
400M
Condition Net
Class Embedder
BERT
Channels
192
192
Channel Multipler
1,2,3,5
1,2,3,5
Number of Heads
1
Batch Size
64
64
Training Iter.
178K
396K
Learning Rate
le-4
le-4
IN-1K. The hyper-parameters of training IN-IK class-conditional LDMs are summarized as follows. We use U-Net with channels of 192 and channel multipliers of 1, the denoising network backbone. We use class embedding, i.e., embedding layer, for computing the embeddings of class labels. The conditional embedding injected to the U-Net with cross-attention. We use DDPM with linear schedule of 1000 steps. The batch size is set to 64 per GPU, and the learning rate is set to le-4 Training IN-1K LDMs for 178K iterations takes about 2.5 days
NVIDIA A100.
CC3M. We use same U-Net as denoising network backbone. We adjust the training iterations to 396K iterations for CC3M, which takes 7.5 days to train 8 NVIDIA A100. We use pre-trained BERT model beettbbas-unncase) for the conditional embeddings, and is fully trainable.
B.4 DIT Pre-training Setup
We pre-train DiT-XL/2 on IN-IK follows Peebles et 11] The hyper-paramesd are shown in Table 4 We train DiT-XL/2 for 400K training iterations using per GPU batch size of 32 on NVIDIA A100, which takes around 2.5 days. Compared to LDM-4, DiT-XL/2 used fine-tuned VQ-VAE with down-sampling factor of 8, a latent space of shape 32 x 32 x 4, and vocabulary size of 16384. DIT-XI/2 has a denoising network backbone based on Transformer architecture and uses Adaptive LayerNorm, initialized with zeros, for injecting the conditional information.
34

---

Table 4:
of IN-IK class-conditional 12/2.
DiT-XL/2 IN-IK 256 256
Down-sampling Factor
Latent Shape
32 x 32
Vocahulary Size
16384
Purams.
675M
Training Iters.
400K
Batch Size
32
Learning Rate
lc-4
B.5 LCM Pre-training Setup
LCM distills the pre-trained Stable Diffusion models to enable faster inference with fewer steps. We choose Stable Diffusion v1.5 as the teacher model and conduct distillation for 35K iterations, which takes 1.5 days on 8 NVIDIA A100 GPUs. We learning rate of le-5.
B.6 ControlNet and T2I T2I-Adapter Personalization Setup
We use the implementation of ControlNet and T2I-adapter of Diffusers |551 for downstream personalization tasks. Default learning rate and batch size from Diffusers are used for these two methods, and we set the training epochs for IN-100 as 10. On NVIDIA V100 GPUs, training ControlNet and T2I-Adapter with LDM-4 takes about hours.
B.7 Evaluation Metrics
We introduce the details of metrics we used to evaluate the diffusion models here. Due to the known difficulties of evaluating generative models, we adopt most of the existing criteria to evaluate the models we have trained.
Fréchet Inception Distance (FID) 633 FID measures the distance between real and generated images the feature space an ImageNet-IK IK pre-trained classifier [16), indicating the similarity and fidelity of the generated images to real images.
sFID 7
sFID utilizes the mid-level features of the inception network [166], which more sensitive to spatial variability.
Inception Score (IS) (64. IS also measures the fidelity and diversity of generated images. It consists of two parts: the first part measures whether each image belongs confidently to single class of an ImageNet- IK pre-trained image classifier [156] and the second part measures how well the generated images capture diverse classes.
Precision and Recall [65]. The real and generated images are first converted to non-parametric representations the manifolds using k-nearest neighbors, on which the Precision and Recall can be computed. Precision is the probability that a random generated image from estimated generated data manifolds falls within the support of the manifolds of estimated real data distribution. Recall is the probability that random real image falls within the support of generated data manifolds. Thus, precision measures the general quality and fidelity of the generated images, and the recall measures the coverage and diversity of the generated images.
Top-1% % Relative Mahalanobis Distance (RMD) Score 667
RMD score measures the sample complexity and difficulty. It is defined as the difference between the Mahalanobis distances of sample induced by the class-specific and class-agnostic Gaussian distributed estimated from the generated data. Given the dataset ((x,, ys) he(()) we first compute the features using the CLIP ViT. B-16 encoder from the images as Gxx). The class-specific Gaussian distribution is then estimated:
P(G(x) y k) N (G(x) Mg. Σ)
Σ G(x))
1 = N kgg=
(38)
Σ
1 Σ
Σ
(G.):) - pu) (G -w))
N
Py.=
35

---

The alass-agnostic Gaussian distribution is estimated over all data as;
P(G(x)) = N (G(x)
Magn Daga)
N
N Σ ((x))
Mugn
(39)
Σ agg
Σ (G(x;) - Maga)) (G Maga))
N
The RMD is defined as:
RMD (Xi, yii) M (xi, Wi) Magn (2it)
M ( y4) (x,)
Myy)
Σ-1 1 (G (xx) Mly))
(40)
Magn (x4) (G Magn)
Σ Eng agn
(G (x,) Maga)
We compute the RMD score for all generated images, and report only the top-1% % of them.
Average Top-5 L2 Distances. As an additional metric of sample diversity, compute the L2 distance of each generated image with the top-5 nearest neighbor training images. To reduce computation requirement of searching over the raw pixel space,
the CLIP ViT-B-16 image encoder [157 to transform images into the feature space before calculating the L2 distance. This metric measures the distance of generated samples with training images, as a proxy evaluation of diversity and memorization.
TopPR F1 72). TopPR is set of reliable evaluation metrics with statistically consistent estimates of generated data and real data. We use the F1 score, computed from the TopPR Precision and Recall as additional metric evaluate the general quality and diversity of generated images.
CLIP Score [66]. CLIP score measures the cosine similarity between the CLIP embedding of an image-text pair. It is widely used as a metric to evaluate the fidelity and alignment of the generated images and the conditional text prompts [
Memorization Ratio [73]. We compute the memorization ratio as the percentage of generated images whose L2 distances with their nearest neighbor training images are below pre-defined threshold. We compute the distances in the feature space of CLIP ViT-B-16 16 image encoder [157] due to the massive size and resolution of the training images and set the threshold as 12. Although there are several studies using the distance comparison between the first and second nearest neighbor as reflection of we found that this metric is not effective for large-scale datasets.
memorization 158 159
Entropy. We compute the entropy metric within the latent space of the pre-trained VQ-VAE [58] 56 Since LDMs (and DiT) learn the data distribution from the latent space of VQ-VAE,
compute the sample entropy H [x)] using the generated and flatten latent vector x E IR NW x and the codebook C of VQ-VAE, where H and W are the height and weights of the original latent vectors, D indicates the dimension of the latent space, and C' denotes the number of embeddings of the codebook. We compute the prohability of each latent vector as Softmax (||x CI3//)), where is a temperature parameter controlling the sharpness of the probability. We compute entropy as:
N HW C
E [[H())] = NHW ΣΣΣ
Softmax ([|x(1) - -C112/T
(41)
C Full Results of Pre-training Evaluation
In this section, present all results of our pre-training evaluation, over different diffusion models, including LDM-4, DiT-XL/2, and LCM-v1.5. and various types of condition corruption.
C.1 Quantitative Results
We present the full evaluation results of IN-IK class-conditional LDMs and CC3M text- conditional LDMs in Fig. [T and Fig. T2 respectively. All the results are com- puted from using set of guidance scales. For IN-IK LDMs, use s
36

---

300
200
0.70
0.70
1-125
RID
200
0.55
80
390
100
050
0.0-
50
5 Guicance Soale
Guidance Scale
Guidance Scaue
Gabdines Stole
(a) FID
(b) IS
(c) Precision
(d) Recall
(7)
200
Mon 3
2910
Tog
15
3.0
Calls
20+
020
280
Guidanno Scala
Guidsnco Seale
Guidanco Scale
Guidecoe Stale
(c) sFID
(f) TopPR F!
g Top-1% RMD
(h) Mem. Ratio
1.33
LES
1166
0..3
C1 Al
0.35
2.35
Gotddnnce Stale
Guidezze Stale
(i) Avg La Dist.
CLIP Score
Figure 11:
Qualitative evaluation results of 50K images generated by class-conditional LDMs pre- trained on ImageNet-IK with synthetic corruptions. The images are generated with various guidance scales using IK class conditions and compared with 50K validation images of ImageNet-IK.
8.0, (1.5, 1.75, 2.0, 2.25, 2,5 4.0, 6.5, 7,5 10.0). For CC3M LDMs, we use E (1.5, 2.0, 2.5, 2.75, 2,75,,30 3.0, including FID, IS, Precision, Recall, sFID, TopPR F1, and CLIP score, we can all observe that slight condition corruption makes LDMs perform better, with improved image quality and diversity. We also observe that, when there condition corruption in the dataset, the memorization ratio based on Le distances actually decreases, in line with observations in Gu al. [158].
6.0. 8.0, 10.0} For all the metrics,
By default we use DPM scheduler for generating the images with 50 inference steps. But we also study the generation of DDIM scheduler with 250 inference steps, as adopted in [9]. Due to the computation cost of running DDIM scheduler for 250 steps, we only study with IN-1K LDM-4.
The results are shown in Fig. 13 One can observe the same trends from the metries using DPM and DDIM, demonstrating our findings are scheduler agnostic.
We then show the pre-training results of DiT-XL/2 and LCM-v1.5 in Fig. 19 where we primarily compute the FID, IS, Precision, and Recall. For DiT-XL/2, we use 2.0, E {1.5, 2.0,, 2.25, 3.0, 4.5,
For LCM-v1.5, we use s E (1.5, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0, 5.5, 6.0, 6.5, 7.0, 7.5, 8.0, 9.0, 10.03. Slight condition corruption also facilitates the performance by using the most suitable guidance scale.
We additionally include the FID and IS trend along training for LDM IN-1K model with no corruption and 2.5% corruption, using a guidance scale of 2.5, as shown in Table 5] Slight corruption begins to be effective the very early stage of training.
Finally, we present the results of LDMs pre-trained on CC3M with LLM corruption and IN-1K with asymmetric corruption, where similar observations still hold.
37

---

01700
0100
1-135
PID
0.40
0.00
0.30
0.0-
0:30
2.00
Guidance Scale
Grndance Scale
Guddance Soal
Gridancs Stale
(a) FID
(b) IS
(c) Precision
(d) Recall
0.50
0.40
200
0:30
RMD
Momnetatten
Zo
0.00
20
0.31
0.1
20
Guidanoe Scala
Guidsnco Scale
Guidanco Scale
Guidinn Seale
(c) sFID
(f) TopPR F!
(g) Top-1% RMD
(h) Mem. Ratio
L.S.
11:0
111 Seamn
2,35
Gutdunnce Stale
Guidenze Stale
(i) Avg Lg Dist.
CLIP Score
Figure 12: Qualitative evaluation results of 50K images generated by class-conditional LDMss pre- trained on CC3M with synthetic corruptions. The images are generated with various guidance scales using 5K text conditions from MS-COCO and compared with validation images of MS-COCO.
0:00
0U20
Rn
22 00
000
0650
[2)
30
0.00
Guidizoe Stule
Guidinoe Scule
Guidance Stale
Guidanee Stale
(a) FID
(b) IS
(c) Precision
(d) Recall
Figure 13: Qualitative evaluation results of 50K images generated by dlass-conditions LDMs pre- trained on ImageNet- KK with synthetic corruptions. The images are generated with various guidance scales using IK class conditions and compared with 50K validation images of ImageNet-1K. We use DDIM scheduler with 250 inference steps for these results.
C.2 Qualitative Results
present more visualization results of class-conditional LDM-4 Fig. 18 class-conditional DiT- XL/2 in Fig. 19, text-conditional LDM-4 in Fig. 20 and text-conditional LCM-v1.5 in Fig. 21 One
can observe that DMs pre-trained with slight condition corruption in general more visually appealing images.
38

---

100
250
Piessiie:
Eccall
22
D.30
168
060
22 23 50 00 Gubdince Stale
133 23 25
40
231
25 10
23 25 10
Cuzdance Stale
Guidunze Stale
Guiduzor Stale
(a) FID
(b)
(c) Precision
(d) Recall
Figure 14: Qualitative evaluation results of 50K images generated by class-conditional DiT-XL/2 pre-trained on IN-IK with synthetic corruptions. The images are generated with various guidance scales using 1K class conditions from IN- 1K and compared with validation images of IN-1K.
bq.
10.2
0.30
2
D63
0.22
060
Goideco Scale
Guédince Stade
Guidhree Stale
Guidsece Stade
(a) FID
(b) IS
(c) Precision
(d) Recall
Figure 15: Qualitative evaluation results of 50K images generated by text-conditional LCM-v1.5 pre-trained on CC3M with synthetic corruptions. The images are generated with various guidance scales using 5K text conditions from MS-COCO and compared with validation images of MS-COCO.
D Full Results of Downstream Personalization Evaluation
We present complete results of downstream personalization here.
D.1
Quantitative Results
We show the results of ControlNet IN-IK LDM-4 in Fig. T2I-Adapter IN-1K LDM- 4 in Fig.
23 ControlNet CC3M LDM-4 in Fig. 24 and T21- CC3M LDM-4 in Fig. 25 For personalization experiments, we compute the results for both Canny and
SAM spatial controls. Guidance scales of (1.25, 1.5, 2.0, 2.25, 2.5, 3.0, 4.0, 5.0, 6.0, 7.0) and (2.0, 3.0, 4.0, 5.0, 6.0, 6.5, 7.0, 7.5, 8.0, 8.5, 9.0, 10.0) are used for IN-IK models and CC3M mod- els, respectively, for all experiments here.
From the results, one can observe that models pre-trained with slight condition corruption also present the best performance in downstream personalization tasks.
D.2 Qualitative Results
We present the qualitative comparison of ControlNet personalization results here. Since T2I-Adapter personalization results are similar but visually worse (quantitatively worse too), we skip their results. The visualizations of ControlNet IN-IK L.M-44 with Canny and SAM conditions are shown in Fig. 26 and Fig. 27 respectively. The visualizations of ControlNet CC3M LDM-4 with Canny and SAM conditions are shown in Fig. 28 and Fig. 29 respectively. Similarly, models pre-trained with slight condition corruption present the best image quality.
39

---

Table 5: FID and IS along training of LDM IN-IK with guidance scale 2.5.
10K
25K
50K
75K
100K
125K
150K
FID
0
71.48
52.02
20.88
14.49
12.66
10:44
10.12
2.5
77.94
51.59
21.16
12.24
9.25
8.98
0
4.86
23,49
71,26
93.85
103.27
164.41
170.2
2.5
13.66
24.40
64.27
97.11
109.39
167.21
175.83
34
1-11
1.4
12
FiD
Reaill
0.53
0.20
-01
91-
0:17
Guidenzee Scale
Guidinze Scale
Guidance
Guisaaze
FID
IS
Precision
(d) Recall
Figure 16: Qualitative evaluation results of 50K images generated by text-conditional LDMs pre- trained on CC3M with LLM re-wir ce-writing corruptions. The images are generated with various guidance scales using 5K text conditions from MS-COCO and compared with validation images of MS-COCO.
E
Full Results of Conditional Embedding Perturbation
E.1
Qualitative Results
More visualizations of CEP, compared with clean and IP pre-trained are shown here. We present the results of IN-IK LDM-4 and DiT-XL/2 Fig. 30(a) and Fig. 30(b) respectively. We also present more results of CC3M LDM-4 and LCM-v1.5 in Fig. (aa) and 3 respectively. CEP generally helps DMs generate more visually appealing and realistic images. We also show the more personalization visualization in Fig. 32 and Fig. 33
E.2
Ablation Study
In Fig. S(a) compute the Le distance of perturbed condition embeddings and the clean ones, as a measurement for the corruption levels (of CEP). Here, we elaborate how we compute the L2 distances. For fixed corruption, caleulate the distances
N
ca- (vf) - Ca- (y.)112.
(42)
N
where 8" is learned from clean data. For CEP, we directly calculate the L2 norm of sampled noise:
(43)
E.3 Comparison with Dropout and Label Smoothing
Here, we additionally compare CEP with dropout and label smoothing on LDM IN-1K models, which alternatives that also introduce perturbations in class embeddings. The results shown in T One can observe that, both dropout and label smoothing have similar regularization effects on training diffusion models, whereas CEP-U and CEP-G more effective.
E.4 Comparison with Fixed and Random Corruption
We further compare with fixed CEP corruption, and random data corruption, to study the effects of fixed and random perturbation to train diffusion models. For fixed CEP-U, we first select the samples
40

---

15.
250
110
22.3
225
0.50
200
Pio Booiaa
Ricill
(1:30
[03)
220
0
Crbdunce Stale
Guidance Stale
Guidunce Stale
Guiduzer Stale
(a) FID
(b)
(c) Precision
(d) Recall
Figure 17: Qualitative evaluation results of 50K images generated by conditioall LDMs pre- trained on IN-IK with asymmetric corruptions. The images are generated with various guidance scales using 1K class conditions from IN- 1K and compared with validation images of IN-1K.
Table 6: Comparison of CEP with dropout and label smoothing LDM IN-1K.
Corruption
FID
IS
Clean
9.44
138.46
+ Dropout 0.1
8.67
145.80
Label Smoothing 0.1
8.49
146.27
CEP-U
7.00
170.73
CEP-G
6.91
180.77
to add perturbation, and then fix them during training. For random data corruption, we randomly choose samples during training to make their label noisy by flipping to other classes. From the results Table we show that CEP works best among all corruption methods. Also fixed CEP is more effective than adding data corrpption (fixed and random). Random data corruption can be viewed as a CaP-variant with embeddings from flipping label instead of adding noise, and thus also more effective than fixed data corruption.
Table 7: Comparison of fixed and random corruption on LDM IN-IK.
Corruption
FID
IS
Clean
9.44
138.46
CEP-U
7.00
170.33
Fixed CEP-U
7.94
154.48
Random Data Corruption
8.13
143.07
Fixed Data Corruption
8.44
140.27
41

---

Clean
2.5%
= 55
7.5%
? = 10%
= 15%
7J 20%
Plarmggan
Carbonara
Geyser
Indigo Bunting
Bald Eagle
Cockatoo
Monarch
Gibboon
Container hpp
Tree Frag
Mud Mud Turtle
Figure 18: Visualization of LDMs IN-IK pre-training results.
42

---

Clean
= 2.5%
= 5%
n 7.5%
= 10%
= 15%
тJ 20%
Tiger Shark
Ostrich
Junco
Redbone
Schooner
Vase
Wing
Yurt
Artichoke
Alp
Seashore
Figure 19: Visualization of DiT-XL/2 IN-IK pre-training results.
43

---

Clean
7
2.5%
7 5%
7 7.5%
10%
?J 15%
20%
A plate topped with different types of food table
"A yellow train is at the station along with tracks
"A suburban street with few houses on the block"
'A man S nnwwboard down the side of snow-covered slope"
'hhee is unmade bed Victorian style room
"A picture a salad and pizza being served"
"A kitchen with wooden walls and cabinets and metal sink
"A train traveling down towards some snow covered mountains"
"An empty street has the lights reflecting off of
Figure 20: Visualization of LDMs CC3M pre-training results.
44

---

Clean
7 2.5%
7 5%
7 7.5%
10%
7j 15%
r
20%
"This is full of yellow bananas"
vase with flowers on it"
"A large fancy chandelier hanging over room full of furmitue
CAR Cu
'A blue motorcycle is parked on the side of a street"
"A beach chair with umbrela beach by the water"
"A teddy bear s itting near a rail over looking crowded beach"
"A car and city bus sitside by side on the street"
A wood bench sitting in lush green field"
"A bunch jajakets hanging on the all in room"
Figure 21: Visualization of LCM CC3M pre-training results.
45

---

24
0.50
0.20
0000
Recall
52.5
0.20
0030
00.0
0.30
47.2
235
12
Guidenco Seale
Galdunce
Guidurce
Guideree
(a) Canny, FID
(b) Canny, IS
(c) Canny, Precision
(d) Canny, Recall
-3
yobli yoitil
07.2
10:0
-5
10.0
Pretiiin
- pm1.222
32 00.7
D.a0
63.2
12.7
0:0
Guidinse Scale
Cuiduace Soale
Guidunce Stale
Guiducee Stale
(e) SAM, FID
(1) SAM, IS
(g) SAM, Precision
(b) SAM, Recall
Figure 22: Qualitative evaluation results of 5K images generated by class-conditional LDMs pre- trained on ImageNet-IK K and personalized on ImageNet-100 using ControlNet. We personalized the models with different control styles, including canny ((a) (d)), segmentation mask from SAM ((e) (h)), and lineart (i) (I)) The images generated using 100 class conditions with various guidance scales, compared with 5K validation images of ImageNet-100.
will
50
1-13
0.50
14-
10.00
Roeall
P1D
30
12
LES
LEX
36-
20%
D/
Guiducoo Scale
Guidanoo Scale
Guidunce Stale
Guidunce Stale
(a) Canny, FID
(b) Canny, IS
(c) Canny, Precision
(d) Canny, Recall
15
135
16-
FiD
131
10
0.20
50
-21
009
10
30
30
Guidizoe Stale
Guidinoo Scale
Guidance Stale
Guidince Scale
(C) SAM, FID
(f) SAM, IS
(g) SAM, Precision
(b) SAM, Recall
Figure 23: Qualitative evaluation results of 5K images generated by class-conditional LDMs pre- trained on ImageNet-1K K and personalized on ImageNet- using T21-Adapter. We personalized the models with different control styles, including canny ((a) (d)), segmentation mask from SAM ((e) (h)), and lineart (ii ()), The images generated using 100 class conditions with various guidance scales, compared with 5K validation images of ImageNet-100.
46

---

--velt
0163
2355
34-
1377
Fil
82
Purbion
0.50
Recall
301
qu 0.30
25-
100
-0
0:30
Guideroo
Guidenoo Scale
Guidurce
Guiduree
(a) Canny, FID
(b) Canny, IS
(c) Canny, Precision
(d) Canny, Recall
151
Dud0
1100
022
Poetiton
FI11
30
80.00
-0:
15
0:47
-3:
Guidinse Scale
Guidinse Scale
Guidunee Stale
Guiducee Stale
(e) SAM, FID
(i) SAM, IS
(g) SAM, Precision
(b) SAM, Recall
Figure 24: Qualitative evaluation results of 5K images generated by text-conditional LDMs pre- trained on CC3M and personalized ImageNet-100 100 using ControlNet. We personalized the models with different control styles, including canny ((a) (d))and segmentation mask from SAM ((e) (h)). The images generated using text captions annotated from BLIP with various guidance scales, compared with 5K validation images of ImageNet-100 100.
154
160
12
0:33
Rocall
Du43
LES
044
501
0:33
Guiducso Scale
Guidanor Scale
Guidance Stale
Guiduace Stale
(a) Canny, FID
(b) Canny, IS
(c) Canny, Precision
(d) Canny, Recall
51
00-
10.00
10
101
26-
05
24
24-
Guidizee Stale
Guidinoee Scale
Guidance Stale
Guidince Scale
(C) SAM, FID
(f) SAM, IS
(g) SAM, Precision
(b) SAM, Recall
Figure 25: Qualitative evaluation results of 5K images generated by text-conditional LDMs pre- trained on CC3M and personalized on ImageNet-100 using T21-Adapter. We personalized the models with different control styles, including canny ((a) - (d)) and segmentation mask from SAM ((e) (h)). The images generated using text captions annotated from BLIP with various guidance scales, compared with 5K validation images of ImageNet-100.
47

---

Control
Clean
= 2.5%
? 5%
1 10%
?) 15%
?i 20%
Lorikeet
Rock Crab
Shih-Tzu
Slipstick
Rocker
Figure 26: Visualization of LDMs IN-IK ControlNet Canny personalization results.
Control
Clean
2.5%
1) = =5%
10%
15%
1| 20%
Coyote
Hyena Dog
Cinema
CE
Pedestal
Rocker
XXXX
Figure 27: Visualization of LDMs IN-1K ControlNet SAM personalization results
48

---

Control
Clean
17 2..5%
= =5%
7 10%
15%
= 20%
"Brightly colored bird with red beak and green body sitting branch
"There is yellow and black vacuum marble floor"
Arafed horse pulling a cart with man sitting on it"
"There are two old cars parked on the side of the street"
Figure 28: Visualization of LDMs CC3M ControlNet Canny personalization results.
Control
Clean
1 = 2.5%
= 5%
77 =00%
= 15%
= 20%
"Thee dog that is standing the grass with trisbee
"There is statue of person the steps of building"
"Arafed monkey sitting on a tree branch in a zoo"
"Araffy pasta with bacon and parsley in white bowl"
Figure 29; Visualization of LDMs CC3M ControlNet SAM personalization results
49

---

Clean
IP
CEP-U
CEP-G
Clean
IP
CEP-U
CEP-6
While Shark
Electric
Cock
Robin
Srail
Counhound
Cricket
Dam
Narfolk
Promentory
Arti
Sea Cucumber
Lakeside
Hare
Vieduct
CIFF
CIR
(a) IN-IK LDM-4
(b) IN-IK DIT-XL/2
Figure 30: Visualization of CEP on IN-IK pre-trained LDM-4 and DiT-XL/2
50

---

Clean
IP
CEP-U
CEP-G
Clean
IP
CEP-U
CEP-G
"A crowded city under the blue blue
'A woman sits on her motorcycle in the street"
TTrain traveling through railyard with trees lining sides"
"A red and white plane flylng undera blue sky"
"A meal o reuben sandwich with fries and coleslaw."
*An outside cafe setting with blee, challs, umbrelas and tree for shading
"A bedroom scene with focus the bed"
"A puppy playing with an tris bee fenced yard
*A that has white flowess in
"A living room with brick faced fireplace*
steel industrial hen lightng
*A sitting on top of toilet bathroom
'A bowl broccoli, noodles and other various foods
piece of chocolate with chocolate drizzlee
"A herd of sheep grazing open field
"A dog lying in with head on pillow"
(a) CC3M I.M-4
(b) CC3M LCM-v1.5
Figure 31: Visualization of CEP on CC3M pre-trained LDM-4 and LCM-v1.5
51

---

Control
Clean
CEP-U CEP-G
Robin
Saluki
Pickup
Robssere
Rotisserie
Lobster
Figure 32: Visualization of CEP on ControlNet adapted IN-1 pre-trained LDM-4
Control
Clean
CEP-U
rArafed spiral staircase window building"
'There is bug the ground"
'There duck that is swimming the water"
"A close up a spider on web black and white photo"
"There is a large boat that is sailing in the water"
Figure 33: Visualization CEP on ControlNet adapted CC3M pre-trained LDM-4
52

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We claimed contribution in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitations in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
53

---

Justification:
Full proofs are shown in Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All model setups, hyper-parameters, inference details shown in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
54

---

Answer: [Yes]
Justification:
We used public available data and all code will be released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pippe.cc/pullic/guiii wwwtc//ggue////eeeeceoocc CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details in main paper and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Mainly becuase computational cost.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
55

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All details shown in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp::/ httpa//ppp...........ctcccccttta
Answer:
[Yes]
Justification:
All research follows NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed at the end of Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
56

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Pre-trained models will be released reuqest and will be equipped with safery checkers.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided relcasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All are cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwpsspipeaattteeeeee thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
57

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
58


Reviews:
[{"review_id": "X7YhSYHxFL", "reviewer": "Reviewer_mxHS", "summary": "This paper claims that slight corruption of the condition is beneficial for conditional diffusion models. Experimentally, they conducted experiments with label flips or text swaps in the dataset and observed the performance improvement over zero noise at noise levels of 2.5% to 5%. Theoretically, they used a Gaussian mixture model to show that generation diversity and quality, as measured by entropy and Wasserstein distance, can be improved with a small amount of state corruption. Based on this, they proposed a method for injecting small amounts of Gaussian noise into the condition embeddings and experimentally demonstrated its superiority.", "questions": "Please see the Weaknesses part.", "limitations": "They mentioned in the last section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Their claim offers a new perspective that may not be intuitive, but has been partially mentioned in the existing literature. They provide a solid experimental and theoretical explanation for this view.", "weaknesses": "* From my understanding, in Theorem 2, $\\gamma$ is $O(1/\\sqrt{n})$, which is the scenario for small corruption. It would be beneficial to address this explicitly in the main text.\n\n* It would be beneficial if the method that determines the noise level could be linked to the theory like using the $\\gamma=O(1/\\sqrt{n})$,.\n\n* They briefly mentioned the various assumptions in their theory as limitations. It would be helpful to provide a detailed explanation of these assumptions and the scenarios in which they hold.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time reviewing this paper. We now address your questions as follows.\n\n---\n\n> From my understanding, in Theorem 2...It would be beneficial to address this explicitly in the main text.\n\nThank you for your suggestion. We have already mentioned that the order of $\\gamma$ is  $O(\\frac{1}{\\sqrt{\\max_k n_k}})$ in Theorem 2 and we  will further emphasize in the main text (page 7, lines 219-220) that the noise level needs to satisfy $\\gamma = O(\\frac{1}{\\sqrt{\\max_k n_k}})$.\n\n\n> It would be beneficial if the method that determines the noise level could be linked to the theory like using the [...]\n\n\nIn the experimental section of Section 5.2, the (Gaussian) noise we used is approximately $0.04\\epsilon$, where $\\epsilon \\sim N(0,I)$, while the theoretical part in Section 4 proves that the noise level is approximately $0.03 \\epsilon$. The noise levels in the experiment and the theory are very close. We will further emphasize their link in the main text (page 8, line 247).\n\n\n> They briefly mentioned the various assumptions in their theory as limitations. It would be helpful to provide a detailed explanation of these assumptions and the scenarios in which they hold.\n\nOur theoretical part includes the main setup that the data distribution follows a mixture of Gaussian distributions, which is a common assumption in the theoretical analysis of diffusion models [1,2,3,4,5]. Follow the previous work [2,6], we accordingly considered a piece-wise linear function as the denoising neural network, which has been proven sufficient to handle the Gaussian mixtures. Based on these previous theoretical works, our theoretical framework, although seems simple, provides a paltform for understanding the improvements in quality and diversity introduced by noised label embeddings.\n\n   [1] Shah, Kulin, Sitan Chen, and Adam Klivans. \"Learning mixtures of gaussians using the DDPM objective.\" *Advances in Neural Information Processing Systems* 36 (2023): 19636-19649.\n\n   [2] Chen, Sitan, Vasilis Kontonis, and Kulin Shah. \"Learning general gaussian mixtures with efficient score matching.\" *arXiv preprint arXiv:2404.18893* (2024).\n\n   [3] Li, Puheng, et al. \"On the generalization properties of diffusion models.\" *Advances in Neural Information Processing Systems* 36 (2024).\n\n   [4] Cui, H., Krzakala, F., Vanden-Eijnden, E., & Zdeborová, L. (2023). Analysis of learning a flow-based generative model from limited sample complexity. arXiv preprint arXiv:2310.03575.\n\n   [5] Li, Yangming, Boris van Breugel, and Mihaela van der Schaar. \"Soft Mixture Denoising: Beyond the Expressive Bottleneck of Diffusion Models.\" *The Twelfth International Conference on Learning Representations*.\n\n   [6] Gatmiry K, Kelner J, Lee H. Learning mixtures of gaussians using diffusion models[J]. arXiv preprint arXiv:2404.18869, 2024.\n\n---\n\nIf you find our response helpful, please consider raising the score for better supporting our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dp9JAiPol3", "reviewer": "Reviewer_6StB", "summary": "In this paper, the authors study the effect of slight corruptions to the training data during pretraining of conditioned diffusion models. They introduce the perturbations to the \"condition\" in the diffusion models and show that this leads to a better model (via FID and other metrics).  They also provide an intuition on why corruptions help by theoretically studying this case on a GMM.", "questions": "- Does the optimal noise perturbation level transfer between the models? If 10% is the best noise level for LDM on one dataset, would it be the best noise for the same-size LDM on another?\n- How do metrics look like throughout training for models trained with different levels of noise?\n- CC3M corruption details are not clear. I went to Appendix B1, you mentioned 5 levels of corruption, but it is still vague! What are these 5 levels? \n- How is the number of training iterations determined for the models? I looked at the appendix and made some rough calculations, it seems like you trained the IN model for 91 epochs and CC3M for 67 epochs (Pls correct me if I'm wrong). On what basis are these numbers decided? This again ties back to the question, when is \"adding noise\" helpful? Are we in an overtraining regime? Is that the reason why adding corruption acts as \"augmentation\" and hence the metrics improve?\n\n- (Minor) A few relevant citations are missing. For example, adding perturbations to the conditioning both in text space and latent space is explored in the context of memorization in last year's Neurips paper [1].\n\n[1] - \"Understanding and mitigating copying in diffusion models.\" Advances in Neural Information Processing Systems 36 (2023): 47783-47803.", "limitations": "n/a", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Well-written paper\n- All the experiments are well thought out, especially the downstream application experiments.\n- Multiple metrics are evaluated and shown that perturbation helps, which makes the claim more sound.", "weaknesses": "- While authors showed extensively that adding noise can help, they missed to show \"when does it help\"? For example, if I'm a practitioner who wants to train a large model from scratch, I usually will not have the budget to train multiple models to determine the amount of noise that needs to be added. That insight is missing in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the reviewer's constructional feedback on this paper.\nWe now address the weaknesses and questions as follows.\n\n---\n\n> While authors showed extensively that adding noise can help, they missed to show \"when does it help\"? For example, if I'm a practitioner who wants to train a large model from scratch...That insight is missing in this paper.\n\nThanks for mentioning this important question. \nHowever, we would like to highlight that core insight from our empirical and theoretical study is *not* to find the \"optimal\" noise ratio in the pre-training dataset that can help.\nInstead, we aim to present insights that, to pre-train diffusion models (and self-supervised models), it is important to ensure the diversity in the pre-training and slight corruption in the pre-training dataset may benefit diversity. \nNo over-filtering on corruption is needed for pre-training data.\n\nInsights for practitioners: the proposed CEP can be simply adapted into pre-training of diffusion models to improve performance, no matter the datasets and models. In Figure 8 (b), we also presented that, on noisy datasets (of different levels) as in practical settings, CEP can also facilitate both the pre-training and downstream transferring performance. \n\n\n> Does the optimal noise perturbation level transfer between the models? If 10% is the best noise level for LDM on one dataset, would it be the best noise for the same-size LDM on another?\n\nThis is indeed an intriguing question. While our empirical study shows that slight corruption universally improves the performance across models and datasets, we would like to mention that it also depends on the pre-training dataset size.\nAs the dataset size scales larger, the slight corruption ratio may also decrease, since we may not expect 10% of our pre-training data of size 2B is corruption.\n\n\n> How do metrics look like throughout training for models trained with different levels of noise?\n\nThanks for mentioning this important question. \nWe use the pre-trained LDM-4 IN-1K clean and noise $2.5$ to generate images at different checkpoints along training, using a guidance scale of $2.5$. \nWe present the FID and IS results as follows:\n\nFID:\n| $\\eta$ (%) \t| 10K \t| 25K \t| 50K \t| 75K \t| 100K \t| 125K \t| 150K |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:| :---:|\n| 0 \t| 71.48 \t|  52.02\t| 20.88 \t| 14.49 \t|  12.66\t| 10.44 | 10.12 |\n| 2.5 \t| 77.94 \t| 51.59  \t| 21.16 \t| 13.08 \t| 12.24 \t| 9.25  \t| 8.98 |\n\nIS:\n| $\\eta$ (%) \t| 10K \t| 25K \t| 50K \t| 75K \t| 100K \t| 125K \t| 150K |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:| :---:|\n| 0 \t| 14.86 \t| 23.49 \t| 71.26 \t| 93.85 \t| 103.27 \t| 164.41 \t| 170.22 |\n| 2.5 \t| 13.66 \t| 24.40  \t| 64.27 \t|  97.11\t| 109.39 \t| 167.21 \t| 175.83 |\n\nFrom the results, we can observe that, with slight corruption, the model performs slightly worse at the beginning of the training (before 50K iter.), while becomes outperforming the clean one towards the end of the training.\n\nWe will include these results in our Appendix. \n\n\n> CC3M corruption details are not clear. I went to Appendix B1, you mentioned 5 levels of corruption, but it is still vague! What are these 5 levels?\n\nSorry for the confusion caused. For CC3M, to introduce corruptions, we randomly sample two image-text pairs, i.e. $(I_1, T_1)$ and $(I_2, T_2)$, and swap the text of these two pairs to make it $(I_1, T_2)$ and $(I_2, T_1)$. \nFor corruption levels, we use the same levels, i.e. {2.5, 5, 7.5, 10, 15, 20}%, as in ImageNet. \nThese are shown in the legend of CC3M figures and mentioned at line 112.\nWe will add more details in Appendix B1 and highlight in the main text to make this more clear. \n\n\n> How is the number of training iterations determined for the models?...On what basis are these numbers decided? \n\nThe number of training iterations mainly follows settings in the previous papers. \nFor example, in LDM paper [1], the main settings of LDM-4 on ImageNet and CC3M is 178K and 390K steps.\nTo replicate this, we used a total batch size of 512 in our experiments, resulting in roughly 2.5K and 5.4k training steps per epoch. Thus we use 71 and 70 epochs of IN-1K (1.28M samples) and CC3M (2.8M samples due to expired url) to make the total training iteration as 178K and 390K.\nThis is similar for DiT and LCM, where we follow their paper to setup the training steps. \n\n> \"this again ties back to the question, when is \"adding noise\" helpful? Are we in an overtraining regime? Is that...and hence the metrics improve?\"\n\nFrom the above results, one can see that, adding noise starts being helpful once the model converges to generate reasonable figures and consistently improves along training after that. \nFrom our paper and ablation results for Reviewer XQzf, you can also see that CEP performs better than other regularization methods, including input perturbation (augmentation), dropout on embedding, label smoothing, etc. \nThis demonstrates the better effectiveness of CEP than traditional regularization.\n\n> A few relevant citations are missing...last year's Neurips paper [1].\n\nThanks for mentioning this relevant work. We will add this work into discussion in our revised paper. Theie differences are:\n\nThis work mainly focuses on memorization and copying of diffusion models.\nThe authors found adding perturbations to condition or condition embedding help relieve the copying issue (but FID remains similar as clean, possibly due to the small-scale dataset and focus on fine-tuning instead of pre-training), which aligns also with our findings as shown in Figure 11 (i) and Figure 13 (i), where large L2 distance of generated images and ground truth are present with pre-training corruption.\n\n---\n\nWe hope the above response can resolve your concern. \nIf you find our revise and response helpful, please consider raising the score to better support of our work.\nAlso please let us know if there is any more question.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "htrD6tYupY", "reviewer": "Reviewer_45qi", "summary": "The paper investigates the impact of slight corruption of conditioning\ninformation in pre-training data on the performance of diffusion models (DMs).\nBy introducing synthetic corruption to ImageNet-1K and CC3M datasets, the study\nevaluates over 50 conditional DMs. Empirical and theoretical analyses reveal\nthat slight corruption enhances the quality, diversity, and fidelity of\ngenerated images. Based on these insights, the work proposes Conditional\nEmbedding Perturbations (CEP) as a method to improve DM training which shows\nsignificant improvements in both pre-training and downstream tasks.", "questions": "None", "limitations": "The authors correctly reflect the limitations of the study. In particular, the\ntheoretical analysis is limited due to the assumptions made, but can still shed\nsome light on the underlying mechanism.  Further, the general issue of\nevaluating image generative models quantitatively is pointed out and\na limitation of the work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- **Clarity, very well written:** The paper is clearly structured and builds up\n   the motivation for the proposed method step by step from experiments and\n   theoretical analysis.\n - **Extensive experiments:** The study examines many conditions training\n   multiple diffusion models with different noise conditions from scratch.\n   This gives a detailed perspective on the performance benefits of using\n   slight conditioning corruption during training and is a valuable\n   addition to the field.  These experiments are evaluated both qualitatively and quantitatively.\n - **Unexpected insight:** The study well describes a generally unexpected\n   phenomenon and shows how it can be leveraged to develop an new method.", "weaknesses": "- **Notion of significance:** The paper claims that slight pre-training\n   corruption yield significantly better performance in terms of FID and IS\n   metrics. Nevertheless, this notion of significance is never formally tested.\n   For this at least a experiment with multiple repetitions should be performed\n   to determine statistical significance.\n - **Typos:**, Page 8, lines 257 and 259 rrecision -> precision, Page 9, lines 299 dataset -> data\n - **Link between theoretical analysis and metrics** should be made explicit.\n   I.e. pointing out that the FID is essentially a Gaussian approximation\n   based estimated for the 2-Wasserstein distance of inception features (preferably at page 7, line 223).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer's acknowledgment on this paper.\nWe now address the weakness raised as follows.\n\n---\n\n> Notion of significance: The paper claims that slight pre-training corruption yield significantly better performance in terms of FID and IS metrics. Nevertheless, this notion of significance is never formally tested. For this at least a experiment with multiple repetitions should be performed to determine statistical significance.\n\nThanks for this great suggestion. \nPlease understand that we will not be able to perform pre-training experiments for multiple repetitions, simply due to the un-affordable cost for that. \nThis is also common in most study of generative models where only a single run is conducted in pre-training.\nAlthough we are not able to run multiple rounds of pre-training, for IN-1K models, we conducted an experiment for generating images with guidance scale of 2.25 of 5 random seeds, and reported average FID and IS with their standard deviation. \nAlso, we conducted a t-test between the results from the clean model and noisy models and reported the p-value here.\nThe results demonstrate that the improvement is indeed significant (extremely significant in terms of the p-value). \n\n| $\\eta$ (%) \t| FID \t| FID p-value \t| IS \t| IS p-value \t|\n|:---:|:---:|:---:|:---:|:---:|\n| 0 \t| 9.79 (0.11) \t|  \t|  138.42 (0.35)\t|  \t|\n| 2.5 \t| 8.49 (0.06) \t| < 0.0001 \t|  155.73 (0.34)\t| < 0.0001 \t|\n| 5 \t| 8.60 (0.07) \t| < 0.0001 \t|  146.77 (0.29)\t| < 0.0001 \t|\n| 10 \t| 9.11 (0.19)\t| = 0.0001 \t|  144.13 (0.86)\t|  < 0.0001\t|\n\n\nDue to the limited discussion time, we will try to include more significant test of other models and downstream applications for the final version of our paper. \n\n\n> Typos:, Page 8, lines 257 and 259 rrecision -> precision, Page 9, lines 299 dataset -> data\n\nThanks for pointing out the typos! \nWe have fixed all of these in the paper.\n\n> Link between theoretical analysis and metrics should be made explicit. I.e. pointing out that the FID is essentially a Gaussian approximation based estimated for the 2-Wasserstein distance of inception features (preferably at page 7, line 223).\n\nThank you for your suggestion. We will emphasize on page 7, line 223 of the manuscript that the FID metric used in the experiments is the 2-Wasserstein distance under the assumption that the inception features follow a multivariate Gaussian distribution, to highlight their connection.\n\n--- \n\nIf you find the above response help resolve your concerns, please consider raising score for better support of our work.\nThanks!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t9lKo47Mpz", "reviewer": "Reviewer_XQzf", "summary": "The authors find that slight corruption in the conditioning of text-to-image diffusion models improves the performance. The authors theoretically analyze this empirical finding in a toy model where the goal is to learn to sample from a Gaussian Mixture and the network is piece-wise linear function. Inspired by their experimental and theoretical findings, the authors propose a technique to improve the performance of text-conditional diffusion models by adding noise to the embedding of the conditioning vector.", "questions": "Apart from the weaknesses mentioned above, I have one more important question regarding this paper. Could the authors please clarify whether the corruption in Sections 3 and Sections 5 happens once for each data sample before the training or whether a new corruption takes place whenever we encounter the same data point across different epochs? To clarify, I want to understand which one of the two is happening:\n\n1) We take a dataset, we corrupt it once, we train with the corrupted dataset\n2) We use the clean dataset and every time we see a sample we corrupt it (by adding a different noise each time to its embedding or by changing each label/text to something else each time we see the same example across epochs).\n\nIf 2) is happening, it seems to me that the proposed method is closely related to regularization. I think this needs to be clarified and the effect of doing 1) or 2) should be studied as an ablation.", "limitations": "The authors have adequately discussed the limitations of their work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The research topic is timely. State-of-the-art diffusion models are trained on billions of image-text pairs. Many of those are potentially noisy.\n* The authors propose a very simple modification to the training of text-to-image diffusion models that leads to improved performance.\n* The finding that slight corruption in the conditioning helps is interesting.\n* The authors validate their findings through an extensive experimental evaluation.", "weaknesses": "* I believe that the authors should emphasize in the title of their paper and in the main body that they study corruption in the conditioning. Numerous works study corruption in the images themselves, e.g. see:\n\n1) Ambient Diffusion: Learning Clean Distributions from Corrupted Data\n2) Consistent Diffusion Meets Tweedie: Training Exact Ambient Diffusion Models with Noisy Data\n3) GSURE-Based Diffusion Model Training with Corrupted Data\n4) Solving inverse problems with score-based generative priors learned from noisy data\n\nIn these works, it has been observed that data corruption decreases the performance of diffusion models. I believe the authors should acknowledge these relevant works and clarify that their paper analyzes corruption in the conditioning.\n\n* The theoretical model is not very relevant, at least not very relevant to Section 3 of the paper. In the experiments of Section 3, the authors mislabeled some of the training examples.  In the theoretical model, it is assumed that the true class is given to the model since the model is using the parameters that correspond to the true class. In a sense, the theoretical model of Section 3 is more related to the proposed algorithm in Section 5. Also, the existence of multiple centers in Section 4 does not seem relevant, unless I am missing something. This is because the model has separate parameters for each center and the centers are given for each example the model sees. I went over the proof and it looks like this is exploited in the proof as well, to obtain a closed-form solution for the optimal network. \n* For the reasons mentioned above, it seems to me that the reason this works is some sort of regularization. I wonder if a similar effect could be obtained by increased dropout in the conditioning or some other form of regularization. Overall, I believe that the reader of the paper doesn't develop much intuition about the origins of the experimental finding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer's efforts on this paper and his constructional suggestions on the ablation study of CEP. \nWe now address the concerns as the follows.\n\n---\n\n> I believe that the authors should emphasize...\n\nThanks for this suggestion.\nWe will make condition perturbation more clear in our title and main paper.\nWe will also include these relevant works into discussion in our revised paper.\n\n> The theoretical model is not very relevant...more related to the proposed algorithm in Section 5. \n\nIn the theoretical part, we considered a more general setting [1,2], where the label embedding is perturbed by Gaussian noise. We proved that slight corruption in pre-training data can lead to more diverse and higher quality generation in diffusion models. Additionally, the results from the theoretical part also inspire and justify the algorithm proposed in Section 5.\n\nThe theoretical framework in Section 4 can also be applied to other types of noise. For example, in Section 3, label flipping can be regarded as a special corrupted label embeddings, i.e., $\\mathbf{c}(y^c) = \\mathbf{c}(y)+ \\boldsymbol{\\xi}$, where the noise $\\boldsymbol{\\xi}$ are vectors with entries 0, 1, and -1. Assuming there are $K$ classes, and each class has an equal sample size (approximately 1300 of each class in the experiment), each label has a probability $p$ of being flipped.Then the noise $\\boldsymbol{\\xi}$ follows a distribution satisfies $\\mathcal{F}(p\\frac{K-2}{K}\\mathbf{1},(p-p^2(\\frac{K-2}{K})^2)\\mathbf{I})$. After obtaining the mean and variance information of the noise  $\\boldsymbol{\\xi}$ , the corresponding optimal linear denoising network and the generative distribution can still be solved using the techniques in Section 4.\n\n[1] Hu W, et al. Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee. ICLR.\n\n[2] Tang Y H, et al. Detecting label noise via leave-one-out cross-validation. arXiv preprint 2021.\n\n> Also, the existence of multiple centers in Section 4 does not seem relevant, unless I am missing something.\n\nThe multiple centers in Section 4 are derived from the assumption that the data follows a mixture of Gaussian distributions, which is a common assumption in the theoretical analysis of diffusion models [1-4]. The multiple centers correspond to the multiple classes of the real data. We consider modeling each center as a simplification of the real problem, and this simplification is sufficient to illustrate how noisy labels affect the training process of the denoising function (Section A.2), thereby influencing the final generative distribution (Section A.3), which shows improvements in both quality and diversity compared to the distribution generated with clean labels (Section A.4).\n\nAlthough the theoretical part considers a simplified single-center version, no other theoretical work simultaneously consider both the training and generative processes of diffusion models, and their interaction, while precisely solving the generative distribution. Our theoretical contributions should not be overlooked. We also agree that employing a more practical theoretical model could improve the theoretical part, which is left for our future study direction.\n\n[1] Shah, Kulin, et al.. \"Learning mixtures of gaussians using the DDPM objective.\" NeurIPS 2023.\n\n[2] Chen, Sitan, et al. \"Learning general gaussian mixtures with efficient score matching.\" *arXiv preprint.\n\n[3] Li, Puheng, et al. \"On the generalization properties of diffusion models.\" NeurIPS 2024.\n\n[4] Li, Yangming, et al. \"Soft Mixture Denoising: Beyond the Expressive Bottleneck of Diffusion Models.\" ICLR.\n\n> For the reasons mentioned above...finding.\n\nThanks for pointing out this point. We also think CEP can be viewed as a regularization method for diffusion training.  By adding small noise during the training process, CEP prevents the trained model from overly collapsing onto the training data, thereby improving the diversity and quality of the generation distribution. \n\nThrough ablation study of other methods, we showed that CEP is superior to other methods (dropout on conditional embedding and label smoothing of class labels for computing conditional embedding) and may function beyond regularization.\n\n| $\\eta$ (%) \t| FID \t| IS \t|\n|:---:|:---:|---:|\n| LDM-4 IN-1K \t| 9.44 \t| 138.46 \t|\n| + Dropout 0.1 \t| 8.67 \t| 145.80 \t|\n| + Label Smoothing 0.1\t| 8.49 \t|  146.27\t|\n| + CEP-U \t|  7.00\t| 170.73 \t|\n| + CEP-G \t|  6.91\t| 180.77 \t|\n\nWhile the regularization methods like Dropout and Label Smoothing can also improve the performance of diffusion, CEP is significantly more effective than these methods.\nWe will include this ablation study into the Appendix of our paper.\n\n> Could the authors please clarify whether the corruption in Sections 3 and Sections 5 happens...\n\nIn section 3, for our empirical study, we adopt fixed corruption, i.e., take a dataset, introduce corruption, and train with the corrupted dataset.\nIn section 5, the proposed CEP is introduced randomly at each training iteration. \nHowever, for results shown in Figure 8 with noisy dataset, we still used the fixed corrupted dataset.\n\n| Corruption \t| FID \t| IS \t|\n|:---:|:---:|:---:|\n| None \t| 9.44 \t| 138.46 \t|\n| CEP-U\t| 7.00 \t| 170.33 \t|\n| Fixed CEP-U \t|  7.94\t| 154.48 \t|\n| Random Data Corruption (2.5%) \t|  8.13\t| 143.07 \t|\n| Fixed Data Corruption (2.5%)\t| 8.44 \t|140.27  \t|\n\nHere we compare these two types of corruption.\nWe showed that CEP works the best among all corruption methods.\nAlso fixed CEP is more effective than adding data corruption (fixed and random).\nRandom data corruption can be viewed as a CEP-variant with embeddings from flipping label instead of adding noise, and thus is also more effective than fixed data corruption.\nWe will include this analysis in our Appendix. \n\n---\nIf you find our revise and response helpful, please consider raising the score for better support of our work.\nWe are open to discuss more if any question still hold.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We first would like to thank all reviewer and AC's efforts and time reviewing this paper and suggestions for making it better.

According to the reviewers' feedback, we summarize several critical points and make a general response here.

---

> Motivation for corruption in pre-training.

- First of all, the motivation to study the problem of pre-training conditional corruption is well-supported by existing research showing that diffusion models are trained on massive (and potentially corrupted) pre-training data, which naturally motivates us to study *what* are the effects of the pre-training corruption, and *whether* and *why* the corruption in pre-training datasets can help.
- Second, the motivation of developing theories are also motivated by our experimental findings. We tie this advantage with the diversity of the learned distribution which is also closer to the ground truth distribution through a theoretical analysis on the mixture of Gaussian.
- Third, from the methodology level, finding an optimal noise ratio to corrupt the pre-training data is not our focus and it is also naturally impossible due to the cost of pre-training. One motivation to design our CEP approach is not to find the optimal noise ratio, but to mimic the pre-training corruption, to improve the diffusion pre-training. We show that CEP generally improves the performance and can also be applied in practical datasets with corruption in it already. 

> Connection between theoretical and empirical findings. And Assumptions in our theoretical analysis.

While we consider a general form for noise in the theoretical analysis part, we demonstrated in the response to Reviewer XQzf that the fixed label noise can be included in this general form too. 
We also showed in the response to Reviewer mxHS that the corruption level used in CEP is pretty close to our theoretical analysis. 

In Section 3, we mainly consider a mixture of Gaussian model for our analysis. 
This helps us derive clearer form to analyze the effect of noise. 
The multi-centroids distribution of GMMs also well aligns with our insights in this paper, i.e., the slight corruption increases the diversity in the learned distribution.


> CEP functions as a regularization method.

Reviewer 6StB and Reviewer 45qi have raised questions regarding the regularization effects of the proposed CEP. 

To show this, we have conducted additional ablation study of:

* FID and IS along training with slight corruption (response to Reviewer 6StB): we demonstrated that, with slight corruption, the noisy model becomes better than the clean one at early training. 
* Comparison of CEP with Dropout and label smoothing on conditional embedding (response to Reviewer 45qi): we demonstrated that CEP is more effective than Dropout and label smoothing as a regularization method. 
* Comparison of fixed data point CEP and random data corruption (response to Reviewer 45qi): we showed that fixed CEP is more effective than random data corruption and fixed data corruption. Also, random data corruption can be viewed as a CEP-variant, which is more effective than fixed data corruption.

Although CEP can also be viewed as a regularization method, through these experiments, we demonstrated its superiority and effectiveness over other methods.

Furthermore, we would like to highlight that while regularization in diffusion modeling is a broad research area, several works in the past have primarily focused on settings limited to only fine-tuning domain [1] and exclusively for inverse problems only [2,3]. We aim to emphasize the practicality of our approach in the context of large pre-training setups which differs from prior work. 

[1] Tang, Wenpin. "Fine-tuning of diffusion models via stochastic control: entropy regularization and beyond." arXiv preprint arXiv:2403.06279 (2024).

[2] Chung, Hyungjin, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. "Improving diffusion models for inverse problems using manifold constraints." Advances in Neural Information Processing Systems 35 (2022): 25683-25696.

[3] Mardani, Morteza, Jiaming Song, Jan Kautz, and Arash Vahdat. "A variational perspective on solving inverse problems with diffusion models." arXiv preprint arXiv:2305.04391 (2023).

Meta-review: In this work, authors present an interesting finding that slightly corrupting the training data used to train diffusion models can actually lead to better performance - they demonstrated it on models trained using ImageNet and CC-3M and also proposed a new algorithm which can be integrated in any standard diffusion model training paradigm. Reviewers acknowledged the fact that the paper is well-written, experiments are robust and show the efficacy of the proposed algorithm. Compute budgets permitting,  I found the work to be rigorous and liked the emphasis on an under-explored aspect within the whole diffusion model training literature and can be beneficial for the field to further explore.

I recommend acceptance - as a suggestion, I'd like to see how this technique can be integrated into large-scale diffusion model training where it can be expensive to tune the noise scale.
Decision: Accept (spotlight)