Title: Slight Corruption in Pre-training Data Makes Better Diffusion Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: pre-training noise

diffusion models

latent diffusion models

diffusion transformers

latent consistency models
Abstract: Diffusion models (DMs) have shown remarkable capabilities in generating realistic high-quality images, audios, and videos. 
They benefit significantly from extensive pre-training on large-scale datasets, including web-crawled data with paired data and conditions, such as image-text and image-class pairs.
Despite rigorous filtering, these pre-training datasets often inevitably contain corrupted pairs where conditions do not accurately describe the data. 
This paper presents the first comprehensive study on the impact of such corruption in pre-training data of DMs.
We synthetically corrupt ImageNet-1K and CC3M to pre-train and evaluate over $50$ conditional DMs. 
Our empirical findings reveal that various types of slight corruption in pre-training can significantly enhance the quality, diversity, and fidelity of the generated images across different DMs, both during pre-training and downstream adaptation stages. 
Theoretically, we consider a Gaussian mixture model and prove that slight corruption in the condition leads to higher entropy and a reduced 2-Wasserstein distance to the ground truth of the data distribution generated by the corruptly trained DMs.
Inspired by our analysis, we propose a simple method to improve the training of DMs on practical datasets by adding condition embedding perturbations (CEP).
CEP significantly improves the performance of various DMs in both pre-training and downstream tasks.
We hope that our study provides new insights into understanding the data and pre-training processes of DMs.

Slight Corruption in Pre-training Data Makes Better Diffusion Models

Abstract

Diffusion models (DMs) have shown remarkable capabilities in generating realistic high-quality images, audios, and videos. They benefit significantly from extensive pre-training on large-scale datasets, including web-crawled data with paired data and conditions, such as image-text and image-class pairs. Despite rigorous filtering, these pre-training datasets often inevitably contain corrupted pairs where conditions do not accurately describe the data. This paper presents the first comprehensive study on the impact of such condition corruption in pre-training data of DMs. We synthetically corrupt ImageNet-1K and CC3M to pre-train and evaluate over 50 conditional DMs. Our empirical findings reveal that various types of slight corruption in pre-training can significantly enhance the quality, diversity, and fidelity of the generated images across different DMs, both during pre-training and downstream adaptation stages. Theoretically, we consider a Gaussian mixture model and prove that slight corruption in the condition leads to higher entropy and a reduced 2-Wasserstein distance to the ground truth of the data distribution generated by the corruptly trained DMs. Inspired by our analysis, we propose a simple method to improve the training of DMs on practical datasets by adding condition embedding perturbations (CEP). CEP significantly improves the performance of various DMs in both pre-training and downstream tasks. We hope that our study provides new insights into understanding the data and pre-training processes of DMs and all models are released at https://huggingface.co/DiffusionNoise .

1 Introduction

Recently, diffusion models (DMs) have been demonstrating unprecedented capabilities in generating high-quality, realistic, and faithful images [1-5], audios [6, 7], and videos [8]. In addition, they exhibit impressive conditional generation results [9-11] when trained with classifier-free guidance [12]. The successes of DMs are often attributed to the massive pre-training on large-scale datasets consisting of paired data and conditions [13-17], which also empowered and facilitated numerous downstream applications and personalization of pre-trained models, such as subject-driven generation [18, 19], controllable conditional generation [20-22], and synthetic data training [23-25].

The large-scale pre-training datasets of paired data and conditions are usually web-crawled. For example, Stable Diffusion [26] was pre-trained on LAION-2B [17], which contains billion-scale image-text pairs collected from Common Crawl [27]. Despite the heavy filtering mechanisms used in collecting pre-training datasets [17, 28], they still inevitably contain corrupted pairs where conditions do not correctly describe or match the data, such as corrupted labels and texts [29-32]. While large-scale datasets are necessary for DMs to perform well, the corruption may lead to unexpected

'Painting  of a young

badger delicately

sniffing a yellow rose'

Clean

Slight

Severe

Samoyed

Catamaran

Valley

'A bundle  of blue

and yellow

flowers in a vase'

'An astronaut  riding

horse in the forest,

water in front lilies'

LDM-4 IN-1K

DiT-XL/2 IN-1K

LDM-4 CC3M

LCM-v1.5 CC3M

'A llama wearing

sunglasses  with Earth

in the  background'

14.91

Avg. FID

96.64

14.23

98.07

16.98

64.98

behavior or generalization performance of models [33-35] during both pre-training and adaptation stages, especially for safety-critical domains such as healthcare [36] and autonomous driving [37, 38].

Conventional wisdom may suggest that training under corrupted conditions could lead to deterioration in performance. For example, Noisy Label Learning [39-42, 29, 43, 44] aims to improve the generalization of models when training with corrupted labels. Label-noise robust conditional generative adversarial nets [45, 46] and DMs [47] have also been studied. However, these works are primarily concerned with supervised learning in downstream scenarios with assumptions of high noise ratios and the same training and testing data distributions. Due to the misalignment with large-scale self-supervised pre-training in practice on filtered datasets with relatively smaller noise ratios, the effects of corruption in pre-training can also differ from those in downstream [48, 49].Understanding the effects of pre-training with such corruption is challenging and still remains largely unexplored.

In this paper, we provide the first comprehensive and practical study on condition corruption in the pre-training of DMs. Through indepth analysis, we empirically, theoretically, and methodologically verify that slight condition corruption in pre-training makes better DMs . We pre-train over 50 class-conditional and textconditional DMs using classifier-free guidance (CFG) [12] on ImageNet-1K (IN-1K) [50] and CC3M [14] with synthetically corrupted conditions, i.e., classes and texts, of various levels. Our study covers a wide range of DM families, including Latent Diffusion Model (LDM) [9], Diffusion Transformer (DiT) [11], and Latent Consistency Model (LCM) [51, 52]. Due to the known obstacles of evaluating generative mod-

(a) FID (

↓

)

(b) IS (

↑

)

LDM-4 IN-1K

DiT-XL/2 IN-1K

LDM-4 IN-1K (Asym.)

+LDM-4 CC3M

→LCM-v1.5 CC3M

+LDM-4 CC3M (LLM)

20

150

15

100

49

10

50

30

0

10

20

0

10

20

Corruption Ratio (%)

Corruption Ratio (%)

els [53-55], we conduct both pre-training and downstream evaluation from the perspectives of image quality, fidelity, diversity, complexity, and memorization, to comprehensively understand the effects of pre-training corruption of DMs. More specifically, for pre-training, we directly evaluate the images generated from the pre-trained models, and for downstream adaptation, we evaluate on the images generated using personalized models with ControlNet [20] and T2I-Adapter [21] from the pre-trained ones. In addition, we theoretically investigate how slight corruption in conditional embeddings benefits the training and generative processes of DMs. Our key findings include:

Empirically, slight corruption in pre-training facilitates the DMs to generate images with higher quality and more diversity, both qualitatively (in Fig. 1) and quantitatively (in Fig. 2).

Theoretically, we employ a Gaussian mixture model to show slight condition corruption improves the diversity and quality of generation by increasing entropy over clean condition generation and reducing the quadratic 2-Wasserstein distance to the true data distribution (in Section 4).

Methodologically, based on our analysis, we propose a simple method to improve the pre-training of DMs by adding conditional embedding perturbations (CEP). We show that CEP can significantly boost the performance of various DMs in both pre-training and downstream tasks (in Section 5).

Going beyond images, we do see the potential of this study in other modalities. Our efforts may also inspire future investigation on other types of corruption and bias inside pre-training datasets. We hope that our work can shed light on the future research of diffusion models and responsible AI.

2 Preliminary

Denoising Diffusion Models. DMs are probabilistic models that learn the data distribution P ( x ) , with x denoting the observed data 3 , over a set of latent variables z 1 , . . . , z T with length T [1, 57]. It assumes a forward diffusion process, gradually adding Gaussian noise to the data with a fixed Markov chain: q ( z t | x ) = N ( √ ¯ α t x , (1 -¯ α t ) I ) , which can be re-parameterized as z t = √ ¯ α t x + √ 1 -¯ α t ϵ with ϵ ∼ N ( 0 , I ) and ¯ α t as constants produced by a noise scheduler. DMs are trained via the reverse process, inverting the forward process as: p θ ( z t -1 | z t ) = N ( µ θ ( z t ) , Σ θ ( z t )) , with a network that predicts the statistics of p θ . Setting Σ θ ( z t ) = (1 -¯ α t ) I to untrained constants, the reverse process is simplified as training equally weighted denoising autoencoders ϵ ( z t , t ) with uniformly sampled t :

After training, new images can be generated by sampling z t -1 ∼ p θ ( z t -1 | z t ) starting with N ( 0 , I ) .

Classifier-free Guidance (CFG). Extra condition information y , such as class labels and text prompts, can be injected into DMs with conditional embeddings c θ ( y ) from modality-specific encoders [9] for conditional generation: p θ ( z t -1 | z t , c θ ( y )) . CFG [12] jointly learns a unconditional model ϵ θ ( z t , t, c θ ( ∅ )) with an empty condition y = ∅ and a conditional model ϵ θ ( z t , t, c θ ( y )) , and combines them linearly to control the trade-off of sample quality and diversity in generation:

where s > 1 denotes the guidance scale. We adopt CFG by default with the training objective:

Condition Corruption. Ideally, each y should accurately describe and match x . However, in practice, due to errors from the collection of web-crawled datasets, conditions y c may un-match x . We define ( x , y c ) as pairs with condition corruption, and assume that c θ ( y c ) = c θ ( y ; η, ξ ) , where ξ denotes certain noise and η denotes corruption ratio that implicitly controls the noise magnitude.

3 Understanding the Pre-training Corruption in Diffusion Models

In this section, we conduct the first comprehensive and practical study on pre-training DMs with condition corruption. Through holistic exploration with synthetically corrupted datasets, we reveal a surprising observation that slight pre-training corruption can be beneficial for DMs.

3.1 Pre-training Evaluation

Pre-training Setup . Here, we adopt Latent Diffusion Models (LDMs) [9] with the pre-trained VQ-VAE [58, 56] and a down-sampling factor of 4 for the latent space of observed data x , denoted as LDM-4. More specifically, we train class-conditional and text-conditional LDM-4 from scratch on synthetically corrupted IN-1K [50] and CC3M [14], respectively, with a resolution of 256 × 256 . We use a class embedding layer and a learnable pre-trained BERT [59] to compute the conditional embeddings of the IN-1K class labels and the CC3M text prompts. To introduce synthetic corruption into the conditions, we randomly flip the class label into a random class for IN-1K, and randomly swap the text of two sampled image-text pairs for CC3M, following [48, 49] (other corruption types studied in Section 3.3). We train models with different corruption ratios η ∈ { 0 , 2 . 5 , 5 , 7 . 5 , 10 , 15 , 20 } % More details on synthetic corruption and pre-training recipes are shown in Appendix B.1 and B.3.

100

200

300

IS

10

15

20

FID

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(a) FID - IS, IN-1K

0

.

2

0

.

4

0

.

6

Recall

0

.

5

0

.

6

0

.

7

0

.

8

Precision

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(b) Prec. - Recall, IN-1K

26

27

28

29

30

CLIP Score

19

22

25

28

FID

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(c) FID - CS, CC3M

0

.

3

0

.

4

0

.

5

Recall

0.5

0.6

0.7

Precision

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(d) Prec. - Recall, CC3M

400

500

600

700

Top-1% RMD

0

2

4

6

8

Density

×

10

-

3

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(a) RMD, IN-1K

2

4

6

3

.

9

4

.

0

4

.

1

4

.

2

4

.

3

4

.

4

Sample Entropy

×

10

-

1

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(b) Entropy, IN-1K

400

600

800

Top-1% RMD

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Density

×

10

-

2

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(c) RMD, CC3M

2

4

6

8

4

.

15

4

.

20

4

.

25

Sample Entropy

×

10

-

1

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(d) Entropy, CC3M

Evaluation of Pre-trained Models . We directly use the pre-trained LDMs to generate images to study the effects of condition corruption in the pre-training stage. We use IN-1K class labels for class-conditional LDMs and MS-COCO text prompts [60] for text-conditional LDMs to generate 50K images and compare with the real validation images. The images are generated using a set of guidance scales s ∈ { 1 . 5 , 2 . 0 , . . . , 10 . 0 } and DPM [61] scheduler with 50 steps for faster inference speed 4 . We adopt Fréchet Inception Distance (FID) [63], Inception Score (IS) [64], Precision, and Recall [65] to evaluate the quality, fidelity, and coverage of the generated images. For CC3M models, we use the CLIP score (CS) [66] to measure the similarity of the generated images and conditional text prompts. From the perspectives of sample complexity and diversity, we compute the top1% Relative Mahalanobis Distance (RMD) [67, 68], calculated from the estimated class-specific and class-agnostic distributions of generated data, and the sample entropy [69, 70], calculated from the VQ-VAE codebook. We also adopt other metrics, including sFID [71], TopPR F1 [72], average L 2 distance, and memorization ratio [73]. More details of the metrics used are shown in Appendix B.7.

Results . We present the main quantitative results of pre-training in Fig. 3 and 4, and the qualitative results in Fig. 5. More results are shown in Appendix C. In summary, we found that slight pretraining corruption 5 can facilitate the quality, fidelity, and diversity of generated images :

Class and text-conditional models pre-trained with slight corruption achieve significantly lower FID and higher IS and CLIP score (Fig. 3(a) and 3(c)). They also present comparable and better Precision-Recall curves (Fig. 3(b) and 3(d)), compared to clean pre-trained models.

Models pre-trained with slight corruption generate images with higher complexity and diversity, with a right-shifted density of RMD (Fig. 4(a) and 4(c)), and larger entropy (Fig. 4(b) and 4(d)).

Qualitatively, models with slight corruption learn a more diverse distribution. Generated images present better variability in the circular walk around the latent space (Fig. 5(a) and 5(b)).

Goldfish

Lorikeet

Increasing  Corruption Ratio

Circular  Walk in Learned Latent  Space

(a) IN-1K

'A volcano with lava pouring  down its slopes'

'A photograph  of a modern kitchen  with appliances'

Increasing  Corruption Ratio

Circular  Walk in Learned Latent  Space

(b) CC3M

More corruption in pre-training can potentially lead to quality and diversity degradation. As η increases, almost all metrics first improve and then degrade. However, the degraded measure with more corruption sometimes is still better than the clean ones (e.g. IS and Entropy).

3.2 Downstream Personalization Evaluation

A common scenario of DMs pre-trained on large-scale datasets is that they can be personalized and customized for more controllable generation [74, 75] after tuning on smaller datasets. Here, we also examine the effects of pre-training condition corruption of DMs at downstream personalization tasks.

Downstream Personalization Setup . We personalize the pre-trained LDMs with two common methods: ControlNet [20] and T2I-Adapter [21]. Both methods can enable the pre-trained DMs to generate more controllable images according to input spatial conditioning. For fair comparison, we automatically annotate the ImageNet-100 (IN-100) dataset to canny edges using the OpenCV canny detector [76] and segmentation masks using SegmentAnything (SAM) [77], similar to Zhang et al. [20]. For text-conditional LDMs, we additionally use BLIP [78] to generate MS-COCO-style text prompts for IN-100. We then fine-tune the LDMs on the annotated training set of IN-100. More details on the annotation and personalization setup are shown in Appendix B.2 and B.6, respectively.

Evaluation of Personalized Models . After tuning, we evaluate the personalized LDMs in the annotated validation set of IN-100. We mainly compute FID, IS, Precision, and Recall to compare the models. We similarly use a set of guidance scales to generate images, but only report results of the best guidance scale in this part due to space limit. The complete results are shown in Appendix D.

Results . Similarly, from the main results in Fig. 6 and Fig. 7, we found that slight pre-training corruption can also benefit the quality of generated images at downstream personalization :

Slight pre-training corruption helps the personalized model generate images with lower FID (Fig. 6(a) and 6(c)) and higher IS score (Fig. 6(b) and 6(d)), whereas more corruption deteriorates.

Qualitatively, the personalization images from slight corruption pre-trained models also present more diversity, better fidelity with the input spatial controls, and higher quality.

3.3 Discussion: Other Types of Pre-training Corruption and Diffusion Models

Our previous studies mainly involve LDMs and symmetric random corruption. Here, we additionally study other types of corruption and DMs to verify that the above observations universally hold.

Condition Corruption . We consider asymmetric (Asym.) label corruption for IN-1K and Large Language Model (LLM) corruption for CC3M. For IN-1K, we introduce corruption only within the

ControlNet

Canny

ControlNet

SAM

T2I-Adapter

Canny

T2I-Adapter

SAM

10

12

14

16

FID

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

ControlNet

Canny

ControlNet

SAM

T2I-Adapter

Canny

T2I-Adapter

SAM

56

59

62

65

IS

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

ControlNet

Canny

ControlNet

SAM

T2I-Adapter

Canny

T2I-Adapter

SAM

20

30

40

50

FID

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

ControlNet

Canny

ControlNet

SAM

T2I-Adapter

Canny

T2I-Adapter

SAM

20

25

30

35

IS

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

Kuvasz

Lamp

Increasing  Corruption Ratio

Canny

SAM

ControlNet

T2I-Adapter

Bowl

Canny

American robin

SAM

'A cat is looking at the camera'

'A bird is sitting on a branch  in the grass '

'Boats are lined up on the shore of a lake in the fall '

'A crab is sitting in the sand'

Canny

SAM

Canny

SAM

Increasing  Corruption Ratio

ControlNet

T2I-Adapter

(a) IN-1K

(b) CC3M

overlapped classes with CIFAR-100 [79], while maintaining others as clean. For CC3M, we prompt GPT-4 [80] to corrupt the texts. More details of the corruption are shown in Appendix B.1.

Diffusion Models . LDMs utilize U-Net [81] as backbone and Cross-Attention for adding conditional information [9]. We pre-train class-conditional diffusion transformers on IN-1K for extra assessment, termed DiT-XL/2 [11], with Transformer [82] as backbone and adaptive LayerNorm [83-86] for conditional information. We also pre-train the recent text-conditional Latent Consistency Models (LCMs) [52, 51] on CC3M, which distill Stable Diffusion v1.5 [26] models to enable swift inference with minimal steps, noted as LCM-v1.5. Detailed setup is shown in Appendices B.4 and B.5.

Results . We present the main results in Fig. 2 due to the space limit. Full results are shown in Appendix C. We find that slight condition corruption of various types universally facilitates the performance of different DMs and consistently makes them outperform the clean pre-trained ones.

4 Theoretical Analysis

In this section, we theoretically analyze condition corruption and find that slight corruption prevents the generated distribution from collapsing to the empirical distribution of the training data and encourages coverage of the entire data space, thereby enhancing diversity and alignment with the ground truth. We present a concise overview here and provide a comprehensive analysis in Appendix A.

Data Distribution. We concentrate on the prototypical problem of sampling from Gaussian mixture models (GMMs). Specifically, we consider the distribution of data x ∈ R d that satisfies:

where y denote class labels of a finite set y ∈ { 1 , . . . , |Y|} . Given any class, x | y follows a Gaussian N ( µ y , I ) , and w y represents the weight of the Gaussian components which satisfies ∑ y ∈Y w y = 1 .

Denoising Networks and Condition Corruption. Inspired by recent works that also target on GMMs [87, 88] of DMs, we parameterize the denoising network as a piece-wise linear function:

where c ( y c ) is the one-hot encoding of corrupted label y c and { W k t , V k t } |Y| k =1 are trainable parameters. Specifically, following a line of existing work [89-91], we adopt a simpler label-noise model by adding Gaussian perturbation to the label embedding, perturbing the clean condition c ( y ) with standard Gaussian noise ξ to obtain c ( y c ) = c ( y ) + γ ξ . Here, the corruption control parameter γ corresponds to the corruption ratio η for a more direct noise magnitude control. While our theoretical framework focuses on Gaussian noise, it can also be extended to distributions such as uniform.

4.1 Generation Diversity: Clean vs. Corrupted Conditions

We employ entropy to evaluate the diversity of generated images, following Wu et al. [70]. Higher entropy suggests a wider spread of data, yielding greater diversity in generated images, while lower entropy implies a more concentrated distribution with less diversity. We present Theorem 1, showing the difference in entropy between generations with corrupted and clean conditions:

Theorem 1. For any class k ∈ Y and sufficiently large length T , assuming the norm of corresponding expectation ∥ µ k ∥ 2 2 is a constant and the empirical covariance of training data is full rank, let z T and z c T be the generation with clean and corrupted conditions respectively, then it holds that

where γ is the corruption control parameter and d is the data dimension.

The proof is provided in Appendix A.4.1. Theorem 1 indicates that for any class k , corrupted conditions enhance image diversity by increasing generation entropy. Moreover, with suitable γ values, image diversity can grow with noise, aligning with observations in Fig. 4.

4.2 Generation Quality: Clean vs. Corrupted Conditions

We then analyze why corrupted conditions benefit the quality of generated images, as also observed in Section 3.1. We employ the 2 -Wasserstein distance as a metric to evaluate the sampling error between the true and the generated distributions, with clean and corrupted conditions. A distributed generated closer to the real data distribution indicates better image quality [63]. In Theorem 2, we analyze the difference in the quality of data generated by corrupted DMs and clean ones:

Theorem 2. For any k ∈ Y and sufficiently large length T , assuming the norm of corresponding expectation ∥ µ k ∥ 2 2 is constant, let P , Q X and Q c X be the ground truth, clean, and corrupted condition distributions over training data X . Then if γ = O (1 / √ max k n k ) , it holds that

where W 2 ( · , · ) denotes the 2 -Wasserstein distance between two distributions, n k is the sample size of k -labeled dataset, and d is the data dimension.

Here the expectation is taken over the random sample of the training dataset from the data distribution. Detailed proof is shown in Appendix A.4.2. Theorem 2 reveals that for any class k , small corruption yields generation distributions closer to the true distribution than clean ones. This partially verifies that the generation quality of the uncorrputly trained diffusion model can be improved by adding slight corruption to the training data. This is also well consistent with our empirical observation in Section 3.1, where the noise we used is approximately 0 . 04 ϵ , close to the theoretical noise level of 0 . 03 ϵ , showing that the FID of the generated images can be improved with a small corruption.

5 Improving Diffusion Models with Conditional Embedding Perturbation

5.1 Method

Our previous analysis demonstrates that slight condition corruption in the pre-training could potentially benefit both the image quality and diversity of DMs, which inspires us to improve the

pre-training of DMs using this conclusion. In practice, it is usually infeasible to directly corrupt the conditions in the pre-training datasets either due to their large-scale nature or difficulties to select which conditions to corrupt. Instead, we propose to add the perturbation directly to the conditional embeddings of DMs, which is termed conditional embedding perturbation (CEP) . Compared to the fixed proportion of condition corruption in datasets we studied before, CEP adds perturbation to every data instance during training on the fly. Specifically, CEP slightly modifies the DM objective:

where δ denotes the perturbation added to conditional embeddings c θ ( y ) . We simply set the perturbation to Uniform, i.e., δ ∼ U ( -γ √ d I , γ √ d I ) , or to Gaussian, i.e., δ ∼ N ( 0 , γ √ d I ) , where the design of the factor γ √ d mainly follows previous works [82, 92-95], d denotes the dimension of c θ ( y ) , and γ controls the perturbation magnitude, mimicing the corruption ratio η . The main purpose of CEP is to learn better DMs with perturbation on relatively clean and heavily filtered datasets, such as CC3M and IN-1K studied in this paper, but it is also applicable to slightly corrupted datasets. Recently, Ning et al. [96] found that adding input perturbations (IP) to latent variables z t during the forward process also helps diffusion training by mitigating exposure bias [97]. Compared to IP, CEP does not alter the marginal data distribution, but encourages the learned joint distribution to be more diverse.

5.2 Experiments

Setup . We pre-trained previous class-conditional LDM-4, text-conditional LDM-4, class-conditional DiT-XL/2, and text-conditional LCM-v1.5 with CEP, and compare with IP and clean pre-trained ones. We use both Uniform and Gaussian perturbation, denoted as CEP-U and CEP-G, respectively. We set γ = 1 for all models, with an ablation study with class-conditional LDM-4 with different γ s.

We evaluated the pre-trained class-conditional models on IN-1K and and text-conditional models on MS-COCO with FID, IS, Precision, and Recall. Additionally, we personalize the pre-trained LDMs with ControlNet on IN-100 to validate the effectiveness of CEP pre-training at downstream.

Results . We present the pre-training results of CEP in Table 1. CEP significantly and universally improves the performance for different class and text-conditional DMs, e.g., 2.53 and 1.25 FID improvement, and 42.31 and 10.27 IS improvement of LDM-4 and DiT-XL/2. CEP also improves precision and recall of DMs. In contrast, IP only achieves marginal improvement and yields slightly worse precision. Adopting CEP in pre-training also benefits the personalization tasks, especially for text-conditional LDMs, with FID improvement

0

1

2

3

4

5

6

Cond. Emb.

L

2

Diff.

7

8

9

10

11

12

FID

Fixed Corrupt.

CEP-U

CEP-G

(a) Ablation of

γ

0

2.5

5

7.5

Corruption Ratio (%)

4

5

6

7

8

9

10

FID

Fixed Corrupt.

CEP-U

CEP-G

(b) Corrupted IN-1K

Figure 8: Ablation with LDM-4 IN-1K. (a) FID and average L 2 distance of conditional embeddings against clean ones with γ = { 0 . 1 , 0 . 5 , 1 . 0 , 5 . 0 , 10 . 0 } , indicated by square points (left to right). We compare with fixed synthetic corruption η = { 2 . 5 , 5 , 10 , 15 } % , shown by circle points. (b) CEP on corrupted IN-1K.

Clean

IP

CEP-U

CEP-G

Fountain

Boathouse

Teapot

Palace

'A beautiful

fall landscape

with a lake'

'A plant  grows at

the bottom of

a shallow  stream'

'A beautiful mountain

with sunset and

lake, highly  detailed'

'A panda bear

with  aviator glasses

on its head'

LDM-4 IN-1K

DiT-XL/2 IN-1K

LDM-4 CC3M

LCM-v1.5 CC3Ms

(a) Pre-training

Clean

IP

CEP-U

CEP-G

Control

'A white rabbit sitting in the snow on the ground'

'A chair with a cushion on it in a room'

Rabbit

Cabbage

(b) Personalization

of 6.08 and 7.02 for Canny and SAM spatial control, as shown in Table 2. Qualitatively, as shown in Fig. 9, images generated from DMs with CEP also look more visually appealing and realistic.

The ablation results of γ are shown in Fig. 8(a). We also compare the average L 2 distance of CEP and fixed corruption against the clean condition embeddings. Interestingly, one can observe that CEP achieves a lower FID with more corruption in the embedding space (larger L 2 from the clean ones), demonstrating its effectiveness. CEP is applicable to slightly corrupted datasets that we may often encounter in practice, as shown in Fig. 8(b), where it also facilitates the performance significantly.

In addition, we also compare the proposed CEP with traditional regularization methods, such as Dropout [98] and Label Smoothing [99], and study the effects of fixed and random perturbation during training in Appendix E.3 and Appendix E.4. The results show that CEP is more effective.

6 Related Work

Diffusion Models . Inspired by thermodynamics, DMs were first proposed by Sohl-Dickstein et al. [100]. DMs have soon been developed into image generation with a fixed Gaussian noise diffusion process [1, 101]. Various techniques have then been proposed for more effective and efficient DMs [102, 4, 5]. One of the most well-known is modeling the diffusion process at the latent space of pre-trained image encoders as a strong prior [58, 56], instead of raw pixels spaces [3, 9, 11], which allows for high-quality image generation with affordable inference speed. Numerous foundational DMs that generate photorealistic images have thus been built [103-107, 10, 108, 26, 109]. These powerful models are generally pre-trained on web-crawled billion-scale data with conditions (usually text), which may inevitably contain corruption [31, 110, 111, 32, 35]. Recently, consistency models [51, 112, 52] were also developed from DMs, allowing generation with much fewer inference steps. These foundational DMs also enabled many downstream applications [20, 21, 113-122]. However, the effects of the pre-training corruption on downstream applications remain unknown.

Learning with Noise . Learning with noise is a long-standing challenge [123-126]. Noisy label learning has been widely studied in classification, from noise correction [127, 40, 128-131, 41, 132, 133, 43, 134, 44, 135] and noise-robust loss functions [39, 136-141, 29, 142]. Learning with noise has also been studied in the context of generative models [143-146]. Robust GANs and DMs [45-47] alleviated the quality degradation and condition misalignment of training generative models with label noise. In contrast, we study a more practical scenario, where the models are trained on corrupted pre-training data with a low noise ratio, and then adapted to downstream tasks.

In fact, more aligned with our work, there are several recent studies on exploring and exploiting the pre-training noise. Chen et al. [48, 49] found that slight label noise in supervised pre-training can be beneficial for in-domain downstream tasks, whereas detrimental for out-of-domain tasks. NoisyTune [147], NEFTune [95], and SymNoise [148] found that introducing noise to the weights and embedding of pre-trained language models can facilitate downstream performance. Ning et al. [96] also found that adding perturbation in the forward diffusion process helps reduce the exposure bias of DMs [97]. Similarly, Naderi et al. [149] introduced noise into the input of image translation

networks for better learning with limited data. Synthetic data (potentially with corruption) have also been found to be useful in pre-training [24, 150]. We demonstrate that slight corruption in conditions of the pre-training DMs can also be beneficial at both the pre-training and downstream.

7 Conclusion and Limitation

We presented the first comprehensive study on condition corruption in pre-training of DMs. Our empirical and theoretical analysis surprisingly demonstrate that slight condition corruption benefits DMs in both the pre-training and downstream adaptation, based on which we proposed CEP as a simple yet general technique that significantly improves the performance of DMs. We hope our findings could inspire more future work on understanding the pre-training data of foundation models.

This work has the following limitations. First, due to a lack of computing resources, we cannot study all types of DMs on larger datasets. Second, the theoretical analysis is based on several assumptions that might be further explored in the future. Third, the evaluation of image generation remains an open question, and we used most of the existing criteria for fair comparison.

Disclaimer

While we study DMs for image generation in this paper, it is important to note that all generations have been selected and verified by human experts to ensure that they are responsible. Although we release all the pre-trained models under different corruption settings, it is possible that these models will generate inappropriate content due to the scale of pre-training and without alignment with human preferences. The main purpose of this research is to raise the awareness of the community on data cleaning and corruption in the research of diffusion models.

References

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations , 2021.

Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems , 34:11287-11302, 2021.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems , 34:8780-8794, 2021.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems , 35:26565-26577, 2022.

Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. AudioLDM: Text-to-audio generation with latent diffusion models. Proceedings of the International Conference on Machine Learning , 2023.

Muqiao Yang, Chunlei Zhang, Yong Xu, Zhongweiyang Xu, Heming Wang, Bhiksha Raj, and Dong Yu. usee: Unified speech enhancement and editing with conditional diffusion models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 7125-7129. IEEE, 2024.

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems , 35:8633-8646, 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems , 35:36479-36494, 2022.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4195-4205, 2023.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM , 59(2):64-73, 2016.

Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL , 2018.

Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR , 2021.

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. Advances in Neural Information Processing Systems , 2021.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems , 35:25278-25294, 2022.

Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618 , 2022.

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22500-22510, 2023.

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3836-3847, 2023.

Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453 , 2023.

Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan-Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems , 36, 2024.

Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition? arXiv preprint arXiv:2210.07574 , 2022.

Lijie Fan, Kaifeng Chen, Dilip Krishnan, Dina Katabi, Phillip Isola, and Yonglong Tian. Scaling laws of synthetic images for model training... for now. arXiv preprint arXiv:2312.04567 , 2023.

Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. Stablerep: Synthetic images from text-to-image models make strong visual representation learners. Advances in Neural Information Processing Systems , 36, 2024.

Stability AI. Stable diffusion. Software available from Stability AI, 2022.

Url https://commoncrawl.org/.

Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems , 36, 2024.

Curtis G. Northcutt, Lu Jiang, and Isaac L. Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research , 70:1373-1411, 2021.

Vijay Vasudevan, Benjamin Caine, Raphael Gontijo Lopes, Sara Fridovich-Keil, and Rebecca Roelofs. When does dough become a bagel? analyzing the remaining mistakes on imagenet. Advances in Neural Information Processing Systems , 35:6720-6734, 2022.

Brian Gordon, Yonatan Bitton, Yonatan Shafir, Roopal Garg, Xi Chen, Dani Lischinski, Daniel Cohen-Or, and Idan Szpektor. Mismatch quest: Visual and textual feedback for image-text misalignment. arXiv preprint arXiv:2312.03766 , 2023.

Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, et al. What's in my big data? arXiv preprint arXiv:2310.20707 , 2023.

Eric Frankel and Edward Vendrow. Fair generation through prior modification. In 32nd Conference on Neural Information Processing Systems (NeurIPS 2018) , 2020.

Melissa Hall, Laurens van der Maaten, Laura Gustafson, Maxwell Jones, and Aaron Adcock. A systematic study of bias amplification. arXiv preprint arXiv:2201.11706 , 2022.

Hao Chen, Bhiksha Raj, Xing Xie, and Jindong Wang. On catastrophic inheritance of large foundation models. arXiv preprint arXiv:2402.01909 , 2024.

Amirhossein Kazerouni, Ehsan Khodapanah Aghdam, Moein Heidari, Reza Azad, Mohsen Fayyaz, Ilker Hacihaliloglu, and Dorit Merhof. Diffusion models in medical imaging: A comprehensive survey. Medical Image Analysis , page 102846, 2023.

Xiaofan Li, Yifu Zhang, and Xiaoqing Ye. Drivingdiffusion: Layout-guided multi-view driving scene video generation with latent diffusion model. arXiv preprint arXiv:2310.07771 , 2023.

Chiyu Jiang, Andre Cornman, Cheolho Park, Benjamin Sapp, Yin Zhou, Dragomir Anguelov, et al. Motiondiffuser: Controllable multi-agent motion prediction using diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9644-9653, 2023.

Aritra Ghosh, Himanshu Kumar, and P. Shanti Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence , 2017.

Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Wai-Hung Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in Neural Information Processing Systems , 2018.

Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In International Conference on Learning Representations , 2020.

Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label-noise learning without anchor points. In International Conference on Machine Learning , pages 6403-6413. PMLR, 2021.

Sheng Liu, Zhihui Zhu, Qing Qu, and Chong You. Robust training under label noise by over-parameterization. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the International Conference on Machine Learning , volume 162, pages 14153-14172. PMLR, 17-23 Jul 2022.

Hao Chen, Ankit Shah, Jindong Wang, Ran Tao, Yidong Wang, Xing Xie, Masashi Sugiyama, Rita Singh, and Bhiksha Raj. Imprecise label learning: A unified framework for learning with various imprecise label configurations. arXiv preprint arXiv:2305.12715 , 2023.

Kiran K Thekumparampil, Ashish Khetan, Zinan Lin, and Sewoong Oh. Robustness of conditional gans to noisy labels. Advances in neural information processing systems , 31, 2018.

Takuhiro Kaneko, Yoshitaka Ushiku, and Tatsuya Harada. Label-noise robust generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2467-2476, 2019.

Byeonghu Na, Yeongmin Kim, HeeSun Bae, Jung Hyun Lee, Se Jung Kwon, Wanmo Kang, and Il-Chul Moon. Label-noise robust diffusion models. arXiv preprint arXiv:2402.17517 , 2024.

Hao Chen, Jindong Wang, Ankit Shah, Ran Tao, Hongxin Wei, Xing Xie, Masashi Sugiyama, and Bhiksha Raj. Understanding and mitigating the label noise in pre-training on downstream tasks. In International Conference on Learning Representations (ICLR) , 2024.

Hao Chen, Jindong Wang, Zihan Wang, Ran Tao, Hongxin Wei, Xing Xie, Masashi Sugiyama, and Bhiksha Raj. Learning with noisy foundation models. arXiv preprint arXiv:2403.06869 , 2024.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems , 25, 2012.

Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469 , 2023.

Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378 , 2023.

Ali Borji. Pros and cons of gan evaluation measures: New developments. Computer Vision and Image Understanding , 215:103329, 2022.

Eyal Betzalel, Coby Penso, Aviv Navon, and Ethan Fetaya. A study on the evaluation of generative models. arXiv preprint arXiv:2206.10935 , 2022.

Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Teufel, Marco Bellagente, et al. Holistic evaluation of text-to-image models. Advances in Neural Information Processing Systems , 36, 2024.

Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12873-12883, 2021.

Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data augmentation. Advances in Neural Information Processing Systems , 36, 2024.

Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in Neural Information Processing Systems , 30, 2017.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems , 35:5775-5787, 2022.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems , 30, 2017.

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in Neural Information Processing Systems , 29, 2016.

Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems , 32, 2019.

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718 , 2021.

Peng Cui, Dan Zhang, Zhijie Deng, Yinpeng Dong, and Jun Zhu. Learning sample difficulty from pre-trained models for reliable prediction. Advances in Neural Information Processing Systems , 36, 2024.

Minhyuk Seo, Diganta Misra, Seongwon Cho, Minjae Lee, and Jonghyun Choi. Just say the name: Online continual learning with category names only via data generation. arXiv preprint arXiv:2403.10853 , 2024.

Claude Elwood Shannon. A mathematical theory of communication. ACM SIGMOBILE mobile computing and communications review , 5(1):3-55, 2001.

Yuchen Wu, Minshuo Chen, Zihao Li, Mengdi Wang, and Yuting Wei. Theoretical insights for diffusion guidance: A case study for gaussian mixture models. arXiv preprint arXiv:2403.01639 , 2024.

Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. In International Conference on Machine Learning , 2021.

Pum Jun Kim, Yoojin Jang, Jisu Kim, and Jaejun Yoo. Topp&r: Robust support estimation approach for evaluating fidelity and diversity in generative models. Advances in Neural Information Processing Systems , 36, 2024.

Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23) , pages 5253-5270, 2023.

Pu Cao, Feng Zhou, Qing Song, and Lu Yang. Controllable generation with text-to-image diffusion models: A survey. arXiv preprint arXiv:2403.04279 , 2024.

Xulu Zhang, Xiao-Yong Wei, Wengyu Zhang, Jinlin Wu, Zhaoxiang Zhang, Zhen Lei, and Qing Li. A survey on personalized content synthesis with diffusion models. arXiv preprint arXiv:2405.05538 , 2024.

Itseez. Open source computer vision library. https://github.com/itseez/opencv , 2015.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4015-4026, 2023.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping languageimage pre-training for unified vision-language understanding and generation. In International conference on machine learning , pages 12888-12900. PMLR, 2022.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18 , pages 234-241. Springer, 2015.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 , 2018.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4401-4410, 2019.

Sitan Chen, Vasilis Kontonis, and Kulin Shah. Learning general gaussian mixtures with efficient score matching. arXiv preprint arXiv:2404.18893 , 2024.

Khashayar Gatmiry, Jonathan Kelner, and Holden Lee. Learning mixtures of gaussians using diffusion models. arXiv preprint arXiv:2404.18869 , 2024.

Wei Hu, Zhiyuan Li, and Dingli Yu. Simple and effective regularization methods for training on noisily labeled data with generalization guarantee. arXiv preprint arXiv:1905.11368 , 2019.

Yu-Hang Tang, Yuanran Zhu, and Wibe A de Jong. Detecting label noise via leave-one-out cross-validation. arXiv preprint arXiv:2103.11352 , 2021.

Jeremy Speth and Emily M Hand. Automated label noise identification for facial attribute recognition. In CVPR Workshops , pages 25-28, 2019.

Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. Freelb: Enhanced adversarial training for natural language understanding. arXiv preprint arXiv:1909.11764 , 2019.

Kezhi Kong, Guohao Li, Mucong Ding, Zuxuan Wu, Chen Zhu, Bernard Ghanem, Gavin Taylor, and Tom Goldstein. Robust optimization as data augmentation for large-scale graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 60-69, 2022.

David Nukrai, Ron Mokady, and Amir Globerson. Text-only training for image captioning using noise-injected clip. arXiv preprint arXiv:2211.00575 , 2022.

Neel Jain, Ping-yeh Chiang, Yuxin Wen, John Kirchenbauer, Hong-Min Chu, Gowthami Somepalli, Brian R Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Aniruddha Saha, et al. Neftune: Noisy embeddings improve instruction finetuning. arXiv preprint arXiv:2310.05914 , 2023.

Mang Ning, Enver Sangineto, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Input perturbation reduces exposure bias in diffusion models. arXiv preprint arXiv:2301.11706 , 2023.

Mang Ning, Mingxiao Li, Jianlin Su, Albert Ali Salah, and Itir Onal Ertugrul. Elucidating the exposure bias in diffusion models. arXiv preprint arXiv:2308.15321 , 2023.

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research , 15(1):1929-1958, 2014.

Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems , 32, 2019.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265. PMLR, 2015.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning , pages 8162-8171. PMLR, 2021.

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 , 2021.

Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems , 34:19822-19835, 2021.

Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. Cogview2: Faster and better text-toimage generation via hierarchical transformers. Advances in Neural Information Processing Systems , 35:16890-16902, 2022.

Wendi Zheng, Jiayan Teng, Zhuoyi Yang, Weihan Wang, Jidong Chen, Xiaotao Gu, Yuxiao Dong, Ming Ding, and Jie Tang. Cogview3: Finer and faster text-to-image generation via relay diffusion. arXiv preprint arXiv:2403.05121 , 2024.

Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. In European Conference on Computer Vision , pages 89-106. Springer, 2022.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning , pages 8821-8831. Pmlr, 2021.

MidJourney Inc. Midjourney. Software available from MidJourney Inc., 2022.

Ryan Webster, Julien Rabin, Loic Simon, and Frederic Jurie. On the de-duplication of laion-2b. arXiv preprint arXiv:2303.12733 , 2023.

Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Understanding and mitigating copying in diffusion models. Advances in Neural Information Processing Systems , 36:47783-47803, 2023.

Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXiv preprint arXiv:2310.14189 , 2023.

Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073 , 2021.

Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18392-18402, 2023.

Nisha Huang, Fan Tang, Weiming Dong, Tong-Yee Lee, and Changsheng Xu. Region-aware diffusion for zero-shot text-driven image editing. arXiv preprint arXiv:2302.11797 , 2023.

Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1921-1930, 2023.

Andrey Voynov, Kfir Aberman, and Daniel Cohen-Or. Sketch-guided text-to-image diffusion models. In ACM SIGGRAPH 2023 Conference Proceedings , pages 1-11, 2023.

Nisha Huang, Yuxin Zhang, Fan Tang, Chongyang Ma, Haibin Huang, Weiming Dong, and Changsheng Xu. Diffstyler: Controllable dual diffusion for text-driven image stylization. IEEE Transactions on Neural Networks and Learning Systems , 2024.

Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Composer: Creative and controllable image synthesis with composable conditions. arXiv preprint arXiv:2302.09778 , 2023.

Dina Bashkirova, José Lezama, Kihyuk Sohn, Kate Saenko, and Irfan Essa. Masksketch: Unpaired structure-guided masked image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1879-1889, 2023.

Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. 2023.

Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22511-22521, 2023.

Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances in Neural Information Processing Systems , 26, 2013.

Bo Han, Quanming Yao, Tongliang Liu, Gang Niu, Ivor W Tsang, James T Kwok, and Masashi Sugiyama. A survey of label-noise representation learning: Past, present and future. arXiv preprint arXiv:2011.04406 , 2020.

Görkem Algan and Ilkay Ulusoy. Image classification with deep learning in the presence of noisy labels: A survey. Knowledge-Based Systems , 215:106771, 2021.

Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE transactions on neural networks and learning systems , 2022.

Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in Neural Information Processing Systems , 31, 2018.

Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In International Conference on Machine Learning , pages 7164-7173. PMLR, 2019.

Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? Advances in Neural Information Processing Systems , 32, 2019.

Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. Advances in Neural Information Processing Systems , 33:7597-7610, 2020.

Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 13726-13735, 2020.

Lei Feng, Senlin Shu, Zhuoyi Lin, Fengmao Lv, Li Li, and Bo An. Can cross entropy loss be robust to label noise? In Proceedings of the twenty-ninth international conference on international joint conferences on artificial intelligence , pages 2206-2212, 2021.

Yivan Zhang, Gang Niu, and Masashi Sugiyama. Learning noise transition matrix from only noisy labels via total variation regularization. In International Conference on Machine Learning , pages 12501-12512. PMLR, 2021.

Shuo Yang, Erkun Yang, Bo Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu. Estimating instance-dependent bayes-label transition matrix using a deep neural network. In International Conference on Machine Learning , pages 25302-25312. PMLR, 2022.

Hongxin Wei, Huiping Zhuang, Renchunzi Xie, Lei Feng, Gang Niu, Bo An, and Yixuan Li. Mitigating memorization of noisy labels by clipping the model prediction. In International Conference on Machine Learning , pages 36868-36886. PMLR, 2023.

Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in Neural Information Processing Systems , 31, 2018.

Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 322-330, 2019.

Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Monazam Erfani, and James Bailey. Normalized loss functions for deep learning with noisy labels. In Proceedings of the International Conference on Machine Learning , 2020.

Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2691-2699, 2015.

Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using a noise adaptation layer. In International Conference on Learning Representations , 2016.

Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda. Earlylearning regularization prevents memorization of noisy labels. Advances in Neural Information Processing Systems , 33, 2020.

X. Li, T. Liu, B. Han, G. Niu, and M. Sugiyama. Provably end-to-end label-noise learning without anchor points. In Proceedings of 38th International Conference on Machine Learning , pages 6403-6413, 2021.

Kiran Koshy Thekumparampil, Sewoong Oh, and Ashish Khetan. Robust conditional gans under missing or uncertain labels. arXiv preprint arXiv:1906.03579 , 2019.

Sandhya Tripathi and N Hemachandra. Gans for learning from very high class conditional noisy labels. arXiv preprint arXiv:2010.09577 , 2020.

Takuhiro Kaneko and Tatsuya Harada. Blur, noise, and compression robust generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13579-13589, 2021.

Lizhen Deng, Chunming He, Guoxia Xu, Hu Zhu, and Hao Wang. Pcgan: A noise robust conditional generative adversarial network for one shot learning. IEEE Transactions on Intelligent Transportation Systems , 23(12):25249-25258, 2022.

Chuhan Wu, Fangzhao Wu, Tao Qi, Yongfeng Huang, and Xing Xie. Noisytune: A little noise can help you finetune pretrained language models better. arXiv preprint arXiv:2202.12024 , 2022.

Arjun Singh and Abhay Kumar Yadav. Symnoise: Advancing language model fine-tuning with symmetric noise. arXiv preprint arXiv:2312.01523 , 2023.

Mohammadreza Naderi, Nader Karimi, Ali Emami, Shahram Shirani, and Shadrokh Samavi. Dynamic-pix2pix: Medical image segmentation by injecting noise to cgan for modeling input and target domain joint distributions with limited training data. Biomedical Signal Processing and Control , 85:104877, 2023.

Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, and Oncel Tuzel. Clip with quality captions: A strong pretraining for vision tasks. 2024.

Hongrui Chen, Holden Lee, and Jianfeng Lu. Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions. In International Conference on Machine Learning , pages 4735-4763. PMLR, 2023.

Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint arXiv:2209.11215 , 2022.

Xuefeng Gao, Hoang M Nguyen, and Lingjiong Zhu. Wasserstein convergence guarantees for a general class of score-based generative models. arXiv preprint arXiv:2311.11003 , 2023.

Christiane Fellbaum. WordNet: An electronic lexical database . MIT press, 1998.

Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https://github.com/huggingface/ diffusers , 2022.

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2818-2826, 2016.

Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , 2021.

Xiangming Gu, Chao Du, Tianyu Pang, Chongxuan Li, Min Lin, and Ye Wang. On memorization in diffusion models. arXiv preprint arXiv:2310.02664 , 2023.

TaeHo Yoon, Joo Young Choi, Sehyun Kwon, and Ernest K Ryu. Diffusion probabilistic models generalize when they fail to memorize. In ICML 2023 Workshop on Structured Probabilistic Inference Generative Modeling , 2023.

Contents

Appendix

A Derivations and Proofs

In this section, we theoretically investigate the behavior of condition corruption on DMs. Our investigation encompasses the DDIM samplers with continuous-time processes. We focus on how slight conditional embedding corruption can affect the training process of DMs, as well as the consequences on the generative process. As our theoretical analysis indicates, slight conditional embedding corruption will benefit both the generation quality and diversity, which aligns with the experimental conclusions in Section 3.

A.1 Preliminaries

We start by giving a concise overview of the problem setup.

Data Distribution. For precise theoretical characterizations, we concentrate on the prototypical problem of sampling from Gaussian mixture models (GMMs). Specifically, we consider that the distribution of the data x ∈ R d satisfies

Here, y is denoted as the class labels with a finite set of values y ∈ { 1 , 2 , · · · , |Y|} . Given any class label y ∈ Y , the data distribution x | y is a Gaussian with the center and covariance as ( µ y , I ) . And the positive w y represents the weights of the Gaussian components which satisfies ∑ y ∈Y w y = 1 .

The diffusion model is a two processes framework: a forward process that transforms the target distribution into Gaussian noise, and a reverse process that progressively denoises in order to reconstitute the original target distribution. In this paper, we consider the continuous-time processes and define the forward process as an Ornstein-Uhlenbeck (OU) process:

Forward Process. At time step t ∈ [0 , T ] , the forward process is

where w t is a d -dimensional standard Brownian motion.

The advantage of considering the forward process as an OU process is that it enables us to directly derive the closed-form expression for the conditional sample distribution at any given time t .

where r t = e -t and σ t = √ 1 -e -2 t .

By the reparameterization trick, x t can be represented as

We explore the widely adopted sampling method, DDIM, augmented with classifier-free guidance. And the associated reverse process is stated as the following ODE implementation:

Reverse Process. We write the reverse process in a forward version by switching time direction t → T -t as

where w ≥ 0 is a hyperparameter that controls the strength of the classifier guidance.

Given our primary focus on the impact of corrupted conditional embedding on generation, we simplify the reverse process by setting w to 0 and concentrate solely on the conditional score network, i.e.,

The remaining task is to estimate the unknown conditional score function ∇ z t log P ( z t | y ) and unconditional score function ∇ z t log P ( z t | y ) via training. In the subsequent analysis, we will indicate that by minimizing Equation (3) and optimizing the denoising network ϵ , we can achieve an estimate of the conditional score.

Denoising Networks. Inspired by recent work that also target on GMMs [87, 88], we parameterize the denoising networks as the following piecewise linear function:

where c ( y ) is the one-hot encoding of label y , 1 y is an indicator function and { W k t , V k t } |Y| k =1 are trainable parameters.

Conditional Embedding Corruption. Weexamine the scenario of conditional embedding corruption, wherein the conditional embedding c ( y ) is no longer matched with the data x , but instead is perturbed by Gaussian noise. Consequently, the corrupted conditional embedding causes the denoising networks to be as follows:

where the d -dimensional corrupted noise ξ ∼ N ( 0 , I ) and γ ≥ 0 serves as the corruption control parameter, mirroring the noise ratio η for direct control of noise magnitude.

Based on the aforementioned setup, our ultimate goal is to obtain the closed form of the final generated data distribution by considering the impact of corrupted noise ξ on the training and generative processes. By comparing the differences in data distribution before and after the addition of corrupted noise, we aim to accurately characterize the impact of embedding corruption on image generation and explain the phenomena observed in experiments.

A.2 The Estimation of Conditional Score

A.2.1 Clean Conditions

We first analyze the case of clean conditional embedding. Note that the denoising networks ϵ θ is a piecewise linear function and the training objective can be represented as

where the class sample size n k := nw k .

We observed that the optimization objective in Equation (17) can be divided into |Y| independent sub-problems based on the label y . This inspires us to analyze the training and generation processes according to different classes.

Given any class k ∈ Y , we present the following lemma for determining the optimal parameters of the corresponding denoising network and the associated conditional score.

Lemma 1. (Clean Conditional Embedding). Given any class k ∈ Y , the optimal linear denoising network is

And the corresponding optimal linear estimation of conditional score ∇ x t log P ( x t | y = k ) is

where Σ k := 1 n k ∑ n k i =1 x i x T i -1 n 2 k ∑ n k i =1 x i ∑ n k i =1 x T i is the empirical covariance of k -labeled dataset and ˆ µ k := 1 n k ∑ n k i =1 x i is the empirical mean of k -labeled dataset, n k is the sample size of k -labeled dataset. And r t = e -t , σ t = √ 1 -e -2 t .

Proof. Given any class k , the training objective is

Since the weights w k is fixed in our analysis, we omit the notation w k without ambiguity in the following contents and for enhanced clarity, we initially omit the subscript/superscript k in n k , W k t , V k t , ˆ µ k , µ k and Σ k . Then Equation (20) can restated as as Equation (21) for simplicity.

where x i is labeled as k .

This training loss can be further simplified as

For simplicity, we denote b t := V t c ( y ) and we get

Then, we define the loss function

The optimal W ∗ t and b ∗ t can be obtained by taking gradient to J ( W t , b t ) such that,

And the optimal W ∗ t and b ∗ t is,

where Σ := 1 n ∑ n i =1 x i x T i -1 n 2 k ∑ n i =1 x i ∑ n i =1 x T i is the empirical covariance of k -labeled dataset and ˆ µ := 1 n ∑ n i =1 x i is the empirical mean of k -labeled dataset.

Based on above analyze, for class k , we then have the following optimal denoising network as the linear estimator of noise ϵ ,

Given any class k , we derive

where data x and x t are labeled with k .

Therefore, the linear estimator of ∇ x t log P ( x t | x , y = k ) given k -labeled data x t is

Since the estimation of ∇ x t log P ( x t | x , y = k ) and ∇ x t log P ( x t | y = k ) are equivalent in optimization, the optimal linear estimator ∇ x t log P ( x t | y = k ) also can be

The proof is completed.

Note that the ground truth functions of conditional score given the label k is

We note the similarities in form and coefficients between the optimal linear estimator (19) and ground truth (22); they both are linear combinations of µ k / ˆ µ k and x t . Specifically, when the empirical covariance Σ k equals the population covariance I and the empirical mean ˆ µ k matches the expected value µ k , the estimated conditional score in Equation (19) coincides with the true conditional score in Equation (22).

A.2.2 Corrupted Conditions

The same analytical approach as in Section A.2.1 will be applied to the scenario where the conditional embedding is perturbed. Given the class k , we propose the following Lemma 2 to describe the optimal linear denoising network and the conditional score estimator with the corrupted conditional embedding ( c ( y ) + γ ξ ) where ξ is the standard Gaussian noise.

Lemma 2. (Corrupted Conditional Embedding). Given any class k ∈ Y , the optimal linear denoising network is

And the corresponding optimal linear estimation of conditional score ∇ x t log P c ( x t | y = k ) is

where Σ k := 1 n k ∑ n k i =1 x i x T i -1 n 2 k ∑ n k i =1 x i ∑ n k i =1 x T i is the empirical covariance of k -labeled dataset and ˆ µ k := 1 n k ∑ n k i =1 x i is the empirical mean of k -labeled dataset, n k is the sample size of k -labeled dataset, and r t = e -t , σ t = √ 1 -e -2 t .

Proof. For enhanced clarity, we initially omit the subscript/superscript k in n k , W k t , V k t , ˆ µ k , µ k and Σ k .

For any class k and standard Gaussian noise ξ , we consider the following training loss

And we further optimize the training loss as

Similarly, we get the optimal W ∗ t and b ∗ t is,

where Σ := 1 n ∑ n i =1 x i x T i -1 n 2 k ∑ n i =1 x i ∑ n i =1 x T i is the empirical covariance and ˆ µ := -1 n ∑ n i =1 x i is the empirical mean of k -labeled dataset.

So for any class k , we get the following optimal denoising network as the linear estimator of noise ϵ ,

The corresponding optimal linear estimator ∇ x t log P c ( x t | y = k ) is

A.3 The Distribution of Generation

Given the class k , after getting the optimal linear estimator of conditional score ∇ x t log P ( x t | y = k ) , we can accurately characterize the reverse process and derive the closed form of generation distribution.

A.3.1 Clean Conditions

According to Lemma 1, we first replace the conditional score ∇ x t log P ( x t | y = k ) in Equation (14) with the optimal linear estimator. Given the class k , we get

Assume the empirical covariance Σ k is full rank, since the empirical covariance is diagonalizable and semi-positive, we decompose Σ k as

where U = [ u 1 | · · · | u d ] is an orthogonal matrix whose columns are the real, orthonormal eigenvectors of Σ k and Λ = diag ( λ 1 , . . . , λ d ) is a diagonal matrix whose entries are the eigenvalues arranged in descending order of Σ k , i.e., 1 ≥ λ 1 ≥ · · · ≥ λ d > 0 .

Therefore, problem (26) can be decomposed into d independent sub-problem as

where u i is i -th eigenvector and λ i is its corresponding eigenvalue.

By analyzing Equation (28), we can obtain the expectation and variance of the final generation distribution separately, thereby inferring the distribution of the ultimately generated data as outlined in the subsequent lemma.

Lemma 3. (Clean Conditional Embedding). For any class k , the distribution of the generated data z T satisfies

where Σ k := 1 n k ∑ n k i =1 x i x T i -1 n 2 k ∑ n k i =1 x i ∑ n k i =1 x T i is the empirical covariance of k -labeled dataset and ˆ µ k := 1 n k ∑ n k i =1 x i is the empirical mean of k -labeled dataset, n k is the sample size of k -labeled dataset.

Proof. For enhanced clarity, we initially remove the subscript k in ˆ µ k , µ k and Σ k . In order to obtain the solution to Equation (28) and achieve the closed form of the generation distribution, we first examine the discrete solution of Equation (28).

Given any class k , we consider the discretization Euler-Maruyama scheme which is widely used in existing work [151-153] and discretize the interval [0 , T ] into N discretization points,

where h := T N is the step size and t j = jh .

According to Equation (30), u T i z ( j +1) h can be regarded as a linear transformation of the initial distribution, which is a standard Gaussian distribution. Therefore, u T i z ( j +1) h still satisfies a Gaussian distribution. The remaining task is analyze the mean and variance of the u T i z ( j +1) h separately.

Expectation. By Equation (30), we have

By telescoping, we get the discretization solution

Since the initial distribution of reverse process is z 0 ∼ N ( 0 , I ) , thus E ( u T i z 0 ) = 0 . We further have

By limiting h → 0 , we can use the identity 1 + hx ≃ exp( hx ) . Then we get the continous version of Equation (31) that when h → 0 , we have

Hence, we have

where e i = 1 -√ λ i e -T √ 1+( λ i -1) e -2 T .

When T →∞ , we get

And the expectation of generation distribution is the empirical mean, i.e.,

Variance. Similarly, by Equation (30), we have the variance

By telescoping, we get the discretization solution

Since Var( u T i z 0 ) = u T i Var( z 0 ) u i = 1 , the variance at time T is

When h → 0 and use the identity (1 + hx ) 2 ≃ exp(2 hx ) , we get

Hence, we have

where v i = λ i 1+( λ i -1) e -2 T .

When T →∞ , we get

We complete the proof.

A.3.2 Corrupted Conditions

We then discuss the distribution of generated data when the conditional embedding is perturbed by noise. Using the same method to derive the data distribution under the corrupted conditional embedding setting, we conclude the following Lemma 4.

Lemma 4. (Corrupted Conditional Embedding). For any class k ∈ Y , the distribution of generation z c T is

where Σ k := 1 n k ∑ n k i =1 x i x T i -1 n 2 k ∑ n k i =1 x i ∑ n k i =1 x T i is the empirical covariance of k -labeled dataset and ˆ µ k := 1 n k ∑ n k i =1 x i is the empirical mean of k -labeled dataset, n k is the sample size of k -labeled dataset. And γ ≥ 0 is the corruption control parameter.

And the expectation of generation distribution is the empirical mean, i.e.,

Proof. To improve clarity, we initially disregard the subscript k in ˆ µ k , µ k and Σ k . Building upon the proof details presented in Lemma 3 and the optimal conditional score estimation outlined in Lemma 2, we arrive at the subsequent findings:

Expectation. Similarly, we derive

Hence, we have

When T →∞ , we get

And the expectation of generation distribution is

Hence, we have

When T →∞ , we get

And the expectation of generation distribution is the empirical mean, i.e.,

A.4 Diversity and Quality: Clean vs. Corrupted Conditions

A.4.1 Generation Diversity

Given any class k , building on previous work [70], we consider entropy to measure the diversity of generated images. In particular, let P and P c be the probability densities for the generated data using clean and corrupted conditional embeddings respectively, we have the following for z t and z c t

Wepropose the following theorem to describe the difference between these two conditional differential entropy.

Theorem 3. (Restatement of Theorem 1) For any class k ∈ Y , assuming the norm of corresponding expectation ∥ µ k ∥ 2 2 is a constant and the empirical covariance of training data is full rank, let z T and z c T be the generation featuring clean and corrupted conditions respectively, then it holds that

where γ is the corruption control parameter and d is the data dimension.

Proof. For the sake of clarity, we begin by omitting the subscript k from ˆ µ k , µ k and Σ k . Given any class k , since both the clean generation z T and the corrupted generation z c T follow multivariate Gaussian distributions, we can derive the closed-form expression for the difference in their differential entropy by Lemma 3 and Lemma 4 as follows

where Σ := 1 n ∑ n i =1 x i x T i -1 n 2 k ∑ n i =1 x i ∑ n i =1 x T i is the empirical covariance of k -labeled dataset and ˆ µ := 1 n ∑ n i =1 x i is the empirical mean of k -labeled dataset.

When the noise ratio γ is small and λ i = ω ( γ ) ,

Therefore,

A.4.2 Generation Quality

Before starting proving that slight noise is beneficial to the quality of generation, we first introduce the lemmas required for the proof.

Lemma 5. Given x 1 , · · · , x n independent and all distributed as a Gaussian N ( µ , I ) . Then,

Proof. The expectation is

Therefore,

By the law of total variance

We consider the Wasserstein distance to measure the distance between the generation distribution and the true data distribution. A smaller Wasserstein distance implies a closer proximity between the generated data distribution and the true data distribution, thereby indicating better generation quality

Given class k , we define 2 -Wasserstein distance between the true data distribution x | y = k and the clean generation z T as d := W 2 ( N ( µ k , I ) , N (ˆ µ k , Σ k ) ) . Similarly, the Wasserstein distance between the true data distribution x | y = k and the corrupted generation z c t is denoted as d c := W 2 ( N ( µ k , I ) , N ( ˆ µ k 1+ γ 2 , Σ k + γ 2 1+ γ 2 ∥ ˆ µ k ∥ 2 2 I ) ) .

Theorem 4. (Restatement of Theorem 2) For any class k ∈ Y , assume the norm of corresponding expectation ∥ µ k ∥ 2 2 is a constant, let P , Q X and Q c X be the ground truth, clean, and corrupted condition distributions, respectively, where X represents the collection of training data points. If γ = O (1 / √ max k n k ) , it holds that

where W 2 2 ( · , · ) denotes the quadratic 2 -Wasserstein distance between two distributions, n k is the sample size of k -labeled dataset and d is the data dimension.

Proof. To express more clearly, we first omit the subscript k of n k , ˆ µ k , µ k and Σ k . According to Lemma 3 and Lemma 4, we then can directly derive the closed form of Wasserstein distance between two Gaussian as

The squared Wasserstein distance between true data distribution and clean generation distribution

The squared Wasserstein distance between true data distribution and corrupted generation distribution

The difference in squared Wasserstein distance is

We consider the expectation error to reduce the randomness of ˆ µ as

We notice that the expectation error can be decomposed into two parts. The error of the first part being caused by the difference in means between the generated distribution and the true distribution. Since the distribution of empirical mean is ˆ µ ∼ N ( µ , 1 n I ) where n k is the sample size of k -labeled dataset, we get

The second part is attributable to the difference in covariance.

By the law of total expectation

(a) and (b) achieve by the Jensen's inequality given 1 √ λ i is a convex function By Lemma 5, we derive

Hence, we get

The second part is

Therefore, with small noise ratio γ , we then can conclude that

Noting that √ n/ ( n -1) -1 = -Θ(1 /n ) when n is large, then if the corruption level γ satisfies γ = O (1 / √ n ) , for any class k , we have

This completes the proof.

B Details of Condition Corruption, Model Training and Evaluation

In this section, we provide detailed training setup of each diffusion models we studied in the main paper, the synthetic corruption for IN-1K and CC3M, the annotation process of IN-100, and the evaluation metrics we adopted.

B.1 Synthetic Condition Corruption

We mainly studied four types of condition corruption in this paper, with two datasets. For IN-1K, we used random symmetric and asymmetric condition corruption. For CC3M, we adopted text swapping and LLM re-writing corruption. We used several levels of corruption η = { 0 , 2 . 5 , 7 . 5 , 10 , 15 , 20 } %.

Symmetric Condition Corruption for IN-1K . To introduce symmetric condition corruption in IN-1K according to a corruption ratio η , we randomly sample a ( x , y ) pair from the dataset, and flip y to another class according to the class prior in IN-1K to obtain y c , until the ratio of y c satisfies η .

Asymmetric Condition Corruption for IN-1K . For asymmetric condition corruption of IN-1K, we first find the class overlap between IN-1K and CIFAR-100 using WordNet [154], denoted as Y C -100 IN -1K . Then, we randomly sample ( x , y ) from the data subset whose y satisfies y ∈ Y C -100 IN -1K and flip y into the remaining classes of the overlapped set Y C -100 IN -1K /y .

Text Swapping Condition Corruption for CC3M . For CC3M, where y is text captions for the images, we randomly sample two pairs and swap the text of these two pairs to introduce condition corruptions. This mainly follows Chen et al. [48], where very disruptive corruption is introduced.

LLM Text Condition Corruption for CC3M . Text swapping corruption may not be common in practice for image-text datasets. Instead, we may encounter captions that have unmatched entities or partially unmatched sentences with the images. To study the text corruption in a more realistic scenarios, we use GPT-4 and prompt it to re-write the captions to introduce corruptions. We pre-define 5 levels of corruption in the prompt, and randomly sample a level as input to GPT-4.

B.2 Automatic ImageNet-100 Annotation

Here, we present the details of annotate ImageNet-100 for personalization of LDMs using ControlNet and T2I-Adapters. A few examples of the annotated images and captions are shown in Fig. 10.

Canny Edge . For canny edge, we directly use the Canny detector from OpenCV to annotate the images. We set the low threshold and high threshold of canny detector to 100 and 200 respectively.

Segmentation Mask from SAM . We use SAM to annotate segmentation masks from IN-100 images. We directly use the colormap of the segmentation masks as input control to ControlNet and T2I-Adapters.

Captions from BLIP . We use BLIP captioning model to generate captions for IN-100 for adapting text-conditional LDMs.

Ground Truth

Canny

SAM

'There is a

small  bird

standing on the

edge of a curb'

'There is a

mushroom

grow ing on the

ground in the

woods'

BLIP Caption

'There is a

snake that is

laying on the

ground in the

grass'

'There is a

snake that is

hiding  in the

bushes'

'There is a

plate of

lobsters and

other food on a

table'

B.3 LDMPre-training Setup

The pre-training setup of LDM-4 mainly follows Rombach et al. [9], as shown in Table 3. For LDM-4 models, we use a VQ-VAE [56] with a down-sampling factor of 4 and a latent space with shape 64 × 64 × 3 . It also has a vocabulary size of 8196 .

IN-1K . The hyper-parameters of training IN-1K class-conditional LDMs are summarized as follows. We use a U-Net with channels of 192 and channel multipliers of 1 , 2 , 3 , 5 as the denoising network backbone. We use class embedding, i.e., embedding layer, for computing the embeddings of class labels. The conditional embedding is injected to the U-Net with cross-attention. We use DDPM with linear schedule of 1000 steps. The batch size is set to 64 per GPU, and the learning rate is set to 1 e -4 Training IN-1K LDMs for 178K iterations takes about 2.5 days on 8 NVIDIA A100.

CC3M . We use a same U-Net as denoising network backbone. We adjust the training iterations to 396K iterations for CC3M, which takes 7.5 days to train on 8 NVIDIA A100. We use a pre-trained BERT model (bert-base-uncased) for the conditional embeddings, and it is fully trainable.

B.4 DiT Pre-training Setup

We pre-train DiT-XL/2 on IN-1K follows Peebles et al. [11]. The hyper-parameters are shown in Table 4. We train DiT-XL/2 for 400K training iterations using a per GPU batch size of 32 on 8 NVIDIA A100, which takes around 2.5 days. Compared to LDM-4, DiT-XL/2 used a fine-tuned VQ-VAE with a down-sampling factor of 8 , a latent space of shape 32 × 32 × 4 , and a vocabulary size of 16384 . DiT-XL/2 has a denoising network backbone based on Transformer architecture and uses Adaptive LayerNorm, initialized with zeros, for injecting the conditional information.

B.5 LCMPre-training Setup

LCM distills the pre-trained Stable Diffusion models to enable faster inference with fewer steps. We choose Stable Diffusion v1.5 as the teacher model and conduct distillation for 35K iterations, which takes 1.5 days on 8 NVIDIA A100 GPUs. We use a learning rate of 1 e -5.

B.6 ControlNet and T2I-Adapter Personalization Setup

We use the implementation of ControlNet and T2I-adapter of Diffusers [155] for downstream personalization tasks. Default learning rate and batch size from Diffusers are used for these two methods, and we set the training epochs for IN-100 as 10. On 4 NVIDIA V100 GPUs, training ControlNet and T2I-Adapter with LDM-4 takes about 6 hours.

B.7 Evaluation Metrics

We introduce the details of metrics we used to evaluate the diffusion models here. Due to the known difficulties of evaluating generative models, we adopt most of the existing criteria to evaluate the models we have trained.

Fréchet Inception Distance (FID) [63]. FID measures the distance between real and generated images in the feature space of an ImageNet-1K pre-trained classifier [156], indicating the similarity and fidelity of the generated images to real images.

sFID [71]. sFID utilizes the mid-level features of the inception network [156], which are more sensitive to spatial variability.

Inception Score (IS) [64]. IS also measures the fidelity and diversity of generated images. It consists of two parts: the first part measures whether each image belongs confidently to a single class of an ImageNet-1K pre-trained image classifier [156] and the second part measures how well the generated images capture diverse classes.

Precision and Recall [65]. The real and generated images are first converted to non-parametric representations of the manifolds using k-nearest neighbors, on which the Precision and Recall can be computed. Precision is the probability that a random generated image from estimated generated data manifolds falls within the support of the manifolds of estimated real data distribution. Recall is the probability that a random real image falls within the support of generated data manifolds. Thus, precision measures the general quality and fidelity of the generated images, and the recall measures the coverage and diversity of the generated images.

Top-1% Relative Mahalanobis Distance (RMD) Score [67]. RMD score measures the sample complexity and difficulty. It is defined as the difference between the Mahalanobis distances of a sample induced by the class-specific and class-agnostic Gaussian distributed estimated from the generated data. Given the dataset { ( x i , y i ) } i ∈ [ N ] , we first compute the features using the CLIP ViTB-16 encoder from the images as G ( x ) . The class-specific Gaussian distribution is then estimated:

The class-agnostic Gaussian distribution is estimated over all data as;

The RMD is defined as:

We compute the RMD score for all generated images, and report only the top1% of them.

Average Top-5 L 2 Distances . As an additional metric of sample diversity, we compute the L 2 distance of each generated image with the top-5 nearest neighbor training images. To reduce computation requirement of searching over the raw pixel space, we use the CLIP ViT-B-16 image encoder [157] to transform images into the feature space before calculating the L 2 distance. This metric measures the distance of generated samples with training images, as a proxy evaluation of diversity and memorization.

TopPR F1 [72]. TopPR is a set of reliable evaluation metrics with statistically consistent estimates of generated data and real data. We use the F1 score, computed from the TopPR Precision and Recall as an additional metric to evaluate the general quality and diversity of generated images.

CLIP Score [66]. CLIP score measures the cosine similarity between the CLIP embedding of an image-text pair. It is widely used as a metric to evaluate the fidelity and alignment of the generated images and the conditional text prompts [9].

Memorization Ratio [73]. We compute the memorization ratio as the percentage of generated images whose L 2 distances with their nearest neighbor training images are below a pre-defined threshold. We compute the distances in the feature space of CLIP ViT-B-16 image encoder [157] due to the massive size and resolution of the training images and set the threshold as 0 . 12 . Although there are several studies using the distance comparison between the first and second nearest neighbor as a reflection of memorization [158, 159], we found that this metric is not effective for large-scale datasets.

Entropy . We compute the entropy metric within the latent space of the pre-trained VQ-V AE [58, 56]. Since LDMs (and DiT) learn the data distribution from the latent space of VQ-V AE, we can compute the sample entropy E [ H ( x )] using the generated and flatten latent vector x ∈ R HW × D and the codebook C ∈ R C × D of VQ-VAE, where H and W are the height and weights of the original latent vectors, D indicates the dimension of the latent space, and C denotes the number of embeddings of the codebook. We compute the probability of each latent vector as Softmax ( ∥ x -C∥ 2 2 /τ ) , where τ is a temperature parameter controlling the sharpness of the probability. We compute entropy as:

C Full Results of Pre-training Evaluation

In this section, we present all results of our pre-training evaluation, over different diffusion models, including LDM-4, DiT-XL/2, and LCM-v1.5, and various types of condition corruption.

C.1 Quantitative Results

We present the full evaluation results of IN-1K class-conditional LDMs and CC3M textconditional LDMs in Fig. 11 and Fig. 12 respectively. All the results are computed from using a set of guidance scales. For IN-1K LDMs, we use s ∈

2

4

6

8

10

8

10

12

14

16

18

20

22

FID

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(a) FID

2

4

6

8

10

50

100

150

200

250

300

IS

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(b) IS

2

4

6

8

10

0.45

0.50

0.55

0.60

0.65

0.70

0.75

0.80

Precision

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(c) Precision

2

4

6

8

10

0.10

0.20

0.30

0.40

0.50

0.60

Recall

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(d) Recall

2

4

6

8

10

20

25

30

35

sFID

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(e) sFID

2

4

6

8

10

0.30

0.40

0.50

0.60

0.70

TopPR F1

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(f) TopPR F1

2

4

6

8

10

280

300

320

340

360

380

400

420

RMD

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(g) Top-

1%

RMD

2

4

6

8

10

0

2

4

6

8

10

12

14

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(h) Mem. Ratio

2

4

6

8

10

0

.

26

0

.

28

0

.

30

0

.

32

0

.

34

0

.

36

L

2

Dist.

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(i) Avg

L

2

Dist.

2

4

6

8

10

27

28

29

30

31

CLIP Score

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(j) CLIP Score

{ 1 . 5 , 1 . 75 , 2 . 0 , 2 . 25 , 2 . 5 , 3 . 0 , 3 . 5 , 4 . 0 , 4 . 5 , 5 . 5 , 6 . 0 , 6 . 5 , 7 . 0 , 7 . 5 , 8 . 0 , 9 . 0 , 10 . 0 } . For CC3M LDMs, we use s ∈ { 1 . 5 , 2 . 0 , 2 . 5 , 2 . 75 , 3 . 0 , 3 . 25 , 3 . 5 , 4 . 0 , 5 . 0 , 6 . 0 , 7 . 0 , 7 . 5 , 8 . 0 , 10 . 0 } For all the metrics, including FID, IS, Precision, Recall, sFID, TopPR F1, and CLIP score, we can all observe that slight condition corruption makes LDMs perform better, with improved image quality and diversity. We also observe that, when there is condition corruption in the dataset, the memorization ratio based on L 2 distances actually decreases, in line with observations as in Gu et al. [158].

By default we use DPM scheduler for generating the images with 50 inference steps. But we also study the generation of DDIM scheduler with 250 inference steps, as adopted in [9]. Due to the computation cost of running DDIM scheduler for 250 steps, we only study it with IN-1K LDM-4. The results are shown in Fig. 13. One can observe the same trends from the metrics using DPM and DDIM, demonstrating our findings are scheduler agnostic.

We then show the pre-training results of DiT-XL/2 and LCM-v1.5 in Fig. 19, where we primarily compute the FID, IS, Precision, and Recall. For DiT-XL/2, we use s ∈ { 1 . 5 , 1 . 75 , 2 . 0 , 2 . 25 , 2 . 5 , 3 . 0 , 3 . 5 , 4 . 0 , 4 . 5 , 5 . 0 , 5 . 5 } . For LCM-v1.5, we use s ∈ { 1 . 5 , 2 . 0 , 2 . 5 , 3 . 0 , 3 . 5 , 4 . 0 , 4 . 5 , 5 . 0 , 5 . 5 , 6 . 0 , 6 . 5 , 7 . 0 , 7 . 5 , 8 . 0 , 9 . 0 , 10 . 0 } . Slight condition corruption also facilitates the performance by using the most suitable guidance scale.

We additionally include the FID and IS trend along training for LDM IN-1K model with no corruption and 2 . 5 %corruption, using a guidance scale of 2.5, as shown in Table 5. Slight corruption begins to be effective at the very early stage of training.

Finally, we present the results of LDMs pre-trained on CC3M with LLM corruption and IN-1K with asymmetric corruption, where similar observations still hold.

2

4

6

8

10

20

25

30

35

40

FID

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(a) FID

2

4

6

8

10

20

22

24

26

28

30

32

34

IS

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(b) IS

2

4

6

8

10

0

.

40

0

.

45

0

.

50

0

.

55

0

.

60

0

.

65

0

.

70

Precision

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(c) Precision

2

4

6

8

10

0

.

25

0

.

30

0

.

35

0

.

40

0

.

45

0

.

50

0

.

55

Recall

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(d) Recall

2

4

6

8

10

44

46

48

50

52

54

sFID

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(e) sFID

2

4

6

8

10

0

.

15

0

.

20

0

.

25

0

.

30

0

.

35

0

.

40

0

.

45

0

.

50

TopPR F1

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(f) TopPR F1

2

4

6

8

10

200

225

250

275

300

325

350

375

RMD

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(g) Top-

1%

RMD

2

4

6

8

10

0

1

2

3

4

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(h) Mem. Ratio

2

4

6

8

10

0

.

34

0

.

35

0

.

36

0

.

37

0

.

38

0

.

39

0

.

40

0

.

41

L

2

Dist.

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(i) Avg

L

2

Dist.

2

4

6

8

10

25

26

27

28

29

30

CLIP Score

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(j) CLIP Score

2

4

6

8

10

8

10

12

14

16

18

20

22

FID

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(a) FID

2

4

6

8

10

30

40

50

60

70

IS

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(b) IS

2

4

6

8

10

0.45

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

Precision

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(c) Precision

2

4

6

8

10

0.10

0.20

0.30

0.40

0.50

0.60

Recall

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(d) Recall

C.2 Qualitative Results

We present more visualization results of class-conditional LDM-4 in Fig. 18, class-conditional DiTXL/2 in Fig. 19, text-conditional LDM-4 in Fig. 20, and text-conditional LCM-v1.5 in Fig. 21. One can observe that DMs pre-trained with slight condition corruption in general more visually appealing images.

1

.

5

2

.

0

2

.

5

3

.

0

3

.

5

4

.

0

4

.

5

5

.

0

5

.

5

7

.

5

10

.

0

12

.

5

15

.

0

17

.

5

20

.

0

22

.

5

FID

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(a) FID

1

.

5

2

.

0

2

.

5

3

.

0

3

.

5

4

.

0

4

.

5

5

.

0

5

.

5

100

150

200

250

300

350

IS

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(b) IS

1

.

5

2

.

0

2

.

5

3

.

0

3

.

5

4

.

0

4

.

5

5

.

0

5

.

5

0.60

0.65

0.70

0.75

0.80

0.85

Precision

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(c) Precision

1

.

5

2

.

0

2

.

5

3

.

0

3

.

5

4

.

0

4

.

5

5

.

0

5

.

5

0.10

0.20

0.30

0.40

0.50

Recall

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(d) Recall

2

4

6

8

10

21

22

23

24

25

26

27

28

FID

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(a) FID

2

4

6

8

10

38

.

0

38

.

5

39

.

0

39

.

5

40

.

0

IS

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(b) IS

2

4

6

8

10

0.63

0.64

0.65

0.66

0.67

0.68

Precision

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(c) Precision

2

4

6

8

10

0.30

0.32

0.34

0.36

0.38

Recall

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(d) Recall

D Full Results of Downstream Personalization Evaluation

We present complete results of downstream personalization here.

D.1 Quantitative Results

We show the results of ControlNet IN-1K LDM-4 in Fig. 22, T2I-Adapter IN-1K LDM4 in Fig. 23, ControlNet CC3M LDM-4 in Fig. 24, and T2I-Adapter CC3M LDM-4 in Fig. 25. For all personalization experiments, we compute the results for both Canny and SAM spatial controls. Guidance scales of { 1 . 25 , 1 . 5 , 2 . 0 , 2 . 25 , 2 . 5 , 3 . 0 , 4 . 0 , 5 . 0 , 6 . 0 , 7 . 0 } and { 2 . 0 , 3 . 0 , 4 . 0 , 5 . 0 , 6 . 0 , 6 . 5 , 7 . 0 , 7 . 5 , 8 . 0 , 8 . 5 , 9 . 0 , 10 . 0 } are used for IN-1K models and CC3M models, respectively, for all experiments here.

From the results, one can observe that models pre-trained with slight condition corruption also present the best performance in downstream personalization tasks.

D.2 Qualitative Results

We present the qualitative comparison of ControlNet personalization results here. Since T2I-Adapter personalization results are similar but visually worse (quantitatively worse too), we skip their results. The visualizations of ControlNet IN-1K LDM-4 with Canny and SAM conditions are shown in Fig. 26 and Fig. 27, respectively. The visualizations of ControlNet CC3M LDM-4 with Canny and SAM conditions are shown in Fig. 28 and Fig. 29, respectively. Similarly, models pre-trained with slight condition corruption present the best image quality.

2

4

6

8

10

20

25

30

35

40

FID

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(a) FID

2

4

6

8

10

24

26

28

30

32

34

IS

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(b) IS

2

4

6

8

10

0

.

45

0

.

50

0

.

55

0

.

60

0

.

65

0

.

70

Precision

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(c) Precision

2

4

6

8

10

0

.

25

0

.

30

0

.

35

0

.

40

0

.

45

0

.

50

Recall

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(d) Recall

E Full Results of Conditional Embedding Perturbation

E.1 Qualitative Results

More visualizations of CEP, compared with clean and IP pre-trained are shown here. We present the more results of IN-1K LDM-4 and DiT-XL/2 in Fig. 30(a) and Fig. 30(b), respectively. We also present more results of CC3M LDM-4 and LCM-v1.5 in Fig. 31(a) and Fig. 31(b), respectively. CEP generally helps DMs generate more visually appealing and realistic images. We also show the more personalization visualization in Fig. 32 and Fig. 33.

E.2 Ablation Study

In Fig. 8(a), we compute the L 2 distance of perturbed condition embeddings and the clean ones, as a measurement for the corruption levels (of CEP). Here, we elaborate how we compute the L 2 distances. For fixed corruption, we calculate the distances as:

where θ ∗ is learned from clean data. For CEP, we directly calculate the L 2 norm of sampled noise:

E.3 Comparison with Dropout and Label Smoothing

Here, we additionally compare CEP with dropout and label smoothing on LDM IN-1K models, which are two alternatives that also introduce perturbations in class embeddings. The results are shown in Table 6. One can observe that, both dropout and label smoothing have similar regularization effects on training diffusion models, whereas CEP-U and CEP-G is more effective.

E.4 Comparison with Fixed and Random Corruption

We further compare with fixed CEP corruption, and random data corruption, to study the effects of fixed and random perturbation to train diffusion models. For fixed CEP-U, we first select the samples

2

4

6

8

10

7

.

5

10

.

0

12

.

5

15

.

0

17

.

5

20

.

0

22

.

5

25

.

0

FID

η

=0%

η

=2.5%

η

=5%

η

=7.5%

(a) FID

2

4

6

8

10

75

100

125

150

175

200

225

250

IS

η

=0%

η

=2.5%

η

=5%

η

=7.5%

(b) IS

2

4

6

8

10

0.55

0.60

0.65

0.70

0.75

0.80

0.85

Precision

η

=0%

η

=2.5%

η

=5%

η

=7.5%

(c) Precision

2

4

6

8

10

0.10

0.20

0.30

0.40

0.50

0.60

Recall

η

=0%

η

=2.5%

η

=5%

η

=7.5%

(d) Recall

to add perturbation, and then fix them during training. For random data corruption, we randomly choose samples during training to make their label noisy by flipping to other classes. From the results in Table 7, we show that CEP works the best among all corruption methods. Also fixed CEP is more effective than adding data corruption (fixed and random). Random data corruption can be viewed as a CEP-variant with embeddings from flipping label instead of adding noise, and thus is also more effective than fixed data corruption.

Ptarmigan

Geyser

Indigo Bunting

Tree Frog

Mud Turtle

Cockatoo

Monarch

Container Ship

Clean

𝜂

= 2.5%

𝜂

= 5%

𝜂

= 7.5%

𝜂

= 10%

𝜂

= 15%

𝜂

= 20%

Ostrich

Junco

Redbone

Schooner

Alp

Seashore

Vase

Wing

Yurt

Artichoke

Clean

𝜂

= 2.5%

𝜂

= 5%

𝜂

= 7.5%

𝜂

= 10%

𝜂

= 15%

𝜂

= 20%

Clean

𝜂

= 2.5%

𝜂

= 5%

𝜂

= 7.5%

𝜂

= 10%

𝜂

= 15%

𝜂

= 20%

Clean

𝜂 = 2.5%

𝜂 = 15%

𝜂 = 20%

1

2

3

4

5

6

7

12

14

16

18

20

22

24

FID

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(a) Canny, FID

1

2

3

4

5

6

7

45

.

0

47

.

5

50

.

0

52

.

5

55

.

0

57

.

5

60

.

0

IS

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

1

2

3

4

5

6

7

0.60

0.65

0.70

0.75

0.80

0.85

Precision

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(c) Canny, Precision

1

2

3

4

5

6

7

0.30

0.40

0.50

0.60

0.70

Recall

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(d) Canny, Recall

1

2

3

4

5

6

7

14

16

18

20

22

24

26

FID

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(e) SAM, FID

1

2

3

4

5

6

7

42

.

5

45

.

0

47

.

5

50

.

0

52

.

5

55

.

0

57

.

5

60

.

0

IS

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(f) SAM, IS

1

2

3

4

5

6

7

0.60

0.65

0.70

0.75

0.80

0.85

Precision

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(g) SAM, Precision

1

2

3

4

5

6

7

0.20

0.30

0.40

0.50

0.60

0.70

Recall

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(h) SAM, Recall

1

2

3

4

5

6

7

15

20

25

30

35

40

FID

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(a) Canny, FID

1

2

3

4

5

6

7

35

40

45

50

55

60

IS

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

1

2

3

4

5

6

7

0.50

0.60

0.70

0.80

Precision

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(c) Canny, Precision

1

2

3

4

5

6

7

0.20

0.30

0.40

0.50

0.60

Recall

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(d) Canny, Recall

1

2

3

4

5

6

7

15

20

25

30

35

40

FID

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(e) SAM, FID

1

2

3

4

5

6

7

35

40

45

50

55

60

IS

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(f) SAM, IS

1

2

3

4

5

6

7

0.45

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

Precision

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(g) SAM, Precision

1

2

3

4

5

6

7

35

40

45

50

55

60

IS

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(h) SAM, Recall

2

3

4

5

6

7

8

9

10

40

45

50

55

60

FID

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(a) Canny, FID

2

3

4

5

6

7

8

9

10

26

28

30

32

34

36

IS

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

2

3

4

5

6

7

8

9

10

0.45

0.50

0.55

0.60

0.65

Precision

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(c) Canny, Precision

2

3

4

5

6

7

8

9

10

0.40

0.45

0.50

0.55

0.60

0.65

0.70

Recall

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(d) Canny, Recall

2

3

4

5

6

7

8

9

10

40

45

50

55

60

FID

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(e) SAM, FID

2

3

4

5

6

7

8

9

10

26

28

30

32

34

36

IS

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(f) SAM, IS

2

3

4

5

6

7

8

9

10

0.40

0.45

0.50

0.55

0.60

0.65

Precision

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(g) SAM, Precision

2

3

4

5

6

7

8

9

10

0.35

0.40

0.45

0.50

0.55

0.60

0.65

Recall

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(h) SAM, Recall

2

3

4

5

6

7

8

9

10

45

50

55

60

65

70

75

80

85

FID

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(a) Canny, FID

2

3

4

5

6

7

8

9

10

22

24

26

28

30

32

34

IS

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

2

3

4

5

6

7

8

9

10

0.30

0.35

0.40

0.45

0.50

0.55

0.60

Precision

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(c) Canny, Precision

2

3

4

5

6

7

8

9

10

0.35

0.40

0.45

0.50

0.55

0.60

0.65

Recall

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(d) Canny, Recall

2

3

4

5

6

7

8

9

10

45

50

55

60

65

70

FID

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(e) SAM, FID

2

3

4

5

6

7

8

9

10

24

26

28

30

32

34

IS

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(f) SAM, IS

2

3

4

5

6

7

8

9

10

0.40

0.45

0.50

0.55

0.60

Precision

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(g) SAM, Precision

2

3

4

5

6

7

8

9

10

24

26

28

30

32

34

IS

η

=0%

η

=2.5%

η

=5%

η

=7.5%

η

=10%

η

=15%

η

=20%

(h) SAM, Recall

Clean

𝜂

= 2.5%

𝜂

= 5%

𝜂

= 10%

𝜂

= 15%

𝜂

= 20%

Control

Lorikeet

Rock Crab

Shih-Tzu

Rocker

Clean

𝜂

= 2.5%

𝜂

= 5%

𝜂

= 10%

𝜂

= 15%

𝜂

= 20%

Control

Coyote

Cinema

Pedestal

Rocker

acea

XIXI:

Clean

𝜂

= 2.5%

𝜂

= 5%

𝜂

= 10%

𝜂

= 15%

𝜂

= 20%

Control

Clean

𝜂

= 2.5%

𝜂

= 5%

𝜂

= 10%

𝜂

= 15%

𝜂

= 20%

Control

White Shark

Clean

IP

CEP-U

CEP-G

Cock

Snail

Cricket

Dam

Promontory

Lakeside

Agaric

Viaduct

Hay

Cliff

Clean

IP

CEP-U

CEP-G

Snail

Coonhound

Norfolk

Arctic Fox

Sea Cucumber

Hare

Guinea Pig

Langur

Cliff

(a) IN-1K LDM-4

(b) IN-1K DiT-XL/2

Clean

IP

CEP-U

CEP-G

'

A meal of a reuben sandwich with fries and coleslaw.

'

'

A herd of sheep grazing in an open field

'

(a) CC3M LDM-4

Clean

IP

CEP-U

CEP-G

'

A woman sits on her motorcycle in the street

'

'

A red and white plane flying under a blue sky

'

'

'

A dog is lying in bed with its head on a pillow

'

(b) CC3M LCM-v1.5

Clean

IP

CEP-U

CEP-G

Control

Saluki

Pickup

Rotisserie

Rotisserie

Clean

IP

CEP-U

CEP-G

Control

'Arafed spiral staircase with a blue window in a building'

'There is a duck that is swimming in the water'

'There is a bug on the ground'

'There is a large boat that is sailing in the water'

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We claimed contribution in Section 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discussed the limitations in Section 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Full proofs are shown in Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All model setups, hyper-parameters, inference details are shown in Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We used public available data and all code will be released.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All details in main paper and Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Mainly becuase computational cost.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All details are shown in Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: All research follows NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discussed at the end of Section 1.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: Pre-trained models will be released by reuqest and will be equipped with safety checkers.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All are cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors find that slight corruption in the conditioning of text-to-image diffusion models improves the performance. The authors theoretically analyze this empirical finding in a toy model where the goal is to learn to sample from a Gaussian Mixture and the network is piece-wise linear function. Inspired by their experimental and theoretical findings, the authors propose a technique to improve the performance of text-conditional diffusion models by adding noise to the embedding of the conditioning vector.\n\nSTRENGTHS:\n* The research topic is timely. State-of-the-art diffusion models are trained on billions of image-text pairs. Many of those are potentially noisy.\n* The authors propose a very simple modification to the training of text-to-image diffusion models that leads to improved performance.\n* The finding that slight corruption in the conditioning helps is interesting.\n* The authors validate their findings through an extensive experimental evaluation.\n\nWEAKNESSES:\n* I believe that the authors should emphasize in the title of their paper and in the main body that they study corruption in the conditioning. Numerous works study corruption in the images themselves, e.g. see:\n\n1) Ambient Diffusion: Learning Clean Distributions from Corrupted Data\n2) Consistent Diffusion Meets Tweedie: Training Exact Ambient Diffusion Models with Noisy Data\n3) GSURE-Based Diffusion Model Training with Corrupted Data\n4) Solving inverse problems with score-based generative priors learned from noisy data\n\nIn these works, it has been observed that data corruption decreases the performance of diffusion models. I believe the authors should acknowledge these relevant works and clarify that their paper analyzes corruption in the conditioning.\n\n* The theoretical model is not very relevant, at least not very relevant to Section 3 of the paper. In the experiments of Section 3, the authors mislabeled some of the training examples.  In the theoretical model, it is assumed that the true class is given to the model since the model is using the parameters that correspond to the true class. In a sense, the theoretical model of Section 3 is more related to the proposed algorithm in Section 5. Also, the existence of multiple centers in Section 4 does not seem relevant, unless I am missing something. This is because the model has separate parameters for each center and the centers are given for each example the model sees. I went over the proof and it looks like this is exploited in the proof as well, to obtain a closed-form solution for the optimal network. \n* For the reasons mentioned above, it seems to me that the reason this works is some sort of regularization. I wonder if a similar effect could be obtained by increased dropout in the conditioning or some other form of regularization. Overall, I believe that the reader of the paper doesn't develop much intuition about the origins of the experimental finding.\n\nQUESTIONS:\nApart from the weaknesses mentioned above, I have one more important question regarding this paper. Could the authors please clarify whether the corruption in Sections 3 and Sections 5 happens once for each data sample before the training or whether a new corruption takes place whenever we encounter the same data point across different epochs? To clarify, I want to understand which one of the two is happening:\n\n1) We take a dataset, we corrupt it once, we train with the corrupted dataset\n2) We use the clean dataset and every time we see a sample we corrupt it (by adding a different noise each time to its embedding or by changing each label/text to something else each time we see the same example across epochs).\n\nIf 2) is happening, it seems to me that the proposed method is closely related to regularization. I think this needs to be clarified and the effect of doing 1) or 2) should be studied as an ablation.\n\nLIMITATIONS:\nThe authors have adequately discussed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the impact of slight corruption of conditioning\ninformation in pre-training data on the performance of diffusion models (DMs).\nBy introducing synthetic corruption to ImageNet-1K and CC3M datasets, the study\nevaluates over 50 conditional DMs. Empirical and theoretical analyses reveal\nthat slight corruption enhances the quality, diversity, and fidelity of\ngenerated images. Based on these insights, the work proposes Conditional\nEmbedding Perturbations (CEP) as a method to improve DM training which shows\nsignificant improvements in both pre-training and downstream tasks.\n\nSTRENGTHS:\n- **Clarity, very well written:** The paper is clearly structured and builds up\n   the motivation for the proposed method step by step from experiments and\n   theoretical analysis.\n - **Extensive experiments:** The study examines many conditions training\n   multiple diffusion models with different noise conditions from scratch.\n   This gives a detailed perspective on the performance benefits of using\n   slight conditioning corruption during training and is a valuable\n   addition to the field.  These experiments are evaluated both qualitatively and quantitatively.\n - **Unexpected insight:** The study well describes a generally unexpected\n   phenomenon and shows how it can be leveraged to develop an new method.\n\nWEAKNESSES:\n- **Notion of significance:** The paper claims that slight pre-training\n   corruption yield significantly better performance in terms of FID and IS\n   metrics. Nevertheless, this notion of significance is never formally tested.\n   For this at least a experiment with multiple repetitions should be performed\n   to determine statistical significance.\n - **Typos:**, Page 8, lines 257 and 259 rrecision -> precision, Page 9, lines 299 dataset -> data\n - **Link between theoretical analysis and metrics** should be made explicit.\n   I.e. pointing out that the FID is essentially a Gaussian approximation\n   based estimated for the 2-Wasserstein distance of inception features (preferably at page 7, line 223).\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nThe authors correctly reflect the limitations of the study. In particular, the\ntheoretical analysis is limited due to the assumptions made, but can still shed\nsome light on the underlying mechanism.  Further, the general issue of\nevaluating image generative models quantitatively is pointed out and\na limitation of the work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors study the effect of slight corruptions to the training data during pretraining of conditioned diffusion models. They introduce the perturbations to the \"condition\" in the diffusion models and show that this leads to a better model (via FID and other metrics).  They also provide an intuition on why corruptions help by theoretically studying this case on a GMM.\n\nSTRENGTHS:\n- Well-written paper\n- All the experiments are well thought out, especially the downstream application experiments.\n- Multiple metrics are evaluated and shown that perturbation helps, which makes the claim more sound.\n\nWEAKNESSES:\n- While authors showed extensively that adding noise can help, they missed to show \"when does it help\"? For example, if I'm a practitioner who wants to train a large model from scratch, I usually will not have the budget to train multiple models to determine the amount of noise that needs to be added. That insight is missing in this paper.\n\nQUESTIONS:\n- Does the optimal noise perturbation level transfer between the models? If 10% is the best noise level for LDM on one dataset, would it be the best noise for the same-size LDM on another?\n- How do metrics look like throughout training for models trained with different levels of noise?\n- CC3M corruption details are not clear. I went to Appendix B1, you mentioned 5 levels of corruption, but it is still vague! What are these 5 levels? \n- How is the number of training iterations determined for the models? I looked at the appendix and made some rough calculations, it seems like you trained the IN model for 91 epochs and CC3M for 67 epochs (Pls correct me if I'm wrong). On what basis are these numbers decided? This again ties back to the question, when is \"adding noise\" helpful? Are we in an overtraining regime? Is that the reason why adding corruption acts as \"augmentation\" and hence the metrics improve?\n\n- (Minor) A few relevant citations are missing. For example, adding perturbations to the conditioning both in text space and latent space is explored in the context of memorization in last year's Neurips paper [1].\n\n[1] - \"Understanding and mitigating copying in diffusion models.\" Advances in Neural Information Processing Systems 36 (2023): 47783-47803.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}

comments: SUMMARY:
This paper claims that slight corruption of the condition is beneficial for conditional diffusion models. Experimentally, they conducted experiments with label flips or text swaps in the dataset and observed the performance improvement over zero noise at noise levels of 2.5% to 5%. Theoretically, they used a Gaussian mixture model to show that generation diversity and quality, as measured by entropy and Wasserstein distance, can be improved with a small amount of state corruption. Based on this, they proposed a method for injecting small amounts of Gaussian noise into the condition embeddings and experimentally demonstrated its superiority.

STRENGTHS:
Their claim offers a new perspective that may not be intuitive, but has been partially mentioned in the existing literature. They provide a solid experimental and theoretical explanation for this view.

WEAKNESSES:
* From my understanding, in Theorem 2, $\gamma$ is $O(1/\sqrt{n})$, which is the scenario for small corruption. It would be beneficial to address this explicitly in the main text.

* It would be beneficial if the method that determines the noise level could be linked to the theory like using the $\gamma=O(1/\sqrt{n})$,.

* They briefly mentioned the various assumptions in their theory as limitations. It would be helpful to provide a detailed explanation of these assumptions and the scenarios in which they hold.

QUESTIONS:
Please see the Weaknesses part.

LIMITATIONS:
They mentioned in the last section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-21