Slight Corruption in Pre-training Data Makes Better Diffusion Models

Authors: hao  chen
yujin  han
diganta  misra
xiang  li
kai  hu
difan  zou
masashi  sugiyama
jindong  wang
bhiksha  raj
william & mary
  mbzuai
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-29 16:48:21

Abstract: Diffusion models (DMs) have shown remarkable capabilities in generating realistic high-quality images, audios, and videos. 
They benefit significantly from extensive pre-training on large-scale datasets, including web-crawled data with paired data and conditions, such as image-text and image-class pairs.
Despite rigorous filtering, these pre-training datasets often inevitably contain corrupted pairs where conditions do not accurately describe the data. 
This paper presents the first comprehensive study on the impact of such corruption in pre-training data of DMs.
We synthetically corrupt ImageNet-1K and CC3M to pre-train and evaluate over $50$ conditional DMs. 
Our empirical findings reveal that various types of slight corruption in pre-training can significantly enhance the quality, diversity, and fidelity of the generated images across different DMs, both during pre-training and downstream adaptation stages. 
Theoretically, we consider a Gaussian mixture model and prove that slight corruption in the condition leads to higher entropy and a reduced 2-Wasserstein distance to the ground truth of the data distribution generated by the corruptly trained DMs.
Inspired by our analysis, we propose a simple method to improve the training of DMs on practical datasets by adding condition embedding perturbations (CEP).
CEP significantly improves the performance of various DMs in both pre-training and downstream tasks.
We hope that our study provides new insights into understanding the data and pre-training processes of DMs.

Main paper: 
 introduction: 
 Recently, diffusion models (DMs) have been demonstrating unprecedented capabilities in generating high-quality, realistic, and faithful images [1] [2] [3] [4] [5] , audios [6, 7] , and videos [8] . In addition, they exhibit impressive conditional generation results [9] [10] [11] when trained with classifier-free guidance [12] . The successes of DMs are often attributed to the massive pre-training on large-scale datasets consisting of paired data and conditions [13] [14] [15] [16] [17] , which also empowered and facilitated numerous downstream applications and personalization of pre-trained models, such as subject-driven generation [18, 19] , controllable conditional generation [20] [21] [22] , and synthetic data training [23] [24] [25] .The large-scale pre-training datasets of paired data and conditions are usually web-crawled. For example, Stable Diffusion [26] was pre-trained on LAION-2B [17] , which contains billion-scale image-text pairs collected from Common Crawl [27] . Despite the heavy filtering mechanisms used in collecting pre-training datasets [17, 28] , they still inevitably contain corrupted pairs where conditions do not correctly describe or match the data, such as corrupted labels and texts [29] [30] [31] [32] . While large-scale datasets are necessary for DMs to perform well, the corruption may lead to unexpected behavior or generalization performance of models [33] [34] [35] during both pre-training and adaptation stages, especially for safety-critical domains such as healthcare [36] and autonomous driving [37, 38] .Conventional wisdom may suggest that training under corrupted conditions could lead to deterioration in performance. For example, Noisy Label Learning [39-42, 29, 43, 44] aims to improve the generalization of models when training with corrupted labels. Label-noise robust conditional generative adversarial nets [45, 46] and DMs [47] have also been studied. However, these works are primarily concerned with supervised learning in downstream scenarios with assumptions of high noise ratios and the same training and testing data distributions. Due to the misalignment with large-scale self-supervised pre-training in practice on filtered datasets with relatively smaller noise ratios, the effects of corruption in pre-training can also differ from those in downstream [48, 49] .Understanding the effects of pre-training with such corruption is challenging and still remains largely unexplored. In this paper, we provide the first comprehensive and practical study on condition corruption in the pre-training of DMs. Through indepth analysis, we empirically, theoretically, and methodologically verify that slight condition corruption in pre-training makes better DMs. We pre-train over 50 class-conditional and textconditional DMs using classifier-free guidance (CFG) [12] on ImageNet-1K (IN-1K) [50] and CC3M [14] with synthetically corrupted conditions, i.e., classes and texts, of various levels.Our study covers a wide range of DM families, including Latent Diffusion Model (LDM) [9] , Diffusion Transformer (DiT) [11] , and Latent Consistency Model (LCM) [51, 52] . Due to the known obstacles of evaluating generative models [53] [54] [55] , we conduct both pre-training and downstream evaluation from the perspectives of image quality, fidelity, diversity, complexity, and memorization, to comprehensively understand the effects of pre-training corruption of DMs. More specifically, for pre-training, we directly evaluate the images generated from the pre-trained models, and for downstream adaptation, we evaluate on the images generated using personalized models with ControlNet [20] and T2I-Adapter [21] from the pre-trained ones. In addition, we theoretically investigate how slight corruption in conditional embeddings benefits the training and generative processes of DMs. Our key findings include:• Empirically, slight corruption in pre-training facilitates the DMs to generate images with higher quality and more diversity, both qualitatively (in Fig. 1 ) and quantitatively (in Fig. 2 ). • Theoretically, we employ a Gaussian mixture model to show slight condition corruption improves the diversity and quality of generation by increasing entropy over clean condition generation and reducing the quadratic 2-Wasserstein distance to the true data distribution (in Section 4).• Methodologically, based on our analysis, we propose a simple method to improve the pre-training of DMs by adding conditional embedding perturbations (CEP). We show that CEP can significantly boost the performance of various DMs in both pre-training and downstream tasks (in Section 5).Going beyond images, we do see the potential of this study in other modalities. Our efforts may also inspire future investigation on other types of corruption and bias inside pre-training datasets. We hope that our work can shed light on the future research of diffusion models and responsible AI. 
preliminary: 
 Denoising Diffusion Models. DMs are probabilistic models that learn the data distribution P(x), with x denoting the observed data 3 , over a set of latent variables z 1 , . . . , z T with length T [1, 57] . It assumes a forward diffusion process, gradually adding Gaussian noise to the data with a fixed Markov chain: q(z t |x) = N ( √ ᾱt x, (1ᾱt )I), which can be re-parameterized as z t = √ ᾱt x + √ 1ᾱt ϵ with ϵ ∼ N (0, I) and ᾱt as constants produced by a noise scheduler. DMs are trained via the reverse process, inverting the forward process as: p θ (z t-1 |z t ) = N (µ θ (z t ) , Σ θ (z t )), with a network that predicts the statistics of p θ . Setting Σ θ (z t ) = (1ᾱt )I to untrained constants, the reverse process is simplified as training equally weighted denoising autoencoders ϵ (z t , t) with uniformly sampled t:L DM = E x,ϵ∼N (0,I),t∼U (1,T ) ∥ϵ -ϵ θ (z t , t)∥ 2 2 .(EQUATIONAfter training, new images can be generated by sampling z t-1 ∼ p θ (z t-1 |z t ) starting with N (0, I).Classifier-free Guidance (CFG). Extra condition information y, such as class labels and text prompts, can be injected into DMs with conditional embeddings c θ (y) from modality-specific encoders [9] for conditional generation: p θ (z t-1 |z t , c θ (y)). CFG [12] jointly learns a unconditional model ϵ θ (z t , t, c θ (∅)) with an empty condition y = ∅ and a conditional model ϵ θ (z t , t, c θ (y)), and combines them linearly to control the trade-off of sample quality and diversity in generation:EQUATIONwhere s > 1 denotes the guidance scale. We adopt CFG by default with the training objective:L DM = E x,y,ϵ∼N (0,I),t∼U (1,T ) ∥ϵϵ θ (z t , t, c θ (y))∥ 2 2 .(EQUATIONCondition Corruption. Ideally, each y should accurately describe and match x. However, in practice, due to errors from the collection of web-crawled datasets, conditions y c may un-match x.We define (x, y c ) as pairs with condition corruption, and assume that c θ (y c ) = c θ (y; η, ξ), where ξ denotes certain noise and η denotes corruption ratio that implicitly controls the noise magnitude. 
understanding the pre-training corruption in diffusion models: 
 In this section, we conduct the first comprehensive and practical study on pre-training DMs with condition corruption. Through holistic exploration with synthetically corrupted datasets, we reveal a surprising observation that slight pre-training corruption can be beneficial for DMs.Pre-training Setup. Here, we adopt Latent Diffusion Models (LDMs) [9] with the pre-trained VQ-VAE [58, 56] and a down-sampling factor of 4 for the latent space of observed data x, denoted as LDM-4. More specifically, we train class-conditional and text-conditional LDM-4 from scratch on synthetically corrupted IN-1K [50] and CC3M [14] , respectively, with a resolution of 256 × 256 . We use a class embedding layer and a learnable pre-trained BERT [59] to compute the conditional embeddings of the IN-1K class labels and the CC3M text prompts. To introduce synthetic corruption into the conditions, we randomly flip the class label into a random class for IN-1K, and randomly swap the text of two sampled image-text pairs for CC3M, following [48, 49] (other corruption types studied in Section 3.3). We train models with different corruption ratios η ∈ {0, 2.5, 5, 7. Evaluation of Pre-trained Models. We directly use the pre-trained LDMs to generate images to study the effects of condition corruption in the pre-training stage. We use IN-1K class labels for class-conditional LDMs and MS-COCO text prompts [60] for text-conditional LDMs to generate 50K images and compare with the real validation images. The images are generated using a set of guidance scales s ∈ {1.5, 2.0, . . . , 10.0} and DPM [61] scheduler with 50 steps for faster inference speed 4 . We adopt Fréchet Inception Distance (FID) [63] , Inception Score (IS) [64] , Precision, and Recall [65] to evaluate the quality, fidelity, and coverage of the generated images. For CC3M models, we use the CLIP score (CS) [66] to measure the similarity of the generated images and conditional text prompts. From the perspectives of sample complexity and diversity, we compute the top-1% Relative Mahalanobis Distance (RMD) [67, 68] , calculated from the estimated class-specific and class-agnostic distributions of generated data, and the sample entropy [69, 70] , calculated from the VQ-VAE codebook. We also adopt other metrics, including sFID [71] , TopPR F1 [72] , average L 2 distance, and memorization ratio [73] . More details of the metrics used are shown in Appendix B.7.Results. We present the main quantitative results of pre-training in Fig. 3 and 4 , and the qualitative results in Fig. 5 . More results are shown in Appendix C. In summary, we found that slight pretraining corruption5 can facilitate the quality, fidelity, and diversity of generated images:• Class and text-conditional models pre-trained with slight corruption achieve significantly lower FID and higher IS and CLIP score (Fig. 3 (a) and 3(c)). They also present comparable and better Precision-Recall curves (Fig. 3 (b) and 3(d)), compared to clean pre-trained models.• Models pre-trained with slight corruption generate images with higher complexity and diversity, with a right-shifted density of RMD (Fig. 4 (a) and 4(c)), and larger entropy (Fig. 4 (b) and 4(d)).• Qualitatively, models with slight corruption learn a more diverse distribution. Generated images present better variability in the circular walk around the latent space (Fig. 5 (a) and 5(b)). • More corruption in pre-training can potentially lead to quality and diversity degradation. As η increases, almost all metrics first improve and then degrade. However, the degraded measure with more corruption sometimes is still better than the clean ones (e.g. IS and Entropy).A common scenario of DMs pre-trained on large-scale datasets is that they can be personalized and customized for more controllable generation [74, 75] after tuning on smaller datasets. Here, we also examine the effects of pre-training condition corruption of DMs at downstream personalization tasks.Setup. We personalize the pre-trained LDMs with two common methods: ControlNet [20] and T2I-Adapter [21] . Both methods can enable the pre-trained DMs to generate more controllable images according to input spatial conditioning. For fair comparison, we automatically annotate the ImageNet-100 (IN-100) dataset to canny edges using the OpenCV canny detector [76] and segmentation masks using SegmentAnything (SAM) [77] , similar to Zhang et al. [20] . For text-conditional LDMs, we additionally use BLIP [78] Evaluation of Personalized Models. After tuning, we evaluate the personalized LDMs in the annotated validation set of IN-100. We mainly compute FID, IS, Precision, and Recall to compare the models. We similarly use a set of guidance scales to generate images, but only report results of the best guidance scale in this part due to space limit. The complete results are shown in Appendix D.Results. Similarly, from the main results in Fig. 6 and Fig. 7 , we found that slight pre-training corruption can also benefit the quality of generated images at downstream personalization:• Slight pre-training corruption helps the personalized model generate images with lower FID (Fig. 6 (a) and 6(c)) and higher IS score (Fig. 6 (b) and 6(d)), whereas more corruption deteriorates. • Qualitatively, the personalization images from slight corruption pre-trained models also present more diversity, better fidelity with the input spatial controls, and higher quality. overlapped classes with CIFAR-100 [79] , while maintaining others as clean. For CC3M, we prompt GPT-4 [80] to corrupt the texts. More details of the corruption are shown in Appendix B.1.Diffusion Models. LDMs utilize U-Net [81] as backbone and Cross-Attention for adding conditional information [9] . We pre-train class-conditional diffusion transformers on IN-1K for extra assessment, termed DiT-XL/2 [11] , with Transformer [82] as backbone and adaptive LayerNorm [83] [84] [85] [86] for conditional information. We also pre-train the recent text-conditional Latent Consistency Models (LCMs) [52, 51] on CC3M, which distill Stable Diffusion v1.5 [26] models to enable swift inference with minimal steps, noted as LCM-v1.5. Detailed setup is shown in Appendices B.4 and B.5.Results. We present the main results in Fig. 2 due to the space limit. Full results are shown in Appendix C. We find that slight condition corruption of various types universally facilitates the performance of different DMs and consistently makes them outperform the clean pre-trained ones. 
theoretical analysis: 
 In this section, we theoretically analyze condition corruption and find that slight corruption prevents the generated distribution from collapsing to the empirical distribution of the training data and encourages coverage of the entire data space, thereby enhancing diversity and alignment with the ground truth. We present a concise overview here and provide a comprehensive analysis in Appendix A.Data Distribution. We concentrate on the prototypical problem of sampling from Gaussian mixture models (GMMs). Specifically, we consider the distribution of data x ∈ R d that satisfies:EQUATIONwhere y denote class labels of a finite set y ∈ {1, . . . , |Y|}. Given any class, x|y follows a Gaussian N (µ y , I), and w y represents the weight of the Gaussian components which satisfies y∈Y w y = 1.Denoising Networks and Condition Corruption. Inspired by recent works that also target on GMMs [87, 88] of DMs, we parameterize the denoising network as a piece-wise linear function:EQUATIONwhere c(y c ) is the one-hot encoding of corrupted label y c and {W k t , V k t } |Y| k=1 are trainable parameters. Specifically, following a line of existing work [89] [90] [91] , we adopt a simpler label-noise model by adding Gaussian perturbation to the label embedding, perturbing the clean condition c(y) with standard Gaussian noise ξ to obtain c(y c ) = c(y) + γξ. Here, the corruption control parameter γ corresponds to the corruption ratio η for a more direct noise magnitude control. While our theoretical framework focuses on Gaussian noise, it can also be extended to distributions such as uniform.We employ entropy to evaluate the diversity of generated images, following Wu et al. [70] . Higher entropy suggests a wider spread of data, yielding greater diversity in generated images, while lower entropy implies a more concentrated distribution with less diversity. We present Theorem 1, showing the difference in entropy between generations with corrupted and clean conditions: Theorem 1. For any class k ∈ Y and sufficiently large length T , assuming the norm of corresponding expectation ∥µ k ∥ 2 2 is a constant and the empirical covariance of training data is full rank, let z T and z c T be the generation with clean and corrupted conditions respectively, then it holds thatEQUATION)where γ is the corruption control parameter and d is the data dimension.The proof is provided in Appendix A.4.1. Theorem 1 indicates that for any class k, corrupted conditions enhance image diversity by increasing generation entropy. Moreover, with suitable γ values, image diversity can grow with noise, aligning with observations in Fig. 4 .We then analyze why corrupted conditions benefit the quality of generated images, as also observed in Section 3.1. We employ the 2-Wasserstein distance as a metric to evaluate the sampling error between the true and the generated distributions, with clean and corrupted conditions. A distributed generated closer to the real data distribution indicates better image quality [63] . In Theorem 2, we analyze the difference in the quality of data generated by corrupted DMs and clean ones: Theorem 2. For any k ∈ Y and sufficiently large length T , assuming the norm of corresponding expectation ∥µ k ∥ 2 2 is constant, let P, Q X and Q c X be the ground truth, clean, and corrupted condition distributions over training data X.EQUATIONwhere W 2 (•, •) denotes the 2-Wasserstein distance between two distributions, n k is the sample size of k-labeled dataset, and d is the data dimension.Here the expectation is taken over the random sample of the training dataset from the data distribution. Detailed proof is shown in Appendix A.4.2. Theorem 2 reveals that for any class k, small corruption yields generation distributions closer to the true distribution than clean ones. This partially verifies that the generation quality of the uncorrputly trained diffusion model can be improved by adding slight corruption to the training data. This is also well consistent with our empirical observation in Section 3.1, where the noise we used is approximately 0.04ϵ, close to the theoretical noise level of 0.03ϵ, showing that the FID of the generated images can be improved with a small corruption.5 Improving Diffusion Models with Conditional Embedding Perturbation 
method: 
 Our previous analysis demonstrates that slight condition corruption in the pre-training could potentially benefit both the image quality and diversity of DMs, which inspires us to improve the pre-training of DMs using this conclusion. In practice, it is usually infeasible to directly corrupt the conditions in the pre-training datasets either due to their large-scale nature or difficulties to select which conditions to corrupt. Instead, we propose to add the perturbation directly to the conditional embeddings of DMs, which is termed conditional embedding perturbation (CEP). Compared to the fixed proportion of condition corruption in datasets we studied before, CEP adds perturbation to every data instance during training on the fly. Specifically, CEP slightly modifies the DM objective:EQUATIONwhere δ denotes the perturbation added to conditional embeddings c θ (y). We simply set the perturba-tion to Uniform, i.e., δ ∼ U -γ √ d I, γ √ d I , or to Gaussian, i.e., δ ∼ N 0, γ √ d I, where the design of the factor γ √ d mainly follows previous works [82, [92] [93] [94] [95] , d denotes the dimension of c θ (y), and γ controls the perturbation magnitude, mimicing the corruption ratio η. The main purpose of CEP is to learn better DMs with perturbation on relatively clean and heavily filtered datasets, such as CC3M and IN-1K studied in this paper, but it is also applicable to slightly corrupted datasets. Recently, Ning et al. [96] found that adding input perturbations (IP) to latent variables z t during the forward process also helps diffusion training by mitigating exposure bias [97] . Compared to IP, CEP does not alter the marginal data distribution, but encourages the learned joint distribution to be more diverse.Setup. We pre-trained previous class-conditional LDM-4, text-conditional LDM-4, class-conditional DiT-XL/2, and text-conditional LCM-v1.5 with CEP, and compare with IP and clean pre-trained ones. We use both Uniform and Gaussian perturbation, denoted as CEP-U and CEP-G, respectively. We set γ = 1 for all models, with an ablation study with class-conditional LDM-4 with different γs. Cond. Emb. L2 Diff. 2 . Qualitatively, as shown in Fig. 9 , images generated from DMs with CEP also look more visually appealing and realistic.The ablation results of γ are shown in Fig. 8(a) . We also compare the average L 2 distance of CEP and fixed corruption against the clean condition embeddings. Interestingly, one can observe that CEP achieves a lower FID with more corruption in the embedding space (larger L 2 from the clean ones), demonstrating its effectiveness. CEP is applicable to slightly corrupted datasets that we may often encounter in practice, as shown in Fig. 8(b) , where it also facilitates the performance significantly.In addition, we also compare the proposed CEP with traditional regularization methods, such as Dropout [98] and Label Smoothing [99] , and study the effects of fixed and random perturbation during training in Appendix E.3 and Appendix E.4. The results show that CEP is more effective. 
related work: 
 Diffusion Models. Inspired by thermodynamics, DMs were first proposed by Sohl-Dickstein et al. [100] . DMs have soon been developed into image generation with a fixed Gaussian noise diffusion process [1, 101] . Various techniques have then been proposed for more effective and efficient DMs [102, 4, 5] . One of the most well-known is modeling the diffusion process at the latent space of pre-trained image encoders as a strong prior [58, 56] , instead of raw pixels spaces [3, 9, 11] , which allows for high-quality image generation with affordable inference speed. Numerous foundational DMs that generate photorealistic images have thus been built [103-107, 10, 108, 26, 109] . These powerful models are generally pre-trained on web-crawled billion-scale data with conditions (usually text), which may inevitably contain corruption [31, 110, 111, 32, 35] . Recently, consistency models [51, 112, 52] were also developed from DMs, allowing generation with much fewer inference steps. These foundational DMs also enabled many downstream applications [20, 21, [113] [114] [115] [116] [117] [118] [119] [120] [121] [122] . However, the effects of the pre-training corruption on downstream applications remain unknown.Learning with Noise. Learning with noise is a long-standing challenge [123] [124] [125] [126] . Noisy label learning has been widely studied in classification, from noise correction [127, 40, 128-131, 41, 132, 133, 43, 134, 44, 135] and noise-robust loss functions [39, 136-141, 29, 142] . Learning with noise has also been studied in the context of generative models [143] [144] [145] [146] . Robust GANs and DMs [45] [46] [47] alleviated the quality degradation and condition misalignment of training generative models with label noise. In contrast, we study a more practical scenario, where the models are trained on corrupted pre-training data with a low noise ratio, and then adapted to downstream tasks.In fact, more aligned with our work, there are several recent studies on exploring and exploiting the pre-training noise. Chen et al. [48, 49] found that slight label noise in supervised pre-training can be beneficial for in-domain downstream tasks, whereas detrimental for out-of-domain tasks. NoisyTune [147] , NEFTune [95] , and SymNoise [148] found that introducing noise to the weights and embedding of pre-trained language models can facilitate downstream performance. Ning et al. [96] also found that adding perturbation in the forward diffusion process helps reduce the exposure bias of DMs [97] . Similarly, Naderi et al. [149] introduced noise into the input of image translation networks for better learning with limited data. Synthetic data (potentially with corruption) have also been found to be useful in pre-training [24, 150] . We demonstrate that slight corruption in conditions of the pre-training DMs can also be beneficial at both the pre-training and downstream. 
conclusion and limitation: 
 We presented the first comprehensive study on condition corruption in pre-training of DMs. Our empirical and theoretical analysis surprisingly demonstrate that slight condition corruption benefits DMs in both the pre-training and downstream adaptation, based on which we proposed CEP as a simple yet general technique that significantly improves the performance of DMs. We hope our findings could inspire more future work on understanding the pre-training data of foundation models. This work has the following limitations. First, due to a lack of computing resources, we cannot study all types of DMs on larger datasets. Second, the theoretical analysis is based on several assumptions that might be further explored in the future. Third, the evaluation of image generation remains an open question, and we used most of the existing criteria for fair comparison. 
 
 Appendix: While we study DMs for image generation in this paper, it is important to note that all generations have been selected and verified by human experts to ensure that they are responsible. Although we release all the pre-trained models under different corruption settings, it is possible that these models will generate inappropriate content due to the scale of pre-training and without alignment with human preferences. The main purpose of this research is to raise the awareness of the community on data cleaning and corruption in the research of diffusion models.In this section, we theoretically investigate the behavior of condition corruption on DMs. Our investigation encompasses the DDIM samplers with continuous-time processes. We focus on how slight conditional embedding corruption can affect the training process of DMs, as well as the consequences on the generative process. As our theoretical analysis indicates, slight conditional embedding corruption will benefit both the generation quality and diversity, which aligns with the experimental conclusions in Section 3.We start by giving a concise overview of the problem setup.Data Distribution. For precise theoretical characterizations, we concentrate on the prototypical problem of sampling from Gaussian mixture models (GMMs). Specifically, we consider that the distribution of the data x ∈ R d satisfiesEQUATIONHere, y is denoted as the class labels with a finite set of values y ∈ {1, 2, • • • , |Y|}. Given any class label y ∈ Y, the data distribution x|y is a Gaussian with the center and covariance as (µ y , I). And the positive w y represents the weights of the Gaussian components which satisfies y∈Y w y = 1.The diffusion model is a two processes framework: a forward process that transforms the target distribution into Gaussian noise, and a reverse process that progressively denoises in order to reconstitute the original target distribution. In this paper, we consider the continuous-time processes and define the forward process as an Ornstein-Uhlenbeck (OU) process:Forward Process. At time step t ∈ [0, T ], the forward process isEQUATIONwhere w t is a d-dimensional standard Brownian motion.The advantage of considering the forward process as an OU process is that it enables us to directly derive the closed-form expression for the conditional sample distribution at any given time t.EQUATIONwhere r t = e -t andσ t = √ 1 -e -2t .By the reparameterization trick, x t can be represented asEQUATIONWe explore the widely adopted sampling method, DDIM, augmented with classifier-free guidance. And the associated reverse process is stated as the following ODE implementation:Reverse Process. We write the reverse process in a forward version by switching time direction t → Tt asEQUATIONwhere w ≥ 0 is a hyperparameter that controls the strength of the classifier guidance.Given our primary focus on the impact of corrupted conditional embedding on generation, we simplify the reverse process by setting w to 0 and concentrate solely on the conditional score network, i.e.,EQUATION)The remaining task is to estimate the unknown conditional score function ∇ zt log P(z t |y) and unconditional score function ∇ zt log P(z t |y) via training. In the subsequent analysis, we will indicate that by minimizing Equation (3) and optimizing the denoising network ϵ, we can achieve an estimate of the conditional score.Denoising Networks. Inspired by recent work that also target on GMMs [87, 88] , we parameterize the denoising networks as the following piecewise linear function:EQUATIONwhere c(y) is the one-hot encoding of label y, 1 y is an indicator function and{W k t , V k t } |Y| k=1 are trainable parameters.Conditional Embedding Corruption. We examine the scenario of conditional embedding corruption, wherein the conditional embedding c(y) is no longer matched with the data x, but instead is perturbed by Gaussian noise. Consequently, the corrupted conditional embedding causes the denoising networks to be as follows:EQUATIONwhere the d-dimensional corrupted noise ξ ∼ N (0, I) and γ ≥ 0 serves as the corruption control parameter, mirroring the noise ratio η for direct control of noise magnitude.Based on the aforementioned setup, our ultimate goal is to obtain the closed form of the final generated data distribution by considering the impact of corrupted noise ξ on the training and generative processes. By comparing the differences in data distribution before and after the addition of corrupted noise, we aim to accurately characterize the impact of embedding corruption on image generation and explain the phenomena observed in experiments.We first analyze the case of clean conditional embedding. Note that the denoising networks ϵ θ is a piecewise linear function and the training objective can be represented asEQUATIONwhere the class sample size n k := nw k .We observed that the optimization objective in Equation ( 17) can be divided into |Y| independent sub-problems based on the label y. This inspires us to analyze the training and generation processes according to different classes.Given any class k ∈ Y, we present the following lemma for determining the optimal parameters of the corresponding denoising network and the associated conditional score. Lemma 1. (Clean Conditional Embedding). Given any class k ∈ Y, the optimal linear denoising network isEQUATION)And the corresponding optimal linear estimation of conditional scoreEQUATIONwhereΣ k := 1 n k n k i=1 x i x T i -1 n 2 k n k i=1 x i n k i=1 x T i is the empirical covariance of k-labeled dataset and μk := 1 n k n k i=1 x i is the empirical mean of k-labeled dataset, n k is the sample size of k-labeled dataset. And r t = e -t , σ t = √ 1 -e -2t .Proof. Given any class k, the training objective isEQUATIONSince the weights w k is fixed in our analysis, we omit the notation w k without ambiguity in the following contents and for enhanced clarity, we initially omit the subscript/superscript k in n k , W k t , V k t , μk , µ k and Σ k . Then Equation ( 20) can restated as as Equation ( 21) for simplicity.EQUATIONwhere x i is labeled as k.This training loss can be further simplified as1 n n i=1 E ϵ [∥W t r t x i + W t σ t ϵ + V t c(y) -ϵ∥ 2 2 ] = 1 n n i=1 ∥W t r t x i + V t c(y)∥ 2 2 + E ϵ [∥(W t σ t -I)ϵ∥ 2 2 ] = r 2 t n n i=1 x T i W T t W t x i + 2r t n e T (y)V T t W t n i=1 x i + e T (y)V T t V t c(y) + Tr (W t σ t -I)(W T t σ t -I) .For simplicity, we denote b t := V t c(y) and we get1 n n i=1 E ϵ [∥W t r t x i + W t σ t ϵ + V t c(y) -ϵ∥ 2 2 ] = r 2 t n n i=1 x T i W T t W t x i + 2r t n b T t W t n i=1 x i + b T t b t + Tr (W t σ t -I)(W T t σ t -I) .Then, we define the loss functionJ(W t , b t ) = r 2 t n n i=1 x T i W T t W t x i + 2r t n b T t W t n i=1 x i + b T t b t + Tr (W t σ t -I)(W T t σ t -I) .The optimal W * t and b * t can be obtained by taking gradient to J(W t , b t ) such that,0 = ∇ Wt J(W t , b t ) = 2r 2 t n k W t n i=1 x i x T i + 2r t n b t n i=1 x T + 2(σ 2 t W t -σ t I), 0 = ∇ bt J(W t , b t ) = 2r t n W t n i=1 x i + 2b t .And the optimal W * t and b * t is,W * t = σ t σ 2 t I + r 2 t Σ -1 , b * t = V * t c(y) = -r t σ t σ 2 t I + r 2 t Σ -1μ.whereΣ := 1 n n i=1 x i x T i -1 n 2 k n i=1 x i n i=1x T i is the empirical covariance of k-labeled dataset and μ := 1 n n i=1 x i is the empirical mean of k-labeled dataset. Based on above analyze, for class k, we then have the following optimal denoising network as the linear estimator of noise ϵ,ϵ θ (x t , y = k) = σ t σ 2 t I + r 2 t Σ -1 x t -r t σ t σ 2 t I + r 2 t Σ -1 μ.Given any class k, we derive∇ xt log P(x t |x, y = k) = - (x t -r t x) σ 2 t = - ϵ σ t ,where data x and x t are labeled with k.Therefore, the linear estimator of∇ xt log P(x t |x, y = k) given k-labeled data x t is -σ 2 t I + r 2 t Σ -1 x t + r t σ 2 t I + r 2 t Σ -1 μ.Since the estimation of ∇ xt log P(x t |x, y = k) and ∇ xt log P(x t |y = k) are equivalent in optimization, the optimal linear estimator ∇ xt log P(x t |y = k) also can be∇ xt log P(x t |y = k) = - ϵ θ (x t , y = k) σ t = -σ 2 t I + r 2 t Σ -1 x t + r t σ 2 t I + r 2 t Σ -1μ.The proof is completed.Note that the ground truth functions of conditional score given the label k isEQUATION)We note the similarities in form and coefficients between the optimal linear estimator ( 19) and ground truth (22) ; they both are linear combinations of µ k / μk and x t . Specifically, when the empirical covariance Σ k equals the population covariance I and the empirical mean μk matches the expected value µ k , the estimated conditional score in Equation ( 19) coincides with the true conditional score in Equation (22) .The same analytical approach as in Section A.2.1 will be applied to the scenario where the conditional embedding is perturbed. Given the class k, we propose the following Lemma 2 to describe the optimal linear denoising network and the conditional score estimator with the corrupted conditional embedding (c(y) + γξ) where ξ is the standard Gaussian noise. Lemma 2. (Corrupted Conditional Embedding). Given any class k ∈ Y, the optimal linear denoising network isEQUATION)And the corresponding optimal linear estimation of conditional score ∇ xt log P c (x t |y = k) isEQUATION)whereΣ k := 1 n k n k i=1 x i x T i -1 n 2 k n k i=1 x i n k i=1 x T i is the empirical covariance of k-labeled dataset and μk := 1 n k n k i=1 x i is the empirical mean of k-labeled dataset, n k is the sample size of k-labeled dataset, and r t = e -t , σ t = √ 1 -e -2t .Proof. For enhanced clarity, we initially omit the subscript/superscriptk in n k , W k t , V k t , μk , µ k and Σ k .For any class k and standard Gaussian noise ξ, we consider the following training lossEQUATIONAnd we further optimize the training loss as1 n n i=1 E ξ E ϵ [∥W t r t x i + W t σ t ϵ + V t (c(y) + γξ) -ϵ∥ 2 2 ] = 1 n n i=1 ∥W t r t x i + V t c(y) + γV t ξ∥ 2 2 + E ϵ [∥(W t σ t -I)ϵ∥ 2 2 ] = r 2 t n n i=1 x T i W T t W t x i + 2r t n e T (y)V T t W t n i=1 x i + e T (y)V T t V t c(y) + γ 2 Tr(V t V T t ) + Tr (W t σ t -I)(W T t σ t -I) .Similarly, we get the optimal W * t and b * t is,W * t = σ t σ 2 t I + r 2 t Σ + r 2 t γ 2 1 + γ 2 ∥ μ∥ 2 2 I -1 , b * t = V * t c(y) = - r t 1 + γ 2 σ t σ 2 t I + r 2 t Σ + r 2 t γ 2 1 + γ 2 ∥ μ∥ 2 2 I -1μ.whereΣ := 1 n n i=1 x i x T i -1 n 2 k n i=1 x i n i=1x T i is the empirical covariance and μ := -1 n n i=1 x i is the empirical mean of k-labeled dataset. So for any class k, we get the following optimal denoising network as the linear estimator of noise ϵ,ϵ c θ (x t , y = k) = σ t σ 2 t I + r 2 t Σ k + r 2 t γ 2 1 + γ 2 ∥ μk ∥ 2 2 I -1 x t - r t 1 + γ 2 σ t σ 2 t I + r 2 t Σ k + r 2 t γ 2 1 + γ 2 ∥ μk ∥ 2 2 I -1μk .The corresponding optimal linear estimator ∇ xt log P c (x t |y = k) is∇ xt log P c (x t |y = k) = - ϵ θ (x t , y = k) σ t = -σ 2 t I + r 2 t Σ + r 2 t γ 2 1 + γ 2 ∥ μ∥ 2 2 I -1 x t + r t 1 + γ 2 σ 2 t I + r 2 t Σ + r 2 t γ 2 1 + γ 2 ∥ μ∥ 2 2 I -1μ.Given the class k, after getting the optimal linear estimator of conditional score ∇ xt log P(x t |y = k), we can accurately characterize the reverse process and derive the closed form of generation distribution.According to Lemma 1, we first replace the conditional score ∇ xt log P(x t |y = k) in Equation ( 14) with the optimal linear estimator. Given the class k, we getEQUATION)Assume the empirical covariance Σ k is full rank, since the empirical covariance is diagonalizable and semi-positive, we decompose Σ k asEQUATIONwhereU = [u 1 | • • • |u d] is an orthogonal matrix whose columns are the real, orthonormal eigenvectors of Σ k and Λ = diag(λ 1 , . . . , λ d ) is a diagonal matrix whose entries are the eigenvalues arranged in descending order of Σ k , i.e., 1≥ λ 1 ≥ • • • ≥ λ d > 0.Therefore, problem (26) can be decomposed into d independent sub-problem asEQUATIONwhere u i is i-th eigenvector and λ i is its corresponding eigenvalue.By analyzing Equation (28), we can obtain the expectation and variance of the final generation distribution separately, thereby inferring the distribution of the ultimately generated data as outlined in the subsequent lemma. Lemma 3. (Clean Conditional Embedding). For any class k, the distribution of the generated data z T satisfies limEQUATIONwhereΣ k := 1 n k n k i=1 x i x T i -1 n 2 k n k i=1 x i n k i=1 x T i is the empirical covariance of k-labeled dataset and μk := 1 n k n k i=1 x i is the empirical mean of k-labeled dataset, n k is the sample size of k-labeled dataset.Proof. For enhanced clarity, we initially remove the subscript k in μk , µ k and Σ k . In order to obtain the solution to Equation ( 28) and achieve the closed form of the generation distribution, we first examine the discrete solution of Equation (28) .Given any class k, we consider the discretization Euler-Maruyama scheme which is widely used in existing work [151] [152] [153] and discretize the interval [0, T ] into N discretization points,EQUATIONwhere h := T N is the step size and t j = jh. According to Equation ( 30), u T i z (j+1)h can be regarded as a linear transformation of the initial distribution, which is a standard Gaussian distribution. Therefore, u T i z (j+1)h still satisfies a Gaussian distribution. The remaining task is analyze the mean and variance of the u T i z (j+1)h separately. Expectation. By Equation ( 30), we haveE(u T i z (j+1)h ) = 1 + 1 - 1 σ 2 tj + r 2 (t j )λ i h E(u T i z jh ) + r(t j ) σ 2 tj + r 2 (t j )λ i u T i μh.By telescoping, we get the discretization solutionE(u T i z N h ) = N -1 k=0 1 + 1 - 1 σ 2 t k + r 2 (t k )λ i h E(u T i z 0 ) + N -1 k=0 r(t k ) σ 2 t k + r 2 (t k )λ i u T i μh N -1 j=k+1 1 + 1 - 1 σ 2 t j + r 2 (t j )λ i h .Since the initial distribution of reverse process is z 0 ∼ N (0, I), thus E(u T i z 0 ) = 0. We further haveEQUATIONBy limiting h → 0, we can use the identity 1 + hx ≃ exp(hx). Then we get the continous version of Equation ( 31) that when h → 0, we haveE(u T i z T ) = T 0 r t σ 2 t + r 2 t λ i exp T t 1 - 1 σ 2 r + r 2 (r)λ i dr u T i μdt = T 0 e t-T 1 + e 2(t-T ) (λ i -1) exp T t 1 - 1 1 + e 2(r-T ) (λ i -1) dr u T i μdt = T 0 e t-T √ λ i [1 + e 2(t-T ) (λ i -1)] 3 2 u T i μdt = λ i e t-T 1 + e 2(t-T ) (λ i -1) T 0 u T i μ = 1 - √ λ i e -T 1 + (λ i -1)e -2Tu T i μ.Hence, we have E(z T ) = Udiag(e 1 , e 2 , . . . , e d )U T μ,where e i = 1 - √ λie -T √ 1+(λi-1)e -2T . When T → ∞, we get e i = 1 - √ λ i e -T 1 + (λ i -1)e -2T → 1.And the expectation of generation distribution is the empirical mean, i.e., E(z T ) = μ.Variance. Similarly, by Equation ( 30), we have the varianceVar(u T i z (j+1)h ) = 1 + 1 - 1 σ 2 tj + r 2 (t j )λ i h 2 Var(u T i z jh ).By telescoping, we get the discretization solutionVar(u T i z N h ) = N -1 k=0 1 + 1 - 1 σ 2 t k + r 2 (t k )λ i h 2 Var(u T i z 0 ). Since Var(u T i z 0 ) = u T i Var(z 0 )u i = 1, the variance at time T is Var(u T i z N h ) = N -1 k=0 1 + 1 - 1 σ 2 t k + r 2 (t k )λ i h 2 .When h → 0 and use the identity (1 + hx) 2 ≃ exp(2hx), we getVar(u T i z T ) = exp T 0 2 1 - 1 σ 2 t + r 2 t λ i dt = exp T 0 2 1 - 1 1 + (λ i -1)e 2(t-T ) dt = λ i 1 + (λ i -1)e -2T .Var(z T ) = Udiag(v 1 , v 2 , . . . , v d )U T .where v i = λi 1+(λi-1)e -2T . When T → ∞, we getv i = λ i 1 + (λ i -1)e -2T → λ i .And the expectation of generation distribution is the empirical mean, i.e., Var(z T ) = Σ.We complete the proof.We then discuss the distribution of generated data when the conditional embedding is perturbed by noise. Using the same method to derive the data distribution under the corrupted conditional embedding setting, we conclude the following Lemma 4.EQUATIONwhereΣ k := 1 n k n k i=1 x i x T i -1 n 2 k n k i=1 x i n k i=1x T i is the empirical covariance of k-labeled dataset and μk := 1 n k n k i=1 x i is the empirical mean of k-labeled dataset, n k is the sample size of k-labeled dataset. And γ ≥ 0 is the corruption control parameter. Proof. To improve clarity, we initially disregard the subscript k in μk , µ k and Σ k . Building upon the proof details presented in Lemma 3 and the optimal conditional score estimation outlined in Lemma 2, we arrive at the subsequent findings:Expectation. Similarly, we deriveE(u T i z c T ) = 1 1 + γ 2 T 0 r t σ 2 t + r 2 t (λ i + γ 2 1+γ 2 ∥ μ∥ 2 2 ) exp T t 1 - 1 σ 2 r + r 2 (r)(λ i + γ 2 1+γ 2 ∥ μ∥ 2 2 ) dr u T i μdt = 1 1 + γ 2 T 0 e t-T 1 + e 2(t-T ) (λ i -1 + γ 2 1+γ 2 ∥ μ∥ 2 2 )) exp T t 1 - 1 1 + e 2(r-T ) (λ i -1 + γ 2 1+γ 2 ∥ μ∥ 2 2 )) dr u T i μdt = 1 1 + γ 2 T 0 e t-T λ i + γ 2 1+γ 2 ∥ μ∥ 2 2 ) [1 + e 2(t-T ) (λ i -1 + γ 2 1+γ 2 ∥ μ∥ 2 2 )] 3 2 u T i μdt = 1 1 + γ 2 λ i + γ 2 1 + γ 2 ∥ μ∥ 2 2 e t-T 1 + e 2(t-T ) (λ i -1 + γ 2 1+γ 2 ∥ μ∥ 2 2 ) T 0 u T i μ = 1 1 + γ 2 1 - λ i + γ 2 1+γ 2 ∥ μ∥ 2 2 e -T 1 + (λ i -1 + γ 2 1+γ 2 ∥ μ∥ 2 2 )e -2Tu T i μ.Hence, we haveE(z c T ) = Udiag(e c 1 , e c 2 , . . . , e c d )U T μ,wheree c i = 1 1+γ 2 1 - λi+ γ 2 1+γ 2 ∥ μ∥ 2 2 e -T 1+(λi-1+ γ 2 1+γ 2 ∥ μ∥ 2 2 )e -2T.When T → ∞, we gete c i = 1 1 + γ 2 1 - λ i + γ 2 1+γ 2 ∥ μ∥ 2 2 e -T 1 + (λ i -1 + γ 2 1+γ 2 ∥ μ∥ 2 2 )e -2T → 1 1 + γ 2 .E(z c T ) = 1 1 + γ 2 μVariance. We getVar(u T i z c T ) = exp T 0 2 1 - 1 σ 2 t + r 2 t (λ i + γ 2 1+γ 2 ∥ μ∥ 2 2 dt = exp T 0 2 1 - 1 1 + (λ i -1 + γ 2 1+γ 2 ∥ μ∥ 2 2 )e 2(t-T ) dt = λ i + γ 2 1+γ 2 ∥ μ∥ 2 2 1 + (λ i -1 + γ 2 1+γ 2 ∥ μ∥ 2 2 )e -2T.Hence, we haveVar(z c T ) = Udiag(v c 1 , v c 2 , . . . , v c d )U T ,whereEQUATION2 )e -2T . When T → ∞, we getv c i = λ i + γ 2 1+γ 2 ∥ μ∥ 2 2 1 + (λ i -1 + γ 2 1+γ 2 ∥ μ∥ 2 2 )e -2T → λ i + γ 2 1 + γ 2 ∥ μ∥ 2 2 .And the expectation of generation distribution is the empirical mean, i.e.,Var(z c T ) = Σ + γ 2 1 + γ 2 ∥ μ∥ 2 2 I.A.4 Diversity and Quality: Clean vs. Corrupted ConditionsGiven any class k, building on previous work [70] , we consider entropy to measure the diversity of generated images. In particular, let P and P c be the probability densities for the generated data using clean and corrupted conditional embeddings respectively, we have the following for z t and z c tEQUATIONEQUATION)We propose the following theorem to describe the difference between these two conditional differential entropy. Theorem 3. (Restatement of Theorem 1) For any class k ∈ Y, assuming the norm of corresponding expectation ∥µ k ∥ 2 2 is a constant and the empirical covariance of training data is full rank, let z T and z c T be the generation featuring clean and corrupted conditions respectively, then it holds thatEQUATION)where γ is the corruption control parameter and d is the data dimension.Proof. For the sake of clarity, we begin by omitting the subscript k from μk , µ k and Σ k . Given any class k, since both the clean generation z T and the corrupted generation z c T follow multivariate Gaussian distributions, we can derive the closed-form expression for the difference in their differential entropy by Lemma 3 and Lemma 4 as followsH(z c T |y = k) -H(z T |y = k) = 1 2 log |Σ + γ 2 1 + γ 2 ∥ μ∥ 2 2 I| - 1 2 log |Σ| = 1 2 d i=1 log 1 + γ 2 (1 + γ 2 )λ i ∥ μ∥ 2 2 ,whereΣ := 1 n n i=1 x i x T i -1 n 2 k n i=1 x i n i=1x T i is the empirical covariance of k-labeled dataset and μ := 1 n n i=1 x i is the empirical mean of k-labeled dataset. When the noise ratio γ is small and λ i = ω(γ),log 1 + γ 2 (1 + γ 2 )λ i ∥ μ∥ 2 2 = γ 2 λ i ∥ μ∥ 2 2 + O(γ 2 ).Therefore,H(z c T |y = k) -H(z T |y = k) = 1 2 d i=1 log 1 + γ 2 (1 + γ 2 )λ i ∥ μ∥ 2 2 = Θ(γ 2 d).Before starting proving that slight noise is beneficial to the quality of generation, we first introduce the lemmas required for the proof. Lemma 5. Given x 1 , • • • , x n independent and all distributed as a Gaussian N (µ, I). Then,EQUATION)Proof. The expectation isE(x i |x 1 + • • • + x n = z) = E(z -x 1 -• • • -x i-1 -x i+1 -• • • , x n |x 1 + • • • + x n = z) = z -(n -1)E(x i |x 1 + • • • + x n = z).Therefore,E(x i |x 1 + • • • + x n = z) = z n .By the law of total varianceE(Var(x i |x 1 + • • • + x n = z)) = Var(x i ) -Var(E(x i |x 1 + • • • + x n = z)) = n -1 n I.We d c := W 2 N (µ k , I), N ( μk 1+γ 2 , Σ k + γ 2 1+γ 2 ∥ μk ∥ 2 2 I) . Theorem 4. (Restatement of Theorem 2) For any class k ∈ Y, assume the norm of corresponding expectation ∥µ k ∥ 22 is a constant, let P, Q X and Q c X be the ground truth, clean, and corrupted condition distributions, respectively, where X represents the collection of training data points. IfEQUATION)where W 2 2 (•, •) denotes the quadratic 2-Wasserstein distance between two distributions, n k is the sample size of k-labeled dataset and d is the data dimension.Proof. To express more clearly, we first omit the subscript k of n k , μk , µ k and Σ k . According to Lemma 3 and Lemma 4, we then can directly derive the closed form of Wasserstein distance between two Gaussian as • The squared Wasserstein distance between true data distribution and clean generation distributiond 2 = ∥ μ -µ∥ 2 2 + Tr(I) + Tr(Σ) -2Tr(Σ 1 2 ) = ∥ μ -µ∥ 2 2 + d + d i=1 λ i -2 d i=1 λ i .• The squared Wasserstein distance between true data distribution and corrupted generation distributiond 2 c = μ 1 + γ 2 -µ 2 2 + Tr(I) + Tr Σ + γ 2 1 + γ 2 ∥ μ∥ 2 2 I) -2Tr (Σ + γ 2 1 + γ 2 ∥ μ∥ 2 2 I)) 1 2 = μ 1 + γ 2 -µ 2 2 + d + d i=1 λ i + γ 2 1 + γ 2 ∥ μ∥ 2 2 -2 d i=1 λ i + γ 2 1 + γ 2 ∥ μ∥ 2 2 .The difference in squared Wasserstein distance isd 2 -d 2 c = - μ 1 + γ 2 -µ 2 2 + ∥ μ -µ∥ 2 2 - d i=1 γ 2 1 + γ 2 ∥ μ∥ 2 2 + 2 d i=1 λ i + γ 2 1 + γ 2 ∥ μ∥ 2 2 -λ i .We consider the expectation error to reduce the randomness of μ asE[d 2 -d 2 c ] = -E μ 1 + γ 2 -µ 2 2 + E ∥ μ -µ∥ 2 2errors caused by the mean We notice that the expectation error can be decomposed into two parts. The error of the first part being caused by the difference in means between the generated distribution and the true distribution. Since the distribution of empirical mean is μ ∼ N (µ, 1 n I) where n k is the sample size of k-labeled dataset, we get+ d i=1 E 2 λ i + γ 2 1 + γ 2 ∥ μ∥ 2 2 -λ i - γ 2 1 + γ 2 ∥ μ∥-E μ 1 + γ 2 -µ 2 2 + E ∥ μ -µ∥ 2 2 = - d n(1 + γ 2 ) 2 -( γ 2 1 + γ 2 ) 2 ∥µ∥ 2 2 + d n ≥ - d n + 2dγ 2 n -o(γ 4 ) + d n = 2dγ 2 n -o(γ 4 ).The second part is attributable to the difference in covariance.d i=1 E 2 λ i + γ 2 1 + γ 2 ∥ μ∥ 2 2 -λ i - γ 2 1 + γ 2 ∥ μ∥ 2 2 ≥ d i=1 E γ 2 ∥ μ∥ 2 2 √ λ i -γ 2 ∥ μ∥ 2 2 -o(γ 4 ) = γ 2 d i=1 E ( 1 √ λ i -1)∥ μ∥ 2 2 -o(γ 4 d).By the law of total expectationd i=1 E ( 1 √ λ i -1)∥ μ∥ 2 2 = d i=1 E ∥ μ∥ 2 2 E 1 √ λ i | μ -d (a) ≥ E d i=1 ∥ μ∥ 2 2 E(λ i | μ) -d (b) ≥ E ∥ μ∥ 2 2 d i=1 E(λ i | μ) -d.(a) and (b) achieve by the Jensen's inequality given 1 √ λi is a convex function By Lemma 5, we deriveTr E Var(x| μ) = Tr E Σ| μ = Tr UE Λ| μ U T = d i=1 E(λ i | μ) = n -1 n d.Hence, we getd i=1 E ( 1 √ λ i -1)∥ μ∥ 2 2 ≥ E ∥ μ∥ 2 2 d i=1 E(λ i | μ) -d = n n -1 -1 dE[∥ μ∥ 2 2 ] = n n -1 -1 d 2 n + d∥µ∥ 2 2 .The second part isd i=1 E 2 λ i + γ 2 1 + γ 2 ∥ μ∥ 2 2 -λ i - γ 2 1 + γ 2 ∥ μ∥ 2 2 ≥ γ 2 n n -1 -1 ( d 2 n + d∥µ∥ 2 2 ) -o(γ 4 d).Therefore, with small noise ratio γ, we then can conclude thatE[d 2 -d 2 c ] = -E μ 1 + γ 2 -µ 2 2 + E ∥ μ -µ∥ 2 2 + d i=1 E 2 λ i + γ 2 1 + γ 2 ∥ μ∥ 2 2 -λ i - γ 2 1 + γ 2 ∥ μ∥ 2 2 ≥ 2dγ 2 n + γ 2 n n -1 -1 d 2 n + d∥µ∥ 2 2 -o(γ 4 d).Noting that n/(n -1)-1 = -Θ(1/n) when n is large, then if the corruption level γ satisfies γ = O(1/ √ n), for any class k, we have E[d 2 -d 2 c ] = Ω γ 2 d n .This completes the proof.In this section, we provide detailed training setup of each diffusion models we studied in the main paper, the synthetic corruption for IN-1K and CC3M, the annotation process of IN-100, and the evaluation metrics we adopted.We mainly studied four types of condition corruption in this paper, with two datasets. For IN-1K, we used random symmetric and asymmetric condition corruption. For CC3M, we adopted text swapping and LLM re-writing corruption. We used several levels of corruption η = {0, IN-1K /y. Text Swapping Condition Corruption for CC3M. For CC3M, where y is text captions for the images, we randomly sample two pairs and swap the text of these two pairs to introduce condition corruptions. This mainly follows Chen et al. [48] , where very disruptive corruption is introduced.LLM Text Condition Corruption for CC3M. Text swapping corruption may not be common in practice for image-text datasets. Instead, we may encounter captions that have unmatched entities or partially unmatched sentences with the images. To study the text corruption in a more realistic scenarios, we use GPT-4 and prompt it to re-write the captions to introduce corruptions. We pre-define 5 levels of corruption in the prompt, and randomly sample a level as input to GPT-4.Here, we present the details of annotate ImageNet-100 for personalization of LDMs using ControlNet and T2I-Adapters. A few examples of the annotated images and captions are shown in Fig. 10 .Canny Edge. For canny edge, we directly use the Canny detector from OpenCV to annotate the images. We set the low threshold and high threshold of canny detector to 100 and 200 respectively.Segmentation Mask from SAM. We use SAM to annotate segmentation masks from IN-100 images. We directly use the colormap of the segmentation masks as input control to ControlNet and T2I-Adapters.Captions from BLIP. We use BLIP captioning model to generate captions for IN-100 for adapting text-conditional LDMs. LCM distills the pre-trained Stable Diffusion models to enable faster inference with fewer steps. We choose Stable Diffusion v1.5 as the teacher model and conduct distillation for 35K iterations, which takes 1.5 days on 8 NVIDIA A100 GPUs. We use a learning rate of 1e-5.We use the implementation of ControlNet and T2I-adapter of Diffusers [155] for downstream personalization tasks. Default learning rate and batch size from Diffusers are used for these two methods, and we set the training epochs for IN-100 as 10. On 4 NVIDIA V100 GPUs, training ControlNet and T2I-Adapter with LDM-4 takes about 6 hours.We introduce the details of metrics we used to evaluate the diffusion models here. Due to the known difficulties of evaluating generative models, we adopt most of the existing criteria to evaluate the models we have trained.Fréchet Inception Distance (FID) [63] . FID measures the distance between real and generated images in the feature space of an ImageNet-1K pre-trained classifier [156] , indicating the similarity and fidelity of the generated images to real images.sFID [71] . sFID utilizes the mid-level features of the inception network [156] , which are more sensitive to spatial variability.Inception Score (IS) [64] . IS also measures the fidelity and diversity of generated images. It consists of two parts: the first part measures whether each image belongs confidently to a single class of an ImageNet-1K pre-trained image classifier [156] and the second part measures how well the generated images capture diverse classes.Precision and Recall [65] . The real and generated images are first converted to non-parametric representations of the manifolds using k-nearest neighbors, on which the Precision and Recall can be computed. Precision is the probability that a random generated image from estimated generated data manifolds falls within the support of the manifolds of estimated real data distribution. Recall is the probability that a random real image falls within the support of generated data manifolds. Thus, precision measures the general quality and fidelity of the generated images, and the recall measures the coverage and diversity of the generated images.Top-1% Relative Mahalanobis Distance (RMD) Score [67] . RMD score measures the sample complexity and difficulty. It is defined as the difference between the Mahalanobis distances of a sample induced by the class-specific and class-agnostic Gaussian distributed estimated from the generated data. Given the dataset {(x i , y i )} i∈[N ] , we first compute the features using the CLIP ViT-B-16 encoder from the images as G(x). The class-specific Gaussian distribution is then estimated:EQUATIONThe class-agnostic Gaussian distribution is estimated over all data as;EQUATIONThe RMD is defined as:EQUATIONWe compute the RMD score for all generated images, and report only the top-1% of them.Average Top-5 L 2 Distances. As an additional metric of sample diversity, we compute the L 2 distance of each generated image with the top-5 nearest neighbor training images. To reduce computation requirement of searching over the raw pixel space, we use the CLIP ViT-B-16 image encoder [157] to transform images into the feature space before calculating the L 2 distance. This metric measures the distance of generated samples with training images, as a proxy evaluation of diversity and memorization.TopPR F1 [72] . TopPR is a set of reliable evaluation metrics with statistically consistent estimates of generated data and real data. We use the F1 score, computed from the TopPR Precision and Recall as an additional metric to evaluate the general quality and diversity of generated images.CLIP Score [66] . CLIP score measures the cosine similarity between the CLIP embedding of an image-text pair. It is widely used as a metric to evaluate the fidelity and alignment of the generated images and the conditional text prompts [9] .Memorization Ratio [73] . We compute the memorization ratio as the percentage of generated images whose L 2 distances with their nearest neighbor training images are below a pre-defined threshold. We compute the distances in the feature space of CLIP ViT-B-16 image encoder [157] due to the massive size and resolution of the training images and set the threshold as 0.12. Although there are several studies using the distance comparison between the first and second nearest neighbor as a reflection of memorization [158, 159] , we found that this metric is not effective for large-scale datasets.Entropy. We compute the entropy metric within the latent space of the pre-trained VQ-VAE [58, 56] . Since LDMs (and DiT) learn the data distribution from the latent space of VQ-VAE, we can compute the sample entropy E[H(x)] using the generated and flatten latent vector x ∈ R HW ×D and the codebook C ∈ R C×D of VQ-VAE, where H and W are the height and weights of the original latent vectors, D indicates the dimension of the latent space, and C denotes the number of embeddings of the codebook. We compute the probability of each latent vector as Softmax ∥x -C∥ 2 2 /τ , where τ is a temperature parameter controlling the sharpness of the probability. We compute entropy as:EQUATIONIn this section, we present all results of our pre-training evaluation, over different diffusion models, including LDM-4, DiT-XL/2, and LCM-v1.5, and various types of condition corruption.We present the full evaluation results of IN-1K class-conditional LDMs and CC3M textconditional LDMs in Fig. 11 and Fig. 12 respectively. All the results are computed from using a set of guidance scales.For IN-1K LDMs, we use s ∈ Guidance Scale {1.5, 1.75, 2.0, 2.25, 2.5, 3.0, 3.5, 4.0, 4.5, 5.5, 6.0, 6.5, 7.0, 7.5, 8.0, 9.0, 10.0}. For CC3M LDMs, we use s ∈ {1.5, 2.0, 2.5, 2.75, 3.0, 3.25, 3.5, 4.0, 5.0, 6.0, 7.0, 7.5, 8.0, 10.0} For all the metrics, including FID, IS, Precision, Recall, sFID, TopPR F1, and CLIP score, we can all observe that slight condition corruption makes LDMs perform better, with improved image quality and diversity. We also observe that, when there is condition corruption in the dataset, the memorization ratio based on L 2 distances actually decreases, in line with observations as in Gu et al. [158] .By default we use DPM scheduler for generating the images with 50 inference steps. But we also study the generation of DDIM scheduler with 250 inference steps, as adopted in [9] . Due to the computation cost of running DDIM scheduler for 250 steps, we only study it with IN-1K LDM-4.The results are shown in Fig. 13 . One can observe the same trends from the metrics using DPM and DDIM, demonstrating our findings are scheduler agnostic.We then show the pre-training results of DiT-XL/2 and LCM-v1.5 in Fig. 19 , where we primarily compute the FID, IS, Precision, and Recall. For DiT-XL/2, we use s ∈ {1.5, 1.75, 2.0, 2.25, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0, 5.5}.For LCM-v1.5, we use s ∈ {1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0, 5.5, 6.0, 6.5, 7.0, 7.5, 8.0, 9.0, 10.0}. Slight condition corruption also facilitates the performance by using the most suitable guidance scale.We additionally include the FID and IS trend along training for LDM IN-1K model with no corruption and 2.5% corruption, using a guidance scale of 2.5, as shown in Table 5 . Slight corruption begins to be effective at the very early stage of training.Finally, we present the results of LDMs pre-trained on CC3M with LLM corruption and IN-1K with asymmetric corruption, where similar observations still hold. Guidance Scale We present more visualization results of class-conditional LDM-4 in Fig. 18 , class-conditional DiT-XL/2 in Fig. 19 , text-conditional LDM-4 in Fig. 20 , and text-conditional LCM-v1.5 in Fig. 21 . One can observe that DMs pre-trained with slight condition corruption in general more visually appealing images. We present complete results of downstream personalization here.We show the results of ControlNet IN-1K LDM-4 in Fig. 22 , T2I-Adapter IN-1K LDM-4 in Fig. 23 , ControlNet CC3M LDM-4 in Fig. 24 , and T2I-Adapter CC3M LDM-4 in Fig. 25 . For all personalization experiments, we compute the results for both Canny and SAM spatial controls. Guidance scales of {1.25, 1.5, 2.0, 2.25, 2.5, 3.0, 4.0, 5.0, 6.0, 7.0} and {2.0, 3.0, 4.0, 5.0, 6.0, 6.5, 7.0, 7.5, 8.0, 8.5, 9.0, 10.0} are used for IN-1K models and CC3M models, respectively, for all experiments here.From the results, one can observe that models pre-trained with slight condition corruption also present the best performance in downstream personalization tasks.We present the qualitative comparison of ControlNet personalization results here. Since T2I-Adapter personalization results are similar but visually worse (quantitatively worse too), we skip their results. In Fig. 8 (a), we compute the L 2 distance of perturbed condition embeddings and the clean ones, as a measurement for the corruption levels (of CEP). Here, we elaborate how we compute the L 2 distances. For fixed corruption, we calculate the distances as:EQUATIONwhere θ * is learned from clean data. For CEP, we directly calculate the L 2 norm of sampled noise:EQUATIONHere, we additionally compare CEP with dropout and label smoothing on LDM IN-1K models, which are two alternatives that also introduce perturbations in class embeddings. The results are shown in Table 6 . One can observe that, both dropout and label smoothing have similar regularization effects on training diffusion models, whereas CEP-U and CEP-G is more effective.We further compare with fixed CEP corruption, and random data corruption, to study the effects of fixed and random perturbation to train diffusion models. For fixed CEP-U, we first select the samples to add perturbation, and then fix them during training. For random data corruption, we randomly choose samples during training to make their label noisy by flipping to other classes. From the results in Table 7 , we show that CEP works the best among all corruption methods. Also fixed CEP is more effective than adding data corruption (fixed and random). Random data corruption can be viewed as a CEP-variant with embeddings from flipping label instead of adding noise, and thus is also more effective than fixed data corruption. "A large fancy chandelier hanging over a room full of furniture""A blue motorcycle is parked on the side of a street" "A beach chair with an umbrella on a beach by the water""A teddy bear sitting near a rail over looking a crowded beach""A car and a city bus sit side by side on the street" "A wood bench sitting in a lush green field""A bunch of jackets hanging on the all in a room" Guidance Scale We use x for images in both the raw pixel space and the latent space of VQ-VAE[56].In Appendix C, we also present the results of IN-1K LDM-4 using the DDIM[62] scheduler with 250 steps.Slight corruption corresponds to η ≤ 7.5%, which might be common in practical large-scale datasets. 


Decision: In this work, authors present an interesting finding that slightly corrupting the training data used to train diffusion models can actually lead to better performance - they demonstrated it on models trained using ImageNet and CC-3M and also proposed a new algorithm which can be integrated in any standard diffusion model training paradigm. Reviewers acknowledged the fact that the paper is well-written, experiments are robust and show the efficacy of the proposed algorithm. Compute budgets permitting,  I found the work to be rigorous and liked the emphasis on an under-explored aspect within the whole diffusion model training literature and can be beneficial for the field to further explore.

I recommend acceptance - as a suggestion, I'd like to see how this technique can be integrated into large-scale diffusion model training where it can be expensive to tune the noise scale.

Reviews: review: paper_summary: paper_summary: This paper claims that slight corruption of the condition is beneficial for conditional diffusion models. Experimentally, they conducted experiments with label flips or text swaps in the dataset and observed the performance improvement over zero noise at noise levels of 2.5% to 5%. Theoretically, they used a Gaussian mixture model to show that generation diversity and quality, as measured by entropy and Wasserstein distance, can be improved with a small amount of state corruption. Based on this, they proposed a method for injecting small amounts of Gaussian noise into the condition embeddings and experimentally demonstrated its superiority.
strength_weakness: strength_weakness: * From my understanding, in Theorem 2, $\gamma$ is $O(1/\sqrt{n})$, which is the scenario for small corruption. It would be beneficial to address this explicitly in the main text.

* It would be beneficial if the method that determines the noise level could be linked to the theory like using the $\gamma=O(1/\sqrt{n})$,.

* They briefly mentioned the various assumptions in their theory as limitations. It would be helpful to provide a detailed explanation of these assumptions and the scenarios in which they hold.
questions: questions: Please see the Weaknesses part.
limitations: limitations: They mentioned in the last section.
paper_summary: paper_summary: In this paper, the authors study the effect of slight corruptions to the training data during pretraining of conditioned diffusion models. They introduce the perturbations to the "condition" in the diffusion models and show that this leads to a better model (via FID and other metrics).  They also provide an intuition on why corruptions help by theoretically studying this case on a GMM.
strength_weakness: strength_weakness: - While authors showed extensively that adding noise can help, they missed to show "when does it help"? For example, if I'm a practitioner who wants to train a large model from scratch, I usually will not have the budget to train multiple models to determine the amount of noise that needs to be added. That insight is missing in this paper.
questions: questions: - Does the optimal noise perturbation level transfer between the models? If 10% is the best noise level for LDM on one dataset, would it be the best noise for the same-size LDM on another?
- How do metrics look like throughout training for models trained with different levels of noise?
- CC3M corruption details are not clear. I went to Appendix B1, you mentioned 5 levels of corruption, but it is still vague! What are these 5 levels? 
- How is the number of training iterations determined for the models? I looked at the appendix and made some rough calculations, it seems like you trained the IN model for 91 epochs and CC3M for 67 epochs (Pls correct me if I'm wrong). On what basis are these numbers decided? This again ties back to the question, when is "adding noise" helpful? Are we in an overtraining regime? Is that the reason why adding corruption acts as "augmentation" and hence the metrics improve?

- (Minor) A few relevant citations are missing. For example, adding perturbations to the conditioning both in text space and latent space is explored in the context of memorization in last year's Neurips paper [1].

[1] - "Understanding and mitigating copying in diffusion models." Advances in Neural Information Processing Systems 36 (2023): 47783-47803.
limitations: limitations: n/a
paper_summary: paper_summary: The paper investigates the impact of slight corruption of conditioning
information in pre-training data on the performance of diffusion models (DMs).
By introducing synthetic corruption to ImageNet-1K and CC3M datasets, the study
evaluates over 50 conditional DMs. Empirical and theoretical analyses reveal
that slight corruption enhances the quality, diversity, and fidelity of
generated images. Based on these insights, the work proposes Conditional
Embedding Perturbations (CEP) as a method to improve DM training which shows
significant improvements in both pre-training and downstream tasks.
strength_weakness: strength_weakness: - **Notion of significance:** The paper claims that slight pre-training
   corruption yield significantly better performance in terms of FID and IS
   metrics. Nevertheless, this notion of significance is never formally tested.
   For this at least a experiment with multiple repetitions should be performed
   to determine statistical significance.
 - **Typos:**, Page 8, lines 257 and 259 rrecision -> precision, Page 9, lines 299 dataset -> data
 - **Link between theoretical analysis and metrics** should be made explicit.
   I.e. pointing out that the FID is essentially a Gaussian approximation
   based estimated for the 2-Wasserstein distance of inception features (preferably at page 7, line 223).
questions: questions: None
limitations: limitations: The authors correctly reflect the limitations of the study. In particular, the
theoretical analysis is limited due to the assumptions made, but can still shed
some light on the underlying mechanism.  Further, the general issue of
evaluating image generative models quantitatively is pointed out and
a limitation of the work.
paper_summary: paper_summary: The authors find that slight corruption in the conditioning of text-to-image diffusion models improves the performance. The authors theoretically analyze this empirical finding in a toy model where the goal is to learn to sample from a Gaussian Mixture and the network is piece-wise linear function. Inspired by their experimental and theoretical findings, the authors propose a technique to improve the performance of text-conditional diffusion models by adding noise to the embedding of the conditioning vector.
strength_weakness: strength_weakness: * I believe that the authors should emphasize in the title of their paper and in the main body that they study corruption in the conditioning. Numerous works study corruption in the images themselves, e.g. see:

1) Ambient Diffusion: Learning Clean Distributions from Corrupted Data
2) Consistent Diffusion Meets Tweedie: Training Exact Ambient Diffusion Models with Noisy Data
3) GSURE-Based Diffusion Model Training with Corrupted Data
4) Solving inverse problems with score-based generative priors learned from noisy data

In these works, it has been observed that data corruption decreases the performance of diffusion models. I believe the authors should acknowledge these relevant works and clarify that their paper analyzes corruption in the conditioning.

* The theoretical model is not very relevant, at least not very relevant to Section 3 of the paper. In the experiments of Section 3, the authors mislabeled some of the training examples.  In the theoretical model, it is assumed that the true class is given to the model since the model is using the parameters that correspond to the true class. In a sense, the theoretical model of Section 3 is more related to the proposed algorithm in Section 5. Also, the existence of multiple centers in Section 4 does not seem relevant, unless I am missing something. This is because the model has separate parameters for each center and the centers are given for each example the model sees. I went over the proof and it looks like this is exploited in the proof as well, to obtain a closed-form solution for the optimal network. 
* For the reasons mentioned above, it seems to me that the reason this works is some sort of regularization. I wonder if a similar effect could be obtained by increased dropout in the conditioning or some other form of regularization. Overall, I believe that the reader of the paper doesn't develop much intuition about the origins of the experimental finding.
questions: questions: Apart from the weaknesses mentioned above, I have one more important question regarding this paper. Could the authors please clarify whether the corruption in Sections 3 and Sections 5 happens once for each data sample before the training or whether a new corruption takes place whenever we encounter the same data point across different epochs? To clarify, I want to understand which one of the two is happening:

1) We take a dataset, we corrupt it once, we train with the corrupted dataset
2) We use the clean dataset and every time we see a sample we corrupt it (by adding a different noise each time to its embedding or by changing each label/text to something else each time we see the same example across epochs).

If 2) is happening, it seems to me that the proposed method is closely related to regularization. I think this needs to be clarified and the effect of doing 1) or 2) should be studied as an ablation.
limitations: limitations: The authors have adequately discussed the limitations of their work.

Comments: title: Thank you for the response
Increased my rating
comment: Thank you for your comments and additional points. We appreciate your thoughtful consideration and suggestions.
Thank you for your feedback and for acknowledging our efforts to address your questions.  
We appreciate for the increased score.
Thank you for your feedback and for revisiting our paper. We're glad our response addressed your concerns, and we appreciate the updated score.
Thank you for the author's response. After checking the author's response and other reviews, most of my concerns have been addressed, so I raise the score.
While I still disagree with the justification for the paper's insights, I appreciate the authors' effort in addressing most of my other questions. I will increase my score.
Thank you for the comments and additional points. While they do clarify some of my concerns, I think my previous score is still appropriate.
We thank the reviewer's feedback and efforts reviewing this work again. 
The changes and additional ablation studies will be included our revised work.
Thanks for your rebuttal. Most of my concerns are now addressed. Please make the appropriate changes in the camera-ready version of your work, as promised.

I increased my rating to 7.