On improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models

Authors: tariq  berrada
pietro  astolfi
melissa  hall
reyhane  askari-hemmat
yohann  benchetrit
marton  havasi
matthew  muckley
karteek  alahari
adriana  romero-soriano
jakob  verbeek
michal  drozdzal
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 20:37:55

Abstract: Large-scale training of latent diffusion models (LDMs) has enabled unprecedented quality in image generation. 
However, large-scale end-to-end training of these models is computationally costly, and hence most research  focuses either on finetuning  pretrained models or experiments at smaller scales.
In this work we aim to improve the training efficiency and performance of LDMs with the goal of scaling to larger datasets and higher resolutions.
We focus our study on two points that are critical for good performance and efficient training: 
(i) the mechanisms used for  semantic level (\eg a text prompt, or class name) and low-level (crop size, random flip, \etc) conditioning of the model, and 
(ii) pre-training strategies to transfer representations learned on smaller and lower-resolution datasets to larger ones.
The main contributions of our work are the following: 
we present systematic experimental study of these points, 
we propose a novel conditioning mechanism that disentangles semantic and low-level conditioning, 
we obtain state-of-the-art performance  on CC12M for text-to-image at 512 resolution.

Main paper: 
 introduction: 
 Diffusion models have emerged as a powerful class of generative models and demonstrated unprecedented ability at generating high-quality and realistic images. Their superior performance is evident across a spectrum of applications, encompassing image [7, 14, 39, 41] and video synthesis [35] , denoising [52] , super-resolution [49] and layout-to-image synthesis [51] . The fundamental principle underpinning diffusion models is the iterative denoising of an initial sample from a trivial prior distribution, that progressively transforms it to a sample from the target distribution. The popularity of diffusion models can be attributed to several factors. First, they offer a simple yet effective approach for generative modeling, often outperforming traditional approaches such as Generative Adversarial Networks (GANs) [3, 16, 24, 25] and Variational Autoencoders (VAEs) [29, 48] in terms of visual fidelity and sample diversity. Second, diffusion models are generally more stable and less prone to mode collapse compared to GANs, which are notoriously difficult to stabilize without careful tuning of hyperparameters and training procedures [23, 30] . Despite the success of diffusion models, training such models at scale remains computationally challenging, leading to a lack of insights on the most effective training strategies. Training recipes of large-scale models are often closed (e.g., DALL-E, Imagen, Midjourney), and only a few studies have analyzed training dynamics in detail [7, 14, 26, 27] . Moreover, evaluation often involves human studies which are easily biased and hard to replicate [17, 56] . Due to the high computational costs, the research community mostly focused on the finetuning of large text-to-image models for different downstream tasks [1, 4, 54] and efficient sampling techniques [34, 36, 45] . However, there has been less focus on ablating different mechanisms to condition on user inputs such as text prompts, and strategies to pre-train using datasets of smaller resolution and/or data size. The benefits of conditioning mechanisms are two-fold: allowing users to have better control over the content that is being generated, and unlocking training on augmented or lower quality data by for example conditioning on the original image size [39] and other metadata of the data augmentation. Improving pre-training strategies, on the other hand, can allow for big cuts in the training cost of diffusion models by significantly reducing the number of iterations necessary for convergence.Our work aims to disambiguate some of these design choices, and provide a set of guidelines that enable the scaling of the training of diffusion models in an efficient and effective manner. Beyond the main architectural choices (e.g., Unet vs. ViT), we focus on two other important aspects for generative performance and efficiency of training. First, we enhance conditioning by decoupling different conditionings based on their type: control metadata conditioning (e.g., crop size, random flip, etc.), semantic-level conditioning based on class names or text-prompts. In this manner, we disentangle the contribution of each conditioning and avoid undesired interference among them. Second, we optimize the scaling strategy to larger dataset sizes and higher resolution by studying the influence of the initialization of the model with weights from models pre-trained on smaller datasets and resolutions. Here, we propose three improvements needed to seamlessly transition across resolutions: interpolation of the positional embeddings, scaling of the noise schedule, and using a more aggressive data augmentation strategy.In our experiments we evaluate models at 256 and 512 resolution on ImageNet-1k and Conceptual Captions (CC12M), and also present results for ImageNet-22k at 256 resolution. We study the following five architectures: Unet/LDM-G4 [39] , DiT-XL2 w/ LN [38] , mDT-v2-XL/2 w/ LN [15] , PixArt-α-XL/2, and mmDiT-XL/2 (SD3) [14] . We find that among the studied base architectures, mmDiT-XL/2 (SD3) performs the best. Our improved conditioning approach further boosts the performance of the best model consistently across metrics, resolutions, and datasets. In particular, we improve the previous state-of-the-art DiT result of 3.04 FID on ImageNet-1k at 512 resolution to 2.76. For CC12M at 512 resolution, we improve FID of 11.24 to 8.64 when using our improved conditioning, while also obtaining a (small) improvement in CLIPscore from 26.01 to 26.17. See Fig. 1 for qualitative examples of our model trained on CC12M. Figure 2 : Influence of control conditions. Images generated using the same latent sample. Top: Model trained with constant weighting of the size conditioning as used in SDXL [39] , introducing undesirable correlations between image content and size condition. Bottom: Model trained using our cosine weighting of lowlevel conditioning, disentangling the size condition from the image content.In summary, our contributions are the following:• We present a systematic study of five different diffusion architectures, which we train from scratch using face-blurred ImageNet and CC12M datasets at 256 and 512 resolutions. • We introduce a conditioning mechanism that disentangles different control conditionings and semantic-level conditioning, improving generation and avoiding interference between conditions. • To transfer weights from pre-trained models we propose to interpolate positional embeddings, scale the noise schedule, and use stronger data augmentation, leading to improved performance. • We obtain state-of-the-art results at 256 and 512 resolution for class-conditional generation on ImageNet-1k and text-to-image generation and CC12M. 
conditioning and pre-training strategies for diffusion models: 
 In this section, we review and analyze the conditioning mechanisms and pre-training strategies used in prior work (see more detailed discussion of related work in App. A), and propose improved approaches based on the analysis.Background. To control the generated content, diffusion models are usually conditioned on class labels or text prompts. Adaptive layer norm is a lightweight solution to condition on class labels, used for both UNets [21, 39, 41] and DiT models [38] . Cross-attention is used to allow more fine-grained conditioning on textual prompts, where particular regions of the sampled image are affected only by part of the prompt, see e.g. [41] . More recently, another attention based conditioning was proposed in SD3 [14] within a transformer-based architecture that evolves both the visual and textual tokens across layers. It concatenates the image and text tokens across the sequence dimension, and then performs a self-attention operation on the combined sequence. Because of the difference between the two modalities, the keys and queries are normalized using RMSNorm [53] , which stabilizes training. This enables complex interactions between the two modalities in one attention block instead of using both self-attention and cross-attention blocks.Moreover, since generative models aim to learn the distribution of the training data, data quality is important when training generative models. Having low quality training samples, such as the ones that are poorly cropped or have unnatural aspect ratios, can result in low quality generations. Previous work tackles this problem by careful data curation and fine-tuning on high quality data, see e.g. [7, 9] . However, strictly filtering the training data may deprive the model from large portions of the available data [39] , and collecting high-quality data is not trivial. Rather than treating them as nuisance factors, SDXL [39] proposes an alternative solution where a UNet-based model is conditioned on parameters corresponding to image size and crop parameters during training. In this manner, the model is aware of these parameters and can account for them during training, while also offering users control over these parameters during inference. These control conditions are transformed and additively combined with the timestep embedding before feeding them to the diffusion model.Disentangled control conditions. Straightforward implementation of control conditions in DiT may cause interference between the time-step, class-level and control conditions if their corresponding embeddings are additively combined in the adaptive layer norm conditioning, e.g. causing changes in high-level content of the generated image when modifying its resolution, see Fig. 2 . To disentangle the different conditions, we propose two modifications. First, we move the class embedding to be fed through the attention layers present in the DiT blocks. Second, to ensure that the control embedding does not overpower the timestep embedding when additively combined in the adaptive layer norm, we zero out the control embedding in early denoising steps, and gradually increase its strength. Control conditions can be used for different types of data augmentations: (i) high-level augmentations (ϕ h ) that affect the image composition -e.g. flipping, cropping and aspect ratio -, and (ii) low-level augmentations (ϕ l ) that affect low-level details -e.g. image resolution and color. Intuitively, high-level augmentations should impact the image formation process early on, while low-level augmentations should enter the process only once sufficient image details are present. We achieve this by scaling the contribution of the low-level augmentations, ϕ l , to the control embedding using a cosine schedule that downweights earlier contributions:γ c (t) = 1 -cos π • (1 -t) α 2 α = 1 α = 2 α = 4 α = 8EQUATIONwhere the embedding functions E h , E l are made of sinusoidal embeddings followed by a 2-layer MLP with SiLU activation, and where γ c is the cosine schedule illustrated in Fig. 3 .Improved text conditioning. Most commonly used text encoders, like CLIP [40] , output a constant number of tokens T that are fed to the denoising model (usually T = 77). Consequently, when the prompt has less than T tokens, the remaining positions are filled by zero-padding, but remain accessible via cross-attention to the denoising network. To make better use of the conditioning vector, we propose a noisy replicate padding mechanism where the padding tokens are replaced with copies of the text tokens, thereby pushing the subsequent cross-attention layers to attend to all the tokens in its inputs. As this might lead to redundant token embeddings, we improve the diversity of the feature representation across the sequence dimension, by perturbing the embeddings with additive Gaussian noise with a small variance β txt . To ensure enough diversity in the token embeddings, we scale the additive noise by σ(ϕ txt ) √ m -1, where m is the number of token replications needed for padding, and σ(ϕ txt ) is the per-channel standard deviation in the token embeddings.Integrating classifier-free guidance. Classifier-free guidance (CFG) [20] allows for training conditional models by combining the output of the uncoditional generation with the output of the conditional generation. Formally, given a latent diffusion model trained to predict the noise ϵ, CFG reads as:ϵ λ = λ • ϵ s + (1 -λ) • ϵ ∅ ,where ϵ ∅ is the uncoditional noise prediction, ϵ s is the noise prediction conditioned on the semantic conditioning s (e.g., text prompt), and λ is the hyper-parameter, known as guidance scale, which regulates the strength of the conditioning. Importantly, during training λ is set alternatively to 0 or 1, while at inference time it is arbitrarily changed in order to steer the generation to be more or less consistent with the conditioning. In our case, we propose the control conditioning to be an auxiliary guidance term, in order to separately regulate the strength of the conditioning on the control variables c and semantic conditioning s at inference time. In particular, we define the guided noise estimate as:EQUATIONwhere β sets the strength of the control guidance, and λ sets the strength of the semantic guidance.Background. Transfer learning has been a pillar of the deep learning community, enabling generalization to different domains and the emergence of foundational models such as DINO [5, 10] and CLIP [40] . Large pre-trained text-to-image diffusion models have also been re-purposed for different tasks, including image compression [4] and spatially-guided image generation [1] . Here, we are interested in understanding to which extent pre-training on other datasets and resolutions can be leveraged to achieve a more efficient training of large text-to-image models. Indeed, training diffusion models directly to generate high resolution images is computationally demanding, therefore, it is common to either couple them with super-resolution models, see e.g. [44] , or fine-tune them with high resolution data, see e.g. [7, 14, 39] . Although most models can directly operate at a higher resolution than the one used for training, fine-tuning is important to adjust the model to the different statistics of high-resolution images. In particular, we find that the different statistics influence the positional embedding of patches, the noise schedule, and the optimal guidance scale. Therefore, we focus on improving the transferability of these components.Positional Embedding. Adapting to a higher resolution can be done in different ways. Interpolation scales the -most often learnable -embeddings according to the new resolution [2, 47] . Extrapolation simply replicates the embeddings of the original resolution to higher resolutions as illustrated in Fig. 4 , resulting in a mismatch between the positional embeddings and the image features when switching to different resolutions. Most methods that use interpolation of learnable positional embeddings, e.g. [2, 47] , adopt either bicubic or bilinear interpolation to avoid the norm reduction associated with the interpolation. In our case, we take advantage of the fact that our embeddings are sinusoidal and simply adjust the sampling grid to have constants limit under every resolution, see App. C. Scaling the noise schedule. At higher resolution, the amount of noise necessary to mask objects at the same rate changes [14, 22] . If we observe a spatial patch at low resolution under a given uncertainty, upscaling the image by a factor s creates s 2 observations of this patch of the form y (i) t = x t + σ t ϵ (i)assuming the value of the patch is constant across the patch. This increase in the number of observations reduces the uncertainty around the value of that token, resulting in a higher signal-to-noise (SNR) ratio than expected. This issue gets further accentuated when the scheduler does not reach a terminal state with pure noise during training, i.e., a zero SNR [32] , as the mismatch between the non-zero SNR seen during training and the purely Gaussian initial state of the sampling phase becomes significant. To resolve this, we scale the noise scheduler in order to recover the same uncertainty for the same timestep.Proposition 1. When going from a scale of s to a scale s ′ , we update the β scheduler according to the following ruleEQUATIONThis increases the noise amplitude during intermediate denoising steps as illustrated in Fig. A1 . The final equation obtained is similar to the one obtained in [14] with the accompanying change of variable t = σ 2 1+σ 2 .Target resolution vs. Pre-training cropping strategies. When pre-training and finetuning at different resolutions, we can either first crop and then resize the crops according to the training resolution, or directly take differently sized crops from the training images. Using a different resizing during pre-training and finetuning may introduce some distribution shift, while using crops of different sizes may be detrimental to low-resolution training as the model will learn the distribution of smaller crops rather than full images, see Fig. 5 . We experimentally investigate which strategy is more effective for low-resolution pretraining of high-resolution models.Guidance scale. We discover that the optimal guidance scale for both FID and CLIPScore varies with the resolution of images. In App. D, we present a proof revealing that under certain conditions, the optimal guidance scale adheres to a scaling law with respect to the resolution, asEQUATION)3 Experimental evaluation 
experimental setup: 
 Datasets. In our study, we train models on three datasets. To train class-conditional models, we use ImageNet-1k [11] , which has 1.3M images spanning 1,000 classes, as well as ImageNet-22k [43] , which contains 14.2M images spanning 21,841 classes. Additionally, we train text-to-image models using Conceptual 12M (CC12M) [6] , which contains 12M images with accompanying manually generated textual descriptions. We pre-process both datasets by blurring all faces. Differently from [7] , we use the original captions for the CC12M dataset.Evaluation. For image quality, we evaluate our models using the common FID [19] metric. We follow the standard evaluation protocol on ImageNet to have a fair comparison with the relevant Table 1 : Comparison between different model architectures. We compare results reported in the literature (top, reporting available numbers) with our reimplementations of existing architectures (middle), and to our best results obtained using architectural refinements and improved training. For 512 resolution, we trained models by fine-tuning models pre-trained at 256 resolution. In each column, we bold the best results among those in the first two blocks, and also those in the last row when they are equivalent or superior. '-' denotes that numbers are unavailable in the original papers, or architectures are incompatible with text-to-image generation in our experiments. '✗' indicates diverged runs. '*' is used for Esser et al. [14] pre-trained on CC12M to denote that FID is computed differently and some details about their evaluation are unclear.ImageNet-1k CC12M 256 512 256 512 literature [3, 15, 38, 41] . Specifically, we compute the FID between the full training set and 50k synthetic samples generated using 250 DDIM sampling steps. For image-text alignment, we compute the CLIP [40] score similarly to [7, 14] . We measure conditional diversity, either using class-level or text prompt conditioning, using LPIPS [55] . LPIPS is measured pairwise and averaged among ten generations obtained with the same random seed, prompt, and initial noise, but different size conditioning (we exclude sizes smaller than the target resolution); then we report the average over 10k prompts. In addition to ImageNet and CC12M evaluations, we provide FID and CLIPScore on the COCO [33] validation set, which contains approximately 40k images with associated captions. For COCO evaluation [33] , we follow the same setting as [14] for computing the CLIP score, using 25 sampling steps and a guidance scale of 5.0.EQUATIONTraining. To train our models we use the Adam [28] optimizer, with a learning rate of 10 -4 and β 1 , β 2 = 0.9, 0.999. When training at 256×256 resolution, we use a batch size of 2, 048 images, a constant learning rate of 10 × 10 -4 , train our models on two machines with eight A100 GPUs each.In preliminary experiments with the DiT architecture we found that the FID metric on ImageNet-1k at 256 resolution consistently improved with larger batches and learning rate, but that increasing the learning rate by another factor of two led to diverging runs. We report these results in supplementary.When training models at 512×512 resolution, we use the same approach but with a batch size of 384 distributed over 16 A100 GPUs. We train our ImageNet-1k models for 500k to 1M iterations and for 300k to 500k iterations for CC12M.Model architectures. We train different diffusion architectures under the same setting to provide a fair comparison between model architectures. Specifically, we re-implement a UNet-based architecture following Stable Diffusion XL (SDXL) [39] 1 and several transformer-based architectures: vanilla DiT [38] , masked DiT (mDiT-v2) [15] , PixArt DiT (PixArt-α) [7] , and multimodal DiT (mmDiT) as in Stable Diffusion 3 [14] . For vanilla DiT, which only supports class-conditional generation, we explore two variants one incorporating the class conditioning within LayerNorm and another one within the attention layer. Also, for text-conditioned models, we use the text encoder and tokenizer of CLIP (ViT-L/14) [40] having a maximum sequence length of T = 77. The final models share similar number parameters, e.g. for DiTs we inspect the XL/2 variant [38] , for UNet (SDXL) we adopt similar size to the original LDM [41] . Similar to [14] , we found the training of DiT with cross-attention to be unstable and had to resort to using RMSNorm [53] to normalize the key and query in the attention layers. We detail the models sizes and computational footprints in Tab. A1.In Tab. 1, we report results for models with different architectures trained at both 256 and 512 resolutions for ImageNet and CC12M, and compare our results (2nd block of rows) with those reported in the literature, where available (1st block of rows). Where direct comparison is possible, we notice that our re-implementation outperforms the one of existing references. Overall, we found the mmDiT [14] architecture to perform best or second best in all settings compared to other alternatives. For this reason, we apply our conditioning improvements on top of this architecture (last row of the table), boosting the results as measured with FID and CLIPScore in all settings. Below, we analyse the improvements due to our conditioning mechanisms and pre-training strategies.Scheduling rate of control conditioning. In Tab. 2a we consider the effect of controlling the conditioning on low-level augmentations via a cosine schedule for different decay rates α. We compare to baselines (first two rows) with constant weighting (as in SDXL [39] ) and without control conditioning. We find that our cosine weighting schedule significantly reduces the dependence between size control and image semantics as it drastically improves the instance specific LPIPS (0.33 vs. 0.04) in comparison to uniform weighting. In terms of FID, we observe a small gap with the baseline (3.04 vs. 3.08), which increases (3.80 vs. 5.04) when computing FID by randomly sampling the size conditioning in the range [512,1024], see Tab. 2b. Finally, the improved disentangling between semantics and low-level conditioning is clearly visible in the qualitative samples in Fig. 2 .Crop and random-flip control conditioning. A potential issue of horizontal flip data augmentations is that it can create misalignment between the text prompt and corresponding image. For example the prompt "A teddy bear holding a baseball bat in their right arm" will no longer be accurate when an image is flipped -showing a teddy bear holding the bat in their left arm. Similarly, cropping images can remove details mentioned in the corresponding caption. In Tab. 2c we evaluate models trained on CC12M@256 with and without horizontal flip conditioning, and find that adding this conditioning leads to slight improvements in both FID and CLIP as compared to using only crop conditioing. We depict qualitative comparison in Fig. 6 , where we observe that flip conditioning improves prompt-layout consistency.Inference-time control conditioning of image size. High-level augmentations (ϕ h ) may affect the image semantics. As a result they influence the learned distribution and modify the generation diversity. For example, aspect ratio conditioning can harm the quality of generated images, when images of a particular class or text prompt are unlikely to appear with a given aspect ratio. In Tab. 2b we compare of different image size conditionings for inference. We find that conditioning on the same size distribution as encountered during the training of the model yields a significant boost in FID as compared to generating all images with constant size conditioning or using uniformly randomly sampled sizes. Note that in all cases images are generated at 256 resolution. Control conditioning and guidance. To understand how control condition impacts the generation process, we investigate the influence of control guidance β (introduced in Sec. 2.1 ) on FID and report the results in Fig. 7 . We find that a higher control guidance scale results in improved FID scores. However, note that this improvement comes at the cost of compute due to the additional control term ϵ c,s .Replication text padding. We compare our noisy replication padding to the baseline zero-padding in Tab. 3. We observe that using a replication padding improves both FID and CLIP score, and that adding scaled perturbations further improves the results -0.35 point improvement in CLIP score and 0.4 point improvement in FID.Dataset shift. We evaluate the effect of pre-training on ImageNet-1k (at 256 resolution) when training the models on CC12M or ImageNet-22k (at 512 resolution) by the time needed to achieve the same performance as a model trained from scratch. In Tab. 4a, when comparing models trained from scratch to ImageNet-1k pre-trained models (600k iterations) we observe two benefits: improved training convergence and performance boosts. For CC12M, we find that after only 100k iterations, both FID and CLIP scores improve over the baseline model trained with more than six times the amount of training iterations. For ImageNet-22k, which is closer in distribution to ImageNet-1k than CC12M, the gains are even more significant, the finetuned model achieves an FID lower by 0.5 point after only 80k training iterations. In Tab. 4b, we study the relative importance of pre-training vs finetuning when the datasets have dissimilar distributions but similar sample sizes. We fix a training budget in terms of number of training iterations N , we first train our model on ImageNet-22k for K iterations before continuing the training on CC12M for the remaining N -K iterations. We see that the model pretrained for 200k iterations and finetuned for 150k performs better than the one spending the bulk of the training during pretraining phase. This validates the importance of domain specific training and demonstrates that the bulk of the gains from the pretrained checkpoint come from the representation learned during earlier stages.Resolution change. We compare the performance boost obtained from training from scratch at 512 resolution vs. resuming from a 256-resolution trained model. According to our results in Tab. 4c, pretraining on low resolution significantly boosts the performance at higher resolutions, both for UNet and mmDiT, we find that higher resolution finetuning for short periods of time outperforms high resolution training from scratch by a large margin (≈ 25%). These performance gains might in part be due to the increased batch size when pre-training the 256 resolution model, which allows the model to "see" more images as compared to training from-scratch at 512 resolution.Positional Embedding. In Fig. 8a , we compare the influence of the adjustment mechanism for the positional embedding. We find that our grid resampling approach outperforms the default extrapolation approach, resulting in 0.2 point difference in FID after 130k training iterations.Scaling the noise schedule. We conducted an evaluation to ascertain the influence of the noise schedule by refining our mmDiT model post its low resolution training and report the results in Fig. 8b . Remarkably, the application of the rectified schedule, for 40k iterations, resulted in an improvement of 0.7 FID points demonstrating its efficacy at higher resolutions.Pre-training cropping strategies. During pretraining, the model sees objects that are smaller than what it sees during fine tuning, see Fig. 5 . We aim to reduce this discrepancy by adopting more agressive cropping during the pretraining phase. We experiment with three cropping ratios for training: 0.9 -1 (global), 0.4 -0.6 (local), 0.4 -1 (mix). We report the results in Fig. 8c . On ImageNet1K@256, the pretraining FID scores are 2.36, 245.55 and 2.21 for the local, global and mixed strategies respectively. During training at 512 resolution, we observe that the global and mix cropping strategies both outperform the local strategy. However, as reported in Fig. 8c , the local strategy provides benefits at higher resolutions. Overall, training with the global strategy performs the best at 256 resolution but lags behind for higher resolution adaptation. While local cropping underperforms at lower resolutions, because it does not see any images in their totality, it outperforms the other methods at higher resolutions -an improvement of almost 0.2 FID points is consistent after the first 50k training steps at higher resolution.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: Proofs are provided Appendix D and contain cross-references to the relevant parts in the main paper.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
conclusion: 
 In this paper, we explored various approaches to enhance the conditional training of diffusion models. Our empirical findings revealed significant improvements in the quality and control over generated images when incorporating different coditioning mechanisms. Moreover, we conducted a comprehensive study on the transferability of these models across diverse datasets and resolutions. Our results demonstrated that leveraging pretrained representations is a powerful tool to improve the model performance while also cutting down the training costs. Furthermore, we provided valuable insights into efficiently scaling up the training process for these models without compromising performance. By adapting the schedulers and positional embeddings when scaling up the resolution, we achieved substantial reductions in training time while boosting the quality of the generated images. Additional experiments unveil the expected gains from different transfer strategies, making it easier for researchers to explore new ideas and applications in this domain. In Appendix B we discuss societal impact and limitations of our work. A Related work Diffusion Models. Diffusion models have gained significant attention in recent years due to their ability to model complex stochastic processes and generate high-quality samples. These models have been successfully applied to a wide range of applications, including image generation [7, 21, 41] , video generation [35] , music generation [31] , and text generation [50] . One of the earliest diffusion models was proposed in [21] , which introduced denoising diffusion probabilistic models (DDPMs) for image generation. This work demonstrated that DDPMs can generate high-quality images that competitive with state-of-the-art generative models such as GANs [16] . Following this work, several variants of DDPMs were proposed, including score-based diffusion models [46] , conditional diffusion models [12] , and implicit diffusion models [45] . Overall, diffusion models have shown promising results in various applications due to their ability to model complex stochastic processes and generate high-quality samples [7, 14, 39, 41] . Despite their effectiveness, diffusion models also have some limitations, including the need for a large amount of training data and the required computational resources. Some works [26, 27] have studied and analysed the training dynamics of diffusion models, but most of this work considers the pixel-based models and small-scale settings with limited image resolution and dataset size. In our work we focus on the more scalable class of latent diffusion models [41] , and consider image resolutions up to 512 pixels, and 14M training images.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We include details required for result reproducibility in Section 3.1. These include details about datasets used for training, evaluation metrics, training parameters, and model architectures. 
open access to data and code: 
 Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: While we are not be able to include code, we do provide specific and thorough detailing of our training and evaluation methods in Section 3.1 to enable experimental reproduction.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Details corresponding to training and testing are included in Section 3.1.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Due to the computational cost of training and performing inference with generative models, we do not include error bars.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: Information about compute resources needed for experiment reproduction is included in Section 3.1.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: This research conforms to the Code of Ethics, including mitigation of potential harms caused by the research process, considerations of social impact, and taking steps for impact mitigation.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss societal impacts and steps taken to mitigate potential harms in Appendix B.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not include crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Model architectures. Early work on diffusion models adopted the widely popular UNet arcchitecture [39, 41] . The UNet is an encoder-decoder architecture where the encoder is made of residual blocks that produce progressively smaller feature maps, and the decoder progressively upsamples the feature maps and refines them using skip connections with the encoder [42] . For diffusion, UNets are also equipped with cross attention blocks for cross-modality conditioning and adaptive layer normalization that conditions the outputs of the model on the timestep [41] . More recently, vision transformer architectures [13] were shown to scale more favourably than UNets for diffusion models with the DiT architecture [38] . Numerous improvements have been proposed to the DiT in order to have more efficient and stable training, see e.g. [7, 14, 15] . In order to reduce the computational complexity of the model and train at larger scales, windowed attention has been proposed [7] . Latent masking during training has been proposed to encourage better semantic understanding of inputs in [15] . Others improved the conditioning mechanism by evolving the text tokens through the layers of the transformer and replacing the usual cross-attention used for text conditioning with a variant that concatenates the tokens from both the image and text modalities [14] .Large scale diffusion training. Latent diffusion models [41] unlocked training diffusion models at higher resolutions and from more data by learning the diffusion model in the reduced latent space of a (pre-trained) image autoencoder rather than directly in the image pixel space. Follow-up work has proposed improved scaling of the architecture and data [39] . More recently, attention-based architectures [7, 14, 15] by scaling the model size further and achieving state-of-the-art performance on datasets such as ImageNet-1k. Efficiency gains were also obtained by [7] by transferring ImageNet pre-trained models to larger datasets.Our research investigates the training of generative image models, which are widely employed to generate content for creative or education and accessibility purposes. However, together with these beneficial applications, these models are usually associated with privacy concerns (e.g., deepfake generation) and misinformation spread. In our paper, we deepen the understanding of the training dynamics of these modes, providing the community with additional knowledge that can be leveraged for safety mitigation. Moreover, we promote a safe and transparent/reproducible research by employing only publicly available data, which we further mitigate by blurring human faces.Our work is mostly focused on training dynamics, and to facilitate reproducibility, we used publicly available datasets and benchmarks, without applying any data filtering. We chose datasets relying on the filtering/mitigation done by their respective original authors. In general, before releasing models like the ones described in our paper to the public, we recommend conducting proper evaluation of models trained using our method for bias, fairness and discrimination risks. For example, geographical disparities due to stereotypical generations could be revealed with methods described by Hall et al. [18] , and social biases regarding gender and ethnicity could be captured with methods from Luccioni et al. [37] and Cho et al. [8] .While our study provides valuable insights into control conditioning and the effectiveness of representation transfer in diffusion models, there are several limitations that should be acknowledged. (i) There are cases where these improvements can be less pronounced. For example, noisy replicates for the text embeddings can become less pertinent if the model is trained exclusively with long prompts. (ii) While low resolution pretraining with local crops on ImageNet resulted in better FID at 512 resolution (see Table 5c ), it might not be necessary if pretraining on much larger datasets (e.g. >100M samples, which we did not experiment in this work). Similarly, flip conditioning is only pertinent if the training dataset contains position sensitive information (left vs. right in captions, or rendered text in images), otherwise this condition will not provide any useful signal. (iii) We did not investigate the impact of data quality on training dynamics, which could have implications for the generalizability of our findings to datasets with varying levels of quality and diversity. (iv) As our analysis primarily focused on control conditioning, other forms of conditioning such as timestep and prompt conditioning were not explored in as much depth. Further research is needed to determine the extent to which these conditionings interact with control conditioning and how that impacts the quality of the models. (v) Our work did not include studies on other parts that are involved in the training and sampling of diffusion models, such as the different sampling mechanisms and training paradigms. This could potentially yield additional gains in performance and uncover new insights about the state-space of diffusion models.Noise schedule scaling. In Fig. A1 we depict the rescaling of the noise for higher resolutions, following Eq. ( 3). Positional embedding rescaling. As illustrated in App. C, rescaling the positional embedding can be integrated in two simple lines of code by changing the grid sampling to be based on reducing the stepsize in the grid instead of extending its limits.# grid_h = arange ( grid_size ) # grid_w = arange ( grid_size ) base_size = grid_size // scale grid_h = linspace (0 , base_size , grid_size ) grid_w = linspace (0 , base_size , grid_size )Pseudocode 1: Rectified grid sampling for positional embeddings.Computational costs. In Tab. A1 we compare the model size and computational costs of the different architectures studied in the main paper. All model architectures are based on the original implemetations, but transposed to our codebase. Mainly, we use bfloat16 mixed precision and memory-efficient attention2 from PyTorch.Experimental details. To ensure efficient training of our models, we benchmark the most widely used hyperparameters and report the results of these experiments, which consist of the choice of the optimizer and its hyperparameters, the learning rate and the batch size. We then transpose the optimal settings to our other experiments. For FID evaluation, we use a guidance scale of 1.5 for 256 resolution and 2.0 for resolution 512. For evaluation on ImageNet-22k, we compute the FID score between 50k generated images and a subset of 200k images from the training set.Training paradigm. We use the EDM [26] abstraction to train our models for epsilon prediction following DDPM paradigm. Differently from [15, 38] , we do not use learned sigmas but follow a standard schedule. Specifically, we use a quadratic beta schedule with β start = 0.00085 and β end = 0.012. In DDPM [21] , a noising step is formulated as follows, with x 0 being the data sample and t ∈ [[0, T ]] a timestep:EQUATIONwhile in EDM, the cumulative alpha products are converted to corresponding signal-to-noise ratio (SNR) proportions, the noising process is then reformulated as:EQUATIONand the loss is weighted by the inverse SNR For almost all learning rates, the optimal batch size is the highest possible. The best performance is obtained when using the highest learning rate that does not diverge with biggest batch size possible.the factor γ itself. In the following we experiment with training a class-conditional DiT model with different batch sizes and learning rates and report results after the same number of iterations.From Tab. A2 we observe an improved performance by increasing the batch size and the learning rate in every learning rate setting. If the learning rate is too high the training diverges. Influence of momentum. We conduct a grid search over the momentum parameters of Adam optimizer, similar to previous sections, we train a UNet model fo 70k steps and compute FID with respect to the validation set of ImageNet1k using 50 DDIM steps. From Fig. A2 , we can see that the default pytorch values (β 1 = 0.9, β 2 = 0.999) are sub-optimal, resulting in an FID of 26.43 while the best performance is obtained when setting (β 1 = 0.9, β 2 = 0.95) improves FID by 4.78 points in our experimental setting.Setting the optimal guidance scale.Proposition 2. The optimal guidance scale λ scales with the upsampling factor s according to the law λ ′ (s) = 1 + s • (λ -1).This is verified in Fig. 7 where the λ ′ (s = 2) = 1 + 2 • (1.5 -1.0) = 2.0 which is the optimal guidance scale at 512 resolution according to the figure. Text padding mechanism. In order to train at a large scale, most commonly used text encoders output a constant number of tokens T that are fed to the denoising model (usually T = 77). Consequently, when the prompt has less than T words, the remaining tokens are padding tokens that do not contain useful information, but can still contribute in the cross-attention, see Figure A3 . This raises the question of whether better use can be made of padding text tokens to improve training performance and efficiency.One common mitigation involves using recaptioning methods that provide longer captions. However, this creates an inconsistency between training and sampling as users are more likely to provide shorter prompts. Thus, to make better use of the conditioning vector, we explore alternative padding mechanisms for the text encoder. We explore a 'replicate' padding mechanism where the padding tokens are replaced with copies of the text tokens, thereby pushing the subsequent cross-attention layers to attend to all With the zero padding mechanism, a significant part of the attention can be used on "dead" padding tokens, potentially de-focusing from the relevant information. Using a replicate padding instead results in redundant information. Noisy replicate padding increases the diversity in the text token representations and therefore acts as a regularizer fostering the model to be robust to local variations in the latent space of the conditioning, e.g., akin to data augmentation.the tokens in its inputs. To improve the diversity of the feature representation across the sequence dimension, we perturb the embeddings with additive Gaussian noise with a small variance β txt . For shorter prompts with a high number of repeats m, we scale the additive noise by √ m -1 to account for the reduction in posterior uncertainty induced by these repetitions:EQUATIONwhere ϕ ch is the standard deviation of the feature text embeddings over the feature dimension. See Figure A4 for an illustration comparing zero-padding, replication padding, and our noisy replication padding.Derivation for Eq. ( 2)Proof. The formula can be obtained by iteratively applying the guidance across the conditions.EQUATIONEQUATIONDerivation for Eq. ( 4)Proof. Assuming that the unconditional prediction ϵ ∅ is distributed around the conditional distribution according to a normal law ϵ ∅ |ϵ c ∼ N (ϵ c , δ 2 I).EQUATIONEQUATIONEQUATIONAfter upsampling by a scale factor of s, the same low resolution patch has s 2 observations, hence the variance is decreased byEQUATIONHence by equalizing the discrepancy between the conditional and unconditional predictions at low and high resolutions we obtain: Derivation for Eq. ( 3)EQUATIONEQUATIONProof. At timestep t, the noisy observation is obtained as :EQUATIONHence x 0 can be estimated using the following formula:EQUATIONThe statistics of the estimate of x 0 are as follows.EQUATION)Consequently, if we already have x t , we have an estimate of x 0 with an error bound given by:EQUATION)At higher resolutions, we have s 2 corresponding observations for the same patch such that the error with respect to the estimate becomes:EQUATIONHence, if we want to keep the same uncertainty with respect to the low resolution patches, the following equality needs to be verified:EQUATIONQualitative results. We provide additional qualitative examples on ImageNet-1k in Fig. A5 . Table A3 : Effects of our changes. We summarize the improvements obtained by each change proposed in the paper. left: Changes relevant to conditioning mechanisms -right: Changes relevant to representation transfer across image resolutions.SD-XL [39] SD3 [14] Ours Guidelines:Sampling Schedule σi<N 1-ᾱi ᾱi i N -1 1-ᾱi ᾱi Forward Process xt x0 + σtϵ / 1 + σ 2 t tx0 + (1 -t)ϵ x0 + σtϵ / 1 + σ 2 t Target ϵ vΘ = ϵ -x0 ϵ Loss scaling σ -2 t 1.0 σ -2 t Timestep sampling U([0, T ]) π(t, m, s) = 1 s √ wπ 1 t(1-t) e (logit(t)-m) 2 /2s 2 U([0, T ]) Guidance mechanism λϵc + (1 -λ)ϵ ∅ λϵc + (1 -λ)ϵ ∅ λ β • ϵc,s + (1 -β) • ϵ ∅,s + (1 -λ) • ϵ ∅,∅′ = ᾱt t ′ = st s ′ +(s-s ′ )t ᾱt ′ = s 2 • ᾱt s ′2 + ᾱt•(s 2 -s ′2 )• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.We only implement the base network, without the extra refiner as in[39]https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_ attention.html 


Decision: This paper proposes an effective pre-training method for diffusion models.

Intial scores were one borderline reject, one borderline acceptance, and one weak acceptance.
Reviewers pointed out some concerns related to experiments and their analysis while repsecting its task importance and effective methods.

During the rebuttal period, the authors successuffly addressed the issues that the reviewers raised, and thus all reviewers have consensus to accept this paper with 6, 6, and 5.

In addtion, ethics reviewer clarified no ethical issue in this paper.

AC also agree with the reviewer's opinions, so recommends accepting this paper.

Reviews: review: paper_summary: paper_summary: This paper studies how to effectively condition image size and crop information, as done by Stable Diffusion XL, and how to implement an effective coarse-to-fine training strategy. For control conditioning, it is designed to be less entangled than traditional methods, resulting in better performance for the same backbone model. The analysis for model transfer in coarse-to-fine training includes noise scheduling and positioning embedding. Additionally, the paper compares backbone models by re-implementing and comparing UNet, DiT, and mDiT. For analysis, ImageNet1K/22K data was used for class2image, and CC12M data for text2image.
strength_weakness: strength_weakness: - The explanation and analysis of the chosen designs are lacking for me. For instance, why was noisy replicate padding selected for text conditioning? It seems to make little difference in terms of FID. Also, why was the cosine function used to prevent entanglement with timestep embedding in control conditioning? Could we just use a simply hardcoded timestep for thresholding or just increasing linearly?
- The number of hyperparameters to tune for optimal performance has increased. Both the control guidance scale and gamma_c for the cosine function need to be adjusted.
- Some experimental results are not very surprising. For example, in Table 2 (b), it is quite obvious that having similar resolution sampling during training and inference yields better results. And the difference of the number in Table 2 (c) is too small.
questions: questions: - I am curious about the additional explanation regarding the model design mentioned in the weaknesses.
- Was there any performance measurement conducted for moving the class embedding position in control conditioning?
- I'm unsure if comparing performance at specific steps is suitable. It would be better to show a graph or observe the best score after running for a sufficiently long time (e.g., the results in Table 5).
- In Table 2 (a) for Control-conditioning, what was the LPIPS compared against?
limitations: limitations: The authors well addressed the limitations on their appendix.
paper_summary: paper_summary: This paper's contributions can be divided into three main points:

1. Proposing better methods for metadata/semantic level conditioning:
   - Instead of using AdaLN, class information is fed into the model through attention.
   - Adjusting the strength of meta conditions related to low-level augmentation according to diffusion timesteps.
   - Suggesting a method of copying text tokens and applying Gaussian perturbation instead of zero padding.
   - Proposing additional guidance for the control variable \(c\) (Eq. 2).

2. Proposing a method of transfer learning from low resolution to high resolution:
   - Using interpolation for positional embedding.
   - Adjusting the noise schedule according to SNR changes with resolution.
   - Experimentally showing that resizing is better than cropping for creating low-resolution images.
   - Proposing a strategy for adjusting guidance strength according to resolution.

3. Reproducing experiments:
   - Demonstrating experimentally that mmDiT has the best performance among various models proposed so far.

They demonstrated the validity of each method through extensive experiments on the Imagenet and CC12M 256x256 (512x512 for finetuning) datasets.
strength_weakness: strength_weakness: 1. Unorganized Experiment Result Presentation
   - One of the key results of this study is Imagenet-1K FID 1.59, CC12M FID 6.79, and 8.64.
   - However, it is difficult to understand how each element claimed by this study contributes to these results.
   - The best presentation would be to examine how removing each configuration affects the Imagenet-1K FID 1.59.
   - Practically, it seems challenging to run 1M iterations for all configurations. It would have been better to show how changing each configuration affects the best config based on 120k iterations.
   - For example, it would be useful to have something like Table 1 from the EDM 2 paper [1].
   - This is not just for easier reader comprehension but is also important for understanding how much the proposed disentangled control conditions contributed to the Imagenet-1K FID improvement from 1.71 to 1.59.
   - If these concerns are well resolved, I am considering raising the score.


2. Method Presentation
   - This study has a very complex design space.
   - However, it is currently difficult to know exactly what the current settings are and how they differ from past settings. Could you include something like Table 1 from the EDM 1 paper [2]?

[1]: Analyzing and Improving the Training Dynamics of Diffusion Models, https://arxiv.org/abs/2312.02696
[2]: Elucidating the Design Space of Diffusion-Based Generative Models, https://arxiv.org/abs/2206.00364
questions: questions: Minor suggestion:
- If adding the flip condition is part of the method, it would be better to introduce it in the method section rather than having it first appear in the experiment section at L264.
limitations: limitations: The author has appropriately described the limitations of this work.
paper_summary: paper_summary: The paper explores the training efficiencies and model performance of latent diffusion models (LDMs) by reimplementing and analyzing various previously published models. The study introduces a novel conditioning mechanism that separates semantic and control metadata inputs, significantly improving class-conditional and text-to-image generation performances on benchmark datasets.
strength_weakness: strength_weakness: (1) The paper is primarily evaluated on large-scale datasets like ImageNet and CC12M, but its effectiveness in low-data or small-scale datasets is not demonstrated.
(2) The paper briefly mentions instability issues with certain models (e.g., DiT with cross-attention), but does not provide detailed insights or solutions to these problems. 
(3) Although the paper presents some ablation studies, they are not comprehensive. More detailed ablation studies isolating the impact of each proposed innovation are recommended.
questions: questions: (1) What are the specific scenarios or cases where your proposed methods did not perform well? 
(2) Have you tested your proposed methods on generative tasks beyond image generation, such as text generation or audio synthesis?
limitations: limitations: (1) The paper mentions instability issues with certain models but does not provide a detailed analysis.
(2) Insufficient ablation experiments.

Comments: comment: We thank the reviewer for their insightful comments and concerns related to bias, fairness and discrimination (BFD) and the generalizability of our results.

As noted by the reviewer, our work is mostly focused on training dynamics, and to ensure reproducibility, we used publicly available datasets and benchmarks, without applying any data filtering. We chose datasets relying on the filtering/mitigation done by their respective original authors. 

In terms of recommendations, we recommend conducting proper evaluation of models trained using our method for BFD risks before releasing them to the public. For example, geographical disparities due to stereotypical generations could be revealed with \[1\], and social biases regarding gender and ethnicity could be captured with \[2\] and \[3\].

\[1\] *Hall, Melissa, et al. "DIG In: Evaluating Disparities in Image Generations with Indicators for Geographic Diversity." Transactions on Machine Learning Research.*  
\[2\] *Luccioni, Sasha, et al. "Stable bias: Evaluating societal representations in diffusion models." Advances in Neural Information Processing Systems 36 (2024).*  
\[3\] *Cho, Jaemin, Abhay Zala, and Mohit Bansal. "Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023\.*
Thanks for your detailed rebuttal and for addressing the concerns raised. 

- Lack of Small-Scale Experiments:

I understand the challenges associated with scaling and convergence when working with state-of-the-art diffusion models. Your justification for focusing on large-scale experiments is noted, and the references to previous studies (Nichol et al., 2021; Karras et al., 2022, 2023) provide additional support for your approach. While small-scale experiments can sometimes provide initial insights, I acknowledge that the scale you chose is more aligned with your objectives and the nature of your work.

- Instability Issues with Certain Models:

Thanks for pointing out the details regarding instabilities in your paper, particularly those related to the SD3 paper solutions. The explanation about the use of RMSnorm to stabilize keys and queries in cross-attention is helpful. 

- Ablations Not Comprehensive:

I appreciate the inclusion of Table R1, which provides a clearer breakdown of the contributions of each improvement. It helps in understanding the specific impact of individual components on the overall performance. 

- Limitations of Your Work:

The expansion of the limitations section is appreciated. Highlighting specific cases where improvements may be less pronounced adds valuable nuance to your findings. The discussion around noisy replicates and conditions such as flip conditioning provides a clearer understanding of the contexts in which your method excels and those where it might face challenges.

- Extension to Audio or Text Generation:

I agree that extending your work to other data modalities, such as audio or text generation, is an intriguing area for future exploration. Acknowledging this as a potential future direction is appropriate, given the scope of your current research.

Overall, your responses have clarified many of the points raised and enhanced the understanding of your contributions. The additional details and context address several of my initial concerns.
We thank the reviewer for their valuable feedback and for raising their score.  
While we acknowledge that many of the contributions we propose are experimentally based, we would highlight that we made an effort to provide theoretical backing for our contributions when possible, [see answer to reviewer rR4N](https://openreview.net/forum?id=B3rZZRALhk\&noteId=y8uNeCscKh).

Regarding the significance of the gains with each modification, there are two points that we would like to raise:

* Differently from works like EDM (Karras et al 2022 and 2023), we are not optimizing for one single metric (FID) but trying to improve different facets of our model (FID, CLIP, LPIPS).  
* We compare our model with very strong baselines, e.g., SD3, which achieve very high scores to start with. Hence, the absolute gap with these methods is small, but the relative improvement is considerable, see Table 1 and R1 below. Notably, our improvements do not carry computational overhead. 

| Tab. 1 | ImageNet-1K |  | CC12M |  |  |  |  |
| :---- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|  | 256 | 512 | 256 |  | 512 |  |  |
|  | FIDtrain↓ | FIDtrain↓ | FIDval↓ | CLIPCOCO↑ | FIDval↓ | FIDCOCO↓ | CLIPCOCO↑ |
| DiT-XL/2 w/ LN | 1.95 (0%) | 2.85 (0%) | — | — | — | — | — |
| DiT-XL/2 w/ Att | 14% | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| mDT-v2-XL/2 w/ LN | \-22% | \-24% | — | — | — | — | — |
| PixArt-α-XL/2 | \-5% | \-7% | ✗ | ✗ | ✗ | ✗ | ✗ |
| mmDiT-XL/2 (SD3) | 14% | \-6% | 7.54 (0%) | 24.78 (0%) | 11.24 (0%) | 6.78 (0%) | 26.01 (0%) |
| **mmDiT-XL/2 (ours)** | **23%** | **3%** | **11%** | **7%** | **79%** | **1%** | **1%** |

| Tab. R1 (a)  | FIDIN1K↓ | FIDCC12M↓ | CLIPCOCO↑ |
| :---- | :---: | :---: | :---: |
| Baseline | 1.95 | 7.54 | 25.88 |
| Ours | 1.59 | 6.79 | 26.6 |
| **Rel. improvement** | **23%** | **11%** | **3%** |

| Tab. R1 (b) | FIDCC12M↓ | CLIPCOCO↑ |
| :---- | :---: | :---: |
| Baseline | 11.24 | 24.23 |
| Ours | 6.27 | 25.91 |
| **Rel. improvement** | **79%** | **6%** |
Thank you for addressing my comments and questions. I believe the authors have adequately explained most of my concerns, including the usefulness of repeated noise and the cosine scheduler. Therefore, I have decided to raise my score to 5. The reason I did not increase the score further is similar to Reviewer rR4N’s—while the method is proposed based on the experimental results without much theoretical background, it does not show significant gains with each modification. However, I believe the experimental results could be valuable to the community working with diffusion models, so I think it is worthy of acceptance.
Thank you for the comments. I think my perspective was somewhat narrow. I apologize for that part and thank you for giving me the opportunity to correct it. It seems like a wrong choice to reduce my confidence in recommending this paper due to its theoretical background. I have increased my confidence to 3 points.

Among your responses, this part especially helped me to refresh my view and focus more on solid improvement in a neglected area:

> We would also note that empirical validation, even when not grounded by theory, has led to generalization several times over the years of DL algorithms development

I fully agree with this statement, and it's actually something I know somewhat trivially in deep learning research, but I had overlooked it. I think this paper, in particular, has done a good job with empirical validation. I definitely agree that this research has achieved solid improvement through extensive experiments in improving conditional mechanisms that have been neglected until now.

Once again, I express my gratitude for the authors' hard work.
We are very glad the reviewer was satisfied by our response, and they increased their score. 

We appreciate the reviewer’s opinion. We see our paper as an in-depth study that: (i) analyzes different model design choices, (ii) provides additional understanding, and (iii) proposes improvements on pre-existing approaches. However, we politely disagree with the premise of the comment that our observations might not generalize due to the lack of theoretical background. All our contributions are validated through an extensive set of experiments at large scale showing good generalization. In particular, we provide results on three datasets of different scales and distributions and for different resolutions. We may remark that previous foundational diffusion models, e.g., LDM and DiT, were trained and tested primarily on Imagenet at 256 and 512 resolutions. We would also note that empirical validation, even when not grounded by theory, has led to generalization several times over the years of DL algorithms development, e.g., MAE (He et al 2022), DINO (Caron et al 2021), spatial pyramid pooling (He et al 2015), ViT (Touvron et al 2021, Dehghani et al 2023\) and ResNet recipes (He, et al 2019, Kolesnikov et al 2020, Bello, et al 2021). Finally, we would like to highlight that some of our proposed improvements are grounded by theory, e.g., decoupled control conditioning leverages the classifier free guidance theory, while noise schedule scaling is backed by Eq.3 (with proof of derivation in Appendix F).

We hope this answer addresses the reviewer's point for lowering confidence. If not, we are happy to discuss any other point that could assist to that end.

\[Touvron, Hugo, et al. "Training data-efficient image transformers & distillation through attention." International conference on machine learning 2021; He, Kaiming, et al. "Masked autoencoders are scalable vision learners." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition 2022; He, Kaiming, et al. "Spatial pyramid pooling in deep convolutional networks for visual recognition." IEEE transactions on pattern analysis and machine intelligence. 2015; Kolesnikov, Alexander, et al. "Big transfer (bit): General visual representation learning." Computer Vision–ECCV 2020; Dehghani, Mostafa, et al. "Scaling vision transformers to 22 billion parameters." International Conference on Machine Learning. 2023; Caron, Mathilde, et al. "Emerging properties in self-supervised vision transformers." Proceedings of the IEEE/CVF international conference on computer vision. 2021\. Bello, Irwan, et al. "Revisiting resnets: Improved training and scaling strategies." Advances in Neural Information Processing Systems 34\. 2021; He, Tong, et al. "Bag of tricks for image classification with convolutional neural networks." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019\. \]
The authors have diligently addressed my original concerns, and I sincerely appreciate their efforts to improve the paper. In light of these improvements, I have increased my score from 5 to 6.

However, I must apologize for an oversight in my first review. Upon further reflection, I've noticed an aspect that I failed to address earlier, which has led me to decrease my confidence from 3 to 2. I feel obligated to bring this to your attention, even at this late stage. The proposed method, while effective, bears similarities to a "bag-of-tricks" approach. While such approaches can yield practical results, they often lack a strong theoretical foundation. This observation doesn't negate the paper's value, but it does raise questions about its generalizability and depth of contribution to the field.

In conclusion, I believe this paper is one that can be seen in NeurIPS, but it's difficult to claim it as strong due to the lack of theoretical background. Again, I apologize for not raising this point earlier.