Title: On improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Generative Models

Generative Modeling

Diffusion

Latent diffusion

Computer vision

text-to-image diffusion
Abstract: Large-scale training of latent diffusion models (LDMs) has enabled unprecedented quality in image generation. 
However, large-scale end-to-end training of these models is computationally costly, and hence most research  focuses either on finetuning  pretrained models or experiments at smaller scales.
In this work we aim to improve the training efficiency and performance of LDMs with the goal of scaling to larger datasets and higher resolutions.
We focus our study on two points that are critical for good performance and efficient training: 
(i) the mechanisms used for  semantic level (\eg a text prompt, or class name) and low-level (crop size, random flip, \etc) conditioning of the model, and 
(ii) pre-training strategies to transfer representations learned on smaller and lower-resolution datasets to larger ones.
The main contributions of our work are the following: 
we present systematic experimental study of these points, 
we propose a novel conditioning mechanism that disentangles semantic and low-level conditioning, 
we obtain state-of-the-art performance  on CC12M for text-to-image at 512 resolution.

On Improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models

Abstract

Large-scale training of latent diffusion models (LDMs) has enabled unprecedented quality in image generation. However, the key components of the best performing LDM training recipes are oftentimes not available to the research community, preventing apple-to-apple comparisons and hindering the validation of progress in the field. In this work, we perform an in-depth study of LDM training recipes focusing on the performance of models and their training efficiency. To ensure apple-to-apple comparisons, we re-implement five previously published models with their corresponding recipes. Through our study, we explore the effects of (i) the mechanisms used to condition the generative model on semantic information ( e.g ., text prompt) and control metadata ( e.g ., crop size, random flip flag, etc .) on the model performance, and (ii) the transfer of the representations learned on smaller and lower-resolution datasets to larger ones on the training efficiency and model performance. We then propose a novel conditioning mechanism that disentangles semantic and control metadata conditionings and sets a new state-of-the-art in classconditional generation on the ImageNet-1k dataset - with FID improvements of 7% on 256 and 8% on 512 resolutions - as well as text-to-image generation on the CC12M dataset - with FID improvements of 8% on 256 and 23% on 512 resolution.

1 Introduction

Diffusion models have emerged as a powerful class of generative models and demonstrated unprecedented ability at generating high-quality and realistic images. Their superior performance is evident across a spectrum of applications, encompassing image [7, 14, 39, 41] and video synthesis [35], denoising [52], super-resolution [49] and layout-to-image synthesis [51]. The fundamental principle underpinning diffusion models is the iterative denoising of an initial sample from a trivial prior distribution, that progressively transforms it to a sample from the target distribution. The popularity of diffusion models can be attributed to several factors. First, they offer a simple yet effective approach for generative modeling, often outperforming traditional approaches such as Generative Adversarial Networks (GANs) [3, 16, 24, 25] and Variational Autoencoders (VAEs) [29, 48] in terms of visual fidelity and sample diversity. Second, diffusion models are generally more stable and less prone to mode collapse compared to GANs, which are notoriously difficult to stabilize without careful tuning of hyperparameters and training procedures [23, 30].

Funny realistic picture of a cat

dressed in a suit counting dollars

Husky puppies sleeping under

a blanket

A car goes by on an icy road

a lion enjoys meat frozen in a block

of ice during hot weather at a zoo

A small cat sitting in a

windowsill

a starry night style painting

of the eiffel tower

A dancing human skeleton

wearing a magician's hat

A beautiful statue of a horse

made of bronze

Despite the success of diffusion models, training such models at scale remains computationally challenging, leading to a lack of insights on the most effective training strategies. Training recipes of large-scale models are often closed ( e.g ., DALL-E, Imagen, Midjourney), and only a few studies have analyzed training dynamics in detail [7, 14, 26, 27]. Moreover, evaluation often involves human studies which are easily biased and hard to replicate [17, 56]. Due to the high computational costs, the research community mostly focused on the finetuning of large text-to-image models for different downstream tasks [1, 4, 54] and efficient sampling techniques [34, 36, 45]. However, there has been less focus on ablating different mechanisms to condition on user inputs such as text prompts, and strategies to pre-train using datasets of smaller resolution and/or data size. The benefits of conditioning mechanisms are two-fold: allowing users to have better control over the content that is being generated, and unlocking training on augmented or lower quality data by for example conditioning on the original image size [39] and other metadata of the data augmentation. Improving pre-training strategies, on the other hand, can allow for big cuts in the training cost of diffusion models by significantly reducing the number of iterations necessary for convergence.

Our work aims to disambiguate some of these design choices, and provide a set of guidelines that enable the scaling of the training of diffusion models in an efficient and effective manner. Beyond the main architectural choices ( e.g ., Unet vs . ViT), we focus on two other important aspects for generative performance and efficiency of training. First, we enhance conditioning by decoupling different conditionings based on their type: control metadata conditioning ( e.g ., crop size, random flip, etc .), semantic-level conditioning based on class names or text-prompts. In this manner, we disentangle the contribution of each conditioning and avoid undesired interference among them. Second, we optimize the scaling strategy to larger dataset sizes and higher resolution by studying the influence of the initialization of the model with weights from models pre-trained on smaller datasets and resolutions. Here, we propose three improvements needed to seamlessly transition across resolutions: interpolation of the positional embeddings, scaling of the noise schedule, and using a more aggressive data augmentation strategy.

In our experiments we evaluate models at 256 and 512 resolution on ImageNet-1k and Conceptual Captions (CC12M), and also present results for ImageNet-22k at 256 resolution. We study the following five architectures: Unet/LDM-G4 [39], DiT-XL2 w/ LN [38], mDT-v2-XL/2 w/ LN [15], PixArtα -XL/2 , and mmDiT-XL/2 (SD3) [14]. We find that among the studied base architectures, mmDiT-XL/2 (SD3) performs the best. Our improved conditioning approach further boosts the performance of the best model consistently across metrics, resolutions, and datasets. In particular, we improve the previous state-of-the-art DiT result of 3.04 FID on ImageNet-1k at 512 resolution to 2.76. For CC12M at 512 resolution, we improve FID of 11.24 to 8.64 when using our improved conditioning, while also obtaining a (small) improvement in CLIPscore from 26.01 to 26.17. See Fig. 1 for qualitative examples of our model trained on CC12M.

64

128

256

512

1024

In summary, our contributions are the following:

We introduce a conditioning mechanism that disentangles different control conditionings and semantic-level conditioning, improving generation and avoiding interference between conditions.

We present a systematic study of five different diffusion architectures, which we train from scratch using face-blurred ImageNet and CC12M datasets at 256 and 512 resolutions.

To transfer weights from pre-trained models we propose to interpolate positional embeddings, scale the noise schedule, and use stronger data augmentation, leading to improved performance.

We obtain state-of-the-art results at 256 and 512 resolution for class-conditional generation on ImageNet-1k and text-to-image generation and CC12M.

2 Conditioning and pre-training strategies for diffusion models

In this section, we review and analyze the conditioning mechanisms and pre-training strategies used in prior work (see more detailed discussion of related work in App. A), and propose improved approaches based on the analysis.

2.1 Conditioning mechanisms

Background. To control the generated content, diffusion models are usually conditioned on class labels or text prompts. Adaptive layer norm is a lightweight solution to condition on class labels, used for both UNets [21, 39, 41] and DiT models [38]. Cross-attention is used to allow more fine-grained conditioning on textual prompts, where particular regions of the sampled image are affected only by part of the prompt, see e.g . [41]. More recently, another attention based conditioning was proposed in SD3 [14] within a transformer-based architecture that evolves both the visual and textual tokens across layers. It concatenates the image and text tokens across the sequence dimension, and then performs a self-attention operation on the combined sequence. Because of the difference between the two modalities, the keys and queries are normalized using RMSNorm [53], which stabilizes training. This enables complex interactions between the two modalities in one attention block instead of using both self-attention and cross-attention blocks.

Moreover, since generative models aim to learn the distribution of the training data, data quality is important when training generative models. Having low quality training samples, such as the ones that are poorly cropped or have unnatural aspect ratios, can result in low quality generations. Previous work tackles this problem by careful data curation and fine-tuning on high quality data, see e.g . [7, 9]. However, strictly filtering the training data may deprive the model from large portions of the available data [39], and collecting high-quality data is not trivial. Rather than treating them as nuisance factors, SDXL [39] proposes an alternative solution where a UNet-based model is conditioned on parameters corresponding to image size and crop parameters during training. In this manner, the model is aware of these parameters and can account for them during training, while also offering users control over these parameters during inference. These control conditions are transformed and additively combined with the timestep embedding before feeding them to the diffusion model.

Disentangled control conditions. Straightforward implementation of control conditions in DiT may cause interference between the time-step, class-level and control conditions if their corresponding embeddings are additively combined in the adaptive layer norm conditioning, e.g . causing changes in high-level content of the generated image when modifying its resolution, see Fig. 2. To disentangle the different conditions, we propose two modifications. First, we move the class embedding to be fed through the attention layers present in the DiT blocks. Second, to ensure that the control embedding

does not overpower the timestep embedding when additively combined in the adaptive layer norm, we zero out the control embedding in early denoising steps, and gradually increase its strength.

Control conditions can be used for different types of data augmentations: (i) high-level augmentations ( ϕ h ) that affect the image composition e.g . flipping, cropping and aspect ratio -, and (ii) low-level augmentations ( ϕ l ) that affect low-level details e.g . image resolution and color. Intuitively, high-level augmentations should impact the image formation process early on, while low-level augmentations should enter the process only once sufficient image details are present. We achieve this by scaling the contribution of the low-level augmentations, ϕ l , to the control embedding using a cosine schedule that downweights earlier contributions:

0

250

500

750

1000

timestep

t

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

g

c

(

t

) =

(

1

-

cos

(

p

·

(

1

-

t

)

a

) )/

2

a

=

1

a

=

2

a

=

4

a

=

8

where the embedding functions E h , E l are made of sinusoidal embeddings followed by a 2-layer MLP with SiLU activation, and where γ c is the cosine schedule illustrated in Fig. 3.

Improved text conditioning. Most commonly used text encoders, like CLIP [40], output a constant number of tokens T that are fed to the denoising model (usually T = 77 ). Consequently, when the prompt has less than T tokens, the remaining positions are filled by zero-padding, but remain accessible via cross-attention to the denoising network. To make better use of the conditioning vector, we propose a noisy replicate padding mechanism where the padding tokens are replaced with copies of the text tokens, thereby pushing the subsequent cross-attention layers to attend to all the tokens in its inputs. As this might lead to redundant token embeddings, we improve the diversity of the feature representation across the sequence dimension, by perturbing the embeddings with additive Gaussian noise with a small variance β txt . To ensure enough diversity in the token embeddings, we scale the additive noise by σ ( ϕ txt ) √ m -1 , where m is the number of token replications needed for padding, and σ ( ϕ txt ) is the per-channel standard deviation in the token embeddings.

Integrating classifier-free guidance. Classifier-free guidance (CFG) [20] allows for training conditional models by combining the output of the uncoditional generation with the output of the conditional generation. Formally, given a latent diffusion model trained to predict the noise ϵ , CFG reads as: ϵ λ = λ · ϵ s +(1 -λ ) · ϵ ∅ , where ϵ ∅ is the uncoditional noise prediction, ϵ s is the noise prediction conditioned on the semantic conditioning s ( e.g ., text prompt), and λ is the hyper-parameter, known as guidance scale , which regulates the strength of the conditioning. Importantly, during training λ is set alternatively to 0 or 1, while at inference time it is arbitrarily changed in order to steer the generation to be more or less consistent with the conditioning. In our case, we propose the control conditioning to be an auxiliary guidance term, in order to separately regulate the strength of the conditioning on the control variables c and semantic conditioning s at inference time. In particular, we define the guided noise estimate as:

where β sets the strength of the control guidance, and λ sets the strength of the semantic guidance.

2.2 On transferring models pre-trained on different datasets and resolutions

Background. Transfer learning has been a pillar of the deep learning community, enabling generalization to different domains and the emergence of foundational models such as DINO [5, 10] and CLIP [40]. Large pre-trained text-to-image diffusion models have also been re-purposed for different tasks, including image compression [4] and spatially-guided image generation [1]. Here, we are interested in understanding to which extent pre-training on other datasets and resolutions can be leveraged to achieve a more efficient training of large text-to-image models. Indeed, training diffusion models directly to generate high resolution images is computationally demanding, therefore, it is common to either couple them with super-resolution models, see e.g . [44], or fine-tune them with high resolution data, see e.g . [7, 14, 39]. Although most models can directly operate at a higher resolution than the one used for training, fine-tuning is important to adjust the model to the different statistics of high-resolution images. In particular, we find that the different statistics influence the positional embedding of patches, the noise schedule, and the optimal guidance scale. Therefore, we focus on improving the transferability of these components.

Positional Embedding. Adapting to a higher resolution can be done in different ways. Interpolation scales the - most often learnable - embeddings according to the new resolution [2, 47]. Extrapolation simply replicates the embeddings of the original resolution to higher resolutions as illustrated in Fig. 4, resulting in a mismatch between the positional embeddings and the image features when switching to different resolutions. Most methods that use interpolation of learnable positional embeddings, e.g . [2, 47], adopt either bicubic or bilinear interpolation to avoid the norm reduction associated with the interpolation. In our case, we take advantage of the fact that our embeddings are sinusoidal and simply adjust the sampling grid to have constants limit under every resolution, see App. C.

Scaling the noise schedule. At higher resolution, the amount of noise necessary to mask objects at the same rate changes [14, 22]. If we observe a spatial patch at low resolution under a given uncertainty, upscaling the image by a factor s creates s 2 observations of this patch of the form y ( i ) t = x t + σ t ϵ ( i ) -assuming the value of the patch is constant across the patch. This increase in the number of observations reduces the uncertainty around the value of that token, resulting in a higher

Low res.

High res.

(

extrapolated

)

High res.

(

interpolated

)

signal-to-noise (SNR) ratio than expected. This issue gets further accentuated when the scheduler does not reach a terminal state with pure noise during training, i.e ., a zero SNR [32], as the mismatch between the non-zero SNR seen during training and the purely Gaussian initial state of the sampling phase becomes significant. To resolve this, we scale the noise scheduler in order to recover the same uncertainty for the same timestep.

Proposition 1. When going from a scale of s to a scale s ′ , we update the β scheduler according to the following rule

This increases the noise amplitude during intermediate denoising steps as illustrated in Fig. A1. The final equation obtained is similar to the one obtained in [14] with the accompanying change of variable t = σ 2 1+ σ 2 .

Pre-training cropping strategies. When pre-training and finetuning at different resolutions, we can either first crop and then resize the crops according to the training resolution, or directly take differently sized crops from the training images. Using a different resizing during pre-training and finetuning may introduce some distribution shift, while using crops of different sizes may be detrimental to low-resolution training as the model will learn the distribution of smaller crops rather than full images, see Fig. 5. We experimentally investigate which strategy is more effective for low-resolution pretraining of high-resolution models.

Global

context

Target

resolution

vs.

Guidance scale. We discover that the optimal guidance scale for both FID and CLIPScore varies with the resolution of images. In App. D, we present a proof revealing that under certain conditions, the optimal guidance scale adheres to a scaling law with respect to the resolution, as

3 Experimental evaluation

3.1 Experimental setup

Datasets. In our study, we train models on three datasets. To train class-conditional models, we use ImageNet-1k [11], which has 1.3M images spanning 1,000 classes, as well as ImageNet-22k [43], which contains 14.2M images spanning 21,841 classes. Additionally, we train text-to-image models using Conceptual 12M (CC12M) [6], which contains 12M images with accompanying manually generated textual descriptions. We pre-process both datasets by blurring all faces. Differently from [7], we use the original captions for the CC12M dataset.

Evaluation. For image quality, we evaluate our models using the common FID [19] metric. We follow the standard evaluation protocol on ImageNet to have a fair comparison with the relevant

literature [3, 15, 38, 41]. Specifically, we compute the FID between the full training set and 50k synthetic samples generated using 250 DDIM sampling steps. For image-text alignment, we compute the CLIP [40] score similarly to [7, 14]. We measure conditional diversity, either using class-level or text prompt conditioning, using LPIPS [55]. LPIPS is measured pairwise and averaged among ten generations obtained with the same random seed, prompt, and initial noise, but different size conditioning (we exclude sizes smaller than the target resolution); then we report the average over 10k prompts. In addition to ImageNet and CC12M evaluations, we provide FID and CLIPScore on the COCO [33] validation set, which contains approximately 40k images with associated captions. For COCO evaluation [33], we follow the same setting as [14] for computing the CLIP score, using 25 sampling steps and a guidance scale of 5 . 0 .

Training. To train our models we use the Adam [28] optimizer, with a learning rate of 10 -4 and β 1 , β 2 = 0 . 9 , 0 . 999 . When training at 256 × 256 resolution, we use a batch size of 2 , 048 images, a constant learning rate of 10 × 10 -4 , train our models on two machines with eight A100 GPUs each. In preliminary experiments with the DiT architecture we found that the FID metric on ImageNet-1k at 256 resolution consistently improved with larger batches and learning rate, but that increasing the learning rate by another factor of two led to diverging runs. We report these results in supplementary. When training models at 512 × 512 resolution, we use the same approach but with a batch size of 384 distributed over 16 A100 GPUs. We train our ImageNet-1k models for 500k to 1M iterations and for 300k to 500k iterations for CC12M.

Model architectures. Wetrain different diffusion architectures under the same setting to provide a fair comparison between model architectures. Specifically, we re-implement a UNet-based architecture following Stable Diffusion XL (SDXL) [39] 1 and several transformer-based architectures: vanilla DiT [38], masked DiT (mDiT-v2) [15], PixArt DiT (PixArtα ) [7], and multimodal DiT (mmDiT) as in Stable Diffusion 3 [14]. For vanilla DiT, which only supports class-conditional generation, we explore two variants one incorporating the class conditioning within LayerNorm and another one within the attention layer. Also, for text-conditioned models, we use the text encoder and tokenizer of CLIP (ViT-L/14) [40] having a maximum sequence length of T = 77 . The final models share similar number parameters, e.g . for DiTs we inspect the XL/2 variant [38], for UNet (SDXL) we adopt similar size to the original LDM [41]. Similar to [14], we found the training of DiT with

cross-attention to be unstable and had to resort to using RMSNorm [53] to normalize the key and query in the attention layers. We detail the models sizes and computational footprints in Tab. A1.

3.2 Evaluation of model architectures and comparison with the state of the art

In Tab. 1, we report results for models with different architectures trained at both 256 and 512 resolutions for ImageNet and CC12M, and compare our results (2nd block of rows) with those reported in the literature, where available (1st block of rows). Where direct comparison is possible, we notice that our re-implementation outperforms the one of existing references. Overall, we found the mmDiT [14] architecture to perform best or second best in all settings compared to other alternatives. For this reason, we apply our conditioning improvements on top of this architecture (last row of the table), boosting the results as measured with FID and CLIPScore in all settings. Below, we analyse the improvements due to our conditioning mechanisms and pre-training strategies.

3.3 Control conditioning

Scheduling rate of control conditioning. In Tab. 2a we consider the effect of controlling the conditioning on low-level augmentations via a cosine schedule for different decay rates α . We compare to baselines (first two rows) with constant weighting (as in SDXL [39]) and without control conditioning. We find that our cosine weighting schedule significantly reduces the dependence between size control and image semantics as it drastically improves the instance specific LPIPS (0.33 vs . 0.04) in comparison to uniform weighting. In terms of FID, we observe a small gap with the baseline (3.04 vs . 3.08), which increases (3.80 vs . 5.04) when computing FID by randomly sampling the size conditioning in the range [512,1024], see Tab. 2b. Finally, the improved disentangling between semantics and low-level conditioning is clearly visible in the qualitative samples in Fig. 2.

Crop and random-flip control conditioning. A potential issue of horizontal flip data augmentations is that it can create misalignment between the text prompt and corresponding image. For example the prompt "A teddy bear holding a baseball bat in their right arm" will no longer be accurate when an image is flipped - showing a teddy bear holding the bat in their left arm. Similarly, cropping images can remove details mentioned in the corresponding caption. In Tab. 2c we evaluate models trained on CC12M@256 with and without horizontal flip conditioning, and find that adding this conditioning leads to slight improvements in both FID and CLIP as compared to using only crop conditioing. We depict qualitative comparison in Fig. 6, where we observe that flip conditioning improves prompt-layout consistency.

Inference-time control conditioning of image size. High-level augmentations ( ϕ h ) may affect the image semantics. As a result they influence the learned distribution and modify the generation diversity. For example, aspect ratio conditioning can harm the quality of generated images, when images of a particular class or text prompt are unlikely to appear with a given aspect ratio. In Tab. 2b we compare of different image size conditionings for inference. We find that conditioning on the same size distribution as encountered during the training of the model yields a significant boost in FID

Flip cond

No flip cond

0

1

2

3

4

5

Guidance scale

7

.

5

10

.

0

12

.

5

FID@256

0

1

2

3

4

5

Guidance scale

5

10

15

FID@512

0

1

2

3

4

Control guidance scale

2

4

FID@256

as compared to generating all images with constant size conditioning or using uniformly randomly sampled sizes. Note that in all cases images are generated at 256 resolution.

Control conditioning and guidance. To understand how control condition impacts the generation process, we investigate the influence of control guidance β (introduced in Sec. 2.1 ) on FID and report the results in Fig. 7. We find that a higher control guidance scale results in improved FID scores. However, note that this improvement comes at the cost of compute due to the additional control term ϵ c,s .

Replication text padding. We compare our noisy replication padding to the baseline zero-padding in Tab. 3. We observe that using a replication padding improves both FID and CLIP score, and that adding scaled perturbations further improves the results 0 . 35 point improvement in CLIP score and 0 . 4 point improvement in FID.

3.4 Transferring weights between datasets and resolutions

Dataset shift. We evaluate the effect of pre-training on ImageNet-1k (at 256 resolution) when training the models on CC12M or ImageNet-22k (at 512 resolution) by the time needed to achieve the same performance as a model trained from scratch. In Tab. 4a, when comparing models trained from scratch to ImageNet-1k pre-trained models (600k iterations) we observe two benefits: improved training convergence and performance boosts. For CC12M, we find that after only 100k iterations, both FID and CLIP scores improve over the baseline model trained with more than six times the

25

50

75

100

125

Iteration (k)

5

.

00

5

.

25

5

.

50

5

.

75

6

.

00

FID@512

Default

Scaled

0

10

20

30

40

Iteration (k)

4

.

0

4

.

5

5

.

0

5

.

5

FID@512

Default

Scaled

0

50

100

150

200

Iteration (k)

5

.

0

5

.

5

6

.

0

6

.

5

7

.

0

FID@512

Mixed

Local

Global

amount of training iterations. For ImageNet-22k, which is closer in distribution to ImageNet-1k than CC12M, the gains are even more significant, the finetuned model achieves an FID lower by 0 . 5 point after only 80k training iterations. In Tab. 4b, we study the relative importance of pre-training vs finetuning when the datasets have dissimilar distributions but similar sample sizes. We fix a training budget in terms of number of training iterations N , we first train our model on ImageNet-22k for K iterations before continuing the training on CC12M for the remaining N -K iterations. We see that the model pretrained for 200k iterations and finetuned for 150k performs better than the one spending the bulk of the training during pretraining phase. This validates the importance of domain specific training and demonstrates that the bulk of the gains from the pretrained checkpoint come from the representation learned during earlier stages.

Resolution change. We compare the performance boost obtained from training from scratch at 512 resolution vs . resuming from a 256 -resolution trained model. According to our results in Tab. 4c, pretraining on low resolution significantly boosts the performance at higher resolutions, both for UNet and mmDiT, we find that higher resolution finetuning for short periods of time outperforms high resolution training from scratch by a large margin ( ≈ 25% ). These performance gains might in part be due to the increased batch size when pre-training the 256 resolution model, which allows the model to 'see' more images as compared to training from-scratch at 512 resolution.

Positional Embedding. In Fig. 8a, we compare the influence of the adjustment mechanism for the positional embedding. We find that our grid resampling approach outperforms the default extrapolation approach, resulting in 0.2 point difference in FID after 130k training iterations.

Scaling the noise schedule. We conducted an evaluation to ascertain the influence of the noise schedule by refining our mmDiT model post its low resolution training and report the results in Fig. 8b. Remarkably, the application of the rectified schedule, for 40k iterations, resulted in an improvement of 0.7 FID points demonstrating its efficacy at higher resolutions.

Pre-training cropping strategies. During pretraining, the model sees objects that are smaller than what it sees during fine tuning, see Fig. 5. We aim to reduce this discrepancy by adopting more agressive cropping during the pretraining phase. We experiment with three cropping ratios for training: 0 . 9 -1 (global), 0 . 4 -0 . 6 (local), 0 . 4 -1 (mix). We report the results in Fig. 8c. On ImageNet1K@256, the pretraining FID scores are 2 . 36 , 245 . 55 and 2 . 21 for the local, global and mixed strategies respectively. During training at 512 resolution, we observe that the global and mix cropping strategies both outperform the local strategy. However, as reported in Fig. 8c, the local strategy provides benefits at higher resolutions. Overall, training with the global strategy performs the best at 256 resolution but lags behind for higher resolution adaptation. While local cropping underperforms at lower resolutions, because it does not see any images in their totality, it outperforms the other methods at higher resolutions - an improvement of almost 0 . 2 FID points is consistent after the first 50 k training steps at higher resolution.

4 Conclusion

In this paper, we explored various approaches to enhance the conditional training of diffusion models. Our empirical findings revealed significant improvements in the quality and control over generated images when incorporating different coditioning mechanisms. Moreover, we conducted a

comprehensive study on the transferability of these models across diverse datasets and resolutions. Our results demonstrated that leveraging pretrained representations is a powerful tool to improve the model performance while also cutting down the training costs. Furthermore, we provided valuable insights into efficiently scaling up the training process for these models without compromising performance. By adapting the schedulers and positional embeddings when scaling up the resolution, we achieved substantial reductions in training time while boosting the quality of the generated images. Additional experiments unveil the expected gains from different transfer strategies, making it easier for researchers to explore new ideas and applications in this domain. In Appendix B we discuss societal impact and limitations of our work.

References

Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin. Spatext: Spatio-textual representation for controllable image generation. In CVPR , 2023.

Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon Kornblith, Xiaohua Zhai, Matthias Minderer, Michael Tschannen, Ibrahim Alabdulmohsin, and Filip Pavetic. FlexiViT: One model for all patch sizes. In CVPR , 2023.

Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR , 2019.

Marlène Careil, Matthew J. Muckley, Jakob Verbeek, and Stéphane Lathuilière. Towards image compression with perfect realism at ultra-low bitrates. In ICLR , 2024.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV , 2021.

Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR , 2021.

Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. PixArtα : Fast training of diffusion transformer for photorealistic text-to-image synthesis. In ICLR , 2024.

Jaemin Cho, Abhay Zala, and Mohit Bansal. DALL-Eval: Probing the reasoning skills and social biases of text-to-image generation models. In ICCV , 2023.

Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, Matthew Yu, Abhishek Kadian, Filip Radenovic, Dhruv Mahajan, Kunpeng Li, Yue Zhao, Vladan Petrovic, Mitesh Kumar Singh, Simran Motwani, Yi Wen, Yiwen Song, Roshan Sumbaly, Vignesh Ramanathan, Zijian He, Peter Vajda, and Devi Parikh. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint , 2309.15807, 2023.

Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In ICLR , 2024.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In ICCV , 2009.

Prafulla Dhariwal and Alex Nichol. Diffusion models beat GANs on image synthesis. In NeurIPS , 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16 × 16 words: Transformers for image recognition at scale. In ICLR , 2021.

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In ICML , 2024.

Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is a strong image synthesizer. In ICCV , 2023.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS , 2014.

Melissa Hall, Samuel J. Bell, Candace Ross, Adina Williams, Michal Drozdzal, and Adriana Romero Soriano. Towards geographic inclusion in the evaluation of text-to-image models. In FAccT , 2024.

Melissa Hall, Candace Ross, Adina Williams, Nicolas Carion, Michal Drozdzal, and Adriana RomeroSoriano. DIG in: Evaluating disparities in image generations with indicators for geographic diversity. TMLR , 2024.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In NeurIPS , 2017.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS Workshop on Deep Generative Models and Downstream Applications , 2021.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS , 2020.

Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. Simple diffusion: End-to-end diffusion for high resolution images. In ICML , 2023.

Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up GANs for text-to-image synthesis. In CVPR , 2023.

Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In CVPR , 2020.

Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In NeurIPS , 2021.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In NeurIPS , 2022.

Tero Karras, Janne Hellsten, Miika Aittala, Timo Aila, Jaakko Lehtinen, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. In CVPR , 2024.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 2015.

Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In ICLR , 2014.

Kwonjoon Lee, Huiwen Chang, Lu Jiang, Han Zhang, Zhuowen Tu, and Ce Liu. ViTGAN: Training GANs with vision transformers. In ICLR , 2022.

Mark Levy, Bruno Di Giorgi, Floris Weers, Angelos Katharopoulos, and Tom Nickson. Controllable music production with diffusion models and guidance gradients. arXiv preprint , 2311.00613, 2023.

Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In WACV , 2024.

T.-Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. Zitnick. Microsoft COCO: common objects in context. In ECCV , 2014.

Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In ICLR , 2023.

Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, Lifang He, and Lichao Sun. Sora: A review on background, technology, limitations, and opportunities of large vision models. arXiv preprint , 2402.17177, 2024.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In NeurIPS , 2022.

Alexandra Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite. Stable bias: Analyzing societal representations in diffusion models. In NeurIPS , 2023.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV , 2023.

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In ICLR , 2024.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML , 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR , 2022.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention , 2015.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) , 115(3):211-252, 2015.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS , 2022.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR , 2021.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR , 2021.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML , 2021.

A. Vahdat and J. Kautz. NVAE: A deep hierarchical variational autoencoder. In NeurIPS , 2020.

Chanyue Wu, Dong Wang, Yunpeng Bai, Hanyu Mao, Ying Li, and Qiang Shen. HSR-Diff: Hyperspectral image super-resolution via conditional diffusion models. In ICCV , 2023.

Tong Wu, Zhihao Fan, Xiao Liu, Yeyun Gong, Yelong Shen, Jian Jiao, Hai-Tao Zheng, Juntao Li, Zhongyu Wei, Jian Guo, Nan Duan, and Weizhu Chen. AR-Diffusion: Auto-regressive diffusion model for text generation. In NeurIPS , 2023.

Han Xue, Zhiwu Huang, Qianru Sun, Li Song, and Wenjun Zhang. Freestyle layout-to-image synthesis. In CVPR , 2023.

Cheng Yang, Lijing Liang, and Zhixun Su. Real-world denoising via diffusion model. arXiv preprint , 2305.04457, 2023.

Biao Zhang and Rico Sennrich. Root Mean Square Layer Normalization. In NeurIPS , 2019.

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV , 2023.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR , 2018.

Sharon Zhou, Mitchell L. Gordon, Ranjay Krishna, Austin Narcomey, Durim Morina, and Michael S. Bernstein. HYPE: a benchmark for Human eYe Perceptual Evaluation of generative models. In NeurIPS , 2019.

Appendix

Table of Contents

A Related work

Diffusion Models. Diffusion models have gained significant attention in recent years due to their ability to model complex stochastic processes and generate high-quality samples. These models have been successfully applied to a wide range of applications, including image generation [7, 21, 41], video generation [35], music generation [31], and text generation [50]. One of the earliest diffusion models was proposed in [21], which introduced denoising diffusion probabilistic models (DDPMs) for image generation. This work demonstrated that DDPMs can generate high-quality images that competitive with state-of-the-art generative models such as GANs [16]. Following this work, several variants of DDPMs were proposed, including score-based diffusion models [46], conditional diffusion models [12], and implicit diffusion models [45]. Overall, diffusion models have shown promising results in various applications due to their ability to model complex stochastic processes and generate high-quality samples [7, 14, 39, 41]. Despite their effectiveness, diffusion models also have some limitations, including the need for a large amount of training data and the required computational resources. Some works [26, 27] have studied and analysed the training dynamics of diffusion models, but most of this work considers the pixel-based models and small-scale settings with limited image resolution and dataset size. In our work we focus on the more scalable class of latent diffusion models [41], and consider image resolutions up to 512 pixels, and 14M training images.

Model architectures. Early work on diffusion models adopted the widely popular UNet arcchitecture [39, 41]. The UNet is an encoder-decoder architecture where the encoder is made of residual blocks that produce progressively smaller feature maps, and the decoder progressively upsamples the feature maps and refines them using skip connections with the encoder [42]. For diffusion, UNets are also equipped with cross attention blocks for cross-modality conditioning and adaptive layer normalization that conditions the outputs of the model on the timestep [41]. More recently, vision transformer architectures [13] were shown to scale more favourably than UNets for diffusion models with the DiT architecture [38]. Numerous improvements have been proposed to the DiT in order to have more efficient and stable training, see e.g . [7, 14, 15]. In order to reduce the computational complexity of the model and train at larger scales, windowed attention has been proposed [7]. Latent masking during training has been proposed to encourage better semantic understanding of inputs in [15]. Others improved the conditioning mechanism by evolving the text tokens through the layers of the transformer and replacing the usual cross-attention used for text conditioning with a variant that concatenates the tokens from both the image and text modalities [14].

Large scale diffusion training. Latent diffusion models [41] unlocked training diffusion models at higher resolutions and from more data by learning the diffusion model in the reduced latent space of a (pre-trained) image autoencoder rather than directly in the image pixel space. Follow-up work has proposed improved scaling of the architecture and data [39]. More recently, attention-based architectures [7, 14, 15] have been adapted for large scale training, showing even more improvements

0

200

400

600

800

1000

timestep

0.0

0.2

0.4

0.6

0.8

1.0

t

(

s

)

s=1

s=2

s=3

s=4

0

200

400

600

800

1000

timestep

0.0

0.2

0.4

0.6

0.8

1.0

t

(

s

)

1+

t

(

s

)

2

s=1

s=2

s=3

s=4

by scaling the model size further and achieving state-of-the-art performance on datasets such as ImageNet-1k. Efficiency gains were also obtained by [7] by transferring ImageNet pre-trained models to larger datasets.

B Societal impact and limitations

Our research investigates the training of generative image models, which are widely employed to generate content for creative or education and accessibility purposes. However, together with these beneficial applications, these models are usually associated with privacy concerns ( e.g ., deepfake generation) and misinformation spread. In our paper, we deepen the understanding of the training dynamics of these modes, providing the community with additional knowledge that can be leveraged for safety mitigation. Moreover, we promote a safe and transparent/reproducible research by employing only publicly available data, which we further mitigate by blurring human faces.

Our work is mostly focused on training dynamics, and to facilitate reproducibility, we used publicly available datasets and benchmarks, without applying any data filtering. We chose datasets relying on the filtering/mitigation done by their respective original authors. In general, before releasing models like the ones described in our paper to the public, we recommend conducting proper evaluation of models trained using our method for bias, fairness and discrimination risks. For example, geographical disparities due to stereotypical generations could be revealed with methods described by Hall et al. [18], and social biases regarding gender and ethnicity could be captured with methods from Luccioni et al. [37] and Cho et al. [8].

While our study provides valuable insights into control conditioning and the effectiveness of representation transfer in diffusion models, there are several limitations that should be acknowledged. (i) There are cases where these improvements can be less pronounced. For example, noisy replicates for the text embeddings can become less pertinent if the model is trained exclusively with long prompts. (ii) While low resolution pretraining with local crops on ImageNet resulted in better FID at 512 resolution (see Table 5c), it might not be necessary if pretraining on much larger datasets ( e.g . > 100M samples, which we did not experiment in this work). Similarly, flip conditioning is only pertinent if the training dataset contains position sensitive information (left vs. right in captions, or rendered text in images), otherwise this condition will not provide any useful signal. (iii) We did not investigate the impact of data quality on training dynamics, which could have implications for the generalizability of our findings to datasets with varying levels of quality and diversity. (iv) As our analysis primarily focused on control conditioning, other forms of conditioning such as timestep and prompt conditioning were not explored in as much depth. Further research is needed to determine the extent to which these conditionings interact with control conditioning and how that impacts the quality of the models. (v) Our work did not include studies on other parts that are involved in the training and sampling of diffusion models, such as the different sampling mechanisms and training paradigms. This could potentially yield additional gains in performance and uncover new insights about the state-space of diffusion models.

C Implementation details

Noise schedule scaling. In Fig. A1 we depict the rescaling of the noise for higher resolutions, following Eq. (3).

Positional embedding rescaling. As illustrated in App. C, rescaling the positional embedding can be integrated in two simple lines of code by changing the grid sampling to be based on reducing the stepsize in the grid instead of extending its limits.

Computational costs. In Tab. A1 we compare the model size and computational costs of the different architectures studied in the main paper. All model architectures are based on the original implemetations, but transposed to our codebase. Mainly, we use bfloat16 mixed precision and memory-efficient attention 2 from PyTorch.

Experimental details. To ensure efficient training of our models, we benchmark the most widely used hyperparameters and report the results of these experiments, which consist of the choice of the optimizer and its hyperparameters, the learning rate and the batch size. We then transpose the optimal settings to our other experiments. For FID evaluation, we use a guidance scale of 1 . 5 for 256 resolution and 2 . 0 for resolution 512 . For evaluation on ImageNet-22k, we compute the FID score between 50k generated images and a subset of 200k images from the training set.

Training paradigm. We use the EDM [26] abstraction to train our models for epsilon prediction following DDPM paradigm. Differently from [15, 38], we do not use learned sigmas but follow a standard schedule. Specifically, we use a quadratic beta schedule with β start = 0 . 00085 and β end = 0 . 012 . In DDPM [21], a noising step is formulated as follows, with x 0 being the data sample and t ∈ [ [0 , T ] ] a timestep:

while in EDM, the cumulative alpha products are converted to corresponding signal-to-noise ratio (SNR) proportions, the noising process is then reformulated as:

and the loss is weighted by the inverse SNR 1 σ 2 t .

Scaling the training. A recurrent question when training deep networks is the coupling between the learning rate and the batch size. When multiplying the batch size by a factor γ , some works recommend scaling the learning rate by the square root of γ , while others scale the learning rate by

the factor γ itself. In the following we experiment with training a class-conditional DiT model with different batch sizes and learning rates and report results after the same number of iterations.

From Tab. A2 we observe an improved performance by increasing the batch size and the learning rate in every learning rate setting. If the learning rate is too high the training diverges.

Influence of momentum. We conduct a grid search over the momentum parameters of Adam optimizer, similar to previous sections, we train a UNet model fo 70k steps and compute FID with respect to the validation set of ImageNet1k using 50 DDIM steps. From Fig. A2, we can see that the default pytorch values ( β 1 = 0 . 9 , β 2 = 0 . 999) are sub-optimal, resulting in an FID of 26.43 while the best performance is obtained when setting ( β 1 = 0 . 9 , β 2 = 0 . 95) improves FID by 4.78 points in our experimental setting.

Setting the optimal guidance scale.

Proposition 2. The optimal guidance scale λ scales with the upsampling factor s according to the law λ ′ ( s ) = 1 + s · ( λ -1) .

This is verified in Fig. 7 where the λ ′ ( s = 2) = 1 + 2 · (1 . 5 -1 . 0) = 2 . 0 which is the optimal guidance scale at 512 resolution according to the figure.

Text padding mechanism. In order to train at a large scale, most commonly used text encoders output a constant number of tokens T that are fed to the denoising model (usually T = 77 ). Consequently, when the prompt has less than T words, the remaining tokens are padding tokens that do not contain useful information, but can still contribute in the cross-attention, see Figure A3. This raises the question of whether better use can be made of padding text tokens to improve training performance and efficiency . One common mitigation involves using recaptioning methods that provide longer captions. However, this creates an inconsistency between training and sampling as users are more likely to provide shorter prompts. Thus, to make better use of the conditioning vector, we explore alternative padding mechanisms for the text encoder. We explore a ' replicate ' padding mechanism where the padding tokens are replaced with copies of the text tokens, thereby pushing the subsequent cross-attention layers to attend to all

0.5

0.95

0.99

0.999

β

2

0.0

0.5

0.9

0.99

β

1

24.02

25.52

24.28

26.11

28.38

25.93

21.79

24.7

25.64

21.65

26.98

26.43

23.4

25.88

27.43

22.42

22

23

24

25

26

27

28

0

10

20

30

40

50

60

70

Token index

-

2

-

1

0

1

2

Token gradient

Number of tokens

𝐾

𝑄

𝑉

Padding tokens

Padding tokens

Image tokens

Text tokens

Image tokens

Text tokens

×

(a)

Zero padding.

𝐾

𝑄

×

Padding tokens

Padding tokens

Image tokens

Text tokens

Image tokens

Text tokens

(b)

Replicate padding.

𝐾

𝑄

𝑉

×

Padding tokens

Padding tokens

Image tokens

Text tokens

Image tokens

Text tokens

(c)

Noisy replicate padding.

the tokens in its inputs. To improve the diversity of the feature representation across the sequence dimension, we perturb the embeddings with additive Gaussian noise with a small variance β txt . For shorter prompts with a high number of repeats m , we scale the additive noise by √ m -1 to account for the reduction in posterior uncertainty induced by these repetitions:

where ϕ ch is the standard deviation of the feature text embeddings over the feature dimension. See Figure A4 for an illustration comparing zero-padding, replication padding, and our noisy replication padding.

D Derivations

Derivation for Eq. (2)

Proof. The formula can be obtained by iteratively applying the guidance across the conditions.

Derivation for Eq. (4)

Proof. Assuming that the unconditional prediction ϵ ∅ is distributed around the conditional distribution according to a normal law ϵ ∅ | ϵ c ∼ N ( ϵ c , δ 2 I ) .

After upsampling by a scale factor of s , the same low resolution patch has s 2 observations, hence the variance is decreased by s 2 .

Hence by equalizing the discrepancy between the conditional and unconditional predictions at low and high resolutions we obtain:

3.-

Derivation for Eq. (3)

Proof. At timestep t , the noisy observation is obtained as :

Hence x 0 can be estimated using the following formula:

The statistics of the estimate of x 0 are as follows.

Consequently, if we already have x t , we have an estimate of x 0 with an error bound given by:

At higher resolutions, we have s 2 corresponding observations for the same patch such that the error with respect to the estimate becomes:

Hence, if we want to keep the same uncertainty with respect to the low resolution patches, the following equality needs to be verified:

E Additional results

Qualitative results. We provide additional qualitative examples on ImageNet-1k in Fig. A5.

Summary of findings. In Table A3 we summarize the improvements w.r.t. the DiT baseline obtained by the changes to the model architecture and training. In Table A4 we compare our model architecture and training recipe to that of SDXL and SD3. In Table A5, we provide a synopsis of the research questions addressed in our study alongside a respective recommendation based on our findings.

Effectiveness of the power cosine schedule We experiment with different function profiles for controlling the conditioning on low-level augmentations. Specifically, we compare the power-cosine profile with a linear and a piecewise constant profile. While the linear schedule manages an acceptable performance in terms of reducing LPIPS (although still higher than the power-cosine profile), it still achieves a higher FID than all the configurations with the cosine schedule. For the piecewise constant profiles, they achieve a higher LPIPS while also having a higher FID. In conclusion, the proposed

power-cosine profile outperforms these simpler schedules in both FID and LPIPS, improving image quality while better removing the unwanted distribution shift induced from choosing different samples during training.

F NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly set the claims made in the paper and match theoretical and experimental results, which are addressed in Section 3.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We include discussion of limitations in Appendix B.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Proofs are provided Appendix D and contain cross-references to the relevant parts in the main paper.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We include details required for result reproducibility in Section 3.1. These include details about datasets used for training, evaluation metrics, training parameters, and model architectures.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: While we are not be able to include code, we do provide specific and thorough detailing of our training and evaluation methods in Section 3.1 to enable experimental reproduction.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Details corresponding to training and testing are included in Section 3.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Due to the computational cost of training and performing inference with generative models, we do not include error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Information about compute resources needed for experiment reproduction is included in Section 3.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This research conforms to the Code of Ethics, including mitigation of potential harms caused by the research process, considerations of social impact, and taking steps for impact mitigation.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss societal impacts and steps taken to mitigate potential harms in Appendix B.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release models nor data in this work.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Use of existing assets is discussed, including which version, and cited throughout the paper. No assets are released.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not include crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the training efficiencies and model performance of latent diffusion models (LDMs) by reimplementing and analyzing various previously published models. The study introduces a novel conditioning mechanism that separates semantic and control metadata inputs, significantly improving class-conditional and text-to-image generation performances on benchmark datasets.\n\nSTRENGTHS:\n(1) The introduction of a novel conditioning mechanism that disentangles semantic and control metadata conditioning is a good advancement. It addresses the interference issue in generative performance, which is a common challenge in diffusion models.\n(2) The paper achieves state-of-the-art results in class-conditional generation on ImageNet-1k and text-to-image generation on CC12M.\n(3) The paper extends theoretical understanding by proposing modifications to the noise scheduling and positional embeddings that are informed by a principled approach.\n\nWEAKNESSES:\n(1) The paper is primarily evaluated on large-scale datasets like ImageNet and CC12M, but its effectiveness in low-data or small-scale datasets is not demonstrated.\n(2) The paper briefly mentions instability issues with certain models (e.g., DiT with cross-attention), but does not provide detailed insights or solutions to these problems. \n(3) Although the paper presents some ablation studies, they are not comprehensive. More detailed ablation studies isolating the impact of each proposed innovation are recommended.\n\nQUESTIONS:\n(1) What are the specific scenarios or cases where your proposed methods did not perform well? \n(2) Have you tested your proposed methods on generative tasks beyond image generation, such as text generation or audio synthesis?\n\nLIMITATIONS:\n(1) The paper mentions instability issues with certain models but does not provide a detailed analysis.\n(2) Insufficient ablation experiments.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper's contributions can be divided into three main points:\n\n1. Proposing better methods for metadata/semantic level conditioning:\n   - Instead of using AdaLN, class information is fed into the model through attention.\n   - Adjusting the strength of meta conditions related to low-level augmentation according to diffusion timesteps.\n   - Suggesting a method of copying text tokens and applying Gaussian perturbation instead of zero padding.\n   - Proposing additional guidance for the control variable \\(c\\) (Eq. 2).\n\n2. Proposing a method of transfer learning from low resolution to high resolution:\n   - Using interpolation for positional embedding.\n   - Adjusting the noise schedule according to SNR changes with resolution.\n   - Experimentally showing that resizing is better than cropping for creating low-resolution images.\n   - Proposing a strategy for adjusting guidance strength according to resolution.\n\n3. Reproducing experiments:\n   - Demonstrating experimentally that mmDiT has the best performance among various models proposed so far.\n\nThey demonstrated the validity of each method through extensive experiments on the Imagenet and CC12M 256x256 (512x512 for finetuning) datasets.\n\nSTRENGTHS:\n1. Extensive experiments:\n   - The performance comparison through re-implementation of all structures is a contribution in itself.\n   - Performance improvements were shown based on FID for all proposed methods. Such an extensive ablation study is rare.\n\nWEAKNESSES:\n1. Unorganized Experiment Result Presentation\n   - One of the key results of this study is Imagenet-1K FID 1.59, CC12M FID 6.79, and 8.64.\n   - However, it is difficult to understand how each element claimed by this study contributes to these results.\n   - The best presentation would be to examine how removing each configuration affects the Imagenet-1K FID 1.59.\n   - Practically, it seems challenging to run 1M iterations for all configurations. It would have been better to show how changing each configuration affects the best config based on 120k iterations.\n   - For example, it would be useful to have something like Table 1 from the EDM 2 paper [1].\n   - This is not just for easier reader comprehension but is also important for understanding how much the proposed disentangled control conditions contributed to the Imagenet-1K FID improvement from 1.71 to 1.59.\n   - If these concerns are well resolved, I am considering raising the score.\n\n\n2. Method Presentation\n   - This study has a very complex design space.\n   - However, it is currently difficult to know exactly what the current settings are and how they differ from past settings. Could you include something like Table 1 from the EDM 1 paper [2]?\n\n[1]: Analyzing and Improving the Training Dynamics of Diffusion Models, https://arxiv.org/abs/2312.02696\n[2]: Elucidating the Design Space of Diffusion-Based Generative Models, https://arxiv.org/abs/2206.00364\n\nQUESTIONS:\nMinor suggestion:\n- If adding the flip condition is part of the method, it would be better to introduce it in the method section rather than having it first appear in the experiment section at L264.\n\nLIMITATIONS:\nThe author has appropriately described the limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies how to effectively condition image size and crop information, as done by Stable Diffusion XL, and how to implement an effective coarse-to-fine training strategy. For control conditioning, it is designed to be less entangled than traditional methods, resulting in better performance for the same backbone model. The analysis for model transfer in coarse-to-fine training includes noise scheduling and positioning embedding. Additionally, the paper compares backbone models by re-implementing and comparing UNet, DiT, and mDiT. For analysis, ImageNet1K/22K data was used for class2image, and CC12M data for text2image.\n\nSTRENGTHS:\nI agree on the necessity of an apple-to-apple comparison of current backbone models. Comparing various backbones under identical experimental conditions and conducting ablation experiments across several attributes will serve as valuable preliminary research for future model training. To be specific, it was good to see the performance improvements brought by the control conditioning design, which is easy to overlook. Additionally, this paper improves the performance of stable diffusion-3 and effectively demonstrates controlled experiments for hyperparameters that were previously unorganized among engineers.\n\nWEAKNESSES:\n- The explanation and analysis of the chosen designs are lacking for me. For instance, why was noisy replicate padding selected for text conditioning? It seems to make little difference in terms of FID. Also, why was the cosine function used to prevent entanglement with timestep embedding in control conditioning? Could we just use a simply hardcoded timestep for thresholding or just increasing linearly?\n- The number of hyperparameters to tune for optimal performance has increased. Both the control guidance scale and gamma_c for the cosine function need to be adjusted.\n- Some experimental results are not very surprising. For example, in Table 2 (b), it is quite obvious that having similar resolution sampling during training and inference yields better results. And the difference of the number in Table 2 (c) is too small.\n\nQUESTIONS:\n- I am curious about the additional explanation regarding the model design mentioned in the weaknesses.\n- Was there any performance measurement conducted for moving the class embedding position in control conditioning?\n- I'm unsure if comparing performance at specific steps is suitable. It would be better to show a graph or observe the best score after running for a sufficiently long time (e.g., the results in Table 5).\n- In Table 2 (a) for Control-conditioning, what was the LPIPS compared against?\n\nLIMITATIONS:\nThe authors well addressed the limitations on their appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}