On improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models
Authors: Tariq Berrada
Pietro Astolfi
Melissa Hall
Reyhane Askari Hemmat
Yohann Benchetrit
Marton Havasi
Matthew J. Muckley
Karteek Alahari
Adriana Romero-Soriano
Jakob Verbeek
Michal Drozdzal
Venue: NeurIPS 2024 poster (2024)
Keywords: Generative Models
Generative Modeling
Diffusion
Latent diffusion
Computer vision
text-to-image diffusion
Primary area: diffusion_based_models

Abstract: Large-scale training of latent diffusion models (LDMs) has enabled unprecedented quality in image generation. 
However, large-scale end-to-end training of these models is computationally costly, and hence most research  focuses either on finetuning  pretrained models or experiments at smaller scales.
In this work we aim to improve the training efficiency and performance of LDMs with the goal of scaling to larger datasets and higher resolutions.
We focus our study on two points that are critical for good performance and efficient training: 
(i) the mechanisms used for  semantic level (\eg a text prompt, or class name) and low-level (crop size, random flip, \etc) conditioning of the model, and 
(ii) pre-training strategies to transfer representations learned on smaller and lower-resolution datasets to larger ones.
The main contributions of our work are the following: 
we present systematic experimental study of these points, 
we propose a novel conditioning mechanism that disentangles semantic and low-level conditioning, 
we obtain state-of-the-art performance  on CC12M for text-to-image at 512 resolution.

On Improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models
Tariq Berrada
Pietro Astolfi
Melissa Hall
Reyhane Askari-Hemmat
Yohann Benchetrit
Marton Havasi!
Matthew Muckley
Karteek Alahari²
Adriana Romero-Soriano 1,34,5
Jakob Verbeek
Michal Drozdzal
'FARR Meta
Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, France
McGGil University
"Mila, Quebec Al institute
Canada CIFAR chair
Abstract
Large-scale training latent diffusion models (LDMs) has enabled unprecedented quality in image generation. However, the key components of the best performing LDM training recipes are oftentimes not available to the research community, preventing apple-to-apple comparisons and hindering the validation of progress in the field. In this work, we perform an in-depth study of LDM training recipes focusing on the performance of models and their training efficiency. To ensure apple-to-apple comparisons, we re-implement five previously published models with their corresponding recipes. Through our study, we explore the effects of (i) the mechanisms used to condition the generative model on semantic information (e.g., text prompt) and control metadata (e.g., crop size, random flip flag, etc.) on the model performance, and (ii) the transfer of the representations learned on smaller and lower-resolution datasets to larger ones on the training efficiency and model performance. We then propose novel conditioning mechanism that disentangles semantic and control metadata conditionings and sets a new state-of-the-art in class- conditional generation on the ImageNet- lk dataset with FID improvements of 7% on 256 and 8% on 512 resolutions - as well as -ext-to-image generation on the CC1 CC12M dataset with FID improvements of 8% on 256 and 23% on 512 resolution.
1 Introduction
Diffusion models have emerged as powerful class of generative models and demonstrated unprece- dented ability at generating high-quality and realistic images. Their superior performance is evident across spectrum of applications, encompassing image [7, 14, 39, 41] and video synthesis [35], denoising [52], super-resolution [49] and layout-to-image synthesis [51]. The fundamental principle underpinning diffusion models is the iterative denoising of an initial sample from trivial prior distri- bution, that progressively transforms to a sample from the target distribution. The popularity of diffusion models can be attributed to several factors. First, they offer simple yet effective approach for generative modeling, often outperforming traditional approaches such as Generative Adversarial Networks (GANs) [3, 16, 24, 25] and Variational Autoencoders (VAEs) [29, 48] in terms of visual fidelity and sample diversity. Second, diffusion models are generally more stable and less prone to mode collapse compared to GANs, which notoriously difficult stabilize without careful tuning of hyperparameters and training procedures [23, 30].
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Funny realistic picture counting
Husky pupples sleeping under blonket
car goes by an icy road
enjoys meat frozen block during hoe
dancing human skeleton wearing a magician's hat
e. starry night style painting of the ciffel tower
A small cat sictiing beautiful stetue of horsz windowsll
made of bronze
Figure 1: Qualitative examples. Images generated using model trained CC12M at 512 resolution.
Despite the success of diffusion models, training such models at scale remains computationally challenging, leading to a lack of insights on the most effective training strategies. Training recipes of large-scale models often closed (e.g., DALL-E, Imagen, Midjourney), and only few studies have analyzed training dynamics in detail [7, 14, 26, 27]. Moreover, evaluation often involves human studies which are easily biased and hard to replicate [17, 56). Due the high computational costs, the research community mostly focused on the finetuning of large text-to-image models for different downstream tasks [1, 4, 54] and efficient sampling techniques [34, 36, 45]. However, there has been less focus on ablating different mechanisms to condition on user inputs such as text prompts, and strategies to pre-train using datasets of smaller resolution and/or data size. The benefits of conditioning mechanisms are two-fold: allowing users to have better control over the content that being generated, and unlocking training on augmented or lower quality data by for example conditioning on the original image size [39] and other metadata of the data augmentation. Improving pre- neetrraninng strategies, on the other hand, can allow for big cuts in the training cost of diffusion models by significantly reducing the number of iterations necessary for convergence.
Our work aims to disambiguate some of these design choices, and provide set of guidelines that enable the scaling of the training of diffusion models in an efficient and effective manner. Beyond the main architectural choices (e.g.. Unet vs. ViT), we focus on two other important aspects for generative performance and efficiency of training. First, we enhance conditioning by decoupling different conditionings based on their type: control metadata conditioning (e.g., crop size, random flip, etc.), semantic-level conditioning based on class names or text-prompts. In this manner, we disentangle the contribution of each conditioning and avoid undesired interference among them. Second, we optimize the scaling strategy to larger dataset sizes and higher resolution by studying the influence of the initialization of the model with weights from models pre-trained on smaller datasets and resolutions. Here, we propose three improvements needed to seamlessly transition across resolutions: interpolation of the positional embeddings, scaling of the noise schedule, and using a more aggressive data augmentation strategy.
In our experiments we evaluate models at 256 and 512 resolution on ImageNet- Ik and Conceptual Captions (CC12M), and also present results for ImageNet-22k at 256 resolution. We study the following five architectures: Unet/LDM-GE [39], DiT-XL2 w/ LN [38], mDT-v2-XL/2 w/ LN [15]. PixArt-a-XL/2 XL/2, and miDiT--XL/2 (SD3) [14]. We find that among the studied base architectures, mmDiT-XX12 (SD3) performs the best. Our improved conditioning approach further boosts the performance of the best model consistently across metrics, resolutions, and datasets. In particular, we improve the previous to 2.76. For CC12M at 512 resolution, we improve FID of 1 11.24 to 8.64 when using our improved conditioning, while also obtaining a (small) improvement in CLIPscore from 26.01 to 26. 17. See Fig. 1 for qualitative examples of our model trained on CC M2M.
DiT result 3.04 FID on ImageNet- 1 at 512 resolution

---

572
Figure Influence control condi- tions. Images generated using the same latent sample. Top: Model trained with constant weighting of the size condition- ing as used in SDXL [39], introducing un- desirable correlations between image con- tent and size condition. Bottom: Model trained using our cosine weighting of low- level conditioning, disentangling the size condition from image content.
In summary, our contributions are the following:
We present systematic study of five different diffusion architectures, which we train from scratch using face-blurred ImageNet and CC12M datasets at 256 and 512 resolutions.
We introduce conditioning mechanism that disentangles different control conditionings and
semantic-level conditioning, improving generation and avoiding interference between conditions.
To transfer weights from pre-trained models we propose to interpolate positional embeddings, scale the noise schedule, and use stronger data augmentation, leading to improved performance.
We obtain state-of-the-art results 256 and 512 resolution for class-conditional generation on ImageNet- Ik and text-to-image generation and CC12M.
2 Conditioning and pre-training strategies for diffusion models
In this section, we review and analyze the conditioning mechanisms and pre-training strategies used in prior work (see more detailed discussion of related work in App. A), and propose improved approaches based on the analysis.
2.1 Conditioning mechanisms
Background. To control the generated content, diffusion models are usually conditioned on class labels or text prompts. Adaptive layer norm is lightweight solution to condition on class labels, used for both UNets [21, 39, 41] and DiT models [38]. Cross-attention is used to allow more fine-grained conditioning on textual prompts, where particular regions of the sampled image are affected only by part of the prompt, see e.g. [41]. More recently, another attention based conditioning was proposed in SD3 [14] within transformer-base across layers. concatenates the image and text tokens across the sequence dimension, and then performs self-attention operation the combined sequence. Because of the difference between the two modalities, the keys and queries are normalized using RMSNorm [53], which stabilizes training. This enables complex interactions between the two modalities in attention block instead of using both self self-attention and cross- cross-attention blocks.
architecture that evolves both the visual and textual tokens
Moreover, since generative models aim to learn the distribution of the training data, data quality is important when training generative models. Having low quality training samples, such as the ones that are poorly cropped or have unnatural aspect ratios, can result in low quality generations. Previous work tackles this problem by careful data curation and fine-tur tine-tuning on high quality data, see e.g. [7. 9]. However, strictly filtering the training data may deprive the model from large portions of the available data [39], and collecting high-quality data not trivial. Rather than treating them as nuisance factors, SDXL [39] proposes an alternative solution where a UNet-based model is conditioned on parameters corresponding to image size and crop parameters during training. In this manner, the model is aware of these parameters and can account for them during training, while also offering users control over these parameters during inference. These control conditions are transformed and additively combined with the timestep embedding before feeding them to the diffusion model.
Disentangled control conditions. Straightforward implementation of control conditions in DiT may cause interference between the time-step, class-level and control conditions if their corresponding embeddings are additively combined in the adaptive layer norm conditioning, e.g. causing changes in high-level content of the generated image when modifying its resolution, see Fig. 2. To disentangle the different conditions, we propose two modifications. First, we move the class embedding to be fed through the attention layers present in the DiT blocks. Second, to ensure that the control embedding
3

---

does not overpower the timestep embedding when additively combined in the adaptive layer norm, we zero out the control embedding in early denoising steps, and gradually increase its strength.
Control conditions can be used for different types of data augmenta- tions: (i) high-level augmentations ((x) that affect the image compo- sition e.g. flipping, cropping and aspect ratio and (ii) low-level augmentations (d1) that affect low-level details - e.g. image resolu- tion and color. Intuitively, high-level augmentations should impact the image formation process early on, while low-level augmenta- tions should enter the process only once sufficient image details are present. We achieve this by scaling the contribution of the low-level augmentations, ot, to the control embedding using cosine schedule that downweights earlier contributions:
x(() =
cos (x- (1 -))
1.0
0.8
0.6
0.4
0.2
0.0
250
750
1000
limestep.
Comb (oh, Φ1, t) = (σr) + ro(t)- Ei(d)).
(1)
Figure 3: Weighting of lowelevel control conditions. The weight is zeroed out early on when im- age semantics are defined, and in- creased later when adding details.
where the embedding functions Fs,, Fg are made of sinusoidal em- beddings followed by 2-layer MLP with SiLU activation, and where Yo is the cosine schedule illustrated in Fig. 3.
Improved text conditioning. Most commonly used text encoders, like CLIP [40], output constant number of tokens T that are fed to the denoising model (usually T = 77). Consequently, when the prompt has less than T tokens, the remaining positions accessible via cross-attention to the denoising network. To make better use of the conditioning vector, we propose noisy replicate padding mechanism where the padding tokens of the text tokens, thereby pushing the subsequent cross-attention layers to attend to all the tokens in its inputs. As this might lead to redundant token embeddings, we improve the diversity of the feature representation across the sequence dimension, by perturbing the embeddings with additive Gaussian noise with small variance 3ix. To ensure enough diversity in the token embeddings, we scale the additive noise by ((ra) Vm - I, where m is the number of token replications needed for padding, and or (dex) is the per-channel standard deviation in the token embeddings.
filled by zero- padddigd but remain
replaced with copies
Integrating classifier guidance. Classifier-free guidance (CFG) [20] allows for training con- ditional models by combining the output of the uncoditional generation with the output of the conditional generation. Formally, given a latent diffusion model trained to predict the noise e, CFG reads as: ed = xx, (1 X) 6g, where cg is the uncoditional noise prediction, 6g is the noise pre- diction conditioned on the semantic conditioning 8 (e.g., text prompt), and À is the hyper-parameter, known as guidance scale, which regulates the strength of the conditioning. Importantly, during training X is set alternatively to 0 or 1, while at inference time it is arbitrarily changed in order to steer the generation be more or less consistent with the conditioning. In our case, we propose the control conditioning to be an auxiliary guidance term, in order to separately regulate the strength of the conditioning on the control variables and semantic conditioning s at inference time. In particular, we define the guided noise estimate as:
> [B cc,s Ec,s
B) =p.s)] + (1 - A) E0.0.
where B sets the strength of the control guidance, and À sets the strength of the semantic guidance.
2.2 On transferring models pre-trained on different datasets and resolutions
Background. Transfer learning has been a pillar of the deep learning community, enabling generalization different domains and the emergence of foundational models such as DINO [5, 10] and CLIP [40]. Large pre-trained text-to-image diffusion models have also been re-purposed for different tasks, including image compression [4] and spatially-guided image generation [1]. Here, we are interested in understanding to which extent pre-training on other datasets and resolutions can be leveraged to achieve a more efficient training of large text-to-image models. Indeed, training diffusion models directly to generate high resolution images is computationally demanding, therefore, it is common to either couple them with auper-resolution models, see e.g. [44], or fine-tune them with high resolution data, see e.g. [7, 14, 39]. Although most models can directly operate at a higher resolution than the one used for training, fine-tuning is important to adjust the model to the different statistics of high-resolution images. In particular, we find that the different statistics influence the positional embedding of patches, the noise schedule, and the optimal guidance scale. Therefore, we focus on improving the transferability of these components.

---

Positional Embedding, Adapting to higher resolution can be done in different ways. Interpolation scales the most often learnable- embeddings according to the new resolution [2. 47]. Extrapolation simply replicates the embeddings of the original resolution to higher resolutions as illustrated in Fig. 4, resulting mismatch between the positional embeddings and the image features when switching to different resolutions. Most methods that use interpolation of learnable positional embeddings, e.g- [2, 47J, adopt either bicubic or bilinear interpolation to avoid the norm reduction associated with the interpolation. In our case, we take advantage of the fact that our embeddings are sinusoidal and simply adjust the sampling grid to have constants limit under every resolution, see App.
Scaling the noise schedule. At higher resolution, the amount of noise necessary to mask objects at the same rate changes [14, 22]. If we observe spatial patch at low resolution under a
High res, (ostripoisted))
High res. (becorpoiscod)
Low res.
given uncertainty, upscaling the image by a factor creates
g² observations of this patch of the form Vi (6) Iz + ate(0) assuming the value of the patch is constant across the patch. This increase in the number of observations reduces the un- higher
Figure Interpolation and extrapo- lation of positional embeddings.
certainty around the value of that token, resulting
signal-to-noise (SNR) ratio than expected. This issue gets further accentuated when the scheduler does not reach a terminal state with pure noise during training, ie., zero SNR [32], as the mismatch
between the non-zero SNR seen during training and the purely Gaussian initial state of the sampling
phase becomes significant. To resolve this, we scale the noise scheduler in order to recover the same uncertainty for the same timestep.
Proposition When going from a scale of to a scale s', we update the 3 scheduler according to the following rule
g2 āt ā₂' s²2 + a. 2s² 22
(3)
This increases the noise amplitude during intermediate denoising steps as illustrated Fig. A1. The final equation obtained is similar to the one obtained in [14] with the accompanying change of variable 1.
1402.
Pre-training cropping strategies. When pre-training and finetuning at different resolutions, we can either first crop and then resize the crops according the training resolution, directly take differently sized crops from the training images. Using a different resizing during pre-training and finetuning may introduce some distribution shift, while using crops of different sizes may be detrimental to low-resolution training as the model will Iearn the distribution of smaller crops rather than full images, see Fig. 5. We experimentally investigate which strategy is more effective for low-resolution pre- training of high-resolution models.
Global context
Target
resolution
VS.
Figure 5: Low-resolution pre- training. Crop size used for pre- training impacts finetuning.
Guidance scale. We discover that the optimal guidance scale for both FID and CLIPScore varies with the resolution images. In App. D, we present proof revealing that under certain conditions, the optimal guidance scale adheres scaling law with respect to the resolution,
X((s) = 1 + -A A-11 -)).
3 Experimental evaluation
3.1 Experimental setup
Datasets. In our study, we train models on three datasets. To train class-conditional models, we use ImageNet-lk [11], which has 1.3M images spanning 1,000 classes, as well as ImageNet-22k [43], which contains 14.2M images spanning 21,841 classes. Additionally, we train exxt-to-image models using Conceptual 12M (CC12M) [6], which contains 12M images with accompanying manually generated textual descriptions. We pre-process both datasets by blurring all faces. Differently from [7], we the original captions for the CC12M dataset.
Evaluation. For image quality, we evaluate our models using the common FID [19] metric. We follow the standard evaluation protocol on ImageNet to have fair comparison with the relevant
5

---

Table I: Comparison between different model architectures. We compare results reported in the literature (top, reporting available numbers) with our reimplementations existing architectures (middle). results obtained using architectural refinements and impraved training. For 512 resolution, we trained models by fine-tuning models pre pre-trained at 256 resolution. In each column, we bold the best results among those in the first two blocks, and also those in the last row when they equivalent superior. "-' denotes that numbers are unavailable in the original papers. or architectures are incompatible with ext-tt--maae generation our experiments. 'x' indicates diverged runs. **' is used for Esser et [14] pre-trained on CC12M to denote that FID is computed differently and some details about their evaluation are unclcar.
our best
ImageNet Ik
CC12M
256
512
256
512
FiD
FlDmi
FID.a
CLIPcoco
FIDui
FIDeoco
CLIP
Results taken from references
UNet (SDILDM G4) [39]
3.60
17.01
9.62
DIT-X12 w/T.N [38]
2.27
3.04
mnDT-v2-X/22 LN [15]
1.79
PisArt-a-XL/211 [7]
10.65
mmDDt-XXL/2 (SD3) [14]
Our re- www.bnplleenttatinnccomm of exiisinn archihectures
UNer (SDXL)
2.05
4.81
8.53
25.36
12.56
7.26
24.79
DIT-XL/ w/ LN
1.95
2.85
X
inDT-v2-XL/2w. w/
2.51
3.75
PisArt-a-XL/2
2.06
3.05
X
X
X
X
mI/iT (SDS)
1.71
3.02
7.54
24.78
11.24
6.78
26.01
improved architecture training mmDDTTXX1/2 (owrs)
1.59
2.76
6.79
26.60
6.27
6.69
26.17
literature [3, 15, 38, 41]. Specifically, we compute the FID between the full training set and 50k synthetic samples generated using 250 DDIM sampling steps. For image-text alignment, we compute the CLIP [40] score similarly to [7, 14]. We measure conditional diversity, either using class-level or text prompt conditioning, using LPIPS [55]. LPIPS is measured pairwise and averaged among ten generations obtained with the same random seed, prompt, and initial noise, but different size conditioning (we exclude sizes smaller than the target resolution): then we report the average over 10k prompts. addition to ImageNet and CC12M evaluations, we provide FID and CLIPScore on the COCO [33] validation set, which contains approximately 40k images with associated captions. For COCO evaluation [33], we follow the same setting as [14] for computing the CLIP score, using 25 sampling steps and guidance scale of 5.0.
Training. To train our models we use the Adam [28] optimizer, with learning rate of 10-4 and 31, 32 = 0.9, 0.999. When training at 256 x 256 resolution, we use a batch size of 2, 048 images, a constant learning rate of 10 10-4. train our models on two machines with eight A100 GPUs each. In preliminary experiments with the DiT architecture we found that the FID metric on ImageNet- Ik 256 resolution consistently improved with larger batches and learning rate, but that increasing the learning rate by another factor of two led to diverging runs. We report these results in supplementary. When training models at 512 512 resolution, we use the same approach but with batch size of 384 distributed over 16 1100 GPUs. We train our ImageNet- k models for 500k to 1M iterations and for 300k to 500k iterations for CC12M.
Model architectures. We train different diffusion architectures under the same setting to provide a fair comparison between model architectures. Specifically, we re-implement UNet-based architecture following Stable Diffusion XL (SDXL) [39] and several transformer based architectures: vanilla DiT [38], masked DiT (mDiT-v2) [15]. PixArt DiT (PixArt-a) [7]. and multimodal DiT (mmDiT) as in Stable Diffusion 3 [14]. For vanilla DIT, which only supports conditooniii generation, we explore two variants one incorporating the class conditioning within LayerNorm and another one within the attention layer. Also, for text-conditioned models, we use the text encoder and tokenizer of CLIP (ViT-L/14) [40] having a maximum sequence length of T 77. The final models share similar number parameters, e.g. for DiTs we inspect the XL/2 variant [38], for UNet (SDXL) we adopt similar size to the original LDM [41]. Similar to [14], we found the training of DiT with
'ee only implement the base network, without the extra retiner in [39]

---

Table 2: Control conditioning. We study different facets of control conditioning and their impact the model performance. (a-b) We report min on ImageNet- 256 using 250 sampling steps. 120k training iterations.
(a) Influence of the parumetrization. LPIPS computa- tion considers all resolutions (64, 128, 256, 512, 1024 while LPIPS/HR exclude 64 and 128
(b) Size conditioning effect FID at inference.
Size sompling (a. b)
baseline
8.0
512
4.48
4.12
weighting
(I)
IPIP5
LPIPS/IR (1)
~ U[[122, 1024])
5.01
3.80
a, (1112, 1024))
4.51
ze1o
3.29
n~ Duii
3.08
3.04
unif.
3.08
0.210
De00
cor(a
1.0)
3.08
0.23
0.076
(c) Influence of control conditioning. Models trained on CC12M@ 256. (124k iterations)
corla
2.0)
3.09
0.18
0.045
costa
4.0)
3.05
0.13
0.025
8.0)
3.04
0.04
0.009
Crop
flip
FID
CLIP (t)
X
8.43
23. 59
cross-attention to be unstable and had to resort to using RMSNorm [53] to normalize the key and query in the attention layers. We detail the models sizes and computational footprints in Tab. A1.
3.2 Evaluation of model architectures and comparison with the state of the art
In Tab. 1, we report results for models with different architectures trained both 256 and 512 resolutions for ImageNet and CC12M, and compare our results (2nd block of rows) with those reported in the literature, where available (Ist block of rows). Where direct comparison possible, we notice that our re-implementation outperforms the one of existing references. Overall, we found the mmDiT [14] architecture to perform best or second best in all settings compared other alternatives. For this reason, we apply our conditioning improvements on top of this architecture (last row of the table), boosting the results as measured with FID and CLIPScore all settings. Below, we analyse the improvements due to our conditioning mechanisms and pre-training strategies.
3.3 Control conditioning
Scheduling rate of control conditioning. Tab. 2a we consider the effect of controlling the conditioning on low-level augmentations via a cosine schedule for different decay rates a. We compare baselines (first two rows) with constant weighting (as in SDXL [39]) and without control conditioning. We find that our cosine weighting schedule significantly reduces the dependence between size control and image semanties as drastically improves the instance specific LPIPS (0.33 0.04) in comparison to uniform weighting. In terms of FID, observe small gap with the bascline (3.04 vs. 3,08), which increases (3.80 vs. 5.04) when computing FID by randomly sampling the size conditioning in the range [512, 1024], see Tab. 2b. Finally, the improved disentangling between semantics and low-level conditioning is clearly visible in the qualitative samples in Fig. 2.
Crop and random-fiij flip control conditioning. A potential issue of horizontal flip data augmentations that it can create misalignment between the text prompt and corresponding image. For example the prompt "A leddy bear holding busebull but in their right arm" will no longer be accurate when an image flipped showing a teddy bear holding the bat in their left arm. Similarly, cropping images can remove details mentioned in the corresponding caption. In Tab. 2c we evaluate models trained on CC12M@ 256 with and without horizontal flip conditioning, and find that adding this conditioning leads to slight improvements in both FID and CLIP as compared to using only crop conditioing. We depict qualitative comparison in Fig. 6, where we observe that flip conditioning improves prompt-layout consistency.
Inference-time control conditioning of image size. High-level augmentations (0), may affect the image semantics. As a result they influence the learned distribution and modify the generation diversity. For example, aspect ratio conditioning can harm the quality of generated images, when images of particular class or text prompt are unlikely to appear with a given aspect ratio. In Tab. 2b we compare of different image size conditionings for inference. We find that conditioning on the same size distribution encountered during the training of the model yields significant boost in FID
7

---

No Fip cund
Flip cond
Figure 6: Illustration of the impact of flip conditioning. Without the flip conditioning, the model may confuse left- right specifications. Including flipping control condition enables the model properly follow left-right instructions.
1d. picture of road showing a blue car the left and red car the right
12.5
FIDe225
FID@512
HDD
10.0
2
0
3
2
5
0
2
Guidance scale
Guidance scale
Control guidance scale
Figure 7: Guidance scales. Left center: The optimal guidance scale varies with the image resolution. Right: Decoupling the control guidance improves FID, the best reported performance is obtained with 8 1.375.
compared to generating all images with constant size conditioning using uniformly randomly sampled sizes. Note that in all cases images are generated at 256 resolution.
Control conditioning and guidance. To understand how control condition impacts the generation process, we investigate the influ- ence of control guidance 3 (introduced in Sec. 2.1 on FID and report the results Fig. 7. We find that a higher control guidance scale results in improved FID scores. However, note that this im- provement comes at the cost of compute due to the additional control term 6c,a.
Table 3: Text padding. Our noisy replication embedding vs. baseline zero-paddee pro-padding. Models trained on CC12M@ 256.
Pudding
FID
CLIP
Replication text padding. We compare our noisy replication padding to the baseline zero-padding in Tab. 3. We observe that using a replication padding improves both FID and CLIP score, and that adding scaled perturbations further improves the results - -0.35 point improvement CLIP score and 0.4 point improvement FID.
replicate
3em
0
6.93
7.19
26.47
replicate
0.02
6.79
26. 60
replicate
0.05
6.82
26.58
replicate
0.1
7.01
26.47
replicate
0.2
7.02
26.41
3.4 Transferring weights between datasets and resolutions
Dataset shift. We evaluate the effect of pre-training on ImageNet-1k (at 256 resolution) when training the models on CC12M or ImageNet-22k (at 512 resolution) by the time needed to achieve the performance as model trained from scratch. In Tab. 4a, when comparing models trained from scratch to ImageNet-Ik Ik pre-trained models (600k iterations) we observe two benefits: improved training convergence and performance boosts. For CC12M, we find that after only 100k iterations, both FID and CLIP scores improve over the baseline model trained with more than six times the
Table 4: Effeect of pre-training across datasets and res- olutions. Number of (pre-)training iterations given in thousands (k) per Relative improvements FID and
(b) Pre-training 512 resolution models on ImageNet-222 before finetuning on CC12M.
CLIP score given percentage in parenthesis.
FiDasza
FiDecuzM
CLiPcouM
2008
150k
5,80
7.15
25.79
(a) Pre-training models at 256 resolution ImageNet- lk
300k
50k
5.11
7.79
25.30
(c) Influence of pre-training at 256 resolution for 512 resolution models (ImageNet-Ik)
IN22k (375k)
5.80
INK
IN22K, (80K)
5.29
(+3.67%)
(110k)
Model
pre-train
Finetune
FID
CC12M (600))
7.54
24.78
UNet
1000k
6,50
INIR
CCTZM (60K)
7.58 (-0.66%)
25.00 (-111
(100k)
(+3.719)
(33.)
INIK
CCTZM (120K)
7:25 (+3.8590)
25.09 (++3.716)
UNer
750L
250k
4.81 (4:29.199)
mmDIT
350R
150k
3.02 (+21112%)
8

---

Figure 8: Resolution shift. Experiments are conducted on ImageNet-Ik Ik at 512 resolution, FID is reported using 50 DDIM steps with respect the ImageNet-Ik Ik validation set.
Default Scalol
5.75
6.5
Local
FID-0:12
5.50
4.5
6.0
4.0
Defmmit
5.5
5.00
Scaled
Bension (3)
Ierrttonn 00
Derratoon OK)
(a) Influence of positional embed-
(b) Influence noise schedule rescaling for training.
Influence pretraining scale on convergence.
ding rexampling on convergence.
amount of training iterations. For ImageNet-22k,, which is closer in distribution to ImageNet- 1k than CC12M, the gains are even more significant, the finetuned model achieves an FID lower by 0.5 point after only 80k training iterations. In Tab. 4b, we study the relative importance of pre-training vs finetuning when the datasets have dissimilar distributions but similar sample sizes. We fix a training budget in terms of number of training iterations N, we first train our model on ImageNet-22k for K iterations before continuing the training on CC12M for the remaining N K iterations. We see that the model pretrained for 200k iterations and finetuned for 150k performs better than the one spending the bulk of the training during pretraining phase. This validates the importance of domain specific training and demonstrates that the bulk of the gains from the pretrained checkpoint come from the representation learned during earlier stages.
Resolution change. We compare the performance boost obtained from training from scratch at 512 resolution vs. resuming from a 250-resolution trained model. According to our results in Tab. 4c, pretraining on low resolution significantly boosts the performance at higher resolutions, both for UNet and mmDiT, we find that higher resolution finetuning for short periods time outperforms high resolution training from scratch by a large margin (~ 25%). These performance gains might in part be due to the increased batch size when pre-training the 256 resolution model, which allows the model to "see" more images as compared to training from-scratch at 512 resolution.
Positional Embedding. In Fig. 8a, we compare the influence of the adjustment mechanism for the positional embedding. We find that our grid resampling approach outperforms the default extrapolation approach, resulting in 0.2 point difference in FID) after 130k training iterations.
Scaling the noise schedule. We conducted an evaluation to ascertain the influence of the noise schedule by refining our mmDiT model post its low resolution training and report the results in Fig. 8b. Remarkably, the application of the rectified schedule, for 40k iterations, resulted in an improvement of 0.7 FID points demonstrating its efficacy higher resolutions.
Pre-training cropping strategies. During pretraining, the model sees objects that are smaller than what it sees during fine tuning, see Fig. 5. We aim to reduce this discrepancy by adopting more agressive cropping during the pretraining phase. We experiment with three cropping ratios for training: 0.9 1 (global), 0.4 0.6 (local), 0.4 (mix). We report the results in Fig. 8c. On ImageNet K @ 256, the pretraining FID scores are 2.36, 245.55 and 2.21 for the local, global and mixed strategies respectively. During training 512 resolution, observe that the global and mix outperfor cropping strategies both outperform the local strategy. However, as reported in Fig. 8c, the local strategy provides benefits higher resolutions. Overall, training with the global strategy performs the best at 256 resolution but lags behind for higher resolution adaptation. While local cropping underperforms at lower resolutions, because it does not see any images in their totality, outperforms the other methods at higher resolutions an improvement of almost 0.2 FID points is consistent after the first 50k training steps at higher resolution.
Conclusion
In this paper, we explored various approaches enhance the conditional training of diffusion models. Our empirical findings revealed significant improvements in the quality and control over generated images when incorporating different coditioning mechanisms. Moreover, conducted

---

comprehensive study on the transferability of these models across diverse datasets and resolutions. Our results demonstrated that leveraging pretrained representations is powerful tool to improve the model performance while also cutting down the training costs. Furthermore, we provided valuable insights into efficiently scaling up the training process for these models without compromising performance. By adapting the schedulers and positional embeddings when scaling up the resolution, we achieved substantial reductions in training time while boosting the quality the generated images. Additional experiments unveil the expected gains from different transfer strategies, making it easier for researchers to explore new ideas and applications in this domain. Appendix B we discuss societal impact and limitations of our work.
10

---

References
[1] Omri Avrahami. Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski. Ohad Fried, and Xi Yin. Spatext: Spatio-textual representation for controllable image generation. In CVPR. 2023.
[2] Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon Kornblith, Xiaohua Zhai, Matthias Minderer, Michael Tschannen, Ibrahim Alabdulmohsin, and Filip Pavetic. FlexiViT: One model for all patch sizes. In CVPR, 2023.
[3] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019.
[4] Marlène Careil, Matthew Muckley, Jakob Verbeek, and Stéphane Lathuilière. Towards image compres- sion with perfect realism ultra-low bitrates. In ICLR, 2024.
[5] Mathilde Caron. Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging propertics in self-supervised vision transformers. In ICCV, 2021.
[6] Soravit Changpinyo, Piyush Sharma, Nan Ding. and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
[7] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang. James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. PixArt-a: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In ICLR, 2024.
[8] Jaemin Cho, Abhay Zala, and Mohit Bansal. DALL-Eval: Probing the reasoning skills and social biases of text-to-image generation models. ICCV, 2023.
[9] Xiaoliang Dai, Ji Hou. Chih- Ma. Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang. Abhimanyu Dubey, Matthew Yu, Abhishek Kadian, Filip Radenovic, Dhruv Mahajan, Kunpeng Li, Yue Zhao, Vladan Petrovic, Mitesh Kumar Singh, Simran Motwani, Yi Wen, Yiwen Song, Roshan Sumbaly, Vignesh Ramanathan, Zijian He, Peter Vajda, and Devi Parikh. Emu: Enhancing image generation models using photogenic needles in haystack. arXiv preprint, 1309.15807, 2023.
[10] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In ICLR, 2024.
[11] Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: large-scale hierarchical image database. In ICCV, 2009.
[12] Prafulla Dhariwal and Alex Nichol. Diffusion models beat GANs on image synthesis. In NeurlPS, 2021.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
[14] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel. Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In /CML, 2024.
[15] Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is a strong image synthesizer. In ICCV, 2023.
[16] Ian Goodfellow, Jcan Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
[17] Melissa Hall, Samuel J. Bell, Candace Ross, Adina Williams, Michal Drozdzal, and Adriana Romero Soriano. Towards geographic inclusion in the evaluation of text-to-image models. In FAceT, 2024.
[18] Melissa Hall, Candace Ross, Adina Williams, Nicolas Carion, Michal Drozdzal, and Adriana Romero- Soriano. DIG in: Evaluating disparities in image generations with indicators for geographic diversity. TMLR, 2024.
[19] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to local Nash equilibrium. In NewrIPS, 2017.

---

[20] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS Workshop Deep Generative Models and Downstream Applications, 2021.
[21] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[22] Emiel Hoogehoom, Jonathan Heek, and Tim Salimans. Simple diffusion: End-to-end diffusion for high resolution images. In JCML, 2023.
[23] Minguk Kang, Jun- Yan Zhu, Richard Zhang, Jacsik Park, Eli Shechtman, Sylvain Paris, and Tacsung Park. Scaling up GANs for text-to- ext-to-imaae synthesis. In CVPR, 2023.
[24] Tero Karras, Samuli Laine, Miika Aittlla, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In CVPR, 2020.
[25] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In NeurlPS, 2021.
[26] Tero Karras, Miika Aittla, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In NeurlPS, 2022.
[27] Tero Karras, Janne Hellsten, Miika Aittala, Timo Aila, Jaakko Lehtinen, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. In CVPR, 2024.
[28] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In JCLR, 2015.
[29] Diederik P. Kingma and Max Welling, Auto-encoding variational Bayes. In ICLR, 2014.
[30] Kwonjoon Lee, Huiwen Chang, Lu Jiang, Han Zhang, Zhuowen Tu, and Ce Liu. ViTGAN: Training GANs with vision transformers. In ICLR, 2022.
[31] Mark Levy, Bruno Di Giorgi, Floris Weers, Angelos Katharopoulos, and Tom Nickson. Controllable music production with diffusion models and guidance gradients. arXiv preprint, 2311 1.000613, 2023.
[32] Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In WACV, 2024.
[33] T.-Y. Lin, M. Maire, S. Belongie, L. Bourdey, R. Girshick, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. Zimick. Microsoft COCO: common objects in context. In ECCV, 2014.
[34] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In /CLR, 2023.
[35] Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan. Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang. Hanchi Sun, Jianfeng Gao, Lifang He, and I Lichao Sun. Sora: A review on background, technology, limitations, and opportunities of large vision models. arXiv preprint, 2402.17177, 2024.
[36] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In NeurlPS, 2022.
[37] Alexandra Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite. Stable bias: Analyz- ing societal representations in diffusion models. In NeurIPS, 2023.
[38] William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
[39] Dustin Podell, Zion English, Kyle Laccy, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In ICLR, 2024.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and 1lya Sutskever. Learning transferable visual models from natural language supervision. In ICMI. 2021.
[41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[42] Olaf Ronneberger. Philipp Fischer, and Thomas Brox U-Net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Asisted Intervention, 2015.
12

---

[43] Olga Russakovsky. Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115())211-252. 2015.
[44] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes. Tim Salimans, Jonathan Ho, David J Flect, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In NeurlPS, 2022.
[45] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[46] Yang Song, Jascha Diek-Dicstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In /CLR, 2021.
[47] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
[48] A. Vahdat and J. Kautz. NVAE: deep hierarchical variational autoencoder. In NeurlPS, 2020.
[49] Chanyue Wu, Dong Wang, Yunpeng Bai, Hanyu Mao, Ying Li, Qiang Shen. HSR-Diff: Hyperspectral image super-resolution via conditional diffusion models. In ICCV, 2023.
[50] Tong Wu, Zhihao Fan, Xiao Liu, Yeyun Gong, Yelong Shen, Jiao, Hai-Tao Zheng, Juntao Li, Zhongyu Wei, Jian Guo, Nan Duan, and Weizhu Chen. AR-Diffusion: Auto-regressive diffusion model for text generation. In NeurIPS, 2023.
[51] Han Xue, Zhiwu Huang, Qianru Sun, Li Song, and Wenjun Zhang. Freestyle layout-to-image synthesis. In CVPR, 2023.
[52] Cheng Yang. Lijing Liang, and Zhixun Su. Real-world denoising via diffusion model. arXiv preprint, 2023.
[53] Biao Zhang and Rico Sennrich. Root Mean Square Layer Normalization. In NeurlPS, 2019.
[54] Lymin Zhang, Anyi Rao, and Mancesh Agrawala, Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
[55] Richard Zhang, Phillip Isola, Alexei Efros, Shechtman, and Oliver Wang. effectiveness of deep features as perceptual metric. In CVPR, 2018.
unreasonable
[56] Sharon Zhou, Mitchell L. Gordon, Ranjay Krishna, Austin Narcomey, Durim Morina, and Michael Bernstein. HYPE: benchmark for Human eYe Perceptual Evaluation of generative models. In NeurlPS, 2019.
13

---

Appendix
Table of Contents
A Related work
14
B Societal impact and limitations
15
Implementation details
15
D Derivations
E Additional results
19
F NeurIPS Paper Checklist
22
A Related work
Diffusion Models. Diffusion models have gained significant attention in recent years due to their ability model complex stochastic processes and generate high-quality samples. These models have been successfully applied to a wide range of applications, including image generation [7, 21, 41], video generation [35], music generation [31], and text generation [50]. One of the earliest diffusion models was proposed in [21]. which introduced denoising diffusion probabilistic models (DDPMs) for image generation. This work demonstrated that DDPMs can generate high-quality images that competitive with state-of-the-art generative models such as GANs [16]. Following this work, several variants of DDPMs were proposed, including score-based diffusion models [46], conditional diffusion models [12], and implicit diffusion models [45]. Overall, diffusion models have shown promising results various applications due their ability to model complex stochastic processes and generate high-qualit samples [7, 14, 39, 41]. Despite their effectiveness, diffusion models also have some limitations, including the need for large amount of training data and the required computational resources. Some works [26, 27] have studied and analysed the training dynamics of diffusion models, most of this work considers the pixel-based models and small-scale settings with limited image resolution and dataset size. In our work we focus on the more scalable class of latent diffusion models [41], and consider image resolutions up 512 pixels, and 14M training images.
Model architectures. Early work on diffusion models adopted the widely popular UNet arcchitec- ture [39, 41]. The UNet an encoder-decoder architecture where the encoder made of residual blocks that produce progressively smaller feature maps, and the decoder progressively upsamples the feature maps and refines them using skip connections with the encoder [42]. For diffusion, UNets are also equipped with cross attention blocks for cross-modality conditioning and adaptive layer normalization that conditions the outputs of the model on the timestep [41]. More recently, vision transformer architectures [13] were shown to scale more favourably than UNets for diffusion models with the DiT architecture [38]. Numerous improvements have been proposed to the DiT in order to have more efficient and stable training, see e.g. [7, 14, 15]. In order to reduce the computational complexity of the model and train at larger scales, windowed attention has been proposed [7]. Latent masking during training has been proposed to encourage better semantic understanding of inputs in [15]. Others improved the conditioning mechanism by evolving the text tokens through the layers of the transformer and replacing the usual tross-attention used for text conditioning with variant that concatenates the tokens from both the image and text modalities [14].
Large scale diffusion training. Latent diffusion models [41] unlocked training diffusion models at higher resolutions and from more data by learning the diffusion model in the reduced latent space of (pre-trained) image autoencoder rather than directly in the image pixel space. Follow-up work has proposed improved scaling of the architecture and data [39]. More recently, attention-baser architectures [7, 14, 15] have been adapted for large scale training, showing more improvements
14

---

Figure A1: Noise schedule scaling law. At higher resolu- tions, keeping the same uncer- tainty means spending more time at higher noise levels, thereby counteracting the un- certainty reduction from the in- crease in the observations for the same patch.
5
0.8
0.6
Sw4
0.6
5-1
V1
0.4
5=3
0.0
0
200
400
600
800
1000
200
400
600
800
1000
timestep
timestep
by scaling the model size further and achieving state-of-the-art performance datasets such as ImageNet-Ik, I.. Efficiency gains were also obtained by [7] by transferring ImageNet pre-trained models larger datasets.
B Societal impact and limitations
Our research investigates the training of generative image models, which are widely employed to generate content for creative education and accessibility purposes. However, together with these beneficial applications, these models are usually associated with privacy concerns (e.g.. deepfake generation) and misinformation spread. our paper, we deepen the understanding of the training dynamics of these modes, providing the community with additional knowledge that can be lever- aged for safety mitigation. Moreover, we promote safe and transparee/hreprouuibbl research by employing only publicly available data, which we further mitigate by blurring human faces.
Our work is mostly focused training dynamics, and facilitate reproducibility, used publicly available datasets and benchmarks, without applying any data filtering. We chose datasets relying on the filteringmmitigationn done by their respective original authors. In general, before releasing models like the ones described in our paper to the public, we recommend conducting proper evaluation of models trained using our method for bias, fairness and discrimination risks. For example, geographical disparities due to stereotypical generations could be revealed with methods described by Hall et al. [18], and social biases regarding gender and ethnicity could be captured with methods from Luccioni et al. [37] and Cho et al. [8].
While our study provides valuable insights into control conditioning and the effectiveness of repre- sentation transfer in diffusion models, there are several limitations that should he acknowledged, (i) There cases where these improvements for the text embeddings can become less pertinent if the model trained exclusively with long prompts. (ii) While low resolution pretraining with local crops on ImageNet resulted in better FID 512 resolution (see Table 5c), it might not be necessary if pretraining on much larger datasets (e.g. > >10M samples, which we did not experiment in this work). Similarly, flip conditioning only pertinent if the training dataset contains position sensitive information (left vs. right in captions, or rendered text in images), otherwise this condition will not provide any useful signal. (iii) We did not investigate the impact of data quality on training dynamics, which could have implications for the generalizability of our findings to datasets with varying levels of quality and diversity. (iv) As our analysis primarily focused on control conditioning, other forms of conditioning such as timestep and prompt conditioning were not explored in as much depth. Further research is needed to determine the extent to which these conditionings interact with control conditioning and how that impacts the quality of the models. (v) Our work did not include studies on other parts that are involved in the training and sampling of diffusion models, such the different sampling mechanisms and training paradigms. This could potentially yield additional gains performance uncover new insights about the state-space of diffusion models.
be less pronounced. For example, noisy replicates
C Implementation details
Noise schedule scaling. In Fig. Al we depict the rescaling of the noise for higher resolutions, following Eq. (3).
15

---

Table A1: Computational comparison between diffferent models. We compute FLOPs for different resolutions the parameter count. FLOPS are computed with batch size of 1.
UNer-SDXL
PitAm-XXL2
mmDit-XL XL/2
Alt-Di-XL/2
mdT-XL/2 (INIK)
Dii-XL/2
FLOPS@256 (G)
189.78
314.68
397.06
407.88
CC12M
(G)
1,
(text)
Params. (M)
$64.31
610.12
791.64
940.49
FLOPS@ 256 512 (G)
275.14
237.93
237
237.1
ImageNet
(daas-ccodl
390
1,200
1. 050
1,050
1. 140
1. 0,0
Params, (M)
679.09
792.44
748.07
679.09
Positional embedding rescaling. illustrated in App. C, rescaling the positional embedding can be integrated in two simple lines of code by changing the grid sampling be based on reducing the stepsize the grid instead of extending its limits.
grid_h = (aaige(gridiig grid_w = arange (grid_gg
size grid_size scale
grid_h linspace (0, base_size, grid_size)
grid_w linspace (0, base_size, grid_size)
Pseudocode 1: Rectified grid sampling for positional embeddings.
Computational costs. In Tab. Al we compare the model size and computational costs of the different architectures studied in the main paper. All model architectures are based on the original implemetations, but transposed our codebase. Mainly, we use bfloat16 mixed precision and memory-efficient attention from PyTorch.
Experimental details. To ensure efficient training of our models, we benchmark the most widely used hyprpaarameters and report the results of these experiments, which consist of the choice of the optimizer and its hyperparameters, the learning rate and the batch size. We then transpose the other experiments. For FID evaluation, resolution and 2.0 for resolution 512. For evaluation on ImageNet-22k,, we compute the FID score between 50k generated images and subset of 200k images from the training set.
optimal settings
a guidance scale of 1.5 for 256
Training paradigm. We use the EDM [26] abstraction to train our models for epsilon prediction following DDPM paradigm. Differently from [15, 38], do not use learned sigmas but follow a standard schedule. Specifically, we use quadratic beta schedule with 3 = 0.00085 and Gond 0.012. In DDPM [21], noising step is formulated as follows, with zg being the data sample and E [0, 7' timestep:
Ft Va + VI-acc
I).
while in EDM, the cumulative alpha products are converted to corresponding signal-to-noise ratio (SNR) proportions, the noising process then reformulated as:
+ 0,E
(6)
V1
and the loss is weighted by the inverse SNR
Scaling the training. A recurrent question when training deep networks the coupling between the learning rate and the batch size. When multiplying the batch size by factor % some works recommend scaling the learning rate by the square root of y, while others scale the learning rate by
nn. functional. oo scaleeeddddddddppoddeeee..
attention. html
16

---

Table A2: Influence of learning rate and batch size convergence. Training performed on ImageNet- @oggeNet-lk@2222 @k@256. Results are reported on model without EMA after 70k training steps. FID is computed using 250 sampling steps w.r.t the training set of ImageNet-
Learning
10
5.10
10-4
10-5
10-6
Batch
Size
w/ UNet
57.56
59.50
104.73
256. X: diverged. For almost all learning rates, the optimal batch size is the highest possible. The best performance is obtained when using the highest learning rate that does not diverge with biggest batch size pos- sible.
512
27.26
19.83
37.98
80.26
1024
X
18.01
19.25
30.00
62.82
2048
14.63
14.73
24.24
61.89
4096
X
49.21
9.26
15.71
w/ DNT
59.22 09.26
104.55
256
39.53
37.08
86,51
512
X
22.48
23.61
83.6
1021
76.15
2048
10.19
12.36
82.62
the factor itself. In the following experiment with training class-conditionas DiT model with different batch sizes and learning rates and report results after the same number of iterations.
From Tab. A2 we observe an improved performance by increasing the batch size and the learning rate in every learning rate setting. If the learning rate too high the training diverges.
Influence of momentum. We conduct grid search over the momentum parameters of Adam optimizer, similar to previous sections, train UNet model fo 70k steps and compute FID with respect to the validation set of ImageNetIk using 50 DDIM steps. From Fig. A2, we can see that the default pytorch values (31 0.9, 32 = 0.999) are sub-optimal, resulting an FID of 26.43 while the best performance is obtained when setting (31 0.9, 32 0.95) improves FID by 4.78 points in our experimental setting.
0.5
78.31
25.05
21.79
21.7
0.59
25
27.43
25
0.56
0,000
Figure A2: Influence of momentum on training dynamics of the UNet. We evalu- ate ImageNetlk@e steps after training the models for 70k steps. The FID is reported w.r.t. the validation set of ImageNet.
Setting the optimal guidance scale.
FID using 50 sampling
Proposition 2. The optimal guidance scale À scales with the upsampling factor according the law (ss) 1 1+ s (X 1).
This is verified in Fig. where the X'ss = 1+ (1.5 1.0) = 2.0 which is the optimal guidance scale at 512 resolution according the figure.
Text padding mechanism. In order to train at large scale, most commonly used text encoders output a constant number of tokens T that fed to the denoising model (usually T 777 Consequently, when the prompt has less than T words, the remaining tokens are padding tokens that do not contain useful information, but can still con- tribute in the cross-altention, the question of whether better use can be made of padding text tokens to improve training performance and efficiency. One common mitigation involves using recaptioning meth- ods that provide longer captions. However, this creates an inconsistency between training and sampling as users are more likely to provide shorter prompts. Thus, to make bet- ter use of the conditioning vector, we explore alternative padding mechanisms for the text encoder. We explore a "replicaee" padding mechanism where the padding tokens are replaced with copies of the text tokens, thereby push- ing the subsequent cross-and coss-attention layers to attend all
Token gradient
Number tokens
Figure A3. This raises
-2
0
10
20
30
40
50
60
70
Token index
Figure A3: Contribution of Padding to- kens. For short prompts, a large number of text tokens do not contain useful information, but may still contribute to the cross-atiention, illustrated by the non-zero gradients w.r.t. tokens after ones coding the prompt (in- dicated by the dashed vertical line).
17

---

Tise nokene
Image solens
Insge: Lakoo
Imrge oihens
Tea Volecte
Tex mikee
Tea
(a) Zero padding.
(b) Replicate padding.
(c) Noisy replicate padding.
Figure A4: Illustration of the attention matrix under different padding mechanisms. With the zero padding mechanism, a significant part of attention can used on "dead" padding tokens, potentially de-focusing from relevant information. Using replicate padding instead results redundant information. Noisy replicate padding increases the diversity in the text token representations and therefore acts as regularizer fostering the model be robust to local variations in the latent space of the conditioning, e.g., akin to data augmentation.
the tokens in inputs. To improve the diversity of the feature representation across the sequence dimension, perturb the embeddings with additive Gaussian noise with small variance Bix. For shorter prompts with high number of repeats m, we scale the additive noise by vm account for the reduction in posterior uncertainty induced by these repetitions:
Au dua vm azh (Pra): Exss
with 6x (),
where deh is the standard deviation of the feature text embeddings over the feature dimension. See Figure A4 for an illustration comparing zero-padding. replication padding, and our noisy replication padding.
D Derivations
Derivation for Eq. (2)
Proof. The formula can be obtained by iteratively applying the guidance across the conditions.
A. dec + (1 - A)E0
(8)
x.3 X(Bec,e + (1 3)c,,) + (1 A)Ea.
(9)
Derivation for Eq. (4)
Proof.
Assuming that the unconditional prediction ep is distributed around the conditional distribution according to a normal law egle, A((E,, 8²2).
εx xe + (1 A)(c de), e~,
(10)
Ex + (1 -)de
(11)
Var(c>) = (1 2)
(12)
After upsampling by a scale factor of s, the same low resolution patch has s observations, hence the variance is decreased by s?.
Var(Ex) (1 //2
(13)
Hence by equalizing the discrepancy between the conditional and unconditional predictions at low and high resolutions we obtain:
Var(ex)==1- Var(c() = (1 11-x)888/5²== (1 2)282
(14)
= () 1)
(15)
18

---

Figure Qualitative examples. Sample from mmDiT-XL/2 trained our method on ImageNet 512 resolution. Samples are generated with 50 DDIM steps and guidance scale of 5.
Derivation for Eq. (3)
Proof. At timestep the noisy observation is obtained as
(D + oxt), e~ N~,
(16)
+
Hence zo can be estimated using the following formula:
To / 1 of ofE
(17)
The statistics the estimate of zy are as follows.
E(xo) 11 + B(xr)) Var(zo) = (1 + Var(z.) + o
(18)
Consequently, if we already have Te, we have an estimate of no with an error bound given by:
Var(#o - v v1+02x;) = of
(19)
At higher resolutions, we have x² corresponding observations for the same patch such that the error with respect to the estimate becomes:
Var
Do
Σ
11+
-o? o
(20)
2² i=1
Hence, if we want keep the same uncertainty with respect to the low resolution patches, the following equality needs to be verified:
(t, = ot''
= 01/8 = o//s'
(21)
1 1 --
1 1 1-
(22)
αt
av
22 at
ōf 22 + (s² -22
(23)
E
Additional results
Qualitative results.
We provide additional qualitative examples ImageNet- 1k in Fig. A5.
19

---

Conditioning
FiD
FiDezuu
CLiPeceo
Resolution transfer
FiDeres
CLiPcoco
Baslliee
1.85
mmll7
1.81
Baseline (train from seratch)
11,24
24.23
cond.
1.76
Low res. pretraining
7.25
25.00
Cuntrol coud separation & schedailig
1.71
7:54
25.88
Local view pretraining
6.89
25.57
Disentongled control guidonce
1.50
7:32
26.13
Positional emb. rescuppling
6.73
25.70
Flip cond.
7:19
26.25
Noise wheddule rescaling
6.27
25.91
Notsed replicaic paidingg for tax condinining
6.79
26.60
Table A3: Effects of our changes. We summarize the improvements obtained by each change proposed in the
paper. left: Changes relevant to conditioning mechanisms right: Changes relevant to representation transfer across image rexolutions.
SD-XL [39]
SI35 [14]
Sumpling
Sciedalk Mscw
TT [1 tje The c 20 1.0
Process at
(2t o.d) +
+ o.d) /
Target
Loss scaling
o
Tissestep sampling
(((D)TT) Are X)aa
te( TT
Guidance mechanism
Ade
AlB 1e ca.. (1 Al 50.0
Network and conditioning
Denoiser architecture
UNet Cinss-dtteation
MMDI Augmested self-atteation
minDT++ Augmented self-atteation RMSNoom
Conditioning meclaniism
Amention Pre-Norm
X
RM15Narm N/A
Contnll conditianing
/
Non-sem Mon-semmnntic cond. schedule 72
1.0
1.0
[I carls (1 12))
conditioning
Canditon disentinglement
tokzn padding
zeto
Zero
noisy replicate
High resolution adaptation
Noise schedule resampling
Positional embedding
x
interpolated
resamppeed
Low res. pietrain. crop scale
[0.9. 1.0) M(a)
(0.S, 1.0] A(+)
(0:4 0.6)
Guidance scale X(a))
a' (M(a) 1)
Optimization
Optimizer
Adam
AdamW
AdamW
Momentmms
0.500
Schedule
Buie-
0.00085.
0.012
B
0.00085, 3.... 0.012 N/A
Auin
0.00085, Bass
Weight docay
0.0
Table A4: Comparison with previous paradigms. We provide a comparative table with previous works,
notably SD-XL [39] and SD3 [14].
Summary of findings. In Table A3 we summarize the improvements w.r.t. the DiT baseline obtained by the changes the model architecture and training. In Table A4 we compare our model architecture and training recipe to that of SDXL and SD3. In Table A5, we provide a synopsis of the research questions addressed in our study alongside a respective recommendation based on our findings.
Effectiveness of the power cosine schedule We experiment with different function profiles for controlling the conditioning on low-level augmentations. Specifically, we compare the power-cosine profile with linear and piecewise constant profile. While the linear schedule manages an acceptable performance in terms of reducing LPIPS (although still higher than the power-cosine profile), it still achieves a higher FID than the configurations with the cosine schedule. For the piecewise constant profiles, they achieve a higher LPIPS while also having higher FID. In conclusion, the proposed
Table A5: Summary of the findings from our experiments in the form of a Q&A.
How to condition on text?
cross-dornin self attentioo [14].
Use control condirioning?
Conditioning
Weight control conditioning with a power cosine schedule?
Use separate guidance scale for control conditions?
Use roplicate padding with goussian perturbations for embeddings?
Pretrain on smaller datasets (ImageNetikk)
Distribution transfer
Pretrain datusets of the same scale but with different distributions Pretrain until convergence?
Diminishing returns after a while
Rescale the noise scheduler Resample the grid coordinates the positional embeddings
Resolution transfer
Pretrain smaller resolution Use more agressive cropping when pretraining at smaller resolutions?
20

---

Table A6: Comparison of the power cosine schedule with other schedules. We report results for linear and step function schedules.
Ini:
weighting
FID (L)
(PIPS(4)
LPIPSHR (4)
zeno wnif.
3.29
2070.
2000.
conte
1.0)
3.08
0.23
0.076
2en0.
2090.
costa 4.0)
3.05
0.13
0.025
2000.
cos(a
8.0)
2000.
linear
3.13
0.07
0.041
2090.
8(1, < 1.0))
3.15
0.09
0.046
2670.
8(a, S 2.0)
8(d, S 6.0)
3.09
0.26
0,170
power-cosine profile outperforms these simpler schedules in both FID and LPIPS, improving image quality while better removing the unwanted distribution shift induced from choosing different samples during training.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly set the claims made in the paper and match theoretical and experimental results, which are addressed in Section 3.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We include discussion of limitations in Appendix B.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Proofs are provided Appendix D and contain cross-references to the relevant parts in the main paper.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We include details required for result reproducibility Section 3.1. These include details about datasets used for training, evaluation metrics, training parameters, and model architectures.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
23

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: While not be able include code, do provide specific and thorough detailing of our training and evaluation methods in Section 3.1 to enable experimental reproduction.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co public/ Bttlic/guieess/odeeee Submii ss ssionPolccy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips. hetuubic/guide//ooo/oo/oo Sudee/ooeSSuiisssee wwriiisssoooooigoo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/ettails
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
Details corresponding to training and testing are included in Section 3.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to the computational cost of training and performing inference with generative models, we do not include error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Information about compute resources needed for experiment reproduction is included in Section 3.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http:///erriisccceee http://ppp.....pooc/ccctttttteee
Question:
Answer: [Yes]
Justification: This research conforms to the Code of Ethics, including mitigation of potential harms caused by the research process, considerations of social impact, and taking steps for impact mitigation.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss societal impacts and steps taken mitigate potential harms in Appendix B.
25

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
We do not release models nor data in this work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Use of existing assets is discussed, including which version, and cited through- out the paper. No assets are relcased.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
26

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paper thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not include crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
27

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "VTkx9G1ST0", "reviewer": "Reviewer_cMfV", "summary": "This paper studies how to effectively condition image size and crop information, as done by Stable Diffusion XL, and how to implement an effective coarse-to-fine training strategy. For control conditioning, it is designed to be less entangled than traditional methods, resulting in better performance for the same backbone model. The analysis for model transfer in coarse-to-fine training includes noise scheduling and positioning embedding. Additionally, the paper compares backbone models by re-implementing and comparing UNet, DiT, and mDiT. For analysis, ImageNet1K/22K data was used for class2image, and CC12M data for text2image.", "questions": "- I am curious about the additional explanation regarding the model design mentioned in the weaknesses.\n- Was there any performance measurement conducted for moving the class embedding position in control conditioning?\n- I'm unsure if comparing performance at specific steps is suitable. It would be better to show a graph or observe the best score after running for a sufficiently long time (e.g., the results in Table 5).\n- In Table 2 (a) for Control-conditioning, what was the LPIPS compared against?", "limitations": "The authors well addressed the limitations on their appendix.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I agree on the necessity of an apple-to-apple comparison of current backbone models. Comparing various backbones under identical experimental conditions and conducting ablation experiments across several attributes will serve as valuable preliminary research for future model training. To be specific, it was good to see the performance improvements brought by the control conditioning design, which is easy to overlook. Additionally, this paper improves the performance of stable diffusion-3 and effectively demonstrates controlled experiments for hyperparameters that were previously unorganized among engineers.", "weaknesses": "- The explanation and analysis of the chosen designs are lacking for me. For instance, why was noisy replicate padding selected for text conditioning? It seems to make little difference in terms of FID. Also, why was the cosine function used to prevent entanglement with timestep embedding in control conditioning? Could we just use a simply hardcoded timestep for thresholding or just increasing linearly?\n- The number of hyperparameters to tune for optimal performance has increased. Both the control guidance scale and gamma_c for the cosine function need to be adjusted.\n- Some experimental results are not very surprising. For example, in Table 2 (b), it is quite obvious that having similar resolution sampling during training and inference yields better results. And the difference of the number in Table 2 (c) is too small.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the thorough feedback. In the following we address it: \n- **Design choices**  \n  - **Noisy replicate padding has minor impact.** In addition to slightly improving the FID and CLIPScore (0.54 and 0.58, respectively), the added noisy replicate can serve as a regularizer with no added cost during training. The intuition is that it disincentivizes the model from ignoring the later copies in the attention, as it would happen with zero-padding (see Figure R3 in the attached pdf). We added Figure R2 to depict this intuition. Moreover, the local variations caused by the added noise acts as a sort of data augmentation in the text encoder’s embedding space, intuitively robustifying the model to small variations in the semantics.   \n  - **Cosine scheduler for conditioning disentaglement**. We chose the power cosine schedule because it provides an easy-to-tune function, in the form of one hyperparameter $\\\\alpha$ (decay rate of the conditioning), and generates a smooth transition, which we intuited to be more inductive for learning than using a step function. To prove our choice and clarify the reviewer's doubt, we have run a comparison against step and linear schedulers, see Table R3. We observe superior performance provided by the cosine scheduler as per FID, LPIPS, and LPIPS at high resolution (LPIPS/HR). We recall that LPIPS is measured across 10 generations obtained with the same random seed, prompt, and initial noise, but different size conditioning (for HR, we exclude sizes smaller than the target resolution); then we report the average over 10K prompts.    \n- **Increased number of hyper-parameters**. While we introduced some new hyperparameters, the model shows improved performance for a significant range of their values. For example, for all $\\\\alpha$ tested in the control scheduler, we obtained lower LPIPS (see Table 2a) and practically removed semantic drift across different size conditionings (see Figure 2 and Table 2b). The control scale, $\\\\gamma\\_c$, also shows consistent performance across datasets for the same values (range \\[0, 2\\], see Figure 7 right). Moreover, when possible, e.g., for the guidance scale, we included theoretical analysis to guide the transfer of these hyperparameters across resolutions (see Eq.4 and its derivation in Appendix F), thereby restricting the search/tuning space.  \n- **Trivial results in Table 2b**. We agree that it is expected to obtain the best FID (distribution overlapping) when conditioning on the original image size. However, this result serves as a reference point to understand how much the choice of the size conditioning affects the distribution of generated images. Indeed, the purpose of Table 2b is to highlight the larger FID increase (i.e., distribution shifting) in the baseline vs ours. The results obtained with our control conditioning suggest that better FID numbers can be obtained without the need of careful tuning of the size conditions for every prompt. We will clarify the reading of Table 2b when updating our paper.  \n- **Class embedding position in control conditioning**. Moving class embeddings into control conditions results in the vanilla DiT architecture, while moving the control conditioning to the cross-attention in mmDiT results in an architecture similar to Hatamizadeh et al (2023), which we found to underperform with respect to the baseline. \\[Hatamizadeh, Ali, et al. \"Diffit: Diffusion vision transformers for image generation.\" (2023)\\]  \n- **Comparing curves instead of specific steps for Table 5\\.** We agree with the reviewer that comparing curves provides richer observations. For this reason, we plotted the experiments presented in Table 5 evaluating runs at intermediate checkpoints. We show the plot in Figure R1. The trend of the curves depicted in all the three panels, (a-c), validate the improved performance of our contributions, which also showcase faster convergence rates. Unfortunately, due to limited time and compute resources, we were not able to run the comparison in panel (b) and (c) until 200K iterations, but we believe these plots could be enough to solve the reviewer's concerns regarding Table 5\\. The runs are still ongoing, and we will update the plots for the final version of the manuscript.  \n- **Clarify LPIPS in Table 2a.** We describe the LPIPS computation at L215 of the main paper, plus we clarify it here above: “*We recall that LPIPS is measured across 10 generations obtained with the same random seed, prompt, and initial noise, but different size conditioning (for HR, we exclude sizes smaller than the target resolution); then we report the average over 10K prompts*.” We will add this extra clarification also in the main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FWOv258nqm", "reviewer": "Reviewer_rR4N", "summary": "This paper's contributions can be divided into three main points:\n\n1. Proposing better methods for metadata/semantic level conditioning:\n   - Instead of using AdaLN, class information is fed into the model through attention.\n   - Adjusting the strength of meta conditions related to low-level augmentation according to diffusion timesteps.\n   - Suggesting a method of copying text tokens and applying Gaussian perturbation instead of zero padding.\n   - Proposing additional guidance for the control variable \\(c\\) (Eq. 2).\n\n2. Proposing a method of transfer learning from low resolution to high resolution:\n   - Using interpolation for positional embedding.\n   - Adjusting the noise schedule according to SNR changes with resolution.\n   - Experimentally showing that resizing is better than cropping for creating low-resolution images.\n   - Proposing a strategy for adjusting guidance strength according to resolution.\n\n3. Reproducing experiments:\n   - Demonstrating experimentally that mmDiT has the best performance among various models proposed so far.\n\nThey demonstrated the validity of each method through extensive experiments on the Imagenet and CC12M 256x256 (512x512 for finetuning) datasets.", "questions": "Minor suggestion:\n- If adding the flip condition is part of the method, it would be better to introduce it in the method section rather than having it first appear in the experiment section at L264.", "limitations": "The author has appropriately described the limitations of this work.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Extensive experiments:\n   - The performance comparison through re-implementation of all structures is a contribution in itself.\n   - Performance improvements were shown based on FID for all proposed methods. Such an extensive ablation study is rare.", "weaknesses": "1. Unorganized Experiment Result Presentation\n   - One of the key results of this study is Imagenet-1K FID 1.59, CC12M FID 6.79, and 8.64.\n   - However, it is difficult to understand how each element claimed by this study contributes to these results.\n   - The best presentation would be to examine how removing each configuration affects the Imagenet-1K FID 1.59.\n   - Practically, it seems challenging to run 1M iterations for all configurations. It would have been better to show how changing each configuration affects the best config based on 120k iterations.\n   - For example, it would be useful to have something like Table 1 from the EDM 2 paper [1].\n   - This is not just for easier reader comprehension but is also important for understanding how much the proposed disentangled control conditions contributed to the Imagenet-1K FID improvement from 1.71 to 1.59.\n   - If these concerns are well resolved, I am considering raising the score.\n\n\n2. Method Presentation\n   - This study has a very complex design space.\n   - However, it is currently difficult to know exactly what the current settings are and how they differ from past settings. Could you include something like Table 1 from the EDM 1 paper [2]?\n\n[1]: Analyzing and Improving the Training Dynamics of Diffusion Models, https://arxiv.org/abs/2312.02696\n[2]: Elucidating the Design Space of Diffusion-Based Generative Models, https://arxiv.org/abs/2206.00364", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. In the following we address it: \n- **Unorganized Experiment Result Presentation.** We appreciate the reviewer's suggestion to adopt a table as in Karras et al (2023) to enhance the presentation of our contributions. We have added Table R1 in the attached pdf, which reports improvements in $256^2$ pre-training (panel (a)) and resolution transfer setting (panel (b)). Notably, rows 2-5 in panel (a) address concerns regarding control conditioning contribution, showing FID improvements from 1.81 to 1.59 on ImageNet and 7.54 to 7.32 on CC12M, along with a slight CLIPScore boost. Given the added value of this table, we consider integrating it into the main paper.  Finally, note that we do not report improvements in *dataset shift* because a similar table is already present in the main paper, see Table 4\\.  \n- **Method presentation / complex design space.** We do agree with the reviewer that the design space might be complex, but we believe that this observation applies to diffusion models as they have many parameters that are left unexplored most of the time. Though, we acknowledge that a table like the one in Karras et al (2022) would clarify many doubts on our configuration and contributions. Hence, we added Table R2 to the attached pdf and to the appendix of the paper. The table compares our model vs. SD3 and SDXL, taking into account the configuration of: Sampling, Network and conditioning, High resolution adaptation, and Optimization.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jdkQEHNpyJ", "reviewer": "Reviewer_JPTZ", "summary": "The paper explores the training efficiencies and model performance of latent diffusion models (LDMs) by reimplementing and analyzing various previously published models. The study introduces a novel conditioning mechanism that separates semantic and control metadata inputs, significantly improving class-conditional and text-to-image generation performances on benchmark datasets.", "questions": "(1) What are the specific scenarios or cases where your proposed methods did not perform well? \n(2) Have you tested your proposed methods on generative tasks beyond image generation, such as text generation or audio synthesis?", "limitations": "(1) The paper mentions instability issues with certain models but does not provide a detailed analysis.\n(2) Insufficient ablation experiments.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) The introduction of a novel conditioning mechanism that disentangles semantic and control metadata conditioning is a good advancement. It addresses the interference issue in generative performance, which is a common challenge in diffusion models.\n(2) The paper achieves state-of-the-art results in class-conditional generation on ImageNet-1k and text-to-image generation on CC12M.\n(3) The paper extends theoretical understanding by proposing modifications to the noise scheduling and positional embeddings that are informed by a principled approach.", "weaknesses": "(1) The paper is primarily evaluated on large-scale datasets like ImageNet and CC12M, but its effectiveness in low-data or small-scale datasets is not demonstrated.\n(2) The paper briefly mentions instability issues with certain models (e.g., DiT with cross-attention), but does not provide detailed insights or solutions to these problems. \n(3) Although the paper presents some ablation studies, they are not comprehensive. More detailed ablation studies isolating the impact of each proposed innovation are recommended.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's feedback and are glad they found our paper a _good advancement_ to the state of the art. In the following, we respond to the points raised:\n- **Lack of small-scale experiments.** The objective of our work is to increase understanding of the design choices in state-of-the-art diffusion models, and propose improvements upon those. However, SOTA diffusion models require sufficiently large scale pre-training to be highly performant. Under this premise, we believe that testing on smaller scales is not necessarily conducive to our objective, since convergence behavior can differ significantly at smaller scales, e.g., due to overfitting of the data-hungry transformers. Also, in the literature, fundamental diffusion studies have been already tested at small-scale, e.g., Nichol et al (2021) and Karras et al (2022 and 2023). \\[Nichol, Alexander Quinn, and Prafulla Dhariwal. \"Improved denoising diffusion probabilistic models.\" (2021); Karras, Tero, et al. \"Elucidating the design space of diffusion-based generative models.\" (2022); Karras, Tero, et al. \"Analyzing and improving the training dynamics of diffusion models.\" (2023)\\]  \n- **Instability issues with certain models not detailed (e.g., DiT with cross-attention)**. As mentioned in our paper at L84-90 and L239-241, these instabilities that we report have been already studied and solved in the SD3 paper. They come from the keys and queries growing larger in amplitude, causing instabilities later on during training. They were solved by using RMSnorm on the keys and queries before every attention operation.  \n- **Ablations are not comprehensive.** We are not sure about which ablations the reviewer would expect us to report. However, for an overall ablation we have added Table R1 in the attached pdf. The added table should better isolate the contribution of each of our improvements. If this would not be enough for the reviewer, we invite them to specify which components they would like to see ablated.   \n- **Limitation of our work**. We refer the reviewer to the limitations section in Appendix C of our paper. In addition, we expand on the limitation section, focusing on failure cases of our method:  \n  - We found our method to provide improvements consistently across the settings tested. However, there are cases where these improvements can be less pronounced. For example, noisy replicates for the text embeddings can become less pertinent if the model is trained exclusively with long prompts.  \n  - Also, while low resolution pretraining with local crops on ImageNet resulted in better FID at 512 resolution (see Table 5c), it might not be necessary if pretraining on much larger datasets (\\~300M samples or more which we did not experiment in this work).  \n  - Similarly, flip conditioning is only pertinent if the training dataset contains position sensitive information (left vs. right in captions, or rendered text in images), otherwise this condition will not provide any useful signal.  \n- **Extension to audio or text generation.** While we find possible extensions to other data modalities intriguing as future works, we believe they are out of the scope of the current work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for thoughtful reviews and precious feedback. We are glad they appreciated our work under many aspects. In particular, reviewers **cMfV**, **rR4N** highlighted our contribution of a much needed apple-to-apple comparison among SOTA models, while **JPTZ** and **cMfV** emphasized contributions on the model, e.g., decoupled control conditioning, noise schedule and positional embedding. Remarkably, all reviewers (**rR4N**, **JPTZ**, **cMfV**) underlined the extensive empirical analysis showcasing SOTA performance, with a special mention for in-depth ablations by **cMfV** and **rR4N**. 

We also received several feedback for improving our work. We did a considerable amount of work to address all of them and in the following we report the outstanding ones: 

- **rR4N** noticed a complex design study of our work suggesting a simplified presentation following Karras et al (2022 and 2023). We find this suggestion extremely useful and beneficial to improve the clarity of our work. Indeed, we report our model contributions in a simplified way via Tables structured as in Karras et al, see attached pdf. Table R1 shows the performance improvement given by our architectural and training choices, in the setting of $256^2$ pre-training (a) and resolution transfer (b), while Table R2 explicits our model configuration vs. SD-XL and SD3.   
- **cMfV** asked for explanations regarding the choice of: (i) noisy replicate padding for text conditioning, and (ii) cosine scheduler in the decoupling of control conditioning. In the attached pdf, Figure R2 and R3 explains the use of noisy replicate padding with an intuitive visualization, and Table R3 validates the performance advantage, in terms of both FID and LPIPS, of our power cosine schedule vs. linear or step schedulers. Moreover, the reviewer raised concerns about comparing strategies for multi-resolution transfer at a specific iteration (as in Table 5 of our paper) rather than plotting their convergence curve. In Figure R1, we plotted the curves for the experiments of Table 5, which ablate the multi-resolution transfer strategies: (a) Pretraining scale, (b) Positional embedding resampling, and (c) noise schedule rescaling.  
- **JPTZ** requested a discussion of our model’s limitations, and ablations to isolate our contributions. For the limitations, we refer to Appendix C of our paper, where we already discuss them. For the additional ablations, as no specific component was mentioned, we refer to the newly produced Table R1, which shows the improvement of each of our contributions, in addition to already reported ablations of *control conditioning*, *text padding*, *transfer from lower to higher resolutions*, *transfer from lower to higher resolutions*, in Tables 2, 3, 4, 5 of the main paper, respectively.

We believe that, by integrating the provided feedback, our contributions will become easier to digest and our design choices better justified, making the whole paper stronger.

Meta-review: This paper proposes an effective pre-training method for diffusion models.

Intial scores were one borderline reject, one borderline acceptance, and one weak acceptance.
Reviewers pointed out some concerns related to experiments and their analysis while repsecting its task importance and effective methods.

During the rebuttal period, the authors successuffly addressed the issues that the reviewers raised, and thus all reviewers have consensus to accept this paper with 6, 6, and 5.

In addtion, ethics reviewer clarified no ethical issue in this paper.

AC also agree with the reviewer's opinions, so recommends accepting this paper.
Decision: Accept (poster)