CADS: Unleashing the Diversity of Diffusion Models through Condition-Annealed Sampling
Authors: Seyedmorteza Sadat
Jakob Buhmann
Derek Bradley
Otmar Hilliges
Romann M. Weber
Venue: ICLR 2024 spotlight (2024)
Keywords: diffusion models
diversity
generative models
Primary area: generative models
TL;DR: We propose a sampling method that increases the diversity of diffusion models without sacrificing image quality.

Abstract: While conditional diffusion models are known to have good coverage of the data distribution, they still face limitations in output diversity, particularly when sampled with a high classifier-free guidance scale for optimal image quality or when trained on small datasets. We attribute this problem to the role of the conditioning signal in inference and offer an improved sampling strategy for diffusion models that can increase generation diversity, especially at high guidance scales, with minimal loss of sample quality. Our sampling strategy anneals the conditioning signal by adding scheduled, monotonically decreasing Gaussian noise to the conditioning vector during inference to balance diversity and condition alignment. Our Condition-Annealed Diffusion Sampler (CADS) can be used with any pretrained model and sampling algorithm, and we show that it boosts the diversity of diffusion models in various conditional generation tasks. Further, using an existing pretrained diffusion model, CADS achieves a new state-of-the-art FID of 1.70 and 2.31 for class-conditional ImageNet generation  at 256$\times$256 and 512$\times$512 respectively.

Published as a conference paper at ICLR 2024
CADS: UNLEASHING THE DIVERSITY OF DIFFUSION MODELS THROUGH CONDITION- AADDITION-ANNEALDD ANNEALED SAMPLING
Seyedmorteza Sadat', Jakob Buhmann', Derek Bradley', Otmar Hilliges', Romann M. Weber
'ETH Zürich, 'DisneyResearch Studios
(seyedmorteza sadat, otmar. hi 1 (elgggeseiiiii ethz. ch [jakob. buhmann,, derek bradley, romann. weber r (di wwwrreearrcccccooomm com
ABSTRACT
While conditional diffusion models are known to have good coverage of the data distribution, they still face limitations output diversity, particularly when sampled with a high classifier-fres free guidance scale for optimal image quality when trained on small datasets. We attribute this problem to the role of the conditioning signal in inference and offer improved sampling strategy for diffusion models that increase generation diversity, especially at high guidance scales, with minimal loss of sample quality. Our sampling strategy anneals the conditioning signal by adding scheduled, monotonically decreasing Gaussian noise to the conditioning vector during inference balance diversity and condition alignment. Our Condition- Annealed Diffusion Sampler (CADS) can be used with any pretrained model and sampling algorithm, and we show that boosts the diversity of diffusion models in various conditional generation tasks. Further, using an existing pretrained diffusion model, CADS achieves a new sfate-of-the-art FID of 1.70 and 2.31 for class-conditional ImageNet generation at 256 256 and 512x512 respectively.
DDPM
CADS (Ours)
DDPM
CADS (Ours)
DDPM
CADS (Ours)
DDPM
CADS (Ours)
Figure 1: Sampling with classifier-free guidance at high guidance scales using standard methods such as DDPM (Ho al., 2020) improves image quality but the cost of diversity. leading sampled images that look similar in composition. We introduce CADS, sampling technique that significantly increases the diversity of generations while retaining high output quality.

---

Published as a conference paper at ICLR 2024
1 INTRODUCTION
Generative modeling aims to accurately capture the characteristics of a target data distribution from unstructured imputs, such as images. To be effective, the model must produce diverse results, ensuring comprehensive data distribution coverage, while also generating high-quality samples, Recent advances in generative modeling have elevated denoising diffusion probabilistic models (DDPMs) (Sohl-Dickstein et al., 2015; Ho et al., 2020) to the state of the art in both conditional and unconditional tasks (Dhariwal & Nichol, 2021). Diffusion models stand out not only for their output quality but also for their stability during training. in contrast to Generative Adversarial Networks (GANs) (Goodfellow et al., 2014). They also exhibit superior data distribution coverage, which has been attributed to their likelihood-buue despite considerable progress in refining the architecture and sampling techniques of diffussion models (Dhariwal & Nichol, 2021; Peebles& & Xie, 2022; Karras et al., 2022; Hoogeboom et al., 2023; Gao et al., 2023), there has been limited focus on their output diversity.
training objective (Nichol & Dhariwal, 2021). However,
This paper serves as comprehensive investigation of the diversity and distribution coverage of conditional diffusion models. We show that conditional diffusion models still suffer from low output diversity when the sampling process uses high classifier-free guidance (CFG) (Ho & Salimans, 2022) or when the underlying diffusion model is trained on a small dataset. One solution would be to reduce the weight of the classifier- free guidance or to train the diffusion model on larger datasets. However, sampling with low classifier-free guidance degrades image quality (Ho & Salimans, 2022; Dhariwal & Nichol, 2021), and collecting a larger dataset may not be feasible in all domains. Even for diffusion model trained on billions of images, e.g., Stable Diffusion (Rombach et al., 2022), the model may still generate outputs with little variability in response certain conditions (Figure 4). Thus, merely expanding the dataset does not provide a complete solution to the issue of limited diversity.
Our primary contribution lies in establishing connection between the conditioning signal and the emergence of low-diversity outputs. We contend that this issue predominantly arises during inference, as a significant majority of samples converge toward the stronger modes within the learned distribution, irrespective of the initial seed. We introduce a simple yet effective technique, termed the Condition-Annealee Diffusion Sampler (CADS), to amplify the diversity of diffusion models. During inference, the conditioning signal is perturbed using additive Gaussian noise combined with an annealing strategy. This strategy introduces significant corruption the conditioning at the outset of sampling and then gradually reduces the noise to zero by the end. Intuitively, this breaks the statistical dependence on the conditioning signal during early inference, allowing more influence from the data distribution as whole, and restores that dependence during late inference. As result, our method can diversify generated samples and simultancously respect the alignment between the input condition and the generated image.
The principal benefit of CADS is that it does not require any retraining of the underlying diffusion model and can be integrated into all diffusion samplers. Also, its computational overhead is minimal, involving only an additive operation. Through extensive experiments, we show that CADS resolves long-standing trade-off between the diversity and quality of the output in conditional diffusion models. Moreover, we demonstrate that CADS outperforms the standard DDPM sampling in several conditional generation tasks and sets a new state-of-the-art FID on class-conditional ImageNet (Russakovsky et al., 2015) generation at both 256x256 and 512x512 resolutions while utilizing higher guidance values. Figure 1 showcases the effectiveness of CADS compared to DDPM on the class-conditional ImageNet generation task.
RELATED WORK
Score-based diffusion models (Song& & Ermon, 2019; Song et al., 2021b; Sohl-Dickstein et al., 2015; Ho et al., 2020) learn the data distribution by reversing a forward destruction process that incrementally transforms the data into Gaussian noise. These models quickly surpassed the fidelity and diversity of previous generative modeling schemes (Nichol & Dhariwal, 2021; Dhariwal & Nichol, 2021), 1), achieving state-of-the-an art results across domains including unconditional image generation (Dhariwal & Nichol, 2021; Karras et al., 2022), text-to-image generation (Ramesh et al., 2022; Saharia et al., 2022b; Balaji et al., 2022; Rombach et al., 2022), image-to-image translation (Saharia
2

---

Published as a conference paper at ICLR 2024
Input
(a) DeepFashion model
(b) SHIIQ model
DeepFashion CADS
Figure 2: Low diversity issue in the pose-to-image generation task. (a) The model trained on DeepFashion generates strongly similar outputs. (b) Training on the larger SHHQ dataset only partially solves the issue. (c) Sampling with CADS significantly reduces output similarity.
et al., 2022a; Liu et al., 2023), motion synthesis (Tevet et al., 2023; Tseng et al., 2023), and audio generation (Chen et al., 2021; Kong et al., 2021; Huang et al., 2023).
Building upon the DDPM model (Ho et al., 2020), numerous advancements have been proposed, including architectural refinements (Ho et al., 2020; Hoogeboom et al., 2023; Peebles & Xie, 2022) and improved training techniques (Nichol & Dhariwal, 2021; Karras et al., 2022: Song et al., 2021b; Salimans & Ho, 2022; Rombach et al., 2022). Moreover, different guidance mechanisms (Dhariwal & Nichol, 2021; Ho & Salimans, 2022; Hong et al., 2022; Kim et al., 2023) provided a means to balance sample quality and diversity, reminiscent of truncation tricks in GANs (Brock et al., 2019).
The standard DDPM model employs 1000 sampling steps and present computational challenges in various applications. As a result, there is an increasing interest in optimizing the sampling efficiency of diffusion models (Song et al., 2021a; Karras et al., 2022; Liu et al., 2022b; Lu al., 2022a; Salimans & Ho, 2022). These techniques aim to achieve comparable output quality in fewer steps, though they do not necessarily address the diversity issue highlighted in this paper.
Somepalli et al. (2023a) discovered that Stable Diffusion (Rombach et al., 2022) may blindly replicate its training data and offered mitigation strategies in a concurrent work (Somepalli et al., 2023b). In contrast, our work addresses the broader issue of reducing the similarity among generated outputs under given condition, even if they are not direct copies of the training data.
Sehwag et al. (2022) presented method to sample from low-density regions of the data distribution using a pretrained diffusion model and likelihood- based hardness score, while our method aims to remain close to the data distribution without specifically targeting low-density regions. Our approach also compatible with latent diffusion models, unlike the pixel-space focus of Sehwag et al. (2022).
Finally, adding Gaussian noise to the input of diffusion model is common in cascaded models to avoid the domain gap between generated data from previous stages and real downsampled data (Ho et al., 2021). However, such models require training with noisy inputs, and the noise addition only happens once at the beginning of the inference as opposed to our scheduled approach.
In summary, work builds on top of recent developments in diffusion models with direct focus on diversity and can be applied to any architecture and sampling algorithm. Additional background on diffusion models is given in Appendix A for completeness.
3 DIVERSITY IN DIFFUSION MODELS
In this paper, diversity defined as the ability of a model to generate varied outputs for a fixed condition when the initial random sample changes. In contrast, models lacking in diversity often generate similar outputs regardless of the random input and tend to focus on a narrower subset of the data distribution.
We trace the low diversity issue in diffusion models to two main factors. First, diffusion models trained on large datasets such as ImageNet (Russakovsky et al., 2015) often require high classifier- free guidance scale for optimal output quality, but sampling with a high guidance scale is known to reduce diversity (Ho& & Salimans, 2022; Murphy, 2023), as shown in Figure 1. Secondly, conditional diffusion models trained on smaller datasets, such as pose-to-image -o-e-age generation on DeepFashion (Liu al., 2016) (=x 10k samples) tend have limited variation, shown in Figure 2a, even with low
3

---

Published as a conference paper at ICLR 2024
CFG scale. In both scenarios, the model establishes a near one-to-one mapping from the conditioning signal to the generated images, thereby yielding limited diversity for a given condition.
One potential solution might involve reducing the guidance scale or training the model on larger datasets, like SHHQ (Fu et al., 2022) (+e 40k samples) for the pose-to-image task. However, decreasing the guidance scale compromises image quality (Ho & Salimans, 2022; Dhariwal & Nichol, 2021), and training on larger datasets only partially mitigates the issue, as demonstrated in Figure 2b.
We conjecture that the low- low-diversity issue is due to highly peaked conditional distribution, particu- larly at higher guidance scales, which leads the majority of sampled images toward certain modes. One way to address this issue is by smoothing the conditional distribution with decreasing Gaussian noise, which breaks and then gradually restores the statistical dependency on the conditioning signal during inference. Next, we introduce novel sampling technique that integrates this intuition into the reverse process of diffusion models. The effectiveness of the method is shown in Figure 2c.
3.1 CONDITION-ANNEALEE DIFFUSION SAMPLER (CADS)
The core principle of CADS is that by annealing the conditioning signal during inference, the model processes different input at each step, leading to more diverse outputs. To maintain the essential correspondence between the condition and the output, CADS employs an annealing strategy that reduces the amount of corruption the inference progresses, causing it to vanish during the final steps of inference. More specifically, inspired by the forward process of diffusion models, we corrupt given condition y via
ÿ V>(1g -
where s determines the initial noise scale, (t) is the annealing schedule, and n ~ NO I). This modification can be readily integrated into any sampler, and significantly diversifies the generations, as demonstrated in Section 4. A discussion on further design choices in CADS follows below.
Annealing schedule function
We propose piecewise linear function for 7() given by
t T1
(()
g-tt
π < T2
12 72
for user-defined thresholds T2, E [0, 1]. Since diffusion models operate backward in time from t = to = 0 during inference, the annealing function ensures high corruption of at early steps and corruption for < T1. In Section 4, we show empirically that this choice of (t) works well in a variety of scenarios.
Rescaling the conditioning signal Adding noise the condition changes the mean and standard deviation of the conditioning vector. In most experiments, we rescale the conditioning vector back toward its prior mean and standard deviation. Specifically, for a clean condition vector y with (scalar) mean and standard deviation Min and Oin. we compute the final corrupted condition ÿ according to
mean(j)
Presalled
Jin + Aiin
(3)
std(g)
inal
v resculed + (1 4))9
for mixing factor 1/r E [0, 1]. This rescaling scheme prevents divergence, especially for high noise scales s, but slightly reduces diversity of the outputs. Therefore, one can trade more stable sampling with more diverse generations by changing the mixing factor th. Section 4.2 contains ablation study on the effect of rescaling and the mixing factor vi.
3.2 INTUITION BEIJIND CONDITION ANNEALING
Consider the task of sampling from conditional diffusion model, where we anneal the condition
according to Equation (1) at each inference step. Using Bayes' rule, the conditional probability at time t can be expressed (z P. Pr( 21 (z.)) p()), and consequently, Vz. log (x ÿ) V2, log (ū) 2,) + V.. (z,). When is close to 1, ((t) xe 0, resulting in ÿ ss sn. This indicates that early in the sampling process, ÿ is independent of the current noisy sample 21, leading
4

---

Published as a conference paper at ICLR 2024
to Vx, log (1 21) O. Hence, the diffusion model initially only follows the unconditional score 2z log (z and ignores the condition. As we reduce the noise, the influence the conditional term increases. This progression ensures more exploration of the space in the early stages and results in high-quality samples with improved diversity. More theoretical insights into condition annealing detailed in Appendices B and C.
3.3 DYNAMIC CLASSIFIER- GUIDANCE
The above analysis motivates us to consider another algorithm to enhance the diversity of diffusion models, in which we directly modulate the guidance weight to initially underweight the contribu- tion of the conditional term. We refer to this method as Dynamic Classifier-Fre FFeee Guidance. More specifically,, in Dynamic CFG, we adjust the guidance weight according to #crG y(t) WCrG. forcing the model to rely more on the unconditional score Vz, log pe (zz) early in inference and gradually move toward standard CFG the end. Section 4.1 details comparison between Dynamic CFG and CADS. We demonstrate that while Dynamic CFG enhances diversity relative to DDPM, it does not match the performance of CADS. We believe that the additional stochasticity in CADS results in more diverse generations and different sampling dynamics than Dynamic CFG.
EXPERIMENTS AND RESULTS
In this section, we rigorously evaluate the performance of CADS on various conditional diffusion models and demonstrate that CADS boosts output diversity without compromising quality.
Setup We consider four conditional generation tasks: class-conditional generation on Ima- geNet (Russakovsky et al., 2015) with DiT-XL/2 (Peebles & Xie, 2022), pose-to-image generation on DeepFashion (Liu et al., 2016) and SHHQ (Fu et al., 2022), identity-conditionee face generation with ID33PM (Kansy al., 2023), and text-to-image generation with Stable Diffusion (Rombach al., 2022). We use pretrained networks for all tasks except the pose-to-image generation, in which we train two diffusion models from scratch. DDPM (Ho al., 2020) is used as the base sampler.
Sample quality metrics We use Fréchet Inception Distance (FID) (Heusel et al., 2017) as the primary metric for capturing both quality and diversity due to its alignment with human judgement. Since FID is sensitive small implementation details (Parmar al., 2022), we adopt the evaluation script of ADM (Dhariwal & Nichol, 2021) to ensure a fair comparison with prior work. For com- pleteness, we also report Inception Score (IS) (Salimans et al., 2016) and Precision (Kynkäänniemi et al., 2019). However, it should be noted that IS and Precision cannot accurately evaluate models with diverse outputs since a model producing high-quality but non-diverse samples could artificially achieve high IS and Precision (as shown in Appendix F).
Diversity metries In addition to FID, we use Recall (Kynkääännemm et al., 2019) as the main metric for measuring diversity and distribution coverage. Furthermore, define two additional similarity metrics. Given a set input conditions, we first compute the pairwise cosine similarity matrix K, among generated images with the same condition, using SSCD (Pizzi et al., 2022) the pretrained feature extractor. The results are then aggregated for different conditions using two methods: the Mean Similarity Score (MSS), which is simple average the similarity matrix Ky, and the Vendi Score (Friedman & Dieng, 2022), which is based on the Von Neumann entropy of Ky.
Implementation details For using CADS, we add noise to the class embeddings in DIT-XL/2, face-ID embeddings in ID3PM, and text embeddings in Stable Diffusion. For the pose-to-image generation models, we directly add noise to the pose image. To have a fair comparison among different methods, we the exact same random seeds for initializing each sampler with and without condition annealing. More implementation details can be found in Appendix G.
4.1
MAIN RESULTS
The following sections describe our main findings. Further analyses are provided in Appendix D.
Qualitative results We showcase generated examples using condition annealing alongside the standard DDPM outputs in Figures 1, 3 and 4. These results indicate that CADS increases the
5

---

Published as a conference paper at ICLR 2024
Input
DDPM
CADS (Ours)
Input
DDPM
CADS (Ours)
(a) Pose-to-image generation
DDPM
CADS (Ours)
DDPM
CADS (Ours)
Input ID
Input ID)
(b)
face synthesis
Figure 3: Comparison between DDPM and CADS on two conditional generation tasks. (a) pose- to-image generation based on DeepFashion, and (b) conditioned face synthesis using the ID3PM model (Kansy et al., 2023). In both cases, CADS enhances the diversity DDPM while maintaining high sample quality.
DDPM
CADS (Ours)
DDPM
CADS (Ours)
Prompt: An astronaut on the Moon
Prompt: A grayscale tiger
Two different results from Stable Diffusion v2.1 (Rombach et al., 2022) sampled with DDPM and CADS. CADS enhances the diversity of Stable Diffusion for prompts that yield highly similar images with DDPM.
Figure 4:
diversity of the outputs while maintaining high image quality across different tasks. When sampling without annealing and using high guidance scale, the generated images realistic but typically lack diversity, regardless the initial random seed. This issue is especially evident the DeepFashion model depicted in Figure 3a, where there is an extreme similarity among the generated samples. Interestingly, even the Stable Diffusion model, which usually produces varied samples and is trained on billions of images, occasionally suffer from low output diversity for specific prompts. As

---

Published as a conference paper at ICLR 2024
Table Quantitative comparison between samples generated with DDPM and CADS for fixed high guidance scale. CADS consistently improves the diversity of the outputs across different tasks as reflected improved FID, recall, and similarity scores.
Dataset
Sampler
FID
Precision
Recall f
MSS
Vendi Score
DDPM
16.36
0.90
0.02
0.80
1.04
DeepFashion (wera 4)
CADS (Ours)
7.73
0.77
0.48
0.30
2.31
DDPM
17.93
0.74
0.17
0.61
1.22
SHHQ (WOr)
CADS (Ours)
10.37
0.65
0.43
0.44
1.65
DDPM
20.83
0.92
0.32
0.19
5.07
ImageNet 256 (werG 5)
CADS (Ours)
9.47
0.82
0.62
0.08
7.98
DDPM
23.10
0.82
0.28
0.20
4.88
ImageNet 512 (WkrG 5)
CADS (Ours)
9.81
0.82
0.52
0.09
7.75
DDPM
16.33
0.65
0.26
0.44
1.71
ID3PM (WCra
CADS (Ours)
11.86
0.67
0.31
0.34
2.44
DDPM
49.48
0.67
0.25
0.21
4.80
Stable Diffusion (wxro
9)
CADS (Ours)
44.16
0.63
0.37
0.15
6.41
Table 2: Benchmark for caadiiconditinnal generation ImageNet 256 256 and 512x512. Sampling with CADS improves the FID) of DiT-X1//2 to the state-of-the at both resolutions while using higher guidance value and without any retraining of the underlying diffusion model.
ImageNet 256x
ImageNet 512x512
Model
FID |
1
Precision
Recall
FID 4
Precision
Recall f
BigGAN-deepp (Brock et al., 2019)
6.95
171.40
0.87
0.28
8.43
177.90
0.88
0.29
StyleGAN-XL (Sauer et al., 2022)
2.30
265.12
0.78
0.53
267.75
0.77
0.52
ADM-G, ADM-U (Dhariwal & Nichol, 2021)
3.94
215.84
0.83
0.53
3.85
221.72
0.84
0.53
LDM-4G (weFF 1.5) Rombach al. (2022)
3.60
247.67
0.87
0.48
RIN-NoiseSchecuule (Chen, 2023)
186.20
216.00
SimpleDiflusion (Hoogehoom
2023)
2.44
256.30
3.02
248.70
VDM++ (Kingma & Gao, 2023)
2.12
267.70
2.65
278.10
DIT-G++ (Kim et al., 2023)
1.83
281.53
0.78
0.64
MDT-G (Gao et al., 2023)
283.01
0.81
0.61
DIT-XL/2-G (MCCG = 1.5) (Peebles & Xie, 2022)
2.27
278.24
0.83
0.57
3.04
240.82
0.84
0.54
DiT-XL/2 with CADS (weFF 2)
1.70
268.77
0.77
0.64
2.53
219.08
0.80
0.61
DIT-XI/2 CADS (WCCF 2.25)
288.10 10
0.78
0.64
233.03
0.60
DiT-XL/2 with CADS (WCFC 2.5)
1.93
297.96
0.78
0.63
2.31
239.56
0.80
0.61
demonstrated Figure 4, CADS successfully addresses this problem. More visual results are provided in Appendix H.
Quantitative evaluation Table quantitatively compares CADS with DDPM across various tasks for fixed high guidance scale. We observe that CADS significantly enhances the diversity of samples, leading to consistent improvements in FID, Recall, and similarity scores for all tasks. This enhanced performance comes without a sigmificant drop in Precision. indicating that CADS increases generation diversity while maintaining high-quality outputs. As expected, the benefits of CADS is less pronounced in Stable Diffusion since the model already capable of diverse generations.
State-of-the-art ImageNet generation By employing higher guidance values while maintaining diversity, CADS achieves a new state-of-thh-art FID of 1.70 for class-conditional generation on ImageNet x56x illustrated in Table 2. Remarkably, CADS surpasses the previous best FID of MDT (Gao et al., 2023) solely through improved sampling and without the need for retraining the diffusion model. Our approach also new otate-of-the-art FID of 2.31 for class-conditional generation on ImageNet 512x512, reinforcing the effectiveness of CADS.
CADS vs guidance scale Classifier-fre guidance has been demonstrated to enhance the quality of generated images at the cost of diminished diversity, particularly higher guidance scales. In this experiment, we illustrate how CADS can substantially alleviate this trade-off. Figure 5 supports this finding. With condition annealing, both FID and Recall exhibit less drastic deterioration as the
7

---

Published as a conference paper at ICLR 2024
FID
Precision
240
CAP6
420
10
300
10
200
110
0.3
The behavior of the evaluation metrics across different guidance scales. CADS exhibits superior ability to balance quality and diversity, evidenced by better performance in FID and Recall.
Figure 5:
Table 3:
Impact of integrating CADS with popular diffusion samplers using the class-conditional ImageNet model (DiT-XL/2). CADS enhances sample diversity across all samplers.
with CADS
without CADS
Sampler
FID J
Recull
FID
Recall f
DDIM (Song et al., 202
9.80
0.59
18.84
0.35
DPM+ (Lu et al., 2022b)
9.63
0.61
18.65
0.36
SDE-DPM++ (Lu et al.. 2022b)
11.25
0,56
19.49
0.35
PNDM (Liu et al., 2022a)
14.60
0.52
20.23
0.32
UniPC (Zhao
2023)
10.10
0.59
18.90
0.35
guidance scale increases, contrasting with the behavior observed in DDPM. Note that because higher guidance values lead to lower diversity, we increase the amount of noise used in CADS relative to the guidance scale to fully leverage the potential of CADS. This is done by lowering Ti and increasing for higher WCFG values. As we increase WCFG. DDPM's 's diversity becomes more limited, and the gap between CADS and DDPM expands. Thus, CADS unlocks the benefits of higher guidance scales without a significant decline in output diversity.
Different diffusion samplers Table 3 demonstrates that CADS is compatible with common off- the-shelf diffusion model samplers. Similar to previous experiments, incorporating CADS into each sampler consistently improves output diversity, as indicated by considerably better FID and Recall.
Dynamic CFG vs CADS We next compare DDPM sam- pling with CADS and Dynamic CFG. Table 4 indicates that although both CADS and Dynamic CFG increase the output diversity of DDPM in the class-conditions model, CADS generally leads to more diverse outputs and out- performs Dynamic CFG in the FID score. This is also reflected in the recall values of 0.62 for CADS and 0.39 for Dynamic CFG. Hence, we contend that CADS per- forms better than simply modulating the guidance weight during inference. Figure also showcases the general similarity between generated samples based on CADS and Dynamic CFG, confirming the theoretical intuition introduced in Section 3.2.
Table 4: Comparison between CADS and Dynamic CFG on class-conditions ImageNet generation.
Sampler
FID.
Recall
DDPM
20.83
0.32
Dynamic CFG
18.42
0.39
CADS
9.47
0.62
Condition alignment To examine the impact of noise on the alignment between the input condition and the generated image, we use three metrics: top-1 accuracy for conditonnii Im- ageNet generation, mean-pert peean-per--oint pose error (MPJPE) for the pose-to-image model, and CLIIP-ccore for text-to-image synthe- sis. Results in Table 5 indicate that pose and text alignments are identical between DDPM and CADS. For class-conditional generation, only minor accuracy decline is observed, likely due increased output variation. Hence, condition annealing does not compromise the adherence to the input conditions.
Table 5: Condition alignment of dif- ferent methods after using CADS.
Alignment metric
DDPM
CADS
Top 1 Class Accuracy t
0.98
0.96
MPJPE J CLIP Score
0.02
0.02
0.31
0.31
4.2 ABLATION STUDIES
This section explores the role of the most important parameters in CADS the final quality and diversity of generated samples. Additional ablations on the role of T2, the distribution of n, and the functional form of 7(t) are provided in Appendix E.
8

---

Published as a conference paper at ICLR 2024
DDPM
Dynamic CFG
CADS
Figure 6: A comparison between condition annealing (CADS) and Dynamic CFG, Both can effectively increase diversity in generated outputs, but CADS generally offers more variations.
Table 6: Ablation study examining various design elements in CADS.
(a) Influence of noise scale s (b) Impact of the cut-off threshold T (c) Effect of the mixing factor
FID 4
Recall
FID L
Recall
FID
Recall
0.025
19.51
0.35
0.2
69.73
0.80
85,25
0.78
15.79
0.52
15.79
0.52
0.56
0.25
42.58
0.69
20.71
0.32
12.18
0.55
0.025
0.25
T1 0.2
T1 0.9
0.0
1.0
(a) Effect of the noise scale
(b) Effect of
(c) Effect of rescaling
Figure 7: Visual illustration how different hyperparameters affect CADS.
Noise scale The effect of the noise scale on the generated images is illustrated in Figure 7a and Table 6a. Introducing minimal noise (s = 0.025) to the conditioning results in reduced diversity, while excessive noise (s 0.25) adversely affects the quality of the samples.
Cut-off threshold Table 6b and Figure 7b show the influence of the annealing cut-off point TI on the final samples. Analogous to the noise scale, too much noise (7₃ = 0.2) degrades image quality, while too little noise (r1 0.9) results in minimal improvement in diversity.
Effect of rescaling Rescaling serves as a regularizer that controls the total amount of noise injected into the condition and helps prevent divergence, especially when the noise scale s high. This effect is illustrated in Figure 7c. Table 6c also reveals that an increase in y (indicating more regularization) typically results in higher sample quality (improved FID) but reduces diversity (lower Recall). We recommend using CADS with ve and decreasing it only if the attained diversity is insufficient.
5 CONCLUSION AND DISCUSSION
In this work, we studied the long- sta-sitnnding challenge of diversity-quality trade-off in diffusion models and showed that the quality benefits of high classifier-free guidance scales can be maintained without sacrificing diversity. This is achieved by controlling the dependence the conditioning signal via the addition of decreasing levels of Gaussian noise during inference. Our method, CADS, does not require any retraining of pretrained models and can be applied on top of all diffusion samplers. Our experiments confirmed that CADS leads to high-quality, diverse outputs and sets a new benchmark in FID score for class-conditional ImageNet generation. Further, we showed that CADS outperforms the naïve approach of selectively underweighting the guidance scale during inference. Challenges remain for applying CADS to broader conditioning contexts, such as segmentation maps with dense spatial semantics, which we consider as promising avenue for future work.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
With the advancement of generative modeling, the ease of creating and disseminating fabricated or inaccurate data increases. Consequently, while progress in Al-generated content can enhance productivity and creativity, careful consideration essential due to the associated risks and ethical implications. We refer the readers to Rostamzadeh et al. (2021) for more thorough treatment of ethics and creativity computer vision.
REPRODUCIBILITY STATEMENT
Our work builds upon publicly available datasets and the official implementations of the pretrained models cited in the main text. The precise algorithms for CADS and Dynamic CFG are detailed in Algorithms and along with the pseudocode for the annealing schedule and noise addition in Figure 15. Further implementation details as well as the exact hyperparameters used to obtain the results presented in this paper are outlined in Appendix G.
ACKNOWLEDGMENT
We would like to thank ETH Zürich Euler computing cluster for providing the infrastructure to run our experiments. Moreover, we thank Xu Chen and Manuel Kansy for helpful discussions during the development of the project and helping out with the final version of the manuscript.
REFERENCES
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yi Yu Liu. ediff-i: Text-to- image diffusion models with an ensemble of expert denoisers. CoRR, 0tss/2211.013244 2022. doi: 10.48550/arXiv.2211 .01322. URL https: /doi. org/ 10. 48500 aarivvaiivv/...///ii 2211. 1...1.00000 32444.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL wwtptttt///// pestps://opat /ptpr//ieeeeeeieeeeeeemee net/Eorum? wwtimmiddddlssse 1ttp:/orrumnd111xsss00000000
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, and William Chan. Waveg- rad: Estimating gradients for waveform generation. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https: wpwnreview.eeeeeeeeeeeeeee/oeeer net / forum? NaMMiiddd--sMaaaaaaaa
Ting Chen. On the importance noise scheduling for diffiusion models. CoRR, abs/2301.10972. 2023. doi: 10.485500arxxiv22011109999 2301.10977
URL httpps://dl) http:///doooooo//... htt.bloooro/11..55//r....... 48550/erxxii5
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans image synthesis. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 8780-8794, 2021. URL https: http:::ss//////eeeeeeeoee neurips wwt.cc/pppppp.ccec ptpppp/ppp/221111 2021 /hash/ wtt.aaacccce0002211115f.aaa Abst Abstt html.
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In JEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 12873-12883. Computer Vision Foundation / IEEE, 2021. doi: URL
10.1109/CVPR244477.2220.000 http///sseer__eee
thecvf. com/ com/content/
for figh Hetoo_Higgh-Relotiiiii Resolution. Image_Synthesi__ccm 2021 pwl.ppapee...m html..
Dan Friedman and Adji Bousso Dieng. The vendi score: A diversity evaluation metric for machine
learning. CoRR, abs/2210.02410, 2022. doi: 10485500axiv.21000240.. URL https: : /doi.
org/10. 41051/0..8550//// 22100 02410.
10

---

Published as a conference paper at ICLR 2024
Jianglin Fu, Shikai Li, Yuming Jiang, Kwan-Yee Lin, Chen Qian, Chen Change Loy, Wayne Wu, and Ziwei Liu. Stylegan-human: A data-centric odyssey of human generation. 13676: 1-19, 2022. doi: 10. 10.1007/978-3331--17771111 1. URL https: http://///o...gg/...o.o 178-33031-193 1011-1111777111111
Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is strong image synthesizer. CoRR, abs/2303. 14389, 2023, doi: 10.48550/arxi..3333.1 14389, URL
https://dole orgeiiorgg000 wtt.:o/.g......./..oo..//
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial networks. CoRR, abs/ 1606,
2014. URL http::t///ce. /ttt:::/axxivv org/ abs 1406.2661
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans time-scale update rule converge to local nash equilibrium. In Isabelle Guyon, Ulrike von I Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. neurips cc cccpappr/20117 h/ass/
trained by
6626-6637, 2017. URL httpst: 00e47077000boom owt.r//0r.00011111111607767666-0 lttml.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. CoRR, abs/2207.125999 2022. doi: 10.48550//rXiv,22277. 125588 https::: http:///.....oo.......oo/ 4tl.l////.....5///aaa/ art.vv.arxxvvv.om 220..
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URLhttpas http::::://///reeeeeeeeeee neurips ctpccc/pappp. httpp/pppe/22//////// /hash/ .tml..
Jonathan Ho, Chitwan Saharia, William Chan, Ja Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. CoRR, abs/2106. 2021. URL
https: / aaraaiiiiiv.///. org.orgggaoooooo awwp////////////o/////////////// 2106.11.. 15282.
Susung Hong, Gyuseong Lee, Wooseok Jang, and Seungryong Kim. Improving sample quality of guidance. CoRR, abs/2210.00939, 2022. doi: 10.48550/arXiv, diffusion models using self-attention guidance.
2210.00939. URL https::tt.. /wwi. www.oo.gooooooooooooo/ooo./....o /1. 48550/ arw.v. 2210. 2210.00939. 00939.
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans, simple diffusion: End-to-end diffusion for high resolution images. CoRR, abs/2301.11093, 2023. doi: 10.48550arxxi.23011.1099 URL https: doi. . org/10.gg.oooo 100. 48550
1101.
Qingqing Huang, Daniel S. Park, Tao Wang, Timo I. Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Havnø Frank, Jesse H. Engel, Quoe V. Le, William Chan, and Wei Han. Moise2music: Text-conditioned music generation with diffusion models. CoRR, abs/2302.03911, 2023, doi: 10445550aaxxi.2222203177 URL /ttttp:://di..rgg.// 48550/arXiv 2302. 03917.
Manuel Kansy, Anton Raël, Graziana Mignone, Jacek Naruniec, Christopher Schroers, Markus Gross, and Romann M. Weber. Controllable inversion of black-box face-recognition models via diffusion. CoRR, abs/2303.1330k 2023. doi: 10.485550arxxw.2303.. 13006. URL https: //doi. org/ wwt.oo/gg//../////// 23033333........00
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. 2022. URL http: /ttp:e///ppppaeeee wttppppppepppppseeeeepppeppppppp mips. cc wwtpp//ppppppeeeeeeee files//iie//eeeeeeeee /aper/
2022/hash/an htt.a////88...eeccccccccaaaaaaaa html.
Dongjun Kim, Ycongmin Kim, Se Jung Kwon, Wanmo Kang, and Il-Chul Moon. Refining generative process with discriminator guidance score-based diffusion models. Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.),
11

---

Published as a conference paper at ICLR 2024
International Conference Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 16567-16598. PMLR, 2023. URL httpps:/ppooceeeeieee wtt.::/s........sseeseeee2e
mlr wwwpppppsssseepppppppppppp. 20t02///mm3333mm html.
Diederik P. Kingma and Ruiqi Gao. Vdm++: Variational diffusion models for high-quality synthesis, 2023.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https:// openreview. net/ forum waa.mmmdd=aa------ YmJJ..
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Buse Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 3929-3938, 2019. URL https:ttph.cmmm https: http::///ococeeeeeeeeeeo meurdidppinnnnnies..p. newrips.crrccoooo cttps.cc/ppeeeeeeeeee wwcpppper/021.. 9/ hash/ 0234 02244c110cccd0008228888833333333
Abstract html.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision Eision-ECCVAA 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, volume 8693 of Lecture Notes in Computer Science, pp. 740-755. Springer. 2014. doi: 10.1007/ 978-3-319- 100-319-1000-11- 10602-00..44 18.48 48.
URL https: /tt///oioooooo -100:///00.11100077------------1
Guan-Horng Liu, Arash Vahdat, De-An Huang, Evangelos A. Theodorou, Weili Nie, and Anima Anandkumar. 12sb: Image-to-image schrödinger bridge. CORR, abs/2302.05872, 2023. doi: 10.485550/rx1i.2322..555
URL https: htt//ddo.oorooo wwo.//o/11... 0tepg/11.48555000 400.04/855///iii.../ htt./aaxvvvvv... 2302. 05872.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. 2022a. URL https::/p/// btttss://opezzeee//e
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models manifolds. In The Tenth International Conference Learning Representationss ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022b. URL Corum?id=s www.ddpkw.KKyyyyB..KK..
n
Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 1096- 1104. IEEE Computer Society, 2016. doi: 10.1109/CVPR.2016 124. URL https: /toe::/ddoo.. /d/oiooooooooooo org/
10. 1109 CVVR. 201 124.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dolesolver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In NewrlPS, 2022a. URL //ap:///appeaeaeeeeee nips . cc/paper wtllleeppppr///// Abstract-
hash// 6aaceeefeeeeeeeeeeceee -ttprr/ettttttttttttttt-------ae wwweett-ooooeeeeeeeeeeene html..
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfci Chen, Chongxuan Li, and Jun Zhu. Dpm-solvert+: Fast solver for guided sampling of diffusion probabilistic models. CoRR, abs/2211.01095, 2022b. doi: 10.488500arXxi..221.0000
URL https: http://o..ooooo/////o carei.orgg1000 48550/ aatpvarxvvv//v. wtt.:iiii..i...................a 2211. 01095.
Kevin P. Murphy. Probabilistic Machine Learning: Advanced Topics. MIT Press, 2023. URL
http:// http::://pobmll...m github. Lo/book2.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event. volume 139 of Proceedings of Machine Learning Research, pp. 8162-8177 8171. PMLR, 2021. URL http: http///pooo.e..../////////./..sp ml wttpps/ss///////// whol: cho 121a html.
12

---

Published as a conference paper at ICLR 2024
Gaurav Parmar, Richard Zhang, and Yan Zhu. On aliased resizing and surprising subtleties in GAN evaluation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA. June 18-24, 2022, pp. 14400-11400 IEEE, 2022. doi: 10. 1109/CVPR52688.2222..11111 URL. https: / 01112.
2022.
William Peebles and Saining Xie. Scalable diffusion models with transformers. CoRR, abs/2212.097488, 2022. doi: 10448550/2Xxx::2222.0774 URL https::/doi. 48550/arxiv 09748.
Ed Pizzi, Sreya Dutta Roy, Sugosh Nagavara Ravindra, Priya Goyal, and Matthijs Douze. A self- supervised descriptor for image copy detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 14512-14522. IEEE, 2022. doi: N0.109//CVR52288.222220000 CVPR52688 688. 2022. 01413.
URL https: /ww. www.oooooooooo..ooo 1109/
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents. CoRR, abs/2204.06125, 2022. doi: 10.48550/ arXiv.2204.06 125. URL https: // doi. otg. (03550/arX1:1 2204. 06125.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA,, USA, June 18-24, 2022, pp. 10674- 10685. IEEE, 2022. doi: 10.1 D0.119//VVR22288.2000 2022.0104. URL https: htt//dooooorrog //o../////o....0.../.. org/10. 1109/CVPR5268 688 2022.22222 01042.
Negar Rostamzadeh, Emily Denton, and Linda Petrini. Ethics and creativity in computer vision. CoRR, abs/2112.03111 2021. URL https:::: http://axxvv........ aataa////111 /2122212..111 2212/.0111.........1111111111111
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Ima- genet large scale visual recognition challenge. Int. J. Comput. Vis., 115(3)221 -222, 2015. doi: 10. URL https: htt.://.iiio// org 10
1007/=11263-015-088
015-0816--110
Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In Munkhtsetseg Nandigjav, Niloy Mitra, and Aaron Hertzmann (eds.), SIGGRAPH '22: Special Interest Group on Computer Graphics and Interactive Techniques Conference, Vancouver, BC, Canada, August 7 11, 2022, pp. 15:1-15:1M ACM, 2022a. doi: 10.1 10.11455332822333533555 URL https: //doi. org/ htt.g//..1111///111
3530757.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Den- ton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. 2022b.
URL http::/paapeee nips http:/pppr_fflleee/eeeeepe http//aap///////// /2022 /hash/ abst html..
ec7
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https: cectps::/openrrviii www.neeviww...o.ooooop net /foumm? prrumidde =id= TIdIXIpzhoI.
Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 2226-2234, 2016. URL htwpt:ttttt/tttetttttt http:::es///////epppppps neurine cc/ Abstract html.
paper 201 hhwah////////eeae 8a3333333aabaa wtt.aeeaeee222222111122111221110 http/////////.33333a222a222222..
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In Munkhtsetseg Nandigjav, Niloy J. Mitra, and Aaron Hertzmann SIGGRAPH '22:
13

---

Published as a conference paper at ICLR 2024
Special Interest Group Computer Graphics and Interactive Techniques Conference, Vancouver, BC. Canada, August 7 11, 2022, pp. 49:1-49:10. ACM, 2022. doi: 10.1145/352823.5555778 URL htthh:t////// doi wwg/ 10 1tt.:/.....1/1/.1.
3530738.
Vikash Sehwag, Caner Hazirbas, Albert Gordo, Firat Ozgenel, and Cristian Canton-Ferrer. Generating high fidelity data from low-density regions using diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 11482-11491, IEEE, 2022. doi: 10.1 wrr.i.org/io 10. 1t..1111/////1 /C/////2V222222 222.2222111111
URL https:
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsu- pervised learning using nonequilibrium thermodynamics. 37:2256-2263 2015. URL http:
//ppooeeeiing mir press / /sthssslaiiikseeiiil55.. 5.tml..
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion or digital forgery? investigating data replication in diffusion models. pp. 6048-6058, 2023a. doi: 10. (099///PR52229.2223...2. 00566.. URL https: /tt.//oiii..//////////// org/ 10/ 1ttp://0.1199/////572200 2023. 00586.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Un- derstanding and mitigating copying in diffusion models. CoRR, abs/2305.20086, 2023b. doi: 10.485500/rXiv.23055 20086. URL https: /doi. /tt.oo..or/.....///////.... 48510/4550/8800 aarpvv..vvv///////// 2305 20086.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In 9th Interna- fional Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
OpenReview.net, 2021a. URL htw.htts:.. htttp:://ppeeeeeeeeeeee /op.neeeeeeeeeeeeeeeeeeeeeeeeeee net/
Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guiide diffu- sion models for inverse problems. In The Eleventh International Conference on Learning Representutiins, ICLR 2023, Kiguli, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: / httpss///pperevieee net ppddfid==_ggMABBB
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché- Buuhe,Bu,.a Emily B. Fox, and Roman Gareett (eds.), Advances in Neural Information Processing Systems 32: Annual Confer- ence on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Van- couver, BC, Canada, pp. 11895-1 11977, 2019. URL https: meuu ps. cc/ paper/2019 9 htt///99/haahhaae hwt.:aaaahhhaaaaaaaeaaaaaa http:/3011e122122 paaps//222.aaa///0.5555553337339 Abst html.
Yang Song. Jascha Sohl-Dickstein Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In 9th Interna- fional Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OvenReview.net, 2021b. URL https: http::////peeeeeeeeeeeee net/ wwt.torumeeeeeeeeeeeo Saarummid-PxIIIOe
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit Haim Bermano. Human motion diffusion model. 2023. URL https: wpwnreveeeeeeeeeee/e/ee wwt./e////d./t met/pdf? ppff/dfff dd SJ1 SJ1kSy02jju.
Jonathan Tseng. Rodrigo Castellon, and C. Karen Liu. EDGE: editable dance generation from music. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 448-458. IEEE, 2023. doi: 10. 1099//VPR529992222.0000 URL wwt.tttts/.......... do org /
0. 11 09 /1//////52222 729. 2023 00051.
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Ming- Ming-Hsuan Yang, and Bin Cui. Diffusion models: A comprehensive survey of methods and applications. CoRR, httx/2299.00966 2022. doi: 10.48550aaxx..2200.00799 URL
https:///oo org/ wtt.:ii.rrg111...1.1.. http://...oogg......//a/a//a..// w...555//arxi..222
Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: A unified predictor-
corrector framework for fast sampling of diffusion models. CoRR, abs/2302.04867, 2023. doi: 10.485550krXiiv22322200o 24877. URL https: : doi. 1tt.:/..............11.1 48t.00.0.././/xxi....... wttp:///aaxxvv..........
14

---

Published as a conference paper at ICLR 2024
A BACKGROUND ON DIFFUSION MODELS
We provide an overview of diffusion models in this section for completeness. Please refer to Yang al. (2022) and Karras et al. (2022) for more comprehensive treatment of the subject.
Let x ~ Paata (2) be a data point, L E [0. 1] be the time step, and 2; = I + o(L)e be the forward process that adds Gaussian noise E (0, to the data for monotonically increasing function (t) satisfying (0) 0 and (11 = maxx > dataa- Karras et al. (2022) showed that the evolution of the noisy samples 2, can be described by the ordinary differential equation (ODE)
dz = (t) 22, log Pt (z)) dt,
(5)
or equivalently, the stochastic differential equation (SDE)
dz = (t) Vz, (x, dt (t))) log):(z;) + 22 duy- (6)
Here, duy is the standard Wiener process, and (z())) the distribution of perturbed samples with Po = Paat and P1 / o m)x Assuming we have access to the time-dependent score function 2x, lo.loop((a (z sampling from diffusion models is equivalent solving the diffusion ODE or SDE backward in time from = 1 to = O. The unknown score function Vz, log (z(z) is approximated using a parameterized denoiser Do (z t) trained on predicting the clean samples from the corre- sponding noisy samples 2,. The framework allows for conditional generation by training a denoiser (2 (, that accepts additional input signals y, such class labels text prompts.
periance-preserving (VP) formulation VP is another way to define the forward process of diffusion models. Note that the process 2, = x + a (L)e has time-dependent scale governed by (t). In contrast, the VP formulation destroys the signal x using ž, axt + o₁e. The parameters og and or determine how the original signal æ is degraded over time, with 0r v a being common choice for a E [0, 1]. This forward process ensures that the variance of the noisy samples 2, is better constrained over time.
Training objective Given noisy sample 2,, the denoiser Dø f,y) with parameters 0 can be trained with the standard MSE loss
arg min (0. [o(, t, ==1²]
It is also common to formulate the denoiser network c0 I, y) that predicts the total added noise e instead of the clean signal x. The training objective for diffusion models this case
mig 1,-)((,1)) [IEo(2e, (,)) -
(8)
0
Recently, Salimans & Ho (2022) found that predicting the "velocity" v. = age o,æ instead of the noise E results in more stable training and faster convergence, similar to the preconditioning step introduced by Karras et al. (2022). After training, the denoiser approximates the time-dependent score function Vz, (ogpi(z, (z,) via
Vz, log))(z)) x Do(2),1.y) o(l)
Eo(2,,1.9) o(t)
Latent diffusion models (LDMs) LDMs (Rombach et al., 2022) are similar to image-space diffusion models but operate on the latent space of a frozen autoencoder. Given an encoder E and a decoder DD trained purely with reconstruction loss, LDMs first convert all training samples to their corresponding latent vectors {(x) and train the diffusion model on We then sample new latents ã from the diffusion model and get the corresponding data through the decoder, with x D(ž).
Classifier-free guidance (CFG) CFG is a method for controlling the quality of generated outputs inference by mixing the predictions of a conditional and an unconditional model (Ho & Salimans, 2022). Specifically, given a null condition Youlll = Ø corresponding to the unconditional case, CFG modifies the output of the denoiser at each step based on
(9( (z( I, Dmult) werr (2( 1, Do( (,yu))))
(10)
where WCHG = corresponds to the non- non-guided case. The unconditional model Do t. Ymu)) is trained by randomly assigning the null condition Ynull @ to the input of the denoiser for portion of training. Similar to the truncation method in GANs (Brock et al., 2019). CFG increases the quality of individual images at the expense of less diversity (Murphy, 2023).
15

---

Published as a conference paper at ICLR 2024
B CONDITION ANNEALING AND LANGEVIN DYNAMICS
The purpose of this section is to show the connection between condition annealing and Langevin dynamics. As stated in Appendix A, sampling from diffusion models can be formulated solving the diffusion ODE given by Equation (5). One straightforward solution to this ODE is through Euler's method: 21-1 2, Pt Vz. log p(z,) for some step size Pt. Note that the denoiser in diffusion models provides an approximation for the score function (Karras et al., 2022). That is, for given condition y, we have
V2, logg(p/.l))) ~ - D(z,,t.))
(1I)
a(t)
For simplicity, assume that we corrupt the condition according to ÿ = + am, where the noise term Jn is small compared to y. Based on the Taylor expansion, denoiser as Dize, t, f) (a t, y) + de V, (z,,, t,y) n, and
approximate the output of the
Zt
Zz, log (z.l()
(t) de Vy D(2r, (1) nn
(12)
Let 77 ce
Pt slt)
V (iz f. f... The update rule in Equation (12) is then equivalent to
2 2, + Pt Vz, logp(zzly) nt
(13)
Hence, incorporating noise into the condition is largely analogous to Langevin dynamics step, albeit with potentially non-isotropic additive Gaussian noise, guided by the underlying diffusion ODE. This parallels conventional Langevin dynamics, where noise addition mitigates mode collapse and improves data distribution coverage. It should be noted that ancestral sampling algorithms like DDPM already incorporate noise addition step in the reverse process. However, our empirical results in Section 4.1 indicate that this step alone is insufficient to ensure output diversity. Conversely, the noise introduced condition condition-annealing, governed by T) appears to provide more effective sampling dynamics.
C CONDITION ANNEALING AS SCORE SMOOTHING
In this section, we provide a rough theoretical analysis of CADS in a manner similar to Song et al. (2023), which is based on Tweedie's 's formula for simple case in which the condition is linearly related to the input, i.e., y = Tw. Note that diffusion models sample from the conditional distribution by following the score V.. log Pe (Ze y) each time step. According Bayes' rule, we have
V,, log
y) Vz, log (zt)) Vz, log Pt (y)
(14)
As the first term does not depend on y. analyze what happens to the second term when we add noise to y via ý + a.n, where az controls the amount of noise at each step. That is, we aim to find an approximation for the conditional score Vz, log (r 2.). First, note that have
p 21) (y) dr.
(15)
The distribution (xx 2,) is intractable, the mean &.. That is, we assume (x) & N(t,, AI) for some X. If the forward process for the diffusion model is given by 2, can be estimated based on Tweedie's formula, namely &, = E[x] = z, + (t) Vz, log P. (z,). By carrying the Gaussian approximation forward and applying the transformation Pe ze) sJ N (T2, XTT ). This means we can write y 2; = TI, + XTu for a random variable u ~ (O I). Hence, the annealed condition is given by ÿ z; Tě, + XTu + n.n, and we have pe z1) = N (T2, TT + o²)). This leads to
approximate it by Gaussian distribution around
a()),, the mean & of the distribution pe p. 2.)
we obtain
Vz, log p. (ūl 2, = (ū - Tě,) (A2TT + + 'T 0b.
(16)
Oz.
Compared to the noiseless case, o. 0, adding noise smooths the inverse term 2T + similar to the effect of &g regularization in ridge regression. This regularization should on average reduce 2. log Pe (Úl 2:]ll and, henee, increase the diversity of generations by preventing the domi- nation of strong modes in the inference process. A similar phenomenon was observed empirically in Dhariwal & Nichol (2021).
16

---

Published as a conference paper at ICLR 2024
-1
Real data
(b) DDPM
(c) CADS (Ours)
Figure 8: Effect of condition annealing on toy problem. The real samples from the data distribution are shown in (a). When sampling with high guidance values, standard DDPM converges the mode of each component (b), but sampling with CADS results in better coverage of each mode (c).
FDD
NETSs
NFCx
NFDs
NFBs
(a) ImageNet
(b) DeepFashion
Figure 9: The behavior of CADS versus different NFEs for the candiionniiional (ImageNet) and pose-to-image (DeepFashion) models. CADS exhibits similar behavior to the standard sampler as the NFE increases, while consistently outperforming DDIM at each NFE.
D ADDITIONAL EXPERIMENTS
We provide additional experiments in this section to extend our understanding of CADS.
D.1 A TOY EXAMPLE
In this section, we visualize the effect of condition annealing on the generated samples with simple toy example. Our goal in this experiment is to show that condition annealing does not result in sampling from unwanted regions according to the data distribution. Consider a two-dimensional Gaussian mixture model with 25 components the data distribution, illustrated in Figure 8a. We train a conditional diffusion model with classifier-free guidance for mapping the component ID (between and 25) to samples from the corresponding mixture component. In Figure 8b, we see that we use the standard DDPM with a high guidance value, the samples roughly converge to the mode of each component. However, condition annealing the class embeddings leads to better coverage of each component as shown in Figure 8c. Also, Figure 8c confirms that using CADS does not result in sampling from regions that not in the support of the real data distribution.
D.2 PERFORMANCE OF CADS VS THE NUMBER OF SAMPLING STEPS
In this section, we study the performance of CADS against the number of sampling steps. also known as neural function evaluations (NFEs). Figure 9 demonstrates that the behavior of CADS similar to the base sampler we change the number of sampling steps, and a consistent gap exists between sampling with and without condition annealing across different NFEs. Interestingly, with a fixed annealing schedule in CADS, the FID metric plots show U-shaped curve over a subset of the range of NFEs in the ImageNet model. This suggests that the annealing schedule used in this experiment may be better suited for certain NFEs over others. Please note that DDPM (Ho et al., 2020) does not perform well when using low NFEs, we switched to the DDIM sampler (Song et al., 2021a) la) to perform this comparison.
17

---

Published as a conference paper at ICLR 2024
Table 7: Quantitative comparison among samples generated with and without CADS for various conditional generation models and diffusion samplers. CADS notably enhances the diversity of the generated samples compared the base sampler.
Dataset
Sampler
FID +
Precision
Recall f
MSS
Vendi Score f
DDIM
14.60
0.93
0.02
0.80
1.03
DeepFashion (weRO 4)
CADS (Ours)
7.90
0.76
0.49
0.35
2.30
DDIM
26.27
0.70
0.15
0.57
1.27
SHHQ (werg
CADS (Ours)
15.14
0.61
0.46
0.36
2.06
DPM++
45.70
0.70
0.29
0.19
5,30
CADS (Ours)
40.35
0.65
0.42
0.13
6.93
Stable Diffusion (werG
PNDM
45.76
0.68
0.28
0.19
5.36
CADS (Ours)
41.37
0.65
0.38
0.13
6.83
D.3 ADDITIONAL EXPERIMENTS WITH DIFFERENT SAMPLERS
Table 7 provides more comparisons on the effectiveness of CADS when used with different diffusion samplers other than DDPM (Ho al., 2020). Specifically, we extend the results of Table for the pose-to-image models using DDIM (Song et al., 2021a) as the base sampler. Additionally. we report different comparisons based on Stable Diffusion with PNDM (Liu al., 2022b) and DPM++ (Lu al., 2022b) methods. Similar to what has been shown in Table 3, CADS consistently improves the diversity of the base sampler without compromising quality.
D.4 TRAINING WITH NOISY CONDITIONS
A natural extension to our approach involves training the diffusion model under noisy conditions, aiming to achieve more diverse results during inference with standard samplers. We tested this on the pose-to-image task by training the underlying diffusion process with noisy pose images. However, as indicated in Figure 10, this approach still yields outputs with limited diversity. The finding indicates that the low diversity issue lies mainly in the inference stage rather than in the training process. In fact, we found that training the diffusion model with noisy poses diminishes the effectiveness of CADS, as the network learns to associate noisy conditions with specific images.
DDPM
CADS (Ours)
DDPM
CADS (Ours)
Figure 10: Comparing DDPM with CADS on a pose-to-image immge model trained on noisy pose images. Training on noisy images does not solve the issue of low-diversity by default, and sampling with CADS is still needed achieve better diversity.
18

---

Published as a conference paper at ICLR 2024
(a) Without noise
(b) Adding noise to joint locations
(c) Adding noise to the 2D pose image
Figure 11: Comparing outputs of CADS when the noise is added to the joint locations and to the 2D pose image. Adding noise the pose image results in better diversity and quality.
D.5 ADDING NOISE TO THE JOINT LOCATIONS
In the pose-to-image task, we also experimented with adding noise to the joint locations instead the entire 2D image. However, we found that this leads to blurry output images, especially at the edges between the subject and background. An illustration of this issue is given in Figure 11. Additionally, this method increases the sampling overhead since it requires the pose image be rendered each step. As result, we opted to add noise directly to the pose image in subsequent experiments.
D.6 REDUCING THE GUIDANCE SCALE
One common way increase the diversity in diffusion models is reducing the scale of classifier-free guidance. However, similar to previous works (Dhariwal & Nichol, 2021; Ho & Salimans, 2022), we show in Figure 12 that reducing the guidance diminishes the quality of generated images. In contrast, Figure 13 shows that CADS maintains high-quality images by improving output diversity at higher guidance scales. Furthermore, when the model is trained on small datasets like DeepFashion, the low diversity issue exists at all guidance scales as evidenced by Figure 13. Therefore, reducing the guidance scale does not resolve the issue of limited diversity.
D.7 MORE DISCUSSION ON DYNAMIC CFG
In the main text, we evaluated the performance of Dynamic CFG and CADS in the context of the class-conditional ImageNet generation model. In this section, we expand our analysis to include pose-to-image generation task. As shown in Figure 14, both Dynamic CFG and CADS offer better diversity than DDPM; however, compared to CADS, sampling with Dynamic CFG results in more artifacts in this domain. This observation is further supported by the FID scores of 9.58 for Dynamic CFG and 7.73 for CADS. These findings reaffirm that CADS outperforms Dynamic CFG, making more effective solution for enhancing diversity in diffusion models.
19

---

Published as a conference paper at ICLR 2024
Cheeseburger
Boston Terrier
Macaw
Fountain
Figure 12: Examples of artifacts when using low classifier-free guidance scale (WCFG 1.5) with the class-conditional model. Reducing CFG increases diversity but hurts image quality. Class labels are given below each image.
DDPM, WCFG 1.0
DDPM, WCFG 1.5
CADS, WCFG 7.0
DDPM, WCRG
1.0
DDPM, MCFG = 1.5
CADS, WCFG = 4.0
Figure 13: Examples of sampling with lower guidance values compared high guidance values with CADS. Reducing the guidance scale deteriorates image quality in the class-conditional ImageNet model and fails to enhance diversity in the DeepFashion mose-to-image model. CADS improves diversity while preserving high-quality images in both cases.
D.8 COMBINING DYNAMIC CFG AND CADS
We empirically evaluate the combination of Dynamic CFG and CADS for class-conditional ImageNet synthesis in Table 8 for two guidance values. Combining CADS and Dynamic CFG enhances performance at high guidance scales but degrades the FID otherwise. This suggests that combining Dynamic CFG and CADS might be beneficial only when the diversity is more limited, e.g., higher guidance scales. We chose to use CADS alone in our main experiments due to its consistent performance across all guidance values.
20

---

Published as a conference paper at ICLR 2024
(a) DDPM
(b) Dynamic CFG
(c) CADS
Figure 14: Comparison of generated images using DDPM, Dynamic CFG, and CADS. While both Dynamic CFG and CADS have better diversity than DDPM, CADS delivers higher sample quality, as evidenced by fewer visual artifacts each image.
Table 8: Combining Dynamic CFG with CADS can be beneficial at higher guidance scales, but may negatively affect the metrics at lower guidance values where the diversity is less limited.
(a) WCFG
(b) WCFG 2.5
Sampling Method
FID 4
Recall
Sampling Method
FID
Recall f
DDPM
20.83
DDPM
12.72
0.49
Dynamic CFG
18.42
0.39
Dynamic CFG
6.58
0.65
CADS
9.47
0.62
CADS
5.02
0.71
Dynamic CFG CADS
8.14
0.66
Dynamic CFG CADS
5.43
0.74
E ADDITIONAL ABLATIONS
In the following sections, we conduct further ablation studies to evaluate the impact of different parameters on CADS performance.
E EFFECT OF 72
The parameter To in Equation (2) determines how many steps in the beginning of sampling are run without using any information from the conditioning signal (since y(t) = 0 for t > T2). Table illustrates the impact of varying 72 on the class-conditional model for two guidance values as well as the pose-to-image model. Since the diversity decreases higher guidance levels, reducing T2 improves both FID and Recall for WCFG 5, but negatively impacts these metrics for WCFG = 2.5. Thus, argue that when the low diversity issue is pronouneed, more noise is needed at the beginning of inference to achieve better diversity. This phenomenon is also evident in the pose-to-image model reducing improves the metrics there.
21

---

Published as a conference paper at ICLR 2024
Table Effect of changing 72 on the conditionl model for two guidance values well as the pose-to-image model. We use fixed = 0.5 for each table.
(a) WCFG =5
(b) WCFG 2.5
(c) Pose-to-image model
72
FID 4
Recall f
FID 4
Recall f
FID
Recall
8.21
0.67
0.5
6.14
0.76
0.5
8.26
0.38
0.9
9.47
0.62
0.9
5.02
0.71
1
10.93
0.19
Table 10: Comparing different choices of (() based on a polynomial annealing schedule with degree and fixed 7 0.6. The piece-wise linear function for (t) with Ti 0.6 and T2 = 0.8 outperforms all polynomial settings. We use 0.15 for all entries.
(t)
4
Recall
Baseline
14.20
0.51
Polynomial, d 15.85
0.45
Polynomial, d 14.98
0.48
Polynomial, d
14.49
0.48
Polynomial, d = 14.32
0.50
E.2 VARYING THE ANNEALING SCHEDULE
Polynomial Anncaling Functions
In this section, we investigate the impact of different y(t) func- tions the performance of CADS. Specifically, we employ polynomial functions of varying degrees for (t) and compare them with piecewise linear function. The polynomial annealing schedule formulated as
0.0
0.4
1ST
7(1) =
(17)
17T.
0.
0.4
0.5
Table 10 indicates that FID and Recall only slightly change as we increase the degree of the polynomial, and the piecewise linear function with appropriate Ti and 72 offers better scores. Therefore, we opted for the piecewise linear annealing function due to its better performance and interpretability.
E.3 CHANNGNGG DISTRIBUTION OF THE NOISE
We now examine various noise distributions, namely Uniform, Gaussian, Laplace, and Gamma, for the noise parameter introduced in Equation (1). Table 11 illustrates that CADS is not sensitive to the particular distribution of the added noise. The only influential factor is the standard deviation of n, which can be absorbed into the noise scale as defined in Equation (1). Based on this, we used Gaussian noise in all of our experiments.
F LIMITATION OF IS AND PRECISION
This section demonstrates that IS and Precision have limitations in accurately evaluating diverse inputs, suggesting cautious interpretation of these metrics. To validate this, we compute the metrics for 10k randomly sampled images from the ImageNet training and evaluation set and report the results in Table 12. As anticipated, FID and Recall show superior performance contrast, DDPM samples exhibit much better Precision and IS scores, largely due to their limited diversity. When sampling with CADS, IS and Precision are lower than DDPM but they do not fall bellow the values computed from the real data. This indicates that CADS enhances diversity while maintaining high image quality.
the real data. In
22

---

Published as a conference paper at ICLR 2024
Table 11: Comparing different noise distributions for Equation (1) on class-conditional ImageNet generation with WCFG = 5. All samples for this table are generated using CADS with 0.5,
72 0.9, 8 = 0.15,, and da 1.0. Gamma distributions are centered to have zero mean.
Noise Distribution
FID +
Recall
Normal(0, 1)
9,47
0.62
Normal(O,
8.89
0.64
Laplace(0, 1) Laplace (0, v1.5)
9.15
0.63
8.86
0.63
Gamma 1, 1)
9.38
0.62
Gamma (, 4)
8.78
0.63
Gamma (1, ()
9.68
0.60
Gamma 77, v7)
9.67
0.61
Uniform (0, 1)
13.42
0.50
Uniform 11
10.54
0.59
Uniform(-3. (33
8.89
0.63
Table 12: Comparing metrics evaluated based on generated and real images. IS and Precision show artificially high values for DDPM, suggesting that FID captures realism and diversity better than these two metrics.
Samples
FID
IS f
Precision f Recall
real, train
4.12
332.49
0.75
0.76
real, eval
4.86
235.85
0.75
0.74
DDPM
20.83
488.42
0.92
0.32
CADS
9.47
417.41
0.82
0.62
G IMPLEMENTATION DETAILS
Sampling algorithms We provide the pseudocode for the annealing schedule and noise addition in Figure 15. In addition, the detailed algorithms for sampling with CADS and Dynamic CFG are provided in Algorithms and
Sampling and evaluation details Table 13 includes the sampling hyperparameters used create each table in the main text. For computational efficiency, 10k samples are employed to calculate the FID across in all the experiments, except the state-of-the-art results (Table 2), where 50k samples are used to align with the current literature. We use 250 sampling steps for computing the metrics for Table 2 and 100 sampling steps for all other experiments. For evaluating the Stable Diffusion model, sample 1000 random captions from the COCO validation set (Lin al., 2014) and generate 10 samples per prompt.
Training details for the DeepFashion and SHHQ models We adopt the same network architec- tures and hyperparameters those in the publicly released LDM models (Rombach et al., 2022), available at https: / truu..ub... com CompVis lltenttddddd ffus ion. Specifically, we train VQGAN autoencoder (Esser et al., 2021) the DeepFashion and SHHQ datasets from scratch with the adversarial and reconstruction losses. Then, we fit diffusion model in the VQGAN's latent space using the velocity prediction formulation (Salimans & Ho, 2022). For detailed information on the exact hyperparameters of the models employed, please refer to the official codebase of LDM, along with the original papers (Esser al., 2021; Rombach et al., 2022).
H MORE VISUAL RESULTS
This section provides more qualitative results demonstrate the effectiveness of CADS compared to DDPM. Figures 16 to 18 show more samples from the class-conditional ImageNet generation task at both x56 and 512x512 resolutions. We also provide more samples from the pose-
23

---

Published as a conference paper at ICLR 2024
Table 13: Sampling parameters used in different experiments.
Dataset
WCFG
72
DeepFashion
4
0.35
0.7
0
SHHQ ImageNet 256
4
0.425
0.6
0.8
0
5
0.5
0.9
0.15
Parameters for Table
ImageNet 512
5
0.6
1
0.1
ID3PM
4
0.5
0.5
Stable Diffusion
9
0.6
0.9
0.25
2
0.525
0.9
0.06
ImageNet 256
2.25
0.5
0.9
0.06
2.5
0.475
0.9
0.06
Parameters for Table 2
2
0.625
1.0
0.07
ImageNet 512
2.25
0.6
1.0
0.07
2.5
0.575
1.0
0.07
0.8
0.9
0.05
1.25
0.75
0.9
0.05
1,5
0.7
0.9
0.05
2
0.55
0.9
0.075
Parameters for Figure 5
ImageNet 256
2.5
0.525
0.9
0.075
3
0.5
0.9
0.1
4
0.5
0.9
0.1
5
0.5
0.9
0.15
7
0.475
0.9
0.2
Parameters for Table 6a
ImageNet 256
5
0.6
1
0
Parameters for Table 6b
ImageNet 256
5
0.1
0
Parameters for Table 6c
ImageNet 256
5
0.6
0.1
to-image model in Figures 19 to 21. Finally, Figures 22 and 23 contain more samples from the ideniityconndtioned face synthesis network.
24

---

Published as a conference paper at ICLR 2024
def dinear_scheuules taul, tau2):
if t <= taul:
return 1.0
t tau2:
return 0.0
ganma (tau2 - t) (tau2 taul)
return gamma
dof add_ncise (y, gamma, poise_scalo, psi, rescalo=ralse):
10
y_mea,, y_std = (y), std (y)
= sgrt (gamma) y noise_scale * sqrt (1 gamma)
randn_ 1ike (y)
if rescale:
13
y_scaled (y mean (y)) / stdly) * y_std y_mean y = psi * y_scaled + (1 psi) y
14
15
return
16
Figure 15: Pseudocode for the annealing schedule function and adding noise to the condition.
Algorithm 1 Sampling with CADS
Require: WCFG: classifier-free guidance strength
Require: y: Input condition
Require: Annealing schedule function 7(t), initial noise scale Initial value: 2 (0,
1:
2: for = ,,,
1, do
Add noise to the input condition and the null condition
3:
(1 + 8 s (t)n, Youll
v (+ Wmull + sv (1)m²
4: Rescale y and yumll needed
o Compute the classifier-free guided output at
5: DeFG(E,,4 v, = D(2,, 1, (mm)) wCFG (D(z,, 1, D(ze, L, mma))
Perform one sampling step (e.g. one step of DDPM)
2 diffusion (aaearseeeDrreeeae 2),
7: end for
8: return 20
Algorithm 2 Sampling with Dynamic CFG
Require: WoPa: classifier-free guidance strength
Require: y: Input condition
Require: Schedule function (()
Initial value: 2 (0,
2: for = 7,..., do
Modulate the guidance weight according the time step 3: Mera =+ttwer
o Compute the classifier-free guided output at
4: Dero(z, t, y) = D(z,, t, y + weru (D(z,, (o(z,,,,y) y, D(ze, t, Vmm)))
Perform one sampling step (e.g. one step of DDPM)
5: 2:-1 diffussionreevrreee (DcrG, Zt,
6: end for
7: return 20
25

---

Published as a conference paper at ICLR 2024
DDPM
CADS (Ours)
Figure 16: Uncurated examples from the class-conditions ImageNet 256 model with wero =
Class labels from top bottom: Fountain (562), Alaskan Wolf (270), Lemon (951), and Horbbill (93).
26

---

Published as a conference paper at ICLR 2024
DDPM
CADS (Ours)
Figure 17: Uncurated examples from the class-conditions ImageNet 256 256x 256 model with were = Class labels from top to bottom: Cabbage (936), Bald Eagle (22), Spider's Web (815), and Golden Retriever (207).
27

---

Published as a conference paper at ICLR 2024
DDPM
CADS (Ours)
Figure 18: Uncurated samples from the class-conditionns ImageNet 512x512 model with WCFG = 6. Class labels from top to bottom are: Volcano (980), Sulphur-Crested Cockatoo (89). Cheeseburger (933), and Siberian Husky (250).
28

---

Published as a conference paper at ICLR 2024
DDPM
CADS (Ours)
DDPM
CADS (Ours)
D6
DDPM
CADS (Ours)
Figure 19: More samples from the DeepFashion pose-to tose-to- model. Each row represents generated images from fixed pose with 8 different seeds. Random seeds are identical between samples of DDPM and CADS.
29

---

Published as a conference paper at ICLR 2024
DDPM
CADS (Ours)
DDPM
CADS (Ours)
DDPM
CADS (Ours)
Figure 20: More samples from the DeepFashion to-saeo- model. Each row represents generated images from fixed pose with 8 different seeds. Random seeds are identical between samples of DDPM and CADS.
30

---

Published as a conference paper at ICLR 2024
DDPM
CADS (Ours)
DDPM
CADS (Ours)
DDPM
CADS (Ours)
Figure 21: More samples from the SHHQ pose-to- to-iage model. Each row represents generated images from fixed pose with 8 different seeds. Random seeds are identical between samples of DDPM and CADS.
31

---

Published as a conference paper at ICLR 2024
DDPM
CADS (Ours)
Figure 22: More results from the identity-conditioned face generation task with ID3PM, Each 4x4 grid contains generated images using fixed identity vector as the input.
32

---

Published as a conference paper at ICLR 2024
DDPM
CADS (Ours)
NNW
e
Figure 23: More results from the identity-conditioned face generation task with ID3PM, Each 4x4 grid contains generated images using fixed identity vector as the input.
33


Reviews:
[{"review_id": "a8rzVamHtp", "reviewer": "Reviewer_fxon", "summary": "Although diffusion models are known for good mode coverage, the sample diversity of conditional diffusion models is still challenging when sampling at a high classifier-free guidance (CFG) scale. This work examines the conditional diffusion models and attributes this problem to how the conditional information should be mixed into the reverse diffusion process. \n\nTo tackle this problem, it introduces an annealed sampling scheme for conditional generation, Conditional Annealed Diffusion Sampler (CADS). The core principle of CADS is to gradually increase the strength of conditional information in the reverse time diffusion process, letting the unconditional score model explore better data modes in the early stage (noisy region) and guiding the sampling converge to the conditional distribution using the conditional information in the final stage (data region).\n\nThis work conducted detailed experiments on class-conditional generation, pose-to-image generation, identity-conditioned face generation, and text-to-image generation. CADS consistently improves the sample diversity of baseline conditional diffusion models without retraining.", "questions": "1. How does this work determine where to add noises to the conditional information? For example, CADS mostly injects noises into the embeddings. However, for the pose-to-image generation, this work adopts the noise injection to the pose image. Is there a guidance to choose the position for conditional noise injection? What if inject noises into the internal layers of the conditional information extractor? What if we also adopt the embeddings noise injection scheme for the face-to-image generation? In other words, how large can the influence of noise injection position be regarding different choices?\n2. Notably, the experiments in this work utilize relatively large sampling steps (>= 100 NFEs). Will the proposed sampler deteriorate at a limited number of function evaluations? How does the conditional generation under CADS change along different sampler steps/NFEs?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This is a good paper for conditional diffusion models. Its merits span the following aspects:\n\n1. The proposed method is intuitive and does not require further finetuning on pretrained diffusion models. This allows for improving a wide range of models and samplers. The straightforward approach should be amenable to the broader audience of ICLR.\n2. This work also provides a theoretical explanation for CADS. I found this explanation helpful. It should be added to the main paper using the extra granted page after acceptance.\n3. Decent experimental results. The proposed sampler offers consistent improvements over the baseline diffusion models.\n4. Detailed ablation studies. The ablation studies clearly show the influence of hyperparameters introduced in this sampler.\n5. Writing clarity. This paper is well presented. The methods and experiment sections are well organized. Please move Appendix C. to the main article.", "weaknesses": "While the paper possesses several strengths, there is room for enhancement in articulating the motivation behind the methodology. The issue of sample diversity has been clearly defined, yet the rationale for the solution could benefit from a stronger motivation. Ideally, the approach should be presented as a natural derivation from the first principle, rather than retroactively justified through theoretical exposition.\n\nThis critique should not be seen as a detriment to the overall quality of the work. It is, in essence, a good paper for improving the conditional information in diffusion models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E8RsaZGlYy", "reviewer": "Reviewer_iK6t", "summary": "Although conditional Diffusion Models have shown impressive performance in good coverage of the real data distribution, it is still limited in covering all the modes of the complex real distribution. In this paper, the authors are proposing a simple but effective sampling method of pretrained Diffusion Models for sampling more diverse results without additional time cost. Comprehensive experiments demonstrate the performance improvements of the proposed method.", "questions": "1. The definition of “diversity” is provided right under Section 3. \nWhat is the meaning of “random seed”? Does it mean the seed of the randomness (e.g., random.seed(1) in Python) or different $x_T$? To me, comparing samples from different seed is less clear than comparing different samples from the fixed seed.\nFor example, we have two $x_T$ sampled from the standard normal, i.e., $x_T^{(0)}$, $x_T^{(1)}$. Let’s say the regular sampling method is $f$ (e.g., DDPM), the proposed sampling method is $g$, and an arbitrary metric of the semantic distance between two images is $h$.\nTo me, the effect of the proposed method would be understandable if $h(f(x_T^{(0)}), f(x_T^{(1)})) < h(g(x_T^{(0)}), g(x_T^{(1)}))$ because we can consider that more modes in the real distribution are covered by the standard gaussian. However, if the noise samples $x_T^{(0)}$, $x_T^{(1)}$ are sampled from different random seed respectively, maybe it's a trivial issue, but it sounds somehow unclear to me. Further clarifications on this point are needed.\n\n2. What is the justification for the higher IS of DDPM across most of $w_{\\text{CFG}}$ in Fig. 5?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Good paper writing. Most parts of the paper is written understandable and reasonable. \n- Simple but effective method.\n- Performance improvement (w.r.t. diversity) in Pose-to-image generation is impressive.\n- Experiments are done comprehensively.\n- The proposed method is theoretically analyzed and also proven to be effective by toy dataset experiment. (in Appendix)", "weaknesses": "1. “Adaptive” may not be an appropriate term. To my knowledge, ‘adaptive’ is used for a method of which parameters are dynamically changed depending on a given input value [1]. Here, $z_t$ is the input I was expecting rather than $t$ since $t$ is a value within a fixed time window.\n\n2. There are a lot of sampling methods including DDIM [2] while only the original DDPM sampler is used to compare. Considering the fact that DDIM is a more widely used sampling method than the original DDPM sampling method, additional comparison with DDIM is needed.\n\n\n[1] Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization, ICCV’17\n[2] Denoising Diffusion Implicit Models, ICLR’21", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BcCTf3siqF", "reviewer": "Reviewer_Djqj", "summary": "In this paper, conditional-annealed diffusion sampler (CADS) is proposed to address the limited diversity of diffusion models with a high classifier-free guidance scale or when trained on small datasets. Specifically, during early inference, the conditional signal is perturbed largely and then restored gradually  during late inference. The proposed method is simple and effective, amplifying the diversity of the generated samples. Besides, it requires minimal computational overhead and easy to implement. In addition, this paper provides extensive experiments on various tasks and ablation studies, validating the effectiveness and novelty of the proposed method.", "questions": "- Empirically, with high guidance weights, the diffusion sampler can produce unnatural images and sometimes even diverges. There is a concern about the proposed method: is it easy to produce bad images?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The proposed method is simple but effective. It is easy to implement and validated with various experiments. Besides, it outperforms a naive and intuitive approach, termed as adaptive CFG, confirming its technical contributions.\n+ To validate the proposed method, this paper provides extensive experiments on various tasks and ablation studies. Thus, it is a solid paper.", "weaknesses": "- In Appendix G, this paper provides sampling and evaluation details, shown in Table 12. The sampling hyperparameters is set deliberately for these experiments. To my knowledge, the training setting is mostly fixed for the same dataset even though it is an ablation study. But, as shown in Table 12, this rule is broken. For example, the setting of ImageNet 256 in Table 1 is different from the setting of ImageNet 256 in Table 6a. How do you set the sampling hyperparameters? \n- As shown in Table 2, there is a missing comparison. The authors should provide the result of DiT-XL/2 with CADS($w_{CFG=1.5}$) or the result of DiT-XL/2 ($w_{CFG=2}$), alleviating the effects of different cfgs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)