Lipschitz Singularities in Diffusion Models
Authors: Zhantao Yang
Ruili Feng
Han Zhang
Yujun Shen
Kai Zhu
Lianghua Huang
Yifei Zhang
Yu Liu
Deli Zhao
Jingren Zhou
Fan Cheng
Venue: ICLR 2024 oral (2024)
Keywords: Image Generation
Generative models
Diffusion models
Primary area: generative models
TL;DR: We theoretically prove and empirically observe the presence of infinite Lipschitz constants near the zero point, and propose a simple but effective approach to address this challenge.

Abstract: Diffusion models, which employ stochastic differential equations to sample images through integrals, have emerged as a dominant class of generative models. However, the rationality of the diffusion process itself receives limited attention, leaving the question of whether the problem is well-posed and well-conditioned. In this paper, we uncover a vexing propensity of diffusion models: they frequently exhibit the infinite Lipschitz near the zero point of timesteps. We provide theoretical proofs to illustrate the presence of infinite Lipschitz constants and empirical results to confirm it. The Lipschitz singularities pose a threat to the stability and accuracy during both the training and inference processes of diffusion models. Therefore, the mitigation of Lipschitz singularities holds great potential for enhancing the performance of diffusion models. To address this challenge, we propose a novel approach, dubbed E-TSDM, which alleviates the Lipschitz singularities of the diffusion model near the zero point. Remarkably, our technique yields a substantial improvement in performance. Moreover, as a byproduct of our method, we achieve a dramatic reduction in the Fréchet Inception Distance of acceleration methods relying on network Lipschitz, including DDIM and DPM-Solver, by over 33\%. Extensive experiments on diverse datasets validate our theory and method. Our work may advance the understanding of the general diffusion process, and also provide insights for the design of diffusion models.

Published as a conference paper at ICLR 2024
LIPSCHITZ SINGULARITIES IN DIFFUSION MODELS
Zhantao Yangl.4
Ruili Feng2.
Han Zhang
Yujun Shen
Kai Zhu
Lianghua Huang
Yifei Zhang 1.4x
Yu Liu
Deli Zhao
Jingren Zhou
Fan Cheng
Shanghai Jiao Tong University
UUniversity of Science and Technology of China
Ant Group
4 *Alibaba Group
{zt.yang 96, ruilifenguste, hzhang9617, shenyujun0302 (atrginnnn002)mmma1a11 gmgmail com
l.s.aiil..tte....c.o (xuangen. srnggen.hihher hih iaaaaiibba-inn com
gidouxiong6199estt..eeo (xy1033e))elllubb--bbeee zhacdeli@gma
Jingren, whogre.. whw.
com chengfanesjtu. edu.
ABSTRACT
Diffusion models, which employ stochastic differential equations to sample im- ages through integrals, have emerged as dominant class of generative models. However, the rationality of the diffusion process itself receives limited attention, leaving the question of whether the problem is well-posed and well-conditioned. In this paper, we explore perplexing tendency of diffusion models: they often display the infinite Lipschitz property of the network with respect time vari- able near the zero point. We provide theoretical proofs to illustrate the presence of infinite Lipschitz constants and empirical results confirm it. The Lipschitz singularities pose a threat to the stability and accuracy during both the training and inference processes of diffusion models. Therefore, the mitigation of Lips- chitz singularities holds great potential for enhancing the performance of diffu- sion models. To address this challenge, we propose novel approach, dubbed E-TSDM, which alleviates the Lipschitz singularities of the diffusion model near the zero point of timesteps. Remarkably, our technique yields a substantial im- provement in performance. Moreover, as byproduct of our method, we achieve a dramatic reduction the Fréchet Inception Distance of acceleration methods relying network Lipschitz, including DDIM and DPM-Solver by over 33%. Extensive experiments on diverse datasets validate our theory and method. Our work may advance the understanding of the general diffusion process, and also provide insights for the design of diffusion models.
INTRODUCTION
The rapid development of diffusion models has been witnessed in image synthesis (Ho et al., 2020; Song et al., 2020; Ramesh et al., 2022; Saharia et al., 2022; Rombach et al., 2022; Zhang & Agrawala, 2023; Hoogeboom et al., 2023) in the past few years. Concretely, diffusion models construct multi-step process to destroy signal by gradually adding noises to it. That way, versing the diffusion process (i.e., denoising) at each step naturally admits a sampling capability. In essence, the sampling process involves solving reverse-time stochastic differential equation (SDE) through integrals (Song et al., 2021b).
Although diffusion models have achieved great success in image synthesis, the rationality of the diffusion process itself has received limited attention, leaving the open question of whether the problem is well-posed and well-conditioned. In this paper, we surprisingly observe that the noise- prediction (Ho et al., 2020) and v-prediction (Salimans & Ho, 2022) diffusion models often exhibit a perplexing tendeney to possess infinite Lipschitz of network with respect time variable near the zero point. We provide theoretical proofs to illustrate the presence of infinite Lipschitz constants
t Corresponding author, Work performed Alibaba Academy, o Project leader

---

Published as a conference paper at ICLR 2024
Different Procedure
Cn
24
E-TSDM
20
DDPM
X
1
Lipschitz Constants
16
Same Procedure
(II) E-TSDM
Xy-1
x
12
100
()) DDPM
XT
XT-1
x
XF
x1
Xta
X
10
100
1000
t=
4
t=t-1
Timesteps
(a) Conceptual comparison
(b) Lipschitz constants
Figure 1: (a) Conceptual comparison between DDPM (Ho et al., 2020) (I) and our proposed early timestep-shared diffusion model (F-TSDM) (II). DDPM trains the network co(:, () with varying timestep conditions í. at each denoising step, whereas E-TSDM uniformly divides the near-zero timestep interval t E [0, with high Lipschitz constants into sub-intervals and shares the condi- tion within each sub-interval. Here, denotes the length of the interval for sharing conditions. When 2 f, E-TSDM follows the same procedure DDPM. However, when f < f, E-TSDM shares timestep conditions. (b) Quantitative comparison of the Lipschitz constants between DDPM and our proposed early timestep-shared diffusion model (E-TSDM). The Lipschitz constants tend to be extremely large near zero point for DDPM. However, our sharing approach allows E-TSMM to be zero, thereby reducing the overall Lipschitz constants in the timestep interval t E [0, E), where i is set as default value 100.
force the Lipschitz constants in each sub-interval
and empirical results to confirm it. Given that noise prediction and v-prediction are widely adopted by popular diffusion models (Dhariwal & Nichol, 2021; Rombach et al., 2022; Ramesh al., 2022; Saharia et al., 2022; Podell et al., 2023), the presence of large Lipschitz constants is a significant problem for the diffusion model community.
Since we uniformly sample timesteps for both training and inference processes, large Lipschitz con- stants w.r.t time variable pose a significant threat both training and inference processes of diffu- sion models. When training, large Lipschitz constants near the zero point affect the training of other parts due to the smooth nature of the network, resulting in instability and inaccuracy. Moreover, since inference requires smooth network for integration, the large Lipschitz constants probably have a substantial impact on accuracy, particularly for faster samplers. Therefore, the mitigation of Lipschitz singularities holds great potential for enhancing the performance of diffusion models.
Fortunately, there is a simple yet effective alternative solution: by sharing the timestep conditions in the interval with large Lipschitz constants, the Lipschitz constants be set zero. Based on this idea, we propose practical approach, which uniformly divides the target interval near the zero point into sub-intervals, and uses the same condition values in each sub-interval, shown in Figure (II), By doing so, this approach can effectively reduce the Lipschitz constants near t = 0 to zero. To validate this idea, we conduct extensive experiments, including unconditional generation on various datasets, acceleration of sampling, and super-resolution task. Both qualitative and quantitative re- sults confirm that our approach substantially alleviates the large Lipschitz constants near zero point and improves the synthesis performance compared to the DDPM baseline (Ho et al., 2020). We also compare this simple approach with other potential methods to address the challenge of large Lip- schitz constants, and find our method outperforms all of these alternative methods. In conclusion, in this work, we thcoretically prove and empirically observe the presence of Lipschitz singularitics issue near the zero point, advancing the understanding of the diffusion process. Besides, propose a simple yet effective approach to address this challenge and achieve impressive improvements.
2 RELATED WORK
The significant advancements of diffusion models have been witnessed in recent years in the domain of image generation. (Karras et al., 2022; Lu et al., 2022b; Dockhorn et al., 2021; Bao et al., 2022b; Lu et al., 2022a; Bao al., 2022a; Zhang et al., 2023). It (Sohl-Diekstein et al., 2015; Ho et al.,
2

---

Published as a conference paper at ICLR 2024
2020: Song al., 2021b) defines a Markovian forward process (x, that gradually destroys the data Xo with Gaussian noise. For any E [0, 7), the conditional distribution go:(xx|xo) satisfies
qo: (x xx) N (xx of
where and oe are referred to as the noise schedule, satisfying a + 1. Generally, ar decreases from to as t increases, to ensure that the marginal distribution of x, gradually changes from the data distribution go to Gaussian. Kingma et al. (2021) further prove that the following stochastic differential equation (SDE) has the same transition distribution qo (x, (xo) as Equation (1) for any t [0, T]:
dx; = f (t) x,dt + g (t) dw:,
Xo ~ q0 (zo)
(2)
where we is the standard Wiener process, f(t dlogge and gt) = 20, log(a,//))
di
dt
Song et al. (2021b) point out that the following reverse-time SDE has the same marginal distribution qt (x(xx) for any / E [0, T]:
dx; = [
Vx, log (xx))dt + g dlw, XT Qr (XT),
where W, a standard Wiener process in the reverse time. Once the score function Vx, log qe (x,) is known, we can simulate Equation (3) for sampling. However, directly learning the score function is problematic, as it involves an explosion of training loss when having a small o,(Snng et al., 202 In practice, the noise prediction model ex(X,,t) t) is often adopted to estimate -o; Vx, log 2: (xx). The network Eo [x, t) can be trained by minimizing the objective:
C (0) == Ex~ xo go (x))) [|lea (a.xo + ote, -
(4)
In this work, our observation of Lipschitz singularities noise-prediction and v-prediction diffusion models reveals the inherent price of such an approach.
Numerical stability near zero point. Achieving numerical stability is essential for high-quality samples in diffusion models, where the sampling process involves solving a reverse-time SDE. Nevertheless, numerical instability is frequently observed near t = 0 in practice (Song et al., 2021a; Vahdat al., 2021). To address this sin- gularity, one possible approach is to set a small non-zero starting time 7 O in both training and inference (Song et al., 2021a; Vahdat et al., 2021).
0.2
0.175 (+40..6%)
DDPM
E-TSDM
Perturbation Error
0.16
0.12
0.090
Kim al. (2022) resolve the trade-off between density estimation and sample generation perfor- mance by introducing randomization to the fixed
(+42.0%%
0.124
0.08
0.026
0.063
T. In contrast, we enhance numerical stability by reducing the Lipschitz constants to zero near t= O. which leads to improved sample quality in diffusion models. It is worth noting that the nu- merical issues observed by aforementioned works are mainly caused by the singularity of transition kernel goe (x, (x)). This transition kernel will de- grade to a Dirac kernel 6(x, x;)p) as o, -> O. However, our observation the infinite Lipschitz constants of the noise prediction model eg (x, t) w.r.t time variable and this is caused by the ex- plosion of das - 0. To the best of our knowl- di edge, this has not been observed before.
0.04
0.0049
1+24.1%
++2.1 %)
0.021
0.0048
0.1
0.2
0.3
Standard Deviation
Figure 2: Quantitative comparison of the er- rors caused by a perturbation on the input be- tween E-TSDM and DDPM (Ho et al., 2020). Results show that E-TSDM is more stable, as its prediction is less affected, e.g., the pertur- bation error of DDPM is 42.0% larger than E- TSDM when the perturbation scale is 0.2.
3 LIPSCHITZ SINGULARITIES IN DIFFUSION MODELS
Lipschitz singularities issue. In this section, we elucidate the vexing propensity of diffusion models to exhibit infinite Lipschitz near the zero point. We achieve this by analyzing the partial derivative θεs(x, t)tOt of the network (x, t). In essence, the emergence of Lipschitz singularities, char- acterized by lim sup
oc, can be attributed to the fact that the prevailing noise schedules conform to the behavior of do co the parameter tends towards zero
dea(x.t)
St
3

---

Published as a conference paper at ICLR 2024
Theoretical analysis. Now we theoretically prove that the infinite Lipschitz happens the point in diffusion models, where the distribution of data is an arbitrary complex distribution. We focus particularly the scenario where the network cx is trained to predict the noises added to images (v-prediction model (Salimans & Ho, 2022) has similar singularity problem, and is analyzed in Appendix C.2). The network eo t) exhibits relationship with the score function Vx log gr (x) that ( t) = - Vx log qu (x) (Song et al., 2021b), where is the standard devi- Nxx a,Xo, oiI). Specifically, ar and oe
ation of the forward transition distribution qor (x| xo) satisfy of + of = 1.
Theorem 3.1 Given noise schedule, since d1 VI-a2 a², we have da,
dre
do.
As t gets
dt
VI-a;
dt
0. close to the noise schedule requires at - L, leading to doz/dt oo as long as The partial derivative of the network can be written as
dag
1=0
deg (x, t) 01
a1 doa
Vx log qt (x) 8V 1xllogo qe (x)
01-
(5)
VI - of dt
δt
Note that at + 1 as t + 0, thus if following two must stand
dag
l + 0, and Vx log qe (x) * O, then one of the
dt
Deg (x
0V x log qi (x) Te 0c.
lim sup 6-0+
+ 0o;
lim sup 6-+
(6)
Ot
Note that das dor 1-- f stands for a wide range of noise schedules, including linear, cosine, and quadratic schedules (see details in Appendix C.I). Besides, we can safely assume that ze (x(x) is smooth process. Therefore, we may often have lim sup (x,((x)) || - co, indicating the infinite Lipschitz constants around = O.
Simple case illustration. Take simple that the distribution of data p(xo) (O, for instance, the score function for any / E [0, T] can be written as
1
X 2 2
Vx log (x) = Vx log
exp
-xx.
2m
Due to the relationship EØ (x t) oo vx log q and the fact that the deviation des tends toward 0o O, have (x.t) co.
de
Case in reality. After theoretically proving that diffusion models suffer infinite Lipschitz near the zero point, we show it empirically. We estimate the Lipschitz constants of a network by
K(I, Ex. Illeo (xe, t) co (x,, +2)]l2]]
(8)
At
where At = ft t'). For a network Es (x,, t') of DDPM baseline (Ho et al., 2020) trained on FFHQ 256 256 (Karras al., 2019) (see training details in Section 5.1 and more results of the Lips- chitz constants K(t, t') on other datasets in Appendix D.1), the variation of the Lipschitz constants K(t,t') t') the noise level varies is seen in Figure 1b, showing that the Lipschitz constants K(t, t') get extremely large in the interval with low noise levels. Such large Lipschitz constants support the above theoretical analysis and pose threat to the stability and accuracy of the diffusion process, which relies on integral operations.
4 MITIGATING LIPSCHITZ SINGULARITIES BY SHARING CONDITIONS
Proposed method. In this section, we propose the Early Timestep-shared Diffusion Model (E- TSDM), which aims to alleviate the Lipschitz singularities by sharing the timestep conditions the interval with large Lipschitz constants. To avoid impairing the network's 's ability, E-TSDM performs a stepwise operation of sharing timestep condition values. Specifically, we consider the interval near the zero point suffering from large Lipschitz constants, denoted as [0, t), where indicates the length of the target interval. E-TSDM uniformly divides this interval into n sub-intervals represented as
sequence T (lo.4n (,, (n), where 0 < t1
< ln f and t to 4

---

Published as a conference paper at ICLR 2024
4096
10
9:50
9.18
E-TSDM
Baseline
26,84
12.00
Modified-NS
E-TSDM
NLTTL M::0
9,40
Regularization
256
171.06
FID-100 10k
7.97
17.48
83.2185.4
6.99
15
13.79
6.62
10
8.67
9.50
16
9.43
9:53
5
256x266
256x256
CelebAHQ
Cosine
Quadratic
Linear
1/
Inverse-Sigmodd
Dutaset
Noise Schedule
Remap Function
(a) Regularization
(b) Modification of noise schedules
(c) Remap
Figure 3: Quantitative analysis of alternative methods evaluated with FID-IOK L. (a) Regulariza-
tion: Experimental results FFHQ 256 x 256 and CelebAHQ 256 256 show that regularization techniques can slightly improve the FID of DDPM (Ho et al., 2020) baseline but performs worse than E-TSDM (b) Modification of noise schedules (Modified-NS), We implement Modified-NS on linear, quadratic, and cosine schedules. Experimental results on FFHQ 256 x 256 dataset indicate that the performance of Modified-NS unstable while E-TSDM achieves better synthesis per- formance. (c) Remap: Quantitative comparison of remap method between uniformly sampling t and uniformly sampling X, during training and inference, on FFHQ 256 x 256. Specifically, U, (0, 1). and 14x is (0, K] for 1/L but U[,K, K] for Inverse-Signoid, where K is large number to avoid infinity. (T) represents the sampling strategy during the training process while (I) represents that during the inference process. Results show that remap is not helpful.
4-1, Vi = 1.2..... n. For each sub-interval, E-TSDM employs a single timestep value (the left endpoint of the sub-interval) the condition, both during training and inference. Utilizing this strategy, E-TSDM effectively enforces zero Lipschitz constants within each sub-interval, with only the timesteps located near the boundaries of the sub-intervals having Lipschitz constant greater than zero. As a result, the overall Lipschitz constants of the target interval E [0, 2) are significantly reduced. The corresponding training loss can be written as
L (co): == EE~U( (T) xorl) (x0) [llco (a,xo + oee, Jr (f)) - <1]]]
where fr(t) maxisisn (1i1 T 4,-1 < t for t E. while fr(t) t for t E. The corresponding reverse process can be represented as
po (xI- xx1) =N Xt-1i
a1-1 xx
3 o,
co (x,, (t)) mn/
(10)
az
01 and ni = Br. E-TSDM easy implement, and the algorithm details are ai provided in Appendix B.2.
where B:
Analysis of estimation error. Then we show that the estimation error of E-TSDM can be bounded by an infinitesimal, and thus the impact of E-TSDM on the estimation accuracy is insignificant. The detailed proof is shown in Appendix C.3.
Theorem 4.1 Given the chosen fr(t), when t E [0, f). the difference between the optimal Ex fr(t)) denoted as e* (x, fr(t)), and E(x,t) t) -o, log (x),, can be bounded by
||c (x. (t)) (x,,t)] S o,K (x) At + B (x) Admax.
(11)
where
V log ai (x)
Vx log (x)
K (x) sup
B (x) sup || log (x) ll.
(12)
t - T|
and Ad = mmmmaaissss J0,, - -Ot-2l- Note that K K(x) and B(x) are finite and lim Admax =
0 for any continuous where At:=i/n, thus the difference converges to 0 as At - 0. Furthermore, the rate of convergence least } orrde with respect to At.
The oodder convergence rate is relatively fast in optimization. Given this bound, we think the introduced emors of E-TSDM controllable.
5

---

Published as a conference paper at ICLR 2024
16
DDPM
E-TSDM
Table 1: Quantitative comparison on FFHQ
14
1.98
(Karras et al., 2019). denotes our reproduced result with the same network as E-TSDM-large.
12
10
9.64
9.5
7-31
/663
7.34
6.62
$.99
7,01
16:72
Model
FID-50k 1
StylGGAN++ADA (Karras al., 2020)
3.62
Soft-Truncation (Kim et al., 2021)
5.54
P2-DM (Choi et al., 2022)
6.92
AFHQ- Cat
AFBQ- Wild
FFHQ CelehAHQ
Lsun-
Laun-
LDM (Rombach et al., 2022)
4.98
Church
DDPM (Ho et al., 2020)
6.88
Figure 4: Quantitative comparison on var- ious datasets with 256 x 256 resolution. All experiments are evaluated with FID-10k 4.
E-TSDM (ours)
5,21
E-TSDM-large (ours)
4.22
Reduction in Lipschitz constants. In Figure 1b, we present the curve of K(t, t') of E-TSDM on FFHQ 256 x 256 (Karras et al., 2019) (we provide results for continuous-time diffusion models and more results on other datasets in Appendix D.I), showing that the Lipschitz constants K K(,,') are significantly reduced by applying E-TSDM.
Improvement in stability. To further verify the stability of E-TSDM, we evaluate the impact of small perturbation added the input. Specifically, we add a small noise with a growing scale the X;, where is set to default value of 100, and observe the resulting difference in the predicted value of Xu, for both E-TSDM and baseline. Our results, as shown in Figure 2, illustrate that E-TSDM exhibits better stability than the baseline, as its predictions are less affected by perturbations.
Comparison with some alternative methods. Although achieving impressive performance as de- tailed in Section 5, E-TSDM introduces no modifications to the network architecture or loss function, thereby not incurring any additional computational cost. 1) Regularization: In contrast, an alter- native potential approach is imposing restrictions on the Lipschitz constants via regularization tech- niques. It necessitates the computation of (x.()) consequently diminishing training efficiency. 2) Modification of noise schedules: Furthermore, E-TSDM preserves the forward process unaltered. Conversely, another potential method involves the modification of noise schedules. Recall that the issue of Lipschitz singularities only arises when the noise schedule satisties das
dt lt= * 0. There- da.
fore, becomes feasible to adjust the noise schedule to meet the requirement dt lt=0 = 0, thus mitigating the problem of Lipschitz singularities. The detailed methods for modifying noise sched- ules are provided in Appendix D.3.2. Although this modification seems feasible, results in tiny amounts of noise the beginning stages of the diffusion process, leading to inaccurate predictions.
3) Remap: In addition, remap is another possible method, which designs a remap function À = f(t) the conditional input of the network, namely, < f()). By carefully designing it can significantly stretch the interval with large Lipschitz constants. For example, f(t) = 1/t
f()).
and (A) sigmoid(A) are two simple choices. In this way, Remap can efficiently reduce the
Lipschitz constants regarding the conditional inputs of the network, @ea(x.() uniformly sample i both in training and inference, what should be focused is the Lipschitz con- stants regarding t. Des (x.f) which can not be influenced by remap. We also consider the situation of uniformly sampling X, which can significantly hurt the quality of generated images. We show the quantitative evaluation in Figure 3 and put the detailed analysis in Appendix D.3.3. Empirically, E-TSDM surpasses not only the baseline but also all of these alternative methods, where the results are demonstrated in Figure 3. For a more in-depth discussions, please refer to Section D.3.
However, since we
DA
Ør.
EXPERIMENTS
In this section, we present compelling evidence that our E-TSDM outperforms existing approaches on a variety of datasets. To achieve this, we first detail the experimental setup used in our studies in Section 5.1. Subsequently, in Section 5.2, compare the synthesis performance of E-TSDM against that of the baseline on various datasets. In Section 5.3, we conduct multiple ablation studies and quantitative analysis from two perspectives. Firstly, we demonstrate the generalizability of

---

Published as a conference paper at ICLR 2024
Table 2: Quantitative comparison between DDPM baseline (Ho al., 2020) and our proposed E-TSDM on both discrete-time and continuous-time scenarios with different noise schedules, on FFHQ 256 x 256 (Karras et al., 2019) using FID- 10k as the evaluation metric. Experimental results illustrate that E-TSDM can be generalized to other noise schedules and is still effective in the context of continuous-time diffusion models.
Noise schedule
Settings
Method
Linear
Quadratic
Cosine
Cosine-shift
Zero- Peoo-termina--NN
Baseline E- E-DDMM
9.50
13.79
27.17
14.51
11.66
Discrete
6.62
9.69
26.08
11.20
8.58
Continuous
Baseline
9.53
14.26
25.65
12.80
10.89
E-TSDM
6.95
9.66
16.80
9.94
8.96
E-TSDM by implementing it on continuous-time diffusion models and varying the noise schedules. Secondly, we investigate the impact of varying the number of conditions n in E [0, 8) and the length of the interval f, which are important hyperparameters. Moreover, we demonstrate in Section 5.4 that our method can be effectively combined with popular fast sampling techniques. Finally, we show that E-TSDM can be applied to conditional generation tasks, such as super-resolution, Section 5.5.
5.1 EXPERIMENTAL SETUP
Implementation details. All of our experiments utilize the settings of DDPM (Ho et al., 2020) (see more details in Appendix B.1). Besides, utilize more developed structure of unet (Dhariwal & Nichol, 2021) than that of DDPM (Ho et al., 2020) for all experiments containing reproduced baseline. Given that the model size is kept constant, the speed and memory requirements for training and inference for both the baseline and E-TSDM are the same. Except for the ablation studies in Section 5.3, all other experiments fix i 100 for E-TSDM and five conditions (n = 5) in the interval t E [0, i), which we have found to be relatively good choice in practice. Furthermore, all experiments are trained on NVIDIA A100 GPUs. Datasets. We implement E-TSDM on several widely evaluated datasets containing FFHQ 256 x 256 (Karras et al., 2019), CelebAHQ 256 x 256 (Karras et al., 2017), AFHQ-Cat 256 x 256, AFHQ- Wild 256 x 256 (Choi et al., 2020). Lsun- Church 256 x 256 and Lsun-Cat 256 x 256 (Yu et al., 2015). Evaluation metries. To assess the sampling quality of E-TSDM, we utilize the widely-adopted Frechet inception distance (FID) metric (Heusel et al., 2017). Additionally, we use the peak signal-to-noise ratio (PSNR) to evaluate the performance of the super-resolution task.
5.2 SYNTHESIS PERFORMANCE
We have demonstrated that E- E-TSDM can effectively mitigate the large Lipschitz constants near f in Figure b. as detailed in Section 4. In this section, we conduct a comprehensive comparison between E-TSDM and DDPM baseline (Ho et al., 2020) on various datasets to show that E-TSDM improve the synthesis performance. The quantitative comparison is presented in Figure which clearly illustrates that E-TSDM outperforms the baseline on all evaluated datasets. Furthermore, as various datasets demonstrate its
depicted in Appendix D.5, the samples generated by E-TSDM ability to generate high-fidelity images. Remarkably, to the best of our knowledge, as shown in Table we set new state-of -fate-of-the-art benchmark for diffusion models on FFHQ 256 x 256 (Karras et al., 2019) using a large version of our approach (see details in Appendix B.
5.3 QUANTITATIVE ANALYSIS
In this section, we demonstrate the generalizability of E-TSDM by implementing it on continuous- time diffusion models and varying the noise schedules. In addition, to gain a deeper understanding of the properties of E-TSDM, we investigate the critical hyperparameters of E-TSDM by varying the length the interval to share the timestep conditions, and the number of sub-intervals n.
5.3.1 QUANTITATIVE ANALYSIS ON THE GENERALIZABILITY OF E-TSDM
To ensure the generalizability of E-TSDM beyond specific settings of DDPM (Ho et al., 2020), we conduct a thorough investigation of E-TSDM other popular noise schedules, as well implement
7

---

Published as a conference paper at ICLR 2024
10
10
9.5
9.5p
FFHQ 256 256
FFHQ 256 256
9
CelebAHQ 256 256
CelebAHQ 256 x 256
FID-100 10k
8
7.64
8.05
8.05
6.85
6.99
6.91
7.23
7.01
7.33
6.99
6.99
7.05
6.85
7
6.87
6.6
6.67
6.96
6.77
7.04
7.09
6,89
6.89
6.67
1
6.85
6.75
6.67
6
0
20
100
150
200
250
300
2
5
10
20
50
100
(a) Interval Length I
(b) Timestep Number n
Figure 5: Ablation study the length of the interval [0, to share the timestep conditions, E, and the number of sub-intervals in this interval, n, using FID-10k J as the evaluation metric. We repeat each experiment three times and provide the error bars.
Table 3: Quantitative comparison on FFHQ 256 x 256 (Karras et al., 2019) between DDPM (Ho al., 2020) and our proposed E-TSDM utilizing different fast samplers, DDIM (Song et al., 2020) and DPM-Solver (Lu et al., 2022b), varying the number of function evalutaions (NFE). FID-10K, is used as the evaluation metric, and 3PM--olver-k represents the k-th-order DPM-Solver. Results indicate that our approach well supports the popular fast samplers.
Fast Samplers
DPM-Solver-3
DPM-Solver-2
DDIM
NFE
20
50
20
50
50
200
DDPM
21.91
24.48
22.21
24.80
21.80
23.16
Method
E-TSDM
16.97
13.52
17.28
14.14
19.34
13.71
a continuous-time version of E-TSDM. Specifically, we explore the three popular ones including linear, quadratic and cosine schedules (Nichol & Dhariwal, 2021), and two newly proposed ones, which are cosine-shift (Hoogeboom et al., 2023) and zero-terminal-SNN (Lin et al., 2023) schedules.
As shown in Table 2, our E-TSDM achieves excellent performance across different noise schedules. Besides, the comparison of Lipschitz constants between E-TSDM and baseline on different noise schedules, illustrated in Appendix 1.1, show that E-TSDM mitigate the singulariis issue besides the scenario of the linear schedule, highlighting that its effects are independent of the specific noise schedule. Additionally, the continuous-time version of E-TSDM outperforms the corresponding baseline, indicating that E-TSDM is effective for both continuous-time and discrete- time diffusion models. We provide the curves of the Lipschitz constants K (t, t') in Figure A1 to compare continuous time E- E-TSDM with its baseline on the linear schedule, showing that E-TSDM can mitigate Lipschitz singularities the continuous-time scenario.
5.3.2 QUANTITATIVE ANALYSTS ON AND i.
E-TSDM involves dividing the target interval E [0, 8) with large Lipschitz constants into n sub- intervals and sharing timestep conditions within each sub-interval. Accordingly, the choices of and have significant impacts on the performance of E-TSDM. Intuitively, i should be relatively small value, therefore representing an interval near zero point. As for n, it should not be too large or too small. If n is too small, it forces the network to adapt to too many noise levels with single timestep condition, thus leading to inaccuracy. Conversely, if the value n is set too large, the reduction of Lipschitz constants is insufficient, where the extreme situation is baseline.
In this section, we meticulously assess the impacts of f and n on various datasets. We present the outcomes on FFHQ 256 x 256 (Karras et al., 2019 10 and CelebAHQ 256 x 256 (Karras et al., 2017) for each hyperparameter in Figure 5, while leaving the remaining results in Appendix D.2. Specifically, in the experiments of E, we maintain the length of each sub-interval, namely, i/n, unchanged, while in the experiments of that E-TSDM performs well when é is relatively small. However, as f increases, the performance of E-TSDM deteriorates gradually. Furthermore, the results for shown in Figure 5 b. from
maintain the & unchanged. The results for f in Figure a demonstrate
8

---

Published as a conference paper at ICLR 2024
which we observe rise in FID when n too small, for instance, when = 2. Conversely, when n is too large, such as n = 100, the performance deteriorates significantly. Although E-TSDM performs well for most and values, considering the results all of the evaluated datasets (see remaining results in Appendix D.2), n = 5 and = 100 are recommended to be good choices to avoid cumbersome searches a good starting point for further exploration when applying B-TSDM.
5.4 FAST SAMPLING
With the development of fast sampling algorithms, it is crucial that E-TSDM can be effectively combined with classic fast samplers, such as DDIM (Song et al., 2020) and DPM-Solver (Lu al., 2022b). To this end, we incorporate both DDIM (Song et al., 2020) and DPM-Solver (Lu et al., 2022b) into E-TSDM for fast sampling in this section. It is worth noting that the presence of large Lipschitz constants can have more detrimental impact on the efficiency of fast sampling compared to full-timestep sampling, as numerical solvers typically depend on the similarity between function values and their derivatives on adjacent steps. When using fast sampling algorithms with larger discretization steps, it becomes necessary for the functions to exhibit better smoothness, which in turn corresponds smaller Lipschitz constants. Hence, it is anticipated that the utilization of E- TSDM will lead to an improvement in the generation performance of fast sampling methods.
As presented in Table 3, we observe that E-TSDM significantly outperforms the baseline when using the same number of function evaluations (NFE) for fast sampling, which is under expectation. Be- sides, the advantage of E-TSDM becomes more pronounced when using higher order sampler (from DDIM to DPM-Solver), indicating better smoothness when compared to the baseline. Notably, for both DDIM and DPM-Solver, we observe an abnormal phenomenon for baseline, whereby the per- formance deteriorates as NFE increases. This phenomenon has been previously noted by several works (Lu et al., 2022b;c; Li et al., 2023), but remains unexplained. Given that this phenomenon is not observed in E-TSDM, we hypothesize that it may be related to the improvement of smoothness of the learned network. We leave further verification of this hypothesis for future work.
5.5 CONDITIONAL GENERATION
In order to explore the potential for extending E-TSDM to conditional generation tasks, we further investigate its performance in the super-resolution task, which is one of the most popular conditional generation tasks. Specifically, we test E-TSDM on the FFHQ 256 x 256 dataset, using the 64 64 -> 256 x 256 pixel resolution our experimental settings. For the baseline in the super-resolution task, we utilize the same network structure and typer-parameter as those employed in the baseline presented in Section 5.1, but incorporate a low-resolution image as additional input. Besides, for E-TSDM, we adopt a general setting with n = and = 100. As illustrated Figure A12, we observe that the baseline tends to exhibit color bias compared real images, which is mitigated by E-TSDM. Quantitatively, our results indicate that F-TSDDM outperforms the baseline on the test set, achieving improvement in PSNR from 24.64 25.61. These findings suggest that E-TSDM holds considerable promise for application in conditional generation tasks.
6 CONCLUSION
In this paper, we elaborate on the infinite Lipschitz of the diffusion model near the zero point from both theoretical and empirical perspectives, which hurts the stability and accuracy of the diffu- sion process. A novel E-TSDM is further proposed to mitigate the corresponding singularities in a -harring- manner. Experimental results demonstrate the superiority of our method in both performance and adaptability to the baselines, including unconditional generation, conditional gen- eration, and fast sampling. This paper may not only improve the performance of diffusion models, but also help to make up the critical research gap in the understanding of the rationality underlying the diffusion process.
Limitations. Although B-TSDM has demonstrated significant improvements in various applica- tions, it has yet to be verified on large-scale text-to-image generative models. As E-TSDM reduces the large Lipschitz constants by sharing conditions, it is possible lead to decrease in the effec- tiveness of lcrge-scale generative models. Additionally, the reduction of Lipschitz constants to zero within each sub-interval in B-TSDM may introduce unknown and potentially harmful effects.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We would like to thank the four anonymous reviewers for spending time and effort and bringing in constructive questions and suggestions, which helped greatly to improve the quality of the paper. We would like to also thank the Program Chairs and Area Chairs for handling this paper and providing valuable and comprehensive comments. In addition, this research was funded by the Alibaba Innovative Research (AIR) project.
REFERENCES
Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, and Bo Zhang. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. arXiv preprini arXiv:2206.07300 07309, 2022a.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dmm: analytic estimate of the optimal reverse variance diffusion probabilistic models. arXiv preprint arXiv:2201.06500 2022b.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018
Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In IEEE Conf. Comput. Vis. Pattern Recog,, 2022.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Adv Neural Inform. Process. Syst., 2021.
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically- damped langevin diffusion. arXiv preprint 1.Xiv:2/12.07066 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge local nash equilibrium. Adv. Neural Inform. Process. Syst., 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Adv. Neural Inform. Process. Syst., 2020.
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. arXiv preprint arXiv:2301. 2023.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv: 100.:9990.0096 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Adv. Neural Inform. Process. Syst., 2020.
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. arXiv preprint arXiv:2206.00336 003664 2022.
Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and II-Chul Moon. Soft truncation: A universal training technique of score-based diffusion model for high precision score estimation. arXiv preprint arXiv:2106.05522 2021.
Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and II-Chul Moon. Soft truncation: A universal training technique of score-based diffusion model for high precision score estimation. In Int. Conf. Mach. Learn., 2022.
Diederík Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Adv Neural Inform. Process. Syst., 2021.
10

---

Published as a conference paper at ICLR 2024
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Shengmeng Li, Luping Liu, Zenghao Chai, Runnan Li, and Xu Tan. Era-solver: Error-robust adams solver for fast sampling diffusion probabilistic models, arXiv preprint arX1::2301.129:: 2023.
Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. arXiv preprint 0rXiv:2305.0889) 2023.
Cheng Lu, Kaiwen Zheng, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Maximum likelihood training for score-based diffusion odes by high order denoising score matching. In Int. Conf. Mach. Learn., 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint 2rXx::2206.00927,, 2022b.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++4 Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint 220i::22211000995 2022c.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Int. Conf. Mach. Learn., 2021.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhor, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01955, 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text- conditional image generation with clip latents. arXiv preprint vrXiv:2204.06/25 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint 1rXi::2205.11487 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.0051/1 2022.
Jascha Sohl-Dicken Dickkiisst,in, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Int. Conf. Mach. Learn., 2015.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training score- based diffusion models. Adv. Neural Inform. Process. Syst. 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. Int. Conf. Learn. Represent.,200 202
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Adv. Neural Inform. Process. Syst., 2021.
Fisher Yu, Ari Seff. Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 750 003660.36, 2015.
11

---

Published as a conference paper at ICLR 2024
Han Zhang, Ruili Feng, Zhantao Yang, Lianghua Huang, Yu Liu, Yifei Zhang, Yujun Shen, Deli Zhao, Jingren Zhou, and Fan Cheng. Dimensionnalit--aryinnn diffusion process. In IEEE Conf. Comput. Vis. Pattern Recog., 2023.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543 2023.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
A OVERVIEW
This supplementary material is organized as follows. First, facilitate the reproducibility of our experiments, we present implementation details, including haper-parameters in Appendix B. and algorithmic details in Appendix B.2. Next, in Appendix C, we provide all details of deduction involved in the main paper. Finally, we present additional experimental results in support of the effectiveness of E-TSDM.
B IMPLEMENTATION DETAILS
B.1 HYPER-PARE PARAMETERS
The hyper-panne ppper-prrameters used in experiments shown in Table AI, and use identical hyper- parameters for all evaluated datasets for both E-TSDM and their corresponding baselines. Specifi- cally, we follow the hyper-pann prper-pprameters of DDPM (Ho al., 2020) but adopt more advanced struc- ture of U-Net (Dhariwal & Nichol, 2021) with residual blocks from BigGAN (Brock et al., 2018). The network employs block consisting of fully connected layers to encode the timestep, where the dimensionality of hidden layers for this block is determined by the timestep channels shown in Table A1. Moreover, we scale up the network to achieve the tfate-of-the-ar art results of diffusion
Table A1: Hyper-parameters of E-TSDM and our reproduced baseline.
Normal version
Large version
T
1,000
1,000
B,
linear
linear
Model size
131M
692M
Base channels
128
128
Channels multiple
(1,1,2,2,4,4)
(1,1,2,4.6,8)
Heads channels
64
64
Self attention
32,16,8
32,64,8
Timestep channels
512
2048
BigGAN block
Dropout
0.0
0.0
Learning rate
le-4
le-1
Batch size
96
64
Res blocks
4
EMA
0.9999
0.9999
Warmup steps
0
Gradient clip
X
models on FFHQ 256 256 (Karras et al., 2019), and therefore we provide the hyper-parameterr of the large version of E-TSDM in Table A1.
B.2 ALGORITHM DETAILS
In this section, we provide a detailed description of the E-TSDM algorithm, including the training and inference procedures shown in Algorithm Al and Algorithm A2, respectively. Our method is simple to implement and requires only a few steps. Firstly, a suitable length of the interval should be selected for sharing conditions, along with the corresponding number of timestep conditions in the target interval E [0, f). While performing thorough search for different datasets can achieve better performance, the default settings = 100 and n = 5 are recommended when E-TSDM is applied without a thorough search.
Next, the target interval t E [0, f) should be divided into n sub-intervals, and the boundaries for each sub-interval should be calculated generate the partition schedule T (l,, .1,...,,,. ).). Finally,
13

---

Published as a conference paper at ICLR 2024
Algorithm A1 Training of B-TSDM
Require: The length of the target interval
Require: The number of conditions
Require: Model Eg to be trained.
Require: Data set D.
1:
Uniformly divide the target interval t E [0. 2) into n sub-intervals to get the corresponding timestep partition schedule T (to, , tn).
2: repeat
3: Xo ~D
4: t~ Uniform ([.....,T) .T))
5:
if < then
6:
i= maxisis E S
7:
else
8:
i-t end if IO.1
9:
11: Take gradient descent step on Volle ep (04x0 + 016, 1)
13:
until converged
Algorithm A2 Sampling of E-TSDM
Require: The length of the target interval
Require: The number of conditions
Require: A trained model ea.
1: Uniformly divide the target interval t (0, into sub-intervals to get the corresponding timestep partition schedule = (to, 11, ti,..,,tt) tn).
2: Xy ~ I)
3: for
=T mart=T.... 1 do i < i then
4:
5:
i= maxi (i- TT: 4-1 $
6:
else
7:
==t end
8:
9:
i > then
I
11:
else
12:
z 0
13:
end if
14: X1-1 ae-1 xx
Ps cx(( 0))
11
as
15:
end for
return Xo
during both training and sampling, the corresponding left boundary for each timestep in the target interval / E [0, should be determined according T, and used as the conditional input of the network instead of l.
C DERIVATION OF FORMULAS
In this section, we provide detailed derivations as a supplement to the main paper. The derivations divided into three parts, firstly we prove that the key assumption of the occurrence of Lipschitz singularities, dt -= + O, holds for mainstream noise schedules including linear, quadratic, and cosine schedules. Therefore, all of the diffusion models utilizing these noise schedules suffer from the issue of Lipschitz singularities. Then we show that Lipschitz singularities also plague the v- prediction (Salimans & Ho, 2022) models. Considering that most of the diffusion models are noise- prediction v-prediction models, the Lipschitz singularities problem is important issue for the
day
14

---

Published as a conference paper at ICLR 2024
24
B-TSDM
20
DDPM
Lipschitz Constants
16
12
100
8
4
100
1000
Timestep
Figure Al: Quantitative comparison of the Lipschitz constants between continuous-time E-
TSDM and continuous-time DDPM (Ho al., 2020). Results show that E-TSDM can efficiently reduce the Lipschitz constants in continuous-timg scenarios.
community of diffusion models. Finally, we demonstrate the detailed derivation Theorem 4.1, showing that the errors introduced by E-TSDM can be bounded by an infinitesimal and thus insignificant.
C.I dat/dt FOR WIDELY USED NOISE SCHEDULES AT ZERO POINT
We have already shown that for an arbitrary complex distribution, given a noise schedule, dos. de t= * 0, then we often have lim sup,- (e.(x.() - oo, indicating the infinite Lipschitz constants around t O. In this section, prove that di Ic-o * 0 stands for three mainstream noise schedules including linear schedule, quadratic schedule and cosine schedule.
Øt
da,
C.1.1
dat/dt FOR LINEAR AND QUADRATIC SCHEDULES AT ZERO POINT
Linear and quadratic schedules are first proposed by Ho et al. (2020). Both of them determine {o by a pre-designed positive sequence {3,) and the relationship a₂ == II: /T Note that {1, 2, .... T} is a discrete index, and {= {B,}%= are discrete parameter se- quences in DDPM. However, ar in do./dd /d refers to the continuous-time parameter determined by
B
the following score SDE (Song et al., 2021b)
dx(r) = - -())((() BB(]dw, r e [0, 1],
(A1)
where w is the standard Wiener process, (()) is the continuous version of {B:)T=] with a continuous time variable - E [0, for indexing, and the continuous-time ambiguity, let (T), [0, denote the continuous version of {a,)T= Thus,
exp (- So (())ds). To avoid
da(7) dr
1 (7) exp (- [ ((s)ds) 1=0
-
= = 3-)
3())
(A2)
Once the continuous function ((r) is determined for specific noise schedule, we can obtain do(r)
immediately by Equation (A2).
r==0
To obtain 3(7), we first give the expression of (3.) linear and quadratic schedules (Ho al., 2020)
Buii
Braax T
imin T
1
Linear: Br
(A3)
T
T
l'
(V
(v
3max T
Bmin
t-
Quadratic: B,
(A4)
T
T
T-1
T
15

---

Published as a conference paper at ICLR 2024
10
DDPM E-TSDM
DDPM E-TSDM
3
Lipschitz Constants
8
Lipschitz Constants
2
2
6
1
6
1
4
0
4
0
100
100
2
90
1
100
1000
100
1000
(a) AFHQ-Cat 256 x 256
(b) AFHQ- Wild 256 256
25
DDPM
7.5
15
DDPM E-TSDM
Lipschitz Constants
E-TSDM
Lipschitz Constants
20
5.0
4
15
2.5
10
2
0.0
0
10
100
100
5
90
9
100
1000
100
1000
(c) Lsun-Cat 256 x 256
(d) Lsun-Church 256 x 256
1.5
20
DDPM
DDPM E-TSDM
15
Lipschitz Constants
E-TSDM
Lipschitz Constants
1.0
15
2
10
0.5
10
0.0
100
100
5
5
9
9
100
1000
100
1000
(c) CelebAHQ 256 x 256
(f) FFHQ 256 x 256 using
quadratic schedule
Figure A2: Quantitative comparison of Lipschitz constants between F-TSDM and DDPM base- line (Ho al., 2020) on various datasets, including (a) AFHQ-Cat (Choi al., 2020), (b) AFHQ-Wii FilQ-Wild (Choi et al., 2020), (c) Lsun-Cat 256 x 256 (Karras et al., 2019), (d) Lsun-Church 256 256 (Karras et al., 2019), and (e) CelebAHQ 256 x 256 (Karras al., 2017) using the lin- ear schedule. (f) Quantitative comparison of Lipschitz constants between E-TSDM and DDPM baseline (Ho et al., 2020) on FFHQ 256 x 256 (Karras et al., 2019) using the quadratic schedule.
where Buin and Bmax are user-defined hyperparameters. Then, we define an auxiliary sequence {3. = T In the limit of T + xo, (3:)-=1 becomes the function B(T) indexed by [0, 1]
Linear: B(T) = =3min + (Bmax - Bmia) T,
(A5)
Quadratic: B(T) = Beain + (V V Amin .7)
+)
2
(A6)
Thus, 3(0) Bmin for both linear and quadratic schedules, which leads do(r))
Bmin-
r=
As common setting, Brin a positive real number, thus da(r)
<
dr 1==
16

---

Published as a conference paper at ICLR 2024
16
- AFHQ-Cat
11.74
- Lsun-Car
LssunCCurch
11
15
256
256
FID-100 10k
10
14
7.5
7.49
7.26
13
8.43
7.01
12
12:46
6.72
7.19
6.52
6.63
7.04
11
11.9811.84
11.93
6.38
6.07
10
5.5
200 250
300
150 200 250 300
100 150 200 250 300 (c) Interval Length
Interval Length
Interval Length
(a) AFHQ-Cat 256 256
(b) LSUN-Cat 256
256
256 256
Figure A3: Ablation study on the length of the interval E [0, 2) to share the timestep conditions, using FID-10K & the evaluation metric.
AFHQ-Car 256 256
- Lsun-Cal 256
- Laun-Church
10
10.18
256
8.5
14.69
9
14
8
FID-10K
8.02
7.5
8.24
7
7.4
7.51
6.63
7.18
7.19
11
11.77
11.98
11.7
11.8811.81
5.5
7.04
6.62
6.72
7.01
5.92
10.97
10
20
50
100
50
100
50
100
# Sub-intervals
# Sub-intervals
# Sub-intervals
(a) AFHQ-Cat 256
256
(b) LSUN-Cat 256
256
(c) LSUN-Church 256
256
Figure A4: Ablation study on the number of sub-intervals in this interval, using FID-10k 4 as the evaluation metric.
C.1.2 / /tt FOR THE COSINE SCHEDULE AT ZERO POINT
The cosine schedule is designed to prevent abrupt changes in noise level near = 0 and (Nichol& & Dhariwal, 2021). Different from linear and quadratic schedules that define {ar]T= by pre-designed sequence {Bs 17 the cosine schedule directly defines )T
{B:)T
f(t) f(0)'
f(t) = cos 1/TT (1/++..
04
1+s ) t= 1,2, ... ,T.
(A7)
where s a small positive offset. The continuous version of (ar )T can be obtained in the limit of T xc as
a(T) = cos
T + π / cos
8 π 2).
T TE [0, 1].
(A8)
1 1 2
1
With Equation (A8), we can easily get da(z)
dr
==
da(r))
π
tan
(A9)
dr
1+ s 2 2(1 + s)
r=0
which leads to da(r)
since >
dr
C.2 LIPSCHITZ SINGULARIES V-PREDICTION DIFFUSION MODELS
In Section 3 of the main paper, prove that noise- prediction diffusion models suffer from Lipschitz singularities issue. this section, we show that the Lipschitz singularities issue is also an important problem for v-prediction diffusion models from both theoretical and empirical perspectives.
17

---

Published as a conference paper at ICLR 2024
25
25
DDPM E-TSDM
DDPM E-TSDM
3
Lipschitz Constants
20
Lipschitz Constants
20
2
15
2
15
10
10
0
1000
100
90
9
100
1000
100
1000
Timestep
Timestep
(a) LSUN-Cat 256 x 256
(b) FFHQ 256 x 256
Figure A5: Quantitative comparison of the Lipschitz constants between E-TSDM and DDPM (Ho al., 2020) using v-prediction (Salimans Ho, 2022) on Lsun-Cat 256 256 (Karras al., 2019) and FFHQ 256 x 256 dataset (Karras et al., 2019). Results show that E-TSDM can efficiently reduce the Lipschitz constants in v-prediction scenarios.
Theoretically, the optimal solution of v-prediction models is
o(x |lgminE[llusxxx t) (ate o,x0) x,
x
= Baac 0xXo|xx = x|
0.6
E
aic Jr
X, X
a1
(A10)
of +
(ar
D x
at
2 + oz
V (x)
0g
(x +
Vx log qe
Ott
where x Vx Vx qe (x) is smooth under the assumption of Theorem 3.1, and
[ta) or
do as
-
Thus, with the same derivation of Theorem 3.1, we can conclude that lim sup, The detailed derivation goes as follows:
dv(z.t)
8t
Firstly, we can obtain the partial derivative of the v-prediction model over as Ou(x,t)
d Lt [x) Vx (x)) Ø: d
a₁ dt (x + V (x))
(A11)
dt at
Note that
03 a1
- (ay, da at di
Vi-a? as 0 under common settings
that og 0. 00 1, and dr 1
is finite, thus if day
de k + 0. and Vx (x) * then
one of the following two must stand
@x t)
lim sup (-0+
oo; lim sup
(x + Vx log (x)(
+ xc.
(A12)
Öt
df
t
Under the assumption that qe (x) is a smooth process, we can conclude that 0e(x.() dt
lim sup
co.
18

---

Published as a conference paper at ICLR 2024
Table A2: Quantitative comparison be- tween E-TSDM and DDPM (Ho et al., 2020) using v-prediction on Lsun-Cat 256 x 256 (Karras et al., 2019) and FFHQ 256 x 256 dataset (Karras et al., 2019) evaluated with FID-10K J. Experimental results indicate that E-TSDM can achieve better synthesis performance.
Table A3: Quantitative comparison among E- TSDM, DDPM (Ho et al., 2020), and DDPM using regularization techniques (DDPM-r) on FFHQ 256 x 256 (Karras et al., 2019) and Cele- bAHQ 256 x 256 (Karras et al., 2017) evaluated with FID-10k J. Experimental results show that DDPM-r can slightly improve the FID but per- forms worse than E-TSDM.
Baseline
E-TSDM
Method
Baseline
E-TSDM
DDPM-
FFHQ
10.85
9.00
FFHQ
9.50
6.62
9.18
Lsun-Cat
18.40
13.86
CelebAHQ
8.05
6.99
7.97
Since most of the diffusion models are noise- coise-prediction and v-prediction models, the Lipschitz sin- gularities issue is an important problem for the community of diffusion models.
Empirically, we can also observe the phenomenon of Lipschitz singularities for v-prediction diffu- sion models, where the experimental results of Lipschitz constants on FFHQ 256 x 256 dataset (Kar- ras et al., 2019) and Lsun-Cat 256 x 256 (Karras et al., 2019) are shown in Figure A5, from which we can tell E-TSDM can effectively mitigate Lipschitz singularities in v-prediction scenario. Besides, we also provide corresponding quantitative evaluations evaluated by FID-10K in Table A2, showing that E- E-TSDM can also improve the synthesis performance in the v-prediction scenario.
C.3 PROOF OF THEOREM 4.1
Here we will first give the derivation of the upper-bound on fx, - cx t)ll when f E [0, t), where e" (x, fr())) denotes the optimal (x, fr())), and E(x, t) V discuss the convergence rate of the error bound.
Vx log q (x). Then, we will
For any (), there exists an i E 21,2, n) such that E 4,). For simplicity, we use h(x, f) to denote the score function Vx log q: (x), and use E,[-] to denote the expectation over T ((44-1,44) 44). Thus, we can obtain
(x,(x, E(x, 1)ll = Ex|e(x,7)) ] - 6(x, !)ll
|r [)] - 0x t)ll
|| |r,,,,(x,, T) - 0, t) + hxx, t) 0, t)]l
|Br|o, (h(x,T) h(x, 1)) |E o,) ()]
(A13)
< [r T) - 1(x, t)ll + +E, ||,, -=e1]l(((x()) f)l
1,, 111(x,)
t((x + (ot, - -0..,)]l((x.t))
or (x) -t-1) + (x) o...))
S (x)(x))1 + A0x))
where K2(x)
supt. - B(x) = suPrE(0.d) (M(x,1)]l and Da maxigisn (0t; 8-7
Bx(x) suPrete.-.ee) [[((x, 1)]1, K(x)
11-
sUPt,
J..-l ). The first equality holds because
E(x,t) arg min Bllin - t, = =x
60
(A14)
DlElT = x, x),
19

---

Published as a conference paper at ICLR 2024
Table A4: Quantitative comparison among E-TSDM, DDPM (Ho et al., 2020), and modification of noise schedules (Modified- NS) on FFHQ 256 x 256 dataset (Kar- ras et al., 2019) evaluated with FID-10K J. Specifically, we implement Modified-NS linear, quadratic, and cosine schedules. Ex- perimental results indicate that the perfor- mance of Modified-NS is unstable while E-TSDM achieves better synthesis perfor-
Table A5: Quantitative comparison of remap method between uniformly sampling and uni- formly sampling X, during training and inference, on FFHQ 256 256 (Karras et al., 2019) evaluated with FID- 10k L. Specifically, 24 is Z((0, 1]. and Mx is 4100 K] for 1/t but U|- K] for Inverse- Sigmoid, where K is large number to avoid in- finity. Results show that remap is not helpful.
Training
Inference
Remap Function
mance.
Strategy
Strategy
1/t
Inverse-Sigmoie
Linear
Quadratic
Cosine
U
t ~ Ut
9.43
9.33
Baseline
9.50
13.79
27.17
~ Up
A ~ Ux
83.71
468.90
E-TSDM
6.62
9.69
26.08
A Ux
t to Ut
83.44
468.19
Modified-NS
8.67
17.48
26.84
A ~ Ux
A ~ Ux
171.06
351.89
and our optimal
e* cx(x, f(t) can be expressed as
c" (x, f(t) c" (x, 14-1)
arg min (r toee -|,]xx
E.) =x] x]
(A15)
Ey-u(tt In) E, x]
E,-u((t 18) (xxx))
As for the rate of convergence, is obvious from Equation (A13) that we only need to determine the convergence rate of A0max Under common settings, is monotonically decreasing and concave for E [0, T), thus
Admax maxx
max 10,, Jo: 11 0be Oo aa 1ss5n
(A16)
where the last equality holds because ot, 00 O. and t i/n At as uniformly divides [0, E) into sub-intervals. Then, verify the convergence rate of Ag max
Ad max
A
lim A--
lim
VAi
At-0
do?
dt
===0
d(1
a?)
(A17)
dl
t=0
da
-20
dt
t=0
da
-
dl
t=
where day 1== is finite and day dr t= < 0. Thus, we can conclude that Admax is at least } -oorder convergence with respect At. and the error bound (x) + B(x)Ao is also at least 3- order convergence. This is a relatively fast convergence speed in optimization, and demonstrates that the introduced errors of E-TSDM are controllable.
20

---

Published as a conference paper at ICLR 2024
12.5
DDPM E-TSOM
DDPM
Lipschitz Constants
10.0
1.0
Lipsehitz Constants
2
7.5
0.5
15
5.0
10
100
100
2.5
0.10
90
100
1000
100
1000
Timestep
Timestep
Figure A6: Quantitative comparison of Lipschitz constants between E-TSDM and DDPM baseline (Ho al., 2020) FFHQ 256 x 256 (Karras et al., 2019) using the co- sine shift schedule.
Figure A7: Quantitative comparison of Lip- schitz constants among E-TSDM, DDPM (Ho et al., 2020), and DDPM (Ho et al., 2020) us- ing regularization techniques (DDPM-r) on FFHQ 256 x 256 (Karras et al., 2019).
D ADDITIONAL RESULTS
D.1 LIPSCIITZ CONSTANTS
In our main paper, we demonstrate the effectiveness of E-TSDM in reducing the Lipschitz constants near / = 0 by comparing its Lipschitz constants with that of DDPM baseline (Ho et al., 2020) on the supplement, we provide additional comparisons FFHQ 256 x 256 dataset (Karras et al., 2019). AFHQ-Cat (Choi et al., 2020) (sce Figure A2a), of Lipschitz constants other datasets, including
AFHQ Wild (Choi et al., 2020) (see Figure A2b), Lsun-Cat 256 256 (Karras et al., 2019) (see Figure A2c), Lsun-Church 256 256 (Karras al., 2019) (see Figure A2d), and CelebAHQ 256 x 256 (Karras et al., 2017) (see Figure A2e). These experimental results demonstrate that E- E-TSDM is highly effective in mitigating Lipschitz singularities in diffusion models across various datasets.
Furthermore, we provide a comparison Lipschitz constants between E-TSDM and the DDPM baseline (Ho et al., 2020) when using the quadratic schedule and the cosine-shift schedule (Hooge- boom et al., 2023). As shown in Figure A2f, we observe that large Lipschitz constants still exist in diffusion models when using the quadratic schedule, and E-TSDM effectively alleviates this prob- lem. Similar improvement can also be observed when using the cosine-shift schedule as illustrated in Figure A6, highlighting the superiority of our approach over the DDPM baseline.
D.2 QUANTITATIVE ANALYSIS OF É AND n
In our main paper, we investigated the impact of two important settings for E-TSDM, the length of the interval to share conditions i, and the number of sub-intervals n this interval. As supplement, we provide additional results on various datasets to further investigate the optimal settings for these parameters.
As seen in Figure A3 and Figure A4, we observe divergence in the best choices of and ī across different datasets. However, we find that the default settings where 7 7= 100 and n consistently yield good performance across a range of datasets. Based on these findings, we recommend the default settings as an ideal choice for implementing E-TSDM without the need for a thorough search. However, if performance is the main concern, researchers may conduct a grid search to explore the optimal values and n for specific datasets.
D.3 ALTERNATIVE METHODS
In this section, discuss three different alternative methods that possibly alleviate Lipschitz sin- gularities. including regularization, modification of noise schedules, and remap. Although seem feasible, they have different problems, resulting in worse performance than E-TSDM.
21

---

Published as a conference paper at ICLR 2024
20
12,000
Cosine
Uniformly Sampling t
Linear
10,000
15
Uniformly Sampling 2
Quadratic
Ratio SNR
8,000
SNR
10
6,000
4,000
2,000
0
200
400
600
800
1000
0
200
400
600
800
1000
Timestep
Timestep
Figure A9: Quantitative comparison of SNR for remap method between uniformly sampling t and uniformly sampling the remapped conditional input A. Results show that when using remap method, uniformly sampling À significantly increases the SNR across all of the timesteps, and thus forces the network to focus too much on the beginning stage of the diffusion process.
Figure A8: Quantitative evaluation of the ratio of SNR of Modified-NS to the SNR of the corresponding original noise schedule. Results show that Modified-NS significantly increases the SNR near zero point, and thus reduces the amounts of added noise near zero point, Specifically, for the quadratic schedule, Modified-NS seriously increases the SNR al- most during the whole process.
D.3.1 REGULARIZATION
As mentioned in the main paper, one alternative method is impose restrictions on the Lipschitz constants through regularization techniques. In this section, we apply regularization on the baseline and estimate the gradient of cg (x, t) by calculating the difference K (t, t'). We represent this method as DDPM-r in this paper. As shown in Figure A7, although DDPM-r can also reduce the Lipschitz constants, its capacity to do so is substantially inferior to that of E-TSDM. Additionally, DDPM- necessitates twice the calculation compared to E-TSDM. Regarding synthesis performance. as shown in Table A3, DDPM-r -r performs slightly better than baseline, but much worse than E-TSDM, indicating that E-TSDM is better choice than regularization.
D.3.2 MODIFYING NOISE SCHEDULES
As proved in Appendix C, the mainstream noise schedules satisfy das singularities as proved in Theorem 3.1. However, it is possible to modity those schedules to force them to have da. dr 1= 0, and thus alleviate Lipschitz. singularities. We represent this method as Modified-NS in this paper. However, as said in Nichol & Dhariwal (2021), das
t= A O, leading to Lipschitz
dt. t=o = 0 means tiny amounts of noise at the beginning of the diffusion process, making hard for the network to predict accurately enough.
To explore the performance, we conduct experiments of Modified-NS on FFHQ 256 x 256 (Kar- ras et al., 2019) for all of the three discussed noise schedules in Appendix C.I. Specifically, for linear and quadratic schedules, since dr
1= 30) (as detailed in Equation (A2)), we implement Modified-NS by setting B(0) = 0. Note that for the quadratic schedule, such modification will significantly magnify the Signal to Noise Ratio (SNR), . across the whole diffusion process, we slightly increase 37 to make its SNR at L = T similar to that of the original quadratic schedule. Meanwhile, ,1....,877
da(z)
are also correspondingly increased due to 3 (v36 + (vBT- Bo )² As for the cosine schedule, we set the offset in Equation (A7) to zero. Experimental results are shown in Table A4, from which we find that the performance of Modified-NS is unstable. More specifically, Modified-NS improves performance for linear and co- sine schedules but significantly drags down the performance for the quadratic schedule. We further provide the comparison of SNR between Modified-NS and their corresponding original noise sched- ules in Figure A8 by calculating the ratio of Modified-NS's SNR to the original noise schedule's 's SNR. From this figure we can tell that for linear and cosine schedule, Modified-NS significantly increase the SNR near zero point while maintaining the SNR of other timesteps similar. In other
22

---

Published as a conference paper at ICLR 2024
0.6
LDM
Lipschitz Constants
E-TSDM
4
0.4
3
0.2
0.0
100
0
10
100
1000
Timestep
Figure A10: Quantitative comparison Lipschitz constants between E-TSDM and LDM (Rombach et al., 2022) on FFHQ 256 x 256(Karras et al., 2019). E-TSDM reduces the overall Lipschitz con- stants near / = 0, and mitigates the Lipschitz singu- larities occurring in LDM (Rombach et al., 2022).
Figure A1l: Qualitative results pro- duced by E-TSDM implemented on LDM (Rombach al., 2022) FFHQ 256 x 256(Karras et al., 2019).
words, the one hand, Modified-NS seriously reduces the amount of noise added near zero point, which can be detrimental to the accurate prediction. On the other hand, Modified-NS alleviates the Lipschitz singularities, which is beneficial to the synthesis performance. As result, for linear and cosine schedules, Modified-NS performs better than baseline but worse than E-TSDM. However, for the quadratic schedule, although we foree the SNR of Modified-NS at = T similar to the SNR of the original schedule, the SNR at other timesteps is significantly increased, leading to a worse performance of Modified-NS compared that of baseline.
D.3.3 REMAP
Except for regularization and Modified-NS, remap another possible method to fix the Lipschitz singularities issue. Recall that the inputs of network ( is noisy image x and timestep con- dition L. Remap is trying to design remap function = =((() on L as the conditional input of network instead of namely, cx(x, ft)). The core idea of remap is to reduce 8xa(x,t) cantly stretching the interval with large Lipschitz constants. Note that although fr of E-TSDM can also be seen as a kind of remap function, there big differences between E-TSDM and remap. Specifically, E-TSDM tries to set the numerator to zero while remap aims to significantly increase the denominator. Besides. fr has no inverse while f(t of remap usually reversible function. We provide two simple choices of ft) in this section as examples, which are f(t) = 1/t and f (A) si)
0t by signifi-
Remap can efficiently reduce the Lipschitz constants regarding the conditional inputs of the net- work, dea(x.() However, since uniformly sample t both in training and inference, what should 9A
be focused on is the Lipschitz constants regarding I, 8ee(x.1), which can not be influenced by remap. In other words, although remap seems to be a feasible method, is not helpful mitigate the Lip- schitz constants we care about, unless we uniformly sample À in training and inference. However, uniformly sampling À may force the network focus on a certain part the diffusion process. We use f(t) = 1/t as example to illustrate this point and show the comparison of SNR between uniformly sampling and uniformly sampling when using remap in Figure A9. Results show that uniformly sampling À maintains high SNR across all of the timesteps, leading to excessive atten- tion to the beginning stage of the diffusion process. As a result, when we uniformly sample À during training inference, the synthesis performance gets significantly worse shown in Table A5. Be- sides, when we uniformly sample both in training and inference, remap makes no difference and thus leads to similar performance the baseline.
23

---

Published as a conference paper at ICLR 2024
Low resolution
Original one
Baseline
PSNR1: 24.64
Ours
PSNR1: 25.61
Figure A12: Qualitative and quantitative results by applying E-TSDM to super-resolution task (i.e., from 64 x to 256 x 256), using PSNR as the evaluation metric. Results show that E-TSDM mitigates the color bias occurring in baseline and improves the PSNR from 24.64 to 25.61, which suggests that our approach well supports conditional generation.
D.4 MORE DIFFUSION MODELS
Latent diffusion models (LDM) (Rombach et al., 2022) is one of the most renowned variants of diffusion models. In this section, we will investigate the Lipschitz singularities in LDM (Rombach et al., 2022), and apply E-TSDM to address this problem. LDM (Rombach et al., 2022) shares resemblance with DDPM (Rombach et al., 2022) but has an additional auto-encoder to encode images into the latent space. As LDM typically employs the quadratic schedule, it is also susceptible Lipschitz singularities, as confirmed Figure A10.
As seen in Figure A10, by utilizing E-TSDM, the Lipschitz constants within each timestep-shared sub-interval are reduced zero, while the timesteps located near the boundaries of the sub-intervals exhibit Lipschitz constant comparable to that of baseline, leading to decrease in overall Lipschitz constants in the target interval [0, f), where is set as the default, namely 100. Consequently, E-TSDM achieves an improvement in FID-50k from 4.98 to 4.61 with the adoption of E-TSDM, when = 20. We provide some samples generated by the E-TSDM implemented on LDM in Figure A11.
Besides, we also implement our E-TSDM to Elucidated diffuusion models (EDM) (Karras et al., 2022), which proposed several changes to both the sampling and training processes and achieves impressive performance. Specifically, we reproduce EDM and repeat it three times on CIFARIO 32 x 32 (Krizhevsky et al., 2009) get FID-50k of 1.904 + 0.015, which is slightly worse than the official released one. Then we apply E- E-TSDM to EDM and repeat it three times to get a FID-50k of 1.797 + 0.016, indicating that E-TSDM is also helpful EDM.
D.5 GENERATED SAMPLES
As supplement, we provide massive generated samples of E-TSDM trained on Lsun-Church 256 x 256 (Karras al., 2019) (see Figure A13), Lsun-Car 256 x (Karras al., 2019) (see Figure A A14), AFHQ-Cat 256 x 256 (Choi et al., 2020), AFHQ-Wild 256 256 (Choi et al., 2020) (see Figure A15), FFHQ 256 256 (Karras et al., 2019) (see Figure A16), and CelebAHQ 256 x 256 (Karras et al., 2017) (see Figure A17).
24

---

Published as a conference paper at ICLR 2024
Figure A13: Qualitative results produced by E-TSDM Lsun ShunChhurch 256 x 256 (Yu al., 2015).
Figure A14: Qualitative results produced by E-TSDM on Lsun-Cat 256 256 (Yu et al., 2015).
25

---

Published as a conference paper at ICLR 2024
Figure A15: Qualitative results produced by E-TSDM on AFHQ-Cat 256 x 256 (Choi et al., 2020)
and AFHQ-Wild 256 256 (Choi al., 2020).
Figure A16: Qualitative results produced by E-TSDM FFHQ 256 256(Karras et al., 2019).
26

---

Published as a conference paper at ICLR 2024
2
2
Figure A17: Qualitative results produced by E-TSDM on CelebAHQ 256 x 256 (Karras et al., 2017).
27


Reviews:
[{"review_id": "syx7W5AtBo", "reviewer": "Reviewer_gj5E", "summary": "Diffusion models, utilizing stochastic differential equations to generate images, have become a leading type of generative model. However, their underlying diffusion process hasn't been thoroughly examined. This paper reveals a concerning tendency in diffusion models: they often display infinite Lipschitz (for $\\sigma_{t} \\cdot \\text{score function}$) near the initial timesteps. Through theoretical and empirical evidence, the presence of these infinite Lipschitz constants is confirmed, which can jeopardize the stability and precision of the models during training and inference. To combat this, the paper introduces a new method, E-TSDM, that uses quantization to reduce these Lipschitz issues. Tests on various datasets support the presented theory and approach, potentially offering a deeper understanding of diffusion processes and guiding future diffusion model design.", "questions": "I am looking for comments from the authors on a few alternative methods:\n1. Learning $\\nabla \\log q_{x}(t)$ directly with weighted least square: can we reduce the weight of the least square when $\\sigma_t$ is small, e.g., learn $E \\sigma_{t}^2\\|\\nabla \\log q_{x}(t) - \\frac{1}{\\sigma_{t}}I\\|^2$?\n2. For Eq (9), what if we only learn $\\epsilon_{\\theta}(\\alpha_{f_T(t)}x_0 + \\sigma_{f_T(t)}\\epsilon, f_{T}(t))$, i.e., only learn the score function for time $f_{T}(t)$ and only use those time steps to do sampling? \n3. Is that $\\sigma_{t}$ an input of the neural network: what if we learn $\\epsilon_{\\theta}(x, \\sigma_{t})$ (the intuition is that $\\sigma_{t}$ will help adjust the network Lipschize automatically). \n\nMinor comments:\n- Eq 10, are $\\beta_{t}$ and $\\eta_{t}$ defined?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This paper highlights a unique and previously unexplored challenge with DDPM: the instability encountered when learning $\\epsilon_{\\theta} = \\sigma_{t} \\cdot \\nabla \\log q_{t}(x)$ during the time steps where $\\sigma_{t}$ is minimal. One might naturally question why DDPM doesn't directly learn $\\nabla \\log q_{t}(x)$. I conjecture that the optimization process for learning $\\nabla \\log q_{t}(x)$, which involves solving $E\\|\\nabla \\log q_{t}(x) - \\frac{1}{\\sigma_{t}} \\|^2$, becomes problematic with a small $\\sigma_{t}$. As a workaround, DDPM employs a transformation to learn $\\sigma_{t}\\cdot \\nabla \\log q_{t}(x)$ directly. However, this paper reveals the inherent price of such an approach (no free lunch indeed).\n\nThe paper validates the infinite Lipschitz problem with $\\epsilon_{\\theta}$ both theoretically and empirically. Moreover, it introduces E-TSDM, an innovative solution that essentially employs a quantization strategy when $\\sigma_{t}$ is minimal, particularly during the initial t=100 steps. Comprehensive experiments demonstrate E-TSDM's enhanced stability and performance, even setting a new benchmark for FFHQ 256×256.\n\nThe paper's novelty is commendable, presenting a compelling and succinct argument with an impressive practical performance. Its insights could significantly influence the diffusion model community. I'm inclined to strongly endorse its acceptance.", "weaknesses": "- One minor suggestion is to avoid saying $t$ being small (rather, it is about $\\sigma_{t}$ being small). Since $t$ is in fact $0, 1, 2, 3, .. 100.$ \n- May add more discussions to the alternative approaches (see Questions below). \n- It may be worth showing that directly learning $\\nabla \\log q_{x}(t)$ with the least square is prohibitve.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1KIyF4ZSWp", "reviewer": "Reviewer_DUEU", "summary": "This paper demonstrates theoretically (and confirms empirically) that the limit of the Lipschitz constant of the noise prediction network for a timestep of zero is infinite. Such a result is a source of instability for using diffusion models in many generative tasks, and the authors propose a technical solution to alleviate this issue and confirm the superiority of the approach with extensive numerical simulations.", "questions": "None", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This is an excellent paper, and the presentation is very well carried out. The authors point out a very interesting theoretical property that could explain some practical instabilities encountered in DDPM samples. They then present a practical solution to the problem. The authors' contribution is excellent for the community, as reducing the instabilities in the generative process, such as diffusion models, has important practical consequences.", "weaknesses": "This paper as it is impeccable in terms of presentation and contribution, both theoretically and practically. The only drawback is that no open-source code is available to experiment with their approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uRmiVcf9KH", "reviewer": "Reviewer_qCuR", "summary": "This paper deals with an exploding Lipschitz constant in the function a neural network is asked to learn in a DDPM model, and the negative effects of trying to learn a function with such.\n\nThe authors present an argument based on taking time derivative of the quantity $-\\sigma_t \\nabla \\log q_t(x)$, where the $\\sigma_t$ are the standard deviations of the forward noising process in the time-discrete forward noising process in the DDPM formulation and $q_t$ is the density of the data distribution diffused to discrete time step $t$.\n\nThey demonstrate that the Lipshitz constant in time explodes to infinity for a most parameter settings of common noising schedules under the DDPM/VPSDE setting.\n\nThe authors propose a method for fixing this issue by applying a transform to the time input of the score network, tying together multiple timestep near t=0 to have the same score.\n\nThey demonstrate significant empirical benefit over a range of diffusion modelling tasks.\n\nThe authors also discuss a number of other possible methods to alleviate the issue of learning high lipshitz constants in diffusion models, but show that these methods despite being theoretically attractive, do not perform as well in practise.", "questions": "1. Could the authors highlight better which lipshitz constant is is that is important, and why we care about it? While I understand I believe which and why it is cared about, it is perhaps not the clearest from reading the paper. The sentence in the abstract \"they frequently exhibit the infinite Lipschitz near the zero point of timesteps\" is a good example of this - it does not specify _what_ function has high lipshitz constant, or why indeed that matters. From reading the paper in depth, the authors care about the lisphitz constant of the quantity $-\\sigma_t \\nabla \\log q_t(x)$ as a) neural networks find it difficult to learn high lipshitz constant functions, and this is the function we are asking the score net to learn, and b) because this quantity is involved in the reverse rollouts, having a term with high lipshitz constant makes discretising the SDE challenging to do accurately, but this should be apparent from the abstract/introduction.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1) The method proposed is simple to implement.\n2) The method clearly demonstrates significant empirical benefit.\n3) The authors discuss alternative proposals and show these are less effective", "weaknesses": "1) The only weakness I would like to highlight is the discussion of the alternative methods presented.\n - I believe 1 of the methods from the appendix is not mentioned in the main text - namely the Remp method (D.3.3).\n - It would be nice to see an expanded discussion of these with some small experiment to show the quantitative difference between the proposed method and these other methods. I appreciate the space limitation, but I think this is really an interesting point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dWS4dZDVrv", "reviewer": "Reviewer_gGRw", "summary": "The paper elaborates upon an important observation concerning the presence of infinite-Lipschitz constants in the diffusion process, made earlier by (Song et al., 2021a; Vahdat et al., 2021). It also proposes a simple yet effective approach to address this challenge.", "questions": "How would you describe the differences in your observation and those of (Song et al., 2021a; Vahdat et al., 2021)? \n\nYou could make your observation more original by noting that the infinite Lipchitz constants mean the SDE need not have a unique strong solution (Øksendal, 2003). Exhibiting multiple solutions would indeed be of interest.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Theorem 3.1 is a nice piece of rigorous analysis of diffusion models, albeit indebted to Song et al.\n\nThe proposed approach to address this infinite-Lipschitz challenge, which is based on improving the resolution of the discretisation, does indeed seem to be effective. \n\nNumerical results in Figures 3 and 4 seem quite impressive.", "weaknesses": "The observation concerning the presence of infinite-Lipschitz constants in the diffusion process is not original (Song et al., 2021a; Vahdat et al., 2021). Concerning it has been observed before, the authors should like to tone down their claims of having observed it first. \n\nSome of the English is stilted (\"vexing propensity of diffusion models\" in the abstract, \"Recently, there have been massive variants that significantly promote the development of diffusion models\" on page 3).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)