Training-Free Adaptive Diffusion with Bounded Difference Approximation Strategy
Authors: Hancheng Ye
Jiakang Yuan
Renqiu Xia
Xiangchao Yan
Tao Chen
Junchi Yan
Botian Shi
Bo Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Adaptive Diffusion
Diffusion Probabilistic Models
Third-order Difference
Primary area: diffusion_based_models
TL;DR: We explore the training-free diffusion acceleration that dynamically selects the denoising path according to given prompts. We design the third-order estimator to indicate the computation redundancy.

Abstract: Diffusion models have recently achieved great success in the synthesis of high-quality images and videos. However, the existing denoising techniques in diffusion models are commonly based on step-by-step noise predictions, which suffers from high computation cost, resulting in a prohibitive latency for interactive applications. In this paper, we propose AdaptiveDiffusion to relieve this bottleneck by adaptively reducing the noise prediction steps during the denoising process. Our method considers the potential of skipping as many noise prediction steps as possible while keeping the final denoised results identical to the original full-step ones. Specifically, the skipping strategy is guided by the third-order latent difference that indicates the stability between timesteps during the denoising process, which benefits the reusing of previous noise prediction results. Extensive experiments on image and video diffusion models demonstrate that our method can significantly speed up the denoising process while generating identical results to the original process, achieving up to an average 2-5x speedup without quality degradation. The code is available at https://github.com/UniModal4Reasoning/AdaptiveDiffusion

Training-Free Adaptive Diffusion with Bounded Difference Approximation Strategy
Hancheng Yel.*, Jiakang Yuan Renqiu Xia Xiangchao Yan',
Tao Chen², Junchi Yan Botian Shi', Bo Zhang
SShnnghai Artificial Intelligence Laboratory School of Information Science and Technology. Fudan University
School of Artificial Intelligence, Shanghai Jiao Tong University
Sthaancheng@pjlale org.
jkyuan2 8m. fudan. cdu. cn, zhangbo@pjlab.org.
Abstract
Diffusion models have recently achieved great success in the synthesis of high- quality images and videos. However, the existing denoising techniques in diffusion models are commonly based on step-by-step noise predictions, which suffers from high computation cost, resulting prohibitive latency for interactive applications. In this paper, we propose AdiptiveDifiusion to relieve this bottleneck by adaptively reducing the noise prediction steps during the denoising process. Our method considers the potential of skipping as many noise prediction steps possible while keeping the final denoised results identical to the original full-step ones. Specifically, the skipping strategy is guided by the thinl-order latent difference that indicates the stability between timesteps during the denoising process, which benefits the reusing of previous noise prediction results. Extensive experiments image and video diffusion models demonstrate that our method can significantly speed up the denoising process while generating identical results to the original process, achieving up to an average ~ 5x speedup without quality degradation. The code is available at https: /tte:://giiiii github. com/ /nn//nnii Unn//nnMMaM Metodal4Red RRasonning AdaptiveDi ffusion.
Introduction
Recently, Diffusson models [1, 11, 30, 33] have emerged as powerful tool for synthesizing high- quality images and videos. Their capability to generate realistic and detailed visual content has made them popular choice in various applications, ranging from artistic creation to data augmentation, e,g. Midjourney, Sora [4], etc. However, the conventional denoising techniques employed in these models involve step-by-step noise predictions, which computationally intensive and lead to significant latency, e.g., taking tens seconds for SDXL [32] to generate a high- nigh-quality image of 1024x 1024 resolutions. Diffusion acceleration mainly focuses on three paradigms: (1) reducing sampling steps [39, 13, 23, 35, 24], (2) optimizing model architecture [15, 42, 9, 27] and (3) parallelizing inference [16, 38].
effective technique has been deeply explored recently, which
Currently, most strategies are designed based on fixed acceleration mode for all prompt data. However, in experiments, it is observed that different prompts may require different steps of noise prediction to achieve the same content as the original denoising process, as presented Fig. Here, compare the denoising paths using two different prompts for SDXL [32]. both of which preserve rich content against the original full-step generation results. The denoising path denotes bool-type sequence, where each element represents whether to infer the noise from the noise prediction model. can be observed that Prompt 2 needs more steps to generate an almost lossless image than Prompt
*Equal contribution, *Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Deneisin predicted Prampt (Prompt dining piants seen)
Denoising predicted by Prompt 2
(Prompt shiny matorcycle has been put on disploy)
Original Image SOXL
Image by Denoiding Path
Original Image SONL
Image by Denoising Path
Two skipping for differest prompts, which
by methed.
Prompt @ Path 1: LPIPS-0,154 PSNR=24.0
Prompt 2 @ Patt 2: LPIPS-0,144 1411, PSMR-24.6
Prompt 2 @ Parth 1 LPIPS-0.20 2466 PSNR-20.2
Reading
30
20
30
40
50
Each should have its denoising
Shaps
Image by Denoising Path
Sanakingg Kath (20 64 SPige shiaa))
Figure I: Different prompts may have different denoising paths to generate the high-quality image. For Prompt 1. we only need 20 steps out of 50 steps for noise predictions generate almost lossless image, while for Prompt 2, we need steps out 50 steps to achieve almost lossless image.
1 (A lower LPIPS [51] value means more similarity between two images generated by the original strategy and strategy). Therefore, it is necessary explore prompt-adaptive acceleration paradigm to consider the denoising diversity between different prompts.
Motivated by this observation, in this paper, we deeply dive into the skipping scheme for the noise prediction model and propose AdaptiveDiffusion, the generation process according to different input prompts. The fundamental concept behind AdaftiveDiffusion is to adaptively reduce the number noise prediction steps according to different input prompts during the denoising process, and meanwhile maintain the quality of the final output. The key insight driving our method is that the redundancy noise prediction is highly related to the third-order differential distribution between can be leveraged to design an effective skipping strategy, allowing us to decide when to reuse previous noise prediction results and when proceed with calculations. Our approach utilizes the third- order latent difference to assess the redundancy of noise prediction at each timestep, reflecting our strategy's 's dependency input information, thus achieving a prompt-adaptive acceleration paradigm.
novel approach that adaptively accelerates
soooaalll-eighhbrinng latents. This relation
Extensive experiments conducted on both image and video diffusion models demonstrate the effec- tiveness of AdaptiveDiffusion. The results show that our method can achieve up to a 5.6x speedup in the denoising process with better preservation quality. This improvement in acceleration quality opens up new possibilities for the application of diffusion models in real-time and interactive environments.
In summary, AdaptiveDiffusion represents substantial advancement in adaptively efficient diffusion, offering a practical solution to the challenge of high computational costs associated with sequentially denoising techniques. The main contribution is threefold: (1) our best knowledge, our method is the first to explore the adaptive diffusion acceleration from the step number reduction of noise predictions that makes different skipping paths for different prompts. (2) We propose novel approach, namely AdaptiveDiffusion, which develops plug-and-pla prediction should inferred reused from the previous noise results. (3) Extensive experiments conducted various diffusion models [32, 33, 52, 44] and tasks [7, 21, 45, 8] demonstrate the superiority of our AdaptiveDiffusion to the existing acceleration methods in the trade-off among efficiency, performance and generalization ability.
criterion to decide whether the noise
2
Related Works
2.1 Diffusion Models
Diffusion models [1, 11, 30, 33, 10] have achieved great success and served as milestone in content
generation. As a pioneer, Denoising Diffusion Probabilistic Models (DDPMs) [11] generate higher-

---

Noise Prediction
Noise Reusing (Skipping Noise Prediction)
Noise Prediction
T.
Ti.
Update
Update
Scheduler
Scheduler
Coche
Latent Coche
Noise Prediction
Noise Prediction Model
Noise Prediction Model
Madel
Third-order Estimator
Third-arder Estimator
Yes
Adaptive Diffusion
Adoptive Diffusion
Neise Cache
Noise Coche
Reuse
Eo (T;, t.)
Eo (z,,, t2)
Eo (Z,-2- 22-2)
Figure 2: Denoising process of the proposed AdaptiveDiffussin: We design third-order estimator (Refer Sec. 3.3 for details), which can find the redundancy between neighhoring timesteps, and thus, the noise prediction model can skipped or inferred according the indicate from the estimator, achieving the adaptive diffusion process. Note that the timestep and text information embeddings are not shown for the sake of brevity.
quality images compared to generative adversarial networks (GANs) [17, 6] through an iterative denoising process. To improve the efficiency of DDPM, Latent Diffusion Models (LDMs) [33] perform forward and reverse processes in latent space of lower dimensionality which further evolves
into Stable Diffusion (SD) family [36, 32]. Recently, video diffusion models [3, 52, 2, 44] have attracted increasing attention, especially after witnessing the success of Sora [4]. Stable Video Diffusion (SVD) [3] introduces a three-stage training pipeline and obtains a video generation model with strong motion representation. 12VGen-XL [52] first obtains a model with multi-level feature extraction ability, then enhances the resolution and injects temporal information in the second stage. Despite the high quality achieved by diffusion models, the inherent nature of the reverse process which needs high computational cost slows down the inference process.
2.2 Accelerating Diffusion Models
Current works accelerate diffusion models can be divided into the following aspects. (1) Reducing Sampling Steps [39, 23, 24, 49, 40, 25, 20, 37, 28, 29, 26]. DDIM [39] optimizes sampling steps by exploring non-Markovian process. Further studies [23, 24, 49] such as DPM-Solver [23] propose different solvers for diffusion SDEs and ODEs to reduce sampling steps. Another way to optimize sampling steps is to train few-step diffusion models by distillation [40, 25, 20]. Among them, consistency models [40, 25] directly map noise data to enable one-step generation. Other works [20, 37] explore progressive distillation or adversarial distillation to effectively reduce the reverse steps. Besides, some works [26, 42, 29] introduce early stop mechanism into diffusion models. (2) Optimizing Model Architecture [9, 19, 46, 27, 5, 47, 22, 41]. Another strategy to accelerate diffusion models to optimize model efficiency reduce the cost during inference. Diff-pruning [9] compresses diffusion models by employing Taylor expansion over pruned timesteps. DeepCache [27] notices the feature redundancy in the denoising process and introduces a cache mechanism to reuse pre-computed features. (3) Parallel Inference [16, 38, 12]. The third line lies in sampling or calculating a parallel way. ParaDiGMS [38] proposes to use Picard iteration to run multiple steps in parallel. DistriFusion [16] introduces displaced patch parallelism by reusing the pre-computed feature maps. Compared with the existing paradigms designing a fixed acceleration mode for all input prompts, method highlights the adaptive acceleration manner with plug-and-play criterion based on the high-order latent differential distribution, which allows various diffusion models with different sampling schedulers to achieve significant speedup with negligible performance drop and deployment cost.
The Proposed Approach
3.1 Preliminary
Reverse Denoising Process. Diffusion models [11, 339 are designed to learn two processes with noise addition (known as the forward process) and noise reduction (known as the reverse process).
3

---

(a) Nevie Prod Latent Updite @8 steps (Full)
(b) Noise Prod @i 25 (Hail)
(c) Noise Pred + Linent Upeltte
Novia Prod + Lalent Updatt
Updase 50 steps (Full)
@ steps
@8 25 steps (Ilal))
Figure 3: Different update strategies. (a) The default SDXL [32] samples 50 steps of noise prediction followed by the latent update process. (b) Our AdaptiveDiffusion skips 25 steps of noise prediction according to the third-order estimator, while the latent is fully updated at all 50 steps. (c) SDXL samples 25 steps of the noisc prediction and latent update process. (d) The default SDXL skips 25 steps of both noise prediction and latent update from its sampled 50 steps.
During the inference stage, only the reverse denoising process is adopted that starts from the Gaussian noise IT (V(0, I) and iteratively denoises the input sample under the injected condition to get the final clean image(s) To. where T' is the predefined number of denoising steps. Specifically, given an intermediate noisy image x; timestep (i = 1, .... T), the noise prediction model co (e.g., UNet [34]) takes F;, timestep f; and an additional condition e (e.g., text, image, and motion embeddings, etc) as input to approximate the noise distribution in 24. The update from I; to I;-1 is determined by different samplers (a..a, schedulers) that can be generally formulated as Eq. (1):
(1) where (i) and g(i are step-related coefficients derived by specific samplers [23, 50]. The com- putation in the update process mainly involves a few element- wise additions and multiplications. Therefore, the main computation cost in the denoising process stems from the inference of noise
Ii-1 f(i 1) x; - g(i - 1) c0(2,,t t,), i......,7
prediction model [16].
Step Skipping Strategy. As proved by previous works [27, 42, 15]. features between consecutive timesteps present certain similarities in distribution, thus there exists set of redundant computations that can be skipped. Previous works usually skip either the whole update process the partial computation within the noise prediction model at redundant timesteps. However, as visualized in Fig. 3, the latent update process in those redundant timesteps may be important to the lossless image generation. Besides, the calculation redundancy of the noise prediction model within each denoising process is still under-explored. To reduce the computation cost from the noise prediction model, we consider directly reducing the number of noise prediction steps from the original denoising process, which will be proved more effective and efficient to accelerate the generation with almost no quality degradation in Sec. 3.3. Given certain timestep i to skip, our skipping strategy for the update process can be formulated using Eq. (2):
[() 2541 g(i) ea co 4++1),
2-1 = f(i f(i-(() 1) I, --- g(i 1)- (0(
3.2 Error Estimation of the Step Skipping Strategy
To validate the effectiveness of our step-skipping strategy, we theoretically analyze the upper bound of the еттог between the original output and skipped output images. For simplicity, we consider skipping one step of noise prediction, e.g., the i-th timestep. Specifically, we have the following theorem about the one-step skipping.
Theorem 1.
Given the skipping timestep i (i 0), the original output x! and the skipped output I;-1, then the following in in-equation holds:
Ei-1 = |1:1111 - = 0 - l4+1) +Ox; x4+1).
(3)
The proof can be found in Appendix A.2.1. From Fq. (3), it can be observed that the error of the noise-skipped output is upper-bounded by the first-order difference between the previous two outputs. Similarly, the error of the continuously skipped output is upper-bounded by the accumulative differences between multiple previous outputs, which can be found in Appendix A.2.2 and A.2.3.
Therefore, it can be inferred that as the difference between the previous outputs x continuously minor, it is possible to predict that the noise prediction at the next timestep be skipped without damaging the output. This inspires us to utilize the distribution of previous outputs to indicate the skip potential of the next-step noise prediction, as detailed in the following section.

---

3.10
Crenty
Crredy Sear
0.25
Isliader Labeet tebel
Indiander Lateel DAI tmil)
4.35
tabe)
0..00
0.30
0..65
0.00
Desnbsttg Ie:dex
Denotete Endex
Denssiitig Tinder
(6)
(b)
(c)
Figure 4: The relation between order differential distributions and the searched optimal skipping path for one prompt. (a) The Ist-order noise diffcrential distribution of the original full-step generation shows no relation with the optimal skipping path. (b) The 1st latent differential distribution indicates the distribution of the optimal skipping path but with no explicit mapping with skipping decisions, while the relative 2nd-order latent differential distribution shows a certain skipping signal in its fluctuation, but this signal is buried in the unstable magnitude. (c) The relative 3rd-ord -rderder latent differential distribution shows clearer signal for skipping decisions.
3.3 Third-order Estimation Criterion
Observations. In this section, we take SDXL [32] with Euler sampling scheduler as an example to describe the effectiveness of the proposed third-order estimator. Before deriving the third-order estimation criterion, one thing is to calculate the optimal skipping path from the given timestep number, so that we can evaluate the effectiveness of our proposed estimator. Considering the explosive searching cost within large search space (e.g., searching the optimal path of skipping N steps within T steps for one prompt requires C.²' search time cost). we design greedy search algorithm to approximate the optimal skipping path under different skipping targets, which can be found in Alg. (1).
Here we randomly take the prompt "A bustling 18th-century market scene with vendors, shoppers, and cobblestone streets, all depicted in the detailed oil painting style." as an example visualize the optimal skipping path searched by the greedy search algorithm under the predefined skipping target constraint. Meanwhile, as the skipping error is upper-bounded by the constraint of the first- order latent difference, it inspires us to explore the relationship between the first-order difference distribution and the ideal skipping path.
As shown in Fig. 4, we visualize two types of first-order difference (one in Fig. 4a representing the noise difference distribution, another in Fig. 4b representing latent difference distribution) in the original full-step diffusion to compare with the skipping path. It can be observed that both two distributions of first-order differences are smooth across the denoising process, showing little relationship with the optimal skipping path. A similar insight can be observed in the distribution of the second-order latent difference, shown in Fig. 4b.
However, when considering the third-order latent difference distribution, it presents a significant fluctuation in the original full-step denoising process. As shown in Fig. 4c, the distribution of the skipping path is related to the distribution of the third-order latent difference, especially in the early denoising process (around 15 timesteps), where the noise densely updates when the third-order latent difference increases and can be skipped when the third-order difference decreases. As for the later denoising process, when most third-order differences are relatively minor, most noise prediction steps are also skipped. According to the first-order latent difference presented in Fig. 4b, the differences between consecutive latent in the early denoising process significantly larger than those in the later denoising process. Therefore, the precise importance estimation of noise predictions in the early process is much more important and the third-order difference distribution can intuitively serve as the indicator of the noise prediction strategy. The theoretical relation between the third-order derivative of latents and the optimal skipping scheme is analyzed in Appendix A.2.4.
Criterion. Based on the above empirical observation of the relationship between the optimal skipping path and the high-order difference distributions, the third-order estimator is proposed to indicate the potential of skipping the noise computation. Specifically, the criterion is formulated as follows:
(z;-1) = ( 23 2 Az:A.
(4)
where (z ) is the indicator that takes x;-1 and previous latents xi, T441, 2442 input to estimate whether the next noise prediction can be skipped. If E(#,-1) 1) returns False, then the noise from the previous step will be reused to update x;-1- 4(3) ((:))_,-, x denotes the third-order latent difference at timestep - 1, i.e. ((3) = ((2)x; 11-1 - _22),r, x, = Ar - 2Ar; + Aret1, and Az, is defined as the difference between 2; and 21+1 (i 0,..., T 1). 8 is hyperparameter thresholding the
5

---

Greedy Searched Path
Thirdardas Extimated Original and thrr
Skats
0.40
6.30
preateo
0.35
0.30
6:38
0.25
0.20
0.30
0.15
8.65
0.50
0.05
0.00
Donveksing Enalirs
Trensiiing Sedkea
Narmiber Sh:1gpiing 5tegps
The effectiveness of the proposed third-order estimator. (a) The third-order estimated skipping path shares similar distribution with the optimal skipping path. (b) The latent error between the full-step update path and the estimated skipping path. (c) The x² stats and p-value hetween the greedy searched paths and the third-order estimated paths different skipping targets.
Figure 5:
relative scale of ((3)x;-1- The reason for selecting Ar; is that x; actually describes the distance between (A 4x,+2)/2 and Ar, Therefore, it is natural to utilize the relative distance against Ar, to indicate the stability of the denoising process. Fig. 4c present a strong relation between / (3) (3) (the blue dashed line) and the optimal skipping path.
Effectiveness of the Third-order Estimator. To validate the effectiveness of the proposed third-order estimator, we compare our third-order estimated path with the optimal skipping path searched by the greedy algorithm, which is shown in Fig. 5a. It can be observed that the distribution of our estimated path is largely similar to the optimal skipping path. The reason for continuous skipping in the later denoising process is that the third-order difference keeps approaching zero illustrated in Fig. 4c. The accumulative error caused by skipping noise predictions is described in Fig. 5b, where it is observed that the error starts increasing quickly after continuously skipping the noise predictions. Thus, it is vital to introduce another hyperparameter, i.e., the maximum step number of continuous skipping Caax. control the accumulative error. Hyperparameter analyses are described in Sec. 4.3.
Furthermore, we analyze the statistical correlation between the estimated path and the optimal path to test whether the designed criterion is significantly correlated to the optimal skipping criterion. As shown in Fig. 5c, we compute the x² stats and p-values under different step numbers of skipping. The results indicate that when the skipping steps are moderate, the estimated skipping path and the optimal skipping path are significantly correlated. For those targeting at small and large numbers of skipping steps, the correlation statistically insignificant. The test details be found in Appendix A.3. The overall skipping algorithm is shown in Alg. (2) of Appendix A.2.5.
4 Experiments
4.1 Experimental Setup
Models. We conduct experiments in three prompt-based settings including text-to-image (T2I), image- to-video (I2V), and text-to-video (T2V) generation tasks. In addition, we also test the effectiveness of AdaptiveDiffusion on the conditional image generation task. For the T2I task, we use Stable Diffusion-vl-5 (SD-1-5) [33] and Stable Diffusion XL (SDXL) [32] and evaluate on three different sampling schedulers (i.e., DDIM [39], DPM-Solver++ [24], and Euler). For the 12V and T2V tasks, we utilize 12VGen-XL. [52] and ModelScopeT2V [44] respectively. Note that we use ZeroScope-v2 instead of the original ModelScopeT22 model to generate watermark-free videos. For conditional image generation, we use LDM-4 [33] as the baseline model.
Benchmark Datasets. Following [27], we use ImageNet [7] and MS-COCO 2017 [21] to evaluate the results on class-conditional image generation and T2I tasks, respectively. For the I2V task, randomly sample 100 prompts and reference images in AIGCBench [8]. For the T2V task, we conduct experiments on widely-used benchmark MSR-VTT [45] and sample one caption for each video in the validation set as the test prompt. More details can be found in Appendix A.3.
Comparison Baselines. We compare AdaptiveDiffusion against DeepCache and Adaptive DPM- Solver in both generation quality and efficiency. Deepcache [27] caches high-level features of UNet to update the lewelevel features at each denoising step, thus reducing the computational cost of UNet. The latter [23] dynamically adjusts the step size by combining different orders of DPM-e..ver.
Evaluation Metrics. For all tasks, we evaluate our proposed method in both quality and efficiency. We report MACs, latency, and speedup ratio to verify the efficiency. For the image generation task, following previous works [17, 16, 18), we evaluate image quality with commonly metries, i.e.,

---

Table 1: Quantitative results on MS-COCO 2017 [21].
Model
Scheduler
Method
PSNR
LPIPS
F
MACs (T)
Mem (MB)
Latency
Speedup Ratio
Deepeache
19.05
0.199
6.96
26
3940
1.7
1,58x
DDIM
Ours
21.74
0.131
5.22
22
3850
1.5
1.77x
SD-1-5
Adaptive DPM
19.0
0.195
6.38
31
3896
3.5
1.25x
DPM++
Ours
23.2
0.092
4.06
25
3852
2.8
1,57
Deepeache
18.89
0.240
7,51
26
3894
1.6
1.57x
Fuler
Ours
20.60
0.157
21
1.3
1,98x
Deepeache
21.9
0.221
7.34
162
14848
7.4
1.75x
DDIM
Ours
25.4
0.141
5.13
186
14460
7.8
1,66x
SDXL
DPM+
Despeache
21.3
0.255
8.48
162
14800
7.6
1.74x
Ours
26.1
0.125
4.59
190
14454
8.0
1.65x
Euler
Despeache
22.0
0.223
7.36
162
14796
6.3
2.16x
Ours
24.33
0.168
6.11
174
14458
6.7
2.01x
Peak Signal Noise Ratio (PSNR). Learned Perceptual Image Patch Similarity (LPIPS), and Fréchet Inception Distance (FID). For the video generation task, we use per-frame PSNR and LPIPS to measure the quality of generated videos. Besides, Fréchet Video Distance (FVD) [43] is also used to quantify the temporal coherence and quality of each frame. Note that since our method achieves adaptive acceleration results, all reported metrics of our method averaged across all prompts.
Implementation Details. We conduct all experiments on RTX 3090 GPUs. For SD-1-5 and SDXL models, the original sampling timesteps ''' are set as 50, and two hyperparameters are set as: 5 = 0.01, Caax 4. For LDM-4, T 250, 5 = 0.005, Cama 10. For 12VGen-XL and ModelScopeTTV. T = 50, 5 = 0.007, Cmax = 4. More details of other methods are listed in Appendix A.3.
4.2 Main Results
4.2.1
Results on Image Generation
We first evaluate our method T2I generation. As shown in Tab. 1, compared to DeepCache and Adaptive DPM methods, Adaptive achieves both higher quality and efficiency in various settings. For example, AdaptiveDiffusion achieves 0.092 LPIPS SD-v1-5 [33], generating almost lossless images compared to those generated by the full-step denoising process. Mcanwhile, the averaged speedup ratio across all testing prompts achieves 2.01 x when using SDXL and Euler sampling scheduler. In addition, when comparing the generation performance between different models (e.g., SD-1-5 and SDXL [32])) and schedulers (e.g., DDIM, DPM- - and Euler), AdaptiveDiffusion shows stronger generalization capability to adapt to different settings.
Table 2: Quantitative results on ImageNet 256 [7].
Model
Method
PSNR
LPPPS
FID
Mom (MB)
MACs (T)
Latency (S)
Speedup Ratio
LDM4
Deepeache
24.98
0.098
5.22
3030
5.6
1.4
6.35x
Ours
25.79
0.090
4.91
2912
6.5
1.6
5.56x
To further verify the effectiveness of our method on different image generation tasks, we also conduct experiments on LDM-4 for the conditional image generation task. As shown in Tab. 2, due to the larger timestep setting in the original denoising process, the slow change of latent change allows us to achieve faster generation with almost lossless quality at nearly 5.6x speedup. Compared with Deepeache, the AdpativeDiffusion obtains a better trade-off between generation quality and efficiency.
4.2.2 Results on Video Generation
We further conduct experiments more tough tasks, e.g., video generation tasks including I2V and T2V generation. As shown in Tab. 3, AdaptiveDiffusion can generate videos of lossless frames and similar video quality with significant speedup against the original models' full-step generated videos. Specifically, our proposed method can achieve a significantly higher quality in single frame evaluation which can be seen by LPIPS and PSNR (e.g., 66.38B PSNR compared to DeepCache using V-n-nnn........ On the other hand, AdaptiveDiftusion can ensure temporal consistency as the original models due to the property of lossless acceleration which can be reflected on FVD.
4.3 Ablation Studies
Ablation Study on Skipping Threshold. As shown in the upper part of Tab. 4, we first analyze the effect of the skipping threshold 8. It can be observed that when the skipping threshold relatively small, there will be fewer skipping steps, resulting in relatively small yet still clear acceleration with
7

---

Table 3: Quantitative results on video generation tasks.
Model
Method
PSNR
I.PPSS
FVDD
Mem (GB)
MACs (T)
Latency (6)
Speedup Ratio
12VGen XL
Deepeache
20.82
0.212
916.0
28.0 (+58.4%)
1878
2.00
Ours
27.20
0.088
274.5
17.7 (+0.3%e)
1983
1.93x
ModeIScopeT2V
Deepcuche
18.24
0.343
3383.0
14.2 (450.8%)
1034
1.37x
Ours
27.70
0.081
470.6
8.7 (-7.8%)
948
42
1,46x
Table 4: Ablation studies on hyperparameters using SDXL [32]. using 50-step Euler sampling scheduler for SDXL on COCO2017.
We conduct the ablation studies
Hyper-purams
Value
PSNR
LPIPS L FID L
MACs (T)
Latency (s)
Speedup Ratio
0.005
34.3
0.023
0.96
262
9.7
1.38
0.008
28.6
0.079
217
7.7
1.74
0.01
24.3
0.168
6.11
174
6.7
2.01
(Cmaz = 4)
0.015
21.0
0.282
9.49
119
5.1
2.64%
0.02
21.0
0.289
9.79
106
4.4
3.06x
4
24.3
0.168
6.11
174
6.7
2.01x
Cmaz 0.01)
6
23.3
0,217
7.74
147
5.7
2.37x
8
22.7
0.256
135
2.59x
10
22.1
0.307
11.91
123
4.7
2.86x
much higher preservation quality. With the threshold gradually increasing, the speedup ratio will be largely improved but at the cost of quality degradation. It can be seen that the image quality does not significantly change with large thresholds (i.e., 0.015 and 0.02). This is because the pre-defined maximum skipping steps prevent the further increase of skip steps. In this paper, we set the skipping threshold to 0.01 which is better trade-off between the generation quality and inference speed.
Ablation Study on Maximum Skipping Steps. Further, we conduct ablation studies on the maximum skipping steps Cmux, as shown in the lower part of Tab. 4. With the increase of the maximum skipping steps, the quality generated images continuously decreases. The reason is that when the timestep I; approaches 0, large number of denoising steps will be skipped due to the minor value of the third-order latent difference when the max-skipp max-skip-step is relatively large. The phenomenon reveals that Cmaxx can effectively prevent the continuous accumulation of generated image errors and ensure image quality.
Analysis on Sampling Steps. To evaluate the effectiveness of AdaptiveDiffusion few-step sampling. can seen from Tab. that AdaptiveDiffusion can further accelerate the denoising process under the few-step settings. Note that the hyperparameters (i.e., 6, and Ciax) should be slightly adjusted according to the original sampling steps due to the varying updating magnitudes in different sampling steps. Specifically, higher threshold 5 and lower max-skip-sted C'axx make better generation quality when reducing the sampling steps.
Table 5: Study on few-step sampling. Acceleration results with different original sampling steps
using SDXL. [32] on COC02017.
Steps
PSNR f
LPIPS 4
FID
MACs (T)
Latency (s)
Speedup Ratio
50 steps
24.3
0.168
6.11
174
6.7
2.01
25 steps
32.9
0.047
1.62
128
5.8
1.31
15 steps
19.9
0.122
5.06
70
3.1
1.38x
10 steps
29.4
0.169
8.28
53
2.5
1.21x
4.4 Visualization Results
Generation Comparisons. To demonstrate the effectiveness of Adaptive more intuitively, show some visualization results. Fig. shows the results of the text-to-image generation task, it can be seen that AdaptiveDiffuion can better maintain image quality compared to Deepcache with nearly equal acceleration. Since AdaptiveDiffusion can adaptively determine which steps can be skipped, unimportant steps that have little impact on the final generation quality will be skipped during inference. Besides, to demonstrate the generalization of AdaptiveDiffusion different tasks, we provide video generation results in Fig. 7. More generalization results can be found in Appendix A.4

---

Original
Cache
Original
DospCache
Prompt: clock front park
Pronpe: sandwick toasted bread
Prompt: two dogs are playing lugether the gnass
Prompt: a majestic bear looks oui acruss a grazs plain
Prompt: yellow bus traveting down a woz park
Prompt topped with bags luggaage and purrne
Figure 6: Qualitative results of text-to-image generation task using SDXL and SD-1-5 MS-COCO 2017 benchmark. Left: SDXL, Right: SD-1-5.
Prompt: Amidet worron alleys of
knight secratly whitpwring animols cuptured style of in pointing
Original (X) speccup))
DeeyCoohe (K2.0 (x2.0 sperdup)
(x2.1 speeduu)
Reference sauge
Frame
Figure 7: Qualitative results of image-to-video generation task using 12VGen-XL. on AIGCBench.
Dapoliiig Jades
Tetl Woober of Noia Updat Soga
(c)
Figure 8: (a) Skipping paths under different skipping targets obtained by the greedy search algorithm. (b) Skipping paths under different skipping thresholds by the third-order estimator, (c) The frequency distribution of the skipping number of noise update steps for SDXL generating images on MS-COCO 2017 benchmark.
Denoising Path Comparisons. Fig. 8 illustrates the distribution of skipping paths at different skipping schemes. It can be observed from Fig. 8a and 8b that when the number noise update steps keeps decreasing (more blank grids in the horizontal lines), both greedy searched paths and third-order estimated paths tend to prioritize the importance of early and late denoising steps. From Fig. 8c, find that most prompts in the MS-COCO 2017 benchmark only need around 26 steps of noise update to generate an almost lossless image against the 50-step generation result.

---

Conclusion
In this paper, explore the training- -free diffusion acceleration and introduce AdaptiveDiffusion which can dynamically select the denoising path according to given prompts. Besides, we perform the emor analyses of the step-skipping strategy and propose computation redundancy. Experiments are conducted on MS-COCO 2017, ImageNet, AIGCBench and MSR-VTT, showing good trade-off between high image quality and low inference cost.
the third-order estimator to indicate the
Acknowledgements
The research supported by the National Key R&D Program of China (Grant No. 20222D0160104), the Science and Technology Commission of Shanghai Municipality (Grant No. 22DZ1 100102), Shanghai Municipal Science and Technology Major Project (Grant No. 2021SHZDZX0102) Shanghai Rising Star Program (Grant No. 23QD1401000) National Key Research and Development Program of China (No. 20222D0160 60101), Shanghai Natural Science Foundation (No. 23/R144 1402900), Shanghai Municipal Science and Technology Major Project (No.2021SHZZ) (0002020HHDD National Natural Science Foundation of China (No. 62071127 and 62101137).
References
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis. Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, ediffi Text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint ar xrXiv:2211.01322 2022.
[2] Omer Bar Tal, Hila Chefer, Omer Tov, Charles Hermaan, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li. Tomer Michaeli, et al. Lumiere: A space-time time diffusion model for video generation. arXiv preprint ar 2401 1294, 2024.
[3] Andreas Blattmann, Tim Dockhor, Sumith Kulal, Daniel Mendelevitch, Macicj Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Stable video diffusion: Scaling latent vidco diffusion models to large datasets. arXiv preprint arXiv:2311.15127 2023.
[4] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024.
[5] Xuanyao Chen, Zhijian Liu, Haotian Tang, Li Yi, Hang Zhao, and Song Han. Sparsevit: Revisiting activation sparsity for efficient high-resolution vision transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2061-2070, 2023.
[6] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Won Ha. Stargan v2: Diverse image synthesis for multiple domains. In CVPR, 2020.
[7] Jia Deng, Wci Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. lece, 2009.
[8] Fanda Fan, Chunjie Luo, Wanling Gao, and Jianfeng Zhan. Aigebench: Comprehensive evaluation of image-to-video content generated by ai. BenchCouncil Transactions on Benchmarks, Standards and Evaluations, 104)1, 2023.
[9] Gongfan Fang, Xinyin Ma, and Xinchao Wang. Structural pruning for diffusion models. Advances in neural information processing systems, 36, 2024.
[10] Ben Fei, Zhaoyang Lyu, Liang Pan, Junzhe Zhang, Weidong Yang. Tianyue Luo, Bo Zhang, and Bo Dai. Generative diffusion prior for unified image restoration and enhancement. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 9935-9946, 2023.
[11] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020.
[12] Akio Kodaira, Chenfeng Xu. Toshiki Hazama, Takanori Yoshimoto, Kohei Ohno, Shogo Mitsuhori, Soichi Sugano, Hanying Cho, Zhijian Liu, and Kurt Keutzer. Streandiffusion: A pipeline-level solution for real-time interactive generation. arXiv preprint 1rX44 2023.
10

---

[13] Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. In ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelikood Models, 2021.
[14] Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
[15] Lijiang Li, Huixia Li, Xiawu Zheng, Jic Wu, Xuefeng Xiao, Rui Wang, Min Zheng, Xin Pan, Fei Chao, and Rongrong Ji. Autodiffusion: Training-free optimization of time steps and architectures for automated diffusion model acceleration. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 7105 10,-7114, 2023.
[16] Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang. Han Cai, Junjie Bai, Yangqing Jia, Ming Yu Liu, Kai Li. and Song Han. Distrifusion: Distributed parallel inference for high-resolution diffusion models, arXiv preprint ar 2402 19481, 2024.
[17] Muyang Li, Ji Lin, Yaoyao Ding, Zhijian Liu, Jun-Yan Yan Zhu, and Song Han. Gan compression: Efficient architectures for interactive conditional gans. In CVPR, 2020.
[18] Muyang Li, Ji Lin, Chenlin Meng. Stefano Ermon, Song Han, and Jun-Yan Zhu. Efficient spatially sparse inference for conditional gans and diffusion models. NeurlPS, 2022.
[19] Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys. Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Snapfusion: Text-to-image diffusion model on mobile devices within two seconds. NeurIPS, 2023.
[20] Shanchuan Lin, Anran Wang, and Xiao Yang. Sdxl-lightming: Progressive adversarial diffusion distillation. arXiv preprint arXiv:2402.1 13929, 2024.
[21] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[22] Zhijian Liu, Zhuoyang Zhang. Samir Khaki, Shang Yang. Haotian Tang, Chenfeng Xu, Kurt Keutzer, and Song Han. Sparse refinement for efficient high-resolution semantic segmentation. arXiv preprint 2rXiv:2407.19014 2024.
[23] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion prohabilistic model sampling in around 10 steps. arXiv preprint arXiv:206.000927, 2022.
[24] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solvert+: Fast solver for guided sampling of diffussin probabilistic models. arXiv preprint arXiv:2211.01095, 2022.
[25] Simian Luo. Yigin Tan, Longbo Huang. Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint 10.042231 2023.
[26] Zhaoyang Lyu, Xudong Xu, Ceyuan Yang, Dahua Lin, and Dai. Accelerating diffusion models via early stop of the diffusion process. arXiv preprint arXiv:2205. 12524, 2022.
[27] Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free. In Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[28] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14297-14306, 2023.
[29] Tachong Moon, Moonseok Choi, EungGu Yun, Jongmin Yoon, Gayoung Lee, and Juho Lee. Early exiting for accelerated inference in diffusion models. In ICML 2023 Workshop on Structured Probabilistic Inference ( Generative Modeling, 2023.
[30] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusioo probabilistic models. In ICML., 2021.
[31] Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In CVPR, 2022.
[32] Dustin Podell, Zion English. Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high righ-resolution image synthesis. In ICLR, 2024.

---

[33] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffuusion models. In CVPR, 2022.
[34] Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assister Citrrventtoo-MMCCAA 2015: 18th International Conference, Munich, Germany. October 5-9, 2015, Proceedings. Part III 18. pages 234-241. Springer. 2015.
[35] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR. 2021.
[36] Axel Sauer, Frederic Boesel, Tim Dockhorn. Andreas Blattmann, Patrick Esser, and Robin Rom- bach. Fast high-resolution image synthesis with latent adversarial difftusion distillation. arXiv preprint arXiv:2403. 12015, 2024.
[37] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint arXiv:2311.17044 2023.
[38] Andy Shih, Suneel Belkhale, Stefano Ermon, Dorsa Sadigh, and Nima Anari. Parallel sampling of diffusion models. NeurlPS, 2023.
[39]
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2020.
[40] Yang Song. Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. 2023.
[41] Haotian Tang. Shang Yang, Zhijian Liu, Ke Hong, Zhongming Yu, Xiuyu Li, Guohao Dai, Yu Wang, and Song Han. Torchsparse**: Efficient training and inference framework for sparse convolution on gpus. In Proceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture, pages 225-239, 2023.
[42] Shengkun Tang, Yaqing Wang, Caiwen Ding, Yi Liang, Yao Li, and Dongkuan Xu. Deediff: Dy- namic uncertainty-awe awarre early exiting for accelerating diffusion model generation. arXiv preprint arXiv:2309.15 17074, 2023.
[43] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019.
[44] Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang. and Shiwei Zhang, Modelscope text-to-video technical report. arXiv preprint 230.0030.. 2023.
[45] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings the IEEE conference computer vision and pattern recognition, pages 2016.
[46] Xingyi Yang, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Diffusson probabilistic model made slim. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22552- 22562, 2023.
[47] Hancheng Ye, Chong Yu. Peng Ye, Renqiu Xia, Yansong Tang, Jiwen Lu, Tao Chen, and Bo Zhang. Once for both: Single stage of importance and sparsity search for vision transformer compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5578-5588, 2024.
[48] Fisher Yu. Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.0 03366.0336500 2015.
[49] Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In ICLR, 2022.
[50] Qinsheng Zhang, Molei Tao, and Yongxin Chen. gddim: Generalized denoising diffusion implicit models. 2022.
[51] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effecctiveness of deep features as perceptual metrie. In CVPR, 2018
[52] Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. 12vgen-xl: High-quality image-to-vides -iiicoo synthesis via cascaded diffusion models, arXiv preprint a Xiv:2311.00145, 2023.
12

---

A
Appendix
Due to the nine-page taee limitation of the manuscript, we provide more details and visualizations from the following aspects:
Sec. A.I: Limitations and Broader Impacts.
See, A.2: Method Explanation.
Sec. A,2.1:
Theoretical Proof of Single-step Skipping.
Sec. A.2.2: Theoretical Proof of Two-step Skipping.
Sec. A.2.3: Theoretical Proof of k-step Skipping.
Sec. A.2.4: Theoretical Relation between the 3rd-order Estimator and Optimal Skipping Strategy.
Sec. A.2.5: Algorithms.
Sec. A.3: More Experimental Details.
Sec. A.3.1: More Evaluation Details.
Sec. A.3.2: Prompts AIGCBench.
Sec. A.4: More Generation Visualizations.
A.1 Limitations and Broader Impacts
Currently, AdaptiveDiffusion only evaluated on image generation and video generation tasks. A long-term vision is to achieve lossless acceleration any modalities generation such as 3D and speech. Besides, although AdaptiveDiffusion can improve the speed on few-step (e.g., steps) settings, it may not work for extreme few-step generation for the rapid changes of the latent features.
A.2 Method Explanations
A.2.1 Error Estimation Induced by Single-step Skipping
In Sec. 3.2, the error between the latents of the continuous noise update and one-step skipping of noise update is stated to be upper-bounded of the difference between the previous two output latents. Here, provide the detailed proof of this statement.
We first make the assumptions that: (1) (x(x,)) t) Lipschitz w.r.t to its paramters x and t;
Assumptions
(2) The 1st-order difference Ar,- T, 24+1 exists and is continuous for 0 i 1
Proof Take i-th step an example to perform the one-step skipping of noise prediction, we can obtain the following update formulations.
2++1 f(i + 1) z1+2 - g(i + 1) c0(222 ti+2);
2: f(i) -:41 - -i() en (2 fz41):
(5)
#: fli -- 11) 1).2.- - g(i 1) (e(241,4t1))
2:-2 f(i -2) -x.- g 2) eo c0(
Here, the step of noise prediction replaced by reusing the i + 1-th noise prediction. Then, the error caused by skipping i-th noise update, E;-1 || - ||. can he derived as follows.
|| = llo(t 1) [eo t441 es (x.,
= lgg(i - 1) [60( eo (Ix, (441) + co(as, f441) co(
(6)
< [ - 1) O44 4+11)
|lg(i - 1) O(x #++1)ll
O(t, t++1) + O(x: 2++1).
The proof utilizes the property that (2,, t:41) ea c0( faa))l )I and [|ca t.) - co (x,, 444)) Illl are upper- bounded O(æ: 2++1) and Ott, 4+11 respectively according the Lipschitz continuity.
13

---

A.2.2 Error Estimation Induced by Two-step Skipping
We further estimate the situation of consecutive two-step skipping of the noise prediction model. Assuming that noise prediction is skipped at the i-th and (2 1)-th steps. then we have the following formulation.
2++1 f(i + 1) -2,-2 - g(i + 1) t0(2442t442):
J, f(i) #1+1 - g(() c (0(
(7)
f(i 1) .x; 1--9 g(1 1) co li41);
2, f(i 2) - g( 2) cs (411))))41)))
The error, 6.-2 = ||:-2 om | can be derived as follows.
6--2
f 2) (z:-1 - g(i 2) (eo
[ 2) g(i - (eo ca (x(411,)4))
g(i 2) fo-
[|A((i 1) -
(8)
g(i 2) (-0
$ ||A 1)- Ott, - (. || - r1+1)|
+ |lo - 2) 0(;--1==:1 -|-1|||
Ol (241) O(2: mm41) + 0 4.) 0 -
From the above derivations, it can be observed that the skipping error clearly related to and upper-bounded by
the accumulation of previous latent differences. Here h² (i- 1) is defined as h2 (i- 1) == (i- (f- 1) (i- 2). The above conclusion of skipping error's upper bound can be casily extended to any situation where finite-step skipping is used.
A.2.3 Error Estimation Induced by k-step Skipping
Take i-th step example perform the k-step (k: 2) skipping of noise prediction, obtain the following update formulations.
Ø; f(i) #441 - g(() c0 c0((111,
x1-1 (i-
(0--))- 1111), - gg(( (i - 1) eo (441)1441);
fi-2 f( 2) -1,--11 (i 2) (c141:441):
:
-i f(i k) 01 9 (i k)) (1141,
ari Fi -
f (i k)) (i--++1 -2- 9 (i k) (co (1441, (00
(t k) 6i-bt+1 + g (1- k:) lleo (01+1,44+1) - eo (T-- 4-x+1)l
(9)
h-m+1 m) 0 (ts-m+1 (=-m+2)
1/4 k-m+1 (i m) (z--m+1 2z mm-
+ l|g (i k)) O (Ii - x-++22) 1lg (i k) O (11-++1 - 4-++2))l
Σ (t-m+t1 +2) + (z1-m+1 x1-m+2).
m=1
The derivation also utilizes the property that (as, t441) c0(41 to ) and [|eo(ms, tx) - cx (0( fu ) are sppee-bbuudded by O(r; #i+1) and O(I 4-11) respectively according to the Lipschitz continuity. Here k- (i m) defined k- (i 1-mm m) = g m) I ( - j).
From the above derivation, it can be observed that the error of an arbitrary k-step skipping scheme is related to and upper-bounded by the accumulation of previous latent differences. Therefore, if the skipping step of noise
14

---

prediction is large, the upper bound of the error will naturally increase, which is also empirically demonstrated by Fig. 5b.
A.2.4 Theoretical Relation between the 3rd-order Estimator and Optimal Skipping Strategy
To explore the theoretical relationship between the third-order estimator and the skipping strategy. need formulatc the difference between the neighboring noise predictions. According Eq. (I), we can get the following first-order differential equations regarding the latent x.
Ar; 2i 2i+1 = [1 1- (f) 2i41 - g (f) 6o 10++); Ar.-1 I: 24 = [1 - f (i - 1)] 0: - g - 1) co (1r, ()).
(10)
Now, let (i) 1- (i- 1), and we further derive the sccond-orded differential equations based on the above equations.
Az;-1 Am;
=u (i)x, u (i 2:41 + (1) (1-11,441 (i 1). (1i,14)
(i) (e; 11-1) + (1) T:-1 u1 (2:41 - 2.) (i 1) x. + (i) co (0:41,441)
- (i 1) -0 (z., t2)
(11)
=u (i) Ari-1 - u (i + Am + (u ( (i) (##+1, f++1) g 1) (0t, f))
=u (i) 4z;-1 u (i 1) Am + 4 (1) (0) (i) (co (1:41,441) (æi, 4:)
[g (i) 9 (i 1)] A0 (2., t.)
=u (i) A7;-1 u (i + 1) Az + A (u x1) (1) A Ag (1) (z),
After simplification of the above equation, we get the following formulation:
f(t (1- Az) Az. - (1) Ac Ag) co (2),4).
(12)
From the above equation, we can observe that the difference between noise predictions Aeg is related to the first- and second-orde derivatives of z., as well as the noise prediction ea (z,, t.). Therefore, it would be difficult estimate the difference without (xx, Now consider the third-order differential equation. From the above equation, we further obtain the following formulation.
(1) Ar; - (( 4m.4 = A [u (1 (1+1)) ( + Ar
Ag (i + 1) cp (0441,441)
*A () Ar:-1] A (1) Az,] (2) 1u (1)
Aeh AA4g (i) co (z,,
*4 Ag (i) (x;, -
(2) If (i 1) Az;-1 + (2) 1u (i) æ
(i) Act
(13)
From the above equation, it be observed that the difference of the neighboring noise predictions is explicitly related third- and sedond-order derivatives of 2z. as well as the second-order derivative of cj. Since lim,-+0 I (i) = 1, limp=+ u (i) = 0, limu-ro g (i) = 0. we can finally get the conclusion that lc-o = O (9) x1-11
A.2.5 Algorithms
Algorithm of greedy search for optimal skipping path. The specific details of the greedy search algorithm used the optimal denoising path shown in Alg. (1).
15

---

Algorithm Greedy Search for the Optimal Skipping Path.
Input: Noise Prediction Model ea. Sampling Scheduler 6, Decoder Fa. Target Skipping Step
Number N. Sample Step T, Conditional embedding c;
1: Initialize Skipping Path S = [True] T, Current Skipping Step Number nskip = 0.
2: Compute "0" by Eq. (1).
3: while Rakip <N do
4: Initialize =0:
5: for in range(T'' do
6:
if S[i]=TTrue then Lemp_path= S.copy(; temp_ ;app_aath[i]=Fale;; Generate temp by Eq. (2);
7:
8:
9:
10:
Compute ε difference Lo 1 -1001 df append((Lo)
temp
11:
12:
end if end for
14:
index argmin(df));
15:
Set Slindex] False:
16: end while
17:
return S.
Algorithm of the overall skipping strategy in AdaptiveDiffusion. The designed skipping strategy used in our AdaptiveDiffusion are elaborated in Alg. (2).
Algorithm 2 AdaptiveDiffusion.
Input: Noise Prediction
Model cp. Sampling Scheduler Φi Decoder Fa, Sample Step T, Conditional
embedding c, Maximum Skipping Step Number Cmax- Threshold 6;
1: Initialize Random Noise z, Skipping Path = [], Previous Differential List Pair []. Previous
Latent List l'asemt [].
2: for in range(T" 1) do if < 2 then
3:
4:
Ope = =0= c(x, i); if i then
5:
6:
Pai Bam-ndd [2 Pintent 1]]l::
7:
end if
8:
else
9:
if
S
S[-1]=Truu then
10:
Ope O e(x, i);
else
12:
O= Opi
end if
14:
end if
Compute ((x) by Eq. (1);
16: if 2 3 then
S.append( {(x)) in Eq. (4);
18:
end if apuem-append(r);:
20:
end for return Fa(x).
21:
A.3
Experimental Details
A.3.1 More Evaluation Details
Statistical Analysis the Estimated Skipping Path. In Fig. 5, we aim to evaluate the correlation between the estimated skipping paths by our method and the searched optimal skipping paths under different target skipping numbers. This evaluation is crucial to validate the effectiveness of our approach in accurately
16

---

predicting skipping paths. The paths are represented as sequences of binary choices (0 for a skipping step. for non-skipping step) each timestep. We employ x² test of independence assess the statistical correlation between the estimated and optimal paths. We first randomly select several prompts from the test set and use the greedy search algorithm to greedily search the optimal skipping paths under different skipping target steps. For the estimated skipping paths, we discard the setting of Cmax to test the effectiveness of the third-order estimation without regularization and refine the threshold 5 in wide range achieve various skipping paths with different skipping step numbers. Then, for each skipping step number, we construct 2 2x2 contingency table from the searched optimal skipping paths and the estimated skipping paths, with the same skipping step numbers. Finally, using the contingency table, we compute the x statistics and the corresponding p-value to assess the independence between the estimated and the searched optimal skipping paths.
Evaluation Tools and Details. Following [16], we use TorchMetries to calculate PSNR and LPIPS, and CleanFID [31]' to calculate FID. Since our method is adaptive method which means can choose suitable skipping path for different prompts, we set batch size during evaluation. The MACs reported in experiments are the total MAC's in the denoising process following [16]. Besides, the reported MACs, speedup ratio, and lateney are the average over the whole dataset.
CodeBase. For a fair comparison, when compared with Adaptive DPM-Solver, we use the official codchase of DPM-Solvers Experiments for LDM-4-G are based on the official LDM codebase and DeepCache codebase". Other experiments are based Diffusers'.
Details of Reproduced DeepCache. In this paper, we mainly compare our AdaptiveDiflusion with DeepCache. To ensure the generation quality, we set cache_interye interall 1-3 and cache_ cache_branch sa___rrnn_____he for SD-v1-5, SDXI, and ModelScope. Since the memory cost will he largely improved on video generation tasks es- pecially high-resolution tasks (e.g., 12VGen-XI.), We set cache_i i nt e eervall -3 and cache_ seree_rrae branch d-C to reduce the memory costs. For class-conditional image generation task, we follow the official setting and set cache interval=10.
A.3.2 Prompts in A AIGCBench
We randomly sample 100 prompts and their corresponding images as the test set for 12V task. Here, we give the list of the selected prompts in AIGCBench in Fig. and Fig. 10. The selected test set includes different styles of images and prompts such as animation, realism, and oil painting.
A.3.3 Experiments on Unconditional Image Generation
We provide the acceleattinn performance of different methods on pure image generation for further comparisons. Specifically, following the experimental setting in Deepcache, we conduct unconditional image generation experiments on CIFAR10 [14] and LSUN [48] datasets. As shown in Table 6. our method achieves a larger speedup ratio and higher image quality than Deepeache on both benchmarks.
Table 6: Performance on Unconditional Image Generation.
Datasets
Methods
FID 1
Speedup Ratio
Deepcache
10.17
2.07x
CIFARIO
Ours
7.97
2.09x
Deepcache Ours
9.13
1.48x
LSUN
7.96
2.35x
A.3.4 Effectiveness on SDE Solver
We also explore the effectiveness of our work SDE solver. Compared with the ODE solver, the SDE solver includes an additional noise item for the latent update, which is unpredictable by previous randomly generated noises. When the magnitude of random noise is not ignorable, the third-order derivative of the neighboring latents cannot accurately evaluate the difference between the neighboring noise predictions. Therefore, to apply our
'ttpps htttpp:://ithuub..oo http:ttwww......ggggoooo//oooooo
9https://00 thub. . http:h/cwa//ccooo//aaaaattttedee
17

---

method to SDE solvers, should design an additional indicator that decides whether the randomly generated noise minor enough relatively unchanged trigger the third-order estimator. In this case, design additional third-order estimator for the sealed randomly generated noise. When third-order derivatives both the latent and the scaled randomly generated noises are under the respective threshold, the noise prediction can be skipped by reusing the cached noisc.
To validate the effectiveness of our improved method, we conduct experiments for SDXL with the SDE-DPM solver COC02017. The results are shown in the following table. Compared with Deepcache, our method can achieve higher image quality with comparable speedup ratio, indicating the effectiveness of AdaptiveDiffusion on SDE solvers.
Table 7: Performance SDE-DPM Solver.
Method
PSNR
LPIPS
FID
Latency (s)
Speedup Ratio
Deepeache
16.44
0.346
8.15
9.2
1.63x
Ours
18.80
0.232
6.03
9.8
1.53x
A.4 More Generation Visualizations
Here, further show qualitative results tested text-to-image task using LDM-4 MS-COCO 2017 text-to-video generation task using ModeIScopeT2V on MSR- VIT, which are illustrated Fig. 11 and Fig. 12 respectively.
18

---

Bchold valiaut knight the thrser exploring surrounded the secret chambers underground laboratory, covisionel as vibrant pixel Discmer a mischiesoos fuicy, sagerty warcking for hidhos ireanore the awn-lit skysrupee a faturiatic city, artfully resdered through abetract lens of pie asso
Bchold playful pando the throer carcfully repairing braken robot sarounded by the dusy strects old western with the soft souh natereolor
high noon, envisioned
Discower a mischiesows fairy, ruoning morathon unidat the tranqusi watere verese lake, artfully rendered with the vilt touch of watercoior Encouster a nobll king they are engaged in valiuntly fighting mouitrou boast mysterion croszrads a mysiccal foresi, all depicied with the suft much watercolar
Amidst the front lines of as ancient battlofield, a gende ogre is butlduu e mowwss captured life a lively cartoon
Behold valiau lnight the thener cleverty solving paosle surroutied by a busting suace nttiow orhiling eorth, envisioned as stauning 3d render of ceraiean
Behold wailing bonchee the themes casting a quell recnunded the cracking van gogh
inpassione strokes of
Behold rugue robut in the throes of exphoring cave swreonded creaking pirate ship, owvisioned with the soft souch of wateroolor
of corlean 11. Amidit sunddapptdd forest vallant knight is casting a spell, captaned with photoroaltatic preciaiue
10. Amidat the comsbing
old searing through the sky capowed or vileant pixel
12. Within the rosbn of the noon-lit shyveropers of fatwrinic city regue rabot carting speil, each moment immortalized in the style of an off painting
13. Amidst the sandy shores of deserted Irland, noble king casting spoll, captured with the start realism photo
14. Discover a vailing banshoe, laugrill catlin a foust anidat
camag a doep juogle, artfilly ronlored with suft touck of watercolar
15. Amida the sondy shover of decerted islond, a heave girl is muticulously investtigating a mysteriou crize snowe, captured as vibvant pixel art
16. Encounter mise bieeus fairy as they engaged conducting on experiment the front lines
anctent battlefield, all depicted as vibrant pirel
17. Discover a ffoeest-dddiiie doelling nyuph, exploring a cave amidst crashing naves of coralean sen, artfuliy rondored through the abstract loss of pleusso
18. Amidat the tranquil waters of sevrar lakae misehiemuu foiry is husting for ghouts, cxpoured with photoraalzzic procision
19. Amidit the carie halls of haunied house, plagft pando sncretly whispering animals, cuptured in the styic of an oil painting old performing mangce
of ceralan 21. Discover a woble bing brewing a pation amidet the sandy shores a deseried island arthully rendered as a suuning 3d render
20 Amidat the cousbing
cuptured like Inwely
22. Amida the sandy shores a deserted island, battle-ccuree by slorg is building snownann cuptured with the stark reatise a photo
deverted islend, a ployful panda casting spell, cuptured 24. Discover a fearsome dragon, secretly whispering to animals amidst the vibrant heart of busting market square, artfully rendered with the reft touch of watcrcolor
Amida the soudy
style an pointing
25. Behold treasure sooking pirate the threes of exploring picano
mirounded the ochoing depths a magical crvisioned through the abstract
26 Disewer a mischiesoer fairy, rutning morathon amidet a bostlin ypare station whhinig earth artfully ronderedi as vibruct pisel ar
27. Behold wise old man throes of painting masterpiece surramdedd by the neon it skoserpeess of funurstiic city, enviiioed ar vibrant pixel art
25. Within the realm of achuing degnis of nagicul cave, gontle agre custing a spell,
moment inmortuliied with the soft toac vuterculor
20 Discower a pleeful ponds, rumning morathon unids the tranquil waterr of a serone lake, arifilly condered is the inparsionnd strkke of von gogh
blooming enchanted garden, ali depicted with photorenlittic precision 31. Within the realm of echuing depris of magical cave, bamil searred cyborg masterfully riding bike, each woment inmortalized through
30. Encounter woble king they engaged building
abstract lows of pieasso
32. Amidat the narrow alieys of mediesal tonn, a vallant buight secretly whispering to animals, captured in the style of uil painting
33. Disewer fovet-buellingg nyuph, deciphoring map awides creaiing pirate ship, artfully rendered like lively
34. Behold gentle agee the throes of marterfally riding a bike nerounded by the backdrop of on alten planet's red shies, envisioned in the impasrioned strokes of van gogh
35. Escouster mystical mormaid they expaged in cleaerly solving passle resttrrioss consorodd mystical fivest alt depicted with the sork
realim a photo
36 Rehold curinus alien in the thooes of playing the violin suroondedd by creaking pirate ship, envisioned as vibrant pixel art
37. Behold curious alten the threes of skilifilly strutiing the guteer sureondedd the narrow alleys modienal envisioned the style an oil painting
38. Rehold noble king tn the thmes of wasting a spell nurrouned by the acon-lit shywrrgeess of fittristii city, ravisiinnd with photorealiviic prerision
39. Discover a brove secredly whisperigg animals anider the duary strects of old western tonn at high ortiilly reniered snunning 3d render
40. Encomnter a playful pando as they are mugaged in playing the violin the tranquil waters of verese lake. all depicted through the absiract lovs of piearro
41. Encounter measure- seehing pirate they engaged in deciphering map the backsinp alien planet's red sioer, all depicted with the stark reaiim a photo
42. Encouster a hrave girl as they are engaged in confscting an experiment the sandy shores of desertedi island, all depictesi with the reft touch of watercolor
43. Within the painiing
evvie balls of hannted huuse, trawwr--eedingg pirnte husting for ghosts, wach inmortaliodd style of oll
44. Encounser a fearsome dragon they are engaged in exploring a caee the sundy shores of deserted Island all depicted in the style an orl paturing
45. Diseowe bease girl, skilifull struaming guitar
sun-duppled furest, artiully readived like livtyy 46 Discuver a valiant lnight, burlding a mowsue amnder the bachdrop of an alien pianet's red skies, artfully rondered in the style of an oil pointing
47. Encrmnter a plagui pando as they are iigaged in conducting ding an experiment the fronty peok of soorc monntaiin alf depicted is samning 3d mende
48. Encounter woble king they engaged funting for ghosts the echoing depths of magical cave, all depicted through the abstract iens of picaseo
49. Encounter a mystical mormald as they are ungaged in hungr cating a feast uncient walls a crambling caslle, all depicted the inpassionnd
strokes of soe pogh
50. Amidit sun-dappied forest, mischievous fairy is carefully repsiring broken robot, captured in the style of oil
51. Encounter a noble king they are eugaged deciphoring map the fresty peak of mony mountals, deplcted the style oll painting
52. Amiddi u wrekinng pirate ship a curinus alwon comnucting an exporiment captored in the impassionnd strnkes of vae gough
53. Discover ew lous alien, curefully repatring broken robut amidi: the carie halls of haunted house, artfally rendored like Eiely cartoon
54. Escrunter a gentus sriestist they engaged souring theough da the vibrant heart busting warket square, depicted like lively cartioe
53. Discover a foreee sholling nymph, dee ihhheinng map amidet the ancient walls of a crumbling cantle, artfully rendered like a lively cartoon
56. Amidst the sandy shores af deserted bllnd, mystical mermald is putnting masterpicce, captsred like lively cartoon auidit ma--duppled
57. Disewer a Huu-zraveling scholan performing magic
arthilly roudoved la the style on painting 58 Amidht the vibrant heart a buzting market square, a mysiccal mermaid is cleverly solving a puzzle, captored as vibront pixei art
59. Behold thne travellig scholar in the throes of paiating masterpicce survunded by run duppled forest, ewvlsioned with the seft touch of watercolor Encouster a gentle agre us they ougaged in susterfully riding a like bunting space stution arbiting surth, all depicted like lively
Figure 9: Prompts used in AIGCBench.
19

---

Encounter an odventuroas astronant they are ongaged brewing a potion bustling space station orbiting earth, all depicted with photorealittic precision
Eacounter plazful panda as they are ongaged in cleverly solving a puzzle narrow alleys of medieval town, depicted derough abstract lens of picusro
63. Discover noble king painting a mastorpiece amidst the craching waves of the cerulean sea, ortfully rendered derough the abstract lons picasse
64. Amidst the ancient walls of crumbling castle, time time-traveling scholar is painting mazterpiece, captured as vibrant pixel art
Discover bruve girl, secrelly whispering to animals amidst a blooming enchanted gurden, artfuily rendered with photorealistic precision
66. Behold gende ogre in the throes of lungrily cating a feart surrounded by the cerie halls of hammted house, envisioned with photoreulistic precision
67. Behold a genius scientist in the throes of secretty whispering animals surrounded by the sundy shores of deserted island, envisiined like lively cartoon
68, Encounter brave girl they are engaged conducting experiment sun-sappped forest, all depicted with the stark realism a photo
the realm of a Mooomnng enchanted gurdle, a rogue robot deciphering map, cach moment immortalized lively cartoon 70. Wishin the realm of sun dappled forest, a curious alien valiantly fighting a monstrous beast, each moment immortalized through the abstract lens of pleasso
71. Encounter valiant lmight they engaged in gracefully dancing under the mounlight the narrow alleys of medieval town, all depicted with the seft much of watercolor
Whthin the realm a creaking pirate ship, mystical mermaid cleverly solving a puzzle, each moment immortalized with the soft touch of watercalor
Amidst sandy shores of deserted island, a fearsome dragon lunting for ghorts, captured with saft touch watercoior
74. Encounter wailing baushee they engaged in conductiing experiment the ruins on uncient temple, all depicted with the stark
realism of photo
Amidst sun-dappled forest, brave girl meticulowsly investigating mysterious crime scene, captured as a stunning 3d render
76. Amidst the frusty peak of snowy mountain, a wise old man is performing magic tricks, captured like lively cartoon
Discover trewsare-seeking pirate, exploring a cave amidst the vibront heart of buttlnng market square, ortfully remdered in the impassioned strokes of gogh
75. Amidst the narrow alleys of medieeaal town, a gentle agre is soaring through the sky, cuptured with the stark realism of a photo
Behold treasure-seeking pirate the throes hungrily eating feart survounded the ruins of ancient temple, envisioned impassioned strukes of van gogh
Behold a wailing hanshee the thrses of exploring cave surrounded
vibrant heart of busting market square, envisioned as
staminn 3d render
81. Amidst the ruins of an ancient tempie, a playful panda is ramning a marathon, captured like a lively cartoon
old man, brewing a potion amidst the cerie halls of a haunted house, artfully rendered in style of oil painting 83. Wishin the realm of the dusty streets of an old wertem town at high noon, a fearsome dragon meticulously investigating a mystrriou crime scone, each moment inmortulized with photovealistic precision
Discover
84, Behold genius scientist the throes of playing the violin survounded the frosty peak of snowy mountain, envisioned in the style of
on oil paintin
Discover noble king, painting a masterpiace amidst the raine ancient temple, ortfully rendered with photorealistic precision
86. Discover an adventurous astronaut, conducting an experiment amidet the rains an ancions tomple, artflll rendered like lively cartoon
87. Discover genias scientist, conducting experiment amidst bustling space station orbiting earth, artfully rendered vibrant pizel art
85, Within the realm the frouty peak snowy mountain, fearsome dragon hunting, for ghosts, each moment immortalized with photoreulistic precision
Encounter treasure-sed seeking pirate as they are engoged in gracefully dancing under the mooulight dusty streets of an old western at high noon, all depicted as stunning 3d render
90. Wishin the realm of the front lines of an ancient battleffeid, mischievous faary meticulously investigating a mysteriou crime scone, each moment immortulized the impossioned strokes of van gogh
91. Encounter rogue robot as they are engaged in deciphering a map the sundy shores of deserted island, all depicted with the soft touch
of matercolor
92. Encounter curious alien they engaged deciphering map the bachdrop of alien planet's red skies, all depicted like a lively cartoon
Discover forest-dwelling nymph, brewing a potion amidst the canopy of deep jungle, artfuily rendered with photoreulistic precision
Enceunter gonlus ocientist they are engaged painting a mastorplace sandy shores of deserted island, all depieted a lively cartoon
95. Wishin the realm of the vibrant heart of buslling market square, a playful panda deciphering a map, each moment immortalited as vihrant picel
96. Wishin the realm of zun dappled forest, a mystical mermaid brewing potion, each moment immorialized in the impastionnd strokes of van gogh
97. Discover carious olien, playing the violin umidst the secret chumbers of on undergronad laboratory, artfully renderedl through the
abstract lens of pleasso
Behold brove girl in throes of playing the violin survounded by mysterious crossronis mysticol forest, envisioned through the
abstract lens of picaszo
Discover mischimoou fairy, pointing a masterpiece amidst the secret chambers of underground laborotory, artfully rendered with photorealittic precision
100. Within the realm of the frosty peok a snowy mountain, a ployful panda moticuloncly investgating mysterious crème scene, each moment immortallted through the abstract lous of picasso
Figure 10: Prompts used in AIGCBench.
20

---

Original
Ours
Original
Ours
Figure 11: Qualitative results text-to-image generation task using LDM-4 on ImageNet 256x256 benchmark.
Phoepe suyting
Oragiial (Xt 10 sponthyy
PopyCaree (XLA sponthep
Chiy (41. sprnsaap)
Prote
Pante 0
Froeer 14
Frone 27
Fiunse Ad
Figure 12: Qualitative results of text-to-video generation task using ModelScopeTTV MSR-VIT benchmark.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The motivations and contributions are well depicted and summarized the abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be pereeived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Iustification:
Limitations are discussed in Appendix A.1.
Guidelines:
The answer NA means that paper has no limitation while answer means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust results are violations these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
grounds for rejection,
Theory Assumptions and Proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
The theory assumptions and proofs are descrihed in Section 3 and Appendix A.2.
Guidelincs:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
22

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
Formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The selected models and benchmarks are clearly and fully presented in Scction 4.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice,
example, if the contribution the contribution is specific model empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the of large language model), releasing of model checkpoint, or other means that are appropriate to research performed.
While NeurIPS does not require relcasing code, the conference does require submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a way to access this model for reproducing the results a way to reproduce model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
new model (e.g.. large language model), then there should either be
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to deseribe the particular way they provide for reprodueibilityy In the case of closcd-soure source models, it may be that access to the model is limited some way (c.g., to registered users), but it should be possible for other rescarchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described supplemental material?
Answer: [Yes]
Justification: The code is available at Adaptives saaativeeiiffssonn
Guidelines:
The answer means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https:
gui guides/ des/ Codesubre ssi onPol PpPolioo Poyllycy) for more details.
While we encourage release of code and data, we understand that might not possible, so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source henchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /ntpss.pp//..////// ec/ for more details.
23

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from script and why.
At submission time, to preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommeneed, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details are carefully presented in Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report x² stats and p-value between the greedy searched paths and the proposed third-order estimated paths at different skipping targets.
Guidelines:
The answer means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful symmetric error bars would yield results that are out of range (e.g. negative error rates). error bars reported tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
show in tables figures
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: The information of the employed compute resources is elaborated in the implementation details of Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
24

---

The paper should provide the amount of compute required for each of the individual experimental well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, every respect, with NeurIPS Code of Ethics https: : /ttpr/neripppppppee. ca http/uulliccee
lines?
Answer: [Yes]
Justification: We make sure that the research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
authors should make sure preserve anonymity (e-g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Our proposed method currently has no apparent societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will he foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example. it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example hy requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
25

---

We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: have cited all papers that we used experiments.
Guidelines:
The answer means that the paper does not use existing assets.
authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset is used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g.. website), the copyright and terms of service of that source should he provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. popular datasets, papers rapersswi theode. com/ cat asets for some datasets. Their licensing guide can help determine the license a dataset.
curated licenses
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does release assets.
Guidelines:
The NA means that the paper does release
Researchers should communicate the details of the dataset/code/model part of their sub- missions structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
answer means that paper does not involve crowdsoureing nor research with human subjects.
Including this information in supplemental material is fine, but main contribution of paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Beard (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants. whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
26

---

Justification:
This paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
27


Reviews:
[{"review_id": "zx5jjS5AXZ", "reviewer": "Reviewer_XP4x", "summary": "To enhance the sampling speed in diffusion models, this paper introduces the AdaptiveDiffusion framework, which utilizes a skipping strategy. Specifically, this strategy is guided by the third-order latent difference, assessing the stability between timesteps throughout the denoising process. Experiments results on image and video diffusion models demonstrate the superiority of the proposed adaptively sampling framework.", "questions": "1. Can you derive the Theorem 1 when the step size is large? Since large step size will magnify the upper bound on the error, and fast sampling method always equals to large step size sampling.\n\n2. Since the proposed method aims to accelerate the sampling speed, can it reduce the NFEs?\n\n3. Is the proposed method effective on pure image generation?", "limitations": "Please see in Weaknesses and Questions. If all of my concerns are addressed, I will definitely improve my score.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The motivation is reasonable: to accelerate the sampling speed by reducing redundant time steps.\n\n2. The contribution is helpful in diffusion community.\n\n3. The figures are pretty and the presentation is readable.\n\n4. The proposed AdaptiveDiffusion is effective on multitask.", "weaknesses": "1. In my humble opinion, some improvements are marginal to me, especially on ImageNet 256×256.\n\n2. I am not sure the Theorem one is guaranteed even in larger sampling step size.\n\n3. Since many methods investigate reducing sampling steps by employing higher-order solutions, this approach lacks novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1: Analysis of Improvements.**\n\nWe describe the advantages of our method in two aspects.\n\n- **Novel Method Design:** Endorsed by three other reviewers, AdaptiveDiffusion is the pioneering framework that accelerates the diffusion process adaptively for diverse prompts. Unlike the SOTA method Deepcache, which caches features of multiple blocks uniformly across all stages and imposes a static caching rule, our approach introduces adaptive acceleration with theoretical underpinnings and memory efficiency via a single cache unit for predicted latent.\n- **Comprehensive Performance Improvements:** As noted by the other reviewers, AdaptiveDiffusion exhibits strong generalization across various models and tasks. Versus SOTA methods like DPM-solver and Deepcache, our method achieves a comparable or higher speedup ratio while maintaining superior image quality in both static image and video generation tasks.  As shown in Tab. 2, due to **the large sampling step number and limited prompt diversity**, the conditional image generation task on ImageNet 256x256 is relatively easy for acceleration, with both methods achieving roughly 6x speedup with negligible quality loss (~0.09 LPIPS). Notably, for specific categories, e.g., 607th and 854th, etc., AdaptiveDiffusion reduces the NFEs to approximately 35 from 250, yielding ~**7x** speedup. When the generation diversity and complexity increases, e.g., video generation, the superiority of AdpativeDiffusion is clearly demonstrated in Tab. 3.\n\n> **Q2: Theorem 1 with A Large Step Size.**\n\nWe would like to clarify that the step size is not mentioned as a condition for Theorem 1. If understood correctly, the large step size mentioned by the reviewer may refer to the large skip step of noise prediction. In this case, as the one- and two-step skipping schemes have been explored in Appendix A.2.1 and A.2.2 respectively, we further derive the error estimation of an arbitrary skipping scheme.\n\nTaking $i$-th step to perform $k$-step ($k\\geq 2$) skipping of noise prediction, we obtain the following update formulations.\n\n$\\quad x_i=f(i)\\ x_{i+1}-g(i)\\ \\epsilon_\\theta(x_{i+1},t_{i+1})$;\n\n$x_{i-1}=f(i-1)\\ x_i-g(i-1)\\ \\epsilon_\\theta(x_{i+1},t_{i+1})$;\n\n$x_{i-2}=f(i-2)\\ x_{i-1}-g(i-2)\\ \\epsilon_\\theta(x_{i+1},t_{i+1})$;\n\n$\\vdots$\n\n$x_{i-k}=f(i-k)\\ x_{i-k+1}-g(i-k)\\ \\epsilon_\\theta(x_{i+1},t_{i+1})$.\n\n$\\Rightarrow\\varepsilon_{i-k}=\\|x_{i-k}-x_{i-k}^{ori}\\|$\n\n$\\quad\\quad\\quad \\ =\\| f(i-k)(x_{i-k+1}-x_{i-k+1}^{ori})-g(i-k)[\\epsilon_\\theta(x_{i+1},t_{i+1})-\\epsilon_\\theta(x_{i-k+1},t_{i-k+1})]\\|$\n\n$\\quad\\quad\\quad \\ \\leqslant f(i-k)\\varepsilon_{i-k+1}+g(i-k)\\|\\epsilon_\\theta(x_{i+1},t_{i+1}) -\\epsilon_\\theta(x_{i-k+1},t_{i-k+1})\\|$\n\n$\\quad\\quad\\quad \\ \\leqslant\\sum_{m=1}^{k-1}{\\|h^{k-m+1}(i-m)\\cdot\\mathcal{O}(t_{i-m+1}-t_{i-m+2})\\|}+\\sum_{m=1}^{k-1}{\\|h^{k-m+1}(i-m)\\cdot\\mathcal{O}(x_{i-m+1}-x_{i-m+2})\\|}$\n\n$\\quad\\quad\\quad\\quad\\ +\\|g(i-k)\\cdot\\mathcal{O}(x_{i-k+1}-x_{i-k+2})\\|+\\|g(i-k)\\cdot\\mathcal{O} (t_{i-k+1}-t_{i-k+2})\\|$\n\n$\\quad\\quad\\quad \\ =\\sum_{m=1}^k{\\mathcal{O}(t_{i-m+1}-t_{i-m+2})+\\mathcal{O}(x_{i-m+1}-x_{i-m+2})}$.\n\nThe derivation utilizes the property that $\\|\\epsilon_\\theta(x_i,t_{i+1})-\\epsilon_\\theta(x_{i+1},t_{i+1})\\|$ and $\\|\\epsilon_\\theta(x_i,t_i)-\\epsilon_\\theta(x_i,t_{i+1})\\|$ are upper-bounded by $\\mathcal{O}(x_{i}-x_{i+1})$ and $\\mathcal{O}(t_i-t_{i+1})$ respectively according to the Lipschitz continuity. Here, $h^{k-m+1}(i-m)\\coloneqq g(i-m)\\prod\\nolimits_{j=1}^{k-m}{f(i-m-j)}$.\n\nIt is observed that the error of $k$-step skipping is upper-bounded by the accumulation of previous latent differences. Thus, if the skipping step of noise prediction is large, the upper bound of the error will naturally increase, as also empirically demonstrated by Fig. 5(b).\n\n> **Q3: Novelty of AdaptiveDiffusion.**\n\nWe first clarify our acceleration mechanism. Then, we will clarify our method's novelty from two aspects.\n\n**1. Mechanism:**\n\nGenerally, the diffusion process comprises two stages at each step: noise prediction and latent update. Given the preset number of inference steps, the main purpose of AdaptiveDiffusion is to **reduce the number of function evaluations (NFEs)** while keeping the latent update number (sampling steps) **unchanged**. By reducing NFEs, we can significantly accelerate the diffusion process.\n\n**2. Novelty:**\n\n- **Motivation-level Novelty:** Prior studies like EDM $^{[1]}$ and DPM-solver leveraged high-order approximations between sampling steps to enhance image quality. In contrast, AdaptiveDiffusion employs these approximations to adaptively reduce NFEs. That is, while earlier high-order methods aimed at refining generation quality given a fixed sampling step number, our work prioritizes adaptive efficiency without compromising image quality.\n- **Algorithm-level Novelty:** Our 3rd-order estimator is distinct in its adaptive acceleration tailored to various prompts. Unlike other high-order methods that flexibly choose solver orders for subsequent high-quality generation, our estimator is both empirically and theoretically constrained to 3rd-order approximations to decide whether to skip noise prediction, as shown in Sec. 3.3 and the global response. To our knowledge, this insight is the first in the field of diffusion model acceleration.\n\nBriefly, AdaptiveDiffusion targets a different motivation and is a novel approach to the acceleration community.\n\n> **Q4: Experiments on Pure Image Generation**\n\nFollowing Deepcache, we perform image generation using DDPMs on CIFAR10 and LSUN-Bedrooms and build our method upon 100-step DDIM for DDPM. As shown in the table below, our method still achieves a larger speedup ratio and higher image quality than Deepcache on both benchmarks.\n|Dataset|Method|FID|Speedup ratio|\n|-|-|-|-|\n|CIFAR10|Deepcache|10.17|2.07x|\n||Ours|**7.97**|**2.09x**|\n|LSUN|Deepcache|9.13|1.48x|\n||Ours|**7.96**|**2.35x**|\n\n***Reference:***\n\n[1] Elucidating the Design Space of Diffusion-Based\nGenerative Models. NeurIPS 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ImCBPbbCOk", "reviewer": "Reviewer_kqHE", "summary": "This paper proposes a strategy to speed up image and video diffusion generative models. The speed-up is achieved by skipping denoising steps. The authors suggest implementing an adaptive skipping schedule, where the decision of which steps to skip depends on the processed image or video. Specifically, the proposed algorithm calculates the norm of the third-order derivative in the latent space. It then skips denoising steps when the norm of the third-order derivative is below a predefined threshold while limiting the number of consecutive skips to a set maximum.\n\nThe authors evaluate the proposed skipping strategy through various experiments with image and video diffusion models, comparing the speed-up and quality of the generated content with DeepCache [1] and Adaptive DPM-Solver [2]. In most experiments, the proposed skipping strategy achieves higher speed-up and better image and video quality than competitors.\n\n[1] Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free. In Conference on Computer Vision and Pattern Recognition (CVPR), 2024\n[2] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.00927, 2022.", "questions": "It would be helpful if the authors could provide any mathematical justification or intuition (in addition to the empirical experiments in the paper) for their choice to employ third-order derivatives as the criterion for skipping denoising steps.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed skipping strategy leverages the generated content to speed up image and video diffusion generative models through an adaptive skipping scheme. In experiments with image and video generation, the authors demonstrate that this adaptive skipping scheme can achieve higher speed and better quality than competitors. In my opinion, the concept of using an adaptive skipping scheme to speed up diffusion generative models is potentially valuable and could be of interest to the research community.", "weaknesses": "The choice of employing the norm of the third-order derivative as a criterion for skipping denoising steps was made empirically. Theorem 1 (Equation 3) explains why it makes sense to consider the first-order derivative as a criterion for skipping. However, the authors empirically demonstrate that the first-order and second-order derivatives barely correlate with an optimal skipping schedule found by a greedy search. There is a lack of mathematical justification for choosing the third-order derivative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">***Q1: Theoretical Analysis of the Relationship between the Third-order Estimator and the Skipping Strategy.***\n\nTo explore the theoretical relationship between the third-order estimator and the skipping strategy, we need to formulate the difference between the neighboring noise predictions. According to Eq.(1), we can get the following first-order differential equations regarding the latent $x$.\n\n$\\quad \\varDelta x_i=x_i-x_{i+1}=[1-f(i)]x_{i+1}-g(i)\\cdot\\epsilon_{\\theta}(x_{i+1},t_{i+1})$;\n\n$\\varDelta x_{i-1}= x_{i-1}-x_i=[1-f(i-1)]x_i-g(i-1)\\cdot\\epsilon_{\\theta}(x_i,t_i)$.\n\nNow, let $u(i)\\coloneqq 1-f(i-1)$, and we further derive the second-order differential equations based on the above equations.\n\n$\\varDelta x_{i-1}-\\varDelta x_i = u( i ) x_i-u( i+1 ) x_{i+1}+g( i ) \\cdot \\epsilon_{\\theta}( x_{i+1}, t_{i+1} ) -g( i-1 ) \\cdot \\epsilon_{\\theta}( x_i, t_i )$\n\n$\\quad\\quad\\quad\\quad\\quad\\ \\ \\ =u( i ) ( x_i-x_{i-1} ) +u( i ) x_{i-1}-u( i+1 ) ( x_{i+1}-x_i ) -u( i+1 ) x_i+g( i ) \\cdot \\epsilon_{\\theta}( x_{i+1}, t_{i+1} ) -g( i-1 ) \\cdot \\epsilon_{\\theta}( x_i, t_i )$\n\n$\\quad\\quad\\quad\\quad\\quad\\ \\ \\ =u( i ) \\varDelta x_{i-1}-u( i+1 ) \\varDelta x_i+\\varDelta [ u( i ) x_{i-1} ] +g( i ) \\cdot \\epsilon_{\\theta}( x_{i+1}, t_{i+1} ) -g( i-1 ) \\cdot \\epsilon_{\\theta}( x_i, t_i )$\n\n$\\quad\\quad\\quad\\quad\\quad\\ \\ \\ =u( i ) \\varDelta x_{i-1}-u( i+1 ) \\varDelta x_i+\\varDelta [ u( i ) x_{i-1} ] +g( i ) [ \\epsilon_{\\theta}( x_{i+1}, t_{i+1} ) -\\epsilon_{\\theta}( x_i, t_i ) ] +[ g( i ) -g( i-1 ) ] \\epsilon_{\\theta}( x_i, t_i )$\n\n$\\quad\\quad\\quad\\quad\\quad\\ \\ \\ =u( i ) \\varDelta x_{i-1}-u( i+1 ) \\varDelta x_i+\\varDelta [ u( i ) x_{i-1} ] -g( i ) \\varDelta \\epsilon_{\\theta}^{i}-\\varDelta g( i ) \\cdot \\epsilon_{\\theta}( x_i, t_i )$.\n\nAfter simplification of the above equation, we can get the following formulation:\n\n$f( i-1 ) \\varDelta x_{i-1}-f( i ) \\varDelta x_i=\\varDelta [ u( i ) x_{i-1} ] -g( i ) \\varDelta \\epsilon_{\\theta}^{i}-\\varDelta g( i ) \\cdot \\epsilon_{\\theta}( x_i, t_i )$.\n\nFrom the above equation, we can observe that the difference between noise predictions $\\varDelta \\epsilon_{\\theta}^{i}$ is related to the first- and second-order derivatives of $x_i$, as well as the noise prediction $\\epsilon_{\\theta}( x_i, t_i )$. Therefore, it would be difficult to estimate the difference without $\\epsilon_{\\theta}( x_i, t_i )$. Now we consider the third-order differential equation. From the above equation, we further obtain the following formulation.\n\n\n$f( i ) \\varDelta x_i-f( i+1 ) \\varDelta x_{i+1}=\\varDelta [ u( i+1 ) x_i ] -g( i+1 ) \\varDelta \\epsilon_{\\theta}^{i+1}-\\varDelta g( i+1 ) \\cdot \\epsilon_{\\theta}( x_{i+1}, t_{i+1} )$.\n\n$\\Rightarrow \\varDelta [ f( i-1 ) \\varDelta x_{i-1} ] -\\varDelta [ f( i ) \\varDelta x_i ] =\\varDelta ^{( 2 )}[ u( i ) x_{i-1} ] -\\varDelta [ g( i ) \\varDelta \\epsilon_{\\theta}^{i} ] -\\varDelta [ \\varDelta g( i ) \\cdot \\epsilon_{\\theta}( x_i, t_i ) ]$.\n\n$\\Rightarrow \\varDelta [ \\varDelta g( i ) \\cdot \\epsilon_{\\theta}( x_i, t_i ) ] =-\\varDelta ^{( 2 )}[ f( i-1 ) \\varDelta x_{i-1} ] +\\varDelta ^{( 2 )}[ u( i ) x_{i-1} ] -\\varDelta [ g( i ) \\varDelta \\epsilon_{\\theta}^{i} ]$.\n\nFrom the above equation, it can be observed that the difference of the neighboring noise predictions is explicitly related to the third- and second-order derivatives of $x_i$, as well as the second-order derivative of $\\epsilon_\\theta^{i}$. Since $\\lim_{i\\rightarrow 0} f( i ) =1,\\lim_{i\\rightarrow 0} u( i ) =0,\\lim_{i\\rightarrow 0} g( i ) =0$, we can finally get the conclusion that $\\varDelta \\epsilon_{\\theta}^{i} \\| \\_{i \\rightarrow 0} \\approx \\mathcal{O} ( \\varDelta ^{( 3 )}x_{i-1} )$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "asPelMD03P", "reviewer": "Reviewer_rWvo", "summary": "## Summary\n* This paper propose a greedy approach that accelerate the Probability Flow ODE solvers for txt to image diffusion models. Empirical results on SD 1.5 and SD XL with mulitple solvers (DDIM, DPM, Euler) demonstrate the advantage of their approach over previous acceleration techniques.", "questions": "## Questions\n* For now the results are reported on 3 different ODE solvers. Can the proposed approach be applied to SDE solvers too? As sometimes the SDE has some advantage over ODE in terms of sample quality (See Closing the ODE-SDE gap in score-based diffusion models through the Fokker-Planck equation).", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "## Strength\n* The idea of prompt adaptative acceleration is interesting and promising. As previous solvers designed for more general PF-ODE are not adaptative to different prompts. It is quite natural for a prompt adaptative approach to obtain better result.\n* The empirical advantage over previous accelerations is obvious, especially in terms of image quality such as PSNR and LPIPS.", "weaknesses": "## Weakness \n* A new trend in image generation diffusion model is the adoptation of Flow matching optimal transport (FM-OT) / Rectified flow (RF) (See Learning to Generate and Transfer Data with Rectified Flow). Those lines of works adopt a forward SDE that is neither VP nor VE. Their special SDE has a constant velocity in the corresponding PF ODE. Rectified flow can even achieve single step sampling with this PF ODE. Their PF-ODE has a path that is very close to straight line and be solved with less steps compared with VP / VE SDE. The latest Stable Diffusion 3 already adopt this type of diffusion. Those efforts in diffusion community also speed up sampling, not by solvers but by different models. Those line of works should be discussed as they have the same goal with this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">***Q1: Discussion of Single-step Sampling Works***\n\nThank the reviewer for the valuable suggestion. We will provide a detailed discussion in the revised manuscript. Here is a brief discussion of single-step sampling works.\n\nIn addition to the acceleration paradigms mentioned in Sec 2.2., a recent training-based acceleration paradigm is increasingly receiving attentions from the community. Different from previous acceleration works reducing sampling steps or model size during inference time, this paradigm adopts the idea of flow-matching optimal transport to directly achieve single-step sampling during training, whose trajectory is approximately a straight line $^{[1,2]}$. Compared with this new trend, our method highlights the training-free acceleration of multi-step sampling models, with no need to train a new efficient diffusion model.\n\n>***Q2: Effectiveness on SDE solvers***\n\nCompared with the ODE solver, the SDE solver includes an additional noise item for the latent update, which is unpredictable by previous randomly-generated noises. When the magnitude of random noise is not ignorable, the third-order derivative of the neighboring latents cannot accurately evaluate the difference between the neighboring noise predictions. Therefore, to apply our method to SDE solvers, we should design an additional indicator that decides whether the randomly-generated noise is minor enough or stably changed to trigger the third-order estimator. In this case, we design an additional third-order estimator for the scaled randomly-generated noise. When the third-order derivatives of both the latent and the scaled randomly-generated noises are under the respective thresholds, the noise prediction can be skipped by reusing the cached model output.\n\nTo validate the effectiveness of our improved method, we conduct experiments for SDXL with the SDE-DPM solver on COCO2017. The results are shown in the following table. Compared with Deepcache, our method can achieve higher image quality with a comparable speedup ratio, indicating the effectiveness of AdaptiveDiffusion on SDE solvers.\n\n|Method|PSNR $\\uparrow$|LPIPS $\\downarrow$|FID $\\downarrow$|Latency (s)|Speedup Ratio|\n|-|-|-|-|-|-|\n|Deepcache|16.44|0.346|8.15|**9.2**|**1.63x**|\n|Ours|**18.80**|**0.232**|**6.03**|9.8|1.53x|\n\n***Reference:***\n\n[1] Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. ICLR 2023.\n\n[2] Flow matching for generative modeling. ICLR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8HxxM69Kz2", "reviewer": "Reviewer_hdh3", "summary": "The paper presents a novel approach titled \"AdaptiveDiffusion\" for accelerating diffusion models used in high-quality image and video synthesis. The core issue addressed is the high computational cost and latency associated with existing denoising techniques in diffusion models, which are typically based on step-by-step noise predictions.\n\n**Contributions of the Paper:**\n1. **Adaptive Diffusion Process**: The paper introduces AdaptiveDiffusion, which adaptively reduces the number of noise prediction steps during the denoising process. This is achieved by skipping steps where the potential redundancy is high, guided by the third-order latent difference that indicates stability between timesteps.\n\n2. **Plug-and-Play Criterion**: A new criterion is proposed to decide whether to infer new noise predictions or reuse previous results based on the third-order difference distribution. This allows for an adaptive acceleration paradigm that is prompt-dependent.\n\n3. **Extensive Experiments**: The method's effectiveness is demonstrated through extensive experiments on both image and video diffusion models. The results show significant speedups of 2 to 5 times on average in the denoising process without quality degradation.\n\n4. **Error Analysis**: The paper provides a theoretical analysis of the upper bound of the error induced by the step-skipping strategy, ensuring that the quality of the final output is maintained.\n\n5. **Adaptive Acceleration**: The approach is designed to be adaptive to different input prompts, offering a practical solution to the high computational costs of sequential denoising techniques.\n\n6. **Generalization Capability**: AdaptiveDiffusion shows a strong generalization capability, being able to adapt to different models and tasks, including text-to-image, image-to-video, and text-to-video generation.\n\nIn summary, the paper offers a substantial advancement in efficient diffusion model acceleration, with the potential to enable real-time and interactive applications of diffusion models.", "questions": "1. **Assumption Validity**: Could the authors elaborate on the conditions under which the Lipschitz continuity assumption for the noise prediction model holds? How do they ensure this assumption is valid across different models and datasets?\n\n2. **Generalization Across Models**: The paper demonstrates results on a few models. What steps have been taken to ensure that AdaptiveDiffusion can generalize across a wider variety of diffusion models, especially those that may not conform to the same architectural patterns?\n\n3. **Hyperparameter Selection**: The paper mentions the importance of hyperparameters δ and Cmax. Can the authors provide more detailed guidelines or methods for selecting these hyperparameters in different contexts or suggest any automated tuning processes?\n\n4. **Memory Usage Discussion**: Given that diffusion models can be memory-intensive, could the authors discuss the memory usage implications of AdaptiveDiffusion, especially when scaling up to larger models or datasets?\n\n5. **Ethical Considerations**: Although the paper focuses on a technical advancement, could the authors comment on any potential ethical implications of the work, particularly related to the generative capabilities of the models?\n\n6. **Reproducibility Assurance**: To ensure the reproducibility of the results, will the authors commit to making their code and datasets publicly available, and if so, when?\n\n7. **Algorithm Visualization**: For better understanding, especially for readers who may be less familiar with the proposed methods, can the authors provide pseudocode or flowcharts illustrating the step-skipping strategy?\n\n8. **Long-Term Viability**: How does the authors' method accommodate or adapt to potential future changes in the architecture or training of diffusion models?\n\n9. **Limitation Discussion**: The paper outlines future work but could benefit from a more explicit discussion of current limitations. Are there specific scenarios or model types where AdaptiveDiffusion might underperform?\n\n10. **Statistical Significance**: The paper reports χ2 stats and p-values for the correlation between estimated and optimal paths. Could the authors provide more details on the statistical tests used and the rationale behind choosing these tests?\n\n11. **Real-World Application**: While the method shows promise in controlled experiments, are there any real-world scenarios or use cases where AdaptiveDiffusion has been tested or is planned to be tested?\n\n12. **Societal Impact**: The paper could be strengthened by a brief discussion on the potential societal impacts, both positive and negative, of the technology. This includes considerations of how the method might be used or misused.\n\n13. **Comparison with State-of-the-Art**: How does AdaptiveDiffusion compare with the state-of-the-art in terms of computational efficiency and quality of results? Are there any specific advantages or disadvantages in particular scenarios?\n\n14. **Documentation and API**: For practical adoption, what level of documentation and API support is available or planned for AdaptiveDiffusion to facilitate its integration into existing systems?", "limitations": "Based on the information provided and the typical guidelines of the NeurIPS Paper Checklist, it appears that the authors have made an effort to address limitations and societal impacts. However,  I offer general advice on how authors can improve their discussion of these topics:\n\n1. **Clear Acknowledgment**: Authors should explicitly acknowledge the limitations of their work in the main text of the paper. This includes potential constraints on the generalizability of their findings, assumptions made, and any conditions under which the method may not perform as expected.\n\n2. **Depth of Discussion**: While acknowledging limitations, authors should provide a thorough explanation of how these limitations might affect the results and the applicability of their method. This could include a discussion of how the method behaves under different conditions or with different types of data.\n\n3. **Societal Impact Analysis**: Authors should consider the broader societal impacts of their work, including both positive and negative outcomes. This discussion should be grounded in the context of the work and consider potential misuse, ethical concerns, privacy issues, and fairness.\n\n4. **Mitigation Strategies**: If there are potential negative societal impacts, authors should discuss possible mitigation strategies. This could involve suggesting guidelines for the responsible use of the technology, potential regulatory frameworks, or technical safeguards.\n\n5. **Ethical Considerations**: The paper should include a section on ethical considerations, especially if the work involves generative models that could be used to create misleading or harmful content.\n\n6. **Transparency**: Authors should be transparent about any potential conflicts of interest, funding sources, or affiliations that might influence the research or its interpretation.\n\n7. **Openness to Feedback**: Authors should demonstrate a willingness to engage with the community for feedback on the societal impacts of their work and be open to adjusting their approach based on this feedback.\n\n8. **Long-Term Vision**: While discussing limitations, authors could also provide a long-term vision for how they anticipate overcoming these limitations in future work.", "rating": 3, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "### Strengths Assessment of the Paper\n\n#### Originality\nThe paper demonstrates a high degree of originality through the introduction of the AdaptiveDiffusion method, which offers a novel perspective on accelerating diffusion models. The approach创造性地 addresses the computational inefficiency inherent in traditional denoising techniques by adaptively reducing noise prediction steps. This innovation is not just a technical tweak but a strategic rethinking of the denoising process itself. The use of the third-order latent difference as a criterion for deciding when to skip steps is an ingenious way to balance efficiency and quality, which has not been explored in prior works.\n\n#### Quality\nThe quality of the paper is reflected in its rigorous theoretical foundation and comprehensive empirical validation. The authors have provided a detailed error analysis to support their method's robustness, ensuring that the acceleration does not compromise the output quality. The experiments are thorough, covering a range of models and tasks, which substantiates the method's effectiveness and generalizability. The paper also discusses the relationship between different orders of latent differences and the optimal skipping path, which adds depth to the understanding of the proposed technique.\n\n#### Clarity\nThe paper is well-structured, with a clear progression from the introduction of the problem to the explanation of the proposed solution, followed by a detailed methodology and extensive experimental results. The figures and tables are used effectively to illustrate the method and results, enhancing the readability and comprehension of the paper. The theoretical proofs and algorithm descriptions are presented in a manner that is accessible to readers with a background in the field.\n\n#### Significance\nThe significance of this paper lies in its potential to transform the applicability of diffusion models. By significantly reducing the computational cost and latency, AdaptiveDiffusion opens up new possibilities for real-time and interactive applications of diffusion models, which are currently limited by their resource-intensive nature. The ability to tailor the denoising process to different prompts is also significant, as it allows for more flexible and responsive generative models that can cater to diverse content creation needs.\n\nIn summary, the paper is a substantial contribution to the field of generative modeling, offering a creative, high-quality, and clearly articulated solution to a pressing problem. Its significance extends beyond technical advancement, promising to enable new applications and use cases for diffusion models.", "weaknesses": "While the paper presents a compelling approach to accelerating diffusion models, there are areas where it could be further strengthened:\n\n### Theoretical Depth\n- **Assumption Limitations**: The paper relies on certain assumptions for its theoretical analysis, such as the Lipschitz continuity of the noise prediction model. It would be beneficial to discuss how violations of these assumptions might impact the method's effectiveness and under what conditions these assumptions hold true.\n\n### Experimental Scope\n- **Diversity of Models**: Although the method is tested on various tasks, the paper could benefit from testing on a broader range of diffusion models to further establish the generalizability of AdaptiveDiffusion.\n- **Real-World Applications**: Demonstrating the method's effectiveness in real-world applications or use cases would provide additional context and significance to the work.\n\n### Hyperparameter Sensitivity\n- **Threshold δ and Cmax**: The paper discusses the impact of these hyperparameters on performance but could provide more guidance on how to select these values in practice, especially given their critical role in balancing speed and quality.\n\n### Computational Complexity\n- **Memory Usage**: While the method aims to reduce computational cost, it would be insightful to have a discussion on memory usage, especially since diffusion models can be memory-intensive.\n\n### Long-Term Viability\n- **Adaptability to Model Updates**: The paper could address how well AdaptiveDiffusion might adapt to future updates in diffusion model architectures or training regimes.\n\n### Societal Impact Consideration\n- **Ethical Considerations**: Although the paper does not explicitly discuss societal impacts, it would be beneficial to include a brief discussion on potential ethical considerations, especially given the generative capabilities of the models involved.\n\n### Reproducibility\n- **Code and Data Availability**: Ensuring that the code and data used for experiments are publicly available would greatly enhance the reproducibility of the results.\n\n### Documentation\n- **Algorithm Pseudocode**: Providing pseudocode or flowcharts for the algorithms could help readers better understand the step-skipping strategy and its integration into the overall process.\n\n### Future Work\n- **Extensions and Limitations**: While the paper outlines future directions, a more detailed discussion on the limitations and potential extensions of the current work would be valuable.\n\nBy addressing these points, the paper could provide a more comprehensive understanding of AdaptiveDiffusion's capabilities and limitations, setting the stage for further research and development in this area.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">***Q1: Method Explanation***\n\n- **Assumption Validity:** We follow the assumptions proposed in DPM-solver $^{[1]}$, which are commonly adopted for the high-order approximation in ODE solvers.\n\n- **Algorithm Pseudocode:** We have provided the pseudocodes of AdaptiveDiffusion and the greed search algorithms in Appendix A.2.3.\n\n>***Q2: Experimental Discussion***\n\n- **Diversity of Models:** We further explore the application of our method on unconditional image generation tasks. The results are shown in the following table. Specifically, following Deepcache, we perform unconditional image generation on CIFAR10 and LSUN-Bedrooms. Shown in the table below, our method still achieves a larger speedup ratio and higher image quality than Deepcache on both benchmarks.\n\n|Dataset|Method|FID|Speedup ratio|\n|-|-|-|-|\n|CIFAR10|Deepcache|10.17|2.07x|\n||Ours|**7.97**|**2.09x**|\n|LSUN|Deepcache|9.13|1.48x|\n||Ours|**7.96**|**2.35x**|\n\n- **Hyperparameter Sensitivity:** We have provided the sensitivity analysis of hyperparameters in Table 4.\n\n- **Memory Usage:** We have listed the memory usage of different models in Table 1, 2, and 3.\n\n>***Q3: Limitation and Future Work***\n\nCurrently, our work mainly focuses on the acceleration of diffusion with ODE solvers. In future work, we should explore the effectiveness of our method on more kinds of solvers and models. For example, the acceleration of SDE solvers should consider the interference of randomly-generated noise in the high-order estimation of the skipping strategy. For consistency models, the acceleration should consider the impact of distillation on the trajectory of image generation, which would change the continuity of noise prediction.\n\n***Reference:***\n\n[1] DPM-Solver: A Fast ODE Solver for Diffusion\nProbabilistic Model Sampling in Around 10 Steps. NeurIPS 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear AC and reviewers,

We are deeply appreciative of the reviewers for their valuable time and thoughtful comments. Their feedback has reinforced our confidence in the paper's **clear presentation and organization** (Reviewer XP4x, kqHE, rWvo, hdh3), **the innovative approach** of AdaptiveDiffusion in addressing computational inefficiencies (Reviewer kqHE, rWvo, hdh3), and **the comprehensive experimental validation** across various tasks and models (Reviewer XP4x, kqHE, rWvo, hdh3) with **notable improvements** in speed and quality (Reviewer kqHE, hdh3).

We have diligently addressed each reviewer's critical feedback. Our goal is to resolve all issues and improve our work through this collaborative process. We will integrate these valuable comments into our revision, confident they will significantly elevate our work's quality and benefit the field.

Here is a summary of what we have done in the rebuttal phase.

- We conduct more experiments to cover the concerns of reviewers.
    - More experiments on pure image generation using DDPMs on CIFAT10 and LSUN.
    - Experiments using SDE solvers.

- We provide the derivation of theorem 1 with a large step size.

- We provide the theoretical analysis of the relationship between the third-order estimator and the skipping strategy.

- We provide a further discussion on the novelty of our work.

- We provide a discussion on single-step sampling works.

Below is the global response that might be commonly mentioned in the responses to several reviewers' comments.

> ***Theoretical Analysis of the Relationship between the Third-order Estimator and the Skipping Strategy.***

We supplement the theoretical relationship between the third-order estimator and the skipping strategy. Specifically, the difference between the neighboring noise predictions is formulated. According to Eq.(1), we can get the following first-order differential equations regarding the latent $x$.

$\quad \varDelta x_i=x_i-x_{i+1}=[1-f(i)]x_{i+1}-g(i)\cdot\epsilon_{\theta}(x_{i+1},t_{i+1})$;

$\varDelta x_{i-1}= x_{i-1}-x_i=[1-f(i-1)]x_i-g(i-1)\cdot\epsilon_{\theta}(x_i,t_i)$.

Now, let $u(i)\coloneqq 1-f(i-1)$, and we further derive the second-order differential equations based on the above equations.

$\varDelta x_{i-1}-\varDelta x_i = u( i ) x_i-u( i+1 ) x_{i+1}+g( i ) \cdot \epsilon_{\theta}( x_{i+1}, t_{i+1} ) -g( i-1 ) \cdot \epsilon_{\theta}( x_i, t_i )$

$\quad\quad\quad\quad\quad\ \ \ =u( i ) ( x_i-x_{i-1} ) +u( i ) x_{i-1}-u( i+1 ) ( x_{i+1}-x_i ) -u( i+1 ) x_i+g( i ) \cdot \epsilon_{\theta}( x_{i+1}, t_{i+1} ) -g( i-1 ) \cdot \epsilon_{\theta}( x_i, t_i )$

$\quad\quad\quad\quad\quad\ \ \ =u( i ) \varDelta x_{i-1}-u( i+1 ) \varDelta x_i+\varDelta [ u( i ) x_{i-1} ] +g( i ) \cdot \epsilon_{\theta}( x_{i+1}, t_{i+1} ) -g( i-1 ) \cdot \epsilon_{\theta}( x_i, t_i )$

$\quad\quad\quad\quad\quad\ \ \ =u( i ) \varDelta x_{i-1}-u( i+1 ) \varDelta x_i+\varDelta [ u( i ) x_{i-1} ] +g( i ) [ \epsilon_{\theta}( x_{i+1}, t_{i+1} ) -\epsilon_{\theta}( x_i, t_i ) ] +[ g( i ) -g( i-1 ) ] \epsilon_{\theta}( x_i, t_i )$

$\quad\quad\quad\quad\quad\ \ \ =u( i ) \varDelta x_{i-1}-u( i+1 ) \varDelta x_i+\varDelta [ u( i ) x_{i-1} ] -g( i ) \varDelta \epsilon_{\theta}^{i}-\varDelta g( i ) \cdot \epsilon_{\theta}( x_i, t_i )$.

After simplification of the above equation, we can get the following formulation:

$f( i-1 ) \varDelta x_{i-1}-f( i ) \varDelta x_i=\varDelta [ u( i ) x_{i-1} ] -g( i ) \varDelta \epsilon_{\theta}^{i}-\varDelta g( i ) \cdot \epsilon_{\theta}( x_i, t_i )$.

From the above equation, we can observe that the difference between noise predictions $\varDelta \epsilon_{\theta}^{i}$ is related to the first- and second-order derivatives of $x_i$, as well as the noise prediction $\epsilon_{\theta}( x_i, t_i )$. Therefore, it would be difficult to estimate the difference without $\epsilon_{\theta}( x_i, t_i )$. Now we consider the third-order differential equation. From the above equation, we further obtain the following formulation.

$f( i ) \varDelta x_i-f( i+1 ) \varDelta x_{i+1}=\varDelta [ u( i+1 ) x_i ] -g( i+1 ) \varDelta \epsilon_{\theta}^{i+1}-\varDelta g( i+1 ) \cdot \epsilon_{\theta}( x_{i+1}, t_{i+1} )$.

$\Rightarrow \varDelta [ f( i-1 ) \varDelta x_{i-1} ] -\varDelta [ f( i ) \varDelta x_i ] =\varDelta ^{( 2 )}[ u( i ) x_{i-1} ] -\varDelta [ g( i ) \varDelta \epsilon_{\theta}^{i} ] -\varDelta [ \varDelta g( i ) \cdot \epsilon_{\theta}( x_i, t_i ) ]$.

$\Rightarrow \varDelta [ \varDelta g( i ) \cdot \epsilon_{\theta}( x_i, t_i ) ] =-\varDelta ^{( 2 )}[ f( i-1 ) \varDelta x_{i-1} ] +\varDelta ^{( 2 )}[ u( i ) x_{i-1} ] -\varDelta [ g( i ) \varDelta \epsilon_{\theta}^{i} ]$.

From the above equation, it can be observed that the difference of the neighboring noise predictions is explicitly related to the third- and second-order derivatives of $x_i$, as well as the second-order derivative of $\epsilon_\theta^{i}$. Since $\lim_{i\rightarrow 0} f( i ) =1,\lim_{i\rightarrow 0} u( i ) =0,\lim_{i\rightarrow 0} g( i ) =0$, we can finally get the conclusion that $\varDelta \epsilon_{\theta}^{i}\| \_{i\rightarrow 0}\approx \mathcal{O} ( \varDelta ^{( 3 )}x_{i-1} )$.

Best Regards,

Authors of Paper230

Meta-review: There are four reviews for this submission. Both Reviewer kqHE and Reviewer rWvo have provided positive scores. While Reviewer XP4x initially rated the submission as a borderline reject, he/she acknowledged during the discussion that most of the concerns were addressed and agreed to raise the score. The comments from Reviewer hdh3 appear to be generated by LLM and were less convincing, which thus were excluded from the final decision. The area chair sees no ground to overturn the consensus and recommends an accept. The decision has been discussed with SAC.
Decision: Accept (poster)