SADA: Stability-guided Adaptive Diffusion Acceleration
Authors: Ting Jiang
Yixiao Wang
Hancheng Ye
Zishan Shao
Jingwei Sun
Jingyang Zhang
Zekai Chen
Jianyi Zhang
Yiran Chen
Hai Li
Venue: ICML 2025 poster (2025)
Keywords: Diffusion Model
Efficient Algorithm
Training-Free Acceleration
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: Training-free acceleration of generative models by adpatively exploit sparsity with principled approximation.

Abstract: Diffusion models have achieved remarkable success in generative tasks but suffer from high computational costs due to their iterative sampling process and quadratic‐attention costs. 
Existing training-free acceleration strategies that reduce per-step computation cost, while effectively reducing sampling time, demonstrate low faithfulness compared to the original baseline. 
We hypothesize that this fidelity gap arises because (a) different prompts correspond to varying denoising trajectory, and (b) such methods do not consider the underlying ODE formulation and its numerical solution. 
In this paper, we propose **Stability-guided Adaptive Diffusion Acceleration (SADA)**, a novel paradigm that unifies step-wise and token-wise sparsity decisions via a single stability criterion to accelerate sampling of ODE-based generative models (Diffusion and Flow-matching). 
For (a), SADA adaptively allocates sparsity based on the sampling trajectory. 
For (b), SADA introduces principled approximation schemes that leverage the precise gradient information from the numerical ODE solver.
Comprehensive evaluations on SD‐2, SDXL, and Flux using both EDM and DPM++ solvers reveal consistent $\ge 1.8\times$ speedups with minimal fidelity degradation (LPIPS $\leq 0.10$ and FID $\leq 4.5$) compared to unmodified baselines, significantly outperforming prior methods.
Moreover, SADA adapts seamlessly to other pipelines and modalities: It accelerates ControlNet without any modifications and speeds up MusicLDM by \(1.8\times\) with \(\sim 0.01\) spectrogram LPIPS.
Our code is available at: [https://github.com/Ting-Justin-Jiang/sada-icml](https://github.com/Ting-Justin-Jiang/sada-icml).

SADA: Saability-guider Adaptive Diffusion Acceleration
Ting Jiang"
Yixiao Wang"
Hancheng Ye"
Zishan Shao 12
Jingwei Sun Hai Li
Jingyang Zhang' Zekai Chen'
Jianyi Zhang'
Yiran Chen
Abstract
1. Introduction
Diffusion models have achieved remarkable suc- cess in generative tasks but suffer from high com- putational costs due to their iterative sampling process and quadratic-attentie training-free acceleration strategies that reduce per-step computation cost, while effectively re- ducing sampling time, demonstrate low faithful- ness compared to the original baseline. We hy- pothesize that this fidelity gap arises because (a) different prompts correspond varying de- noising trajectory, and (b) such methods do not consider the underlying ODE formulation and its numerical solution. In this paper, we pro- pose Stability-guider Adaptive Diffusion Ac- celeration (SADA), a novel paradigm that uni- fies step-wise and token-wing wise sparsity decisions via single stability criterion accelerate sam- pling of ODE-based generative models (Diffusion and Flow- Nlw-maathingg... For (a), SADA adaptively allocates sparsity based on the sampling trajec- tory. For (b), SADA introduces principled approx- imation schemes that leverage the precise gradi- ent information from the numerical ODE solver. Comprehensive evaluations on SD-2, SDXL, and Flux using both EDM and DPM++ solvers re- veal consistent > 1.8x speedups with minimal fi- delity degradation (LPIPS < 0.10 and FID < 4.5) compared to unmodified baselines, significantly outperforming prior methods. Moreover, SADA adapts seamlessly to other pipelines and modali- ties: It accelerates ControlNet without any modi- fications and speeds up MusicLDM by 1.8x with 0.01 spectrogram LPIPS. Our code is avail- able at: httpp::/githubbcco. com.h//...ooooooccoee..eeeeeeeea Ting- Jww.uutttt---nn/salee
Recent advancements in diffusion models have set new benchmarks across various tasks, including image, video, text, and audio generation (Sohl-Dickstein et al.) 2013 Song et al. a). significantly enhancing productivity and creativ- ity. However, the deployment of these models at high res- olutions faces two fundamental efficiency bottlenecks: (i) the iterative nature of the denoising process and (ii) the quadratic complexity of attention mechanisms. To address these challenges, existing training-free acceleration meth- ods have primarily focused two corresponding fronts: (i) reducing the number of inference steps (Song et al. Lu ct al. 2022b Karras et al.
costs. Existing
2022). (ii) reducing the compu-
tational cost per step (Bolya & Hoffman 2023 Ma et al.
2024b Zhao et al. 2024 Wang et al. 2024 Zou et al. 2024
et a.
2024).
Sampling with generative models can be cast as transporting between two distributions through a reverse ordinary differ-
ential equation (ODE) (Song & Ermon 2019 Song et al.,
b) Lipman et al. 2023 Liu et al. 2022). Numerical ODE
solvers (Karras et al. 2022 Lu et al. 2022a b) that signifi-
cantly reduce the number of inference steps while preserving sample fidelity have become a fundamental component for practical workflows. Orthogonal these advanced sched- ulers, recent works that reduce per-step computational cost in category (ii) exploit the sparsity empirically observed in pretrained architectures (Rombach et al. 2022 Chen et al., 2024b Podell et al. Fisse et al.
2024) during the sampling
procedure, either on a token-wisss wise
(Bolya & Hoffman 2023
Kim et al. 2024 Zou et al.
2024) or step-wise (Ma et al.
2024b Zhao
2024
Ye al. 2024) granularity.
While effectively reducing sampling latency, these architec- tural strategies in category (ii) demonstrate low faithfulness compared to original samples, as measured in LPIPS (Zhang et al. 2018) and FID (Heusel et al.) 2017). We hypothesize
icml
that this fidelity gap arises because: (a) Fixed (Ma et al.,
Equal contribution 1 'Department of Electrical and Computer Engi- ncering, Duke University, Durham, U.S.A Department of Statisti- cal Science, Duke University, Durham, U.S.A. Correspondence to: Ting Jiang <justin,jiang&& @duke.ddu>.
2024b) or pre-searched (Yuan et al. 2024) sparsity patterns
cannot adapt to the variability of each prompt's denoising trajectory, and (b) Such methods do not explicitly leverage the underlying ODE formulation of the denoising process, nor interplay with the specific ODE-solver used.
Proceedings of the 12nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Motivated by these observations, we introduce Stability-

---

SADA: Stability Stability-uuided guided Adaptive Diffusion Acceleration
SADA ds or SOTA
Faster
2.02 x
SADA ds or SOTA
Faster
1.86 X
Figure 1. Accelerating (Flux, SDXL, SD-2) by (2.02x, 1.86x, 1.80x with Stability- Adaptive Diffusion Acceleration with 50 (Diffusion, Flow-matching) inference steps.
guided Adaptive Diffusion Acceleration (SADA), training- free framework that dynamically exploits both step-wise and token-wise sparsity via a unified stability criterion. SADA addresses (a) by adaptively allocating computation along the denoising trajectory (See Section B.3). and (b) by employing approximation schemes that leverage precise trajectory gra- dient information from the chosen ODE solver (See Section
2024ch and downstream tasks (Zhang et al. 2023 without additional training. all tested scenarios,
(Chen et al.
SADA achieves substantially better faithfulness (S 0.100
LPIPS) than existing strategies(Ye Ye al., 2024) Ma al.)
2024b Liu al. 2025a), while consistently delivering
2 1.8x speedup. These results establish SADA prac- tical plug-in for high-throughput, high-fidelity generative sampling.
3.4
Specifically, leveraging the second-order difference of the precise gradient y/ dæ. according to its definition (Song al. Lipman et al., 2023). SADA makes better de- cisions on sparsity allocation and principled approxima- tion. For (a), we propose the stability criterion (Crite- rion 3.4p as the second-order difference of yr, which mea- sures the local dynamics of the denoising trajectory. Imple- mented in a plug-and-play fashion, SADA dynamically iden- tifies the sparsity mode (token-wise, step-wise, mul 1 stis-wiewwiii s at each timestep. For (b), we incor- porate the gradient calculated by specific ODE solver into both the stability criterion and the approximation correction. derive two approximation schemes that unify the =if and Iy trajectory, yielding principled estimate of the per-step clean sample zá compatible with advanced diffusion schedulers.
In summary, the core contributions of our work are: (i) We introduce SADA, a training-free framework that leverages a stability criterion adaptively sample the denoising process with principled approximation. (ii) To our best knowledge. SADA is the first paradigm that directly bridges numerical solvers of the sampling trajectory with sparsity-aware archi- tecture optimizations. (iii) Extensive experiments conducted on various baselines, solvers, and prediction frameworks demonstrate the effectiveness of SADA to existing accelera- tion strategies. Moreover, SADA can be easily adapted to new generation pipelines and modalities.
de
In particular,
2. Related Work
Diffusion models 2020 Song et al. Nichol & Dhariwal 2021 have emerged as a transformative framework for generative mod- eling. They generate high-fidelity samples by iteratively reversing stochastic process that gradually converts data into pure noise, delivering -fate-of-the-- art results across
(Sohl-Dickstein et al. 2015 Ho et al.
Both theoretical and empirical analyses indicate that SADA compatible with different backbone architectures (Ron-
neberger al. 2015 Peebles & Xie 2023) and solvers, and able to accelerate generative modeling in various modalities
2

---

SADA: Stability- Adaptive Diffusion Acceleration
wide range of tasks. In the realm of image synthesis, Stable Diffusion (Rombach al. 2022 enhances computational efficiency by mapping high-dimensional pixel inputs to compact latent space via a Variational Autoencoder (Kingma & WWlling 2014). Moreover, it incorporates text condition- ing through a pre-trained text encoder (Radford et al. 2021 Oquab et al. 2024), driven by classifier-free guidance (Ho &
in deeper layers.
Feature caching methods (Zhao al.
2024
Ma et al. Saghatchian et al., 2025 Liu et al.
2024a Wimbauer et al.
2024
Zhen al.
2025
2024 Chen al.
2024d
Shen et al.) Liu et al., 2025b) shorten sampling latency by reusing attention outputs. PF-Diff (Wang et al. 2024) lever- ages previous model outputs to predict a look-ahead term in the ODE solver, analogous to the use of Nesterov momen- tum. These step-wise approaches typically employ fixed via hyperparameter-
Salimans 2022). Building upon this foundation, later mem-
bers of the latent diffusion family (Podell et al.) Chen al.]
guide the sparsity pattern during sampling. In parallel, token reduc-
2024b Fores--orest-Lbbs. 2024) incorporate Transformers
( Vassaani et al. 2017) as the primary architectural backbone,
tion strategies (Bolya & Hoffman 2023 Kim et al.) 2024 Zhen et al. 2025 Saghatchian et al., 2025) exploit the re- dundancy in image pixels to eliminate unnecessary tokens, thereby reducing the attention module's computational load. ToCa series (Zou et al. 2024 Zhang et al. 2024) combines
enabling improved scalability (Peebles & Xie 2023) and
higher-resolution generation (Chen et al.] 2024a Gao et al.
2024). However, these advancements come at the cost of efficiency: the iterative nature of the denoising process and the quadratic complexity of self-attention significantly slow down inference.
caching and token pruning. DiTFastAttn (Yuan et al.) [022]] Zhang et al., 2025 compresses the attention module lever- aging the redundancies identified after a brief search.
Accelerating Diffusion Models To mitigate the above two bottlenecks, existing acceleration strategies typically focus on two fronts: (1) reducing the number of inference steps, and (2) reducing the computational cost per step.
Despite these advancements, none of these approaches dy- namically allocate step-wise and token-wise sparsity to ac- celerate sampling in multi- craliiatrrllarttyy levels. Moreover, their end-to-end acceleration configurations typically hinge on specific hyperparameters or pre-trained models. Adaptive Diffusion (Ye et al. (024)) offers promising perspective by adjusting its acceleration mode based on the prompt. How- ever, it still requires hyperparameter tuning and does not correct for approximation error. In contrast, we frame dif- fusion accelcration as a -eeeiit-prriitiinn problem rather than merely controlling error accumulation. SADA requires minimal hyperparameter tuning and incorporates principled approximation schemes that directly match our stability cri- terion.
The first paradigm involves interpreting the underlying stochastic process, developing numerical methods, and intro- ducing novel objectives and frameworks. DDIM (Song et al. reformulates the diffusion process as a non-Markovian procedure. Subsequent work identifies the diffusion pro- cess as stochastic differential equation (Song & Ermon 2019) and converts it into solving a probabilistic-flow ordi- nary differential equation (PF-ODE) that could be param- eterized by various model output objectives (Song et al.
b) Chen et al. 2023 Kim et al.) Kingma & Gao 2023). Building on this perspective, advanced numerical solvers significantly decrease the required number of function eval- uations. Among these, the Euler Discrete Multistep (EDM) solver (Karra et al.) 2022 Liu et al., 2023) has been ap- plied effectively, while the DPM-Solver series (Lu et al. 2022a Zheng et al. 2023) further enhance efficiency and stability by computing the linear component analytically with respect to the semi-linearity of PF-ODE. More recent works further explore the sampling trajectory of generative modeling. Consistency models (Song et al., 22033 Lu & Song, 2024) establish a theoretical one-step mapping from any point on the PF-ODE to the data distribution, while the
3. Proposed Method
3.1. Preliminary
Diffusion Models
(Sohl-Dickstein et al. 2015 Ho et al.,
2020 Song et al. ap Nichol & Dhariwal 2021) formulate
sample generation as iterative denoising process. Start- ing from a Gaussian sample zT ~ (O, I). these models progressively recover a clean signal over T timesteps. The forward process is defined by variance schedule { with a = 1 - 3g and āg = =I ag. The reverse (sampling) process is then modeled as:
flow matching (Liu et al. 2022 Albergo & -ajddnnEijnden 2022, Lipman et al., 2023) casts generative modeling as directly learning the ODE that transports samples from the noise distribution to the data distribution.
po(0:-1 zz)) = N (ae-1 B,I)
(I)
where the mean ua 1) vove
xx 1- (z 1) is
The second paradigm mainly leverages either step-wise or token-wise sparsity in diffusion models to reduce compu- tation overhead. On the step-wise front, DeepCache (Ma et al. 2024b) accelerates the U-Net (Ronneberger et al. based denoising model by caching high-level features
parameterized by noise prediction model co (z t). Mean- while, the data reconstruction æf could be calculated as:
1 1(xe v1 -aveo(z)n)t)
2/1
Var (xt
(2)
2015
3

---

SADA: Stability- Adaptive Diffusion Acceleration
0.371
1.50
Model
X
1.8
x
E ft+1
Yo. Yt+1. Yt+2
1.41
0,100
ourut
Sgracient
See Equation (16)
Xt-1
2 &
Salver
X
Spredap (1)
LIMPS ())
Model
1
E
X0
True
ê ft
Cache
0.255
Criterion 3.4
Fv Et
P1-1 Yv Y(+1
Spodk
1.56
LAVS
False
xc-1
LGC
6.4
Model
é
DP
80
Salver
Xt=2
Spentop (t)
LPEPS
Mask
DeepCeche
Adptiieiiitsiis
Figure Overview paradigm of SADA. The sparsity mode (middle: step-wise se, bottom: token-wi timestep f- 1 is adaptively identified by the stability Criterion 3.4er fresh computation at timestep t. Note that "DP" in the pipeline stands for "Data Prediction". Visualization of SADA and baseline methods' performance terms laithfulness and efficiency. Our methods significantly on both metrics, using (SD-2 (Top), SDXL (Bottom)) with
Right:
outperforms existing baselines (DeepCache, SeaptiveDiftue DPM-solver++ 50 steps.
For text-to-image synthesis, generation typically occurs a learned latent space (Rombach et al. 2022). this setting, the denoising model is usually implemented using deep neural network composed of L transformer-based layers. At the 1-th transformer layer timestep the input latent (1)
work Yee et al. 2024) leverages a third-order difference of the latent feature map x, to identify temporal redundancy in the sampling process. Let A (1) = 2t # as backward finite difference. At step f, given the noise e, and threshold T, if:
representation is denoted by X₁ E BBxHxx xc where B the batch size, and H, W, C' are the spatial and embedding dimensions, respectively.
(IA) (1) r ll + (A())) 1)/2 |40
ST. (5)
| (1)
then the denoising model is bypassed for the subsequent timestep 1, and 6--1 is reused by es.
Sampling with diffusion models can be reformulated as solving reverse- reverse-time ordinary differential equation (ODE). In particular, previous works (Song & Er- mon 2019 Song et b) defined the Probability-flow ODE that characterizes the continuous- time evolution samples:
Diffusion ODEs
Token-wise Sparsity accelerates diffusion models with high representation granularity. The token indices of the input sequence in a transformer layer are partitioned into Treduce and Tiss aiming to maximize [Teduuee while main- taining acceptable image quality. An index mapping / [1. + {I...., N.) is defined, where N' A standard token pruning (Bolya & Hoffman 2023 Kim
da
(²)
f(t)x; + ep (I+: 20,
(3)
x=
where f(1) log Von, g²(()= da, 2 (log Var) or. and 02 = /I āg. Here, time is assumed to be contin- uous with & ~ ((00 1)). Recent works explored rectified flows (Liu et al. 2022 Albergo & Vanden EijndnnEijnden] 2022 Lipman et al. 2023), where the reverse transformation is
dt
et al. 2024 process then discards Ireduce and retains only Lnx:
x|i] = x(i'], = N', where 1(i') = j and E Iax- After self-attention A(-), the feature map is reconstructed by:
(6)
achieved with a learned vector field (x t) predicted by the denoising model:
x||] A() (1()),
i=
(7)
dx
= uo(oE, E)-
(4)
Removing more tokens often degrades generation quality. Our analysis in Appendix shows that token merging be- haves as a low-pass filter, motivating our choice to build upon token pruning.
dt
Without loss of generality, we mark de For brevity, we further denote y(re. 1) Ut. ex 1) as et. and (I t) as Ut
de l=z, as y(re, t).
3.2. Overall Paradigm
Step-wise Sparsity accelerates the diffusion model by adaptively reducing the noise prediction steps. The recent
Figure illlstrates the overall paradigm of SADA. We lever- age the precise gradient from the sampling trajectory to

---

SADA: Stability-guided Adaptive Diffusion Acceleration
measure the denoising stability, as described in Sections 3.3 This Boolean measure serves an overall criterion for our acceleration process. Specifically, when Criterion 3.4 returns True, we apply step-wise cache- assisted pruning (refer to Section 3.4) with dual approximation scheme {step-wwid wwis,, multistep-wistt
To derive our stability criterion, we now present the follow- ing supplemental theorems:
Theorem 3.2. The expected value of xi over the joint dis- tribution of Fo, G, and timestep t satisfies: Erzo,e,([ee] = Vā.- [zo]
Conversely, when Criterion 3.4 returns Fe,se, we apply token-wise cache- assisted pruning (refer Section 3.5)
(i) From Theorem 3.2 we know that the trajectory { is continuous in expectation. Consequently, the empirical average Iy also exhibits continuity by the law of large num- bers (Feller et al. 1971).
3.3. Modeling the Stability of the Denoising Process
We formulate the acceleration of the denoising process as a stability prediction problem. Given a prr--rraimee trained diffusion architecture with noise prediction 6, timestep i, we seek dynamic criterion for step-wise and token-wise cache- assisted pruning (i.e., optimal sparsity allocation) that re- duces computation at timestep - without degrading sample fidelity.
Theorem 3.3. Let the network eo (x., be trained with the standard mean-sene mean-squared error (MSE) objective:
( Exgg,,, [[]c co (2,=,,))1]
(11)
Suppose O minimizes C, and the training is sufficiently converged. Then, following Assumption 1 that cø is Lipschitz in It and t, we have the following consis- tency property for sampling-time inputs iy:
The criterion for acceleration should match the specific ap- proximation method that mitigates the loss resulting from step-wise pruning. AdaptiveDiffusion (Ye et al., 2024) di- rectly caches and reuses the noise across steps. However, approximating é = E; introduces a mismatch between the sample state It-1 and its corresponding noise, causing error to accumulate. A more principled approach should incorporate historical information to correct this approxi- mation.
Ezg.c,fle - cg- (2x, - 0 as 12. - zell + 0. (12)
(ii) From Theorem 3.3 well-trained denoiser permits us to treat the sampling trajectory as continuous process, preserving structural consistency.
We therefore assume that, in a stable regime, the sign of consecutive third-order differences remains consistent: sign A (.))
Concretely, propose a third-order extrapolation of the sample state at í - 1, which implicitly carries historical noise along the ODE trajectory. A straightforward third- order backward finite-difference baseline can be written as f = 3z,- 30,41 + Z142. This yields a correction term,
sign (8) Consequently, if the ex- trapolation error It-1 -z is aligned with the true curva- ture A (3) 1; then - serves as a directionally accurate approximation. To incorporate precise gradient information, we leverage the always-hold identity A (2) A (3) x, < 0 by construction. Combining the sign measure with the identity, we substitute A (3) x, with A (3) 2t-1 4t and yield the following criterion:
Ft-1 - - = A (13)x,11- x4-1- Moreover, - is theoretically
bounded by Theorem 3.1 which guarantees its stability under small At.
Theorem 3.1. Let f E (*(aa,) b] be a smooth function and let xo == I, with equally spaced grid points Ta == ih for i = 0, 1....,k 1. Define (t) as the degree-(k: 1) Lagrange interpolant of at ( k-1
Criterion 3.4. A timestep t is considered stable and
eligible for acceleration if the extrapolation error is anti- aligned with the local curvature of velocity:
Then, the extrapolated value at x h satisfies:
(11-1-21-1) A < O.
(13)
k-1
hh) Σ axf(),)) (h),,,
(8)
This stability measure ensures that the extrapolated state â- lies in the correct direction with respect to curvature correction.
where the weights given by:
3.4. Step-wise Cache-Assisted Pruning
a: = --1) 1: (-)))(++)) i- O........k k-
(9)
This section proposes two complementary approxima- tion schemes for Step-wise Cache-Assisted Pruning: (step-wise) and (multistep-ww mation scheme produces a clean-sample estimate af. which is then fed into advanced samplers (e.g., DPM-Solver++ or
The error bound of the remainder term is:
Either approxi-
R4 = )hht).
(10)
5

---

SADA: Stability guidee Adaptive Diffusion Acceleration
3rd order FDM
Mean Squared Error
0.08
Ours
0.06
0.04
0.02
0.00
20
Baseline (SDXL)
Order (LPIPS 0.072)
3 Order (LPIPS 0.054)
Step Index
Step-wise pruning result of the third-order finite difference method and third-order Adams-Moulet method. Right: Mean Squared Error comparison between two strategies, averaged over randomly selected prompts. Shaded regions indicate the standard deviation at each step.
Figure 3. Comparison of ap-approximation strategies. Left:
Multistep-whee wttp:-/wwwwhgtootitttttee Agproximnation
The full derivation and the corresponding bound provided in Proposition B.II Theorems 3.1 B.2 and 3.5 in the Appendix B.2 To quantify the benefit of these precise updates, we measure per-step reconstruction error on 50 randomly sampled MS-COCO prompts (Lin et al. 2014). Our Adams-Moulton scheme results lower mean error and smaller standard deviation, compared to third-order finite difference, as illustrated in Figure
Semantic-planning Stuge
Data Prediction
Step-si Approximation
Approximated State
bi Equation w/ previous
Equation 16 w/ previous xf
Given that zá captures structural information and serves as the initial input to all schedulers, accurately reconstructing z is critical. Theorem 3.6 establishes an upper bound on the reconstruction error at timestep t. based on the third- order estimator in Theorem 3.5 and incorporating the effect of noise reuse.
x
Fidelity -mpprvvigg Stage
Figure Illustration the proposed dual approximation scheme on the x/f and trajectory.
Theorem 3.6. Let i'g denote the reconstruction of zo at time t. Then, the final reconstruction i6 satisfies the following error bound:
EDM (Karras al. 2022 Lu al. 2022 This unified framework aligns the z and 2, trajectories (Fig. 4). Below, we detail the design choices for each approximation.
|| - -18ll O OAAr))
(15)
Step-wise Approximation As noted in Section 3.3 simple baseline for approximating #z-1 is third-order backward finite difference. Empirical experiments demon- strate low reconstruction errors, as shown in Figure 3 Note that we reuse the noise prediction at step 6, formulated as é < ét, under this scheme. However, since we have exact derivative information Yt (See Eq. 3 and third-order Adans-Moulton method (Iserles 2009) along the ODE trajectory:
This theorem characterizes the reconstruction error as a first-order term in both the scheduler resolution At and the variation Ax, The detailed proof can be found in Ap- pendix B.2
Multistep-wise Approximation Prior work (Liu al. 2025b) empirically divides the denoising trajectory into a semantic-planin stage and a subsequent sideeuuunnfieellyyii--eee fidelity-iproovin stage, corresponding to regions where data z is inherently stable (see Figure In these stable regions, one can safely use larger effective step sizes by compensating with higher- order interpolation. Building on this insight, we implement uniform step-wise pruning strategy with Lagrange inter- polation, once the trajectory enters the stable regime.
we adopt
Theorem 3.5. Using the second- and hhird-order method, we define the estimator:
5At
5At
2At
81-1 ==
U1
Ut+1
+
3 V142, (14)
6
For example, consider a 50-step process. To achieve a step- wise pruning interval after stabilization (i.c., compute every 4th step fully and interpolate the skipped steps via
whose local truncation error satisfies: 21-1 It-1 (
6

---

SADA: Mability-guidee Adaptive Diffusion Acceleration
Our algorithm is formulated follows:
(i) Cache Initialization: Let T denote the starting timestep for Cache-Assisted Pruning and the caching interval. For an input at timestep t 1 and transformer layer I, if (t 1 T*) mod O, we initialize the cache after full computation:
Denoising Model
Rocosruucs
Datn Prodiction
C = A(x BBBNCC
(18)
where C₁ is the feature map stored in the cache for layer I, and it is updated throughout the denoising process.
Mask
(ii) Cache Update: When (t 1 - T*) mod * O, we prune the input data into x E BB Xxx length N' [Zxx] After the Transformer (/Attention), update C, with fresh tokens:
x where its
Figure 5. Illustration of the proposed Token-wise strategy. The criterion at token-level generates a "mask" to guide the pruning process at input of (-th attention layer. The pruned feature map is then reconstructed at the output of the layer using its cached representation C₁.
c A( (())) for Iax-
(19)
(iii) Cashe-Assisted Reconstruction: The pruned tokens approximated by their cached representations.:
Lagrange), we store zó every 4 steps before stabilization. Their indices define the fixed-size set I, which is a rolling buffer to limit memory usage. For any skipped t:
A(x ((4()) if Iax: CG
(20)
if i E Treduce-
Theorem 3.7. Let T (0, (....... k} be ls distinct indexes with known cached values (xt hier. For any skipped timestep & {ti htr, define the interpolated reconstruction as:
We keep the reconstructed sequence x x synchro- nized with Ct for subsequent timesteps.
4. Experiments
16 = - SEI (ErV())
II
4-4 t, 11
r
(16)
4.1. Experiment Settings
Model Configurations
To verify the generalization of the proposed approach, we evaluate set of widely used text-to-image models employing different backbone archi- tectures: SD-2 (U-Net), SDXL (Podell et al.) (modified U- Net), and Flux. 1-dev Fores-Forest-Labs 2024 (DiT). We perform evaluations using two sampling schedulers-
Then, under the assumption that 27 is (k + 1)-times continuously differentiable over T E [lming lmax], the interpolation error satisfies:
| - = O(h
(17)
where h is the maximum step spacing among {li).
Discrete Multistep (EDM) (Karras al. 2022) Solver (first- order) and DPM-Solver++ (Lu et al. 2022b) (second-order) -each configured with 50 sampling steps. All pipelines are implemented using the Huggingface Diffusers frame- work. Experiments with Flux. I-dev are executed on single NVIDIA A100 GPU, while the remaining experiments are run on a single NVIDIA A5000 GPU.
This multi-step cache-assisted pruning strategy significantly induce step-wise sparsity while preserving sample fidelity.
3.5. Token-wise Cache-Assisted Pruning
At timestep the Criterion 3.4 returns False, we con- tinue evaluate our stability measure under high gran- ularity level. The core idea of our token-wise algorithm is aligned with its step-wise counterpart: (i) fix unstable tokens for full calculation (ii) reduce the stable token and approximate it by previous representation in latent cache. This procedure partitions the tokens into two sets, Iíx (un- stable tokens) and Iestuse (stable tokens), which define our adaptive pruning configuration for the subsequent timestep.
We compare our proposed paradigm with widely-adopted training-fire acceleration strategies, DeepCache, AdaptiveDiffusion and TeaCache. (Ma et al.) 2024b Ye et al. 2024 Liu et al. (025a)). DeepCache caches and reuses the latent feature in the middle layer of the U-Net architecture. Adfptive skips the noise predictor and reuses the previous predicted noise guided by a third- order estimator. TeaCache introduces caching threshold
Evaluation Metries

---

SADA: Stability-guided Adaptive Diffusion Acceleration
Table Quantitative results on MS-COCO 2017 (
2014
Model
Scheduler
Methods
PSNR 1
LPIPS
FID 4
Speedup Ratio
DeepCache
17.70
0.271
7.83
1.43x
DPM+
AdaptiveDiffusion
24.30
0,100
4.35
1.45x
SADA
26.34
0.094
4.02
1.80
SD-2
DeepCache
18.90
0.239
7.40
1.45%
Fuler
AdaptiveDiffusion
21.90
0.173
7.58
1.89 x
SADA
26.25
0.100
4.26
1.81x
DeepCache
21.30
0.255
8.48
1.74
DPM++
AdaptiveDiffusion
26.10
0.125
4.59
1.65x
SADA
29.36
0.084
3.51
1.86
SDXL
DeepCache
22.00
0,223
7.36
2.16x
Euler
AdaptiveDiffusion
24.33
0.168
6.11
2.01
SADA
28.97
0.093
3.76
1,85x
TeaCache
19.14
0,216
4.89
2.00x
Flux
Flow-matching
SADA
29.44
0.060
1.95
2.02
FID down from 8.48 to 3.51 (59% 4) on SDXL with OPM--Slver+++ and Flux. 1-dev from 4.89 1.95 (60%
Table Ablation study on few-step sampling across schedulers.
Results on MS-COCO 2017.
L). Across every configuration, SADA maintains LPIPS < 0.100 relative to the original samples-sune substasi-sssta bet- ter than competing
SD-2
Scheduler
Steps
PSNR
LPIPS 4
FID L
Specdup
only negligible perceptual deviation from the unmodified baseline. Cru- cially, these significant quality improvements incur no extra compute beyond the acceleration itself, yielding consistent 1.8-2x speedups, outperforming the competing methods in the majority of cases, underscoring SADA's effectiveness loss-free acceleration framework.
26.34
0.094
4.02
1,80
DPM++
25
28.15
0.073
3.13
1,48
15
29.84
0.072
3,05
1.24x
50
26.25
0.100
4.26
1.81 x
Euler
25
26.83
0,088
3,87
1,48x
29.34
0.076
3.70
1.25x
SDXL.
Scheduler
Steps
PSNR
LPIPS 4
FID 4
Speedup
4.3. Ablation Studies
50
29.36
0.084
3.51
1.86x
We provide justification for choice of base step T = 50 in Figure A.3 To evaluate SADA's performance under few- step sampling, Table reports results on {SD-2, SDXL using {Euler, DPM-Solver++) while varying the number of inference steps {50, 25, 15). As the step count decreases,
DPM++
25
30.84
0.073
2.80
1.52x
31.91
0.073
2.54
1.29
2
50
28.97
0.093
3.76
1,85x
Buler
29.42
0.085
3.13
1,50x
15
31.28
0.084
3.26
1,26x
SADA achieves higher similarity: PSNR rises from 26. 6..25 dB to 29.34 dB, FID falls from 4.26 to 3.70, and LPIPS drops from 0.100 0.076. We attribute this trend reduced eπror accumulation when using fewer steps. Mcanwhile, SADA can further accelerate sampling under few-step setting. The speedup ratio maintains ~ 1.5x under 25 steps denoising, and ~ 1.25x under 15 steps denoising, high- lighting SADA's effectiveness under low computational budget. Note that the Lagrange interpolation parameters are slightly adjusted match the shorter denoising schedules in these few-step settings.
that measures the error accumulation. All experiments are conducted using the MSCOCO-2017 validation set as gen- cration prompts under identical conditions assess effi- ciency and quality. We report speedup ratios compared to the baseline measure of generation efficiency. Gener- ation quality is evaluated using the Peak Signal-to-Noise Ratio (PSNR), Learned Perceptual Image Patch Similarity (LPIPS), and Fréchet Inception Distance (FID) between original generated and accelerated samples.
4.4. Downstream Tasks and Data Modalities
4.2. Main Results
This section demonstrates that SADA has potential accel- erate any generative modeling with an iterative generative process, regardless of the downstream task or data modality.
As shown in T SADA consistently outperforms Deep- Cache, Difftive and TeaCache across all set- tings. Compared to other acceleration strategies, drives
8

---

SADA: Stability guided Adaptive Diffusion Acceleration
"Chtte ay Sixtory Menns hop
LPEPS 0.008
5. Conclusion
We present Stability-guided Adaptive Diffusion Accelera- tion (SADA), training- free paradigm that adaptively ac- celerates the sampling process of ODE-based generative models (mainly Diffusion and Flow- (low-matching). Leverag- ing trajectory gradient calculated from the numerical solver, SADA dynamically exploits step-wise and token-wise spar- sity during the procedure with principled and error-bounded approximation, bridging between the sampling trajectories and sparsity- -awae optimizations. Extensive experiments on SD-2, SDXL, and Flux across EDM and DPM++ solvers both demonstrate consistent 2 1.8x speedups with negli- gible fidelity loss (LPIPS < 0.10, FID I 4.5) compared to unmodified baselines, significantly outperforming exist- ing strategies. Moreover, we show that SADA generalizes across modalities acchaities-achievingg ~ 1.81 x acceleration on Mu- sicLDM and 1.41 on ControlNet withoou additional tuning.
00
"Farkionalle Pap Case Siecto"
LPPPS 0007
52-
"4, "4,ooiiiic piace musicc with the dogp sound everippiig places
LPPP - 0017
2,574
SADA: L42s
Figure 6. SADA deployment MusicI.DM on different text prompts. SADA accelerates MusicLDM by 1.81x while main- taining the spectrogram LPIPS under 0.020.
"A deer."
"Seowy forent."
Acknowledgment
This material is based upon work supported by the U.S. National Science Foundation under award No.211 This work is also supported by ARO W91 N911NF-23-220022 and NAIRR Pilot project NAIRR240270. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the U.S. National Science Foundation, ARO, NAIRR, and their contractors. In addition, we thank the area chair and reviewers for their valuable comments.
Bose Image
ControlNet Bascline (4.76
SADA (3.36 s)
Figure 7. SADA deployment on ControlNet. We demonstrate the SD-I.5-based ControlNet pipeline trained on canny edges as con- ditional input. SADA accelcrates ControlNet by 1.41 while preserving fidelity.
Impact Statement
This paper aims to advance the field of adaptive accelcration of generative models. We believe that our work has signifi- cant potential impact for the deployment and inference of generative models in any modalities. While there are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Data Modality We evaluate SADA on music and au- dio generation using the MusicLDM (Chen et al., (202c) pipeline to synthesize 8-second clips. We compare both perceptual quality and spectrogram similarity between ac- celerated samples and the unmodified baseline. As shown in Figure 6 SADA achieves an LPIPS of ~ 0.010 between accelerated and baseline audio, while reducing sampling time by ~ 1.81x. This implies that SADA has the potential implemented in future baselines models for any-modality generation with little to no modifications.
References
Albergo, M. S. and Vanden-Eijnden, E. Building normal- izing flows with stochastic interpolants. arXiv preprint
M:Xiv:2209//5777 2022.
To validate cross- cross-pipeline compatibil- ity, we apply SADA directly on top of ControlNet (Zhang et al. 2023) without any additional fine-tuning or architec- tural changes. Figure[]demonstraal faithfulness between ControlNet-conddiioneee outputs while accelerating the sampling by ~ 1.41x. This implies that SADA could be seamlessly deployed in professional work- flows.
Downstream Task
Forck-Forest-Labs. Flux. https://qithub hlt.
black forest- flux 2024.
Bolya, D. and Hoffman, J. Token merging for fast stable diffusion. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pp. 9p.4599-4603 2023.
that SADA preserves
Chen, J., Ge, C., Xie, E., Wu, Y., Yao, I.., Ren, X., Wang,

---

SADA: Stability-guided Adaptive Diffusion Acceleration
Z., Luo, P., Lu, H., and Li, Z. Pixart-o: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. In European Conference on Computer Vision, pp. 74-91. Springer, 2024a.
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35: 2022.
Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wang, Z., Kwok, J. T., Luo, P., Lu, H., and Li, Z. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In International Conference on Learning Rep- resentations, 2024b.
Kim, D., Sony, A., Lai, C.-H., Liao, W.-H., Murata, N., Takida, Y., He, Y., Mitsufuji, Y., and Ermon, S. Consis- tency trajectory models: Learning probability flow ode trajectory of diffusion.
Kim, M., Gao, S., Hsu, Y.C.,, Shen, Y., and Jin, H. Token fusion: Bridging the gap between token pruning and token merging. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1383- 1392, 2024.
Chen, K., Wu, Y., Liu, H., Nezhurina, M., KirkgKKrkaa,rick, T., and Dubnov, S. Musicldm: Enhancing novelty in text-to-music generation using beat-synchronous mixup strategies. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1206-1210. IEEE, 2024c.
Kingma, D. and Gao. R. Understanding diffusion objectives as the elbo with simple data augmentation. Advances in Neural Information Processing Systems, 36:65484 65516, 2023.
Chen, P., Shen, M., Ye, P.. Cao, J., Tu, C., Bouganis, C.-S., Zhao, Y., and Chen, T. d-dit: A training-free acceleration method tailored for diffusion transformers. arXiv preprint arXiv:2406.01 125, 2024d.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. In Proceedings of the International Conference on Learning Representations (ICLR), 2014.
Chen, S., Daras, G., and Dimakis, A. Restoration- degradation beyond linear diffusions: A non-asymptotic analysis for ddim-typs samplers. International Con- ference on Machine Learning, pp. 4462-4484. PMLR, 2023.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. European Conference on Computer Vision, pp. 740-755. Springer, 2014.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F,, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and
Le, M. Flow matching for generative modeling. In The Eleventh International Conference on Learning Repre- sentations, 2023.
Feller, W. et al. An introduction to probability theory and its applications. 1971.
Liu, E., Ning. X., Yang. H., and Wang, Y. unified sam- pling framework for solver searching of diffusion proba- bilistic models. In The Twelfth International Conference on Learning Representations, 2023.
Gao, P., Zhuo, L.,, Liu, D., Du, R., Luo, X., Qiu, L., Zhang, Y., Lin, C., Huang, R., Geng, S., et al. Lumina- 12x: Transforming text into any modality, resolution, and du- ration via flow-based large diffusion transformers. arXiv preprint arXiv:2405.0594 2024.
Liu, F., Zhang, S., Wang, X., Wei, Y., Qiu, H., Zhao Y., Zhang, Y., Ye, Q., and Wan, F. Timestep embedding tells: It's time to cache for video diffusion model. arXiv preprint arXiv:2411.19108, 2024.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Liu, F., Zhang, S., Wang, X., Wei, Y., Qiu, H., Zhao Y., Zhang, Y., Ye, Q. and Wan, F. Timestep embedding tells: It's time to cache for video diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 2025a. URRp::ttts://///p thecvf com/ Fiitual/2255posteeree Hall D, Poster Session 2.
Ho, J. and Salimans, T. Classifier-fres free diffusion guidance. arXiv preprint arXiv:2207.12598 2022.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, (116840-685.) 2020.
Poster #190, Ex-
Iserles, A. A first course in the numerical analysis of dif- ferential equations. Number 44. Cambridge university press, 2009.
Liu, H., Zhang, W., Xie, J., Faccio, F,, Xu, M., Xi- ang, T., Shou, M. Z., Perez-Rua, J.-M., and Schmid- huber, Faster diffusion through temporal attention
10

---

SADA: Stability guiided Adaptive Diffusion Acceleration
decomposition. Transactions Machine Learning Re- search, 2025b. ISSN 2835-8856. URL https://
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint ar arXiv:2209.03000
2022.
Ronneherger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmenta- tion. In Medical image computing and computer-assisted intervention-MiCCA 2015: 18th international confer- ence, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234-241. Springer, 2015.
Lu, C. and Song, Y. Simplifying, stabilizing and scal- ing continuous-time consistency models. arXiv preprint arXiv:2410. 1081, 2024.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C.,, and Zhu, J. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling around 10 steps. Advances in Neurul Information Processing Systems, 35:5775-5745 5787, 2022a.
Saghatchian, O., Moghadam, A. G., and Nickabadi, A. Cached adaptive token merging: Dynamic token reduc- tion and redundant computation elimination in diffusion model. arXiv preprint arXiv:2501.0094 2025.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2022b.
Shen, M., Chen, P., Ye, P., Xia, G., Chen, T., Bouganis, C.- S., and Zhao, Y. Md-dit: Step-aware mixture-of de-tss for efficient diffusion transformers. In Adaptive Founda- tion Models: Evolving for Personalized and Efficient Learning.
Ma, X., Fang, G., Bi Mi, M., and Wang, X. Learning- to-cache: Accelerating diffusion transformer via layer caching. Advances in Neural Information Processing Systems, 37:133282-133304 2024a.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In International conference on machine learning, pp. 2256-2265. pmlr, 2015.
Ma, X., Fang, G., and Wang, X. Deepeache: Acceler- ating diffusion models for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15762-15772, 2024b.
Song, J., Meng. C., and Ermon, S. Denoising diffusion implicit models. In International Conference on Learning Representations, a.
Nichol, A. Q. and Dhariwal, P. Improved denoising diffu- sion probabilistic models. In International conference on machine learning, pp. 8162-8171. PMLR, 2021.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El- Nouby, A., et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research Journal, pp. 2024.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, b.
Peebles. W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195-4205, 2023.
Song. Y., Dhariwal, P., Chen, M., and Sutskever, I, Consis- tency models. In International Conference Machine Learning, pp. 1-22222. PMLR, 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations.
Wang, G., Cai,, Y., Peng, W., Su, S.-Z., et al. Pfdiff: Training- free acceleration of diffusion models combining past and future scores. In The Thirteenth International Conference Learning Representations, 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Wimbauer, F., Wu, B., Schoenfeld, E., Dai, X., Hou. J., He, Z., Sanakoyeu, A., Zhang, P., Tsai, S., Kohler, J.

---

SADA: Stability-guided Adaptive Diffusion Acceleration
et al. Cache me if you can: Accelerating diffusion models through block caching. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1211-0 2024.
Yang, Z., Feng, R., Zhang, H., Shen, Y., Zhu, K., Huang, L.. Zhang, Y., Liu, Y.. Zhao, D., Zhou, J., et al. Lips- chitz singularities diffusion models. In The Twelfth International Conference on Learning Representations, 2023.
Ye. H., Yuan, J. Xia, R., Yan, X., Chen, T., Yan, J., Shi, B., and Zhang, B. Training- free adaptive diffusion with bounded difference approximation strategy. arXiv preprint ar wiiv:2410.09873. 2024.
Yuan, Z., Zhang, H., Lu, P., Ning, X., Zhang, L.,, Zhao, T., Yan, S., Dai, G., and Wang, Y. Ditfastatin: Attention compression for diffusion transformer models. arXiv preprint 2tt.:2406008555,, 2024.
Zhang, E., Xiao, B., Tang. J., Ma, Q., Zou, C., Ning, X., Hu, X., and Zhang, L. Token pruning for caching better: times acceleration on stable diffusion for free. arXiv preprint ar 00w:2200..00375. 2024.
Zhang, H., Su. R., Yuan, Z., Chen, P., Fan, M. S. Y., Yan, S., Dai, G., and Wang. Ditfastattnv2: Head-wist atten- tion compression for multi-modality diffusion transform- ers, 2025. URL https://arr/w..or////bbbt 22796
Zhang, L., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3836-3847, 2023.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-599 2018.
Zhao, X., Jin, X., Wang, K., and You, Y. Real-time video generation with pyramid attention broadcast. CoRR, 2024.
Zhen, T., Cao, J., Sun, X., Pan, J., Ji, Z., and Pang. Y. Token- aware and step-aware acceleration for stable diffusion. Pattern Recognition, 164:11 1is:111479,, 2025.
Zheng, K., Lu, C., Chen, J., and Zhu, J. Dpm-solver-v3: Im- proved diffusion ode solver with empirical model statis- tics. Advances in Neural Information Processing Systems, 36:55502-55544 2023.
Zou, C., Liu, X., Liu, T., Huang, S., and Zhang, L. Ac- celerating diffusion transformers with coken-wise feature caching. arXiv preprint 17X 2024.
12

---

SADA: Stability-guided Adaptive Diffusion Acceleration
A. Appendix
Due to the page limit of the main manuscript, we provide more theoretical and implementation details in the appendix, orgamized as follows:
Sec. B Mathematical Foundations
Sec. B.1
Theoretical Assumptions.
Sec.
Proofs of Main Theorems.
Sec. Analysis on Existing Token Reduction Methods.
Sec. D
Additional Experimentss.
B. Mathematical Foundations
B.1. Theoretical Assumptions
The network output (2 t) is jointly Lipschitz continuous in both 2; and t. In our experiments, we skip the first and last time steps to avoid potential issues with infinite Lipschitz (Yang et al. 2023) constants near the boundaries.
B.2. Proofs of Main Theorems
Theorem 3.2. Let Tt = V ae e, where e ~ _N I) and æo ~ p(za) is independent Then the expected value of 24 over the joint distribution of zo, e, and timestep & satisfies
[xo.,, Va.- Ezo [200].
(A.1)
Proof. By the definition of the forward process in diffusion models, the latent variable timestep t is given by:
Te Vārzo + vI ag
(A.2)
Taking expectation zu p(xo), NoO, 1), and ~ Uniform((1,.... TT),, get:
E E [ma] = E, Ero (orrol + Bc[V1 oe
(A.3)
Since (0, we have [E[]] = 0, and thus:
E Ex..e.((22] = Ee (Vā Ex, [xol] Vā.- Exy [2o].
(A.4)
This concludes the proof.
Theorem 3.3 (Consistency of the network estimator under trajectory approximation). Let the network eg (z, t) be trained with the standard -qua-e error (MSE) objective:
((0)
Exo.e.t [|le - co(a,, 1)
2, v 04,00 + OrE.
(A.5)
Suppose 4* minimizes C, and the training is sufficiently converged. Then, following Assumption that ( t) is
13

---

SADA: Stability-guided Adaptive Diffusion Acceleration
Lipschitz in dy and I, have the following consistency property for sampling-time inputs dy.
Exc.e,cle cg. (8c, as c zell +
(A.6)
Proof. Since 24 is linear deterministic combination of (æn, <) time step the distribution p(a+, () is induced from the joint distribution of 20, e, and Therefore, minimizing the MSE loss over (zo, t) is equivalent to minimizing it over (z4, t). The optimal solution in the L sense is the conditional expectation:
Eg- (2t, Ala re,
(A.7)
Taking expectation again:
Exg,e,e Eg. (xres [)] O.
(A.8)
Now, for any approximation é, by Lipschitz continuity:
||ea- (#e, t) cg. (2e, < L|(e, xzll-
(A.9)
Thus,
[| - Eg- (2,, t)ll & [LEv.e,rEe- cg. (Ie, t)ll + Ezalet Illea- (xe, t) - eg. (8,, t)ll
(A.10)
The first term zero; the second vanishes 0, - Ptt- Therefore,
Exo., co. (2c, t)] + 0,
(A.11)
as | Jeel + 0.
Theorem 3.7 (Lagrange Interpolation for Cache-Assio Assisted Reconstruction). Let T {0, 1, 1,..., k- be k + 1 distinet
indexes with known cached values (ah) Fiez. For any skipped timestep & {tilee, define the interpolated reconstruction as:
$/1 = Σ
II jENVE)
t tj
z
(A.12)
t
SET
Then, under the assumption that v7 is (k + timtimes continuously differentiable over T E [tming (max], the interpolation error satisfies:
| x691 O(h+t),
(A.13)
where h is the maximum step spacing among {tz}.
Proof. The expression in Equation A.12 is the Lagrange interpolation formula for approximating function unobserved point using + 1 known values {x]] at points {t:} I.
By classical interpolation theory, the interpolation error at for a (k + 1)-times continuously differentiable function zū satisfies:
1 d++1ag x
2/6 2/
II SET
42)
for some & E [fmnny /max)
(A.14)
(k 1)!
dgh+1
14

---

SADA: Stability- Adaptive Diffusion Acceleration
Taking and bounding the derivative gives:
1 1 =b8l - (k 1)!
d 20 de
max &
II iE!
tyl
(A.15)
all time steps are approximately uniformly spaced with step size h, then It - t,l S Ch for some constant C, and hence:
1 =fol o(
(A.16)
as claimed.
Theorem 3.1 (Backward Extrapolation via Lagrange Interpolation). Let f E C* [a, bj be a smooth function and let
Zo := z, with equally spaced grid points 2; == x + ih for l....... 1. Define 1t) as the degree-(k: 1) Lagrange interpolant of f at (=-))=---
Then, the extrapolated value at h satisfies:
f(x ((- Σ
an (x))).
(A.17)
i=
where the weights are given by:
k (,=1) i= 0, o......,kk 1,
a: = (-1)²
(A.18)
and the remainder term is:
(E)
Ra(
(x h x.),
E h, (k 1)4].
(A.19)
k!!
and the error bound of the remainder term is:
R4 (h) O(h*)
(A.20)
Proof. Let 24 = x + for O,...,k We construct the Lagrange interpolating polynomial:
k-1
(t-1(t) = Σ ((x))
(A.21)
1
where the Lagrange basis functions are:
k-
X.
((t)
(A.22)
1=0
Evaluating gives:
f(z - h)) = P4-1(x - 1)) + Rx(h),
(A.23)
where the remainder term is the standard Lagrange error:
(E)
k-1 Il 1=
(h)
(x h - 2z), E [r - h + (I - 1)h).
(A.24)
kd
15

---

SADA: Stability-guided Adaptive Diffusion Acceleration
Since 3j x jh, have: (h) O( Now compute the weights 0g le(x h1), we have:
k-1
x - h)) (x (x
1j
4(1 - 11) =
(A.25)
x + ih)
(I: jh)
(i-5)h
1 j
Canceling h and simplifying signs:
(k: - 1)!
i+1 (-1)² ( k
i+t1 )
l(x - h) = --11)=-1
(A.26)
i!(h:: 1 i)!
Therefore:
f(x h
k-1
k i (+ f(x:)
((t))))))))))) + Re(h),
(A.27)
claimed.
Proposition B.1 (High-order backward difference as weighted combination). Let == (-1) be the extrapolation coefficients in Theorem 3.1 Then the following linear combination defines the k-th order backward finite difference:
k-1 aff(x ih) = A (8)
f(z h) h) Σ
f(x fl)
h),
(A.28)
1=
where (k)) is the standard k-th order backward difference operator:
A (8) f(x - 11) Σ (-11)) (1) [(x + ih).
(A.29)
Proof. The result follows directly by substituting the expression for a, from Theorem 3.1 into Equation A.28 and matching terms with the standard definition of (k))
Theorem B.2 (Adams-Moulton Method via Forward Lagrange Quadrature). Let y(r) E C* bj and let xig == x with
equally spaced grid points I; := x + ih for = o, 1. Let Px-1(t) be the degree-(k -k1 Lagrange interpolant of f. which is derivative of y. {z:)=-)
Define the one-step approximation of the integral:
(" f(s)ds.
y( -() h)) -))=
(A.30)
J0-h
Then, the Adams-Moulton method of order l is:
Σ Bulnii P
in-1 Un
(A.31)
where: Dn+j = y(2z), fats f(z)), and the quadrature weights B-1,Bo... 3k-11 are given by:
16

---

SADA: Stability- Adaptive Diffusion Acceleration
Bj = [( for = -1,0, -,.....,,,
(A.32)
where E5(8) are the Lagrange basis polynomials defined on nodes Tj = j. with T-1 for the future value Yn-1- The local truncation error
0n- Yn-1 O(h
(A.33)
Proof. We aim to approximate the integral:
u( - h) - u(x)) - -fa x-hh
f(s) ds.
(A.34)
Let us define variable substitution 8 = x + Th, so that:
[e (()) h
hhh/// f f(x h Th)
(A.35)
..K & Let Tj j for j = ......
and T-1 -1 (the future point), and define Lagrange basis polynomials:
(T()
T TJ T3
(A.36)
Let P&(T)= = - 4 (T)))((()) be the Lagrange interpolant of f(x + rh) using nodes Tj-
Now approximate the integral:
[ f(x f(x- (1x-h44 h rh) = ( Pr(T))r= = Σ ([(((taa f(x h h+
1-1
(A.37)
jm-l
Define:
Bj 8/:= = f/ ((())dr j ==,,,,.... ...,..... 1.
(A.38)
Then we get:
rth
k-1
f(8) =
Σ
8, -
h t1t))))
( - h) E
(A.39)
1=
17

---

SADA: Stability-guidel Adaptive Diffusion Acceleration
Substitute into the ODE integral:
R-1
9n-1 = =n + h B-15n- +
> 5=0
Syfnts
(A.40)
which is exactly the MountMMoulton method of order k.
Finally, since this is based on Lagrange interpolation over k + nodes (including the implicit node at future time), the quadrature error is +1)++)), leading to method of order k:
We now instantiate the backward Adams-Moulton method of order 3 from Theorem B.2 and quantify its effect in discrete difference form, including enror propagation.
Theorem 3.5 (Third-order backward difference via Adams-Moulton estimation). Let 2; be the trajectory satisfying the ODE: day, yt. Consider estimating J4-1 by third-order backward difference A (3) using forward values T1+11 Tt+2, where the step size is At. Furthermore, we using second and third order of Adams-Moulton method to define the estimator:
5A
5
2A1
8 := Dt
Dt-
+
V++2-
(A.41)
6
6
3
This estimate consistent with the discrete ODE, and the local truncation error satisfies:
h - 1-1-0-11000
(A.42)
Proof. By Theorem B.2 the third-order Adams-Moulton method reverse-time (backward extrapolation) reads:
Al
Ft-
F4.
Vt+1 + 59t-1)- (8y:
(A.43)
12
In our scenario, we aim to express everything in terms of xı, 2441, 2442, and Vt, V6+1, V1+2-
By using the identity:
A (3) xt-1 1
Fz-1 3x, + 3r.41 Fe+2,
(A.44)
we reverse it get the original estimation of 8;-1:
Tt-1 = 3x, 3x.+1 + 1142 = T, + 2(x, x441) - (x+41 - xl42).
(A.45)
Then we plug in a higher-order correction from the Adams-Moulton method:
AL (Ve41 + (2+22 + O(At)).
F(41
(5%r + 8y:+1 - V1+2) - O(At²)
F+41 T142
(A.46)
12
Combining these, we define the estimator -z-1:
5At
5At
2At
St.
1/+
Uh++
B4+2-
(A.47)
18

---

SADA: Stability-guided Adaptive Diffusion Acceleration
Thus have the error bound
5At
5
2
8 Tt-1 =
9t-1
+
Vt+2
1t-1
6
6
3
-i-- + O It-1
-(AI))
(A.48)
as claimed.
Theorem 3.6 (Error bound for final reconstruction a/)). Let & denote the reconstruction of zy at time t, obtained using an extrapolated estimate - from Theorem 3.5 which incurs an error of order ( Then, the final reconstruction o satisfies the following error bound:
|
=f6l O(Ai) O(Am))
(A.49)
Proof. This follows from the linearity of the prediction formula for S/ which is linear combination of xı and é. Therefore, by following Assumption 1 that ( is Lipschitz in Ie and
|
z1 < Cilla =ell + C2llér - -ell = O(At) + O(Az,),
(A.50)
where 11 and C2 constants depending the schedule.
C. Analysis on Existing Token Reduction Methods
Compared to the baseline, token merging with a high merging ratio significantly loses detailed information, while token pruning at the same ratio partially preserves fine-grained details. However, pruning still leads to the loss of a substantial amount of information in the generated image. In this section, we provide a short analysis of token merging, token pruning, and the unmerging process.
Token merging
We represent the token-merging procedure via an N' x N matrix M:
I. if S
My.
0, otherwise,
where each set S, C (l...., N} groups tokens deemed similar. The merged sequence x' is then obtained by:
x' Mx.
Within each subset Sy. the operation is an average: hy = operation has low-puss frequency response described by the sine function:
|s, IT_ From a signal-processinnn perspective, such an averaging
H(w) = sin π (Sy))
π u SS
Hence, merging tokens suppresses high-frequency components while retaining lower-frequency content.
Token pruning Token pruning also cause information loss, as any unique feature captured by the pruned tokens are no longer available for subsequent computations. Since it removes tokens without averaging, token pruning does not exhibit the low-pass filtering effect, resulting in the loss of specific spatial information rather than a smoothing the input sequence.
19

---

SADA: Stability- Adaptive Diffusion Acceleration
Unmerging process Furthermore, due the non-linear mapping of self-attention, the high similarity of two input tokens v/i..<] (where Sj) does not guarantee the high similarity of corresponding outputs after attention computation. Therefore, the unmerging procedure, which duplicates the processed merged tokens back their original position results in an inherent downsampling effect. This duplication does not recover the high-frequeng details lost during merging, thus further degrading the detailed features the output.
D. Additional Experiments
I'm pinds dome, Al is will rit, Transhonig trough wabin From draaed be srrees new world
I'm yodu
in picdi
Al is aaf will vtc
Al is sugh Mrtee
Texeafowd Frregh wabine From droas new
Traefooo machi
From dreamrhh
NAN well wer
Baseline
SADA
TeaCache
Figure A. Comparison between SADA and TeaCache on Fl.ux.1 1 Dev. Our method shows significantly hetter faithfulness under an identical speedup ratio of x.0x.
20

---

SADA: Stability-uiided Adaptive Diffusion Acceleration
Baseline
SADA
AdaptiveDiffusion
Figure A.2. Comparison between SADA and AdaptiveDiflusion SDXL with 50 steps DPM++. Our method shows better faithfulness with much faster speedup of 1.81 x vs 1.65x.
15 steps
25 steps
50 steps
80 steps
250 steps
Figure A.3. The generated samples from DPM-SOO Solver first dramatically change, then demonstrate convergence after setting the sample step to Therefore, we believe our baseline setting reasonable.
21


Reviews:
[{"review_id": "b4L5685qYO", "reviewer": "Reviewer_DBkM", "summary": "The paper proposes SADA, a training-free acceleration method for diffusion models that unifies step-wise (temporal) and token-wise (spatial) sparsity using a stability criterion based on the denoised latent $ x_0 $. Specifically,  the paper uses a unified $ x_0 $-guided sparsity criterion for step skipping and token pruning, leveraging $ x_0 $'s structural alignment with modern ODE solvers.  A second-order Adams-Moulton method to approximate skipped steps and a token cache to reconstruct pruned tokens.  Experiments on Stable Diffusion 2 and XL show speedups of up to 1.5× while maintaining image quality.\n\n## update after rebuttal\nThank you to the authors for their response and additional experiments, which have provided me with a deeper understanding of SADA's effectiveness. However, the baselines compared in this paper are somewhat non-repetitive, and some important papers on cache-based DiT acceleration, such as Learning-to-Cache and $\\Delta - Dit$, were not included in the comparison. Additionally, my concerns about the novelty of this paper remain. The caching method proposed in the paper does not differ fundamentally from previous approaches. Although the work most similar to this paper, [1], was published during the review period, earlier works like [2, 3, 4] also bear significant similarity in methodology, especially TeaCache [4]. A comprehensive experimental comparison with these papers, along with a detailed explanation of the differences in approach, is necessary. I still recommend rejecting this paper.\n\n[1] Token-aware and step-aware acceleration for stable diffusion\n\n[2] Cached Adaptive Token Merging: Dynamic Token Reduction and Redundant Computation Elimination in Diffusion Mode\n\n[3] Accelerating diffusion transformers with token-wise feature caching\n\n[4] Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for comprehensive comments.\n\n**Q1: Aggressive configuration of SADA**\n\n**A1:** To implement an aggressive version, we leverage the inherent robustness of the per‐step data reconstruction $x_0^t$. When the $x_0^t$ trajectory demonstrates high stability (e.g., the second half of Fig.2), we could employ larger step sizes compensated by higher-order approximations. Building on this insight, we implement a uniform step-wise pruning strategy after the stability of the denoising process with Lagrange interpolation for correction.\n\nFor example, consider a 50‐step process. To achieve a step-wise pruning interval of 4 after stable (i.e., compute every 4th step fully and interpolate the skipped steps via Lagrange), we store $\\hat{x}_0^t$ every 4 steps before stabilization. Their indices define the fixed-size set $I$, updated dynamically to limit memory usage. For any skipped $t$:\n\n$$\n\\hat{x}_0^{t}\\gets\\sum _{i \\in I}\\prod _{j\\in I\\setminus \\\\{ i \\\\}}\\frac{t-t_j}{t_i-t_j}\\hat{x}_0^{t_i}\n$$\n\nUnder this setting, we yield a $\\geq 1.8\\times$ speedup regardless of models or solvers. To balance the degradation, we raise the Adams-Moulton approximation from second to third order, allowing $x_0^t$ to leverage information from the previous three steps (instead of two), thereby improving numerical accuracy and robustness. Our updated result in Table 1 demonstrates the effectiveness of the above improvements.\n\n**Table 1: Quantitative results on MS-COCO 2017**\n|**Model**|**Scheduler**|**Methods**|**PSNR**|**LPIPS**|**FID**|**Speedup Ratio**|\n|-|-|-|-|-|-|-|\n|SD2|DPM++|DeepCache|17.70|0.271|7.83|1.43|\n|||AdaptiveDiffusion|24.30|0.100|4.35|1.45|\n|||SADA|**26.34**|**0.094**|**4.02**|**1.80**|\n|SD2|Euler|DeepCache|18.90|0.239|7.40|1.45|\n|||AdaptiveDiffusion|21.90|0.173| 7.58|**1.89**|\n|||SADA | **26.25**|**0.100**|**4.26**|1.81|\n|SDXL|DPM++| DeepCache|21.30|0.255|8.48|1.74|\n|||AdaptiveDiffusion| 26.10|0.125|4.59|1.65|\n|||SADA| **29.36**| **0.084**|**3.51**|**1.86**|\n|SDXL|Euler|DeepCache|22.00|0.223|7.36|**2.16**|\n|||AdaptiveDiffusion|24.33|0.168|6.11|2.01|\n|||SADA|**28.97**|**0.093**|**3.76**|1.85|\n\n**Q2: Comparison between $x_0$ and $x_t$-based criterion**\n\n**A2:** To our best knowledge, we are the first work that considers $x_0$ as acceleration criterion and approximation objective. We compare our paradigm with AdaptiveDiffusion, which leverages the third-order difference of $x_t$, as demonstrated in Table 1. \n\nResiding in an image representation space, $x_0$ demonstrates structural alignment with the final output while evolving in a more robust trajectory (as shown in Fig. 2). It captures semantics and thus yields a more consistent sparsity allocation decision. \n\n**Q3: Comparisons to recent works**\n\n**A3:** We thank the reviewers for listing the four recent works, and we will cite and discuss them in the camera-ready version. The four works explore different Caching mechanisms within diffusion architectures that accelerate sampling. However, we believe our work **fundamentally differs** from the four works listed. Note that [1] is published after submission, which is impossible be addressed at the moment.\n\n**a.** *Methodology*: The four works mentioned above accelerate diffusion in a fixed configuration (e.g., fixed caching interval and pruning ratio), while SADA is adaptive to different prompts. In addition, to our best knowledge, SADA is the first work that unifies token- and step-wise sparsity by a single criterion from the perspective of the ODE-solver process, achieving a multi-granularity adaptive acceleration strategy. The novelty is strongly supported by Reviewer km7j.\n\n**b.** *Motivation*: SADA formulates the acceleration of the ODE-based generative modeling (e.g., Diffusion, Flow-matching) as a **stability measure** of the denoising trajectory, while the four works focus only on the redundancy of the denoising architecture with relatively weak theoretical justification.\n\n**c.** *Experiment*: We believe the objective of post-training acceleration is to preserve the similarity (faithfulness) between original generated and accelerated samples while maximizing speed. Therefore, we evaluate using LPIPS and FID computed between these samples—unlike previous work, which only compares the FID of accelerated samples against the dataset. As shown in Table 2 of our response to Reviewer km7j, our method significantly outperforms [4] on FLUX.1 in terms of faithfulness at the same speed-up ratio.\n\n**Q4: CLIP/Pick Score**\n\n**A4:** Our objective is to preserve the original generation quality through our sparsity framework—metrics like these do not reflect that goal. For completeness, we have provided the requested metrics, along with generation samples and comparisons, via the link below:\n\nhttps://drive.google.com/file/d/168ovZu9fxcfY5PfE8F5AgkN4la6dvH9f/view?usp=sharing\n\n**Q5: Computational overhead analysis & validation of Theoretical claim**\n\nPlease refer to Q2, Q3 in our response to Reviewer km7j.", "claims_and_evidence": "- **Claim 1**: $ x_0 $-based pruning improves stability and aligns with solvers.\n    \n    *Evidence*: Theoretical analysis (Theorem 3.1) links $ x_0 $ to step stability; experiments show lower LPIPS/FID than $ x_t $-based methods (Table 1).\n    \n    *Problems*: Limited comparison to other $ x_0 $-aligned methods; no ablation on $ x_0 $ vs. $ x_t $.\n    \n- **Claim 2**: Unified sparsity outperforms isolated strategies.\n    \n    *Evidence*: SADA outperforms DeepCache/AdaptiveDiffusion in FID/LPIPS (Table 1).\n    \n    *Problems*: Missing comparisons to recent works[1,2,3,4]\n\n[1] Token-aware and step-aware acceleration for stable diffusion\n\n[2] Cached Adaptive Token Merging: Dynamic Token Reduction and Redundant Computation Elimination in Diffusion Mode\n\n[3] Accelerating diffusion transformers with token-wise feature caching\n\n[4] Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model", "methods_and_evaluation_criteria": "- **Methods**: The $ x_0 $-aligned criterion and reconstruction mechanisms are well-motivated. Adams-Moulton provides a principled ODE-based approximation.\n- **Evaluation**: COCO-2017 benchmarks with standard metrics (LPIPS, FID) are appropriate. But no clipscore or pick-socre reported, The former evaluated the text alignment of images generated by models after acceleration, while the latter assessed aesthetic scores.", "theoretical_claims": "- **Theorem 3.1** (Lindeberg condition): Correct under the assumption of token independence, but real-world spatial correlations in images may affect validity.\n- **Theorem 3.2** (Error bound): Relies on Lipschitz continuity of $ \\epsilon_\\theta $, which is not empirically validated.", "experimental_designs_or_analyses": "- **Strengths**: Broad evaluation across schedulers (DPM++, Euler) and models (SD2, SDXL).\n- **Weaknesses**: \n   + No analysis of computational overhead from the token cache or varying pruning ratios.\n   + No clipscore or pick-socre reported, The former evaluated the text alignment of images generated by models after acceleration, while the latter assessed aesthetic scores.\n   + Missing comparisons to recent works[1,2,3,4]\n   + The acceleration effect obtained is not significant compared to previous work; it's around 1.5x, which is incremental in nature. The ideas presented in this article do not differ significantly from those based on cache methods previously [1,2,3,4], with contributions being incremental as well.\n\n[1] Token-aware and step-aware acceleration for stable diffusion\n\n[2] Cached Adaptive Token Merging: Dynamic Token Reduction and Redundant Computation Elimination in Diffusion Mode\n\n[3] Accelerating diffusion transformers with token-wise feature caching\n\n[4] Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model", "relation_to_broader_scientific_literature": "Aligns with ODE-based solvers (DPM-Solver++) and token reduction (ToMe). Missing discussion of previous similar work[1,2,3,4]\n\n[1] Token-aware and step-aware acceleration for stable diffusion\n\n[2] Cached Adaptive Token Merging: Dynamic Token Reduction and Redundant Computation Elimination in Diffusion Mode\n\n[3] Accelerating diffusion transformers with token-wise feature caching\n\n[4] Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model", "essential_references_not_discussed": "The following references should be discussed because they are similar in method to this paper.\n\n[1] Token-aware and step-aware acceleration for stable diffusion\n\n[2] Cached Adaptive Token Merging: Dynamic Token Reduction and Redundant Computation Elimination in Diffusion Mode\n\n[3] Accelerating diffusion transformers with token-wise feature caching\n\n[4] Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model", "other_strengths_and_weaknesses": "no", "other_comments_or_suggestions": "This paper is poorly written and difficult to read.", "questions_for_authors": "see Weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MohS08bDxU", "reviewer": "Reviewer_km7j", "summary": "The paper proposes SADA (Stability-guided Adaptive Diffusion Acceleration), a method to accelerate diffusion models by jointly optimizing step-wise and token-wise sparsity using a unified criterion based on the denoised latent \\( x_0 \\). Key contributions include: (1) alignment of pruning decisions with \\( x_0 \\)-based solvers for stability, (2) a second-order Adams-Moulton approximation for skipped steps, and (3) a token cache mechanism to mitigate information loss. Experiments on Stable Diffusion 2 and SDXL demonstrate up to 1.52× speedup while maintaining image quality (e.g., LPIPS of 0.118 on SDXL). The method outperforms baselines like DeepCache and AdaptiveDiffusion in metrics such as FID and LPIPS.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's thoughtful feedback and kind support for our work.\n\nBased on the suggestions from Reviewer 4E6r and DBkM, we implement an aggressive version of SADA:\n\n1. Implementing uniform step-wise pruning when the $x_0^t$ trajectory is stable, using Lagrange interpolation.\n\n2. Mitigating degradation by upgrading the Adam-Moulton Approximation from second to third order.\n\nDetailed motivation and formulation are provided in our response to Reviewer 4E6r, and updated results are shown in Table 1.\n\n**Q1: Comparison between $x_0$ and $x_t$-based criterion**\n\n**A1:** We compare our $x_0$ driven paradigm with AdaptiveDiffusion, which leverages the third-order difference of $x_t$ as acceleration criterion. As shown in Table 1, our method consistently delivers superior generation quality—achieving higher PSNR, lower LPIPS and FID—while maintaining a stable speed-up ratio of $\\geq 1.8 \\times$ regardless of model and scheduler. \n\nThe $x_0$ representation is naturally aligned with the final output, capturing essential semantic structures and enabling a more robust criterion than $x_t$. \n\n**Table 1: Comparison between $x_0$ based and $x_t$ based criterion**\n|**Model**|**Scheduler**|**Methods**|**PSNR**|**LPIPS**|**FID**|**Speedup Ratio**|\n|-|-|-|-|-|-|-|\n| SD2|DPM++|AdaptiveDiffusion| 24.30|0.100| 4.35|1.45|\n|||SADA|**26.34**|**0.094**|**4.02**|**1.80**|\n|SD2|Euler|AdaptiveDiffusion|21.90|0.173|7.58|**1.89**|\n|||SADA|**26.25**|**0.100**|**4.26**|1.81|\n|SDXL|DPM++|AdaptiveDiffusion| 26.10| 0.125|4.59|1.65|\n|||SADA| **29.36**| **0.084** | **3.51**| **1.86**|\n|SDXL| Euler| AdaptiveDiffusion|24.33|0.168|6.11|**2.01**|\n|||SADA|**28.97**|**0.093**|**3.76**|1.85|\n\n**Q2 Theoretical claims**\n\n**A2-1 (CLT):** The independence assumption holds for $x_t$ as $\\epsilon_t$ is sampled i.i.d. from Gaussian. For $\\hat{x}_t=\\sqrt{\\bar{\\alpha}_t}\\hat{x}_0^t+\\sqrt{1-\\bar{\\alpha}_t}\\hat{\\epsilon}_t$, we write $\\hat{\\epsilon}_t=\\epsilon_t+(\\hat{\\epsilon}_t-\\epsilon_t)$. The first term is i.i.d. Gaussian with zero mean by Law of Large Number (LLN). For the second, the training objective $E| \\epsilon-\\hat{\\epsilon}_t|^2$ implies $\\hat{\\epsilon}_t\\to E[\\epsilon\\mid x_t,t]$, so $E[\\hat{\\epsilon}_t-\\epsilon_t]\\to 0$, and by LLN, the sample mean $\\overline{\\hat{\\epsilon}_t-\\epsilon_t}\\to 0$.\n\n**A2-2 (Lipschiz):** The Lipschiz continuity of $\\epsilon_\\theta$ is widely assumed by preliminary works such as Adaptive Diffusion and DPM-Solver.\n\n**Q3 Computational overhead**\n\n**a.** *Memory*: For step-wise pruning with third-order Adam-Moulton, after reformulate we only need to store 1 previous $x_0^t$ and 2 previous $dx/dt$ in the cache. For token-wise pruning, we store 1 previous representation $\\mathbf{x}^l_t$ for only transformers with the highest resolution. For example, we observe only a neglectable increase in memory usage in the SD-XL model (from 14981 MB to 15127 MB).\n\n**b.** *Complexity*: All computation in the SADA framework is addition and scaling, $O(N)$. Note that SADA does not include any quadratic complexity computation (e.g. cosine similarity, matrix calculation) as in previous works.\n\n**Q4. Qualatative examples & Fewer step generation**\n\nWe provide the following link for more generation samples and comparison with previous strategy. In Addition, the CLIP score for generation quality is provided. \n\nhttps://drive.google.com/file/d/168ovZu9fxcfY5PfE8F5AgkN4la6dvH9f/view?usp=sharing\n\nWe appreciate for pointing out the better similarity when decreasing sampling steps. We believe the extent accumulation of error decrease when reducing sampling steps. This trend could be clearly found in our ablation table. \n\n**Q5. Comparison with other token-wise sparisty strategies**\n\nTable 2 shows that our method significantly outperforms ToMeSD. DiTFastAttention is limited to traditional Diffusion Transformer because its windowed attention cannot handle mixed-modality inputs (e.g., MM-DiT modules in SD-3 and Flux). In contrast, our approach easily adapts to these architectures, as demonstrated later.\n\n**Table 2: Quantitative results on MS-COCO 2017**\n\n|**Model**|**Scheduler**|**Methods**|**PSNR**|**LPIPS**| **FID**|**Speedup Ratio**|\n|-|-|-|-|-|-|-|\n|SD2|DPM++|ToMeSD|16.29|0.41|13.70|1.10|\n|||SADA|**26.34**|**0.094**|**4.02**|**1.80**|\n\n**Q6. SADA for Flow-matching & DiT Architecture**\n\nUnder the flow matching objective, the model directly predicts the transportation vector field $dx/dt$ between noise and data distributions. Since the denoising trajectory is ODE-based, our criterion effectively measures its stability. Table 3 on Flux (DiT) shows that our method significantly outperforms the most recent work suggested by reviewer DBkM.\n\n**Table 3: Quantitative results on MS-COCO 2017**\n\n|**Model**|**Scheduler**|**Methods**|**PSNR**|**LPIPS**| **FID**|**Speedup Ratio**|\n|-|-|-|-|-|-|-|\n| Flux|Flow-matching|TeaCache|19.14|0.216|4.89|2.00|\n|||SADA|**29.44**| **0.060**|**1.95**|**2.02**|", "claims_and_evidence": "The claims are largely supported by experiments, but some aspects need clarification:  \n- The assertion that \\( x_0 \\)-based pruning is \"more stable\" than \\( x_t \\)-based methods is validated via metrics (Table 1), but direct ablation studies comparing \\( x_0 \\) vs. \\( x_t \\) criteria are missing.", "methods_and_evaluation_criteria": "- **Methods**: Combining step/token pruning via \\( x_0 \\)-alignment is novel and sensible. The Adams-Moulton approximation and token cache are well-motivated.  \n- **Evaluation**: COCO-2017, SD2/SDXL, and standard metrics (LPIPS, FID) are appropriate. However, user studies or qualitative examples (beyond Fig. 5) would strengthen claims about preserved visual details.", "theoretical_claims": "Overall, the theoretical proof is solid. There are two possible concerns:\n- **Theorem 3.1** (global token average): Proof in Appendix A.1 applies Lindeberg-Feller CLT but assumes independent tokens, which diffusion latents may not satisfy?\n- **Theorem 3.2** (error bound): The proof assumes Lipschitz continuity of \\( \\epsilon_\\theta \\), which is standard but not empirically verified. Or it would be better to have some literature support.", "experimental_designs_or_analyses": "- Table 1 shows strong results, but baselines like ToMeSD or concurrent methods (e.g., DiT-FastAttn) are omitted.  \n- The ablation study (Table 2) reports improved quality with fewer steps. The authors should clarify if this stems from their method’s stability or experimental setup.", "relation_to_broader_scientific_literature": "The work builds on diffusion acceleration via step skipping (DPM-Solver++, AdaptiveDiffusion) and token reduction (ToMeSD, DeepCache). It unifies these paradigms, addressing limitations in prior isolated approaches. The \\( x_0 \\)-alignment aligns with modern ODE solvers (Karras et al., 2022), extending their utility to sparsity decisions.", "essential_references_not_discussed": "N/A. However, the following works could be included in the related work section or used as baselines to enhance the quality of the paper, as they also focus on training-free acceleration of diffusion models.\n- Delta-DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformers\n- Faster Diffusion: Rethinking the Role of UNet Encoder in Diffusion Models\n- Cache Me if You Can:  Accelerating Diffusion Models through Block Caching", "other_strengths_and_weaknesses": "- **Strengths**: Solid theoretical proof, Novel unification of step/token pruning, strong empirical results, and practical speedup.  \n- **Weaknesses**: No analysis of computational overhead from the cache mechanism.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Few-Step Improvement: Why does LPIPS improve with fewer steps (Table 2)? Could you provide some insightful explanations?\n2. Baseline Comparison: It would be better to add ToMeSD or concurrent methods such as DiT-FastAttn for comparison.\n3. Some of the latest diffusion models are trained based on flow matching loss, and whether this method is also suitable for such models.\n4. The diffusion model of DiT architecture has also received a lot of attention recently, whether this method is also applicable to this architecture, and if so, increasing the experimental results of this architecture will help to improve the quality of the paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X7pXKpMFxt", "reviewer": "Reviewer_4E6r", "summary": "This paper proposes SADA, a novel paradigm that unifies step-wise and tokenwise sparsity decisions using a shared criterion based on the denoised latent x0. By aligning with modern numerical solvers that rely heavily on x0, SADA offers more stable pruning decisions and preserves important visual details throughout the denoising trajectory. Extensive experiments on SD 2 and SDXL demonstrate that SADA significantly accelerates inference without compromising image quality.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the insightful and constructive comments.\n\n**Q1: Can its speedup be extended to $2 \\times$ or beyond?**\n\n**A1:** Yes, it can. To apply an faster configuration, we leverage the inherent stability of the per‐step data reconstruction $x_0^t$. When the $x_0^t$ trajectory demonstrates high stability (e.g., the second half of Fig.2), we could employ larger step sizes compensated by higher-order approximations. Building on this insight, we implement a uniform step-wise pruning strategy after the stability of the denoising process with Lagrange interpolation for correction.\n\nFor example, consider a 50‐step process. To achieve a step-wise pruning interval of 4 after stable (i.e., compute every 4th step fully and interpolate the skipped steps via Lagrange), we store $\\hat{x}_0^t$ every 4 steps before stabilization. Their indices define the fixed-size set $I$, which is updated dynamically to limit memory usage. For any skipped $t$:\n\n$$\n\\hat{x}_0^{t}\\gets\\sum _{i \\in I}\\prod _{j\\in I\\setminus \\\\{ i \\\\}}\\frac{t-t_j}{t_i-t_j}\\hat{x}_0^{t_i}\n$$\n\nUnder this setting, we yield a $\\geq 1.8 \\times$ speedup regardless of models or solvers. The acceleration would be even more aggressive if further increasing the step size. \nTo balance the degradation, we raise the Adams-Moulton approximation from second to third order, allowing $x_0^t$ to leverage information from the previous three steps (instead of two), thereby improving numerical accuracy and robustness. Our updated result in Table 1 demonstrates the effectiveness of the above improvements. Notably, we achieve a $2.02 \\times$ speedup on the most powerful Flux.1 model with impressive $0.06$ LPIPS and $1.95$ FID.\n\n**Table 1: Quantitative results on MS-COCO 2017**\n|**Model**|**Scheduler**|**Methods**|**PSNR**|**LPIPS**|**FID**|**Speedup Ratio**|\n|-|-|-|-|-|-|-|\n|SD2|DPM++|DeepCache|17.70|0.271|7.83|1.43|\n|||AdaptiveDiffusion|24.30|0.100|4.35|1.45|\n|||SADA|**26.34**|**0.094**|**4.02**|**1.80**|\n|SD2|Euler|DeepCache|18.90|0.239|7.40|1.45|\n|||AdaptiveDiffusion|21.90|0.173| 7.58| **1.89**|\n|||SADA | **26.25**|**0.100**|**4.26**|1.81|\n|SDXL|DPM++| DeepCache|21.30|0.255|8.48|1.74|\n|||AdaptiveDiffusion| 26.10|0.125|4.59|1.65|\n|||SADA| **29.36**| **0.084**|**3.51**|**1.86**|\n|SDXL|Euler|DeepCache|22.00|0.223|7.36|**2.16**|\n|||AdaptiveDiffusion|24.33|0.168|6.11|2.01|\n|||SADA|**28.97**|**0.093**|**3.76**|1.85|\n|Flux|Flow-matching|TeaCache|19.14|0.216|4.89|2.00|\n|||SADA|**29.44**|**0.060**|**1.95**|**2.02**|\n\n**Q2: Ablations on DPM-Solver++**\n\nWe appreciate the reviewers for highlighting the practical importance of DPM++ and its ability to achieve high quality with fewer steps.\n\n**a. SADA performs worse than AdaptiveDiffusion?**\n\nTo counterbalance the aggressive configuration, we increased the order of the Adams-Moulton approximation from second to third order. This enhancement incorporates additional information from the previous denoising trajectory, which in turn improves both accuracy and stability. As shown in Table 1, SADA now significantly outperforms AdaptiveDiffusion when used with DPM++.\n\n**b. Set the sampling step to 20?**\n\nTable 2 presents a comprehensive ablation study across various sampling steps. Our method achieves a 1.5× acceleration in the 25-step scenario with negletable difference. Furthermore, we observe that as the number of inference steps increases, the images generated by DPM++ initially change dramatically before converging when the base step is set to 25. An illustration, along with additional generation examples and comparisons, is available at the following link:\n\nhttps://drive.google.com/file/d/168ovZu9fxcfY5PfE8F5AgkN4la6dvH9f/view?usp=sharing\n\n**Table 2: Ablation studyon sampling steps**\n|**Model**|**Scheduler**|**Steps**|**PSNR**|**LPIPS**|**FID**|**Speedup Ratio**|\n|-|-|-|-|-|-|-|\n|SD-2|DPM++|50|26.34|0.094|4.02|1.80|\n|||25|28.15|0.073|3.13|1.48|\n|||15|29.84|0.072|3.05|1.24|\n||Euler|50|26.25|0.100|4.26|1.81|\n||| 25| 26.83| 0.088|3.87|1.48|\n|||15|29.34|0.076|3.70|1.25|\n|SDXL|DPM++|50|29.36|0.084|3.51|1.86|\n|||25|30.84|0.073|2.80|1.52|\n|||15|31.91|0.073|2.54|1.29|\n||Euler|50|28.97|0.093|3.76|1.85|\n|||25|29.42|0.085|3.13|1.50|\n|||15|31.28|0.084|3.26|1.26|\n\n\n**Q3. SADA for Flow-matching & DiT Architecture**\n\nUnder the flow matching objective, the model directly predicts the transportation vector field $dx/dt$ between noise and data distributions. Since the denoising trajectory is ODE-based, our criterion effectively measures its stability. Table 3 on Flux (DiT) shows that our method significantly outperforms the most recent work suggested by reviewer DBkM.\n\n**Table 3: Quantitative results on MS-COCO 2017**\n\n|**Model**|**Scheduler**|**Methods**|**PSNR**|**LPIPS**| **FID**|**Speedup Ratio**|\n|--|--|--|--|--|--|--|\n| Flux|Flow-matching|TeaCache|19.14|0.216|4.89|2.00|\n|||SADA|**29.44**| **0.060**|**1.95**|**2.02**|", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "This paper proposed a new cache-based accelerating method in the diffusion model area.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "**Strengths:**  \n1. The paper is well-organized and includes comprehensive technical details.\n2. The authors provide sufficient theoretical analysis for the proposed method, including the proof of error bound.\n3. The images generated by the proposed method are consistent with those produced by the original diffusion models.\n\n\n**Weakness:**\n1. This paper only presents experimental results on U-Net-based diffusion models. Since the state-of-the-art image diffusion models now primarily use DiT or MM-DiT architectures, it is essential to demonstrate the effectiveness of SADA on models like PixArt, Flux, or SD 3.  \n2. The evaluation setting for DPM-Solver++ uses 50 sampling steps. However, the main advantage of DPM-Solver is its ability to achieve high-generation quality with fewer sampling steps. Therefore, it would be more reasonable to set the sampling step of DPM-Solver++ to 20.  \n3. The proposed SADA performs worse than AdaptiveDiffusion when evaluated with DPM-Solver++. However, DPM-Solver++ holds greater practical value compared to the Euler solver.  \n4. As shown in the results, the acceleration ratio of SADA is around 1.5×. Can its speedup ratio be extended to 2× or beyond at the cost of some performance?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please see the weakness", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces SADA (Stability-guided Adaptive Diffusion Acceleration), a method designed to accelerate diffusion models by jointly optimizing step-wise and token-wise sparsity through a unified criterion based on the denoised latent variable $x_0$. SADA is evaluated on Stable Diffusion 2 and SDXL, demonstrating up to a 1.5× speedup while preserving image quality.

The review scores are 1–3–5. The main concerns raised by Reviewer DBkM relate to the similarity of the proposed approach to existing work (see references [1]–[4] in the corresponding review). Although these references were unpublished or unavailable at the time of the ICML submission deadline, the authors should address and discuss them in the updated manuscript.

Reviewer 4E6r’s concerns appear to have been addressed. We encourage the authors to reflect these updates in the revised version.
Additionally, the reviewers suggest further improvements, and we recommend that the authors incorporate relevant points from their rebuttal to strengthen the final version of the paper.
Decision: Accept (poster)