ASDSV: Multimodal Generation Made Efficient with Approximate Speculative Diffusion and Speculative Verification
Authors: Kaijun Zhou
Xingyu Yan
Xingda Wei
Xijun Li
Jinyu Gu
Venue: NeurIPS 2025 poster ()
Keywords: Speculative Diffusion
Diffusion model
Multimodel Generation
Inference acceleration
Primary area: infrastructure

Abstract: Diffusion in transformer is central to advances in high-quality multimodal generation 
but suffer from high inference latency due to their iterative nature. 
Inspired by speculative decoding's success in accelerating large language models, 
we propose Approximate Speculative Diffusion with Speculative Verification (ASDSV), 
a novel method to enhance the efficiency of diffusion models. 
Adapting speculative execution to diffusion processes presents unique challenges. 

First, the substantial computational cost of verifying numerous speculative steps 
for continuous, high-dimensional outputs makes traditional full verification prohibitively expensive. 
Second, determining the optimal number of speculative steps $K$ 
involves a trade-off between potential acceleration and verification success rates. 

To address these, ASDSV introduces two key innovations: 
1) A speculative verification technique, which leverages the observed temporal correlation between draft and target model outputs, 
efficiently validates $K$ speculative steps by only checking the alignment of the initial and final states, significantly reducing verification overhead. 
2) A multi-stage speculative strategy that adjusts $K$ according to the denoising phase—employing smaller $K$ during volatile early stages 
and larger $K$ during more stable later stages to optimize the balance between speed and quality. 

We apply ASDSV to state-of-the-art diffusion transformers, 
including Flux.1-dev for image generation and Wan2.1 for video generation. 
Extensive experiments demonstrate that ASDSV achieves up to 1.77$\times$-3.01$\times$ speedup 
in model inference with a minimal 0.3\%-0.4\% drop in VBench score, 
showcasing its effectiveness in accelerating multimodal diffusion models without significant quality degradation. 
The code will be publicly available once the acceptance of the paper.

ASDSV: Multimodal Generation Made Efficient with Approximate Speculative Diffusion and Speculative Verification
Kaijun Zhou, Xingyu Yan, Xingda Wei, Xijun Li, Jinyu Gu* School of Computer Science, Shanghai Jiao Tong University
(a) Flux (text-to- immat)
(b) Wan (text-to-video)
ASDSV
Original
Baseline
Figure 1: Visual comparisons of different methods on various generation tasks. From left to right: Approximate Speculative Diffusion with Speculative Verification (ASDSV), original model, Baseline (TeaCache). The red boxes highlight regions that differ significantly from the original image/video.
Abstract
Diffusion in transformer is central to advances high-quality multimodal genera- tion but suffer from high inference latency due to their iterative nature. Inspired by speculative decoding's success in accelerating large language models, we pro- pose Approximate Speculative Diffussion with Speculative Verification (ASDSV), novel method to enhance the efficiency of diffusion models. Adapting speculative execution diffusion processes presents unique challenges.
First, the substantial computational cost of verifying numerous speculative steps for continuous, high-dimensional outputs makes traditional full verification pro- hibitively expensive. Second, determining the optimal number of speculative steps involves a trade-off between potential acceleration and verification success rates.
To address these, ASDSV introduces two key innovations: 1) A speculative verifi- cation technique, which leverages the observed temporal correlation between draft and target model outputs, efficiently validates K speculative steps by only check- ing the alignment of the initial and final states, significantly reducing verification
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

overhead. 2) A multi-stage speculative strategy that adjusts K according to the denoising fiaase-employin smaller K during volatile early stages and larger K during more stable later stages to optimize the balance between speed and quality. We apply ASDSV to state-of-the-art diffusion transformers, including Flux. 1 1-dev for image generation and Wan2.1 for video generation. Extensive experiments demonstrate that ASDSV achieves up to with a minimal 0.3%-0.4% drop in VBench score, showcasing its effectiveness accelerating multimodal diffusion models without significant quality degradation. The code will be publicly available once the acceptance of the paper.
x speedup in model inference
1 Introduction
Multimodal generation has achieved rapid progress in recent years, with diffusion models emerging as dominant choice for producing high-quality, semantically coherent outputs across modalities such as images, videos, and text [3]. As these models scale in capacity and complexity complexty-exeemplffee
by recent breakthroughs like Sora [], Flux [18), and 1an2.1 37
their generative capabilities have
reached unprecedented levels.
However, the inference latency of diffusion models remains a critical performance challenge due to the inherently iterative nature of the denoising process M. Each inference requíres tens to hundreds of sequential denoising steps and each step is computationally expensive. This challenge is particularly pronounced in high-resolution image synthesis or long video generation, where the cumulative latency poses significant barriers to real-time applications and scalable deployment [41 22]
To address this challenge, previous works have explored three primary directions: distillation, archi- tectural modifications, and caching-base strategies. Distillation methods [29] compress pretrained models into faster variants but demand substantial computational resources for retraining and often de- grade output quality. Architectural optimizations, such as attention compression and sparse attention mechanisms [43] 38) reduce per-step computation but also require retraining and are tightly coupled with model designs, limiting generalizability. Caching-basee
investigate the similarity between adjacent steps in the denoising process and propose skip-stepping strategies to reduce the number of denoising steps. Yet, this aggressive strategy may lead to a noticeable degradation in generation quality, as shown in Figure (differences between Original and Bascline).
approaches [45] 42 24
In language model inference, speculative decoding 19 has emerged as a powerful paradigm for latency reduction. It typically employs a lightweight draft model to predict K tokens sequentially, which are then verified in paraliel by the target model in a single inference batch, achieving speedups without compromising the output results. Inspired by its success, we explore adapting speculative decoding to diffusion models. A key observation motivates this direction: multimodal generation models often have corresponding draft models readily available, either as lightweight variants from the same family (e.g.. Wan2.1-1.3B for Wan2.1-14B (37). Stable- Soable-Diffusio 3.5 medium and large variants [9] ) or through community-driveer model compression efforts (c.g., Flux-lite for Flux. 1-dev (18).
However, adapting this strategy to diffusion models presents unique challenges. First, efficient verification is significant hurdle. The continuous and high-dimensional nature of image and video data demands substantially more computational resources and memory for verification compared to discrete token sampling in language models. This can restrict batch sizes and has limited existing speculative approaches in diffusion (2) to low-resolution tasks such as CIFAR-10 [17. To overcome this, we introduce approximate speculative verification strategy. Instead of verifying all K intermediate outputs from the draft model, our method only checks the initial and final speculative outputs. If both align with the target model's outputs within a defined tolerance, all intermediate steps are accepted. The insight for this efficient verification is that the outputs from draft and target models tend to exhibit similar variations across adjacent denoising steps (detailed in Sed3.I).
Second, determining the optimal number of speculative steps, K, involves a critical trade-off. A larger K' could theoretically yield greater acceleration, but it also increases the likelihood of the draft model's predictions diverging from the target model, leading to higher verification failure rates. To address this, we propose multi-stage speculative strategy that adopts multiple K values based on the characteristic dynamics of diffusion models: early denoising steps exhibit rapid output changes, while later steps show gradual convergence [45] 10). Consequently, our strategy employs smaller

---

K' during the volatile early stages to minimize error accumulation and a larger K in the more stable later stages to maximize performance.
In all, our proposed system, which we term Approximate Speculative Diffusion with Speculative Verification (ASDSV), begins approximate speculative diffusion after letting the target model complete the first N denoising steps. The initial denoising steps exhibit large variations, making them unsuitable for speculative execution. Then, ASDSV enters an interleaved execution pattern: the draft model executes K steps, speculative verification is performed, and the target model executes a single step. This approach effectively reduces total computations while striving to preserve generation quality.
We apply ASDSV to a wide range of multimodal models, including two state-of- -hrte-rr diffusion transformers: Flux. I-vvv [8] (image generation) and Wan2.1 [37] (video generation). Our extensive experiments demonstrate that ASDSV achieves up to 1.77x-3.01 x speedup on model inference speed over the vanilla execution, while only incurring minimal performance drop of 0.3%-0.4% in VBench score.
2 Related Work
2.1 Diffusion Model
Multimodal generation has achieved remarkable progress, with diffusion models continuously break-
ing state-of-the-ar records R [2]. Diffusion models consist of an iterative process of denoising steps,
where inference gradually removes noise from the initial data sample [15] 31 By using attention
mechanisms KE and various optimization algorithm designs [25 26 23 diffusion in transformer
(DiT) (30 models demonstrate excellent performance in text-to-image and text-to-video generation.
State-of-thee art (SOTA) models have evolved from Pixart-Alpha (1.2B) [4] to Flux (12B) [18 in
image generation; from Open-Sora (I.IB) [46] to W (14B) [37] and Step- Video (20B) [2 in video generation.
2.2 Diffusion Model Acceleration
Distillation and Architectural Optimization. Besides distillation methods 32 28
which requires
substantial computational resources to re-train, many acceleration methods focus on optimizing the attention mechanism to reduce the computational cost. DistFastAtten 4 introduces spatial and temporal window attention with residual sharing, while Efficient-vDiT [8 and Sparse VideoGen (38] leverage the sparse properties of 3D full-attention in video generation reduce the number of tokens involved in attention computation. Yet, such architectural optimizations either requires extra training or are tightly coupled with specific model designs.
Cache. Since the outputs of diffusion models after each denoising step exhibit strong similarity, cache-based methods have been proposed to reduce the number of denoising steps 42
45
PAB [45] uses a step-skipping strategy (fixed-step) to skip some of the denoising steps. The blind step-skipping strategy may lead performance degradation. TeaCache (24) designs cache indicator to determine when to reuse the cached results (i.c., the output of the previous step) based on the similarity of the inputs of two adjacent steps, which avoids fixed-step skipping. However, directly reusing the cached outputs would still cause performance degradation, as shown in experiments.
Parallelization. Parallelization methods distributing computations across multiple GPUs, also be effective for accelerating diffusion models (e.g., PipeFusion [1]. DistriFusion (00. Classifier-Free Guidance Parallelism 100) and orthogonal work.
2.3 Speculative Decoding
To reduce the latency of autoregressive language model inference, Speculative Decoding introduces a Draft-then-Vent Verify decoding paradigm [39] 40 19 It needs to train small model (draft model) for large language model (target model). For decoding, it first efficiently generates multiple tokens by the draft model and then verifies all these tokens in parallel (in one batch) using the target model. Thereby, the target model inferences once but generates multiple tokens, which faster than the original autoregressive decoding.
3

---

One prior work, Speculative Diffusion (2. make the first attempt to extend Speculative Decoding to diffusion models. However, is only applied to low-resolution image-generation tasks (32x32) [17]. as the exsiting verification process is too computationally expensive to be efficiently done for high- resolution tasks. Notebly, the target model needs to generate all candidate steps one batch for quickly verifying the draft steps, which is impractical for SOTA multimodal models 3 37 27
3 Method
3.1 Approximate Speculative Diffusion with Speculative Verification
12V craft Hode Draat Made
100 langa
cxrga
0.10
0.25
0100
0075
0.0
0.2
3.50
0000
(a)
(b)
(c)
Figure 2: (a) shows strong temporal correlation between draft and target models the diffusion
process. (b) and (c) shows similar trends patterns of verification results: early steps show gradual
followed by an abrupt divergence the few last steps of the verification process.
Our design of ASDSV is motivated by two observations.
(1) The draft models for SOTA multimodal generation models are off-the-shee shelf, facillitating specula- tive diffusion. First, multimodal generative models typically exhibit large parameter sizes and high computational demands, they are usually released with variants of varying sizes to accommodate diverse hardware capabilities. For examples, 1-1.3B for 1an2.11141 and
also provide a rich pool of draft 3.5's medium and large variants. Second, the resource-constrained environments. models through model compression techniques for
(2) The output trends of draft and target models show temporal correlation. The output trend depicts that the differences between adjacent steps in the denoising process of diffusion model, while difference between two steps defined as L1 loss between the outputs them.
L1 D T = mean( [D - -)
D, and T; denote the predicted outputs from the draft and target models step respectively. As shown in (a), the output trends of draft and target models are close to each other. This refers our key empirical observation: the strong temporal correlation between draft and target models in the diffusion process, Specifically, the magnitude of change of the draft model's output and the target model's output between consecutive steps is highly similar (i.e., ||, D;- [[Z; -T-ill after the first N initial steps. This observed characteristic holds for both text-to-image (T21) and text-to-video (T2V) generation models.
For any adjacent K denoising steps for the draft and target models, we mark their outputs as DN- DN+1, D K, and Tx, TN+1, .... TN+K. k. respectively. Since the output trends of draft and target models are close to each other, if the differences (L1_loss) between DN+1 and TN+1 and between DN+ K and + both smaller than threshold, we can speculatively conclude that all the differences between intermediate DN++ and TN+; are also within the threshold. Thereby, unlike traditional speculative decoding that verifies every output (T'+1, ... TN-K) of the draft model, we propose speculative verification strategy which only verifies the output of the draft model at the beginning and end of the sequence (TN+1 and T TN+K), which significantly reduces the verification cost for diffusion models. Here's 's formalized expression for speculative verification:
DN+I Mg(TN), DN+i M,(DN+i-1), ....., K
TN+1, TN+K & M,, MD(DN+k), Mp and M, has strong temporal correlation If If 1, TN+1) & and I.1 L.1_los( TN+K) S 5, then accept D ,nnt...,,DDk. else reject and generate from T Ty+2

---

Demoising Process
Initial steps
Speculative Diffusion
Speculative Verilication
Target Step and Next round
7
Tx
Ti++
Tire
Target generation
DraAi model gemeration
Dgys
Dusi-
Daee
Dy-1
Dx.
Dyek-1
Prompt: A man takes a picture of an airplane taking off.
Accept verification: Til. (mM,) o04))
Figure 3: Overview of ASDSV compared with vanilla speculative diffusion. After N Mtarget model steps, ASDSV sequentially samples K steps within draft model, and the target model verifies the noises by comparing the first and last step's noises. In contrast, vanilla speculative diffusion verifies the noise of each step.
Workflow. Figure 3 illustrates the workflow of a denoising process with ASDSV. The initial N steps are generated by the target model without any speculative steps, where the input for each step consists of the timestep embedding at the step and the output image of the previous step. After that, the repeated Speculative Diffusion and Speculative Verification process starts.
For each round of Speculative Diffusion, the draft model sequentially samples K steps, where the initial step's input image comes from the target model's output of the previous round; the target model only executes for the first step and the last step (used for verification), where the last step's 's input image comes from the draft model's output (DN+ The following Speculative Verification will calculate two differences between D and +1 and between D +++ and T TN The verification succeeds if both differences are smaller than threshold hyper-parameter, 8). If so, next round of Speculative Diffusion and Speculative Verification starts. Otherwise, the target model generates TN+2 and then new round starts.
Batched Pipeline Optimization. As shown in Figure 3 the target model needs to generate the K step as the last step of the current round's verification process, and the K + step as the first step of subsequent round. If the verification succeeds, TN+K is similar with Dy=k, the target model can generate TN+K++ K+1 based on DN+K. This allows for pipeliming the current round's verification process and the next round's generation process in parallel. Thereby, ASDSV batches the draft sampling results DN+K-1 and + as the input of the target model, which then generates TN4K and K+1 in parallel.
3.2 Multi-Stage Speculative Strategy
also observe that during the denoising process, the differences between adjacent steps are big in the early stages and small in the later stages, which is an intrinsic property of diffusion models [45 If we choose large K value (speculative steps), the verification failure probability will be high in the early stages, leading to more wasted computation. In contrast, choose small K value, the verification failure probability will remain low in different stages, but the acceleration ratio will be low. Therefore, unlike traditional speculative decoding approaches that employ fixed number of speculative steps K throughout the entire decoding process, propose a multi-stage speculative strategy which chooses different K values for different stages during the denoising process.
According to Figure 2 a),
Specifically, ASDSV uses a three-stage strategy. Stage-O: The initial steps is the warmup stage which is used to establish the foundation for high-quality generation. Stage- The subsequent warmup stage employs a smaller 7/ as value, to preserve a high success rate of verification during the early stage. Stage-2: The main generation stage employs larger 7/ K value. to achieve higher acceleration ratio. Such a strategy can achieve an optimal balance between generation speed and generation quality.
choose the hyperparameters based on the step-metric results in both image and video scenarios. We illustrate with a red vertical line in Figure (b) and (c), the verification error curve exhibits sharp
5

---

increase at the 9-th step. This inflection point shows that the draft model's outputs are accumulatiig errors and starting to diverge significantly from the target model's. Therefore, it represents the appropriate moment to halt the drafting phase. So we set 72 as for Stage-2 to achieve both high success rate of verification and high acceleration ratio. More details about other stages are provided in the Appendix A.3
3.3 Walltime Improvement
We denote the draft model as M, and the target model as Mp. Let ? and 72 be the number of speculative steps in the Stage-1 and Stage-2, respectively. The running time of one step of M, is TM, and that of Mp is MM Let c, the cost coefficient, be the ratio of TM, to TM,, The number of steps in Stage-O is denoted as N. Assuming Stage-1 and Stage-2 respectively complete aı and ag speculative rounds and all verifications are successful, ASDSV can achieve a maximum speedup as follows.
N + 0171 + 0222 Total steps
Vanilla Diffusion: Total steps x TM,
(2)
ASDSV: NTM, + Ta,, (Total steps N) TM, (1 a1 + 02)
Note that Mp generates the last step of the current round and the first step of the next round in parallel
for each round, except for the first step of the first round. Therefore, in ASDSV, the cost for Mp is NTM, + Tas, (1 + a1 + a2). The theoretical upper bound of speedup is:
N + 01 + 02?2
N a+ +1) + 02 (2 + 1)
We also provide the expected speedup of ASDSV and its proof in Appendix A.5
V1 3
2.32x
Stage-0: Initial N steps
V/2
Stage-1: Mg generation
Stage-2: Mg generation
== V/2 -
1.88x
Mp generation
Vanilla
Wall time (seconds)
+
Figure 4: A simplified trace diagram to show ASDSV under different settings on Wan2.1 using 50 steps. The purple blocks and pink blocks represent steps where results are generated by the draft model in Stage-1 and Stage-2, respectively, N is set to 10% of total steps. The first row shows ASDSV with 71 = 3 and 7/2 = 9, the second row shows ASDSV with 71 = 2 and 7/2 = 5, and the third row shows vanilla diffusion.
Figure 4 presents the concrete theoretical upper bound of speedup of ASDSV under different settings. However, since there can be verification failures in the speculative steps, the theoretical upper bound is hard to achieve. Furthermore, in our experiments, was 0.60 for text-to-image models and 0.19 for text-to-video models. In contrast, the cost coefficient e much smaller large language models (often less than 0.05 (19). As this work focuses making speculative diffusion effective in SOTA multimodal generation models, the exploration of optimal draft model orthogonal.
Experiments
4.1 Settings
Models and Baselines. We evaluate ASDSV on two representative diffusion models: Flux. 1-dev [8 for image generation tasks and Wan2.1 (37 for video generation tasks. We compare ASDSV with two statc-of-the-art studies on accelerating diffusion inference: Teacache [24 (cache-based method)

---

for both image and video generation, and Sparse VideoGen [3 (architectural optimization) for video
generation acceleration.
Metrics and Datasets. We evaluate ASDSV and other baselines on two aspects: generation quality and generation efficiency. For image generation with Flux. 1-dev, evaluate the generation quality
using Frechet Inception Distance (FID) [4), Inception Score (IS)
[33 Learned Perceptual Image
Patch Similarity (LPIPS) 44,, Peak Signal-to-Noise Ratio (PSNR) 335 and CLIP score [3]. Each
comparison method generates 10k images using the COCO Captions 2014 dataset [3]. FID, LPIPS and PSNR are metries for similarity assessment, i.e., how different the generated images are from the ground truth (the output of the original model), CLIP score measures the semantic alignment between the input prompt and the generated images and IS measures the diversity of the generated images.
For video generation with Wan2.1, 1, like prior studies [2]], we evaluate the generation quality using VBench Score [16] evaluate the visual quality of the generated videos, LPIPS, PSNR and Structural Similarity Index Measure (SSIM) [36] to evaluate the similarity between the generated videos and the ground truth. Each comparison method generates 2k videos using the VBench dataset [16].
For generation efficiency evaluation across both models, we use Floating Point Operations (FLOPs) and inference latency as metrics.
Implementation Details. We implement two variants of our method: ASDSV- and ASDSV. slow. The ASDSV-slow variant performs the speculative verification and samller K for the Stage-1 speculative step to achieve better visual quality. The ASDSV-fast variant skips the verification entirely, assuming the draft samples pass the verification, and uses the same K for the Stage-1 and Stage-2 speculative steps to achieve higher efficiency, which is a more aggressive speculative diffusion strategy.
For both image and video generation, we set the total number of denoising steps to 50. For text- to-image generation, we use FluX-SVDIII as the draft model. For ASDSV-slow, we set 71 = 3, 02 9. and warmup ratio to 15% For ASDSV-fat fast, we use 31 F2 9. We set initial steps N to 8% of total steps for both variants and verification threshold (6) of 0.02. For text-to-video generation, we use Wan2. 1-..3B as the draft model. For ASDSV slow, we set 71 2, 722 9, and warmup ratio to 25%. For ffas, we use 71 72 = 9. We set initial steps N to 10% of total steps for the fast variant and 15% for the slow variant with (5) 0.2.
As for the baselines, we use the same settings as in the original papers: Teacache 24 threshold is 0.25 for slow and 0.6 for fast in text-to-image generation, and 0. 14 for slow and 0.2 for fast in text-to-video generation. Sparse VideoGen (38) is used with sparsity 0.10 for fast and 0.20 for slow, without enabling custom kernel and FP8 quantization. We measure the latency per sample on single NVIDIA A800 GPU using Pytorch 2.6.0 and CUDA 12.4.
4.2 Main Results
Comparison with Draft Models. Besides the above-mentioned baselines, we also compare ASDSV with using the different draft models for multimodal generation, which can be regarded as distillation- based methods. As provided in the Appendix [ the result is that ASDSV consistently outperforms all draft models, which also shows the effectiveness and broad applicability of ASDSV across diverse compression techniques.
Results of Evaluation Metrics. Table D and 2 present the quantitative evaluation of efficiency and visual quality using the evaluation metrics described in Section 4.1] Compared to both cache- based and architectural optimization methods, ASDSV delivers better visual quality and enhanced generation efficiency across different modalities, base models, resolutions and video length.
In Table I
(text-to-image) ASDSV slow achieves the best visual similarity metrics (FID, LPIPS, PSNR) and attaining nearly 1.3x x speedup. Although ASDSV fast achieves the lowest FLOPs count, doesn't 't attain the minimum latency. This discrepancy arises because the quantized draft model doesn't 't reach optimal acceleration on A800 GPUs. On hardware supporting efficient low-precision computation (e.g., 50-series GPUs), our approach would likely achieve ideal acceleration, fully realizing its theoretical efficiency advantages.
7

---

Table 1: Efficiency and visual quality comparison of ASDSV and other baselines on image generation.
Flux-SVD the draft model in ASDSV.
Efficiency
Visual Quality
Method
FLOPs (T)
Latency (8)
Speedup
FID
CLIPT
LPIPS
IS
PSNR
FLUX.I-dey (512x512)
FLUX.I-dev 50)
1082.5
9.18
31.14
26.32
TeaCache-fast
330.4
4.25
2.16x
8.16
31.34
0.30
25.66
28.89
TeaCache-slow
566.8
5.73
1.6x
4.33
31.20
0.14
26.7
31.67
ASDSV-fast
206.3
6,25
1.47x
4.11
31.15
26.66
32.95
ASDSV-slow
292.3
7.3
1.26x
3.90
31.15
0.10
26.52
33.14
Despite not achieving the highest IS and CLIP scores, ASDSV maintains scores closest to those of the original model, which aligns with our primary goal of achieving higher fidelity to the original model's s output as the ground truth.
Table 2: Efficiency and visual quality comparison of ASDSV and other baselines on video generation.
Efficiency
Visual Quality
Method
FLOPs (P) +
Latency (s) 4
Speedup
VBench f
LPIPS +
PNSR
SSIM
Wan2.1 (81 frames, x 832x480)
Wan2.1 (T' 50)
168.1
924
1x
82.69
Sparse-fast
138.4
761
1,21x
78.97
0.75
8.6
0.27
Sparse-slow
153.1
789
1.17x
81.91
0.70
0.30
TeaCache-faid
84.2
462
81.83
0.45
14.2
0.49
TeaCache-slow
114.4
628
1.47x
82.17
0.37
15.75
0.55
ASDSV-fast
52.6
307
3.01 X
82.36
0,41
14.89
0.51
ASDSV-slow
74.4
521
1.77x
82.29
0.33
16.58
0.58
In Table 2 (text-to-video), ASDSV fast achieves the highest 3.01 x speedup with better VBench score compared TeaCache-slow, Additionally, ASDSV-slow achieves the most faithful reproduction (0.3%-0.49) VBench score degradation), along with higher acceleration 1.77 x than TeaCache-slow. These results demonstrate that ASDSV not only preserves the better enerative performance of the models but also achieves greater acceleration than existing methods especially in text-to-video generation.
Visualization. Figure presents the images and the videos (the first frame and last frame of the video) generated by ASDSV compared with those by Teacache and the vanilla. Experimental results show that ASDSV not only produces higher visual quality, but also achieves better alignment with the ground truth (original images and videos). Specifically, our method better preserves the color fidelity, positioning and details of characters compared with Teacache, Additional visual comparisons can be found in the Appendix
4.3 Ablation Study
Verification Strategy Comparison. We configure ASDSV-medium with = 3 and = 9, using a warmup ratio of 20%. As shown Figure we design three verification schemes to compare the trade-off between efficiency and quality. Our method offers multiple configurable variants, allowing users to select the optimal configuration based on their specific requirements.

---

16.8
0.65
ASDSV
0.57
16.0
ASDEV
Tachachh
TaaChhh
Quality (I-LPIPS) 1
15.2
Quality (SSIM)
0.51
Quality (PSNR)
0.55
0.300
Sparse
0.300
Sparss
0.275
0.275
8.8
0.250
800
Latency(s)
Latency(s)
Latency(s)
Figure 5: Quality-latency comparison of text-to-video diffusion models. The proposed ASDSV method demonstrates superior visual fidelity while maintaining lower inference latency compared to TeaCache [4] and Sparse VideoGen evaluated under the Wan2.1 model.
(a) prompt: A boat sailing leisurely along the Seine River with the Eiffel Tower in background, surreal- ism style.
(b) prompt: couple in formal evening going home get caught in heavy downpour with umbrel- las, zoom in.
Figure 6: Visual comparisons of different verification strategies on text-to-video generation. From left to right: ASDSV-fast ASDSV-medium) and ASDSV-slow Each column presents the first and last frames of the generated video.
Performance at different Resolution and Length. Figure illusstaae that ASDSV maintains ro- bust acceleration performance across varying video lengths and resolutions. This sustained efficiency highlights ASDSV's 's capacity to expedite sampling for extended and high-fideling video content, thereby addressing practical and effective requirements more complex applications.
388.00
1.79k
3.06x
2065.00
71X
2.92x
3352.00
1.70X
2,91X
1505.00 1.19x
1.46x
Fime (swcordd))
Three concont
1344.00
232,00
1205.00
1965.00
Trow
127,00
200.00
1153.00 00
O4p/ral
Origal
AGD AN
015 AG0IAS ON POT
0r mmm 1633V-S0w AMDPVFMM
480P, 41 frames
480P, 141 frames
(c) 720P, 81 frames
(d) 1024x 1024, 1 frame
Figure 7: Inference efficiency at different video lengths and resolutions. (a), (b), and (c) show
different resolutions and video lengths using Wan2.1, while (d) corresponds to different resolutions using Flux.1 1-dev.
Sensitivity Analysis and Tuning Process for 7. Both 71 7/ depend on the temporal correlation between draft and target model on different stages, so ASDSV is sensitive to these parameters. Nevertheless, since the temporal correlation can be profiled offline, we can systematically search for the best candidates for these hyperparameters for each model offline. We generated 200 videos per setting for this specific study on Wan2.1 model, following are the detailed analysis and tuning methods of hyperparameters %

---

Analysis of 71. First, we use brute force to search `/1 because the space is small: it can only be an integer in (1,5). Below is the search space of Wan2.1, We only present one model result for simplicity. other models are the same.
Table 3: Ablation study on the choice of speculative step 71 in Stage-1.
LPIPS|
PSNRI
SSIMI
Speedup
0.23
18.57
0.61
1.70x
3 0.30
16.80
0.59
1.71x
0.30
16.76
0.59
1.72x
Analysis of 7/2- Because the search space of 12 is larger than "/ (an integer selected from range (1,15)), we developed a two-step iterative-guided search process for "/2 to balance speed-fidelifg trade-off: (1) Profile Model Dynamics: We conduct small-scale sampling run profile the model's behavior, generating plots similar to Figure 2
This allows us to observe the output change dynamics and step-wise verification loss, which are crucial for setting the "/2 and decide the verfication threshold. (2) Iteratively Tune %2 and 8: We start with a candidate "/2 based on the theoretical speedup ceiling for our target. Next, using the verification loss curve (like Figure (c))., we set a corresponding threshold ò above the observed loss at the 72-th step. We then incrementally increase "/2 and adjust 6, evaluating the trade-off on validation set until the desired speedup met with minimal quality impact. The following results show typical search results on Wan2. model, others are the same:
Table 4: Ablation study on the choice of speculative step "/2 and verification threshold ō in Stage-2.
72
0
LPIPS
PSNR
SSIM
Speedup
3
0.1
0.24
18.89
0.60
1.2x
9
0.2
0.23
18.57
0.61
1.7x
12 80
0.25
18.47
0.60
3.1x
The baseline for these comparisons is the original model, corresponding to the setting "1 = 0, 72 0. The setting of 5 = inf represents an "always-accept" strategy that effectively bypasses the verification stage to achieve maximum acceleration, corresponding to ASDSV-fast variant in Figure []] We have summarized the tuning process described above into an automated method, which is detailed in Appendix B
Quality-Efficiene Tradeoff. Figure 5 shows the trade-off between the quality and the efficiency of our method compared with other baselines. Notably, our method achieves the best scores not only on the reference-free VBench, but also on similarity metrics. This performance stems from the fact that ASDSV most closely aligns with the outputs of the original model, resulting in quality scores that are nearly indistinguishable from those of SOTA diffusion models.
5 Conclusion and Future Work
This paper introduces Approximate Speculative Diffusion with Speculative Verification (ASDSV), a novel speculative decoding method for multimodal generation. Different from traditional speculative decoding, ASDSV employs speculative verification strategy specifically for diffusion models to minimize verification cost, by leveraging the temporal correlation between draft and target models. Meanwhile, we propose a multi-stage speculative strategy based on the dynamics of diffusion process to to balance the trade-off between generation speedup and performance. Experiments demonstrate that ASDSV achieves significant inference acceleration while maintaining high similarity and robust performance in both video and image generation models.
Limitations and Future Work. First, ASDSV performance depends on the cost coefficient between models, with limited gains when target models are already optimized or draft models lack efficiency. Second, our current verification failure handling discards all subsequent samples. Advanced recovery strategies could potentially enhance performance, such as employing binary search techniques to efficiently identify the last acceptable draft sample. Third, integration with parallelism strategies requires further research to maximize performance in distributed environments.
10

---

Acknowledgments
We sincerely thank the anonymous reviewers, whose reviews, feedback, and suggestions have signifi- cantly strengthened our work. This research was supported in part by New Generation Information Technology Program from Shanghai Committee of Science and Technology (NO.2551 1104100), National Natural Science Foundation of China (No. 62432010), and the Fundamental Research Funds for the Central Universities.
References
[1] Shubham Agarwal, Subrata Mitra, Sarthak Chakraborty, Srikrishna Karanam, Koyel Mukherjee, and Shiv Kumar Saini. Approximate caching for efficiently serving text-to-image diffusion models. Proceedings of the 2 USENIX Symposium on Networked Systems Design and Implementation, NSDI'24, USA, 2024. USENIX Association. ISBN 978- 978-1-939133-39-7
[2] Valentin De Bortoli, Alexandre Galashov, Arthur Gretton, and Arnaud Doucet. Accelerated diffusion models via speculative sampling, 2025.
[3] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng. Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL hRtLhhtss////eeaaa com/research/ hoorddsimmultors
dedeoogeneratton--i mwweetiiioooodmll---ee
[4] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis, 2023. URL hatp::://aaxivvorpppp /bs/ $310.00423
[5] Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitiick. Microsoft coco captions: Data collection and evaluation server, 2015. URL https://arxii.... org 1504. 00325
[6] Javier Martín Daniel Verdú. Flux. I lite: Distilling fluxl.des ldde for efficient text-to-image genera- tion. 2024.
[7] Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis, 2021. URL
[8] Hangliang Ding, Dacheng Li, Runlong Su, Peiyuan Zhang, Zhijie Deng, Ion Stoica, and Hao Zhang. Efficient-vdit: Efficient video diffusion transformers with attention tile, 2025. URL
https://arrae org/ abs//502..61.
[9] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis, 2024. http::tt: wwweeoorgggaooo /ba/ 2403. .32066
[10] Jiarui Fang, Jinzhe Pan, Xibo Sun, Aoyu Li, and Jiannan Wang. xdit: an inference engine for diffusion transformers (dits) with massive parallelism. arXiv preprint arXiv:2411.0173: 2024.
[11] Jiarui Fang, Jinzhe Pan, Jiannan Wang, Aoyu Li, and Xibo Sun. Pipefusion: Patch-level pipeline parallelism for diffusion transformers inference, 2024. ttt.
14430
[12] Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image dif- fusion models without specific tuning, 2024. URL https:t//rxiii...t/t http://aaaxii...r/aa///0//// 2ttp://227700425555 04725
[13] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning, 2022. URIp setttps://arree //arxiv. org/ abs/ 2104, 2104.08718
11

---

[14] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler. and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, page 6629-660 6640, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
[15] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato. R. Hadsell, M.F. Balcan, and H. Lin, editors. Advances in Neural Information Processing Systems, volume 33, pages 6840-6851, Curran Associates, 2020. URL http::t:tp/p/ceeeeeeeee neurips. cc/ hecepaper_ffite wateor_filleeppppeeee htt.le//pppper222///11111 .pf
[16] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang. Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Vbench: Comprchensive benchmark suite for video generative models, 2023. URL htttp::/arxxv..rgg/ 2ttp:/211..179999 17982
[17] Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical report, 2009.
[18] Black Forest Labs. Flux.
htt.:tthhb..omm/aaaaaee www.oo//ooaaooc.o...eo......aaee booeetffssckf-st-.ooo.oo labs 2024.
[19] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding, 2023. htt.Attppee//arr....rr////////// 17192
[20] Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang. Han Cai, Junjie Bai, Yangqing Jia, Ming- Yu Liu, Kai Li, and Song Han. Distrifusion: Distributed parallel inference for high-resolution diffusion models, 2024. https://:
org/ //////b///////// 22022 19481
[21] Muyang Li Yujun Lin Zhekai Zhang*, Tianle Cai, Xiuyu Li. Junxian Guo, Enze Xie, Chenlin Meng, Jun- Yan Zhu, and Song Han. Svdquant: Absorbing outliers by low-rank components for 4-bit diffusion models. In The Thirleenth International Conference on Learning Representations, 2025.
[22] Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Snapfusion: Text-to-image diffusion model on mobile devices within two seconds. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 20662-20678 Curran Associates, Inc., 2023. URL.atapss./prrcceednggg wwwrips.eiippppee. wwtps//wwww..p..p.pppppee.eeee files/paper/ pdf
[23] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling, 2023. URL https::/arxii..orrrg wrggg/bs///1///00 02747
[24] Feng Liu, Shiwei Zhang, Xiaofeng Wang, Yujie Wei, Haonan Qiu, Yuzhong Zhao, Yingya Zhang, Qixiang Ye, and Fang Wan. Timestep embedding tells: It's time to cache for video diffusion model. arXiv preprint arXiv:2411. 19108, 2024.
[25] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan LI, and Jun Zhu. Dpm- solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 5775-5788 Curran Associates, patpp//pp2//iiaap// #Boo44cccessaabaaacccettrreeeeep Poree//eeeeeepeeeeeooooeeeeeeeee pdf
Inc., 2022. URL /ttp/tttss// wwwpeediiiiinnnnnnunneeeeeeeeeee neurips. cc/ wwtpsser.pppppppeeeeeeee htppp/aper__ffa fiwes/leeppppp patpe__iieespppprr/22//////1//ee
[26] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models, 2023. URL httttttpps//artttee org/ wtt.a///1....0
[27] Guoqing Ma, Haoyang Huang, Kun Yan, Liangyu Chen, Nan Duan, Shengming Yin, Changyi Wan, Ranchen Ming. Xiaoniu Song, Xing Chen, Yu Zhou, Deshan Sun, Deyu Zhou, Jian Zhou, Kaijun Tan, Kang An, Mei Chen, Wei Ji, Qiling Wu, Wen Sun, Xin Han, Yanan Wei, Zheng Ge, Aojie Li, Bin Wang, Bizhu Huang. Bo Wang. Brian Li, Changxing Miao, Chen Xu, Chenfei Wu, Chenguang Yu, Dapeng Shi, Dingyuan Hu, Enle Liu, Gang Yu, Ge Yang, Guanzhe Huang, Gulin Yan, Haiyang Feng, Hao Nie, Haonan Jia, Hanpeng Hu, Hanqi Chen, Haolong Yan, Heng
12

---

Wang, Hongcheng Guo, Huilin Xiong, Huixin Xiong, Jiahao Gong, Jianchang Wu, Jiaoren Wu, Jie Wu, Jie Yang, Jiashuai Liu, Jiashuo Li, Jingyang Zhang, Junjing Guo, Junzhe Lin, Kaixiang Li, Lei Liu, Lei Xia, Liang Zhao, Liguo Tan, Liwen Huang. Liying Shi, Ming Li, Mingliang Li, Muhua Cheng, Na Wang. Qiaohui Chen, Qinglin He, Qiuyan Liang, Quan Sun, Ran Sun, Rui Wang, Shaoliang Pang, Shiliang Yang, Sitong Liu, Siqi Liu, Shuli Gao, Tiancheng Cao, Tianyu Wang. Weipeng Ming, Wenqing He, Xu Zhao, Xuelin Zhang, Xianfang Zeng. Xiaojia Liu, Xuan Yang. Yaqi Dai, Yanbo Yu, Yang Li, Yineng Deng, Yingming Wang. Yilei Wang, Yuanwei Lu, Yu Chen, Yu Luo, Yuchu Luo, Yuhe Yin, Yuheng Feng, Yuxiang Yang. Zecheng Tang, Zekai Zhang, Zidong Yang. Binxing Jiao, Jiansheng Chen, Jing Li, Shuchang Zhou, Xiangyu Zhang. Xinhao Zhang, Yibo Zhu, Heung- Yeung Shum, and Daxin Jiang. Step-video- (2v technical report: The practice, challenges, and future of video foundation model, 2025. URL https://arxiv.. org/ abs 2abs/22502.. 10248
[28] Xinyin Ma, Gongfan Fang, Michael Bi Mi, and Xinchao Wang. Leanengg---nngtt--cccceeee Accelerating diffusion transformer via layer caching, 2024. htt.itttss//rziw.or////aa
[29] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik P. Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models, 2023. URL. https
//arxiv. org/ www.or/g//////////////.//ee.ee// 2210.2220.... 2210.0314220222
[30] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4195-4205, October 2023.
[31] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[32] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models, 2022. URL httt::tts:://rrrt htttp:://arrtv..rr//bb//// 00512.
[33] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016. URL pproceeeeoo wwwpriiiininnnnnneeeeeeeeeeeeeee http:pape_fflle/pppeeeeeeee pdf
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, ed- itors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, wwtpp//s..s....pppppppaapeeeeeee
Inc., 2017. URL httpp:ttttts:: neurips. cc/ pape/rpperrrrr httplappapea/a//lllllllllaaeeeea wtt.ss//ppper///////iL.. wttprr/sssspppppppeteeeee/e//pee
pddf
[35] Zhou Wang and A.C. Bovik. A universal image quality index. IEEE Signal Processing Letters, 93):811-84 2002. doi: 10.1 101119999999922.
[36] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE Transactions on Image Processing, 1 1344):600-612, 2004. doi: 10. 109/T1P.2003..00 001.P.22000.11988886 819861.
[37] WanTeam, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang. Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang. Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang, Tianyi Gui, Tingyu Weng. Tong Shen, Wei Lin, Wei Wang, Wei Wang. Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulín Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv: 2503.2230000314 2025.
13

---

[38] Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li. Jianfei Chen, lon Stoica, Kurt Keutzer, and Song Han. Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity, 2025. URL
https://arxiv.. wrg. Sawlg/abs/25000 abtpabb....001111 01776
[39] Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation, 2023. URL butps://arxxr org/ abs/ /203, 16487
[40] Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding, 2024. URL htthtttttt//attt www. abs/ 2211//220100111
[41] Yuchen Xia, Divyam Sharma, Yichao Yuan, Souvik Kundu, and Nishil Talati. Modm: Efficient serving for image generation via mixture-of-diffusion models, 2025. URL https://axxiv org/ hts/abbs25003.. 11972
[42] Mengwei Xu, Mengze Zhu, Yunxin Liu, Felix Xiaozhu Lin, and Xuanzhe Liu. Deepcache: Principled cache for mobile deep vision. In Proceedings of the 24th Annual International Conference Mobile Computing and Networking, MobiCom '18, page 129-144, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450359030. doi: 10.1145/ 3241539.3241563, URL httph://dooi orgooo////// 1145/324153 3241563
[43] Zhihang Yuan, Hanling Zhang, Lu Pu, Xuefei Ning, Linfeng Zhang, Tianchen Zhao, Shengen Yan, Guohao Dai, and Yu Wang. DiTFastattn: Attention compression for diffusion transformer models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL htt.thtst///peeeviiiiiiiie wwt.:eee/oooooo net/forumiid=5 11Hppk0y3t
[44] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 586-595, 2018. doi: 10.1109/VVP.2208..0000
[45] Xuanlei Zhao, Xiaolong Jin, Kai Wang, and Yang You. Real-time video generation with pyramid attention broadcast, 2025. URL
12588
[46] Zangwei Zheng, Xiangyu Peng, Tianji Yang. Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all. arXiv preprint arXiv:2412.204000 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
They are properly stated.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
They are properly stated in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
They are properly stated in Section 3.3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
They are included in
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
Codes provided in supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
htapit///uiidss//deeubmmmm cessionnoiccyl for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//guiii wwwiiicgguie///eeeeeee CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
They are properly stated in Section 4.1 and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We do not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: They are properly stated in Section 4.1 implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:::////uis./eeeeeeeee www.eerris.ccp.uuuuieee
Answer:
[Yes]
Justification: conforms in every aspect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Discussed in Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
18

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We have models/datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Credits for all assets are well given.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswppteeeee wwworeewwthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
19

---

Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subject.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subject.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
20

---

Justification:
The LLM is only used for writing, editing, It is not used as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy hattps://neuris.. 220ps//ee///00ffeeeee//22/////// for what should or should be described.
21

---

A Additional Experiments
A.1 Results of Draft Model Comparison
In autoregressive language models, draft models (M₂) are typically selected from existing off-the- shelf smaller transformers with the same architecture as the target model (Mp). For example, T5-XXL (IIB) as M, and T5-large (800M) Mg. Following this practice, evaluate the performance of different draft models on text-to-image generation. Flux.1-der is the state-of ohettrr text-to-image model, and Flux-lite (distilled model) and Flux-SVD (quantized model) are two efficient draft variants of I-de..
Table illustrates that ASDSV consistently outperforms both draft models, which demonstrates the effectiveness and broad applicability of ASDSV across diverse compression techniques. The quantized model Flux-SVD shows the best overall performance and is chosen as the draft model for 1-dev in our experiments. FID, LPIPS and PSNR are reference- the similarity between generated results and Flux. 1-dev original outputs. CLIP score measures the semantic alignment between the input prompt and the generated images and IS measures the diversity of the generated images.
metrics for measuring
Table Performance of FLUX.1 FLUX.I-dev compared with different draft models. ASDSV w/ Flux-lite and ASDSV w/ Flux-SVD are ASDSV that use Flux.I-dev as the target model and Flux-lite and Flux-SVD as draft model, respectively.
Model
FLUX.1-dev
Score
FID
CLIP t
LPIPS
IS
PSNR
Flux.
31.14
26.32
Flux-lite
9.51
31.07
0.40
25.59
29.73
Flux-SVD
5.52
31.18
0.19
26.26
30.53
VSDSV w/ Flux-lite
6.62
31.20
0.22
25.68
30.49
ASDSV w/ Flux-SVD
4.97
31.24
0.11
26.36
32.69
Additionally, improved the draft model baseline by combined the Stage-0 into the draft model diffusion process. ASDSV-slow and ASDSV-fax are the two variants of ASDSV with different settings, detailed in Experiments Settings of the main paper. Specifically, we use the target model to generate the first N = 5 steps as the draft model's initial steps (the same as the ASDSV-fast), and then use the draft model generate the remaining steps. The evaluation metric results are shown in Table 6 The visual results are shown in Figure g ASDSV outperforms the Improved-Draft baseline in both visual quality metric (VBench score) and similarity metries (LPIPS, PSNR, and SSIM).
Table Efficiency and visual quality comparison of ASDSV and improved draft model baseline on text-to-video generation.
Efficiency
Visual Quality
Method
FLOPs (P)
Latency (s) J Speedup
VBench
LPIPS +
PNSR
SSIM
Improved-Draft
28.0
256
3.67x
82.23
0.42
14.70
0.50
ASDSV-fast
52.6
307
3.01x
82.36
0.41
14.89
0.51
ASDSV-slow
74.4
521
1.77x
82.29
0.33
16.58
0.58
A.2 Multi-Seed Stability Analysis
Our method is insensitive the random seed. This is because the seed in the diffusion process only affects the initial noise, whereas the efficacy of ASDSV depends on the temporal correlation between the draft and target models (as shown in Figure
We conducted new set focused experiments on three additional seeds (3461, 772190542, 1 1182700022244990399 for our main comparisons (Original vs. ASDSV). For this analysis,
22

---

SDSV-slow
Original
Improved-
Figure 8: Visual comparisons of improved draft model baseline text-to-video generation.
generated 2000 images per seed for the Text-to-lmage task and 200 videos per seed for the Text- to Video task. Performance was consistent across all seeds (ASDSV achieves better performance than other acceleration methods). The standard deviations over multiple seeds results on Text-to- Image(T2I) over are shown in Table 7
Table Standard deviations over multiple seeds for T21 generation.
Images Metric
Ratio (Original ASDSVV
CLIP
100.04% = 0.10% 103.84% + 3.7%
Since most of the Text-to- Video(T2V) metrics are reference-based, we cannot calculate relative ratio (original/accelerated method). Therefore, we used comparison between the accelerated methods (ASDSV vs. Teacache) shown in Table R and ASDSV is about 1.5x faster than Teacache.
Table 8: Standard deviations over multiple seeds for T2V generation.
Videos Metric
Ratio (ASDSV Teaccche)
LPIPS.
88.02% 2.85%
PSNR SSIM
106.10% + 1.25% 104.95% + 1.84%
A.3 Analysis of Multi-Stage Speculative Strategy
ASDSV employs a multi-stage speculative strategy to balance the trade-off between generation speedup and performance: first N initial steps (Stage-O) combined with the subsequent Stage-1 serve the warmup period for the diffusion process, establishing the foundation for high-quality generation, followed by the main generation stage as Stage-2. A key characteristic of the warmup period is the relatively large variation between outputs of adjacent steps, unlike the highly similar outputs observed between consecutive steps in Stage-2. To ensure higher quality model outputs, modifications to the architecture are typically avoided during the warmup period.
We evaluated two warmup approaches: the 2 steps method and a more conservative fixed ratio of 20% of total diffusion steps. After completing their respective warmup phases, they employ the same speculative strategies as ASDSV in Stage-2. Our experimental results in visual difference. Configuration (a) with minimal warmup steps significantly reduces target model invocations, improving speed but compromising generation quality. In contrast, configuration (c) with larger warmup steps maintains higher quality output but with reduced acceleration. To balance this trade-off, we divided the warmup phase into two distinct components (Stage-O and Stage- 1): first allowing the target model establish critical structural elements, then employing small speculative steps to accelerate the remaining warmup procedure. As demonstrated in configuration (d), which
a clear
23

---

uses 2 steps for Stage-0 followed by speculative steps with K for Stage-1, I, then employing the
same 9 speculative steps for Stage-2 as configurations (a) and (c). This multi-stage approach achieves nearly 2.1 x acceleration over the vanilla diffusion process while preserving better visual quality compared to configuration (a).
Speedup: 2.4x
Speedup: 1x
Speedup: 1.9x
Speedup: 2.1x
TTCP
STOP
STOP
STOP
TTCP
STOP
STOP
STOP
(a) Warmup step=2
(b) Original
(c) Warmup step=20%
(d) Multi-stage
Figure 9: Comparison of different warmup and stage strategies: (a) minimal warmup with 2 steps followed by ASDSV Stage-2, (b) vanilla diffusion process, (c) conservative warmup with 20% of total steps (10 steps) followed by ASDSV Stage-2, (d) proposed multi-stage strategy with 2 steps for Stage-0 and 8 steps for Stage-1 1 (20% total diffusion steps as warmup), followed by the same ASDSV Stage-2.
A.4 Results of FID against Real Images
ASDSV aims to preserve the generation quality of original models, that we only present the FID compared with the original generationed images in the paper. We now provide the FID scores against real images COCO Captions 2014 in Table 9
Table 9: FID scores against real images on COCO Captions 2014.
Method
FID (based ground truth) FID (based original) $
original
35.63
ASDSV-fast
35.25
4.1
ASDSV-slow
35.31
3.90
The scores based ground truth here are very close, with change of only 0.9%. Although the score for ASDSV-fast is higher (0.17% higher than ASDSV-slow), ASDSV-slow is actually closer the original score, which also meets our expected goal.
A.5 Expected Speedup of ASDSV
Here is the theoretical formula for the expected speedup, which incorporates the verification failure rate E(3). The expected speedup factor for each stage is as follows:
(1 - E(B)) + (1 (1 - E(9))2)
Seepected
(1)
YC + 2
Proof. Let the cost of running a single step of Mp by T. An unoptimized round of ASDSV costs yeT + 2T. This cost covers running the draft model My times and running the target model M,

---

twice (at the first and last step). Assuming the two verification checks are independent, the expected number of steps advanced per cycle is given by:
Elsteps] (1 E(3) (1 - E(O))))
(1-(1_(()))
This calculation assumes simple failure handling, where a failed verification results in advancing a single step. Thus, the effective cost per advanced step is the total cycle cost divided by the expected steps advanced:
yCT + 27
E(3) + (1 Since the cost of generating single step with the target model M,is T, get the desired result as equation (1).
E())))
is pesteemddppenden constant, determined by the specific hardware and software implementation. A smaller faster draft model) reduces the drafting overhead. However, it may imply a simpler model that struggles to approximate the target model's 's output, thus increasing the verification failure rate BB)). In our experiments, conservatively-choss y leads few failure cases.
As stated in Section 3.3
B Hyperparameter Auto-Tuning Process
As a follow-up to the sensitivity analysis in Section 4.3 this section details the principled, semi- automated process for determining the key hyperparameters for new models. We address each hyperparameter below:
For and Initial Steps N (Generalization): We found " to be less sensitive hyperparameter for the short Stage generalizing well across models. fixed, conservative value (typically 3) is robust choice. The initial step count N is set to 5, analogous to the warm-up phase in other methods 10 20
For the Stage Boundary (Profile Model Dynamics): We determine the boundary between Stage 1 and Stage 2 via a small-scale profiling run. By generating a small number of samples (as in Figure ())), we can observe the model's output dynamics. This empirical data is crucial for setting the stage boundary.
We start with a candidate 72 based on the theoretical speedup ceiling for target (Section B.3). Next, using the verification loss curve obtained from the profiling stage (e.g., ()))) we set a corresponding threshold slightly above the observed loss at the 72-th step. Finally, we iteratively increase 72 and adjust 6, evaluating the trade-off a validation until the desired speedup is met with minimal quality impact.
For "/2 and (Iterative Tuning):
C Social Impact
The acceleration of diffusion models provided by ASDSV reduces computational resources and latency, improving real-time applicability of state-of-the-art diffusion models while promoting environmental sustainability through reduced energy consumption. However, is important to note that ASDSV focuses primarily on efficiency gains and does not address inherent challenges such as privacy, bias, and fairness the underlying diffusion models.
D Additional Visualization
We provide comprehensive visual comparisons between ASDSV and bascline acceleration methods for both text-to-image and text-to-video generation, as shown in Figure TO and Figure [T] Extensive results demonstrate the superior visual fidelity of ASDSV across different generation scenarios.
25

---

Original
ASDSV-slow
ASDSV-fast
TeaCache-slow
TeaCache-fast
Figure 10: Comparison different accelerating methods on text-to-image generation using Flux. dev at 512x51 12 resolution.
26

---

Dolll
Original
ASDSV-slow
TeaCache-slow
Sparse-slow
Figure 11: Comparison of different accelerating methods on text-to-video generation using Wan2.1 480P resolution with 81 frames.
27


Reviews:
[{"review_id": "Myufk3PWVy", "reviewer": "Reviewer_C7bR", "summary": "This paper introduces Speculative Diffusion with Speculative Verification (SDSV). The core idea is to leverage a compressed, lightweight model as a draft model in the diffusion process. To reduce the cost of verification, SDSV verifies only the initial and final states of the speculative steps. Furthermore, the authors divide the denoising process into multiple stages, assigning different speculative steps to each stage to enhance sample quality. Experiments on state-of-the-art models, FLUX and WAN, demonstrate the effectiveness of SDSV.", "questions": "What draft models are used in Table 1?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- The idea of using off-the-shelf compressed model as a draft model is interesting. It is also intuitive and easy to understand.\n- The method is validated on leading image and video generation models (FLUX and WAN), demonstrating strong empirical performance.\n\n\n## Weaknesses\n\n- As acknowledged in the limitations, the method requires an additional draft model, which may not always be readily available. Acquiring such models can require significant effort. Additionally, using an extra model increases memory consumption.\n- SDSV does not generalize well to low-step diffusion models; all experimental results are based on 50-step settings.\n- The paper's writing needs to be improved:\n  - Some mathematical notations are undefined prior to use—for instance, $D$, $T$, and $i$ in Line 131.\n  - Figure 2’s plots are unclear. What is the \"output trend\" referred to in Line 134? How is it defined mathematically?\n  - Lines 182–187: The intuition behind choosing $\\gamma_2 = 9$ from Figure 2(b)(c) is not clear to me.", "was_revised": true, "final_justification": "The rebuttal addressed my concerns about the paper. I will keep my rating to accept this paper.", "rebuttal": "We thank the reviewer for the insightful and valuable comments. We especially appreciate the recognition that our work addresses an important/interesting problem. We respond to your comments as follows and sincerely hope that our rebuttal could properly address your concerns.\n\n***Q1. What draft models are used in Table 1?***  \n\nThe draft model used in Table 1 is Flux-SVD. As we described in Lines 232-233, Flux-SVD is the Flux.1-dev model quantized by the SVDQuant [21]. We will explicitly label \"Flux-SVD\" as the draft model in the caption of Table 1 in the revised version.\n\n\n***W1. Using an extra model increases memory consumption.***\n\nYes, but it can be alleviated. We measured the peak memory overhead to be +9GB (29.7%) for Text-to-Image and +5GB (8.0%) for Text-to-Video. However, the target and draft models do not need to run concurrently. This allows the extra memory consumption to be almost entirely eliminated by using asynchronous swapping without affecting inference latency (i.e., swapping the inactive model's parameters between GPU and CPU).\n\n***W2. SDSV does not generalize well to low-step diffusion models; all experimental results are based on 50-step settings.***\n\nSDSV focuses on accelerating diffusion models which have more steps and thus face the issue of slow inference. Low-step diffusion models are out-of-scope. We will clarify the positioning of our work.\n\nOur experiments are based on the 50-step setting because most SOTA text-to-image and text-to-video models operate in this setting to ensure high-fidelity results. For example, leading models like FLUX.1-dev (T2I), Wan2.1 (T2V), and CogVideoX1.5 (I2V) all specify 50 steps as their default configurations.\n\n***W3. The paper's writing needs to be improved.***\n\nThanks for the detailed feedback. We will further proofreed our manuscript.\n\n1. We will define all key mathematical symbols at their first appearance. For example, we will state that \"$D_i$ and $T_i$ denote the predicted outputs from the draft and target models at step $i$, respectively,\" ensuring all notations are unambiguous throughout the paper.\n\n2. We will provide a more precise definition for the term \"output trend\" in Line 134. This refers to our key empirical observation: the strong temporal correlation between draft and target models in the diffusion process. Specifically,  as shown in Figure 2(a), the magnitude of change of the draft model's output and the target model's output between consecutive steps is highly similar (i.e., $||D\\_i - D\\_{i-1}||\\_1 \\approx ||T\\_i - T\\_{i-1}|\\|_1$) after the first $N$ initial steps.\n\n3. We will add a detailed explanation for our choice of $\\gamma_2$ in Line 183. As we now illustrate with a red vertical line in Figure 2(b)(c), the verification error curve exhibits a sharp increase at the 9th step. This inflection point shows that the draft model's outputs are starting to diverge significantly from the target model's. Therefore, it represents the appropriate moment to halt the drafting phase."}, {"review_id": "EmpgySvZEZ", "reviewer": "Reviewer_9xwA", "summary": "The authors propose to improve inference efficiency of diffusion models by adapting speculative decoding to diffusion transformers. They show their approach leads to significant speedup (up to 3x) with minimal drop in quality.", "questions": "- Can the authors include standard deviations over multiple seeds in their reported results?\n- Can the authors comment on hyperparameter sensitivity to $\\gamma_1$, $\\gamma_2$?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n- The proposed speculative decoding method leads to significant inference-time improvements (even under different video lengths and resolution) with minimal degradation in quality, which will have significant impact in the research community\n- The proposed strategy of only verifying the first and last denoising steps is novel and impactful\n- The authors provide extensive ablations and clear analysis of the theoretical speedup\n\n**Weaknesses**\n- Missing analysis statistical uncertainty. Error bars for quality-latency tradeoff in Figure 5 are not provided\n- Missing analysis on hyperparameter sensitivity of $\\gamma_1$, $\\gamma_2$, which are manually selected", "was_revised": true, "final_justification": "I reserve my score of \"accept\"\n- The authors provided extensive follow-up results that addressed my remaining concerns, in particular, statistic significance and hyperparamter sensitivity", "rebuttal": "We thank the reviewer for the insightful and valuable comments. We especially appreciate your recognition of an important problem, a novel method, and a clear analysis. We respond to your comments as follows and sincerely hope that our rebuttal could properly address your concerns.\n\n***W1/Q1: Can the authors include standard deviations over multiple seeds in their reported results?***\n\nOur method is insensitive to the random seed. This is because the seed in the diffusion process only affects the initial noise, whereas the efficacy of SDSV depends on the temporal correlation between the draft and target models (as shown in Figure 2). \n\nTo quantitatively verify this and address the reviewer's request, we further conducted a new set of focused experiments on three additional seeds (3461, 772190542, 118270042274490399) for our main comparisons (Original vs. SDSV) during the rebuttal period. Performance was consistent across both seeds (SDSV achieves better performance than other accerlation methods). The standard deviations over multiple seeds results on Text-to-Images over are below:\n\n| Images Metric | Ratio (Original / SDSV) |\n| :---: | :---: |\n| CLIP | 100.04% ± 0.10% |\n| IS | 103.84% ± 3.7% |\n\nSince most of the video metrics are reference-based, we cannot calculate a relative ratio (original/accelerated method). Therefore, we used a comparison between the accelerated methods (SDSV vs. Teacache), and SDSV is about 1.5x faster than Teacache.\n\n| Videos Metric | Ratio (SDSV / Teacache) |\n| :---: | :---: |\n| LPIPS↓ | 88.02% ± 2.85% |\n| PSNR↑ | 106.10% ± 1.25% |\n| SSIM↑ | 104.95% ± 1.84% |\n\n***W2/Q2: Can the authors comment on*** ***hyperparameter*** ***sensitivity to $\\gamma1, \\gamma2$?***\n\nBoth $\\gamma_1$, $\\gamma_2$ depend on the temporal correlation between draft and target model on different stages, so SDSV is sensitive to these parameters (see Figure 2). Nevertheless, since the temporal correlation can be profiled offline, we can systematically search for the best candidates for these hyperparameters for each model offline. Please check the detailed analysis below and we will add them to the revised version.\n\n1. **Analysis of $\\gamma_1$ and its search method** \n\nFirst, we use brute force to search $\\gamma_1$ because the space is small: it can only be an integer in (1,5). Below is the search space of Wan2.1. We only present one model result for simplicity, other models are the same.\n\n| $\\gamma_1$ | LPIPS↓ | PSNR↑ | SSIM↑ | Speedup↑ |\n| :---: | :---: | :---: | :---: | :---: |\n| 2 | 0.23 | 18.57 | 0.61 | 1.70x |\n| 3 | 0.30 | 16.80 | 0.59 | 1.71x |\n| 4 | 0.30 | 16.76 | 0.59 | 1.72x |\n\n\n2. **Analysis of $\\gamma_2$ and its search method**\n\nBecause the search space of $\\gamma_2$ is larger than $\\gamma_1$ (an integer selected from range (1,15)), we developed a two-step iterative-guided search process for $\\gamma_2$ to balance speed-fidelity trade-off:\n\n- **Profile Model Dynamics:** We conduct a small-scale sampling run to profile the model's behavior, generating plots similar to Figure 2. This allows us to observe the output change dynamics and step-wise verification loss, which are crucial for setting the $\\gamma_2$ and decide the verfication threshold.\n- **Iteratively Tune $\\gamma_2$ and $\\delta$:** We start with a candidate $\\gamma_2$ based on the theoretical speedup ceiling for our target (Section 3.3). Next, using the verification loss curve (like Fig. 2b/c), we set a corresponding threshold $\\delta$ above the observed loss at the $\\gamma_2$-th step. We then incrementally increase $\\gamma_2$ and adjust $\\delta$, evaluating the trade-off on a validation set until the desired speedup is met with minimal quality impact.\n\nThe following results show typical search results on Wan2.1 model, others are the same:\n\n| $\\gamma_2$ | $\\delta$ | LPIPS↓ | PSNR↑ | SSIM↑ | Speedup↑ |\n| :---: | :---: | :---: | :---: | :---: | :---: |\n| 3 | 0.1 | 0.24 | 18.89 | 0.60 | 1.2x |\n| 9 | 0.2 | 0.23 | 18.57 | 0.61 | 1.7x |\n| 12 | $\\infty$ | 0.25 | 18.47 | 0.60 | 3.1x |\n\nDue to the rebuttal format constraints, we are unable to show visual results here, but they will be included in the revised version."}, {"review_id": "zEmmmdF8Op", "reviewer": "Reviewer_e9nk", "summary": "The paper introduces speculative decoding in LLMs to diffusion models and proposes Speculative Diffusion with Speculative Verification. The proposed method uses a smaller version of the target model as a draft model to speed up the inference. The proposed method speeds up the inference of the diffusion model.", "questions": "1. In the proposed method, for every K steps, only the first and last steps’ noises will be compared and verified. The last step takes the output of the (N+K-1)-th step of the draft model as inputs, how to guarantee that the target model samples every step from N+1 to N+K-1, and the output of the (N+K-1)-th step of the target model will be the same?\n2. In Table 1, I think the FID should also be calculated against the COCO Captions 2014 dataset. Or any other real datasets. Only calculating FID with the images generated by the base model is not enough, because it only indicates how similar the generated images are to those of the original model. We also care about the fidelity of the generated images compared to the real images.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Paper Strengths:\n1. The paper introduces the idea of speculative decoding in LLMs to diffusion models, which is significant to the vision community\n2. The paper is well-written and easy to follow.\n3. The proposed method speeds up the inference and does not harm the visual quality of generated images.\n\nPaper Weaknesses:\n1. In the proposed method, for every K steps, only the first and last steps’ noises will be compared and verified. The last step takes the output of the (N+K-1)-th step of the draft model as inputs, how to guarantee that the target model samples every step from N+1 to N+K-1, and the output of the (N+K-1)-th step of the target model will be the same?\n2. In Table 1, I think the FID should also be calculated against the COCO Captions 2014 dataset. Or any other real datasets. Only calculating FID with the images generated by the base model is not enough, because it only indicates how similar the generated images are to those of the original model. We also care about the fidelity of the generated images compared to the real images.", "was_revised": true, "final_justification": "The rebuttal addressed my questions regarding the details of the proposed method. I will keep my rating to accept the paper.", "rebuttal": "We thank the reviewer for the insightful and valuable comments. We especially appreciate the recognition that our work addresses a significant problem and represents a promising approach. We respond to your comments as follows and sincerely hope that our rebuttal could properly address your concerns.\n\n***W1/Q1: In the proposed method, for every K steps, only the first and last steps’ noises will be compared and verified.How to guarantee that the target model samples every step from N+1 to N+K-1, and the output of the (N+K-1)-th step of the target model will be the same?***\n\nAs detailed in Section 3.1, our speculative verification method chooses an approximative (threshold-based) while compution-less acceptance strategy instead of guaranteeing exactly the same output. We think this is a practical tradeoff between generation fidelity and latency.\n\nSpecifically, as illustrated in Figure 2(a), after an initial warm-up phase, the step-wise change is highly consistent for draft and target models. That is, the magnitude of change between consecutive steps, measured by L1\\_Loss becomes highly similar(i.e., $||D\\_i - D\\_{i-1}||\\_1 \\approx ||T\\_i - T\\_{i-1}||_1$). Based on  the draft and target model both trajectories evolve with a similar trend, if the start and last step of the current round are verified to be similar, and then the intermediate points are also highly likely to be valid (i.e., their verification loss would be below the threshold $\\delta$). \n\nThis high fidelity is demonstrated by the extensive quantitative metrics (e.g., FID, IS) and qualitative visual results presented in our evaluation. \n\n***W2/Q2: Calculating*** ***FID*** ***between the generated images and the real images.*** \n\nSDSV aims to preserve the generation quality of original models, so that we only present the FID compared with the original generationed images in the paper. Following your advice, we now provide the FID scores against real images on COCO Captions 2014: \n\n| Method | FID (based ground truth) | FID (based original) ↓ |\n| :---: | :---: | :---: |\n| original | 35.63 | -- |\n| SDSV-fast | **35.25** | 4.11 |\n| SDSV-slow | 35.31 | **3.90** |\n\nThe scores based ground truth here are very close, with a change of only 0.9%. Although the score for SDSV-fast is higher (0.17% higher than SDSV-slow), SDSV-slow is actually closer to the original score, which also meets our expected goal."}, {"review_id": "QSXiR42Tij", "reviewer": "Reviewer_1KPp", "summary": "This paper introduces Speculative Diffusion with Speculative Verification (SDSV), a novel method for accelerating multimodal diffusion models. It employs a lightweight \"draft\" model to propose future steps, which are then checked by the larger \"target\" model. The first innovation is \"speculative verification\" which cuts down the computational costs by only verifying the first and final outputs of a speculative sequence. Secondly, SDSV uses a multi-stage strategy that adjusts the number of speculative steps, employing fewer steps in the early stages and more in the stable later phases to optimize the speed-quality trade-off. On SOTA models like Flux.1-dev and Wan2.1, this approach achieves up to a 3.01x inference speedup and only 0.3%-0.4% drop in VBench quality scores.", "questions": "1. The hyperparameters are empirically chosen for specific model pairs. How well do they generalize to new models? Are there simpler, automated methods for tuning them?\n\n2. Can you provide a direct cost comparison against traditional method (Speculative Diffusion) on a high-resolution task?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths:\n\n1. Adequate experiments and compelling results: The method is supported by strong empirical evidence. The experiments are adequate, evaluating on SOTA models for both image and video generation and comparing against strong baselines like TeaCache and Sparse VideoGen. The speedups are substantial and the quality degradation is minimal.\n\n2. Easy-for-implementation: SDSV is a plug-and-play inference optimization without the need to retrain or modify the original model architecture, which makes it highly practical.\n\n# Weaknesses\n\n1. Simple failure handling: The current strategy for handling a verification failure is to discard all subsequent speculative steps and revert to the target model. This approach is inefficient, especially when using a large number of speculative steps, as it wastes the computation from the draft model.\n\n2. Complexity of Hyperparameter selection: The overall workflow needs many hyperparameters including the initial steps N, speculative steps for Stage-1 and Stage-2, and the number of rounds for each stage. The paper states these are chosen based on step-metric results (as in Fig 2b, 2c), which means that the tuning process is needed for every new model pair.\n\n3. Lack of comparison with traditional Speculative Methods: The paper states that prior work's (Speculative Diffusion) verification process is too costly for high-resolution tasks. But a specific comparison in computation resources or time is missing.", "was_revised": true, "final_justification": "Thanks for the rebuttal. My concerns have been well addressed, and I will keep my positive score to accept this paper.", "rebuttal": "We thank the reviewer for the insightful and valuable comments. We especially appreciate the recognition that our work is a pratical solution and provides a thorough evaluation. We respond to your comments as follows and sincerely hope that our rebuttal could properly address your concerns.\n\n***W1: Simple failure handling*** ***may waste the computation from the draft model.***\n\nAs discussed in Section 5,  a binary search can be employed instead of discarding the entire K-step sequence. When a failure is detected at the final step, SDSV would recursively check the mid-step of the sequence to efficiently find the largest valid prefix of accepted steps. Since of our conservative hyperparameter setting, we encounter few failures and thus currently do not employ such a failure handling method.\n\n***W2/Q1: The hyperparameters are empirically chosen for specific model pairs. How well do they generalize to new models? Are there automated methods for*** ***tuning*** ***them?*** \n\nYes, our hyperparameters can be determined for new models through an automated process. We address each parameter below:\n\n- **For $\\gamma_1$ and Initial Steps $N$ (Generalization):** We found $\\gamma_1$ to be a less sensitive hyperparameter for the short Stage 1, generalizing well across models. A fixed, conservative value (typically 2 or 3) is a robust choice. The initial step count N is set to 5, analogous to the warm-up phase in other methods [10,20].\n- **For the Stage Boundary (Profile Model Dynamics):** We determine the boundary between Stage 1 and Stage 2 via a small-scale profiling run. By generating a small number of samples (as in Fig. 2a), we can observe the model's output dynamics. This empirical data is crucial for setting the stage boundary.\n- **For $\\gamma_2$ and $\\delta$ (Iterative** **Tuning**):  We start with a candidate $\\gamma_2$ based on the theoretical speedup ceiling for our target (Section 3.3). Next, using the verification loss curve obtained from the profiling stage (e.g., Fig. 2b/c), we set a corresponding threshold $\\delta$ slightly above the observed loss at the $\\gamma_2$-th step. Finally, we iteratively increase $\\gamma_2$ and adjust $\\delta$, evaluating the trade-off on a validation set until the desired speedup is met with minimal quality impact.\n\n***W3/Q2: Can you provide a direct cost comparison against traditional method (Speculative Diffusion) on a*** ***high-resolution*** ***task?***\n\nYes. Here is one direct comparison. On the Wan2.1 (Text-to-Video) model, traditional Speculative Diffusion takes 1,120s compared to the original model's 930s, i.e., increases the inference latency by 20%.\n\nThis slow down is because the core assumption of the original speculative diffusion (Unet)—efficient parallel verification of draft steps—breaks down on SOTA, compute-bound diffusion in transformer (DiT) models."}, {"review_id": "l3210e8ylX", "reviewer": "Reviewer_yuAv", "summary": "This paper proposes a method named SDSV, which aims to accelerate the inference of high-quality diffusion models (e.g., Flux and Wan2.1) by leveraging a lightweight \"draft\" model. The authors claim inspiration from the field of Speculative Decoding in Large Language Models (LLMs). The method's core idea is to use a \"speculative verification\" mechanism, which checks only the first and last steps of a generated sequence to decide on acceptance, combined with a multi-stage strategy to dynamically adjust the speculation length.", "questions": "* The Rigor of the Theoretical Analysis: Can you provide a more complete theoretical model for the expected speedup that incorporates the verification failure rate (as a function of δ and K), rather than assuming 100% success? How would this more realistic model change your discussion on the importance of the cost coefficient (c)?\n* The Sensitivity of the Core Hyperparameter δ: Your paper only provides fixed values for δ for specific tasks. The choice of δ directly controls the trade-off between approximation error and acceleration efficiency, making it the central balancing point of your entire method. Can you please provide a sensitivity analysis for δ? For instance, how do the model's quality metrics (e.g., FID) and the measured speedup change as δ is varied from 0 to a larger value? This is critical for understanding the true performance envelope of your method.\n* About writing: All formulas in the article are unnumbered.", "limitations": "Please see Weaknesses and Questions.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "## Strengths\n* Engineering Effectiveness: It is undeniable that the proposed set of techniques, combining a draft model, partial verification, and a multi-stage schedule, achieves significant speedups in practice. As an acceleration heuristic, the method demonstrates that it can keep visual quality degradation to a minimum under specific hyperparameter settings, which is a valuable reference.\n\n## Weaknesses\n* Fundamental Misuse of the \"Speculative\" Concept: This is the most critical flaw of the paper. In the LLM domain, speculative decoding has an unshakeable theoretical cornerstone: it is an unbiased acceleration technique that mathematically guarantees that its final output sequence has the exact same probability distribution as the sequence generated by the original target model autoregressively. It is a form of lossless speedup. The proposed SDSV, however, does not possess this property at all.\nThe core of SDSV relies on an L1 loss and a threshold δ to determine the acceptance of the draft model's output. This means its acceptance criterion is approximative. The distribution of its generated results will inevitably deviate from the original target model's distribution; it is, by definition, a form of lossy approximation.\nThe paper's own experiments, which repeatedly measure and report degradations in quality metrics (e.g., in FID, PSNR, and a 0.3%-0.4% drop in VBench score ), serve as proof against its own \"speculative\" framing. If the method truly guaranteed distributional identity, there would be no quality drop to measure, only a change in speed.\n\nTherefore, naming the method \"Speculative Diffusion\" is highly misleading. It incorrectly implies that the method possesses the same theoretical guarantees as speculative decoding in the LLM field, which is an abuse of a core concept.\n\n* Superficial Theoretical Analysis: The paper's theoretical analysis of the speedup ratio is also problematic.\nThe speedup formula presented in Section 3.3 is derived under the unrealistic assumption that \"all verifications are successful\". This completely ignores the most critical variable in the entire method: the verification failure rate, which is governed by the threshold δ and the step length K.\nIn true speculative decoding, the cost coefficient (c) and the acceptance rate are the two central factors that determine the final expected speedup. This paper's analysis fixes the acceptance rate at 100%, rendering the entire analysis superficial and incapable of revealing the method's true performance dynamics. A more rigorous analysis would model the expected speedup, E[speedup], as a function of c, K, and δ.\n\n* Limited Novelty Beyond Heuristics: If we strip away the misleading \"speculative\" label, the method's core idea can be seen as \"using a proxy model to generate candidates, then using a metric (L1 loss) to decide whether to skip the original model's computation.\" This is conceptually similar to existing cache-based skipping methods. For instance, TeaCache also uses a threshold to determine if adjacent inputs are similar enough to reuse a cache (i.e., skip a step). SDSV merely changes the comparison from \"adjacent inputs\" to \"draft vs. target outputs\" and the action from \"reusing a cache\" to \"accepting the draft result.\" While the implementation is more complex, the novelty of the core concept—\"threshold-based approximate skipping\"—is questionable", "was_revised": true, "final_justification": "The authors' rebuttal addresses my main concerns. Their method shows a clear advantage in balancing speed and quality, tackling the important problem of diffusion model acceleration.\n\nHowever, the original manuscript had significant flaws in its theoretical framing and rigor, particularly regarding the misleading use of terminology (framing an approximative method as \"speculative\" in the lossless sense) and a lack of theoretical rigor in its analysis. Therefore, while I am raising my score based on their promised rebuttal, my recommendation remains cautiously optimistic. The paper's final quality is contingent on the authors thoroughly implementing the proposed changes, particularly renaming the method and strengthening the theoretical analysis.", "rebuttal": "We appreciate the reviewer for the insightful and constructive comments. We respond to your comments as follows and sincerely hope that our rebuttal could properly address your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know your further concerns, and we will continue actively responding to your comments and improving our paper.\n\n***W1:***  ***Misuse of the \"Speculative\" Concept. It incorrectly implies that the method possesses the same theoretical guarantees as speculative decoding in the*** ***LLM*** ***field.***\n\nWe do apologize for causing confusion. Please let us clarify our original intent: 1) We never try to imply our method is lossless: we presented the quality drop in abstract, introduction, and evaluation in the submission. 2) We deliberately add \"Speculative\" before \"Verification\", with the intent to explain our verification uses an \"approximate\" way.\n\nIn the revised version, we will emphasize the difference with speculative decoding in the LLM field at the beginning of the paper and use the name of \"Approximate Speculative Diffusion with Speculative Verification (ASDSV)\".\n\n***W2/Q1: The Rigor of the Theoretical Analysis：1) Can you provide a more complete theoretical model for the expected speedup that incorporates the verification*** ***failure rate*** ***(as a function of δ and K)? 2) How would this more realistic model change your discussion on the importance of the cost coefficient (c)?***\n\n1\\) Yes. Here is a more complete formula for the expected speedup, which incorporates the verification failure rate $E(\\beta)$. The expected speedup factor for each stage is as follows: \n\n$S_{expected} = \\frac{(1-E(\\beta))^2\\gamma + (1 - (1-E(\\beta))^2)}{\\gamma c + 2} \\tag{1} $\n\n**Proof.** Let the cost of running a single step of $M_p$ by $T$. An unoptimized round of SDSV costs $\\gamma cT+2T$.  This cost covers running the draft model $M_q$ $\\gamma$ times and running the target model $M_p$ twice (at the first and last step). Assuming the two verification checks are independent, the expected number of steps advanced per cycle is given by: $E[\\text{steps}] = (1-E(\\beta))^2 \\gamma + (1 - (1-E(\\beta))^2) \\cdot 1$ This calculation assumes simple failure handling, where a failed verification results in advancing a single step. Thus, the effective cost per advanced step is the total cycle cost divided by the expected steps advanced: $\\frac{\\gamma cT+2T}{(1-E(\\beta))^2\\gamma + (1 - (1-E(\\beta))^2)}$ Since the cost of generating a single step with the target model $M_p$is $T$, we get the desired result. \n\n2\\)  As stated in our submission, $c$ is still a system-dependent constant, determined by the specific hardware and software implementation [19].  A smaller c (a faster draft model) reduces the drafting overhead. However, it may imply a simpler model that struggles to approximate the target model's output, thus increasing the verification failure rate $E(\\beta)$. In our experiments, conservatively-chosen $\\gamma$ leads to few failure cases.\n\nWe will add this analysis to Section 3.3 of our revised manuscript. Thank you for helping us strengthen our theoretical discussion.\n\n***Q2: The Sensitivity of the Core*** ***Hyperparameter*** ***δ***\n\nBoth $\\gamma_1$, $\\gamma_2$ depend on the temporal correlation between draft and target model on different stages, so SDSV is sensitive to these parameters (see Figure 2). Nevertheless, since the temporal correlation can be profiled offline, we can systematically search for the best candidates for these hyperparameters for each model offline. Please check the detailed analysis below and we will add them to the revised version.\n\n1. **Analysis of $\\gamma_1$ and its search method** \n\nFirst, we use brute force to search $\\gamma_1$ because the space is small: it can only be an integer in (1,5). Below is the search space of Wan2.1. We only present one model result for simplicity, other models are the same.\n\n| $\\gamma_1$ | LPIPS↓ | PSNR↑ | SSIM↑ | Speedup↑ |\n| :---: | :---: | :---: | :---: | :---: |\n| 2 | 0.23 | 18.57 | 0.61 | 1.70x |\n| 3 | 0.30 | 16.80 | 0.59 | 1.71x |\n| 4 | 0.30 | 16.76 | 0.59 | 1.72x |\n\n\n2. **Analysis of $\\gamma_2$ and its search method**\n\nBecause the search space of $\\gamma_2$ is larger than $\\gamma_1$ (an integer selected from the range (1,15)), we developed a two-step iterative-guided search process for $\\gamma_2$ to balance the speed-fidelity trade-off:\n\n- **Profile Model Dynamics:** We conduct a small-scale sampling run to profile the model's behavior, generating plots similar to Figure 2. This allows us to observe the output change dynamics and step-wise verification loss, which are crucial for setting the $\\gamma_2$ and deciding the verification threshold.\n- **Iteratively Tune $\\gamma_2$ and $\\delta$:** We start with a candidate $\\gamma_2$ based on the theoretical speedup ceiling for our target (Section 3.3). Next, using the verification loss curve (like Fig. 2b/c), we set a corresponding threshold $\\delta$ above the observed loss at the $\\gamma_2$-th step. We then incrementally increase $\\gamma_2$ and adjust $\\delta$, evaluating the trade-off on a validation set until the desired speedup is met with minimal quality impact.\n\nThe following results show typical search results on Wan2.1 model, others are the same:\n\n| $\\gamma_2$ | $\\delta$ | LPIPS↓ | PSNR↑ | SSIM↑ | Speedup↑ |\n| :---: | :---: | :---: | :---: | :---: | :---: |\n| 3 | 0.1 | 0.24 | 18.89 | 0.60 | 1.2x |\n| 9 | 0.2 | 0.23 | 18.57 | 0.61 | 1.7x |\n| 12 | $\\infty$ | 0.25 | 18.47 | 0.60 | 3.1x | \n\nThe baseline for these comparisons is the original model, corresponding to the setting $\\gamma_1=0, \\gamma_2=0$. The setting of $\\delta=\\infty$ represents an \"always-accept\" strategy that effectively bypasses the verification stage to achieve maximum acceleration, corresponding to our SDSV-fast variant in Figure 5.\n\n***W3:  Limited Novelty Beyond Heuristics. It is conceptually similar to existing cache-based skipping methods.***\n\nFirst, SDSV continues the line of using a fast path to accelerate multi-modal generation but introduces a new paradigm for the \"fast path\" in multi-modal generation: approximate speculative generation. Compared with caching (e.g., PAB [45], Teacache [24]), the fast path is augmented with a draft model to significantly enhance the quality and fidelity (as shown in Figures 1 and 5). Compared with non-approximate speculative (e.g., Traditional speculative diffusion [2]), the generation latency speedup is improved by 2x with negligible quality drop thanks to the significantly reduced verification overheads.\n\nSecond, the design of an approximative speculation framework is far more challenging than simply deciding a threshold for skipping. As far as we know, we are the first to show that the vanilla draft-then-verify, which was designed for LLM, does not work well for SOTA, high-resolution multi-modal generation. Instead, we propose a draft-then-approximate-verify paradigm. It requires us to address which verification steps can be skipped, and how to balance the speed-fidelity trade-off. We propose systematic solutions including a multi-stage verification strategy and a system-level batched pipeline (deciding threshold is just one component of our overall design), and present promising results (e.g., 1.8-3X speedup with a minimal 0.3%-0.4% drop in VBench score) on diverse tasks using SOTA models like FLUX.1-dev and Wan2.1.\n\n***Q3:  About writing: All formulas in the article are unnumbered.***\n\nThanks for your suggestion. We will fix it in the revised version."}]

Author rebuttal:

Meta-review: This paper explores a new technical perspective for accelerating multimodal diffusion models: formulating a speculative verification strategy and a multi-stage speculative strategy, conditioned on a dual use of a draft model and a target model, inspired by speculative decoding. By these two strategies, the initial and final speculative outputs of K speculative steps by the draft and target model need to be aligned, facilitated by different-stage-value settings of K. The efficacy of the proposed method is validated on Flux.1-dev for image generation and Wan2.1 for video generation.

The paper initially|finally scored (4,5,5,4,2)|(4,5,5,4,3) by five knowledgeable reviewers, who mostly recognized the motivation, the idea and the basic performance of the proposed method. Meanwhile, the reviewers also raised some concerns about 1) improvements of the writing in some parts; 2) more experiments to study the generalization of the proposed method; 3) method limitations, e.g., for every K steps, only the first and last steps’ noises of the draft and target models will be verified, which is not optimal; 4) missing ablations, e.g., studying the choice of hyperparameters.

The authors provided detailed responses to these concerns, which were recognized by all of five reviewers. Finally, 4 reviewers maintained their positive scores, 1 reviewer (yuAv) increased the score from Reject to Borderline reject, criticizing the misleading use of word "speculative" in method formulation. The AC read the paper, the reviews, the rebuttal, the author-reviewer discussion and the reviewers' feedback, and agree with most of reviewers' assessment. Therefore, I recommend to accept this paper (poster). The authors are encouraged to carefully consider the reviewers' comments/suggestions and their rebuttal in the final paper revision.
Decision: Accept (poster)