BLADE: Block-Sparse Attention Meets Step Distillation for Efficient Video Generation
Authors: Youping Gu
XIAOLONG LI
Yuhao Hu
Chen Minqi
Bohan Zhuang
Venue: ICLR 2026 Poster (2026)
Keywords: sparse attention; video generation; step distillation
Primary area: generative models

Abstract: Diffusion transformers currently lead the field in high-quality video generation, but their slow iterative denoising process and prohibitive quadratic attention costs for long sequences create significant inference bottlenecks. While both step distillation and sparse attention mechanisms have shown promise as independent acceleration strategies, effectively combining these approaches presents critical challenges---training-free integration yields suboptimal results, while separately training sparse attention after step distillation requires prohibitively expensive high-quality video data. To overcome these limitations, we propose $\textit{BLADE}$, an innovative data-free joint training framework that introduces: (1) an Adaptive Block-Sparse Attention (ASA) mechanism for dynamically generating content-aware sparsity masks to focus computation on salient spatiotemporal features, and (2) a sparsity-aware step distillation paradigm, built upon Trajectory Distribution Matching (TDM), directly incorporates sparsity into the distillation process rather than treating it as a separate compression step and features fast convergence. We validate BLADE on text-to-video models like CogVideoX-5B and Wan2.1-1.3B, and our framework demonstrates remarkable efficiency gains across different scales. On Wan2.1-1.3B, BLADE achieves a 14.10$\times$ end-to-end inference acceleration over a 50-step baseline. Moreover, on models such as CogVideoX-5B with short video sequence lengths, our framework delivers a robust 8.89$\times$ speedup. Crucially, the acceleration is accompanied by a consistent quality improvement. On the VBench-2.0 benchmark, BLADE boosts the score of CogVideoX-5B to 0.569 (from 0.534) and Wan2.1-1.3B to 0.570 (from 0.563), results that are further corroborated by superior ratings in human evaluations.

Published as a conference paper at ICLR 2026
BLADE: BLOCK-SPARSE ATTENTION MEETS STEP DISTILLATION FOR EFFICIENT VIDEO GENERATION
Youping Gul
Xiaolong Li
Yuhao Hu?
Minqi Chen?
Bohan Zhuang'
'Zhejiang University
Centtal Media Technology Institute, Huawei Technologies
[youpgu71, xiaolong, ziplab, bohan. zhuang) @gma i 1 com.
aa-raatereepprrrerrs..e.a com, chenmingi @wwwweeeehhwwiieeee com
ABSTRACT
Diffusion transformers currently lead the field in high-quality video generation, but their slow iterative denoising process and prohibitive quadratic attention costs for long sequences create significant inference bottlenecks. While both step dis- tillation and sparse attention mechanisms have shown promise independent acceleration strategies, effectively combining these approaches presents critical challenges-ttraning---eee integration yields suboptimal results, while separately training sparse attention after step distillation requires prohibitively expensive high-quality video data. To overcome these limitations, we propose BLADE, an innovative data-free joint training framework that introduces: (1) an Adaptive Block-Sparss Attention (ASA) mechanism for dynamically generating content- aware sparsity masks to focus computation on salient spatiotemporal features, and (2) a sparsity-aware step distillation paradigm, built upon Trajectory Distri- bution Matching (TDM), directly incorporates sparsity into the distillation pro- cess rather than treating it as a separate compression step and features fast con- vergence. We validate BLADE on text-to- text-to-video video models like Cog VideoX-5B and 1-12.1 and our framework demonstrates remarkable efficiency gains across different scales. On Wan2.1-1.333 BLADE achieves 14.10x end-to-end infer- ence acceleration over 50-step baseline, and an 8.89x speedup on the short- sequence model X-g Crucially, the acceleration achieved while maintaining generation quality comparable to the original 50-step baseline. On the VBench-2.0 benchmark, BLADE boosts the score of Cog VideoX-5B 0.569 (from 0.534) and Wan2.1-1.33 11.BB to 0.570 (from 0.563), results that are further cor- roborated by superior ratings human evaluations. The Project available at http:// / ztpbbb.bbbb/...///.. co/ co/BLADE HoODE--eHepp./....
INTRODUCTION
Diffusion models have emerged as the state-of-the-art for a wide array of generative tasks (Dhariwal & Nichol, 2021), achieving unprecedented quality in image synthesis (Cao et al., 2024; Esser et al., 2024; Labs et al., 2025) and now pushing the frontier in the complex domain of video generation (Blattmann et al., 2023; Xing et al., 2024). By modeling generation as a gradual reversal of noising process (Ho et al., 2020), these models can produce diverse and high-fidelity content. However, for diffusion transformers, this power comes severe computational cost (Shen al., 2025). The introduction of the temporal dimension dramatically inflates the complexity of the attention mechanism, which scales quadratically with sequence length (Wan et al., 2025; Yang et al., 2024; Kong et al., 2025), This, combined with the iterative nature of the denoising process, results in prohibitively slow inference speeds that hinder practical deployment.
To mitigate this critical efficiency bottleneck, two primary research directions have gained promi- nence: reducing the number of inference steps via step distillation (Song et al., 2023; Salimans & Ho,
"Equal contribution.
I Corresponning author.

---

Published as a conference paper at ICLR 2026
2022: Liu et al., 2024; Zheng et al., 2024; Gu et al., 2023; Goodfellow et al., 2014; Yin et al., 2024) and lowering the per-step cost via sparse attention (Zhang et al., 2025b; Yuan et al., 2024; Zhang et al.. 2025a; Li et al., 2025; Xu et al., 2025; Dao et al., 2022). However, effectively integrating these two powerful paradigms non-trivial and presents a critical dilemma. A naive, training- frree combination, where pre-trained sparse attention mechanism is applied distilled model, yields suboptimal results because the distillation process is agnostic to sparse attention. Conversely, a sequential training pipeline that involves first performing step distillation and then fine-tuning the model for sparsity is equally impractical, as it re-ree-iie introduces the need for prohibitively large and ex- pensive high-quality video datasets, counteracting the key benefits of modern data-free distillation methods (Gu et al., 2023; Sauer et al., 2024; Luo et al., 2025).
The challenge of designing an appropriate sparse attention mechanism further exacerbated in the video domain. Many existing methods rely static, content-agnostic sparsity patterns (Zhang et al., 2025b; Li al., 2025; Xi al., 2025). These fixed patterns, such as rigid local windows or pre- determined striding, fail to adapt to the dynamic and diverse spatiotemporal features of video con- tent. Consequently, they often struggle to preserve important details and long-range dependencies, leading to significant quality degradation, especially at higher sparsity levels required for meaning- ful acceleration. In contrast, another line of work explores dynamically generated attention masks, which allow the sparsity pattern to adapt content-specific structure. While dynamic masking methods such as VSA (Zhang et al., 2025c) improve the trade-off between efficiency and fidelity, they conceptually operate on structured 3D token grids. For irregular latent shapes, this design typically necessitates padding dimensions to align with tile boundaries, introducing computational overhead that can diminish practical sparsity gains. On the other hand, SpargeAltention (Zhang et al., 2025a) supports vaining-free inference but cannot be trained and exhibits limited sparsity. Limited flexibility and applicability restrict the widespread adoption of dynamic sparse attention in video generation.
This landscape highlights a clear need for a sparse attention mechanism that is computationally ficient, dynamically content-aware, and flexible enough to support arbitrary resolutions and both training-free and training-aware modes at high sparsity without sacrificing visual fidelity. To this end, we introduce ASA, training-free sparse attention framework with dynamic token selection, pable of adapting to input content while maintaining high generation quality across various settings. For cases where training is permitted, further present ASA_G, distillation-bart based variant that leverages global token prediction to enable end-to-end training. Together, ASA and ASA G offer a unified solution both inference and training scenarios in efficient video generation.
Overall, this paper argues that a truly effective solution requires moving beyond treating distillation and sparsity as separate, post-hoc optimizations. We introduce BLADE (BLock-sparse Attention Meets step Distillation for Efficient video generation), novel framework that pioneers the syn- ergistic, data-free joint training of dynamic sparsity and step distillation. Our approach directly incorporates awareness into the distillation process, allowing the student model to learn a compact and efficient trajectory from the teacher, conditioned on dynamic attention pattern.
The main contributions of this work are as follows:
1. We propose BLADE, a data-free joint training framework that synergistically integrates an adaptive sparse attention mechanism directly into a sparsity-aware step distillation process, overcoming the limitations of prior sequential or training-fres integration approaches.
2. We introduce Adaptive Block- SpeckSS Attention (ASA), a dynamic. content-aware, and hardware-friende attention mechanism that generates sparsity masks on-the-fly to focus computation on salient features.
3. We demonstrate significant end-to-en end inference accelcration on diverse models, achieving a 14.10x speedup on 1. and a robust 8.89 x on the shorter- -horuer-sequence Cog VideoX- 5B. Crucially, this accelcration is accompanied by a consistent quality improvement, with VBench-2.0 scores increasing for both Wan2.1-1.30 (0.563 - 0.570) and Cog V5g (0.534 + 0.569).
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORK
2.1 VIDEO GENERATION WITH DIFFUSION MODELS
Recent years have witnessed remarkable progress in video generation, largely driven by the success of diffusion models (Ho et al., 2020; Song et al., 2021; Ma et al., 2025; Cao et al., 2024; He et al., 2023). These models have become the de facto standard for synthesizing high-fidelity and tempo- rally coherent video content, achieving state-of- sate-of-tteee ---tthe-rttttt results on various benchmarks (Huang al., 2024; Zheng et al., 2025).
The operating principle of diffusion models is to learn the reversal of fixed data corruption process. Specifically, a noisy sample x, is generated by corrupting a clean sample Xg ~ Preal using a simple formulation: X, Xx o,e, where e ~ N I) is standard Gaussian noise. The positive scalars ar and or are dictated by a noise schedule, which controls the signal-to-noise ratio at each timestep t (Karras et al., 2022).
The model's s task is to learn this reversal. A network, often termed a denoiser [4a( t), is trained to predict the original clean sample Xo from its corrupted version Xt. This learned denoiser provides an estimate of the score function (Song et al., 2021):
so (exx,, = Vx, log ( Xt. (xpe)(
(1)
of
Generation is then achieved by starting with pure noise XT (O, I) and iteratively applying the learned denoising function to reverse the corruption process, step-by-step, until a clean sample x is obtained.
2.2 ACCELERATION VIA STEP DISTILLATION
Step distillation has emerged a primary strategy for accelerating diffusion models (Song al., 2023; Salimans & Ho, 2022; Liu et al., 2024; Zheng et al., 2024; Gu et al., 2023; Goodfellow et al., 2014). The goal is transfer the knowledge from slow "teacher" model (e.g., 50-step sampler) to faster "student" model that can generate comparable results in very few steps (e.g.. 1-8 steps). Early methods like Progressive Distillation (Salimans & Ho, 2022: Luhman & Luhman, 2021) iter- atively halve the number of sampling steps. Distillation strategies can be broadly categorized into output distillation, which trains the student match the final output of multi-step teacher process, and trajectory distillation (Luhman & Luhman, 2021; Song et al., 2023), which guides the student to follow the teacher's intermediate generation path. Trajectory Distribution Matching (TDM) rep- resents recent and sophisticated advancement in this area (Luo al., 2025). TDM unifies the concepts of distribution matching and trajectory matching. Instead of enforcing a strict instance- level match of the trajectory, it aligns the distribution of the student's intermediate samples with the teacher's s corresponding diffused distributions at each step. A key advantage of TDM is that is data-free method; it does not require access the original, often proprietary, training dataset, relying only on the pre-trained teacher model to generate guidance signals. This makes it highly practical and versatile distillation framework, which we adopt as the foundation for work.
2.3 VIDEO-SPECIFIC SPARSE ATTENTION
Several promising approaches have been proposed to accelerate attention computation, each with distinct mechanisms and trade-offs. Early methods such as STA (Zhang et al., 2025b) and Radial Attention (Li al., 2025) primarily utilize static attention masks. STA employs fixed local window, a design choice that makes it most effective for specific input dimensions, while Radial Attention proposes heuristic whose resulting sparsity is less pronounced on shorter sequences, limiting its adaptability. To introduce more dynamism, SVG (Xi et al., 2025) selects between two pre-defined masks, a binary choice that offers limited granularity and may create a trade-off between quality and sparsity. Other methods like SpargeAttenttin (Zhang et al., 2025a) also shows potential in training-free free scenarios. However, is not applicable to training, and its sparsity level must be kept moderately low preserve video quality. VSA (Zhang al., 2025c) introduces training and offers finer- finer-grained grained control via fixed attention cubes, a design that influences the range of applicable resolutions. To bridge these varied trade-offs, we propose Adaptive Block-Sparse Attention (ASA),
3

---

Published as a conference paper at ICLR 2026
X
Sparse Generator
Distribution Matching
VgDKL
Real Score
Gaussian Noise
Fake Score
Fy.5 (Go244
Deterministic Sampling (Backward)
Diffusion Loss
Diffusion Process(Forward)
Figure 1:
&z). The training mechanism of BLADE within a single distillation interval (8,-1, The
Sparse Generator (Ga) denoises the input xs, to produce the sample Xt,-1' Crucially, this output then re-corrupte with Gaussian noise to create an intermediate sample Xt,- dedicated Fake Score model evaluates this re-noised sample. Its output is contrasted with the score from the Real Score model (which is the pre-trained teacher model) to compute the Distribution Matching Loss 1.).) This loss directly updates the student generator, forcing it to align its generation trajec- tory with the teacher's at a distributional level.
dynamic, content-aware mechanism that generates hardware-friente sparsity masks on-the-fly, providing unified solution for both training-free and distillation -base scenarios.
METHOD
3.1
OVERALL ARCHITECTURE
BLADE is holistic framework for accelerating video diffusion models by synergistically inte- grating dynamic sparsity into a powerful step distillation process. illustrated in Figure 1, our architecture is based student-teacher paradigm. The teacher, før is pre-trained, high-quality but computationally expensive multi-step diffusion model. The student, Gg. initially shares the same Transformer-basee (DiT) (Peebles & Xie, 2023) architecture and weights the teacher. Our key innovation, designed to enable few-step generation, is the replacement of the standard self- attention layers within the student with our proposed Adaptive Block- -lock-S Attention (ASA) mech- anism. The training process follows the Trajectory Distribution Matching (TDM) (Luo et al., 2025) paradigm. In each iteration, the sparse student model Go generates an intermediate trajectory. This trajectory is then guided to match the distribution of the teacher's trajectory via data-free score distillation loss. This ensures the student learns produce high-quality outputs while operating under the computational constraints imposed by ASA.
3.2 PRELIMINARIES: TRAJECTORY DISTRIBUTION MATCHING (TDM)
Trajectory Distribution Matching (TDM) (Luo al., 2025) is an advanced distillation framework designed to create efficient, few-step diffusion models. Its core idea is align the entire generation trajectory of a student model with that teacher model at the distribution level, rather than re- quiring exact instance-level match. This is operationalized through a data-free score distillation process that relies on three key components:
1. The pre-trained teacher model fø- which provides the real data score 86-
2. The student generator Gg. which generates samples through K denoising steps defined by the time sequence (t, K where 0 to ta lx = T.

---

Published as a conference paper at ICLR 2026
3. A fake score model fø. which is trained to approximate the student's intractable sample score and provides the estimated score Sy.
The training process involves two intertwined objectives, one for the fake score model and one for the student generator.
Training the fake score model (f+). The score distillation process requires the student model's score function Vx po,()x),, which intractable. TDM resolves this by introducing fake score model Ju. neural network trained concurrently to approximate the student's 's score. To ensure this approximation is accurate, the fake score model parameterized as denoiser and trained using the following objective:
K Σ
((w)
Ex, pp.., Ex, w (x(,, Io (x):)) xr,
where the -gueent-eenerated sample xt, at timestep t, serves as the denoising target, A noisy sample Xj is then created by perturbing Xt, to intermediate timestep j. and the model learns recover X1, from X..
Training the student generator (Ge). With access to both the teacher's score so and the stu- dent's score estimate &ø (derived from the denoiser output fu), the student generator Go can be trained. The objective is to minimize the KL divergence between the student's 's trajectory distri- bution and the teacher's trajectory distribution. This alignment is performed across the K sampling intervals. The core distillation loss is:
K-1
(
Σ
X DRI. (x),s,())ll (x),,,
In practice, minimizing this KL divergence is achieved by matching the scores. The gradient this objective is computed by replacing the student's 's intractable true score with the estimated score sp- This results the following gradient approximation:
Ox:: 00
VoL(O) = Σ
V (xg) s (x)
(4)
i=
j=t,
K-1 Σ
Fx::
Σ Xg(sp(X,.)) so(x,, 00
Following the TDM framework (Luo et al., 2025), this process is made both practical and memory- efficient through two key implementation choices. First, ensure the distillation intervals [t,, f++1) are non-overlapping. This design allows a single fake score model f to be sufficient for all stages. Second, conserve GPU memory, backpropagation through the student generator is constrained to only one ODE step at time.
3.3 ADAPTIVE BLOCK- -PARSE ATTENTION (ASA)
A core design of our work is the Adaptive -lock-Sparse Attention (ASA) mechanism, developed upon block-sparse attention. ASA leverages the prior that neighboring tokens in the latent repre- sentations of video often share similar semantics, which makes it reasonable for queries within the same block to share a mask, and pooling operation can keep meaningful semantic information. By allowing each query in block to selectively attend only the most relevant keys and values, ASA achieves superior performance over traditional static masks. In the following, we provide a detailed introduction to method.
token rearrangement. The input matrix Q, K, and V, repre- senting flattened sequence of video tokens, are first restructured into blocks. A critical preliminary step rearranging the tokens to preserve their inherent spatial locality, which is often disrupted by standard raster-scan tokenization. To this end, employ Gilbert space-fillin filling curve (Zhang al.,
Preprocessing:
5

---

Published as a conference paper at ICLR 2026
Step1. Efficient Atteation Prober (b-125, 28, k-1
Step2. Threshold-basee Mask Generation &
Takeu Gatel Contral
Block-Sparse MM
(pelnns) parligggr
Markk
(3444 aati
softnax(O"") (
K
block vine
Redaa adilitee
01
01
0.7
Sklyped region
Regien to compated
82
833
Tarobakk-ead stection
sample
0.5
Max pooling
Recken conpuned
Sktyped
Randomly semple
loketa bluck
P
M
Figure 2: The two-stage process for Adaptive Block-Sparse Attention mask generation. (1) The Efficient Attention Prober samples a few representative tokens (c.g., k 16) from each block to compute low-cost max-pooled attention matrix P. (2) The Threshold- basddd Mask Generator sorts the scores P and selects the top blocks that contain a specified threshold (e.g., 95%), producing the final binary mask M. To enrich the context for training, concatenating it with pooled version: denotes mean pooling over window of size n. During attention computation, the original K region uses the binary block mask M, while the pooled region receives a fixed additive mask of In n,, softly guiding attention without disrupting sparsity.
augment the key matrix K by Concat( (K, MeanPool, (o),,,K)), (K)), where MeanPool,, (K)
2025a) to reorder the tokens before blocking. This ensures that the resulting blocks are more mantically coherent, containing spatially contiguous information, which significantly enhances the effectiveness of the subsequent threshold-basee pruning.
Step 1: Efficient block importance estimation. Conceptually, one could compute the full, dense attention matrix P = softmax QKT /vdx), partition it into blocks of size b b, and then apply max-pooling over each block. This would yield a downsampled importance matrix, Pimps where each element signifies the importance of the corresponding block. A sparse mask could then be generated by applying a threshold to each row Pimp. allowing each query block to focus only on the most salient key-value blocks. However, the initial computation of the full matrix P makes this method impractical for achieving actual acceleration.
To overcome this limitation, propose efficient online approximation. Instead of the full matrix, we sample & representative tokens (k < b) from each block of Q and K to form smaller matrices, Q. and K.. We then compute much smaller, low-resolution attention map. Papprox. from these sampled tokens. The block importance matrix Pimp is derived from this approximate map. This approach reduces the complexity of mask generation from O(N2) to approximately (k.2. (k/b)2), where N is the sequence length. This makes online mask generation feasible. This sampling-bais scheme not only reduces the complexity of mask generation, but also improves the accuracy of block importance estimation. Unlike SpargeAttention (Zhang et al., 2025a), which collapses each block into single mean token and derives importance from a coarse N/b x N/b attention map, ASA retains intra-block structure by computing attention over sampled tokens and then applying max- pooling within each sub-block. This finer-grained approximation enables ASA to better capture salient patterns within each block. Detailed experimental comparisons with SpargeAttention are provided in Table 8.
Step 2.1:
Sparse mask construction. Once the block importance matrix Pimp is obtained, we generate the final sparse attention mask based threshold- masking strategy. Specifically, we sort each row of Pimp in descending order and include the minimal number of key blocks such that their cumulative attention scores exceed specified threshold (e.g.. 90%). This tireshol-bbase dynamic pruning preserves the most salient attention paths while skipping less informative blocks, offering flexible trade-off between accuracy and efficiency.

---

Published as a conference paper at ICLR 2026
Table 1: Video Quality Evaluation VBench-2.0.
Model
Method
Sparsity Total Creativity Commonsense Controllability Human Physics Speedup
Baseline
0,534
0.458
0.523
0.341
0.808
0.539
1x
CogvideoX-SDD
FA2
0.539
0,458
0.498
0.354
0.813
0,570
x
ASA_G
0.82
0.569
0.546
0.514
0.367
0.802
0.618
8.89
Baseline
0.563
0.508
0.549
0.338
0.820
0,600
1x
FA2
0.580
0.631
0.485
0.311
0.841
0.631
9.37x
1...B
STA
0.74
0.528
0.504
0.471
0.265
0.855
0.543
10.53x
ASA_G
0.8
0.570
0.472
0.532
0.312
0.918
0.617
14.10x
Note: Baseline refers to the official 50 steps baseline. All methods except the Baseline are distilled to 8 steps using TDM.
The resulting binary mask is then used to restrict the computation of attention during both training and inference, ensuring that the majority of computational resources are focused on the most relevant interactions. We provide the pseudocode of ASA Algorithm 3.
Step 2.2: Computation. Based on this mask generation technique, we introduce two variants of our mechanism tailored to different application scenarios:
1) Standard ASA (Training-Free Free): In its primary form, the generated binary sparse mask M is directly integrated with block-sparse attention kernel. This variant can be applied to pre-trained models without any retraining, offering direct inference speed-up by focusing computation on fine-grained, salient information.
2) ASA with Global Tokens (for Distillation): To mitigate potential global information loss high sparsity ratios, we introduce an enhanced variant. We augment the Key (K) and Value (V) by creat- ing set of "global tokens". These are generated by applying mean pooling
window of size n, reducing the sequence length to 1/n of the original lengths of K and V. The augmented K are formed as Kang Concat( K, MeanPool, (K)) (and similarly for V). During attention computation, a query's interaction with the original region is governed by the binary sparse mask M, preserv- ing fine-grained details. For the augmented "global tokens" region, of (n( to the pre-softmax scores. This bias compensates for the averaging effect of mean pooling, ensuring that each global token contributes attention as if it represents the full importance of its constituent fine aee-grained tokens. This softly guides every query to maintain awareness of the global context, preventing catastrophic information loss when most blocks are pruned.
apply fixed additive mask
Throughout this paper. we refer to the standard implementation as ASA and the augmented version as ASA with Global Tokens (ASA in short).
3.4 SPARSITY-AWAR DISTILLATION
A cornerstone of the BLADE framework is the principle of sparsity-awant distillation. Unlike pre- vious approaches that apply sparsity as a post-training compression step, we integrate ASA directly into the TDM training loop. At every training iteration, the student model Ge generates its trajec- tory using the ASA mechanism. The distribution matching loss then updates the student's 's weights improve its output quality given these dynamic sparsity constraints. This co-design strongly reg- ularizes the model, forcing it to learn robust, semantic representation that often yields superior perceptual quality.
4 EXPERIMENT
4.1 EXPERIMENTAL SETUP
Models. We evaluate BLADE on two text-to-video diffusion models: CogVideoX-5B (Hong al., 2022) and 1-n2.1-1.3B (Wan et al., 2025). These models represent different architectures and scales, allowing test the generalizability of our approach.
7

---

Published as a conference paper at ICLR 2026
Dataset. Our training process is guided by dataset of 10,000 text prompts. These prompts were sampled from the JourneyDB benchmark (Sun et al., 2023) and subsequently enhanced for quality and diversity using the Qwen2.5-3B In-struct (Team, 2024) model.
Implementation details. Unless otherwise specified, we use a block size b = 128, k = 16 sam- pled tokens per block for the attention prober. Distillation is typically run for 100-200 iterations. Experiments on Cog VodeoXX5BB and Wan2.111.333 were conducted on a cluster of 8 A800(80GB) GPUs. We use suite of standard metrics to evaluate performance: Peforrmance:BBBnch- VBench-2.0 (Zheng et al., 2025), SSIM & PSNR (Hor & Ziou, 2010), Human Evaluation.
(Huang al., 2024),
Compared methods. ASA_G, ASA, STA (Zhang al., 2025b), and RaA (Li et al., 2025) respec- tively denote using our adaptive attention, its training-free variant. Sliding Tile Attention (Zhang al., 2025b), and Radial Attention (Li al., 2025). FA2 refers to FlashAttention-2 (Dao, 2024).
4.2
MAIN RESULTS: EFFICIENCY AND QUALITY
Our experiments demonstrate that Video-BLADE achieves significant acceleration without compro- mising, and often improving, generation quality.
Quality Analysis. Table 1 presents the VBench-2.0 benchmark results for Cog CogVideoX-5B and Wan2.1-1.3B across several methods, including our proposed ASA.G, the sparse baseline STA, FA2, and the 50-step dense bascline.
For CogVideoX-5BB ASA G delivers consistent and comprehensive improvements across all major quality dimensions. achieves the highest overall VBench-2.0 score (0.569), outperforming both the 50-step baseline and FA2, and leads in Creativity, Controllability, and Physicsall key for gener- ating plausible and engaging video content. Notably, ASA G achieves this performance using only 8 decoding steps over a short 17k-token sequence, resulting in an 8.89x speedup while simultane- ously improving generation quality. These results demonstrate that even with extremely constrained sequence lengths, ASA G achieves robust generation quality.
For Wan2. 1-1.3B, ASA.G continues to show clear advantages. It achieves a strong VBench-2.0. score (0.570), the highest Human Fidelity (0.918), and strong Physics performance, all while operating at just 7.09 % of the original inference time (14.10 speedup). Compared to STA, which shares similar sparsity, ASA, G performs significantly better in almost all metrics. Although FA2 slightly outperforms ASA G in total score, its performance on controllability is weaker and comes at higher computational cost. A gallery-style visual comparison, showcasing video samples across diverse models and inference strategies, is presented in the Appendix F.
An intriguing observation from our results is that BLADE, despite its high sparsity and few inference steps, can surpass the quality of the 50-step dense baseline. We attribute this phenomenon to regularization effect induced by our joint training framework. The long, iterative trajectory of the 50-step teacher can sometimes accumulate numerical errors overfit to noisy, less coherent details. In contrast, our sparsity-aware distillation compels the student model to learn a more direct and stable generation path (a principle that echoes findings in prior works like DMD2 (Yin al., 2024)), forcing it to capture the most essential semantics while implicitly filtering out the "detours" and noise from the teacher's process. The adaptive sparsity further aids this by focusing computation only on the most salient features. We provide a visual corroboration of this effect with attention map analyses the Appendix B. The resulting model is therefore not merely a faster approximation but can be more robust and coherent generator. We evaluate our models on VBench-2.0, which places greater emphasis on semantic Taithfulnessasessingg how well the generated videos preserve high-level meaning rather than just pixel-wise accuracy. This aligns closely with the strengths of our approach.
Efficiency analysis. At the kernel level, our ASA implementation achieves x.30 speedup over the standard dense attention used in the 8-step FA2 bascline (22.21 ms vs. 73.25 ms), benefiting from an effective sparsity rate of 0.798. This low-level gain directly translates substantial end- to-end acceleration: our ASA-based model completes generation in 24.00 seconds, compared to 36.11 seconds for its dense counterpartyielding a 1.504x E2E speedup.
Notably, while the kernel speedup is more than xx, the E2E gain is sub-linear. This suggests that attention is no longer the dominant bottleneck in the distilled model; instead, other operations (e.g..
8

---

Published as a conference paper at ICLR 2026
Table 2: Efficiency analysis on Wan2. 1-1.3B
Table 3: Comparison of training-free sparse tention methods on Wan2.1-1.31 (8-step dis- tilled model).
(test on an H20).
Metric
FA2-50
FA2-8
ASA--8
Method
Sparsity
PSNR
SSIM
Kernel Time (ms)
73.25
73.25
22.21
Kernel Speedup
1.00x
1.00x
3.30x
STA
0.74
16.72
0.6190
E2E Time (s)
338.41
36.11
24.00
SVG
0.75
16.68
0.6390
E2E Speedup
1.00x
9.37x
14.10
ASA
0.75
19.55
0.7433
Note: The number suffix (e.g. FA2-50) indicates the
RaA
0.50
22.07
0.8191
number of inference steps used in each model.
ASA
0.50
22.20
0.8290
the VAE encoder/decoder and non-attention layers within the transformer) begin to dominate the runtime. This shift validates the effectiveness of our targeted kernel optimization in minimizing attention overhead within modern diffusion pipelines.
4.3 COMPARISON OF SPARSE ATTENTION MECHANISMS
To isolate the performance of the ASA mechanism itself, we compare it against other sparse attention methods in a training- ffree inference setting
1an2.111.333 For sparse inference, the first two steps adopt FA2, while the remaining steps use sparse attention. Table shows that at a similar sparsity level, ASA significantly outperforms STA (Zhang al., 2025b), RaA (Li al., 2025) and SVG (Xi et al., 2025) in both PSNR and SSIM, establishing its superiority as dynamic attention mechanism. Videos sampled by different methods are shown in Figure 3. Further ablation studies, including human evaluation results, are provided in the Appendix A.
FA2
ASA (Ours)
STA
SVG
Figure 3: Comparison of generated videos at frame 0,40,80 for the prompt "A tranquil tableau of bedroom". ". Each row shows the same frame index across 4 methods.
5 CONCLUSION AND FUTURE WORK
In this paper, we have presented BLADE, novel framework that effectively addresses the critical efficiency challenge in video diffusion models. By synergistically co-designing a dynamic, content- aware adaptive block-sparse attention mechanism with data-free trajectory distribution matching distillation process, our method achieves significant inference acceleration without sacrificing gen- eration quality. Our results demonstrate that by making the model sparsity aware during training, often achieves superior visual quality and intrinsic faithfulness (Zheng et al., 2025) compared to both the original multi-step teacher and densely distilled student model. Our contributions

---

Published as a conference paper at ICLR 2026
validated through extensive experiments on various video models, demonstrating marked improve- ments in kernel-level efficiency, end-to-end inference speed, and generation quality as measured by both automated benchmarks (VBench-2.0) and human evaluations.
Limitations and future work. While BLADE exhibits strong performance, we acknowledge sev- eral limitations that point to promising directions for future research. First, current experiments are limited to video sequences of moderate length. Extending and validating the ASA mechanism for generating minute-long videos with hundreds of thousands of tokens remains an important next step. Additionally, our current ASA kernel is implemented in Triton for simplicity, which prevents from fully realizing its theoretical speedup. Future work will focus on developing a more opti- mized CUDA implementation to better leverage the efficiency potential of ASA. These directions underscore the importance of evaluating ASA in more demanding settings and exploring further architectural enhancements. Lastly, the idea of sparsity-aware training as a form of regularization shows promise and could extended to other generative domains beyond video synthesis, such as 3D content generation and high-resolution image synthesis.
ACKNOWLEDGEMENT
This work was supported by the Central Media Technology Institute, Huawei Technologies, through a technology cooperation.
REFERENCES
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models, 2023. URL https:/: http///axvvv//ea//////ee//aaae wttp://///........////
08818.
Pu Cao, Feng Zhou, Qing Song, and Lu Yang. Controllable generation with text-to-image diffusion models: A survey, 2024. URL https: : /tt///aaxvvvv//e////a//eeee.aav/
hbtp:/ab///200 wwtp:///rra/bb/////00/ 04279.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In ICLR, 2024.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher R. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. URL httttth//axxii.. org. abs/ 2205.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis, 2021. URL
https: //tl///aaxiii /awxivvivvv//v////////aaa//a/ava org/ als// /2105. 05233.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Mller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion En- glish, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformens for high-resolution image synthesis, 2024. URL https://arxiv..or 03206.
lan J.
Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014. URL https: hacbbb///4.... 2661.
/ wttp://rrgabb/aaee/.
Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Lingjie Liu, and Josh Susskind. Boot: Data-free distillation of denoising diffusion models with bootstrapping, 2023. URL https:////ax .rg/ abs/ 2306. 05
Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion mod- els for high- figh-fidelity long video generation, 2023. URL 13221.
org. 22111
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020. URL https:/ //tpii/aaiivvvv//i orggabs /
a00p//0s//0006///6///// 6006. 1 11239.
10

---

Published as a conference paper at ICLR 2026
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scate scale pre- training for rext-to-video generation via transformers. arXiv preprint 1rXi::2200.158688 2022.
Alain Hor and Djemel Ziou. Image quality metries: Psnr vs. ssim. 2010 20th International Conference on Pattern Recognition. pp. 2669, 2010. doi: 10. 10.1109/ICR.2210..777
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang. Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite for video generative models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- lion, 2024.
Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models, 2022. URL https: //axivv. www.eiiwwwwiooooooooooooo abs 2206. 00364.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong. Xin Li, Bo Wu, Jianwei Zhang, Kathrina Wu, Qin Lin, Junkun Yuan, Yanxin Long, Aladdin Wang, An- dong Wang, Changlin Li, Duojun Huang, Fang Yang, Hao Tan, Hongmei Wang, Jacob Song, Jiawang Bai, Jianbing Wu, Jinbao Xue, Joey Wang, Kai Wang, Mengyang Liu, Pengyu Li, Shuai Li. Weiyan Wang, Wenqing Yu, Xinchi Deng, Yang Li, Yi Chen, Yutao Cui, Yuanbo Peng, Zhen- tao Yu, Zhiyu He, Zhiyong Xu, Zixiang Zhou, Zunnan Xu, Yangyu Tao, Qinglin Lu, Song- tao Liu, Dax Zhou, Hongfa Wang. Yong Yang, Di Wang, Yuhong Liu, Jie Jiang, and Caesar Zhong. Hunyuanvideo: A systematic framework for large video generative models, 2025. URL https:: / org/ abss 2412. 03603.
Black Forest Labs, Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, Zion English, Patrick Esser, Sumith Kulal, Kyle Lacey, Yam Levi, Cheng Li, Dominik Lorenz, Jonas Miler, Dustin Podell, Robin Rombach, Harry Saini, Axel Sauer, and Luke Smith. Flux.1 kontext: Flow matching for in-context image generation and editing in latent space, 2025. URL https: /ttp/a/aiii///.////////eee/...a. org/ abs / /2..//55.......
Xingyang Li, Muyang Li, Tianle Cai, Haocheng Xi, Shuo Yang, Yujun Lin, Lvmin Zhang, Songlin Yang, Jinbo Hu, Kelly Peng, Maneesh Agrawala, Ion Stoica, Kurt Keutzer, and Song Han. Radial attention: on log n) sparse attention with energy decay for long video generation, 2025. URL https:/ /taps:://axiii org/ als///////////// 6506. 9852.
Xingchao Liu, Xiwen Zhang. Jianzhu Ma. Jian Peng, and Qiang Liu. Instaflow: One step is enough for high-quality diffusion- text-to-image generation, 2024. URL htttttts///aiiii /rrg abs/ /2099 06380.
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed, 2021. URL httttt/////aaattt http//aawivvraaaeaaaeeeae/sseeee / 2101.11000 02388.
Yihong Luo, Tianyang Hu, Jiacheng Sun, Yujun Cai, and Jing Tang. Learning few-step diffusion models by trajectory distribution matching, 2025. URL https:ttpt.... http///www....oraa///////// 06674.
Xin Ma. Yaohui Wang, Xinyuan Chen, Gengyun Jia, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation, 2025. URL https://
arxiv. orgo /tbb//4411 03048.
William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023, URL https: //arxiv. hbtp/bbs/22212 09748.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models, 2022. attp/bbss22222 00512.
URL htttts://aaxii....../////
Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rom- bach. Fast high-resolution image synthesis with latent adversarial diffusion distillation, 2024.
URL htw.t:ttp:/eeee /eattss:/arxiv...o .rgg wtw./bbs///eeeee /2003. 12015.

---

Published as a conference paper at ICLR 2026
Hui Shen, Jingxuan Zhang. Boning Xiong, Rui Hu, Shoufa Chen, Zhongwei Wan, Xin Wang, Yu Zhang, Zixuan Gong, Guangyin Bao, Chaofan Tao, Yongfeng Huang, Ye Yuan, and Mi Zhang. Efficient diffusion models: A survey, 2025. URL https:// http:aaxiiiii...... abs/2222. 06805.
Yang Song. Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations, 2021. URL https: //axivi org/ aaaaabbs/22111 13456.
Yang Song. Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469, 2023.
Kegiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, Jifeng Dai, Yu Qiao, Limin Wang, and Hongsheng Li. Journeydb: A benchmark for generative image understanding, 2023. URL https://arxiv, .rg. abs/
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL http:///://
github. httpti/////////o//...o...ooooott http//log//we/222 qwen2... 5/.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng. Jiayu Wang, Jingfeng Zhang. Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng. Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang. Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang. Wei Wang. Wenmeng Zhou, Wente Wang. Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang. Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng. Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and advanced large-scale video generative models. arXiv preprint 25X3v:2503.200 20314, 2025.
Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, Jianfei Chen, lon Stoica, Kurt Keutzer, and Song Han. Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity, 2025. URL /tttp:://rxxv..or org/ als 2502. 01776.
Zhen Xing, Qijun Feng. Haoran Chen, Qi Dai, Han Hu, Hang Xu, Zuxuan Wu, and Yu- Yu-Gang Jiang. A survey on video diffusion models, 2024. URLhttps:: arxi www.rwwwww....aoo.ee..eooeeeee/e abs /211 00..10000 1064 7.
Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, and Song Han. Xattention: Block sparse 166288,..16......
attention with antidiagonal scoring, 2025. URL https: //wt:/aaaaav///aa///aa/aaaaawava
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-tovvideo diffusion models with an expert transformer. arXiv preprint arXiv:2408.0610 060722 2024.
Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang. Eli Shechtman, Fredo Durand, and William T Freeman. Improved distribution matching distillation for fast image synthesis. In NeurlPS, 2024.
Zhihang Yuan, Hanling Zhang, Pu Lu, Xuefei Ning, Linfeng Zhang, Tianchen Zhao, Shengen Yan, Guohao Dai, and Yu Wang. Ditfastattn: Attention compression for diffusion transformer models, 2024. URL https: / arxiv. wwo//og/sss... 220.. 08552.
Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. Spargeattention: Accurate and training-free sparse attention accelerating any model inference, 2025a. URL hww.t:tpp//oo //ap:ii////aav./a/aa hwtp:/orgg////ooot......o. abs/ 18137.
Peiyuan Zhang, Yonggi Chen, Runlong Su, Hangliang Ding, Ion Stoica, Zhengzhong Liu, and Hao Zhang. Fast video generation with sliding tile attention, 2025b. URL https: : //wxivi.oooo../ org/ 22022 04507.
12

---

Published as a conference paper at ICLR 2026
Peiyuan Zhang, Haofeng Huang, Yongqi Chen, Will Lin, Zhengzhong Liu, lon Stoica, Eric Xing, and Hao Zhang. Vsa: Faster video diffusion with trainable sparse attention, 2025c. URL https: //arxiv. /tt.:aas//05.111.1...
13389.
Dian Zheng, Ziqi Huang, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Yuanhan Zhang, Jingwen He, Wei-Shi Zheng, Yu Qiao, and Ziwei Liu. VBench-2.0: Advancing video generation benchmark suite for intrinsic faithfulness. arXiv preprint arXiv:2503.2175,, 2025.
Jianbin Zheng, Minghui Hu, Zhongyi Fan, Chaoyue Wang, Changxing Ding. Dacheng Tao, and Tat-Jen Cham. Trajectory consistency distillation: Improved latent consistency distillation by semi-linear consistency function with trajectory mapping, 2024, URL htttttss///aiii....//./ abs/ 19159.
13

---

Published as a conference paper at ICLR 2026
A ADDITIONAL EXPERIMENTS
We conducted comprehensive set of experiments to analyze the contribution of each component and investigate the performance of BLADE under broader settings.
A.1 IMPACT OF REARRANGEMENT STRATEGY
Table 4: Ablation results for the token rearrangement strategy, evaluated with the VBench- 1.0 qual-
ity score.
Configuration
Quality Score
Without Rearrange With Rearrange (Ours)
0.779
0.788
We validated the importance of the Gilbert rearrangement strategy. As shown in Table 4, CogVideoX-5t model distilled (using ASA) with this strategy achieve higher VBench-1.0 quality score (0.788) compared to those without (0.779), confirming role in preserving spatial locality for more effective block-wise pruning.
A.2 IMPACT OF ADDITIVE MASK AND GLOBAL TOKEN ASA_G
Table 5: Effect of Global Token (G) and Additive Mask (AM) ASA on CogVideoX-5B (VBench-
2.0).
Config
Sparsity (%)
VBench-2.0
0.8
0.539
ASA.G
0.82
0.569
ASA G_W/o_MM
0.82
0.559
Baseline-50
0.534
Note: G Global Token, AM = Additive Mask. Baseline-50 is the original 50-step FA2 model.
We conduct ablation studies on VBench-2.0 to validate the effectiveness of our key designs: the Global Token (G) and the Additive Mask (AM). shown Table 5, our base model, ASA, al- ready surpasses the baseline (0.539 vs. 0.534). Upon integrating the GT, the performance of our model, ASA significantly leaps to 0.569. This substantial gain underscores the critical role of GT in aggregating global spatio-temporal information. Furthermore, removing the AM from the full model (i.e., ASA_G_ A.A._W//AAMM leads to a noticeable performance drop to 0.559, which confirms the necessity of AM in preserving model integrity under the sparse attention mechanism. Collec- tively, these results demonstrate that both GT and AM are indispensable components, synergistically contributing to the superior performance of our final model.
A.3 IMPACT OF BLOCK SIZE CONFIGURATION
Table 6: Ablation study on block size configuration for ASA on Wan2.1-1.3B (sparsity ratio 0.8). Smaller block sizes offer finer granularity but ineur higher overhead.
Block Size (QxK)
PSNR
SSIM
LPIPS |
64 64
22.24
0.818
0.144
128 x 64
22.05
0.803
0.162
128 128
21.75
0.793
0.169
Block size determines the granularity the attention masking, acting as a pivotal hyperparameter in the ASA mechanism. We analyze three block configurations (64 x 64, 128 64, and 128 128)
14

---

Published as a conference paper at ICLR 2026
under constant sparsity ratio of 0.8. Table 6 demonstrates monotonic improvement in generation quality as block size decreases. Specifically, the 64 x 64 configuration outperforms the coarser 128 x 128 setting by 0.49 in PSNR and 0.025 in SSIM.
This performance gain stems from the finer granularity smaller blocks, which allows the ASA mechanism preserve salient regions with higher precision, adapting more effectively local mantic structures. However, finer blocking introduces non-negligible computational overhead dur- ing mask generation and memory access. Consequently, adopt 128 128 in our main experiments as a strategic trade-off, prioritizing system throughput while maintaining competitive generation quality.
A.4 IMPACT OF ATTENTION THRESHOLD
Table 7: Ablation study on the attention threshold (r) in ASA. Experiments conducted on 1-1.3B (50 steps; ASA enabled after a 12-step warm-up). governs the sparsity-quality frontier.
Threshold (r)
Sparsity
PSNR
SSIM
LPIPS
0,9
0.73
23.93
0.856
0.106
0.8
0.80
22.08
0.812
0.153
0.7
0.84
20.46
0.750
0.209
0.6
0.87
19.20
0.712
0.243
0.5
0.92
15.48
0.583
0.436
The threshold 7 directly modulates the trade-off between computational sparsity and generation fidelity. By varying from 0.5 to 0.9, we observe in Table 7 that higher thresholds preserve more attention blocks, naturally leading to superior metrics. threshold of = 0.9 yields near-dense quality (PSNR 23.93) but results in lower sparsity of 0.73, Conversely, aggressive pruning with T = 0.5 achieves high sparsity (0.92) but causes a structural collapse in quality (PSNR drops to 15.48), indicating that critical attention contexts are being discarded.
We identify т = 0.8 as the optimal operating point, achieving a sparsity of 0.80 without severe degradation in perceptual quality (SSIM > 0.8). This ablation further shows that ASA supports flexible deployment: higher thresholds are suitable for quality-critical applications, while lower thresholds benefit latency-sensitive scenarios where minor artifacts are acceptable. Visual examples across different sparsity levels are provided in Appendix F.1.
A.5 SCALABILITY LARGER MODELS WITH LONGER SEQUENCES: WAN2.1-14B
Table 8: Comparison of training- free sparse attention methods Wan2.1-14B. ASA achieves the best trade-off between sparsity and visual quality.
Method
Sparsity
PSNR t
SSIM
LPIPS
SVG
0.75
24.86
0.823
0.094
Sparge
0.77
24.03
0.808
0.117
STA
0.75
25.00
0.845
0.079
ASA (Ours)
0.77
26.05
0.865
0.050
To validate the scalability of ASA to larger architectures and longer video sequences, benchmark it against state-of-the-ar training-fre frree sparse attention methods on W-n2.1-14B. As shown in Ta- ble ASA maintains superior performance at comparable sparsity levels ( 0.75-0.77), Notably, our method achieves an exceptionally low LPIPS of 0.050, outperforming the strongest baseline, STA, by a significant margin (0.029 improvement). Compared to SpargeAttention,, ASA improves PSNR by over 2.0 and reduces perceptual loss by more than 50% (0.050 vs. 0.117), underscoring its capability to preserve fine-gre mme-grained semantic details even when processing extended contexts. These results, consistent with the 1.3B evaluation, demonstrate that ASA's dynamic masking strategy is
15

---

Published as a conference paper at ICLR 2026
robust across both model scales and sequence lengths, effectively mitigating the quality degradation observed in static pruning methods on targe-scale, context foundation models.
A.6 FEASIBILITY ANALYSIS AT LOW-STEP INFERENCE
To investigate the feasibility of BLADE under strict computational constraints, we extend the distil- lation process low-step regime of four sampling steps. Table 9 directly compares these 4-step sparse models against the standard 50-step dense baselines.
Notably, method achieves substantial end-to-end inference speedups of 15.2x Videoo- 5B and 17.6x on Wan2. 1.1.3B. Despite this aggressive reduction in sampling steps, both models maintain superior VBench Total scores compared the 50-step baselines (CogVideoXX 0.562 vs. 0.534; Wan2.1: 0.570 vs. 0.563). These results validate that our joint training framework remains highly effective even few-step scenarios, ensuring high-quality generation alongside extreme ac- celeration.
Table 9: VBench-2.0 comparison of 4-step BLADE models against 50-step baselines.
Model
Method
Creativity Commonsense
Controllability
Human
Physics
Total
Baseline
0.458
0.523
0.341
0.808
0.539
0.534
CogVideoX-SB
ASA G
0.425
0.553
0.389
0.840
0.606
0.562
Baseline
0.508
0.549
0.338
0.820
0.600
0.563
W-n2.1-1.3B
ASA G
0.467
0.564
0.333
0.897
0.594
0.570
A.7
HUMAN EVALUATION RESULTS
Table 10: Human preference: 8-step models vs. 50-step baseline.
Comparison
Win
Lose
Tic
Cog Vogeeeo0
ASA_G (Ours) vs. Baseline
Won2.1-1.3B
ASA_G (Ours) vs. Baseline
10
12
28
STA vs. Baseline
0
26
24
We conducted a human preference study to evaluate our efficient 8-step ASA_G A_G modelssparsity ratio 0.8) and 8-step STA model(sparsity ratio 0.74) against the standard 50-step baseline. The evaluation was performed using 50 diverse video prompts. The aggregated results are shown in Table 10.
For the Cog X5B model, ASA_G was preferred or rated equally in 80% of comparisons, while achieving an 8.89 x speedup in inference time. For Wan2.1-1. ASA_G achieved a 56% tie rate, yielding 76% non-inferiority rate overall, while reducing inference time just 7.09 % of the base- line. In contrast, STA was consistently outperformed by the baseline, with O wins and a 52% loss rate. These results highlight that ASA G maintains high visual fidelity despite aggressive accelera- tion, validating its effectiveness for practical deployment.
B MASK VISUALIZATION
To elucidate the mechanism by which our proposed sparse attention method achieves both significant acceleration and enhanced quality in video generation, conducted a visualization analysis of the model's 's internal attention patterns. We hypothesize that constraining the model to operate within a limited computational budget compels it to disregard low-information, redundant areas (c.g., static backgrounds) and instead concentrate its focus more efficiently on core semantic objects within the scene.
16

---

Published as a conference paper at ICLR 2026
(a) Sparse Mask
(b) Full Attention
(c) Mask on Frame
(a) Our Sparse Mask
(b) Full Attention
(c) Mask on Frame
(a) Our Sparse Mask
(b) Full Attention
(c) Mask on Frame
Figure 4: Visualization and analysis of attention masks. Our sparse method (a) is shown to capture the most salient regions identified by full attention (b), effectively focusing on key semantic objects within the frame (c).
Figure offers a direct and intuitive validation of this hypothesis. Each row illustrates the attention behavior for single sample, comparing our method against standard full attention baseline across diverse scenes. Specifically, the three sub-figures within each composite image correspond to:
(a) Our Sparse Mask: This visualizes the attention mask produced by our sparse method for single query patch Q. The white areas denote the spatial positions of key patches K that are retained for the attention score calculation. Conversely, the extensive black regions the positions pruned by our method, where attention is not computed, effectively masking out non-salient information prior the softmax operation.
(b) Full Attention Map: As a baseline, this map displays the raw attention weight distribution for the same query patch without sparsity constraints. Brighter colors (e.g., yellow) indicate higher attention scores.
(c) Mask Overlaid on Frame: The sparse mask, highlighted as cemi-transparent red overlay, is superimposed on the actual video frame to intuitively show the spatial locus of attention.
Across the three distinct scenes (a bird, a cartoon, and an elderly man), it is evident that although our method prunes a substantial number of computations (as shown in column a), the retained attention regions precisely cover the core semantic objects, such as the bird's beak, the cartoon character's tail, and the man's beard.Ne Notably, the regions selected by our sparse mask exhibit a high degree of overlap with the highest-scoring areas in the full attention map. This provides strong evidence that sparsity strategy efffectiely identifies and preserves the most salient semantic information while filtering out redundant background noise. This focusing mechanism offers a plausible explanation for the unexpected improvement in the model's generation quality.
17

---

Published as a conference paper at ICLR 2026
C PSEUDOCODE
Inspired by the pseudocode in ScerAttention, adapt to implementation to get the max- pooling of attention map P from the downsampled Q, input. The process is detailed in Algorithms 2 and 1.The pseudocode of ASA is detailed in 3.
Algorithm GetMaxXPolledAttaaa
Input: Q, K RHxSxd pooling size b, scale factor
Output: e [ T, [
Initialize M as -co with shape (H,
2: Initialize l as with shape (H, $)
3: Initialize R as -0o with shape (H, S,T.))
4: for each head h do
5: Split Qn K₁, into T, blocks: Q1 Q....., Qr. Ki...., Kr.
6: for
T, do
7:
M +- M(,, (i - 1) * b ij
8:
L + E(h, (i 1) b:
9:
R R(h, (i - 1) b b.:]
10:
for j + to T, do
11:
Sij + Q Kj
12:
my + rowmax(s;)). Poz + exp(sis mij)
E, rowsum P.j), Maew + max(M, mij)
13:
1 M Mine . m mar Ris
14:
15:
M -mnw: R(::5] mij
16:
end for
for
j
1 to T, do
18:
8ij 6- RM
8ij +- 8ig1
19:
i max(sii)
20: end for
end for
22: end for
Algorithm 2 Compute Block Importance Score
Input: Query Q, Key K E #xkdd block size = 128, tokens per block & = 16, scale factor Output: P #x2, where T, [S/b]
I: Make length divisible by b: Qp + Pad(Q, b). K2 < Pad(K,b) b)
2: Sample & tokens per block:
3: Q f BlockSample((p... k)) K t (hockSampleKKp.bb k)
4: P + GelMax K,kks)
18

---

Published as a conference paper at ICLR 2026
Algorithm 3 ASA Mask Generation
Require: Q. E IR xd, block size b, sample size k, threshold - 1: Rearrange tokens using Gilbert curve
2: Partition Q, K into N₂ N/b blocks
Randomly sample & tokens from each block to get Qs Kg E Rxxd
4: Compute attention: P = softmax(Q,K] / /vd)
5: MaxPool over k blocks to get mimp E R RNN.M.
6: for each row in Piup do (im
7: Pimp(i,3) - Σx Pim (1,p(i,, (i,h) k)
8: Sort Pimple [i descending +
9: Find smallest such that 2j=1 8j 2 then clamp m within the range defined by minimum
and maximum retention ratios
Set Mii.j]] for top m indices, others
11: end for
return Binary mask M
D MODEL CONFIGURATION DETAILS
Table 11 presents the detailed model configurations. For the distillation training phase (iterations 100-200), experiments were conducted on x NVIDIA A800 (80GB) GPUs using DeepSpeed ZeRO-2. The training took approximately 10 hours with global batch size of 128, and the peak memory usage reached 76 GB.
Table 11: Detailed configuration parameters for Wan2.1-1.3B and Cog VideoX-5B models.
Category
Parameter
Wan2.1-1.3B
CogVideoX-5B
Number of Layers
30
42
Number of Attention Heads
12
48
Attention Head Dimension
128
64
In/Out Channels
16
16
Temporal Compression Ratio
4
4
Prediction Dtype
flow
velocity
Model Architecture
Sequence Length
32760
17550
Text Dimension
4096
4096
Patch Size
[1,2,2]
[1,2,2]
Vocab Size
256384
32128
Number of Timesteps
1000
1000
Student learning rate
lc-4
lc-4
Fake model learning rate
5e-4
5e-4
LoRA Enabled
True
Thue
LoRA alpha
64
64
Optimizer
AdamW
AdamW
Adam Betal
0
0
Training & Inference
Adam Beta2
0.95
0.95
Gradient Clipping
1.0
1.0
Seed
42
42
CFG
5
6
Video Resolution
480x832
480x720
Sample FPS
16
8
Gradient Checkpointing
True
True
Training Mode
Zero2
Zero2
19

---

Published as a conference paper at ICLR 2026
E RUNTIME BREAKDOWN OF THE ADAPTIVE OPERATOR
ATTENTION
We provide additional runtime analysis of our adaptive block-sparss attention (ASA) operator, in- cluding per-component breakdowns under two sequence lengths (100k and 18k), followed by sparsity sweep comparing against FlashAttention2 (FA2) and FlashAttention3 (FA3), All results are measured with 30 heads and head dimension 128.
For clarity, the first row each table reports the end-to-end forward time of ASA, while the remain- ing decompose this total into three main stages: (i) Block Importance Estimation (computing the importance scores for each KV block), (ii) Mask Construction (converting importance scores into binary block-sparse mask), and (iii) Block-Sparse Attention Compute (the main attention kernel executed on the selected blocks), plus a small Other category for residual overheads (re- shapes, indexing, etc.).
E.1 COMPONENT-LEVEL BREAKDOWN
At sparsity T 0.8,, simple FLOP-based model suggests that a block-sparse attention kernel should cost roughly (1 r) times the full-attention baseline (FA22, i.e., a 5x theoretical speedup. In our setting, FA2 at 100k tokens takes = 440 ms, so the ideal runtime under 0.8 sparsity would be 88 ms. At 18k tokens, FA2 takes 13.16 ms, yielding ideal 0.8-sparse runtime of 2.63 ms.
Runtime breakdown at 100k and 18k tokens (sparsity 0.8). Table 12 reports component-lewe breakdown of the Adaptive Sparse Attention (ASA) operator at two sequence lengths under the same FLOP-equivalent sparsity of 0.8.
Table 12: Runtime breakdown of Adaptive Sparse Attention (ASA) at two sequence lengths (sparsity 0.8).
100k tokens
18k tokens
Component
Time (ms)
%
Time (ms)
%
Adaptive Sparse Attention (Total)
116.99
100.0%
8.07
100.0%
Block Importance Estimation
9.34
8.0%
1.36
16.8%
Mask Construction
1.03
0.9%
0.41
5.1%
Block- Sparse Attention Compute
106.32
90.9%
6.06
75.1%
Other
0.31
0.3%
0.25
3.0%
100k tokens, sparsity 0.8. At this sequence length, runtime is dominated by the block-sparse compute kernel: over 90% of the ASA time is spent in the block-sparse attention compute, while importance estimation and mask construction together account for only ~ 9% (Table 12). Compared to the FLOP- based ideal of z 88 ms, the measured ASA runtime at 100k tokens is 116.99 ms, i.e., 1.33x higher than the ideal. The resulting zs 29 ms gap can be decomposed into Ro 10.4 ms from importance estimation mask construction (~ 36% of the gap, only 2.4% of the dense FA2 time), and As 18.6 ms from the block-sparse compute kernel itself (padding, load imbalance, and memory/system overhead that are not captured by the simple FLOP count). In other words, long sequences we recover about 3.8x speedup over FA2 (vs. the 5x theoretical limit), and the dominant source of the gap is the block-sparse kernel rather than mask building.
18k tokens, sparsity 0.8. For shorter sequences, the relative share of preprocessing becomes more visible, and fixed kernel overheads dominate the deviation from the ideal FL.OP-level speedup (Ta- ble 12). Here, the FLOP-based ideal runtime is 2.63 ms (i.e., 0.2 13. 16 ms), whereas the measured ASA runtime is 8.07 ms. This corresponds to a 1.63 x speedup over FA2, recovering only 33% of the theoretical 5x limit. The x 5.4 ms gap to the ideal can again be decomposed into 1.8 ms from importance estimation mask construction (~ 33% of the gap, 13.5% of the dense FA2 time) and 3.7 ms from the block-sparse compute kernel. Thus, in the short-sequence regime, the main discrepancy between theoretical and realized speedup comes from fixed kernel and
20

---

Published as a conference paper at ICLR 2026
system overheads (e.g., launch latency, limited parallelism, and padding). while the cost of building the sparse mask remains small in absolute terms.
Overall, the gap between the FLOP- ideal and the realized ASA runtime arises from both the mask--generation stage and the practical inefficiencies of the block-sparse compute kernel. In the short-sequence regime, the relative overhead of mask generation appears larger because several per- layer operations incur fixed costs that do not diminish proportionally with sequence length. As the sequence length increases, these fixed components become quickly amortized, causing the mask- generation share to shrink, and leaving the kerneltheoretical mismatch as the dominant contributor the remaining gap.
F
VISUAL COMPARISON GALLERY
F.1 ASA PERFORMANCE UNDER DIFFERENT SPARSITY LEVELS ON WAN2.1-1.3B
Each panel shows rows corresponding to different threshold values (r E (0.5, 0.6, 0.8, 0.9)), with sparsity levels shown alongside. Lower thresholds exhibit severe structural distortion due to aggres- sive pruning, while higher thresholds restore coherent motion and fine- fine-grained details. Top: results for prompt "[A small boy, head bowed in determination, sprints through a torrential downpour as lightning crackles and thunder rumbles in the distance. Sheets of rain lash the ground, while the faint silhouette of a cozy home in the background glows like a small beacon of safety and warmth. l". Bottom: results for prompt "IA golden retriever in black sunglasses sprints across rain- rain-damp rooftop lerrace, its long fur rippling in the breeze. Seen from distance, the dog bounds toward the camera, tail wagging, as water droplets sparkle on the concrete and its golden coat stands against the overcast sky. ]". The final row in each panel shows the dense-attention baseline.
21

---

Published as a conference paper at ICLR 2026
Threshold: Sparsity:0.92
Threshold: 0.6 Sparsity:0.89
Threshold: 0.8 Sparsity:0.82
Threshold: Sparsity:0.75
Baseline
Threshold: Sparsity:01.22
Threshold: 0.6 Sparsity:0.89
Threshold:
Sparsity:0.82
Threshold: 0.9
Sparsity:0.75
Bascline
Figure 5: Effect of attention threshold 7 visual quality across different prompts.
22

---

Published as a conference paper at ICLR 2026
F.2 WAN2.1-1.3B RESULTS
Baseline
ASA_G
Figure "The camera orbils around. Taj Mahal, the camera circles around."
Baseline
ASA.G
Figure 7: "One person opens the door for another person."
Baseline
ASA_G
Figure 8: "A bird is in front of table, then the bird flies to the right of the table."
Baseline
ASA G
Figure 9: "A drone is floating in the air."
23

---

Published as a conference paper at ICLR 2026
F.3
COGVIDEOX-SB 5B RESULTS
Baseline
ASA.G
Figure 10: "The camera orbits around. Rocket, the camera circles around."
Baseline
ASALG
Figure 11: "A person is eating spaghett with fork."
Baseline
ASA.G
Figure 12: "A butterfly's wings change from yellow to white."
Baseline
ASA_G
Figure 13: "Princess Elsa plunged her northern kingdom into eternal winter."
This section presents qualitative comparisons between baseline models and our ASA.G-SAAAO G-distilled 8- step models. For each comparison, the top row shows results from the baseline 50-step model, while the bottom row shows results from our ASA-G method using only 8 steps(with sparsity ratios of 0.8 for Wan2.1-1.3B and 0.82 for Cog X-deo Each row displays 4 sampled frames from the generated video sequence, demonstrating temporal consistency and visual quality across different prompts.
24

---

Published as a conference paper at ICLR 2026
G ADVANTAGES OF JOINT TRAINING OVER TWO-STAGE PIPELINE
We compare our approach against a conventional two-stage pipeline that separates optimization into Iwo independent tasks: Step Distillation then Sparse Fine-tuning. Our joint training strategy offers two critical advantages:
1. Elimination of Dataset Bias. A decoupled two-stage approach inherently relies on external real datasets for the sparse fine-tuning stage. This introduces significant dataset biasthe resulting model's quality becomes heavily dependent on the domain alignment and quality of the collected data. In contrast, BLADE fully data-free. By jointly optimizing sparsity with distillation, we leverage the supervisory signals generated by the Teacher model guidance. This ensures the student aligns perfectly with the teacher's distribution without introducing external dataset bias.
2. Superior Training Efficiency. As analyzed in Appendix D, our joint method achieves conver- gence within the same iteration budget (100-200 steps) as standard dense distillation. Since sparsity is applied during training, the computational cost per iteration is reduced compared to dense pro- cessing. Thus, BLADE achieves both acceleration targets (step reduction and sparsity injection) in single, efficient pass, avoiding the substantial computational overhead of separate sequential fine-tuning stage.
25


Reviews:
[{"review_id": "x5bnA5MZkl", "reviewer": "Reviewer_YBVV", "summary": "This paper presents BLADE, a novel framework designed to accelerate video diffusion models. BLADE significantly improves inference speed while maintaining or even enhancing generation quality by combining the dynamic, content-aware Adaptive Block-Sparse Attention (ASA) mechanism with the data-free Trajectory Distribution Matching (TDM) distillation process. Extensive experiments on various video models demonstrate significant improvements in kernel-level efficiency, end-to-end inference speed, and generation quality.", "questions": "1. The paper mentions that ASA performs excellently in accelerating the generation process, but increasing sparsity might negatively impact generation quality. Specifically, how can the generation quality be ensured when sparsity is very high, while still maintaining significant computational acceleration?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The innovative BLADE framework effectively addresses the computational bottleneck in accelerating inference for video diffusion models by jointly training the sparse attention mechanism (ASA) with trajectory distillation (TDM). This solution not only accelerates the generation process but also maintains high-quality outputs, especially in high sparsity conditions, achieving high-quality video generation with fewer steps, outperforming traditional methods. The paper is clearly motivated, well-written, and the diagrams are easy to understand, ensuring good readability.", "weaknesses": "1. Although ASA's performance is compared with traditional sparse attention methods (e.g., STA, RaA, SVG), the paper does not delve into the impact of different sparsity patterns (e.g., varying threshold settings, block sizes) on generation quality. Ablation experiments with different sparse configurations could provide further insights.\n2. BLADE optimizes generation performance by jointly training sparse attention and trajectory distribution matching. The core innovation here is the fusion of sparsity with the distillation process. However, there is a lack of further experimental evidence to demonstrate the advantages of joint training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XWDrRKmJda", "reviewer": "Reviewer_asWM", "summary": "The paper introduces BLADE, a framework that integrates Adaptive Block-Sparse Attention (ASA) with step distillation for efficient video generation. It proposes a data-free joint training approach, leveraging ASA to generate dynamic, content-aware sparsity masks and sparsity-aware Trajectory Distribution Matching (TDM) to enhance quality. Experiments on CogVideoX-5B and Wan2.1-1.3B demonstrate significant speedups (up to 14.10×) and quality improvements, validated by VBench-2.0 and human evaluations.", "questions": "Please refer to the **Weaknesses** above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Integration of adaptive block-sparse attention with step distillation, enabling data-free joint training for efficient video generation.\n- ASA mechanism dynamically generates content-aware sparsity masks that enable high sparsity levels, achieving hardware-friendly acceleration without quality loss when combined with distillation training.\n- Demonstrates substantial speedups (up to 14.10×) on diverse models like CogVideoX-5B and Wan2.1-1.3B, with consistent quality improvements on VBench.", "weaknesses": "This paper lacks details on experimental settings and comparative results, for example:\n- Lack of reporting on specific GPU hours, training batch size, and memory usage for the 100-200 distillation iterations.\n- Lack of inference results demonstrating video quality across low-to-high sparsity levels to illustrate the impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p2y4gD6IxX", "reviewer": "Reviewer_WLSr", "summary": "The paper proposes BLADE, which unifies Adaptive Block-Sparse Attention (ASA) and Sparsity-Aware Step Distillation, introducing a new sparse attention operator to enhance efficiency. The method achieves 8.9×–14.1× acceleration on CogVideoX-5B and Wan2.1-1.3B with maintained or improved quality. The idea is clear, technically solid, and promising for efficient video generation, though validation on larger models (e.g., Wan2.1-14B) is still needed.", "questions": "Have you considered including training-free results on larger models such as Wan2.1-14B to strengthen the evaluation?\n\nCould you provide a more detailed breakdown of the runtime for each component of ASA to better explain the performance gap?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Effective under both training-free and distillation-based settings.  \n\n- Large speedups with stable quality (VBench and human evaluation confirmed).  \n\n- Robust at high sparsity (~80%), outperforming similar methods.  \n\n- Detailed pseudocode and source code are provided, making the method easy to follow and reproduce.", "weaknesses": "- Lacks large-scale and long-sequence experiments;\n\n- ASA is currently implemented in a custom Triton kernel and Block Sparse Attention library, and a more detailed analysis of the runtime contribution of each component would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zLj8chEsUL", "reviewer": "Reviewer_Sg3J", "summary": "BLADE unifies sparse attention and step distillation to accelerate video diffusion transformers. It introduces Adaptive Block-Sparse Attention (ASA), which dynamically selects salient regions for computation, and integrates this sparsity directly into a Trajectory Distribution Matching (TDM) distillation process. This joint, data-free approach enables efficient few-step video generation without quality loss. Tested on CogVideoX-5B and Wan2.1-1.3B, BLADE achieves up to 14× faster inference and improved VBench-2.0 scores, showing that sparsity-aware distillation yields both speed and fidelity gains", "questions": "See weakness section.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Unifies two dominant acceleration axes—few-step distillation (via TDM) and sparse attention—by making sparsity part of the distillation loop rather than a post-hoc swap. The results matches current practice in few-step distillation (Trajectory Distribution Matching).\n\n2. A diverse set up metric, including VBench, human evaluation, and PSNR/SSIM.\n\n3. Offering both ASA (inference-only) and ASA-G (distillation-aware with global tokens) gives a practical path for immediate speedups and better quality when (light) training is allowed.", "weaknesses": "1. The paper discusses SpargeAttention and VSA but does not include either as baselines. I find ASA (inference-only) to highly resembles SpargeAttention, while VSA represents a trainable sparse attention design closely related to the ASA-G variant.  Moreover, the claim that VSA is limited by video resolution is inaccurate — VSA is not constrained by resolution in their open-sourced implementation. I feel including SpargeAtteniton as baseline is necessary (published at ICML) and including VSA is optional (a more recent work), but the author should discuss the difference.\n\n2.  The distillation component essentially follows Trajectory Distribution Matching (TDM). The overall framework is thus a straightforward combination of two existing techniques. While I find incremental A+B paper to be acceptable, it should be supported by strong results. However,  this paper’s experimental evidence is weak, as discussed in later points.\n\n3. One of the central arguement of this paper is combining sparse attention and distillation in a single training stage is better than doing them sequentially. However, there is not comparions against a. sparse attention tuning and then distillation. b. distillation and then sparse attention tuning.\n\n4. The experimental evaluation is limited to small or medium-sized models and short video sequences at low resolution. No experiments are conducted on Hunyuan Video or Wan 2.1 14B, and no results are presented for long-sequence or high-resolution scenarios — where sparse attention truly matters due to quadratic scaling of attention cost. \n\n5. The paper claims that ASA improves generation quality, but this is not supported by sufficient evidence. On Wan 1.3B, full attention after distillation achieves a higher VBench score. The only improvement shown is on CogVideoX, an arguably old model released last year, which does not strongly support the generality of the quality improvement claim.\n\n6. In Table 2, the authors evaluate on the H20 GPU but use FlashAttention-2 (FA2) as the dense baseline. On H20, FlashAttention-3 (FA3) should be the standard baseline, as it is roughly 40% faster than FA2. Given the reported effective sparsity rate of 0.798, a theoretical 5× speedup over FA3 is expected, yet only a 3.3× gain over FA2 is reported — indicating significant implementation inefficiencies and optimization headroom that are not analyzed.\n\n7. This claim  “Moreover, on models such as CogVideoX-5B with short video sequence lengths, our framework delivers a robust 8.89× speedup” is misleading in the abstract, the speedup mostly comes from distillation (7.93), putting this sentence make people think sparse attention play a huge part.\n\n8. I believe DMD2 alone can reduce the number of inference steps to 3 or 4 steps on Wan 2.1, which is faster than the proposed sparse attention + TDM solution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)