Title: Training-Free Efficient Video Generation via Dynamic Token Carving
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video Generation

Training-Free

Efficient Generation

Diffusion Transformers
Abstract: Despite the remarkable generation quality of video Diffusion Transformer (DiT) models, their practical deployment is severely hindered by extensive computational requirements. This inefficiency stems from two key challenges: the quadratic complexity of self-attention with respect to token length and the multi-step nature of diffusion models. To address these limitations, we present Jenga, a novel inference pipeline that combines dynamic attention carving with progressive resolution generation. Our approach leverages two key insights: (1) early denoising steps do not require high-resolution latents, and (2) later steps do not require dense attention. Jenga introduces a block-wise attention mechanism that dynamically selects relevant token interactions using 3D space-filling curves, alongside a progressive resolution strategy that gradually increases latent resolution during generation. Experimental results demonstrate that Jenga achieves substantial speedups across multiple state-of-the-art video diffusion models while maintaining comparable generation quality (8.83$\times$ speedup with 0.01\% performance drop on VBench). As a plug-and-play solution, Jenga enables practical, high-quality video generation on modern hardware by reducing inference time from minutes to seconds---without requiring model retraining.

Training-Free Efficient Video Generation via Dynamic Token Carving

Abstract

Despite the remarkable generation quality of video Diffusion Transformer (DiT) models, their practical deployment is severely hindered by extensive computational requirements. This inefficiency stems from two key challenges: the quadratic complexity of self-attention with respect to token length and the multi-step nature of diffusion models. To address these limitations, we present Jenga, a novel inference pipeline that combines dynamic attention carving with progressive resolution generation. Our approach leverages two key insights: (1) early denoising steps do not require high-resolution latents, and (2) later steps do not require dense attention. Jenga introduces a block-wise attention mechanism that dynamically selects relevant token interactions using 3D space-filling curves, alongside a progressive resolution strategy that gradually increases latent resolution during generation. Experimental results demonstrate that Jenga achieves substantial speedups across multiple state-of-the-art video diffusion models while maintaining comparable generation quality (8.83 × speedup with 0.01% performance drop on VBench). As a plug-and-play solution, Jenga enables practical, high-quality video generation on

modern hardware by reducing inference time from minutes to seconds-without requiring model retraining.

1 Introduction

The advancement of Latent Diffusion Models [1, 2, 3, 4, 5, 6, 7] has significantly propelled the development of image and video generation. Recently, Diffusion Transformers (DiT) [8, 9, 10, 11, 12, 13, 14, 15, 16] have emerged as the predominant architecture for foundation models due to their inherent scalability and superior generative capabilities. As high-resolution video generation techniques continue to advance and DiT-based models scale to unprecedented sizes, the computational efficiency of generating high-quality content has become critically important. For example, generating a mere 5-second 720P video using HunyuanVideo [12] on a single NVIDIA H800 GPU requires approximately 27 minutes, severely limiting its practical applications in real-world scenarios.

This challenge stems from two orthogonal factors: (1) Self-Attention versus massive token length N . The continuously increasing token length for high-resolution generation causes a computational bottleneck due to the O ( N 2 ) computational complexity of self-attention in Transformers. Even with efficient attention mechanisms [17], self-attention in HunyuanVideo [12] still consumes 77.8% of the total processing time. (2) The multi-step nature of Diffusion models. The denoising process requires forwarding through the DiT architecture T times, introducing T -fold computational overhead compared to non-diffusion models [18, 19] of similar specifications.

To address these challenges, various approaches have been explored. One branch focuses on operator-based acceleration, particularly attention optimization, to eliminate computational bottlenecks. STA [20], CLEAR [21], and SVG [22] predefine head-aware attention sparsity patterns in temporal or spatial dimensions. However, these approaches inadequately account for dynamic variations in attention patterns across inputs and achieve only modest speedup ratios (1.5-2 × ), insufficient for practical deployment. Orthogonal approaches optimize the diffusion generation pipeline through distillation [23, 24, 25, 26], quantization [27, 28, 29], or feature reuse [30, 31, 32]. However, distillation incurs significant training costs while often degrading output quality. Similarly, feature reusing and quantization methods also face limitations in achieving adequate acceleration ratios necessary for practical applications.

Based on the two orthogonal factors identified, we propose Jenga , a progressive, fully sparse inference pipeline with a dynamic, generalizable Attention Carving kernel. Studies have shown that the diffusion denoising process progresses from low to high-frequency generation [33, 34], where earlier steps establish content structures while later steps refine details. The core idea of Jenga is: Early denoising steps do not require high-resolution latents, while later steps do not require dense full attention . Once video content is established, the inherent redundancy in video latents means that not every token must participate in attention computations; at high resolutions, attention is inherently sparse, and fine details can be generated without full attention. Accordingly, Jenga designs a device-friendly Attention Carving kernel that decomposes latents into contiguous latent blocks using space-filling curves, and employs block-wise attention to selectively compute key-value pairs, creating an efficient attention mechanism. As illustrated in Fig. 1 (a), video details can be preserved even when we only keep 1% key-value blocks using Attention Carving.

Generating content layouts does not require huge latent inputs, so we introduce a multi-stage Progressive Resolution (ProRes) strategy that generates video through phased resizing and denoising of latents, effectively reducing the token interactions. Under this strategy, we face the challenge of generating resolution-dependent variations in the field of view that affect content richness. For example, low-resolution generation focuses on zoomed-in details rather than global scenes. To counteract this, we introduce a text-attention amplifier that reduces local neighborhood focus, enhancing condition information utilization, and producing more content-informative results, which are similar to generating content directly using high-resolution.

As illustrated in Fig. 1(b), Jenga is a combination of two complementary techniques: ProRes handles robust content generation with lower resolutions, while Attention Carving processes sparse attention, reducing token interactions. Like optimally arranged real-world Jenga blocks, these techniques deliver efficient, high-quality video generation with high block sparsity. Empowered by Jenga, we achieve impressive results across multiple state-of-the-art DiT-based video diffusion models. For instance, we obtain 4.68-8.83 × speedup on HunyuanT2V [12] while maintaining comparable performance

on VBench [35]. Similarly, we demonstrate significant acceleration on HunyuanVideo-I2V (4.43 × ), the distilled model AccVideo [25] (2.12 × ), and Wan2.1 1.3B [13] (4.79 × ). Further, when deployed on an 8 × H800 GPU computing node, Jenga reduces the DiT inference time to 39 seconds for HunyuanVideo and 12 seconds for AccVideo.

Our contributions are threefold: (1) we propose a novel dynamic block-wise attention carving approach that enables high-efficiency sparse attention computation for video generation; (2) we introduce Progressive Resolution, which decouples the content generation and detail refinement stages, reducing token interactions and achieving further acceleration; and (3) as a plug-and-play inference pipeline, Jenga achieves unprecedented speedup across various modern video DiT architectures.

2 Related Works

Efficient attention design in Transformers represents a critical research direction focused on mitigating computational demands arising from the quadratic O ( N 2 ) complexity relative to the token sequence length N . In Language Models, efficient attention methods like MInference [36], HIP [37, 38], MoBA [39], and NSA [40, 41, 42, 43] adopt partial or hierarchical key-value selections for efficient long-context understanding. To process dense vision features, efficient attention designs are also adopted in ViT and Diffusion Models, including linear attention [44, 45] and cascade attentions [46]. All these approaches aim to reduce the number of tokens actively participating in attention computations, thereby achieving acceleration and decreasing memory requirements.

Efficient video generation has garnered substantial research interest concurrent with the rapid evolution of video Diffusion Transformers (DiTs) [8, 11, 12, 47, 13]. Early acceleration techniques focused on reducing sampling steps, primarily through step distillation methodologies [25, 26] or training-free approaches that leverage step-wise feature reuse, such as TeaCache [31] and RAS [32, 48]. Bottleneck Sampling [49] employs a variable resolution strategy across different sampling stages, thereby utilizing fewer tokens during intermediate computational phases. Complementary to step reduction strategies, various efficient attention mechanisms for DiTs have emerged, including CLEAR [21], STA [20], and SVG [22], which operate on the fundamental assumption of localized attention distribution patterns. While this localization assumption preserves consistent attention structures, it inherently constrains the model's capacity for long-range feature aggregation. Recent advancements in block-wise attention architectures, such as SpargeAttn [50, 27] and AdaSpa [51], implement selective processing based on block-level mean values, achieving approximately two-fold acceleration in video generation pipelines. Nevertheless, their optimization potential remains limited by rigid block partitioning structures and attention sparsity parameters that require further finetune.

3 Jenga: Token-Efficient Optimization for Video Diffusion Transformers

Latent Diffusion Models (LDMs) [1] learns to reverse a noise corruption process, transforming random noise into clean latent-space samples. At time step t ∈ { 0 , . . . , T } , the model predicts latent state x t conditioned on x t +1 : p θ ( x t | x t +1 ) = N ( x t ; µ θ ( x t +1 , t ) , σ 2 t I ) , where θ represents the model parameters, µ θ denotes the predicted mean, and σ t is the predetermined standard deviation schedule. For Diffusion Transformers [8], during each timestep t , the model processes noisy visual latent tokens x t together with tokenized conditional embeddings x c (e.g., text prompt), predicting the noise component ϵ added at that timestep. A scheduler [52] then guides the progressive denoising process to compute the next denoised state x t -1 = scheduler ( x t , ϵ, t ) , gradually yielding a fully denoised video latent x 0 , which is then converted back to pixel space with a pre-trained V AE decoder.

The overview of our method is illustrated in Fig. 2. Jenga aims to minimize the computational complexity by reducing the number of tokens processed in each operation within video DiTs [8]. This is achieved through two primary optimizations: (1) enhancing the efficiency of the self-attention mechanism (Sec. 3.1) and (2) streamlining the inference pipeline (Sec. 3.2). In video DiT, we typically process N v = numel( z v ) = thw visual tokens, where t , h , and w represent the temporal length, height, and width of the video latent z v in the latent space, after the visual patch embedding layer, z v = patchemb( x v ) .

3.1 Block-Wise Attention Carving

As observed in [20, 50], the proportion of time spent on self-attention operations within transformer forward passes becomes increasingly dominant as the number of tokens grows. The 3D full-attention

skip

Outside DiT: Progressive Resolution

skip

skip

skip

skip

skip

skip

One-Hot Block-wise

Mask

m

threads

m

threads

Each block: fully

parallel attention

Block-wise attention kernel

3D Latent Block

Partition

Space filling curve

Block-wise

attention

!

skip

I am a 720P

video…

Content

Details

Sampling timesteps

…

Inside DiT: Attention Carving

parallel

GPU

I am a 720P

video…

I am a 720P

video…

Imagination

360P

540P

720P

3D Latent Re-Ordering & Block Partition

Space Filling Curve

(SFC) Re-order

thw Ordering

8

48

0

56

16

40

24

32

7

6

0

1

4

5

3

2

Next Block

3

12

48

51

60

63

4

47

31

15

32

16

0

…

…

…

…

Recursive SFC

inside a block

!(#)

Block-wise

attention

Mean pooling

Block adjacency pre-compute

Adjacency Mask

3D-neighbors should

be calculated

Condition Mask

Text condition should

be calculated

Importance Mask

Top-important blocks

should be calculated

One-Hot Block-Wise Attention Masks

Video Latent

Condition

mechanism in video transformers can be represented in its most fundamental form as:

where Q i , K i , V i ∈ R N × d k represent the query, key, and value features for the attention head i , respectively. We define d as the embedding dimension, and h as the number of attention heads with d k = d/h . N = N v + N c denotes the total number of tokens, comprising N v vision tokens and N c condition tokens. In the context of video diffusion models, this attention operation incurs significant computational overhead due to its quadratic complexity O ( N 2 ) concerning the token count across spatial and temporal dimensions.

Due to the inherent redundancy in video latents, a direct approach to improve efficiency is to reduce the number of key-value pairs each query attends to. We adopt a block-wise coarse key-value selection method, as shown in Fig. 3. FlashAttention [53, 17] and other GPU-optimized approaches [50, 20] uniformly divide Q and KV into M blocks with m = N/M tokens each, corresponding to m parallel threads in the attention computation, to compute exact attention results across all M 2 blocks through parallel processing. For simplicity, we assume N v and N c are padded lengths divisible by m . Our objective is therefore to reduce KV pairs at the block level. First, to obtain tokens with higher internal similarity within 3D blocks, we reorder the 1D vision tokens z thw (flattened along thw dimensions) into a block-wise order z blk before subsequent partitioning. The reordering and its inverse process are represented by:

where G ( · ) represents an index permutation function implemented via the Generalized Hilbert reordering [54, 55, 56], a toy example of which is illustrated in the left part of Fig. 3. Compared with vanilla linear hwt ordering, this space-filling curve (SFC) ordering ensures that tokens in 1D proximity within z blk effectively preserve their 3D neighborhood relationships from the original space. Thus, this approach enables uniform partitioning directly in the flattened dimension when computing attention operations.

re-noise

𝜖̃

~ N(0, I)

Timestep Skip & Multi-Stage Latent Rescale

Block-wise attention kernel

skip

skip

skip

skip

skip

skip

skip

Video latent

Condition

Q K

Add a

bias

on

the video-text

attention score

Low

High

Text attention affects the FOV, videos are generated from a 360P initial stage

…

Timestep Skip

stage s

stage s+1

𝜖

!"#

reuse

𝜖

!"#

𝜖

!$#

𝑥

!"#

𝑥

!

𝑥

!$#

𝑥

!$%

𝑥 #

&

'

resize

Direct denoising

Text-Attention Amplifier

model default resolution

lower resolution

Field of View (FOV)

difference

For KV-block selection, we build a one-hot block-wise 2D mask B ∈ R M × M for each attention head to represent the selection result of the block-sparse attention. It is a union of three masks, as shown in the right part of Fig. 3: (1) Importance Mask B top. For importance-based block attention selection, inspired by MoBA [39] from large language models, we employ block-wise mean values to compute an attention probability map that roughly identifies which block pairs require attention computation. Specifically, for the reordered inputs, we express the relevance between blocks R using:

where ( ˆ · ) is a mean-pooling operator for each block of size m . Then for the i -th query block, we set a rate k , and keep the top kM key-value blocks in R . Meanwhile, for each query block, we set a constraint to fulfill the cutoff approximate accumulated probability. This means after all kM blocks are selection, we still need to select blocks for some block-head combination with top probabilities, until the accumulated probability meets a cutoff softmax probability threshold p , defined by ∑ j ∈ B top [ i ] R [ i ][ j ] > p . This constraint is set to avoid global context lost, especially for some attention heads to aggregate global information.

(2) Condition Mask. B cond = { i > N v /m ∨ j > N v /m } , where i, j are mask indices in querykey block dimensions. This means all condition-related attentions should be fully computed. (3) Adjacency Mask. B adja = { adja ( i, j ) } , which represents whether i -th and j -th blocks are adjacent in the 3D thw space. The adjacency mask is beneficial in fixing border artifacts between spatially adjacent blocks. In Jenga, B cond and B adja are pre-computed, and only determined by the resolution and the partition function G . The final selection array is defined as the union of three one-hot masks, B = B top ∪ B cond ∪ B adja .

For the block-wise attention, we skip the computation of indices that B [ i ][ j ] = 0 , hence achieve an attention complexity O ( N ′ N ) , in which N ′ = m ∑ B /M is the average number of selected tokens.

3.2 Progressive Resolution

Block-wise Attention Carving significantly reduces the latency of each DiT forward pass, but since diffusion sampling is an iterative process, compressing the number of tokens at the diffusion pipeline level is also crucial for accelerating generation. Leveraging the coarse-to-fine nature of diffusion denoising [33, 57], we decompose the generation inference process of T timesteps into S stages, starting from a low resolution R 1 = { t , h 1 , w 1 , r , d } and progressively increasing the resolution at each stage until reaching the final target resolution R S = { t , h , w , r , d } , where r represents the latent patch size and d is the channel dimension. The stage switch is illustrated in the left part of Fig. 4. At the end of each intermediate stage s at timestep t , we predict the clean latent at ˆ x s 0 ∈ R R s and resize it to a higher resolution R s +1 , then re-noised following an approach similar to [49]. The progressive resolution process between stages is defined as:

Here U ( · ) is a latent upsample function in 3D space, for which we employ area interpolation. ϵ t is the prediction at timestep t , and σ t is the time-dependent standard deviation in the scheduler [52]. By reducing resolution, the earlier stages involve significantly fewer tokens in inference, while the denoising at higher resolutions ensures the generated videos maintain high-quality details.

Text-Attention Amplifier. Unlike bottleneck-style sampling [49], ProRes determines video content and structure during the low-resolution stage, without preserving the original resolution in the initial stage. While Video DiT generates coherent low-resolution videos, we observe that the Field of View (FOV) degrades with decreasing resolution, effectively transforming ProRes into a super-resolution process on videos with a constrained FOV. We illustrate this phenomenon in Fig. 4, which occurs because tokens at lower resolutions disproportionately attend to their spatial neighborhoods.

To maintain a stable FOV across resolutions, we introduce a text-attention amplifier with a resolutiondependent bias β that "hypnotizes" the model in the first low-resolution stage by enhancing textattention weights, thereby reducing the focus on spatial neighborhoods. This concept is illustrated in Figs. 2 and 4. Specifically, when processing a vision query block q v and a condition key block k c in attention, the biased vision-condition attention score is calculated as: q v k ⊺ c + β where β = -ρ log(numel( R s ) / numel( R S )) is computed based on the token count ratio between the current stage resolution R s and the target resolution R S , with ρ serving as a balancing factor.

Case-Agnostic Timestep Skip. Timestep reduction is one of the most common optimization directions in efficient diffusion pipelines. Methods like TeaCache [31, 32] approximate outputs by caching input features to dynamically determine which steps can be skipped. However, in practical implementation, we observe that TeaCache's skip mechanism is effectively a static timestep scheduler, rather than a truly case-wise dynamic step skipping approach. Therefore, we employ a fixed timestep skip setting (23 steps, same as TeaCache-fast) that samples more densely at the beginning and end while sampling sparsely in the middle, eliminating the additional computation overhead of TeaCache.

4 Experiments

4.1 Implementation Details.

Settings. Our experiments are primarily conducted on the HunyuanVideo [12] architecture with a 50-step configuration. All generated HunyuanVideo videos maintain a resolution of 125 × 720 × 1280 , corresponding to a patchified video latent size of t × h × w = 32 × 45 × 80 , approximately 115K tokens. Unless specified, all experiments are performed on one NVIDIA H800 GPU.

For Attention Carving block partitioning, we employ Generalized Hilbert [54] as G ( · ) with a block size of m = 128 . We implement the Attention Carving kernel using Triton [58] and adopt a progressive top-K selection strategy when computing the importance mask: k = 0 . 3 at stage 1, and k = 0 . 2 for subsequent stages. The probability threshold is set to p = 0 . 3 . When calculating the adjacency mask B adja, it incorporates a 26-neighborhood in 3D latent space. For ProRes stages, we provide two basic configurations-Base and Turbo-corresponding to implementations using 1 (straight 720P) and 2 stages (starting with 540P, 50% steps each stage). We also introduce a 2-stage Jenga-Flash setting, which applies smaller k values in both stages to further enhance efficiency. The balancing factor of the text-attention amplifier is set to ρ = 0 . 5 . After timestep skipping, 23 of the original 50 timesteps are retained, while additional steps will be added after the stage-switch process. We adopt TeaCache-style [31] latent reuse, where features are reused before the image unpatchify layers. Comprehensive details are provided in Appendix B.1.

Multi-GPU Adaptation. Our method seamlessly integrates into multi-GPU parallel processing configurations. We have implemented adaptations based on xDiT [59, 60] within our approach using the HunyuanVideo [12] framework. This enables parallel processing of attention operations across the head dimension, while all operations except patchification are parallelized across multiple GPUs along the token dimension. Utilizing an 8-GPU parallel configuration, Jenga-Flash achieves a further 6.28 × speedup (245s → 35s) with identical computational operations, which is also 5.8 × faster than the official 8-GPU implementation in HunyuanVideo [12]. Detailed latency results are shown in Tab. 2b. We provide specifications of this implementation in Appendix B.2.

Distilled Model & Image-to-Video. Jenga demonstrates considerable generalizability across diffusion model architectures. It not only achieves substantial acceleration ratios in Text-to-Video (T2V) models [12, 13], but our adaptive attention carving technique can also be effectively implemented in models refined through step-distillation [25, 26] with a 3.16 × speedup. Furthermore, when applied

185s

One monster, kneeling

Two monsters,

standing

Two monsters, standing

Temporal

flickering

HunyuanVideo,

1634s

SVG,

985s

223s

226s

Behind the rock

TeaCache,

711s

225s

Next to the octopus

345s

Jenga+HunyuanVideo-I2V,

383s

(a) Visual Comparisons

Prompt: Animated scene features a close-up of a short fluffy monster kneeling beside a melting red candle

…

Jenga-3stage,

163s

Prompt: A petri dish with a bamboo forest growing within it that has tiny red pandas running around.

Prompt: A large orange octopus is seen resting on the bottom of the ocean … The octopus is unaware of a king crab that is crawling towards it from behind a rock …

Prompt: The camera follows behind a white vintage SUV with a black roof rack as it speeds up a steep dirt road surrounded by pine trees on a steep mountain slope …

📷

Similar FOV

360P

first-stage

720P

first-stage

540P

first-stage

O

(b) Visual Examples of More Settings

to Image-to-Video (I2V) models [12], our approach achieves 4.43 × speed improvement in I2V generation [12] tasks even without employing ProRes. Detailed results are shown in Tab. 2.

Evaluation and Metrics. For speed assessment, we report the diffusion time consumed-specifically the DiT forward pass time-as the VAE decoding component remains constant across all configurations. We also report FLOPs and step-wise FLOPs to provide an intuitive comparison of computational complexity. For qualitative evaluation, we employ the widely adopted CLIP-based metric CLIPScore [61] to measure text-video alignment, and utilize the comprehensive benchmark suites VBench [35] and VBench-I2V [62] with their original full-set prompts. We evaluate each prompt with a fixed random seed to ensure both evaluation consistency and statistical reliability. Additionally, we conducted a user study to assess human preference rates between Jenga and various efficient generation baselines, including a direct comparison with vanilla inference.

4.2 Comparisons

Attention Efficiency. We benchmarked our Attention Carving (AttenCarve) approach against stateof-the-art training-free attention optimization methods, specifically MInference [36], CLEAR [21], and SVG [22], as shown in Tab. 1. From a theoretical perspective, CLEAR (3D local window) and SVG (spatial-temporal windows) can be viewed as specialized instances of our more general Jenga framework. To establish a robust block-selection baseline, we adapted MInference [36] for video generation by removing causal masks and modifying selection optimizations. Jenga's dynamic block selection mechanism more effectively identifies crucial key-value pairs in video content while preserving important local information aggregation. Consequently, AttenCarve achieves superior acceleration ratios (2.17 × ) with reduced computational requirements while maintaining higher generation quality, particularly in terms of semantic adherence, compared to existing approaches.

Sampling Efficiency. We further compared our Progressive Resolution (ProRes) approach with TeaCache [31] in Tab. 1. We observe that ProRes and timestep skipping represent orthogonal solutions that address different aspects of efficient sampling. By incorporating ProRes, we achieve a significant reduction in step-wise FLOPs while maintaining high-quality video outputs. Qualitative evaluations confirm that our Progressive Resolution strategy effectively preserves generated video quality while substantially improving computational efficiency (3.28 × speedup).

(a) Jenga on HunyuanVideo-I2V [12] (row 1-4) and Wan2.1 [13] (row 5-7), while we report a timestep skip result as the efficiency baseline.

(b) Jenga on distilled model [25] (row 1-4) and multiGPU inference (row 5-7). For multi-GPU, benchmark results are the same as Jenga-Flash in Tab. 1.

Methods

NFE

VBench

latency

speedup

HunyuanI2V [12]

50

87.49%

1499s

1.00

×

+ TimeSkip

23

87.67%

2.08

×

+

Jenga

23

87.75%

338s

4.43

×

Wan2.1-1.3B [13]

50

83.28%

115s

1.00

×

+ TeaCache-fast [31]

15

82.63%

34s

3.48

×

+

Jenga

15

82.52%

17s

6.52

×

Methods

VBench

CLIP

latency

speedup

AccVideo [25]

1

83.82%

31.23

161s

1.00

×

+

Jenga

1

83.29%

31.12

51s

3.16

×

+

Jenga-8GPU

8

83.29%

31.12

7s

23.00

×

# GPU speed-up rate

1 8.8

×

2 7.9

×

4 7.0

×

8 5.8

×

VBench

HunyuanVideo [12]

1625s

225s

82.74%

+

184s

107s

63s

39s

82.73%

37.1%

35.7%

21.8%

52.1%

56.8%

64.6%

10.7%

7.5%

13.6%

25.7%

21.8%

25.7%

59.6%

64.6%

59.6%

14.6%

13.6%

14.6%

44.6%

43.9%

41.8%

48.6%

51.1%

47.9%

6.8%

5.0%

10.4%

Better

Same

Worse

Overall Quality

Semantic Quality

Visual Quality

Jenga-Base (347s) vs HunyuanVideo (1625s)

Jenga-Flash (225s) vs TeaCache-fast (703s)

4.68× Faster

3.12× Faster

4.39× Faster

Qualitative Evaluation. By orthogonally combining AttenCarve and ProRes with different stage configurations, we developed three variants of Jenga: Jenga-Base (1-stage), Jenga-Turbo (2-stage), and Jenga-Flash (2-stage, higher sparsity). These variants effectively balance generation quality and speed, achieving 4.68-8.82 × acceleration while maintaining high-quality outputs. Notably, both Jenga-Base and Jenga-Turbo surpass the baseline on VBench [35] metrics, with particularly significant improvements in the semantic score (72.84% → 77.48%). This demonstrates that our approach not only accelerates inference but can also enhance the semantic coherence of generated videos. It is worth highlighting that when combining AttenCarve with timestep skipping alone (Jenga-Base), our quality metrics were not negatively affected. Jenga's focus on key information selection improves semantic performance. We provide visualization cases in Fig. 5. Meanwhile, our static case-agnostic timestep skip schedule performs similar behaviour to the TeaCache in both HunyuanVideo and HunyuanVideo-I2V [12]. Results are reported in Tabs. 1 and 2a.

User Study. We conducted a user study employing the standard win-rate methodology to evaluate our approach. Questionnaires were constructed, each containing 12 randomly selected videos generated using Sora prompts [63]. The videos were presented in randomized order, and participants were asked to evaluate them along three dimensions: visual, semantic, and overall quality. We collected a

mask type

No Mask (FA)

TopK only

TopK, SFC

+ Prob. Constraint w/o Adjacency

w/o Condition w/o Importance

All Mask

(b) Block partition & block selection masks (left and right are two separate tables, in row 1-3), stage numbers (row 47), and text amplifier bias (row 8-10). VB-Q/S represents VBench Quality and Semantic.

(d) Ablation studies on key hyperparameters. (Top): Progressive resolution design with varying low-res step ratios. (Bottom): Cutoff probability threshold analysis.

B

top

Full

Part

Part

✓

✓

✓

✓

B

cond

Full

✓

✓

✓

B

Full

✓

✓

✓

VBench

82.57%

81.35%

81.41%

81.87%

82.42%

81.82%

77.41%

83.07%

latency

495s

220s

220s

223s

222s

221s

140s

225s speed

×

1.00

×

2.25

×

2.25

×

2.22

×

2.23

×

2.24

3.54

×

×

2.20

total of 70 completed feedback forms, with results presented in Fig. 6. The findings demonstrate that our method is perceptually indistinguishable from multiple efficient generation baselines [22, 12, 31] when subjected to human evaluation.

4.3 Ablation Study and Discussions

To rigorously validate the effectiveness of our proposed method, we conducted comprehensive ablation studies on both Attention Carving and Progressive Resolution, with results in Tab. 3.

Attention Carving. As shown in Tab. 3a, we ablated selection rates k and truncation probability p . Our results demonstrate robust performance even with a smaller k (82.73% for 0.1-0.2 selection rate). The findings reveal a gradual decline in both latency and generation quality as selection rates increase in the second stage, while k in the first stage has minimal impact on latency. The probability constraint enhances global information gathering, as illustrated in Fig. 7, but a large cutoff value ( p = 0 . 5 ) disrupts the selection balance among attention heads, leading to slight performance degradation. Tab. 3b ablates latent-reorder and block selection strategies. Our experiments revealed that conventional linear partitioning can introduce shift artifacts in videos. Furthermore, this scanning approach disregards locality and consequently requires more blocks than space-filling curve (SFC) partitioning, resulting in marginally increased latency. Fig. 7 and Tab. 3b also validate the effectiveness of incorporating the adjacency mask B adja and condition mask B cond, demonstrating their necessity.

For cutoff probability analysis in Tab. 3d, lower values (0.0-0.3) achieve higher effective sparsity (80.3%-80.1%) by selecting the most important blocks, while higher values approach full attention behavior. The effective sparsity significantly exceeds theoretical selection rates, revealing the inherently local nature of video attention patterns.

Mask Selection Strategy. Tab. 3c provides a comprehensive analysis of different mask components' contributions to overall performance. The results demonstrate that while the importance mask B top alone achieves significant speedup (2.25 × ), incorporating probability constraints and space-filling curve (SFC) ordering provides marginal improvements. The condition mask B cond and adjacency mask B adja prove essential for maintaining generation quality, with their removal causing noticeable performance degradation. Notably, removing the importance mask entirely leads to substantial quality

Accumulated

Gradients

Accumulated

Gradients

Ghosting

Artifacts

Clear

Background

w/o

𝑝

Full model

artifact

smooth

w/o

𝜝

!"#!

Full model

loss (77.41% vs. 83.07%), confirming its critical role in preserving video generation fidelity while enabling efficient sparse attention.

Progressive Resolution. The ablation studies presented in Tab. 3b demonstrate the effectiveness of our multi-stage approach. We found that a 2-stage configuration maintains strong generation quality, while increasing to S = 3 stages introduces some quality degradation due to latent alignment challenges. Nevertheless, the 3-stage variant still delivers satisfactory quality while achieving a 10.35 × speedup. Additionally, we evaluated the impact of various text-attention amplifier scales on generation quality. As shown in Tab. 3b, excessively high amplifier values introduce more global context and a shift in softmax distribution, resulting in some quality reduction. However, appropriately scaled amplifiers enhance content richness without compromising generation quality.

Resolution Scheduling. Tab. 3d reveals key insights about trajectory-guided resolution scheduling. The denoising trajectory serves as guidance for optimal resolution scheduling, with the 50% lowresolution step configuration achieving the best balance between quality (83.07%) and efficiency (225s). We observe dramatic quality degradation when low-resolution steps exceed 50%, while too few low-resolution steps also reduce quality, validating that low-resolution content generation requires higher attention density to establish proper structure.

Limitation Analysis & Future Works. While Jenga demonstrates compelling efficiency gains, some limitations remain. Foremost is maintaining latent alignment during resolution transitions-direct latent resizing offers computational advantages over pixel-domain operations (after V AE processes), but occasionally produces boundary artifacts. We found that these artifacts can be mitigated using detailed and comprehensive prompts. Our current implementation employs non-adaptive SFC block partitioning without leveraging semantic context for token importance, presenting a clear improvement opportunity. Future work could integrate learnable attention carving strategies during training rather than applying them post-hoc, potentially yielding optimal token selection while preserving Jenga's efficiency benefits. Detailed limitations are discussed in Appendix C.1.

5 Conclusion

In this paper, we introduce Jenga, a training-free inference pipeline that addresses computational bottlenecks in DiT-based video generation by dynamically managing token interactions. Our approach combines block-wise Attention Carving with Progressive Resolution, effectively decoupling content generation from detail refinement to significantly reduce computational complexity while preserving generation quality. Extensive experiments demonstrate substantial speedups up to 8.83 × across leading models, including text-to-video, image-to-video, and step distilled models. As a plug-and-play solution requiring no model retraining, Jenga represents a significant advancement toward making high-quality video generation more practical and accessible for real-world applications.

Acknowledgements This work was supported in part by the Research Grants Council under the Areas of Excellence scheme grant AoE/E-601/22-R. This work is partially supported by Hong Kong General Research Fund (14208023, 14206825), Hong Kong AoE/P-404/18, and the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under InnoHK supported by the Innovation and Technology Commission.

References

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR , pages 10684-10695, 2022. 2, 3

Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR , pages 12873-12883, 2021. 2

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-toimage diffusion models with deep language understanding. NeurIPS , 35:36479-36494, 2022. 2

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS , 33:68406851, 2020. 2

James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, Yunxin Jiao, and Aditya Ramesh. Improving image generation with better captions. Technical report, OpenAI, 2023. 2

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023. 2

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurIPS , 34:8780-8794, 2021. 2

William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV , pages 4195-4205, 2023. 2, 3

Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. TMLR , 2025. 2

Jiaqi Xu, Xinyi Zou, Kunzhe Huang, Yunkuo Chen, Bo Liu, MengLi Cheng, Xing Shi, and Jun Huang. Easyanimate: A high-performance long video generation method based on transformer architecture. arXiv preprint arXiv:2405.18991 , 2024. 2

Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072 , 2024. 2, 3

Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603 , 2024. 2, 3, 6, 7, 8, 9, 25, 26, 27, 31

Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314 , 2025. 2, 3, 6, 8, 25, 31

Black Forest Labs. Flux. https://github.com/black-forest-labs/flux , 2024. 2

Jinbo Xing, Long Mai, Cusuh Ham, Jiahui Huang, Aniruddha Mahapatra, Chi-Wing Fu, Tien-Tsin Wong, and Feng Liu. Motioncanvas: Cinematic shot design with controllable image-to-video generation. arXiv preprint arXiv:2502.04299 , 2025. 2

Yuechen Zhang, Yaoyang Liu, Bin Xia, Bohao Peng, Zexin Yan, Eric Lo, and Jiaya Jia. Magic mirror: Id-preserved video generation in video diffusion transformers. arXiv preprint arXiv:2501.03931 , 2025. 2

Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691 , 2023. 2, 4, 23, 26, 29

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139144, 2020. 2

Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In CVPR , pages 10124-10134, 2023. 2

Peiyuan Zhang, Yongqi Chen, Runlong Su, Hangliang Ding, Ion Stoica, Zhenghong Liu, and Hao Zhang. Fast video generation with sliding tile attention. arXiv preprint arXiv:2502.04507 , 2025. 2, 3, 4, 30

Songhua Liu, Zhenxiong Tan, and Xinchao Wang. Clear: Conv-like linearization revs pre-trained diffusion transformers up. arXiv preprint arXiv:2412.16112 , 2024. 2, 3, 7, 8, 26, 30, 31

Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, et al. Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity. arXiv preprint arXiv:2502.01776 , 2025. 2, 3, 7, 8, 9, 26, 29, 31

Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. ICML , 2023. 2

Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In CVPR , pages 14297-14306, 2023. 2

Haiyu Zhang, Xinyuan Chen, Yaohui Wang, Xihui Liu, Yunhong Wang, and Yu Qiao. Accvideo: Accelerating video diffusion model with synthetic dataset. arXiv preprint arXiv:2503.19462 , 2025. 2, 3, 6, 8, 25, 26, 31

Hangliang Ding, Dacheng Li, Runlong Su, Peiyuan Zhang, Zhijie Deng, Ion Stoica, and Hao Zhang. Efficient-vdit: Efficient video diffusion transformers with attention tile, 2025. 2, 3, 6

Jintao Zhang, Haofeng Huang, Pengle Zhang, Jun Zhu, Jianfei Chen, et al. Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration. arXiv preprint arXiv:2410.02367 , 2024. 2, 3

Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In ICCV , pages 17535-17545, 2023. 2

Chenglin Yang, Celong Liu, Xueqing Deng, Dongwon Kim, Xing Mei, Xiaohui Shen, and Liang-Chieh Chen. 1.58-bit flux. arXiv preprint arXiv:2412.18653 , 2024. 2

Felix Wimbauer, Bichen Wu, Edgar Schoenfeld, Xiaoliang Dai, Ji Hou, Zijian He, Artsiom Sanakoyeu, Peizhao Zhang, Sam Tsai, Jonas Kohler, et al. Cache me if you can: Accelerating diffusion models through block caching. In CVPR , pages 6211-6220, 2024. 2

Feng Liu, Shiwei Zhang, Xiaofeng Wang, Yujie Wei, Haonan Qiu, Yuzhong Zhao, Yingya Zhang, Qixiang Ye, and Fang Wan. Timestep embedding tells: It's time to cache for video diffusion model. arXiv preprint arXiv:2411.19108 , 2024. 2, 3, 6, 7, 8, 9, 26, 30, 31

Jiacheng Liu, Chang Zou, Yuanhuiyi Lyu, Junjie Chen, and Linfeng Zhang. From reusing to forecasting: Accelerating diffusion models with taylorseers. arXiv preprint arXiv:2503.06923 , 2025. 2, 3, 6

Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. ICLR , 2022. 2, 5

Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-toprompt image editing with cross attention control. ICLR , 2023. 2

Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In CVPR , pages 21807-21818, 2024. 3, 7, 8, 31, 34

Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir Abdi, Dongsheng Li, Chin-Yew Lin, et al. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. NeurIPS , 37:52481-52515, 2024. 3, 7, 8, 23, 24, 26, 31

Heejun Lee, Geon Park, Youngwan Lee, Jina Kim, Wonyoung Jeong, Myeongjae Jeon, and Sung Ju Hwang. Hip attention: Sparse sub-quadratic attention with hierarchical attention pruning. arXiv e-prints , pages arXiv-2406, 2024. 3

Heejun Lee, Geon Park, Jaduk Suh, and Sung Ju Hwang. Infinitehip: Extending language model context up to 3 million tokens on a single gpu. arXiv preprint arXiv:2502.08910 , 2025. 3

Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, et al. Moba: Mixture of block attention for long-context llms. arXiv preprint arXiv:2502.13189 , 2025. 3, 5, 26

Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. ICLR , 2020. 3

Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. ICLR , 2023. 3

Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context llm inference. arXiv preprint arXiv:2406.10774 , 2024. 3

Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint arXiv:2502.11089 , 2025. 3

Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, et al. Sana: Efficient high-resolution image synthesis with linear diffusion transformers. arXiv preprint arXiv:2410.10629 , 2024. 3

Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using focused linear attention. In ICCV , pages 5961-5971, 2023. 3

Xinyu Liu, Houwen Peng, Ningxin Zheng, Yuqing Yang, Han Hu, and Yixuan Yuan. Efficientvit: Memory efficient vision transformer with cascaded group attention. In CVPR , pages 14420-14430, 2023. 3

Guoqing Ma, Haoyang Huang, Kun Yan, Liangyu Chen, Nan Duan, Shengming Yin, Changyi Wan, Ranchen Ming, Xiaoniu Song, Xing Chen, et al. Step-video-t2v technical report: The practice, challenges, and future of video foundation model. arXiv preprint arXiv:2502.10248 , 2025. 3

Ziming Liu, Yifan Yang, Chengruidong Zhang, Yiqi Zhang, Lili Qiu, Yang You, and Yuqing Yang. Region-adaptive sampling for diffusion transformers. arXiv preprint arXiv:2502.10389 , 2025. 3

Ye Tian, Xin Xia, Yuxi Ren, Shanchuan Lin, Xing Wang, Xuefeng Xiao, Yunhai Tong, Ling Yang, and Bin Cui. Training-free diffusion acceleration with bottleneck sampling. arXiv preprint arXiv:2503.18940 , 2025. 3, 5, 6, 23

Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. Spargeattn: Accurate sparse attention accelerating any model inference. arXiv preprint arXiv:2502.18137 , 2025. 3, 4, 29

Yifei Xia, Suhan Ling, Fangcheng Fu, Yujie Wang, Huixia Li, Xuefeng Xiao, and Bin Cui. Training-free and adaptive sparse attention for efficient long video generation. arXiv preprint arXiv:2502.21079 , 2025. 3

Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. ICLR , 2023. 3, 6

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memoryefficient exact attention with io-awareness. NeurIPS , 35:16344-16359, 2022. 4

Jakub ˇ Cervený and contributors. Gilbert: Space-filling curve for rectangular domains of arbitrary size. https://github.com/jakubcerveny/gilbert , 2025. Accessed: April 16, 2025. 4, 6, 24

Hans Sagan. Space-filling curves . Springer Science & Business Media, 2012. 4

Xiaoyang Wu, Li Jiang, Peng-Shuai Wang, Zhijian Liu, Xihui Liu, Yu Qiao, Wanli Ouyang, Tong He, and Hengshuang Zhao. Point transformer v3: Simpler faster stronger. In CVPR , pages 4840-4851, 2024. 4

Shilong Zhang, Wenbo Li, Shoufa Chen, Chongjian Ge, Peize Sun, Yida Zhang, Yi Jiang, Zehuan Yuan, Binyue Peng, and Ping Luo. Flashvideo: Flowing fidelity to detail for efficient high-resolution video generation. arXiv preprint arXiv:2502.05179 , 2025. 5

Philippe Tillet. Introducing triton: Open-source gpu programming for neural networks. https:// openai.com/index/triton/ , 2021. 6

Jiarui Fang, Jinzhe Pan, Xibo Sun, Aoyu Li, and Jiannan Wang. xdit: an inference engine for diffusion transformers (dits) with massive parallelism. arXiv preprint arXiv:2411.01738 , 2024. 6, 26

Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. arXiv preprint arXiv:2310.01889 , 2023. 6

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. EMNLP , 2021. 7, 8

Ziqi Huang, Fan Zhang, Xiaojie Xu, Yinan He, Jiashuo Yu, Ziyue Dong, Qianli Ma, Nattapol Chanpaisit, Chenyang Si, Yuming Jiang, et al. Vbench++: Comprehensive and versatile benchmark suite for video generative models. arXiv preprint arXiv:2411.13503 , 2024. 7, 8

OpenAI. Video generation models as world simulators. 8

Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing , 568:127063, 2024. 22, 29

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In ICML , 2024. 23

Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, and Horace He. Flex attention: A programming model for generating optimized attention kernels. arXiv preprint arXiv:2412.05496 , 2024. 26

Junxian Guo, Haotian Tang, Shang Yang, Zhekai Zhang, Zhijian Liu, and Song Han. Block Sparse Attention. https://github.com/mit-han-lab/Block-Sparse-Attention , 2024. 26

Ruoyi Du, Dongliang Chang, Timothy Hospedales, Yi-Zhe Song, and Zhanyu Ma. Demofusion: Democratising high-resolution image generation with no $. In CVPR , pages 6159-6168, 2024. 27

Zhen Yang, Guibao Shen, Liang Hou, Mushui Liu, Luozhou Wang, Xin Tao, Pengfei Wan, Di Zhang, and Ying-Cong Chen. Rectifiedhr: Enable efficient high-resolution image generation via energy rectification. arXiv preprint arXiv:2503.02537 , 2025. 27

Revital Dafner, Daniel Cohen-Or, and Yossi Matias. Context-based space filling curves. In Computer Graphics Forum , volume 19, pages 209-218. Wiley Online Library, 2000. 28

Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. NeurIPS , 37:68658-68685, 2024. 29

Alexandros Stergiou and Ronald Poppe. Adapool: Exponential adaptive pooling for information-retaining downsampling. TIP , 32:251-266, 2022. 29

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS Paper Checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction part claim the paper's scope (efficient video generation) and contribution (a training-free efficient video generation method).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper do discuss the limitation of the work, and will provide an analysis in the Supplementary.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper is not discussing an theoretical method, most of the related theoretical result are correctly referenced, the novel part of the method is also correctly explained with assumptions (scopes) noted.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We includes the core part of our method with implementation details in both the main paper and the supplementary.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We will open-source the code as committed.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specified all details in the main paper and the supplementary.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Due to the page limit and computation cost, we do not include error bars in the main paper. We will report the error bar mainly in the latency results in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Yes, we report detailed computer resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the social impacts in the supplementary.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our method is training-free and is based on the current open-sourced models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Yes.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [No]

Justification: The method is training-free.

Guidelines: The paper does not release new assets.

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes]

Justification: We report the instructions and screenshots of the user study in the supplementary.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: N/A.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/ LLM ) for what should or should not be described.

Appendix

Training-Free Efficient Video Generation via Dynamic Token Carving

Overview of the Supplementary Material

T

HTML/index.html

Appendix File (This PDF)

This appendix contains extended discussions, technical details of Jenga, and additional qualitative and quantitative results that supplement the main paper.

HTML website

https://julianjuaner.github.io/projects/jenga

(open in your browser)

It contains extensive video results, comparisons, ablation studies, and limitation analyses that cannot be fully represented in a static document.

A. Technical Details

Are there any tricks used in this method?

An algorithmic implementation is provided to help understand the full pipeline of Jenga

Block partition details

It consists 4 parts.

B. Implementation Details

A detailed parameter selection cross different settings (HunyuanT2V, HunyuanI2V, Wan model series)…

Detailed implementation for Multi-GPU adaptation

Adaptation details for I2V models and distilled models .

Details for user study.

C. Discussions and Analysis

Limitation and alternative solutions .

What is the attention patterns looks like in real cases?

In-depth speed and computation analysis.

How to push the limit of the speedup ratio?

More examples and analysis about variant FOV .

D. More Results

VBench result on full 16 dimensions.

More visualizations.

Figure 8: Overview of the Supplementary. We hope all readers enjoy this work in detail. We summarize common possible questions and important technical points here to arrange the supplementary. We strongly recommend that all readers open the link https://julianjuaner.github. io/projects/jenga/ in your browser for video result visualizations.

A Algorithmic Implementation

For a more comprehensive understanding of the method component of Jenga, we provide pseudo-code algorithmic workflows in Algorithm 1 (Progressive Resolution), Algorithm 2 (Attention Carving pipeline), and Algorithm 3 (building block mask B ).

A.1 Details in Pipeline and ProRes

In the Progressive Resolution algorithm, we highlight three key technical details that were not fully elaborated in the main text.

Frequency re-ordering. Prior to each attention layer, input latent patches undergo positional embedding operations such as RoPE [64], which typically establish frequency maps based on the standard thw ordering. Since we employ G to re-order the latents, we similarly apply f blk = G ( f ) to re-order the frequency components f across different dimensions, ensuring alignment with the latent ordering. As this operation is performed only once per stage, its computational overhead is negligible.

Ordering back before unpatchify. Since the block selection in AttenCarve occurs after patchification, and both patchify and unpatchify operations need to be performed in the thw space,

Require: Text prompt c , stage number S , resolutions R 1 , . . . , R S , block size m , block selection rates k 1 , . . . , k S , cutoff probability p , text-amplifier ρ , timestep lists T 1 , . . . , T S

Algorithm 1 Progressive Resolution Framework for Jenga Video Generation Ensure: Diffusion model M θ , flow-matching scheduler 1: Text tokens: x c = LM( c ) 2: for s = 1 to S do 3: Initial noise ˜ ϵ ∼ N (0 , I ) ∈ R R s , x T ← ˜ ϵ if s = 1 4: Compute block reordering G , G -1 and adjacency masks B adja 5: Remap positional frequencies: f blk ← getFreq ( R s , G ) 6: for t in T s do 7: Reorder tokens: z t ←G ( patchfiy ( x t )) 8: Apply sparse attention: z t ← M θ ( z t , x c , k s , ρ, f blk , B adja ) 9: Restore order: ϵ t ← unpatchfiy ( G -1 ( z t )) 10: Denoise step: x t -1 ← scheduler( x t , ˜ ϵ t , t ) 11: end for 12: if s > 1 then 13: Predict clean latent: ˆ x s 0 ← x t -σ t ϵ t 14: Resolution transition: x t -1 ← (1 -σ t ) ×U (ˆ x s 0 ) + σ t ˜ ϵ 15: Reset text amplifier: ρ ← 0 for s > 1 16: Increase sampling shift: α ← α +2 17: end if 18: end for 19: return Final prediction x 0

Algorithm 2 Block-Sparse Attention with Conditional Enhancement

Require: Query Q , Key K , Value V , topk , block size m , text blocks M c , probability threshold p , adjacency mask B adja

Ensure: Attention output

1: Get visual blocks M v ←⌊ N/m ⌋ -M c

2: if M v > 0 then

3: Extract Q v from first vision blocks × M tokens

4: B ← BuildMask ( Q v , K, k, p, M c ∪ B adja

5: O v ← AttenCarve ( Q normal , K, V, B )

6: end if

7: if M c > 0 then

8: Extract Q c from remaining tokens

9: O c ← FullAttention ( Q c , K, V ) : Text blocks see all.

10: end if

11: return concat ( O v , O c )

Algorithm 3 Build Block-wise Attention Mask

Require: Query Q v , Key K , topk , probability threshold p , visual blocks M v , adjacency mask B adja

Ensure: Block selection mask B

2: Block attention scores: S ← ˆ Q ˆ K ⊺ / d k

1: ˆ Q, ˆ K ← BlockPool ( Q v ) , BlockPool ( K ) , mean pooling per block. √

3: Convert to probabilities: R ← softmax ( S )

4: Sort probabilities: R sorted , I ← sort ( R , desc = True )

5: C ← cumsum ( R sorted )

6: N k ← max( sum ( C ≤ p ) + 1 , k · M v )

7: Initialize: B top ← zeros ( B,H,M v , M total )

8: Fill B top using indices I [: , : , : , 0 : N k ]

9: B cond ←{ i > M v ∨ j > M v }

10: B ← B top ∪ B adja ∪ B cond

11: return B

we must execute reordering after patchification. Subsequently, before unpatchification, we apply the inverse operation G -1 from Eq. (2), ensuring that all transformations are performed in the appropriate space.

Scheduler re-shift. Following the re-noise process in Eq. (4), although theoretically we maintain the same noise strength, the clean state ˆ x s 0 still exhibits a discrepancy from the true distribution. To address this, we employ an approach similar to BottleNeck Sampling [49, 65], progressively increasing the timestep shift factor α of the rectified flow scheduler across stages.

A.2 Details in AttenCarve

The implementation of AttenCarve builds upon the official codebase of block-wise MInference [36]. To enhance attention efficiency, we decoupled the vision and text query blocks as Q = concat ( Q v , Q c ) , and applied FlashAttention2 [17] directly to the condition blocks. For the cutoff probability constraint when constructing the importance mask B top, we formulate the optimization

Algorithm 4 Block-Sparse Attention with Text Amplification Kernel

Require: Query Q , Key K , Value V , sequence lengths, qk scale, text amplifier ρ , text block start index, block

mask B , block dimensions Ensure: Output features 1: start_m ← program_id (0) // Current query block 2: off_hz ← program_id (1) // Batch * head index 3: Load sequence length and check bounds 4: Initialize offsets for data loading 5: Load query block q and scale by qk_scale 6: Initialize accumulators m i ←-∞ , l i ← 0 , acc ← 0 7: for block_idx = 0 to NUM_BLOCKS -1 do 8: is_valid_block ← B [ off_hz , start_m , block_idx ] 9: if is_valid_block then 10: Load key-value block k , v at offset block_idx × BLOCK_N 11: Compute attention scores qk ← q · k T 12: Apply sequence length mask to qk 13: // Apply text amplification 14: is_text_block ← block_idx ≥ text_block_start 15: qk ← qk + ρ if is_text_block else qk 16: Compute attention weights p ← exp( qk -max( qk )) 17: Update accumulators with standard attention updates 18: end if 19: end for 20: Normalize: acc ← acc /l i 21: Write results to output

problem as minimizing the number of selected blocks:

To satisfy this constraint, our implementation employs a sort-then-greedily-select approach. For block index selection operations, we leverage vectorized indexing techniques to circumvent large-scale for loops, thereby substantially improving computational efficiency. In line 2 of Algorithm 3, we address an omission in the original Eq. (3) by explicitly incorporating the dimension d k in multi-head attention. Additionally, we implemented several engineering optimizations based on the MInference [36] block selection mechanism, including replacing the original einsum operations with CUBLAS-optimized torch.bmm() functions for enhanced latency performance.

A.3 Index Re-Order and Block Partition

Block Partition, 900 Blocks

SFC Visualization (32*45*80)

Adjacent Latents

Each

color

represents a

block

Block #450

Check mask

𝐁

!"#!

[450]

[426, 427, 447, 448, … 479]

20 blocks

To provide readers with a better understanding of the block partition characteristics in Jenga, beyond the toy example in Fig. 3, we demonstrate the Space-Filling Curve (SFC) implementation in a real 720P video latent space in Fig. 9. We employ Generalized Hilbert curves to overcome the limitation of standard Hilbert curves, which are only suitable for (2 n , 2 n , 2 n ) 3D spaces. It is important to

note that each block in Jenga is not a regular rectangular prism, but rather a local cluster of tokens that are naturally partitioned. This design provides Jenga with minimal constraints regarding video dimensions-without requiring padding along physical dimensions, it only necessitates that the total token count thw be divisible by the block count m . The continuity property of SFC in the original space also ensures a certain degree of semantic similarity among tokens within each block.

We further demonstrate how to utilize the Adjacency Mask B adja to identify blocks that are spatially adjacent in 3D space based on their SFC representation. As illustrated, for block 450, by identifying the blocks to which neighboring tokens belong, we located 20 adjacent blocks that are subsequently incorporated into the attention computation for the current block.

B Implementation Details

B.1 Detailed Parameter Settings

In Tab. 4, we provide a comprehensive list of almost all key parameters used in this work. It is worth noting that although Jenga-Base employs a single-stage pipeline, we utilized different drop rates (0.7, 0.8) at different timesteps, effectively dividing our steps into two segments. We discovered that using a higher cutoff probability (i.e., p = 0 . 9 ) in Wan2.1 [13] significantly improved results without incurring additional computational time, suggesting the presence of a few attention heads that concentrate on global features. We briefly describe our ProRes adaptation specifically for HunyuanVideo [12] (i.e., Jenga-Base). We will implement ProRes adaptation for Wan2.1 [13] in the future.

B.2 Multi-GPU Adaptation

Video

[thw]

Re-Order + Flatten, [N

v

, C]

cuda:0

cuda:G

cuda:1

cuda:2

…

[

N

v

/G, C]

[

N

v

/G, h, d]

Gather All to All

…

[N

v

, h/G, d]

cuda:0

cuda:1

cuda:2

cuda:G

State B: Head-Split Attention (Jenga kernel)

Gather All to All

*K Blocks

…

Unflatten,

[thw]

For multi-GPU parallelism, we adapted our approach based on the xDiT [59] foundation used in HunyuanVideo. As illustrated in Fig. 10, we implemented parallelization across G GPUs. The parallelism within Transformer blocks remains consistent with the original implementation (i.e., state A : parallelization along the token dimension before and after attention, and state B : parallelization along the head dimension within attention). We modified the corresponding LongContextAttention interface to make AttenCarve compatible with this parallel paradigm. Additionally, we discovered that when utilizing multi-GPU parallelism, the block selection process becomes the performance bottleneck. As explained in Appendix A.2, employing more efficient torch.bmm operations significantly accelerates multi-GPU execution (reducing processing time from 77s to 34s with 8 GPUs).

For parallelism outside transformer blocks, since we have naturally serialized tokens using SFC, we can directly partition them according to their SFC indices before feeding them into state A. This straightforward implementation also eliminates the previous requirement that latent sizes be divisible by G along specific dimensions.

B.3 Image-to-Video & Distilled Model

For Image-to-Video [12] adaptation, two specific details warrant clarification. Since this model performs specialized modulation operations on image conditions (latent at t [0] ), we provide an additional token-level mask G ( m ) , m = { 1 if t = 0 , else 0 } when inputting tokens into the model. This enables decoupled modulation operations on the re-ordered latents. Additionally, the condition mask B cond incorporates both text conditions and conditioning features from the first frame. Given that the first frame already contains the overall content of the video, we did not implement the text-attention amplifier.

For the distilled model AccVideo [25], which inherently requires fewer sampling steps, we employed a single-stage Jenga-Base setting as detailed in Tab. 4. Other configurations, including multi-GPU implementation, remain consistent with our HunyuanVideo setup.

B.4 Compared Baselines

To establish a uniform evaluation standard, we standardized the test prompts, utilized the more widely adopted FlashAttention2 [17], and maintained consistent input video dimensions across experiments. Below are the specific configurations for comparison methods beyond the baseline:

CLEAR [21]. We implemented based on the original FlexAttention [66] with a 3D radius r = 32 . When calculating FLOPs, since CLEAR does not account for GPU parallelism capabilities, we used the actual block sparsity (11.1% instead of the theoretical 56%) to compute effective FLOPs. Combined with the kernel optimization overhead of FlexAttention itself, the resulting generation speed could not even surpass the baseline.

MInference [36]. As explained in Sec. 4.2, we enhanced the block-wise attention mechanism from MInference. We removed the causal mask designed for LLMs and implemented a selection rate of k = 0 . 3 . Notably, several approaches similar to MInference exist, such as block-sparse attention [67] and MoBA [39], which employ essentially identical methodologies.

SVG [22]. We utilized SVG's original implementation and resolution, incorporating its optimized RoPE and Normalization kernels with a sparsity setting of 0.2.

TeaCache [31]. We employed the official thresholds (0.1 for slow, 0.15 for fast configurations). For Wan2.1, we set the threshold to 0.2 and enabled the use_ret_step parameter, which provided further acceleration while preserving result quality.

B.5 Details about User Study

Fig. 11 presents the Google Form questionnaire and anonymous website interface used to display video assets in our user study. We randomly sampled 12 prompts from a pool of 63 paired results and randomized the left-right ordering of videos within each comparison pair. To ensure data quality, we excluded invalid responses with completion times less than 5 minutes or greater than 1 hour. We also removed 3 submissions exhibiting highly homogeneous selection patterns (e.g., consistently choosing the "left video" or "same" for all comparisons). The results from the remaining 70 valid questionnaires are presented in Fig. 6.

5/18/25, 12:58 AM

All changes saved in

User Study on Efficient Video Generation

Methods

For each pair of videos, please choose one of the options A. Left Video B. Same C. Right Video for

each question. We evaluate the visual quality (temporal consistency, aesthetic quality) and semantic

quality (prompt consistency) of the generated video.

Finally, please provide your overall quality choice based on your preference.

Left Video

Same

Right Video

Visual Quality

Semantic Quality

Overall Quality

1. A petri dish with a bamboo forest growing within it that has tiny red pandas running

around.

A petri dish with a bamboo forest growing

within it that has tiny red pandas running

Drive

(a) Google Form for user study

(b) HTML for video preview

+

User Study onEfficientVideo

GenerationMethods

For each pair of videos, please choose one of the options A.Left Video B.

SameC.RightVideoforeachquestion.Weevaluate thevisual quality

(temporal consistency,aesthetic quality)and semantic quality

(prompt consistency) of the generated video.Finally,please provide your

overall quality choice based on your preference.

1. A young man at his 20s is sitting on a piece of cloud in the sky, reading a

book.

Left Video

Right Video

https://docs.google.com/forms/d/1wfy2qNLwXTyx2dK65Dqc1AVFGdC7pgybSlTkOZcRbBs/edit

1/1

C Discussions and Analysis

C.1 Limitation Analysis & Alternative Solutions

Dynamic scene

:

less artifact

Clear boundary

:

latent mis-alignment

Complex texture

:

less artifact

Static scene

:

latent mis-alignment

As discussed in Sec. 4.3, Jenga faces certain challenges when implementing Progressive Resolution (ProRes). Several studies [68, 69] have examined the disparities between latent-space resizing and pixel-space resizing. Even with substantial re-noising ( σ t > 0 . 9 ), we cannot guarantee that edges in

Untitled form - Google Forms

the pixel space will be perfectly denoised in the final result. Since our work focuses on transformer acceleration, we opted against using the V AE decode-resize-encode approach, as tiled decode-encode operations during stage transitions would introduce additional latency of nearly 50 seconds. Fig. 12 illustrates some failure cases and usage scenarios of our current solution in 3-stage Jenga (results shown in Tab. 3b, 10.35 × faster). We observed that generation quality occasionally deteriorates in static scenes or scenarios with clear boundaries (as well as in the Image-to-Video scenario). However, these issues tend to diminish when generating more complex textures or scenes with intricate motion patterns. We validated both the baseline and multi-stage results on VBench using enhanced prompts, as shown in Tab. 5. This enables users to obtain satisfactory video results with significant acceleration when using more complex prompts (such as Sora-style prompts, as demonstrated in Fig. 5 (b), the SUV case).

Beyond the training-based improvements discussed in Sec. 4.3, another promising direction for optimization is developing enhanced block partition methods. While the current SFC approach possesses many desirable properties, it remains fundamentally static. Extending context-based SFC approaches [70] into 3D video latent space could potentially yield better utilization of block selection.

C.2 Block Selection: Attention Patterns

Sink & Global

Semantic-Aware (floor)

Location-Aware

Local Pattern

We visualize the block-aware attention scores in Fig. 13. Our analysis reveals four key characteristics in the attention patterns: (1) In shallow layers, most patterns exhibit strong locality features, or (2) attention patterns highly correlate with position, forming stripe or planar distributions. In deeper layers of the model, (3) semantic-aware attention patterns emerge, where attention shifts according to the video's semantic content. (4) Simultaneously, we observe hybrid patterns combining the three aforementioned characteristics, as well as global patterns with attention sinks. Our cut-off probability threshold is specifically designed to capture information from these latter heads. These visualized patterns not only demonstrate the inherent sparsity characteristics of attention mechanisms but also highlight the necessity for dynamic block selection in our approach.

C.3 Resolution-Aware Field of View

In addition to the influence of the text-attention amplifier on Field of View (FOV) demonstrated in Figs. 4 and 5, we present additional examples in Fig. 14 showing dynamic FOV changes achieved by adjusting the factor ρ . We observed that in certain scenarios, not utilizing the text-attention amplifier results in an overly localized focus, ultimately reducing the content coverage in the frame. By introducing the bias parameter β , we can exert a degree of control over different field-of-view ranges.

Prompt: In a still frame, the Temple of Hephaestus, with its timeless Doric

grace, stands stoically against the backdrop of a quiet Athens

Prompt: a boat accelerating to gain speed

!

low

high

0.037s

0.445s

0.059s

0.037s

0.132s

0.059s

28160

63360

115200

68s

256s

708s

360P

540P

720P

360P

540P

720P

(a) Number of tokens

(b) Inference time (23 steps)

AttenCarve k=0.2

Pre-attention

Attention

Post-Attention

Pre-attention

Attention

Post-Attention

Block Selection

2.80%

Vis Block-wise Attention

88.8%

Text Attention

1.13%

Post Reshape

3.40%

Others

3.78%

(c) In-block inference time

(d) AttenCarve profile

C.4 Speed Analysis & Additional Overheads

In this section, we provide an in-depth analysis of our method's latency. First, as illustrated in Fig. 15 (a)-(b), we demonstrate the necessity of directly reducing token count by adjusting resolution. At 360P, only 1/4 of the input tokens, the generation speed achieves a 10 × improvement compared to 720P. In Fig. 15 (c), we specifically evaluate the acceleration achieved by AttenCarve compared to FlashAttention2 [17], which achieves a 3.7 × speedup in attention computation. Furthermore, Fig. 15 (d) provides a detailed time breakdown across different components of AttenCarve, showing that Block selection introduces only 2.8% computational overhead. Additionally, we analyzed the memory efficiency of our approach. Without any specialized optimizations, when generating 720P videos, Jenga introduces a minimal additional memory overhead of only 3.7% (71.84 → 74.49 GiB).

Despite the series of optimizations in Jenga, numerous avenues remain for potential performance improvements. These include incorporating quantization optimizations mentioned in SVG [22] and SpargeAttn [50], as well as kernel optimizations for RoPE [64] and normalization operations. From a hardware perspective, adapting FlashAttention3-based [71] attention kernels on the Hopper architecture shows significant speed enhancement potential. Additionally, parallelization and sparsification strategies for the V AE component have not been fully explored. These directions represent promising areas for future engineering optimizations and continued investigation in our work.

C.5 Structural Fidelity Evaluation and Comparative Analysis

While our primary evaluation focuses on VBench for comprehensive video quality assessment, we recognize the value of complementary metrics that capture different aspects of video quality. Structural fidelity metrics like FVD (Fréchet Video Distance) provide additional validation of our method's effectiveness by evaluating the distance between training data distribution and inference data distribution. As a training-free method, we adopt two evaluation protocols: (1) the official VBench test results with different sampling seeds, and (2) the high-quality video dataset Inter4K [72] as real data distributions to evaluate FVD.

Independent Acceleration Analysis. To demonstrate that Jenga achieves substantial acceleration independently and can be combined with orthogonal methods, we conducted comprehensive comparisons with TeaCache [31], a feature reuse technique. Tab. 6 shows results on both HunyuanVideo and Wan2.1 models, revealing that our acceleration is fundamentally independent of feature caching techniques.

The results demonstrate that Jenga preserves video generation quality comparable to baselines while achieving superior acceleration. For HunyuanVideo, Jenga-Turbo achieves an FVD score of 141 versus 144 for the baseline, maintaining structural fidelity while delivering 7.22 × speedup. Similarly, on Wan2.1, our method achieves competitive FVD scores across different evaluation protocols. The FVD results align with VBench trends, confirming that our acceleration techniques do not compromise generation quality.

Importantly, while TeaCache focuses on reusing computed features across timesteps, Jenga reduces computation through progressive resolution and selective attention while boosting generation quality. This orthogonality means the methods can potentially be combined for even greater acceleration benefits. Our results show that Jenga's acceleration is fundamentally independent of feature caching techniques, providing a complementary approach to efficient video generation that maintains high structural fidelity.

Design Choice: Static SFC vs. Adaptive Partitioning. Our choice of static Space-Filling Curve (SFC) construction is motivated by several key considerations that balance effectiveness and computational efficiency. First, SFC block partition exhibits inherent locality properties that align with the predominantly local characteristics of attention computation in high-resolution video data. The generalized Hilbert curves naturally possess local-neighborhood properties where neighbors on the 1D curve correspond to neighbors in 3D space, as validated in Tab. 3b where SFC improves both latency and performance while reducing line-wise drifting artifacts compared to linear partitioning.

Second, SFC demonstrates remarkable parameter insensitivity compared to hand-crafted strategies. While local window approaches (as in STA [20] and CLEAR [21]) restrict models to specific resolutions with dimensional padding severely impacting performance, SFC's 1D nature makes it insensitive to resolution and block size parameters. As shown in Tab. 7, our 3D-SFC requires only 112 padding tokens and 0.19% additional computation, compared to 19,440 tokens and 35.32% overhead for STA's tiled local windows, demonstrating superior efficiency.

Furthermore, adaptive partitioning faces fundamental challenges in text-to-video generation. Since generation starts from pure noise, extracting meaningful semantic information for adaptive token partitioning in early denoising timesteps is problematic. We experimented with dynamic approaches,

including changing SFC dimension scanning directions during interleaved forward attention passes to enhance block interactions (similar to Swin-window shift). This approach yielded no significant quality improvements while introducing substantial processing overhead ( 20s per video) due to memory discontinuity and defragmentation costs. Our static SFC approach avoids these overheads entirely through pre-computation while maintaining the locality benefits essential for efficient sparse attention. While this represents a limitation with room for future improvement, the current design effectively balances computational efficiency with generation quality.

D Additional Results

D.1 Detailed Benchmarks

Tab. 8 provides comprehensive evaluation results across all 16 dimensions of VBench [35]. As shown, Jenga achieves notable advantages in multiple semantic score dimensions while maintaining high performance in quality metrics.

Regarding detailed results in Tab. 8, there are two key points to clarify. First, we discovered that compared to the static local patterns used in CLEAR [21], our query/head-aware dynamic patterns significantly enhance the dynamic degree of generated results (45.83% → 70.83%). Overall, Jenga introduces larger motion amplitude at the quality level, while presenting some trade-offs in subject

consistency when the selection rate is small (Jenga-Flash). At the semantic level, Jenga demonstrates substantially better semantic adherence across multiple dimensions (color, object class, scene, and overall consistency).

D.2 More Visual Results

We showcase additional results of Jenga in different settings, as illustrated in Fig. 16, and Fig. 17. We recommend viewing the video files in the provided HTML to better evaluate the effectiveness of our method.

E Social Impacts

This paper introduces a novel framework for efficient video generation that is based on current pretrained Diffusion Transformers. Although this application has the potential to be misused by malicious actors for disinformation purposes, significant advancements have been achieved in detecting malicious generation. Consequently, we anticipate that our work will contribute to this domain. In forthcoming iterations of our method, we intend to introduce the NSFW (Not Safe for Work) test for detecting possible malicious generations. Through rigorous experimentation and analysis, our objective is to enhance comprehension of video generation techniques and alleviate their potential misuse.

Jenga-Base 347s, 4.68×

Several giant wooly

mammoths approach

treading through a snowy

meadow…

Extreme close up of a 24 year

old woman's eye blinking,

standing in Marrakech

during magic hour…

A close up view of a glass

sphere that has a zen garden

within it. There is a small

dwarf in the sphere who is

raking the zen garden and

creating patterns in the sand.

225s, 7.22×

This close-up shot of a

Victoria crowned pigeon

showcases its striking blue

plumage and red chest…

3D animation of a small,

round, fluffy creature with

big, expressive eyes explores

a vibrant, enchanted forest. ..

The camera follows behind a

white vintage SUV with a

black roof rack as it speeds

up a steep dirt road

surrounded by pine trees on

a steep mountain slope, dust

kicks up from it's tires…

184s, 8.83×

Historical footage of

California during the gold

rush.

A movie trailer featuring the

adventures of the 30 year old

space man wearing a red

wool knitted motorcycle

helmet, blue sky, salt desert,

cinematic style, shot on

35mm film, vivid colors.

A young man at his 20s is

sitting on a piece of cloud in

the sky, reading a book.

Jenga-Super

157s, 10.35×

Photorealistic closeup video

of two pirate ships battling

each other as they sail inside

a cup of coffee.

A gorgeously rendered

papercraft world of a coral

reef, rife with colorful fish

and sea creatures.

A drone camera circles

around a beautiful historic

church built on a rocky

outcropping along the Amalfi

Coast…

An aerial view of a futuristic

building on a cliff overlooking a

body of water, camera pans left

A young girl smelling a pink

flower

A person riding a horse in a

polo match

A seagull is flying towards a

person's hand

HunyuanI2V+Jenga

338s, 4.43×

A person is shaking head

A cute happy Corgi playing in

park, sunset, tilt up

A storm trooper vacuuming the

beach.

A boat sailing leisurely along

the Seine River with the Eiffel

Tower in background by

Wan2.1-1.3B+Jenga

24s, 4.79×

A shark is swimming in the

ocean, featuring a steady and

smooth perspective

A cute happy Corgi playing in

park, sunset, racking focus

A couch and a potted plant

A motorcycle slowing down to

stop

AccVideo+Jenga

76s, 2.12×

Peer reviews:
comments: SUMMARY:
This paper presents a training-free and efficient video generation method called Jenga. It involves two aspects: dynamic attention carving and progressive resolution generation, which is inspired by the observed insight (1) early denoising steps do not require high-resolution latents, and (2) later steps do not require dense attention. The experiments are extensively conducted and the performance is superior, for example, x8.83 speedup with 0.01% performance drop on VBench.

STRENGTHS AND WEAKNESSES:
Strengths:

Clear to follow

Extensive and logical experiment

Nice illustration

Weaknesses:

Latent alignment and artifacts during resolution transitions

Uses static space-filling curves (SFCs) for latent block partitioning

purely heuristic and post-hoc attention carving, without adaptability

QUESTIONS:
I am a little bit confused about the title and content. It seems the the title is highly related with attention carving. But Jenga involves progressive resolution generation. The two parts, attention carving and progressive resolution generation, are independent. Could the author explain it? In my view, it could be simply a combination of the two tricks. 


It seems that the authors partially ablate different masking strategies. It could be better to increasingly add different masking strategies.

LIMITATIONS:
The authors have highlighted the limitations.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents Jenga, a plug-and-play, training-free method to accelerate video diffusion Transformers, reducing inference time from minutes to seconds. Contributions include the introduction of block-wise attention carving for sparse computation, as well as the incorporation of progressive resolution for further acceleration. Additionally, Jenga is applied during inference without requiring any training.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tThe paper has a well-structured format and the illustrations are visually appealing.\n2.\tThe training-free design enhances its generalizability.\n3.\tThe design of the masks in the method is quite clever and insightful.\n4.\tThe experiments are conducted across multiple methods, demonstrating the generalizability of Jenga.\n\nOverall, the design of the mask and the complete experiments make me inclined to accept this work.\n\n---\n\nWeakness:\n\n1.\tThe observation that early denoising steps in diffusion do not require high-resolution latents is not novel and has been previously discussed in the literature. The progressive resolution design appears more heuristic than principled. Additionally, there seems to be a potential inconsistency between the claim in Lines 45–47 that early steps establish global structure, and the observation in Line 60 that low-resolution generation tends to emphasize local, zoomed-in details. Clarifying this relationship would strengthen the motivation for ProRes.\n2.\tThe sparse attention mechanism proposed in Jenga shares similarities with prior work such as CLEAR [1] and SVG [2], which also adopt token or block sparsity. While the proposed block-wise dynamic selection is interesting, it may not represent a fundamentally new direction compared to these existing techniques.\n3.\tIn Table 2(a), the acceleration improvement of Jenga over TeaCache for Wan2.1 appears relatively modest (from 34s to 24s). It would be helpful to clarify whether this result is achieved by applying Jenga on top of TeaCache or independently. This distinction is important for evaluating the true incremental benefit of the method.\n4.\tThe evaluation currently focuses primarily on perceptual and semantic alignment metrics such as VBench and CLIPScore. However, the absence of structural fidelity metrics (e.g., FVD[3]) makes it difficult to assess the overall visual consistency and realism of the generated videos. Including such measures would provide a more comprehensive assessment of generation quality.\n\n[1] Liu S, Tan Z, Wang X. CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up[J]. arXiv preprint arXiv:2412.16112, 2024.\n\n[2] Xi H, Yang S, Zhao Y, et al. Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity[J]. arXiv preprint arXiv:2502.01776, 2025.\n\n[3] Unterthiner T, Van Steenkiste S, Kurach K, et al. Towards accurate generative models of video: A new metric & challenges[J]. arXiv preprint arXiv:1812.01717, 2018.\n\nQUESTIONS:\n1.\tRegarding novelty: It is recommended that the authors reframe the motivation and its correspondence with the proposed method to better highlight the novelty of the approach.\n2.\tRegarding acceleration performance: The authors are encouraged to provide more detailed explanations. If the acceleration is based on TeaCache, the speedup appears to be modest.\n3.\tRegarding metrics: It is suggested that the authors include objective metrics such as FVD to provide a more comprehensive evaluation.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel training-free and efficient method for video generation. The approach is based on two key insights: (1) early denoising steps do not require high-resolution latents; and (2) later steps do not require dense attention. Based on these observations, the authors introduce a block-wise attention mechanism using 3D space-filling curves, which effectively reduces the number of key-value blocks in attention. Additionally, they present a multi-stage Progressive Resolution strategy that uses varying latent resolutions across different diffusion timesteps, thereby reducing token interactions throughout the process. By integrating a few-step approach, the method achieves 4–8× acceleration on HunyuanVideo-T2V without compromising video quality.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Compared to prior methods with hand-crafted attention strategies, the proposed dynamic Token Carving mechanism more effectively identifies crucial key-value pairs in video content. It also preserves global context through accumulated probability.\n\n2. The method maintains video generation quality while significantly improving inference efficiency.\n\n3. Comprehensive experiments are conducted to validate the effectiveness of the proposed approach.\n\nWeaknesses:\n1. How are the diffusion timesteps for different stages determined in the Progressive Resolution strategy? What impact do different transition points have on the final video quality?\n\n2. In line 287, the paper states that \"a large cutoff value (p = 0.5) disrupts the selection balance among attention heads, leading to slight performance degradation.\" Could the authors elaborate on the mechanism behind this observation? A more detailed explanation or visualization would be helpful.\n\n3. What is the speedup achieved when using only AttenCarve and ProRes (without applying timeskip)?\n\nQUESTIONS:\nSee the weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Jenga, a training-free inference acceleration framework for video diffusion transformers (DiTs). It addresses two primary bottlenecks in DiT-based video generation: the quadratic complexity of self-attention and the multi-step nature of diffusion. Jenga achieves acceleration through two core components:\n\n1. Attention Carving: A dynamic block-wise sparse attention mechanism using 3D space-filling curves to reduce redundant token interactions.\n\n2. Progressive Resolution (ProRes): A multi-stage generation strategy where early denoising occurs at low resolutions and progressively upsamples latents, reducing the total number of processed tokens.\n\nThe framework is shown to be broadly applicable across text-to-video, image-to-video, and distilled models, achieving up to 8.83× speedup on state-of-the-art video diffusion models like HunyuanVideo, AccVideo, and CogVideoX, with minimal degradation in generation quality (e.g., less than 0.01% drop on VBench).\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. Proposes a training-free, plug-and-play framework for efficient video generation that achieves up to 8.83× speedup across various DiT architectures, including HunyuanVideo and AccVideo, without model modification or retraining.\n\n2. Introduces Attention Carving, a dynamic, block-wise sparse attention mechanism guided by 3D space-filling curves and masks based on block importance, adjacency, and condition relevance—offering a clean and GPU-friendly design.\n\n3. Leverages a Progressive Resolution (ProRes) strategy that denoises at lower resolutions in earlier stages and progressively increases resolution, aligning with the coarse-to-fine nature of diffusion processes.\n\n4. Includes a novel text-attention amplifier to stabilize the field of view (FOV) during early low-resolution steps, which would otherwise generate zoomed-in or truncated content.\n\n5.  strong empirical results across text-to-video, image-to-video, and distilled models, maintaining semantic quality and CLIP scores even under aggressive speedup (e.g., Jenga-Flash, 8.83×).\n\n## Weaknesses\n\n1. While the paper compares against cache-based methods, Jenga does not reuse intermediate states or features, which limits its benefit for very low-step (e.g., <8) samplers or consistency models, where reuse is more effective.\n\n2. The block-wise attention masking procedure, though efficient, is still heuristic-based and uses static top-k and probability thresholds. No adaptive or learned sparsification is explored.\n\n3. he motivation for combining curvature estimation with ProRes is underdeveloped: how the geometry of the sampling trajectory informs resolution transitions isn’t fully explained or connected.\n\n4. The novelty is somewhat incremental, as both sparse attention (e.g., SpargeAttn) and progressive resolution (e.g., Bottleneck Sampling) have prior art. Jenga’s contribution lies more in effective integration than in introducing new algorithmic ideas.\n\nQUESTIONS:\n1. Clarification of Block-Level Attention Mechanism: In Figure 2 and Section 3.1, it's unclear how frequently the block-wise attention mask is updated. More clarification is needed on the roles and update frequencies of the importance mask, condition mask, and adjacency mask—specifically, which are static and which contribute most to performance. Additionally, please elaborate on why the condition and adjacency masks are solely determined by resolution.\n\n2. Connection Between Trajectory Curvature and Resolution Scheduling: The paper introduces curvature analysis to guide block selection, while the progressive resolution strategy (ProRes) appears independently designed. Is there any correlation between curvature changes in the sampling trajectory and the decision points for resolution upsampling?\n\n3. Applicability to Low-Step or Consistency Models: Has Jenga been evaluated in low-step regimes (e.g., 4–8 denoising steps) or on consistency-based models like SD3? These increasingly popular settings may challenge assumptions in both ProRes and the sparsity design of Attention Carving.\n\n4. Adaptivity of Sparsity Masks: The current block selection uses fixed thresholds (e.g., top-k, softmax cutoff). Could adaptive or learned sparsity mechanisms further improve the balance between quality and efficiency, especially across diverse prompt distributions or content styles?\n\nLIMITATIONS:\nThe paper does address some limitations, especially regarding resolution transitions and heuristic design, but the discussion could be more thorough. In particular:\n\n1. Lack of Generalization to Consistency and Low-Step Models: The current method assumes a moderately long sampling process (e.g., 24–50 steps). It may not generalize to ultra-fast samplers like consistency models or low-step DMs, where resolution transitions and sparsity patterns can break down.\n\n2. Static Heuristics for Sparsity: The block selection (top-k, threshold-based) is fixed per stage. While effective, this limits adaptability to different prompts or domains and may underperform in highly dynamic content.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}