VMoBA: Mixture-of-Block Attention for Video Diffusion Models
Authors: Jianzong Wu
Liang Hou
Haotian Yang
Ye Tian
Pengfei Wan
Di ZHANG
Yunhai Tong
Venue: ICLR 2026 Poster (2026)
Keywords: Video Generation
Sparse Attention
Training Acceleration
MoBA
Primary area: generative models

Abstract: The quadratic complexity of full attention mechanisms poses a significant bottleneck for Video Diffusion Models (VDMs) aiming to generate long-duration, high-resolution videos. While various sparse attention methods have been proposed, many are designed as training-free inference accelerators or do not optimally capture the unique spatio-temporal characteristics inherent in video data when trained natively. This paper introduces Video Mixture of Block Attention (VMoBA), a novel sparse attention mechanism specifically adapted for VDMs. Motivated by an in-depth analysis of attention patterns within pre-trained video transformers, which revealed strong spatio-temporal locality, varying query importance, and head-specific concentration levels, VMoBA enhances the original MoBA framework with three key modifications: (1) a layer-wise recurrent block partition scheme (1D-2D-3D) to dynamically adapt to diverse spatio-temporal attention patterns and improve efficiency; (2) global block selection to prioritize the most salient query-key block interactions across an entire attention head; and (3) threshold-based block selection to dynamically determine the number of attended blocks based on their cumulative similarity. Extensive experiments demonstrate that VMoBA significantly accelerates the training of VDMs on longer sequences, achieving 2.92$\times$ FLOPs and 1.48$\times$ latency speedup, while attaining comparable or even superior generation quality to full attention. Furthermore, VMoBA exhibits competitive performance in training-free inference, offering 2.40$\times$ FLOPs and 1.35$\times$ latency speedup for high-res video generation.

Published as a conference paper at ICLR 2026
VMoBA: MIXTURE-OF-BLOCK ATTENTION FOR VIDEO DIFFUSION MODELS
Jianzong Wu 1, 1 Liang Hou?, Haotian Yang', Ye Tian' Pengfei Wan', Di Zhang', Yunhai Tong'
Peking University *Kling Team, Kuaishou Technology
Code: https: //github.... com/Kwai VGI /VMoBA Email: jzwu stu, ekw...n
Aull Attention
Score: 58.25
Scow 66.60
Score: 67.78
Trannl with KudA, cosl 226 GPU Hours
(1) Compariienn
MudA Score: %t..4
Wioo concoroonn 1 SANAI
Incemsed Edfcinorr Decrested Quality X
Trainol with
VMbBA [Dun)
Fromp:: X Luffy wiite sheep dashes serces a ush green mescioe"
Jal tatiee compattion VVNus and baudine methods
10 offiieny and quality
Figure 1: (a) VMoBA performs better than Full Attention while reducing training time. (b) VMoBA faster when the sequence length goes higher. The 'x' points represent tested latencies, and the lines are fitted with quadratic functions. (c) VMoBA is efficient and effective adaptation of MoBA.
ABSTRACT
The quadratic complexity of full attention mechanisms poses significant bot- tleneck for Video Diffusion Models (VDMs) aiming to generate Long-duration high-resolution videos. While various sparse attention methods have been proposed, many are designed as training-free inference accelerators or do not optimally cap- ture the unique spatio-temporal characteristies inherent in video data when trained natively. This paper introduces Video Mixture of Block Attention (VMoBA), novel sparse attention mechanism specifically adapted for VDMs. Motivated by in-depth analysis of attention patterns within pre-trained video transformers, which revealed strong spatio-tempore locality, varying query importance, and head- specific concentration levels, VMoBA enhances the original MoBA framework with three key modifications: (1) layer-wise recurrent block partition scheme (ID-2D-3D) dynamically adapt to diverse spatio-temporal attention patterns and improve efficiency; (2) global block selection to prioritize the most salient query- key block interactions across an entire attention head; and (3) hhreshold-based block selection to dynamically determine the number of attended blocks based on their cumulative similarity. Extensive experiments demonstrate that VMoBA sig- nificantly accelerates the training of VDMs on longer sequences, achieving 2.92 FLOPs and 1.48 x latency speedup, while attaining comparable or even superior generation quality to full attention. Furthermore, VMoBA exhibits competitive performance training-free inference, offering 2.40x FLOPs and 1.35x latency speedup for high-res video generation.
"The work done during Jianzong Wu's internship in Kling Team, Kuaishou Technology.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Modeling long-sequence data a critical challenge in modern deep learning, particularly for Video Diffusion Models (VDMs) for generating high-resolution, long- ong-duration videos (Saharia et al., 2022; Rombach et al., 2022; Guo et al., 2024; Wang et al., 2023b;a; Chen et al., 2023; 2024a; Ma et al., 2024b; Brooks et al., 2024; 2023; Kong et al., 2024; Wang et al., 2025). The ubiquitous full attention mechanism, while powerful, exhibits a computational complexity that scales quadratically with the sequence length (Vaswani et al., 2017). This bottlenecks the ability to process longer sequences. For example, a 720p video can exceed 76K tokens (Section 4).
To mitigate this, various sparse attention mechanisms have been proposed (Lu et al., 2025; Yuan et al., 2025; Ma et al., 2024a; Yuan et al., 2024; Zhang et al., 2025b; Xi et al., 2025; Zhang et al., 2025a). The core idea is to let each query interact with only subset of key-value pairs, thereby reducing computational complexity. Sparse attentions for VDMs are primarily designed as training- free methods (Xi et al., 2025; Yuan et al., 2024; Zhang et al., 2025a). However, these methods show suboptimal results without training (Section 4). Leveraging sparse attention to accelerate the training of VDMs, while being a realistic demand, remains a relatively underexplored domain.
Recently, Mixture of Block Attention (MoBA) (Lu et al., 2025) is introduced as a sparse attention mechanism for training. It demonstrates effectiveness in Large Language Models (LLMs) training for long-context data. Unfortunately, our initial attempt to directly apply MoBA to the training of VDMs yields unsatisfactory results, as illustrated in Fig. 1a, where MoBA shows a significant quality drop (VBench Score 68.25 + 56.88). We attribute this to MoBA's design being primarily optimized for textual data, where the locality spans on the ID space, leading to a performance gap for the video generation task, where the locality lays on the 3D space.
To bridge this gap, we conduct a task-specific analysis of attention mechanisms within pre-trained Video Diffusion Transformers (DiTs). Our analysis (detailed in Section 3) reveals several key insights:
Observation I: Full attention map has 1-2-3D attention patterns. Video data exhibits strong spatio- temporal locality, with attention mechanisms showing focusing on tokens within local one, two, or three dimensional neighborhoods (Fig. 3). MoBA's original key block partitioning - flattening the latent space into one-dimensional sequence and then uniformly splitting into blocks fundamentally disrupts this locality. Spatially adjacent tokens may be assigned to different blocks, diluting the repre- sentational power of block means. Existing sparse attention methods, such as SparseVideoC et al., 2025), categorizes attention heads into spatial and temporal. However, the role of any particular head can shift with changes in prompt, diffusion step, layer. Lacking reliable prior on which heads should be spatial or temporal, Sparse VideoGen computes a small pilor attention before each layer to classify head types, incurring substantial overhead that, as sequence length grows, can render it even slower than full attention, as shown in Fig. lb.
(Xi
Observation 2: Queries have various importance. Attention maps indicate that different query tokens receive varying degrees of attention, and their top similarity scores with key tokens can differ significantly (Fig. 4). Selecting the same number of key blocks for each query might under-allocate resources to queries that inherently have stronger affinities with keys.
Observation 3: Heads have different concentration levels. The concentration level of similarity scores varies across different attention heads (Fig. 5). Some heads exhibit highly concentrated similarities, while others show more smoothed patterns. A fixed Top-K selection, as in MoBA, may not be optimal because it ignores that different heads have different similarity-scort distributions.
Motivated by these observations, we propose VMoBA. a sparse attention specifically adapted for VDMs training. There are three key innovations corresponding to the observations:
Innovation 1: 1-2-3D block partition. Key blocks are partitioned using a cyclical ID-2D-3D scheme across layers. This allows the model to learn appropriate spatio-temporal attention patterns during training. It also improves efficiency compared to a uniform 3D partitioning.
Innovation 2: Query-global block selection. For each attention head, the top-scoring key blocks are selected from global pool aggregated across all query-key block products. This prioritizes blocks corresponding to the most important query-key interactions.
2

---

Published as a conference paper at ICLR 2026
Innovation 3: Threshold-based block selection. We dynamically determine the number of selected blocks based on their cumulative similarity score, rather than a fixed Top-K. This allows for a more adaptive approximation of full attention by catering to the diverse concentration levels across heads.
Through these targeted innovations, VMoBA enables efficient training of VDMs on longer sequences, significantly outperforming vanilla MoBA and achieving comparable superior quality full atten- tion, as shown in Fig. la and Fig. Ic. Furthermore, VMoBA demonstrates competitive performance in training-free settings, offering substantial speedups for long sequences, shown in Fig. lb.
Our contributions are summarized as follows: 1) We conduct an in-depth analysis of attention mecha- nisms in video DiT's, revealing key observations, including spatio-temporal characteristies, varying query importance, and diverse similarity distribution patterns. 2) We propose VMoBA, the first mixture-of-block sparse attention specifically for training VDMs on long sequences. VMoBA incor- porates novel layer-wise recurrent block partition, global block selection, and threshold- block selection strategies. 3) Extensive experiments demonstrate that VMoBA significantly accelerates training and inference for long video sequences while maintaining or improving generation quality compared to full attention and baseline sparse attention methods.
2 RELATED WORK
Diffusion model inference acceleration. The technologies of inference acceleration broadly fall into two categories. First, methods that reduce the number of denoising steps. These approaches modify only the inference procedure. Many edit the samplers noise schedules to cut down diffusion timesteps (Song et al., 2020; Liu et al., 2022; Lu et al., 2022: Liu et al., 2023; Song et al., 2023; Luo al., 2023). For example, high-order ODE solvers and multi-step methods can generate videos in tens of steps instead of hundreds (Liu et al., 2022: Lu et al., 2022). Other works exploit the redundancy between successive diffusion steps: by caching and reusing features (e.g., attention maps), they avoid repeated computation (Ma et al., 2023; Chen et al., 2024b; Zhao et al., 2024; Lv et al., 2024; Liu et al., 2025). However, these training-free free methods are highly sensitive to hyperparameters and can produce degraded fidelity (Zhao et al., 2024; Chen et al., 2024b; Liu et al., 2025). Second, model distillation techniques train a student network to mimic the original model with fewer steps (Salimans & Ho, 2022; Yin et al., 2024). However, Distillation needs extra training on large-scale datasets, and the student model often generates on the same or smaller resolutions, struggling to scale the data to larger resolutions (Salimans & Ho, 2022).
Efficient attention mechanisms. Linear attention methods, such as sate-space models (Gu & Dao, 2023; Dao & Gu, 2024; Liu et al., 2024; Wang et al., 2024a) and RNN-derived architectures (Peng et al., 2023) achieve linear complexity by reformulating the attention operation. However, these schemes typically cannot be seamlessly interchanged with full attention. Moreover, they are observed falling short on some tasks (Yu & Wang, 2024) compared with full attention. Alternatively, sparse attention mechanisms limit each token's 's attended set. Approaches like MoBA (Lu et al., 2025), NSA (Yuan et al., 2025), and -itek--Atentiion (Ma et al., 2024a) allow Transformers to transition between full and sparse attention modes seamlessly. For video diffusion models, specialized sparse attention techniques have been introduced to accelerate generation at inference time, including Sliding Tile Attention (Zhang et al., 2025b), VSA (Zhang et al., 2025c), DiTFastAttn (Yuan et al., 2024), SparseVideo VideoGen (Xi et al., 2025), and SpargeAttt (Zhang et al., 2025a). However, these techniques often serve training freely, offering limited benefit for speeding up model training. On the contrary, our proposed VMoBA is a sparse attention mechanism for video diffusion models. It accelerates inference with lossless quality and speeds up model training towards longer sequences.
METHOD
3.1 OVERVIEW
VMoBA operates in three steps, as illustrated in Fig. 2. Step 1: Partition and Mean Key Blocks. The input key tensor K is first partitioned into non-overlapping blocks. Crucially, the partitioning strategy (ID, 2D, or 3D) is determined layer-wise, as detailed in Section 3.2. After partitioning, the mean of each key block is computed, resulting in of key blocks B. Step 2: Select Key Blocks. For
3

---

Published as a conference paper at ICLR 2026
Step 1: Partition and Mean Key Blocks
Key Blocks
Frame
Mean
Number of glodde
Layer-wise
Blocks
Heiggtt
Key
Block Partition
OR
OR
blocks
eicampl
Width
Current Layer
Temporal Partition
Spatial Fartition
Spatial-Tempordl Temporal Partition
Number of Heads
Step Key
Step Calculate Sporse Selected Blocks
Mask
Query-Key Blocks Similanity Map
Query Key Multiplication Softmax
Value
Key Blocks for Head
Ranki
Rank20
Banki6
Rank11
RankkO
Koy Tokens for Rood
Rank19
Ronk12
Ranka
Ranv3
Rankil
Soffmaa
Rank1B
Ranva
Rank15
Repeat Step 2 for Each Head
Ronk13
Ronk17
fanks
Ranee
Rank?
Output for Head
Query
Querr
this example,, suppose we have 5 query tokens, 4 key blocks, and 2 attention heads
RankX
Sumi
[Sum)
Sumi
Figure 2: The overall pipeline of VMoBA. first partition key blocks with Layer-wise Recurrent BLock Partition, then select the blocks using Global Block Selection and Threshold-based Block Selection. Finally, the attention is computed only with the selected blocks.
each attention head, a similarity map is computed between all query tokens Q and all the key blocks derived in Step Instead of selecting fixed number of key blocks for each query independently, VMOBA employs two essential modifications: 1) Global Block Selection (Section 3.3): The most significant query-key block interactions are identified from a global pool across all queries for the current head. 2) Threshold-be -rrsseood-baedd Selection (Section 3.4): The number of selected key blocks is dynamically determined based on their cumulative similarity score relative the total similarity, controlled by threshold r. Step 3: Calculate Sparse Attention on Selected Blocks. Each query token performs attention only with the key-value pairs belonging to selected blocks. The outputs from all attention heads are concatenated form the final attention layer output. Note that while this describes the conceptual process, our implementation leverages FlashAttention (Dao et al., 2022) for efficient training, ensuring the output is equivalent to this theoretical procedure.
Computational complexity analysis (FLOPs). The computational complexity of VMOBA primarily consists of two parts: 1) Block Selection: Calculating the similarity matrix between query tokens (each of dimension d) and N2 key blocks. This is approximately (sdNN,) = ( 2 d/ st), where &b is block size. 2) Sparse Attention: Performing attention between query tokens and their selected kavg key-value pairs on average. The complexity is roughly O(skavg Note that here we omit the attention head, assuming is the whole hidden dimension with all the heads. The total FLOPs can be approximated as ((s((s/ (s/ + kavg&))). This indicates that larger block sizes and fewer selected blocks per query contribute lower FLOPs.
3.2 MAYER-WISL RECURRENT BLOCK PARTITION
Video data inherently possesses strong spatio-temporal locality. Our analysis of attention maps from full atten- tion DIT, Wan 2.1 1.3B (Wang et al., 2025), is illustrated Fig. 3. We find several representative attention patterns across layers. Specifically, different layers might focus on temporal, spatial, or combined spatio-temporal neigh- borhoods. For example, layer 27 prioritizes interactions along the temporal axis (1D neighbor), layer 3 focuses on spatial relationships within frame (2D neighbor), and layer 20 attends to local 3D ppatooltempora volumes (3D neighbor). Specifically, for the 3D neighbor pattern, the
Meck
Leer 21 reipter
Lever 223 aeptee
Loor michoe
Qutty Danns salltadto Fight.id
Figure 3: The query-key block attention map shows 1-2-3D distinct patterns.
4

---

Published as a conference paper at ICLR 2026
queries belonging to the first block attend primarily to the 1st, 2nd, 3rd, and 5th key blocks, which are accurate themselves and their neighbors in width, height, and time.
To accommodate these varying patterns without costly dynamic selection (Xi al., 2025). VMoBA employs a Layer-wise Recurrent Block Partition strategy. We define three types of key block partitioning: 1) Temporal (1D) Partition. Blocks are formed along the time axis, grouping tokens from the near frames. 2) Spatial (2D) Partition. Blocks are formed along the height and width dimensions, grouping tokens from the local spatial position across all frames. 3) Spatio-temporal (3D) Partition. Blocks are formed as local 3D patches in the spatio-temporal latent space. The three partition functions are performed recursively across layers in our model, which can be formulated as:
(NE) (s H W),
if mod 3 = 0
K' rearrange E (T H ())
(N NW (T Shy Sbg )
if mod 3 1 (1)
(N N NW) (5h Sby 5bg
W
if modd 3 = 2
(NZ),, (NH NW) (Ng
if / mod 3 3=0
= (ea) E
if mod
(2)
by
N by N
if mod 3 =
where K' is the key after rearrange, and B is the key block tensor. is the layer number. T, H, and W represent the frame, height, and width of the video latent. No.- N and N,W. the number of blocks in time, height, and width dimensions for each partition pattern, respectively. of,, se and shy are the corresponding block sizes. The subscript x E {1, 2, indicates the partition pattern (1-2-3D). We omit batch size and hidden size here for simplicity. Thanks to the training nature of VMoBA, this cyclical ID-2D-3D scheme allows the model to learn spatio- apatio-temporal attention patterns automatically, This approach improves efficiency by reducing the total number of blocks compared to uniform 3D partition across all layers, as VMoBA's speed is sensitive to the total block number. Furthermore, this structured partitioning helps ensure that semantically related tokens are more likely be grouped within the same block, leading to more representative mean key blocks.
3.3 GLOBAL BLOCK SELECTION
Full attention maps indicate that different query tokens receive varying degrees of attention, and their top similarity scores with key tokens can differ significantly. As shown in Fig. 4, the sum of top 25% query-key similarities varies greatly across queries. For example, regions in "as- tronaut" and "ground" have high top similarity scores while regions in "ball" and "sky" have lower scores. This indicates that some queries inherently have stronger affinities or importance regarding key interactions. MoBA's original strategy selects fixed number of key blocks for each query independently. This can under-allocate resources to queries that inherently have stronger affinities with keys and could benefit from interacting with more blocks, or over-allocate to less important queries. To address this, the proposed VMoBA implements Global Block Selection algorithm. For each attention head, instead of selecting blocks on a per-query basis, we first compute all query-key block similarities. Then, the top-scoring key blocks are selected from this global pool of similaritics, aggregated across all query-key block products. This prioritizes blocks corresponding to the most important overall query-key interactions, allowing sparsity to be allocated more effectively based on the collective signal strength of interactions. This can be formulated as:
Output Frame)
Top Query Similarities
Summation of top 25% query-key simi- larities. Different queries have varying importance.
Figure 4:
M, = bf, k), i (1,
where the TopkMask(a, b) function selects the top b values in and returns mask with selected positions set to 1. i is the attention head index, from to h, the number of attention heads. qi in shape x d] contains all the query tokens for head i, and b, in shape N,, x d is all the key blocks for head i.
5

---

Published as a conference paper at ICLR 2026
q. is matrix production get their similarity matrix. M; in shape N, s the selected query- key block pair mask. From this formula, VMoBA selects blocks globally, considering the different importance of queries. Here l is the number of selected blocks, further defined in Section 3.4.
3.4 THRESHOLD-BED BASED BLOCK SELECTION
The concentration level of similarity scores varies not only across queries but also across different attention heads. Some heads exhibit highly concentrated similarities towards a few query-key block pairs, while others show more diffuse or smoothed patterns. This is illustrated in Fig. 5, which shows the distribution of sorted query-key block similarities for different heads. The varying slopes indicate different concen- tration levels. Specifically, head 4 rapidly in- creases the similarity on the most right part (the largest similaritics), while head I requires more blocks to capture a significant portion of the to- tal similarity. For better understanding of the concentration level, we draw the 30% and 50% cutoff lines for each head, observing significant differences across the heads. The number of query-block pairs required to reach 50% cumu- lative similarity for heads 1 and 4 differs by nearly 25,000, which is highly significant. Under such an observation, fixed Top-K selection of blocks, in the original MoBA, which ignores these head-specific distribution differences, may not be optimal. It might select too few blocks for diffuse heads, thus losing information, or too many low-similarity blocks for concentrated heads, thus wasting computation. Therefore, VMoBA introduces Threshold-basd the number of selected blocks dynamically. After obtaining the global query-key block similarity scores, these similarity scores in descending order. We then select blocks starting from the highest similarity, accumulating their normalized similarity scores until this cumulative sum exceeds predefined threshold. The process dynamically define k can be written as:
Normalized Similirity Valure
1900
200m
Query-Key Similarity Index
Figure 5: Sorted query-key block similarity and top 30%/50% cutoff lines. The right parts of the cutoff lines contain corresponding cumulative sum- mations of query-key block similarity.
Block Selection to determine
min (S)t 2 r].
where S q.bp" is the computed query-key block similarity matrix. T is a hyperparameter to define the cumulative similarity threshold. This adaptive approach allows VMoBA to approximate full attention more effectively by catering to the diverse concentration levels across heads. It ensures that high-similarity pairs are captured while efficiently pruning low-similarity ones, acting an effective compression of sparsity based informational content, rather than fixed k.
4 EXPERIMENTS
4.1 SETUP
Datasets. For training VMoBA and the baseline models, we use the Koala-36M dataset (Wang et al., 2024b). We train different resolutions to assess VMoBA's generalization capability. Importantly, in each experimental setting, all models are trained on the identical data to eliminate any confounding effects due to data variation. To evaluate the video generation ability, we use the VBench prompt list (Huang al., 2024). For training free inference experiments, the traiiin-bassed results, we use the prompts after optimization (Yang et al., 2024) because training prompts from Koala-36M (Wang et al., 2024b) long.
the original prompt list. For
Metrics. We adopt five assessment aspects of VBench (Huang et al., 2024) to evaluate the video gen- eration ability. They Overall Consistency (TextConsis), Dynamic Degree (Dynamic), Background

---

Published as a conference paper at ICLR 2026
Table 1: Training-free free inference comparison of VMoBA and baseline methods.
Vidao Size
Merhbad
Spunity
Similarity ESNK
Quality
FiBriency
TeatConais
BCC mmiss
ImagsQuel
SalCoriis 1
FLOIS
Lairnyy
FullAna
27,8570
044,4755
50.70%
103s
81 x 450 x 532
DITROsAm (Yean et ol. 2024
0.50
25,07
37. 37.38%
60.37%
54.57%
182 7IT (1.56x)
39 (1.15%)
SVG DXG al. 2025)
0,50
12:64
27.30%
93.675
60.8%%
$9.84%
182.00 927 (1.54x)
90s (1.170
X9K Tikons
an 1009)
0.25
18.47%
60009
(2.12x)
126c (0K1.)
PallAm
27,54/
95.345
50.5
1246
4M5
61x 720 x. 1280
EN BiTTaaAAii (Yean et 2014g
0.50
24.50
26.09%
920075
65 59599
02 25%
2220 (8-75xc
(1.50
DVG es al., 3025)
0.50
25.85
27.6488
92.11
63.56R
95. 1899
720 (1.75k)
314s 11.201
T6K Tokers
2005)
30.46
38.47%
64
424
(2.75x)
180t (1.13x)
Table 2: Training-based results. Training time is measured in GPU hours. Training-free free methods apply to tuned Full Attention models.
Vadeo Stze
Mishoo
Spardity
Quality
Iilrirncy
TextConsn
Dynami:
BCCC
ImragsQual]
SubCoenii
FLOPN
Training
Pretraised
28,229
62.97%
92.40%
Fullaun
24.51%
61.58%
04 19.9%%
59.49%
90.567
705.027 (1 (0
2756 (1.00x
x 576 1074
INTTEasAAm (Yoo ot, 2014))
0.90
13:447
61 17.57%
181
67
$3.199
413
129 (1 [6
0.90
33.179
47%
S5K Tokers
McSA (Le etal. 2025)
0.25
25,00%
5,80%
07.0%
63.75%
$4.30%
262
(2 40-0
2256 (1 22-1
VMEBA
0.19
25.89%
56.90%
96.06%
55/45%
94.22%
248.8RT
(2.83
187H (1-881
Pretruined Wan 2.1
27.11%
56.59%
63.49%
92.06%
724
FallAun
21,929
43.01%
02.30%
$4.367
92.58%
924.97 (1 (00x
262h (1.004
141 x 45000 x 838
ArT Piaon et 2020)
0.90
21. 11.399
60.50%
89-77
63 30.6
$6.009
43T (1 67H
SVG et 2015)
0.90
21.179
43.07
03.77
65. 63.72%
92.189
434.647 (1 67%
56K Tokeus
Mc8A (Le VMBBA
0.25
25.239
11.07%
65,00%
$3.409
269h 10T (2.51=)
2090 (1
0.18
23.70%
31.36%
05
67.88%
08.787%
248.390 (2.92-)
1828 (E388)
Consistency (BGConsis), Imaging Quality (ImageQual), and Subject Consistency (SubConsist). Furthermore, we use PSNR (Fardo et al.. 2016) to evaluate the similarity of sparse and full attention generated videos in the training-free inference setting. We also count the generation latency, training time, and the corresponding speedup to compare the efficiency of methods.
Baselines. We compare VMoBA with its most relevant native training-targeted sparse attention method, MoBA (Lu et al., 2025). We also compare with two training- fenning--reeeeee free sparse attention methods designed for DiT's, DiTFastAttn (Yuan et al., 2024) and SVG (Xi et al., 2025). Full Attention (Wang et al., 2025) is considered a ground-truth in the training-free setting. In the training-based experiments, it becomes a comparable baseline method. For a thorough insight into the results, we only compare the Attention mechanism with baselines, eliminating other tricks like cache or quantization.
Implementation details. We use Wan 2.1 1.3B (Wang et al., 2025) as the base model for all the experiments. For VMoBA, we set the threshold T to 0.25, while its token sparsity may vary across different settings (always smaller than because of the selection algorithm). The block size and block number differ in various resolutions. We primarily set the number of spatial-temporal blocks to 72, which is sufficient for most cases. Following previous works (Xi et al., 2025; Zhao al., 2024; Li et al., 2024; Ly et al., 2024; Liu et al., 2025), we remain the first 25% of denoising steps with full attention. Training experiments are finished with 2000 steps. Additional implementation details can be found in the Appendix.
4.2 QUANTITATIVE RESULTS
Training-free inference. We conduct training-free inference experiments on two resolutions. One is the original resolution of Wan (Wang et al., 2025) (81 x480x832, 33K tokens), the other is an extended token length x81x720x 1280, 76K tokens). The results are presented in Table 1. At the original resolution, VMoBA achieves a 1.01 x speedup over Full Attention, surpassing MoBA's x.83x. While its speedup is slightly less than training-fres methods, VMoBA demonstrates strong generation quality. It ranks second in TextConsis and BGConsis, with other scores being comparable (within 1 % of the top baseline). Regarding similarity to Full Attention, as measured by PSNR, VMoBA is second only to DiTFastAttn. When extending to longer sequences, VMoBA's advantages in speed and quality become more pronounced. Specifically, VMoBA achieves a 1.35x x speedup, which is 19% higher than vanilla MoBA. This improvement is primarily because MoBA's ID partitioning leads to excessive number of blocks. In terms of quality, VMoBA leads in BGConsis, is second in TextConsis and ImageQual, and is comparable in SubConsist. Its PSNR is lower than some methods,
7

---

Published as a conference paper at ICLR 2026
indicating that VMoBA's generated videos have less similarity to Full Attention outputs in this longer token setting. This might reflect a characteristic of VMoBA: for long tokens, it can generate videos that are not identical to Full Attention but still maintain high quality. In summary. despite being designed for training, VMoBA demonstrates acceptable performance in training-free settings.
Training on longer sequences. To evaluate VMoBA's ability fine-tune pre-trained models on longer sequences, we train Wan (Wang et al., 2025) on two extended resolutions: 1) Spatially extended video: 93x576x 1024 (55K tokens). 2) Temporally extended video: 14I x480x832 (56K tokens), Results are shown in Table 2. For spatially extended videos, VMoBA, with the lowest token sparsity (0.19) and training time, achieves quality comparable to or even exceeding Full Attention. The VBench mean score is 68.34 for VMoBA and 68.25 for Full Attention. Full Attention required 276 GPU Hours for training, while VMoBA only needed 187 GPU Hours, resulting in 1.48x speedup. The vanilla MoBA's score is particularly low on Dynamic Degree (5.80%). indicating its outputs have minimal dynamies and are almost static. This is likely because MoBA's 1D block partitioning struggles to capture spatio-temporal motion information effectively. DiTFastAttn and SVG', applied to the tuned Full Attention model, yield lower scores than Full Attention and do not offer training acceleration. For temporally extended videos, VMoBA achieves 1.44x speedup over Full Attention, again delivering comparable or superior performance. Notably, VMoBA outperforms Full Attention by 3.3% in Image Quality. In summary, the proposed VMoBA accelerates DiT training significantly while ensuring comparable or improved generation quality on longer sequences.
Brongt de screee diyee in anteoirraaa
Prongt spnnca in dishen
5WS
VM688
30.00
B1e490e632 frum
Blx720e1280 Trairing
Abauczh beact spligg wou lipingg LIne
Pronuct Bay francer sirnght
Fulldrr
64
McBA 23.00
65.07
VIKUBA 25:88
67.66
95x570x1304 Fine tuned
[d) 141:8005322 Fine tuned
Figure 6: Qualitative comparison of VMoBA and baseline methods.
4.3 QUALITATIVE RESULTS
Fig. 6 showcases four examples, illustrating qualitative results. (a) Displays training-free inference results at the original resolution. Even with sparsity of 0.5, SVG (Xi et al., 2025) exhibits clear blurry strips at the top and bottom of the video, contributing to its very low PSNR (12.64). In contrast, VMoBA closely reproduces the video content. Interestingly, in the ground truth video generated by Full Attention, the seagull's head appears in an incorrect orientation, an artifact not present in (b) Presents results for high-definition video (81 x80x0 1280) with the prompt "A panda cooking in the kitchen." Full Attention generates a panda and a chopping board but misses the "cookiig" aspect. VMoBA, however, generates dishes and condiments, aligning more closely with the prompt. This observation helps explain VMoBA's higher TextConsis score 28.06% vs. 27.99% for Full Attention. SVG's output more similar to Full Attention's 's resulting in higher PSNR but TexiConsis score 0.42% lower than VMoBA's. (c) Illustrates results after training models on the spatially extended resolution. VMoBA generates an aesthetically pleasing scene of coastal waves in spring, consistent with the prompt. Full Attention, on the other hand, lacks a clear depiction of the coast. MoBA's output is notably static and less aesthetically appealing. This demonstrates that VMoBA, after training, can generate videos comparable or better than Full Attention while
VMoBA's generation.
8

---

Published as a conference paper at ICLR 2026
achieving significant training speedup (1.48x). (d) shows results after training temporally extended resolution. VMoBA's output exhibits better image quality and more realistic scene depiction than other methods. In summary. VMoBA offers significant speed improvements for long sequences. After training, it can achieve visual quality consistent with, and in some aspects, even better than, full attention mechanisms, particularly in terms of prompt alignment and imaging quality.
Table 3: Ablation studies. 'DD', 'Q', and 'SC' represent Dynamic, ImageQual, and SubConsis. Training time is measured in GPU hours.
(a) Key block partition method ablation.
(b) Block choice design ablation.
Method
DD
f
SC f
Time
Method
DD f
IQ f
SC
1-2D
55.49%
58.51%
86.12%
176
topk local
54.87%
65.59%
91.64%
1-3D
28.57%
66.71%
91.34%
187
threshold + local
55.19%
65.31%
92.43%
2-3D
57.01 %
66.02% 67.45%
94.75%
202
topk global
55.29%
64.58%
92.86%
1-2-3D
56.91 %
94.72%
187
threshold global
56.91%
67.45%
94.72%
(c) Ablation on threshold r.
(d) Ablation on the number of blocks. 'a-b-c' the number of 1-2-3D blocks, correspondingly.
Sparsity
DD f
1Q
SC
Time
0.15
0.13
55.46%
66.82%
91.12%
162
#Blocks
DD f
IQ 1
SC f
Time
0.25
0.19
56.91%
67.45%
94.72%
187
8-24-36
57.25%
66.57%
93.59%
172
0.35
0.27
55.00%
67.63%
94.35%
8-48-72
56.91%
67.45%
94.72%
187
0.50
0.39
56.74%
68.17%
94.42%
24-48-144
57.70%
68.60%
94.42%
222
4.4
ABLATION STUDY
Block partitioning strategy. We ablate the layer-wise recurrent block partitioning strategy by removing one of the ID, 2D, 3D partitioning patterns and show the results in Table 3a. Removing any single partitioning patter generally leads to drop in performance. For instance, The 2-3D strategy achieves slightly higher score but at the cost of increased training time. This suggests that the ID partition contributes to efficiency. This highlights the benefit of incorporating diverse partitioning schemes capture different aspects of spatio-temporal locality.
Block choice design. We ablate the key components our block selection mechanism: global block selection and threshold-based selection, detailed in Table 3b. The results show that our proposed "threshold + global" strategy achieves the best performance across all metrics. Removing either leads to performance drop. This indicates that both adaptively determining the number of blocks via threshold and selecting blocks from a global pool are crucial for VMoBA's effectiveness.
The influence of threshold т. We investigate the impact of the threshold 7. Results are presented in Table 3c. We observe that a larger threshold generally leads to better performance. However, this comes at the cost of increased training time. This is intuitive, as a larger T means selecting more blocks, thereby increasing computational load but also enhancing the model's capacity. A threshold of 0.25 offers a good balance.
The influence of block number. We evaluate the influence of the number of blocks, shown in Table 3d. Generally, using more blocks tends to bring better performance. However, this also increases the training time. Our default setting of "8-48-72" achieves comparable scores with lower training cost, which provides a good trade-off.
CONCLUSION
This paper introduces VMoBA, mixture-offblocc sparse attention mechanism designed to address the computational challenges of training long-sequence data in VDMs. Our approach was motivated by a task-specific analysis of attention mechanisms in pre-trained video DìTs. VMoBA incorporates three key innovations: layer-wise recurrent block partitioning, global block selection, and threshold- based block selection. Extensive experimental results demonstrate VMoBA's effectiveness. achieves speedups of up 1.48x while maintaining or even improving generation quality compared to full attention. Moreover, VMoBA provides competitive speedups in training-fre inference settings, showcasing its versatility.

---

Published as a conference paper at ICLR 2026
Acknowledgement. This work is supported by the National Key Research and Development Program of China (No. 2023YFC3807600.
ETHICS STATEMENT
Our research is grounded in ethical practices, with particular attention paid to the responsible use of data. All datasets employed in this study publicly available and well-established within the computer vision community. Specifically, our training data includes Koala-36M (Wang al., 2024b), while benchmarking conducted on VBench (Huang al., 2024). Our of this data is in accordance with their provided licenses and intended academic purpose.
REPRODUCIBILITY STATEMENT
We are committed to ensuring the reproducibility of the research presented in this paper. To this end, comprehensive implementation details for models and experiments provided in Appendix, including the training procedures and all hyperparameters used. Furthermore, upon acceptance of this paper, all source code, datasets, and trained model checkpoints will be made publicly available.
REFERENCES
Tim Brooks, Bill Peebles, Connor Holmes, Yufei Guo Will DePue, Li Jing. David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Zeroscope, 2023.
Tim Brooks, Bill Peebles, Connor Holmes, Yufei Guo Will DePue, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Sora: Creating video from text, 2024. 2
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310. 19512, 2023. 2
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang. Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. arXiv preprint arXiv:2401.09044, 2024a.
Pengtao Chen, Mingzhu Shen, Peng Ye, Jianjian Cao, Chongjun Tu, Christos-Savvas Bouganis, Yiren Zhao, and Tao Chen. 5-dit: A training-free acceleration method tailored for diffusion transformers. CoRR, 2024b.
Tri Dao and Albert Gu. Transformers are ssms: generalized models and efficient algorithms through structured state space duality. In ICMI., 2024.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. NeurIPS, 2022. 4, 14
Fernando A Fardo, Victor Conforto, Francisco de Oliveira, and Paulo $ Rodrigues. formal evaluation of psnr as quality measurement parameter for image segmentation algorithms. arXiv preprint arXiv: 01/600007///.. 0 16, 2016. 7
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, arXiv
preprint ar Xriv:223/2.00752 2023. 3
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. /CLR, 2024.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang. Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In CVPR, 2024. 6, 10
10

---

Published as a conference paper at ICLR 2026
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03601, 2024.
Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang. Han Cai, Junjie Bai, Yangqing Jia, Kai Li, and Song Han. Distrifusion: Distributed parallel inference for high-resolution diffusion models. In CVPR, 2024.
Feng Liu, Shiwei Zhang, Xiaofeng Wang, Yujie Wei, Haonan Qiu, Yuzhong Zhao, Yingya Zhang, Qixiang Ye, and Fang Wan. Timestep embedding tells: It's time to cache for video diffusion model. In CVPR, 2025. 3.7
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In ICLR, 2022. 3
Xingchao Liu, Chengyue Gong, al. Flow straight and fast: Learning generate and transfer data with rectified flow. In ICLR, 2023. 3
Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu. Vmamba: Visual state space model. NeurIPS, 2024.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In NeurIPS, 2022.
Enzhe Lu, Zhejun Jiang, Jingyuan Liu, et al. Moba: Mixture of block attention for long-context Ilms. arXiv preprint arXiv:2502.13/88 2025. 3, 7, 13, 14
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378. 2023. 3
Zhengyao Lv, Chenyang Si, Junhao Song. Zhenyu Yang, Yu Qiao, Ziwei Liu, and Kwan- Yee K Wong. Fastercache: Training-fres video diffusion model acceleration with high quality. arXiv preprint arXiv:2410.1985 19355, 2024. 3,7
Dongyang Ma, Yan Wang, and Tian Lan. Block-attention for efficient prefilling. ICLR, 2024a. 2,
Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.00 003048 2024b.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. DeepCache: Accelerating diffusion models for free. arXiv preprint arXiv:2312.00888, 2023. 3
Bo Peng, Bric Alcaide, Quentin Gregory Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Nguyen Chung, Leon Derczynski, et al. Rwkv: Reinventing mns for the transformer era. In EMNLP. 2023. 3
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022. 2
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS, 2022. 2
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2020. 3
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. ICML, 2023.
11

---

Published as a conference paper at ICLR 2026
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017.
Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen- Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, et al. Wan: Open and advanced large-scale video generative models. arXiv preprini arXiv:2503.203
2025. 2, 4, 7, 8, 14
Hongjie Wang, Chih- Yao Ma, Yen-Cheng Liu, Ji Hou, Tao Xu, Jialiang Wang, Felix Juefei-Xu Xu, Yaqiao Luo, Peizhao Zhang, Tingbo Hou, et al. Lingen: Towards high-resolution minute-length text-to-video generation with linear computational complexity. arXiv preprint arXiv:2412.09856 2024a.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Mod- elscope text-to-video technical report. arXiv preprint arXiv:2308.06577 2023a.
Qiuheng Wang. Yukai Shi, Jiarong Ou, Rui Chen, Ke Lin, Jiahao Wang, Boyuan Jiang, Haotian Yang. Mingwu Zheng, Xin Tao, et al. Koala-36m: A large-scale video dataset improving consistency between fine- gra-ira conditions and video content. arXiv preprint vrXiv:2410.08200 2024b. 6, 10
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang. Yinan He, Jiashuo Yu. Peiqing Yang, al. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309. 15100, 2023b.
Haocheng Xi, Shuo Yang. Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, al. Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity. arXiv preprint arXivv2502.0177 2025. 2, 5, 8, 13
Zhuoyi Yang. Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, al. Cogvideox: Text-to-video diffusion models with an expert transformer. CoRR, 2024. 6
Tianwei Yin, Michaël Gharbi, Taesung Park. Richard Zhang. Eli Shechtman, Fredo Durand, and William T Freeman. Improved distribution matching distillation for fast image synthesis. In NeurlPS, 2024.
Weihao Yu and Xinchao Wang. Mambaout: Do really need mamba for vision? CoRR, 2024. 3
Jingyang Yuan, Huazuo Gao, Damai Dai, al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint UrXiv:2502./0088 2025. 3
Zhihang Yuan, Hanling Zhang, Lu Pu, Xuefei Ning, Linfeng Zhang, Tianchen Zhao, Shengen Yan, Guohao Dai, and Yu Wang. Ditfastattn: Attention compression for diffusion transformer models. NeurlPS, 2024. 2, 7,
Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. Spargeattn: Accurate sparse attention accelerating any model inference. arXiv preprint arXiv:2502.18137, 2025a. 2.3
Peiyuan Zhang, Yongqi Chen, Runlong Su, Hangliang Ding, Ion Stoica, Zhenghong Liu, and Hao Zhang. Fast video generation with sliding tile attention. arXiv preprint arXiv:2502. 2025b. 2,
Peiyuan Zhang, Haofeng Huang, Yonggi Chen, Will Lin, Zhengzhong Liu, Ion Stoica, Eric Xing, and Hao Zhang. Faster video diffusion with trainable sparse attention. arXiv preprint arXiv:2505. 13389, 2025c.
Xuanlei Zhao, Xiaolong Jin, Kai Wang, and Yang You. Real-time video generation with pyramid attention broadcast. CoRR, 2024. 3,
12

---

Published as a conference paper at ICLR 2026
Appendix
Table of contents. The supplementary includes the following sections:
Section A. Implementation details of the experiments.
Section B. Additional experiments.
Section C. Pre-training loss comparison between VMoBA and Full Attention.
Section D. Limitations and future work.
A
IMPLEMENTATION DETAILS
Table 4: Implementation detials of experiments in the main paper.
Training-fre free
Training
Video Resolution
81 480 832
81 720 1280
93 576
1024
141 480 832
Latent Resolution
21 x 30 x 52
21 x 45 x 80
24 x 36 64 3
36 x 30 x 52 3
Temporal Block Size
3
Spatial Block Size
5 x 13
x 10
6 x
5 x 13
Spatial- Temporal Block Size
7 x 13 7
x 15 x 20
8 x 12 x
12 x 10 x 13 7
Temporal Blocks
Spatial Blocks
24
40
48 72
24 36
Spatial- Temporl Blocks
72
36
Block Selection Type
Local + TopK 216 18
Local + Topk 211019
Global Threshold
Global + Threshold
&
0.25
0.25
The detailed hyperparameter settings of the main experiments are shown in Table Note that apart from the 3D block partition algorithm, we use local and topk query-key block selection in the training-free experiments, which is the same option as the original MoBA (Lu et al., 2025). We empirically find that adopting VMoBA's block selection strategy will cause vibration effects the generated videos. On the contrary, the original local and topk options can better preserve the generation stability, probably due their closer modeling nature to full attention. However, the full VMoBA setting in all training experiments, observing a stronger performance and efficiency gain compared MoBA, which demonstrates the effectiveness of VMoBA as a sparse attention algorithm designed for training video diffusion models.
B ADDITIONAL EXPERIMENTS
Table 5: Training results on original and shorter sequence lengths.
Vidco Size
Methed
Sparsity
Quality
Elacieecy
TeuCCrsis 1
Dynank 1
BOConsis 1
InageQual 1
SubCwaist
FLOPs
Triisin Time
Oriin 87 487 x XY2
FullAtts
20.32%
98-43%
92.39%
6..18
65.09%
382.64
104 GPU Houns (L00c)
DDTTRatAmm (Vaan al, 2004)
70.95%
95.17%
850115
563665
$1.22%
182 730 55
SVG (Ki at. 2025)
19.11%
97.309
950955
00.136
M4.12%
(1.564)
(100S)
Prorattd Wan 3.1
12.19
96.14
91.176
01 005
67171T
Shunee
(1 (0
(0x))
(1
SVG 2025)
20/9/8
48.009
07.815
621355
$1.17 37
15K Tikees MolBA clase 2003)
0:3
34.09%
S00WS
N2 30.855
07. 707
42 24.44
(1 58-3
118 CIU Flems 75x)
VMoBB
24.39%
35.005
62.9015
98.78%
#0JAT (1,75))
100 CPU (BU
We conduct additional training experiments on the original and shorter sequence lengths and show the results in Table 5. For the original sequence length (33K), VMoBA's speed is very close to the pre-trained full attention model, while for a shorter sequence length (13K), VMoBA becomes even slower. This is reasonable because the acceleration of VMoBA is more significant when the sequence length goes longer, as shown in Fig. 1. Moreover, in these two experiments, VMoBA consistently outperforms MoBA in both performance and efficiency, further demonstrating its effectiveness.
13

---

Published as a conference paper at ICLR 2026
C PRE-TRAINING LOss COMPARISON
Table 6: Model configuration of the pre- pre-training experiments.
Parameter
Layer
Head
Hidden Size
FFN Dim
5.6M
384
896
60.4M
217M
4480
526M
1152
M1.50M
SALAR44
632
Validation Loss
6.40
Validation Loss
2110
6330
1200
VINESA SVAM
6250
0175
.065
12892
Training Step
Training Step
(a) 11k sequence length.
(b) 46k sequence length.
Figure 7: Validation loss comparison for pre-training between VMoBA and full attention. Blue and red lines are full attention and VMoBA, respectively. Deeper colors represent bigger model sizes.
We conduct training ferhiing-from-seee from-scratch experiments to evaluate the pre-training ability of VMoBA, com- pared with full attention. Following MoBA (Lu al., 2025), we train two sequence lengths, 11k (77 x 288 x 512) for shorter length, and 46k (77 x 576 x 1024) for longer length. The 1-2-3D block numbers of VMoBA are 10, 48, and 60. The threshold 7 is shown in Table 6. We use the same architecture as Wan 2.1 (Wang et al., 2025). For VMoBA models, we replace all the self-attention layers with VMoBA, which is the only change. The results for 11K length are shown in Fig. 7a. Despite VMoBA's validation loss curves being consistently higher than those of full attention, the difference between them shrinking as the model size increases. This indicates that VMoBA performs closely with full attention for larger model scales. Moreover, the results for 46K length are shown in Fig. 7b. We observe close overlapped validation loss curves between VMoBA and full attention pre-training, further demonstrating that VMoBA potentially performs better at longer sequences, which is also revealed by MoBA (Lu et al., 2025). In conclusion, the pre-training experiments demonstrate that VMoBA is reliable alternative to full attention for the pre-tree pre-training of video generation models.
0.25. The model configuration are
D LIMITATIONS AND FUTURE WORK
As shown in Table 5, the speed up of VMoBA is sub-optimal when the sequence length is short, even though its FLOPs are lower. This may due the inconsistent memory distribution of selected blocks. The current FlashAttention (Dao et al., 2022)-based implementation may not be able to optimize the speed of the sparse attention mechanism. Future work may focus on more efficient and hardware- hardware-friendly implementations to make the practical speed of VMoBA closer to the theoretical FLOPs.
E USE OF LLMS
In preparing this paper, LLMs utilized ceneral-purponts assistive tool. Specifically, the of LLMs is strictly limited to proofreading the author author-written written text for grammatical errors, spelling corrections, and improvements language clarity. This application consistent with the of
14

---

Published as a conference paper at ICLR 2026
conventional grammar--checkng software and did not extend research ideation, data analysis, or the generation of any substantive content.
15


Reviews:
[{"review_id": "uBB859N8Ql", "reviewer": "Reviewer_UCKK", "summary": "The paper proposed VMoBA, an improved version of Mixture of Block Attention (MoBA) for accelerating the training process of Video Diffusion Models on long sequence inputs. The authors point out the deficiency of the original MoBA's 1D partitioning scheme and apply layer-wise recurrent block partition, equipping the model with more spatio-temporal awareness. Then, they substitute global block selection for MoBA's query-wise key blocks number selection, which resolves resource under-allocation by precomputing all the QK block similarities for each attention head and generating block masks for top-K key blocks. The author replace fixed Top-K selection with threshold-based block selection mechanism, which enables the model to dynamically adjust the number of blocks selected across different heads and better fit the varying nature of similarity scores. They benchmark VMoBA against several training-free approaches under VBench, and conclude that VMoBA indeed reduces training time and FLOPS while still providing comparable performance. Lastly, they conduct relatively complete ablation studies, and conclude that the design choice is indeed a proper one.", "questions": "see weaknesses", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, the paper could be considered as an all-round incremental extension for MoBA. \n1. VMoBA provides 2.92 times FLOPS reduction and 1.44 times speed up compared to original model, which shows certain practical feasibility in terms of training.\n2. The ablations are relatively complete, which ensures the model design is informed.", "weaknesses": "There several points about the main experiment that needs to be further clarified:\n1. In training-based part of the main experiments, no trainable sparse attention pattern is included, which lacks certain universality in terms of benchmarking. \n2. In the training-based part of the main experiments, only one baseline method designed for accelerating training is presented.\n3. The reason why methods for inference acceleration is used as a baseline for benchmarking training acceleration has not been clearly clarified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LIEVBLfnsy", "reviewer": "Reviewer_9PZV", "summary": "This paper proposes Video Mixture of Block Attention (VMoBA), a novel sparse attention mechanism tailored for Video Diffusion Models (VDMs). The authors introduce three key innovations: (i) a layer-wise recurrent 1D- 2D-3D block partitioning scheme, (ii) global block selection that selects the most salient query-key block interactions across all queries per head, and (iii) threshold-based dynamic block selection that adapts the number of attended blocks based on cumulative similarity. Experiments on long-sequence video generation show that VMoBA achieves faster training and fewer FLOPs than full attention, while matching or even surpassing its generation quality. VMoBA also performs competitively in training-free inference settings.", "questions": "- The hyperparameter \\tau controls the trade-off between speed and quality. If there exists a general value for all video types? If not, how to choose a proper \\tau for new videos or resolutions?\n- The paper notes that attention heads have different concentration levels. Have the authors analyzed whether VMoBA’s threshold-based selection leads to more specialized head behavior compared to full attention?\n- Have you tested VMoBA within a few-step distilled or consistency diffusion frameworks to verify compatibility with fast-sampling variants?\n- Could the cyclical 1-2-3D partitioning be learned end-to-end rather than fixed by layer index?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The three proposed modifications (1D-2D-3D partitioning, global selection, threshold-based sparsity) are well-motivated by the observed limitations of applying MoBA naively to video. Each component is clearly linked to a specific empirical observation.\n- The authors evaluate VMoBA in both training-based and training-free settings across multiple resolutions, using standard metrics (VBench, PSNR) and complete ablation studies. VMoBA achieves FLOPs reduction and training-time speedup with minimal loss in generation quality. Qualitative results further support the claims.\n- The writing is good and is easy to follow. Method illustration and implementation details are well documented.", "weaknesses": "- The study omits some recent linear or hybrid video attentions that could serve as stronger baselines, such as STA[1] and RainFusion[2].\n- The paper should include more human evaluation. Human judgment on video quality and video consistency is crucial for assessing the performance.\n- In the global selection part, this module prioritizes key blocks with the highest overall significance, but may overlook certain keys that are locally relevant to queries yet have low global scores. The high-frequency details in generated videos may be affected.\n- Have you tried other fusion methods to fuse the tokens in a block? Does mean pooling harm the diversity within a block?\n\n[1] Fast Video Generation with Sliding Tile Attention\n\n[2] RainFusion: Adaptive Video Generation Acceleration via Multi-Dimensional Visual Redundancy", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bvhRiH8jn9", "reviewer": "Reviewer_MLuW", "summary": "VMoBA introduces a trainable sparse attention mechanism tailored for Video Diffusion Models. Building on MoBA, it integrates a 1D–2D–3D recurrent block partition, global block selection, and threshold-based adaptive sparsity to better capture spatio-temporal locality in video data. Compared to full attention, VMoBA achieves up to 2.9× FLOPs reduction and 1.5× speedup during training, while maintaining comparable or improved generation quality, and also accelerates training-free inference for high-resolution videos。", "questions": "1. You use “sparsity” to mean the fraction of computation retained vs. full attention. Standard usage is density = kept fraction, sparsity = skipped fraction?\n\n2. Does 𝜏 =0.25 mean accumulating normalized attention mass to 25%? If so, it would seem a bit counterintuitive. I would think only covering 25% of attention score to bring huge quality loss. \n\n3. With thresholding, how much does actual density/e2e runtime vary across prompts and seeds? \n\n4. How is the sparsity in Table 2 calculated? Does it take into a account the fact that the first 25% timestep is full attention?\n\n5. Table 2 lists MoBA dynamic degree ≈ 5.8%. Is this value correct? Why is it so low compare to other methods?\n\n6. Why is “Dynamic” absent in Table 1, and why are only five VBench aspects reported?\n\n6. Why presents training-free results at 81×720×1280  for Wan-2.1-1.3B, a model trained at 480p?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper diagnoses three phenomena in DiT attention—1D/2D/3D locality, uneven query importance, and head-wise concentration—then maps them to three design choices (1–2–3D recurrent partitioning, global selection, and thresholded selection). This “observations → mechanisms” linkage is well argued.\n2. Unlike training-free sparse attention method, VMoBA is a trainable block-sparsity scheme intended to replace full attention during training . This positions it to deliver training compute savings where many inference-only methods (DiTFastAttn, SVG) cannot.\n3. On Wan-2.1-1.3B fine-tuning with extended tokens (55K/56K), VMoBA reports 2.83–2.92× FLOPs reduction with 1.44–1.48× end-to-end training time speedup, while matching or slightly exceeding Full Attention on several VBench aspects (e.g., ImageQual +3.3% at 141×480×832).", "weaknesses": "1. The paper reports ~2.40× FLOPs reduction but only ~1.35× end-to-end latency speedup in the training-free 720p setting. A deeper breakdown (kernel MFU, QK/softmax/AttnV time, IO cost) is needed to explain the under-translation from theoretical to realized speed.\n2. Several contributions hinge on MoBA being less efficient, yet the paper does not thoroughly analyze why MoBA is inefficient and why the proposed methods can make it more efficieint.\n3. Strong, recent sparse-attention baselines are missing. In particular, SpargeAttn (universal, training-free) and STA (trainable) both report substantial end-to-end gains on state-of-the-art video DiTs; including them would calibrate the claimed benefits. \n4. Since VMoBA is trainable, the paper should report backward attention timing, MFU in backward, or end-to-end training throughput (e.g., it/s or tokens/s) againt full attention.\n5. Only five VBench aspects are reported. Given VBench’s breadth, either report the full score or provide a principled selection rationale; when PSNR drops, include human blind-preference to reflect perceptual quality in Table 1. Also consider including VBench 2.0.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TdAHRgxp6e", "reviewer": "Reviewer_89qF", "summary": "The paper examines the quadratic complexity of full attention mechanisms and proposes a novel and efficient sparse attention mechanism for video diffusion models. The presented method is called Video Mixture of Block Attention (VMoBA), which is an improvement form existing Mixture of Block Attention (MoBA) from three different key modifications. The contributions of this work lies in three different aspects as outlined: a cyclical 1D-2D-3D block partition, a query-global block selection and a threshold-based block selection. All these contributions are well documented and related to the observations and motivations.", "questions": "1. Do the cyclical 1D-2D-3D scheme generate the tokens independentlyor eventually resulting in a 3D patch token based on the Eqns. (1) and (2)?\n\n2. Can the proposed scheme generalize to other full attention DiTs? If yes, what is the common assumption for this generalization? Have you explored other full attention DiTs for the proposed method?\n\n3. The paper relies on many insightful observations, which is good. But how will these observations generalize, especially when certain hyper-parameters were used?\n\n4. Could the paper also elaborate on how the proposed schemes are particularly useful for the long video sequence generation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main strengths of this paper is to adapt the existing Mixture of Block Attention (MoBA) for Video Mixture of Block Attention (VMoBA). This is not a trivial adaptation as the paper performs insightful analysis to draw the useful conclusions. \n\nThe paper performs the analysis on a full attention DiT model and observe that different layers may have its own focus on spatial, temporal and its combination with respect to different layers. This put forwards the idea of layerwise block partition for 1-2-3D. This types of pattern distinction seem not be widely confirmed in the previous video diffusion models. \n\nRegarding the global block selection, the paper also makes a good observation by identifying the limitation of query-independent selection. For each attention head, all the queries are grouped to perform the block section. To determine the number of blocks selected, the paper adopts a threshold-based Block Selection to determine the number of selected blocks dynamically. This strength might not be that significant compared to the other two.", "weaknesses": "The paper also shows several weakness that requires further attentions. The paper uses Wan 2.1 1.3B as the base model and implements the proposed idea. All assumptions such as the 1-2-3D attention patterns were derived from such full attention DiT. It is unclear whether this VMoBA can also be adapted to other video diffusion models based on full attention DiT. This needs to be further validated with other full attention DiT models. This could be the main weakness of this paper so far. \n\nThe paper could better lay out the performances for the proposed method compared to the others listed in Tables. The paper uses the highlighted color to indicate the performance from the proposed method. However, it is better to indicate which metric the proposed algorithm performs the best, similar to other used baselines. After all, based on the given tables results, the proposed method does not perform the best among all the metrics. This could present some confusions. \n\nThe assumption about heads having different concentration levels motivate the design threshold-based block selection. This assumption also relies on some empirical observation where 30% and 50% cutoff lines for each head was used. These numbers could be different across different models. It will be nice to have a more generalized treatment of these selections. \n\nThe paper shows some interesting results on training longer sequences, which is critical in video diffusion models. How this long-sequence capability modeling is related to the proposed innovations are not very well explained. The paper could offer more insights on why the proposed designs are more suitable, not just improving efficiency and maintaining the accuracy simultaneously.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)