MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generation
Authors: Weinan Jia
Yuning Lu
Mengqi Huang
Hualiang Wang
Binyuan Huang
Nan Chen
Mu Liu
Jidong Jiang
Zhendong Mao
Venue: ICLR 2026 Poster (2026)
Keywords: video generation
Primary area: generative models

Abstract: Long video generation with Diffusion Transformers (DiTs) is bottlenecked by the quadratic scaling of full attention with sequence length. Since attention is highly redundant, outputs are dominated by a small subset of query–key pairs. Existing sparse methods rely on blockwise coarse estimation, whose accuracy–efficiency trade-offs are constrained by block size. This paper introduces Mixture-of-Groups Attention (MoGA), an efficient sparse attention that uses a lightweight, learnable token router to precisely match tokens without blockwise estimation. Through semantic-aware routing, MoGA enables effective long-range interactions. As a kernel-free method, MoGA integrates seamlessly with modern attention stacks, including FlashAttention and sequence parallelism. Building on MoGA, we develop an efficient long video generation model that end-to-end produces minute-level, multi-shot, 480p videos at 24 fps, with a context length of approximately 580k. Comprehensive experiments on various video generation tasks validate the effectiveness of our approach. Project website: https://jiawn-creator.github.io/mixture-of-groups-attention/ .

Published as a conference paper at ICLR 2026
MoGA: MIXTURE-OF-GROUR: ATTENTION FOR END- TO-END LONG VIDEO GENERATION
Weinan Jia 1, Yuning Lu Menggi Huang" Huullang Wang21,
Binyuan Huang Nan Chen', Mu liu?, Jidong Jiang", Zhendong Mao!
1 University of Science and Technology of China FanqieAl, ByteDance China
3 Hong Kong University of Science and Technology Wuhan University
j llaw@maall 1. ust C. edu. cn, Layuningebytedann com, huangmguustc.edu
ABSTRACT
Long video generation with Diffusion Transformers (DiTs) is bottlenecked by the quadratic scaling of full attention with sequence length. Since attention is highly redundant, outputs are dominated by small subset of query-key key pairs. Existing sparse methods rely on blockwise coarse estimation, whose accuracy-efficiency trade-offs are constrained by block size. This paper introduces Mixture-of-Grrups Attention (MoGA), an efficient sparse attention that uses a lightweight, learnable token router precisely match tokens without blockwise estimation. Through semantic-aware routing, MoGA enables effective long-range interactions. As a kernel-free method, MoGA integrates seamlessly with modern attention stacks, including FlashAttention and sequence parallelism. Building on MoGA, we de- velop an efficient long video generation model that end-to-end produces minute- level, multi-shot, 480p videos at 24 fps, with context length of approximately 580k. Comprehensive experiments on various video generation tasks validate the effectiveness of our approach. Project website: hettp:://Laann-craate creator.. C.c.uub.oomiittuee of groups attrass-ttenniion
INTRODUCTION
growing body of research indicates that scaling laws are a primary driver of progress toward
artificial general intelligence (Brown et al. 2020 Team al. 2023 Kaplan et al. (020). As model parameters and data scale to billions, transformer- foundation models (Vaswani et al. 2017)
often exhibit emergent capabilities (Wei al. 2022 Kaplan et al. 2020 Radford et al.) 2021). In
video generation, given the inherently temporal nature, progress requires not only scaling parameters and data but, more critically, scaling the effective context length. This need is especially salient for long-form video generation (e.g., movies), where persistent memory is essential for maintaining consistency of environments and characters (Yu et al. 2025).
The main challenge of vanilla attention (Vaswani et al. 2017) for long sequences is its computa- tional cost, which grows quadratically with the context length. To mitigate the challenge, prior work (Zhuang et al. 2024 Tian et al. 2024 [Huang et al.) 2025b) Xiao et al.) 2025; Wang et al.) 2025a) adopts a multi-stan stage pipeline that first generates key frames and then synthesizes intermedi- ate frames. However, this design yields disjoint objectives that are not directly optimized for the end task, leading to error accumulation across stages. It also introduces hand-crafted inductive biases, hindering scalability.
For end-to-end long video generation, one line of work compresses historical content to accommo- date longer contexts (e.g., via recurrent layers (Dalal et al. (2025) or FramePack (Zhang & Agrawala, 2.25)), which inevitably results in information loss. A complementary direction exploits sparse al- Tention Zaheerr et al., 2020 by restricting computation
selected subset of salient query-key pairs. Existing selection strategies generally fall into two categories: (i) static selection, iee, prior-
Corresponding authors.
Work done during internships at FanqicAl, ByteDance China.
6 Projecc leader.

---

Published as a conference paper at ICLR 2026
Irput Video Tokons
Input Video Takens
Input Video Takens
Input Video
Pooling
Taken Router
5 Emmonn Salorny b Scows
Group Self Att--inn
Softmax
Token
Attention Score
Blocks
Weight
Choup
Croup
Groug
Token Rearnanger
Blocks Similarity Map
Dense Compute
Similority Confusion
Efficient Sporse
(a) Full Attention
(b)
Block Sparse Attention
(c) Mixture-of-Groups Attention
Figure Illustration of our motivation. (a) Full attention suffers from dense computing when dealing with long sequences. (b) Block sparse attention may fail when block-level similarity is confused, resulting in unreliable attention. (c) Mixture-of-groups attention uses a lightweight token router (i.e., single linear layer) that assigns tokens to specialized groups, enabling groupwise attention and efficient long-context modeling.
driven heuristics that emphasize local spatiotemporal neighborhoods, which is efficient but limited in capturing dynamic long-range dependencies (Li et al., 2025 Xi et al. 2025 Gao et al. 2025
Seawead et al. 2025); and (ii) coarse-to-fine dynamic selection, which first estimates block-level important scores, routes query tokens to the top-k blocks, and then applies fine-grained attention
within the selected blocks (Wu et 2025 Cai et al.] 2025) Yang et al. (225 Yuan et al. [025
Lu et al. 2025). As shown in Fig I (b), the latter introduces an efficiency-perfommace
trade-off: using larger blocks with small top-k reduces the computational cost of the coarse stage but reduces selection performance.
In this work, we reveal that such gaasee-raiied estimation is unnecessary and each token should be precisely allocated. To achieve this, we propose Sexxure-oofGroupee Attention (MoGA), simple and efficient dynamic token routing solution for end-to-end long video generation. lightweight router (i.e., a single linear layer) is employed to assign tokens to specific groups, as illustrated Fig. [Tc) inspired by Mixture-of-Experts (MoE) (Jacobs et al. 1991). Full attention then performed within each group, where the groupwise attention integrates seamlessly with modern attention kemnels, e.g., FlashAttention (Dao) 2023). Intuitively, the linear router's 's weight can be viewed implicit cluster centers, enabling direct assignment of tokens to learnable anchors, without global similarity estimation. Furthermore, to balance long-range coherence and local fidelity, we couple MoGA with the spatiotemporal window attention (Gao al. (025), which can be considered as the groupwise attention with static and pre-defined groups. In addition, extended context alone insufficient because single global prompt cannot reliably control scene transitions or orchestrate events at precise time points in long videos. We therefore introduce shot-level textual conditioning via modal attention, where each shot is guided by concise description (Gu al.) 2025 Wang et al. 20256). To support this, we build data pipeline that produces minute-level video samples with dense, multi-shot captions and reliable shot segmentation.
We propose MoGA, an effective sparse attention mechanism that replaces block- level scoring with precise group assignment via lightweight token router, enabling effective mod- eling of long contexts. Building on MoGA, we introduce a video generation model capable of pro- ducing minute-level, multi-shot, 480p videos at 24 fps with a context length about 580k tokens. Extensive evaluations show consistent improvements over state-of-the-arr (SoTA) sparse attention baselines and a multi-shot video generation model.
Our contributions:
2 METHOD
In this section, we introduce MoGA for efficient long video generation. The overview of the whole architecture is shown in Fig. 2 We first present the preliminaries, then detail MoGA, and finally describe the pipeline for constructing multi-shot long-video training data.
2

---

Published as a conference paper at ICLR 2026
Model Architecture
(a) Visual Attention
Spotial
Full
Temporal
sho
shot
ShOP
Spatial Temtioll
VAE Decoder
Video Tokens
Group Attention
Output
Projection Unpotchify
Sexcture-of-Groute
Tokens
Attention
NN
Cross-Madal Attention
(b) Mixture-of Groups Attention
DIT
Visual Attention
Text Encoder
- -00 000 DOC
shor
AnnnM
Token Rearranger
Token Router
Group
Patchify
Encoder
Caption Captunn
Group
man is
sitlin
showa
shot
ShoL
shat
tbke
Group Attention
Figure 2: Left: Our model adopts DiT architecture with interleaved Visual Attention and Cross- Modal Attention blocks. Visual Attention exclusively processes visual content, whereas Cross- Modal Attention enables shot-level text conditioning, instantiated via either cross-attention (Wan
al. 2025) or multi-modal attention (Kong et al. 2024] Esser et al. 2024). Top-right (a): Visual Attention combining MoGA with Spatial-Temppral Group Attention for global-local consistency. Bottom-right (b): MoGA, where Router groups tokens and performs intra-group attention, en- abling long-range global interactions.
2.1 PRELIMINARY
Vanilla self-attention Vaswani al. (201)) plays crucial role in video generation with Diffusion
Transformer (DiT) Peebles & Xie 2023). ). Consider an input sequence X I Nxd where N
X represents the total number of tokens across the latent spatial dimensions (h w) and
the latent temporal dimension (f), with denoting the model's 's hidden dimension. For simplicity, we consider a single query case, where æ is token from the input sequence and q is its corresponding query. Vanilla self-attention (SA) is computed as:
qK* Vd
SA(q, K, V) = softmax
V.
(1)
where K and V denote the keys and values. While self-attention excels at capturing long-range dependencies via global information aggregation, it incurs a quadratic computational complexity of O(N2). The computational burden becomes particularly prohibitive in long-video generation. For example, generating a I-minute video 480p with approximately 1,600 tokens per frame across 961 frames (16 fps) yields total token count approaching 384k] Performing full-attention such a long sequence is intractable.
Beyond computational cost, full attention is not ideally aligned with the structure of videos. In videos, softmax attention is inherently sparse Xi et al. (2025) because nearby tokens exhibit strong local spatiotemporal correlation, while only few globally shared, dynamic semantics persist across frames. Most query-key pairs contribute little, whereas a small subset dominates [Ge et al. (2023). For long videos, attention should leverage this sparsity by prioritizing important query-key interac- tions to reduce redundancy.
2.2 MIXTURE-OF-GROUS ATTENTION (MoGA)
MoGA addresses the above challenge via efficient token routing, where a lightweight trainable router assigns correlated tokens to groups and performs self-attention within each group. Specif- ically, the router
linear projection followed by softmax gating, similar to MoE Fedus et al. 2022). MoE scales model parameters by routing tokens to expert FFNs. In contrast, MoGA scales with respect to sequence length by modifying attention and routing tokens into different attention
Following
Wan et al. 2025 VAF downsampling factors of (t/h/w (4,8,8 and patchify sizes is (1,222).

---

Published as a conference paper at ICLR 2026
Shot
group visuelization
Shat
group visuelization
Shot 3
group visuclizaaion
Figure 3: Visualization of dynamic router grouping. We visualize the visual patches of tokens from the same group assigned by an intermediate block's router. They focus on specific visual concept across frames.
groups. Given token Id and predetermined number of groups M, the router computes routing scores r E RM as:
r Router(
The group assignment probabilities are computed as:
pii ar) noftmax(());
and the token assigned to the group with highest probability:
g (x) arg max p(ilæ:). iE[M]
(4)
After grouping, we apply self-attention within each group independently. The output of MoGA is:
MoGA(x) p(g(x))x) SA(q, K g(z)), Va(ar),
(5)
where K ((()) and V ((()) are the keys and values of the group g(x), and is the query feature of æ.
This grouped attention mechanism reduces computational complexity from O(N2) to theoretical minimum of ((N2/M) under uniform group assignment.
As illustrated in Fig. 3 we extract the grouping assignments from an intermediate- layer router dur- ing the video generation process and visualize one representative group. After end-to-end training, the router assigns the man's 's head, hands, and portions of his clothing to the same group, indicating its ability to capture semantically coherent structures that span shot boundaries.
MoGA builds on groupwise attention and remains compatible with high-performance kernels such as FlashAttention (Dao (023) (see Alg. [T. Beyond sparse attention, a second pillar of long-contex modeling is sequence parallelism (Jacobs et al.) 2023), with which MoGA is also compatible. Before the sequence gather and head scatter step in each attention layer, MoGA computes routing scores over tokens (with whole heads) and then aggregates the routing results across all tokens.
Group Balancing Loss. A potential issue with token assignment is that the router may collapse by routing most tokens to only few groups, which would degrade MoGA into full attention. To encourage adaptive token allocation across groups, we introduce an auxiliary group balancing loss, inspired by the load balancing loss (Fedus al. 2022) used MoE. The loss is defined as:
Czb= = a- M Σ F. P.
(6)
where a is loss weight and F; is the fraction tokens assigned to group i,
F \ -)).
(7)
where 1 is the indicator function, and P3 the mean routing probability allocated for group i,
P.
Σ p(g(x))æ))
(8)
g(Ø)=1
Minimizing Lgb encourages uniform token assignment across groups, as this objective attains its minimum under uniform distribution (Fedus et al. 2022).
Spatial-Tempore Group Attention. Although MoGA captures long-range coherence, it lacks local continuity. We complement with local spatiotemporal group attention (STGA) (Gao al. 2025
4

---

Published as a conference paper at ICLR 2026
Algorithm MoGA Pseudocode with FlashAttention
Q, K, are the query, key and value of tokens X
2: = router(XX XX)
> MoGA routing results
3: Q, K, V, cu seq len, max seq len, permute index = permute(Q, V,9)
4: 0 flash attm Q K, V, len, max seq_see
5: O repermute O, permute.index)
>
MoGA recovers the original token positions
Raw Videos
video-level
shot-level
Training Samples
Video Score
Video Filter
Shot Split
Shot Score
Shot
Shot
Shot Merge
Filter
Crop
Caption
Figure 4: Multi-shot long video data pipeline.
Zhang et al. 2025c), which restricts self-attention to local windows latent video space, as shown in Fig. 2 a). This captures short-range dependencies with bounded compute.
We first partition the latent video into fixed spatial windows and then group frames along the tem- poral axis. Frames from different shots are assigned to distinct temporal groups. We empirically find that completely removing inter-shot interactions causes flicker in the first frame after a shot cut. To mitigate this, when computing group attention, augment the keys and values with two latent frames from adjacent shots (without augmenting the queries). This preserves continuity at shot boundaries with negligible additional compute. To enable intra-frame information exchange, we also perform per-frame attention by grouping tokens within each latent frame. Each token there- fore receives outputs from multiple groups (one dynamic group and two static groups), and take their mean as the final output.
2.3 DATA PIPELINE
We construct a pipeline that converts raw long videos into one-minute, multi-shot clips with dense annotations for long video generation. The pipeline has two stages: video-level phase and shot- level phase (Fig.
Video-level. We first analyze raw videos using visual quality assessment (VQA) models (e.g.. aes- thetics (Schuhmann et al. 2022b). clarity, exposure) and simple operators (e.g.. black-border detec- tion) obtain metadata and quality scores. We then filter raw videos with source-specific, calibrated thresholds to remove low-quality content. Because long-video samples requires temporally coher- ent, relax clip- filterin(Zheng al., Kong et 2024) but apply stricter filtering the source (raw-video) level. Next, we segment each vidco into single-shot clips. using AutoShot (Zhu) al. 2023) and PySceneDetect (Breakthrough & Contributors 2014). AutoShot shows higher sen- sitivity to fades and gradual transitions. Combining both tools" predictions allow us to label whether a boundary clean affected by transition overlap. This stage yields pool of single-shot clips.
We process single-shot clips using VQA and optical character recognition (OCR) models and discard low-quality clips. Based on OCR results, we compute maximal- areea crop that excludes watermarks and subtitles while preserving the original aspect ratio. Clips with insufficient retained area are discarded. Next, we generate captions of cropped clips by employing multimodal large language model (Bai al. 2025). Finally, merge temporally adjacent single-shot clips into multi-shot training samples (up to 65 seconds) and trim a few frames from clips affected by transition overlap to ensure clean boundaries.
Shot-level.
EXPERIMENTS
Training settings. We fine-tune MoGA on existing DiT-based short video generation models with the rectified flow objective Esser et al. (2024). For a fair comparison with baselines, we train MoGA on the open-source Wan2. models (I.3B and 14B) (Wan et al.) 2025). The resulting model stably generates 477 frames at 16 fps (30-second) at 480p. with a context length 187k. We use a constant learning rate of le-5. The loss weight is set 0.1. We set the number of groups to M 5 and
5

---

Published as a conference paper at ICLR 2026
Method
Base Medel
Croustiacy
Subject
Background Consalency
Motion Smowtlnes
Acuihatic Quality
Imsge Quality
TextZVilee
CLIP
Orand Comsitenc
Tanpeest Flickrrigg
Sporsity
(Cingreal)
DITIFa Nin Thaikinghhaoo froouh
0.8706
0.900
0.5369
0/466
0.1611
0.1661
0,89
50 50:005
SVG (hairlee--tre)
0J8976
819771
0.577
0A757
03516
0.1090
03714
91:015
(Trining
McGA (Dan)
W.r2.3-1.38
0.0533
69962 0:9522
0.5656
0.5319
9,5533
0,500
0,159
0.921
71.35%
McGA (Dun)
Ra.2211148
0.9699
B5997
0.501
0,699
0.2576
0.1743
0.811
71,35%
Table 1: Quantitative comparison of 5-second single-shot video generation.
Method
Base Model
Subject Consistency
Background Consistency
Smoothness
Austhetic Quality
Image Quality
Text2Viden
CLIP 1
Cross-hhots
DINO 1
Cros-Shots
CLIP t
IC-Lora+Wan Wan
Wan2.1-1
0.9476
0,9538
0.9901
0.5237
0.6684
0,2381
0.4669
0.7169
Echoshot
Wan2.1-1.3
0.9544
0.9518
0,9939
0.5718
0.6534
0.2535
0.5961
084699
MoGA (Ours)
Wan2.1-1 1-1.3B
0,9549
0,9597 0.9679
0.9919
0.5890
0.6729
0,2582 0.2783
0.6623
0,8654
MoGA (Ours)
Wan2.1-11 1-14B
0.6867
0.8629
Table 2: Quantitative comparison of 10-second multi-shot video generation.
partition the spatial grid into 2 x 2 groups. We adopt multistage training strategy: 3k steps on 10-second clips followed by 1 steps on 30-second clips.
Because MoGA is a general sparse attention, we also apply it to video generation model built on MMDIT (Esser et al., 2024, Kong et al.) 2024). Unlike Wan, this model replaces cross-attention with MMDIT to perform cross-modal attention. It partitions space into 4x4 groups and sets the router's group number to M = 20, enabling a much longer context length. This MMDiT-based model generates 1,441 frames at 24 fps (60-sccond) at 480p, with context length of 578K.
Baselines. To evaluate our method, we compare with multiple baselines. For mulli-shof long video generation, we include the keyfraame-bet based pipeline IC-LoRA+Wan (Huang et al. 2024a Wan jet al., 2025) and EchoShot (Wang et al. 2025b), which natively supports multi-shot generation. For sparse video generation, compared against sparse attention methods, including the training- based method DiTFastAutn (Yuan et al. 2024) and the training-free methods SVG (Xi et al. 2025 and VMOBA (Wu al. 2025).
Evaluation metrics. Following prior work, we evaluate all methods using the metrics introduced by VBench (Huang al.] 2024b). Specifically, subject consistency and background consistency measure how well the main subjects and backgrounds of sampled frames are preserved throughout the video. Motion smoothness measures motion fluidity, penalizing jitter and abrupt transitions. We also report aesthetic quality and image quality to quantify the visual appeal and technical fidelity of each frame. Video CLIP measures the semantic consistency between generated video and its text prompt. Temporal Flickering reflects the temporal stability of videos. To compute cross-shot consistency, we first sample fixed number of frames from different shots. We then compute feature similarities across shots using CLIP (Radford et al. (021)) and DINOv2 (Oquab et al. 2023) feature similarities across shots, referred to as Cross-Shot CLIP and Cross-Shot DINO. For single-shot 5-second video generation, we constructed diverse test set comprising 300 prompts. For multi- shot 10-second video generation, we use the 100 multi-shot prompt sets, which introduced from (Wang et al.) 2023b). We evaluate long video generation with test set of 11 scripts comprising 105 prompts. Each script contains 8-10 shots to produce a 30-second video.
3.1 QUANTITATIVE RESULTS
First, we compare MoGA with prior sparse attention methods for single-shot, short video generation, following their evaluation settings to ensure fairness. As shown in Tab. [ despite higher sparsity, MoGA achieves consistent improvements over existing sparse baselines across metrics. It is worth noting that although our method highly sparse, it can still match or surpass original Wan (full attention) on multiple dimensions.
Next, we compare MoGA with other multi-shot video generation methods. Tab. 2 reports quan- titative comparisons among MoGA, IC-LoRA+WAN and EchoShot. Despite relying sparse attention, our method outperforms the full attention baseline (EchoShot) on most metrics, indicating that preserving interactions among salient tokens not only reduces FLOPs but also suppresses noise from irrelevant content. This leads to stronger character identity consistency and improved temporal scene coherence.

---

Published as a conference paper at ICLR 2026
Subject Consistency
Background Consistency
Motion Smoothness
Acsthetic Quality
Image Quality
Tex12Vidco
Method
Base model
CLIP
IC-Lora+ Wan
Wan2.1-144
0.8946
0.9169
0.9872
0.5759
0.6835
0.2547
MoGA (Ours)
Wam2. 1-1.3B
0.9218
0.9204
0,9846
0.5731
0.6829
0.2579
MoGA (Ours)
Wan2.1-14 1-14,
0.9572
0.9475
0,98993
0.5789
0.6993
0.2634
MoGA (Ours)
0.9305
Table Quantitative comparison of 30-second multi-shot long video generation.
T2I (IC-Lora)
Inconsistent
I2V (Wan2.1)
EchoShot
Incon
[Shott]
basket
[Sharā]
with aladd
[Sho18] weman smile
[Shord] pororomic thot Tham
MoGA (Ours)
[Shat5] womar
[5hot6] fill face ani
[Sha
[Shal8] wittron looks laff
Figure 5: Qualitative of MoGA and other methods. We present eight representative shots, demon- strating long-range coherence, character consistency, and visual quality.
Finally, we benchmark long video generation against the baseline. Because few open-source meth- ods can produce 30-second, multi-shot videos, we compare MoGA (with two backbones) to IC- LoRA+Wan As shown Tab. 3 MoGA substantially outperforms IC-LoRA+War under the same backbone, highlighting the benefits of end-to-end modeling over multistage pipelines. Notably, even under aggressive sparsity, MoGA with MMDiT maintains high visual fidelity, indicating scalable path to longer context lengths.
3.2 QUALITATIVE RESULTS
In this subsection, present qualitative results on 30-second videos across representative base- lines. Because EchoShot cannot natively produce 30-second outputs, we concatenate video clips generated by EchoShot to form the full sequence. As shown in Fig. 5 the IC-LoRA+Wal pipeline is constrained by its per-iteration image cap (typically three frames), which limits its ability to cover larger number of shots. Consequently. often exhibits subject drift and background inconsis- tency as the sequence progresses. EchoShot scales to more shots but still shows notable cross-shot inconsistencies long temporal distances. In contrast, MoGA maintains stable, coherent content

---

Published as a conference paper at ICLR 2026
x10+s
Full Attention
MoGA (M=10)
MoGA (M=5)
(M=20)
Compute (FLOPS)
5
15
30
Figure 6: Computational efficiency. The x-axis denotes the generated video duration (s). As the number of groups increases, MoGA's Flops decrease substantially.
Group num
Cross-shot CLIP
Cross-shot DINO
Sparsity
PFlops
0.8206
0,5919
0%%
0.88
2
0.8589
0.6761
41.25%
0.59
4
0,8672
0.6853
66.25%
0.42
8
0.8606
0.6910
78. 78.75%
0.36
16
0.8569
0.6896
81.25%
0.35
Table 4: Results of consistency for MoGA with Wan2.1-1.3B on 10s videos.
over extended durations. For example, even without repeated or explicit specification across shots, the woman's 's hat remains consistently preserved. Since STGA lacks explicit cross-shot informa- tion exchange, this consistency can be attributed to MoGA, which effectively selects and maintains shot-spanning identity and context. We provide more visualization in appendix A.1
3.3 ABLATION STUDY
Computational Efficiency. Fig. 6 plots the relationship between the number of groups (M) and FLOPs for the Wan3.1-1.3B model. Our experiments show that even with relatively small groups count M 5 for 30-second videos, MoGA achieves substantial computational savings compared to full attention (2.26 PFlops vs. 6.94 PFlops). Meanwhile, it also delivers a 1.7x speedup in both training and inference. Notably, unlike alternative sparse attention such as VMoBA, which ineur additional memory overhead due to their block-based mechanisms, our approach maintains memory efficiency without additional memory consumption.
Routing Group number M. We conduct an ablation study on the number of groups under fixed computational budget (in Tab. 4.. Cross-shot DINO and CLIP scores exhibit rise-then-fall trend the number of groups increases. This suggests that moderate level of grouped sparsity strikes a balance between global consistency and efficiency, yielding near-optimal consistency while main- taining computational efficiency.
Effectiveness of MoGA and STGA. As shown in Fig 7 MoGA and STGA play complementary roles in enabling context-consistent long video generation. Using MoGA alone lacks local infor- mation exchange and fails to produce meaningful visual content. Conversely, using only STGA limits long-range shot interactions, leading poor cross-shot consistency and weakened narrative coherence. When combined. the model achieves strong cross-shot consistency. These results indi- cate that MoGA effectively routes and preserves shot-spanning identity and context at relatively low computational cost.
Controllability of subject consistency. Fig. g provides a comparison between MoGA and full attention. Both models are trained on 10-second data with Wan2.1-14B. The left panel illustrates MoGA's ability to maintain subject identity across multiple scenes, while the right panel demon- strates its robustness to appearance changes (e.g., clothing) when preserving identity consistency. Despite 71.25% sparsity, MoGA achieves narrative coherence and content editability on par with full attention, and in some cases delivers superior performance.
Quantitative analysis of token routing. We treat the router's token grouping as a form of unsu- pervised semantic segmentation and use SAM2 Ravi et al. (2024) to generate foreground masks for each video frame as ground truth (GT). For each GT mask, we match the group whose prediction yields the highest Intersection- cereetioooovv---eeee Union (IoU) and take that IoU as the per-frame score. Final per- formance reported as the average over different cases. Trained MoGA clearly surpasses random and untrained routers (28.6% vs. 15.6% 10%61885%% Tab. 5). Over timesteps, IoU rises from 17.6%
8

---

Published as a conference paper at ICLR 2026
Only MoGA
Only STGA
MoGA STGA
Shot
inconsistent
Consistent
Shot 2
Figure 7: Visual ablation of MoGA and STGA.
Full Attn
MoGA
Full Attn
MoGA
Figure 8: Visual comparison of MoGA vs. full attention on multi-shot generation with single subject. Left column shows the subject wearing the same outfit across different shots. Right column shows the subject changing outfits at shot transitions according to the text instructions.
1=999 to 28.6% and then stabilizes (Tab. 6 Across different DiT blocks, middle layers are strongest (up to 31.0%: Tab. 7.. For more details, please refer to A.10
RELATED WORK
4.1 LONG VIDEO GENERATION
Previous work long video generation beyond typical duration limits has converged on three main paradigms. Multistage methods decompose long video generation into multiple steps YYn
al. 2023 Zhuang al. 2024 Tian et al., 2024| Huang al.) 2025b] Xiao al. [025 Wang
et al. 2025a For example, Captain Cinema (iiao et al.] 2025) adopts hierarchical planning with top-down keyframe generation and bottom-up synthesis for narrative coherence. Multistage ap- proaches introduce hand-crafted inductive biases and pose challenges for end-to-end optimization.
Autoregressive approaches generate videos through sequential segment synthesis (Chen ct al. 2024
Huang et al. 2025c; Yin et al. 2025 Henschel al. 2025 Gu et al. 2025 et al.) 2025 Dif-
fusion Forcing (Chen et al.) 2024) adapts denoising schedules for variable sequence lengths. Ca-
sus Vid (Yin 20025) distills bidirectional models into efficient autoregressive model. Stream-
ing (Henschel et al. 2025) combines short- and long-term memory for streaming video exten-
sion. FAR (Gu al. (025)) introduces hierarchical causal representations for multiscale dependen- cies. MAGE-T (ai et al., 2025) demonstrates the scaling capability of this paradigm. Context com-
pression methods address computational constraints by compressing historical content (Dalal al.
2025 Zhang & Agrawala 2025 Diang et al.) 2025 Huang et al.) (025a). TTT (Dalal et al. 2025
compresses long context via bidirectional recurrent layer. FramePack (Zhang Agrawala 2025
employs importance- based frame compression to maintain fixed computational budget. Further- more, M4V (Huang al. 2025a) augments the original MMDiT blocks with modified Mamba (Gu & Dao 2024) tailored for multimodal and spatiotemporal modeling, achieving lower FLOPs and re- ducing training/infereee
latency while maintaining quality. However, these methods either produce
videos of limited duration (Chen et al. 2024 Huang al.
2025c al al. 2025 fail to gener-
ate multi-shot videos in real-word wollwwrld scenes Yin et al. [2025]
Henschel et al.
2025 Gu et al., 2025]
Dalal et al. 2025
Zhang & Agrawala 2025 Jiang al.
2025). closely related line of work

---

Published as a conference paper at ICLR 2026
Random
MoGA (Before Training)
MoGA (After Training)
IoU (%)
15.6
18.5
28.6
Table 5: Comparison of IoU before and after training.
Time Step
999
956
931
884
853
499
438
364
272
155
IoU (%)
17.6
22.2
25.9
25.1
24.8
27.8
27.6
27.3
28.0
28.6
Table 6: Comparison of IoU across different time steps.
Block Index
10
15
20
25
30
35
38
IoU (%)
19.2
18.4
31.0
23.6
26.6
28.9
28.0
30.6
24.5
Table 7: Comparison of IoU across different blocks.
LCT (Guo et al. 2025), which models interleaved multi-shot prompts and videos within local context window using full attention. While pioneering end-to-end multi-shot long video generation, LCT remains constrained by the quadratic cost of full attention.
4.2 SPARSE ATTENTION FOR VIDEO GENERATION
Attention blase foundation models unify many domains and consistently exhibit a common spar- sity structure (Lu et al. 2025 Yuan al. 2025 DeepSeek-AT 2025). In video generation, given the inherent sparsity, natural approach to efficient generation is to select important query-key pairs. Prior work broadly falls into two categories: static priors (Zhang et al.) [025a Xi et al. [0255] Li
ett al. (225)) and coarse-to-fin dynamic routing (Wu et al. 2025: Yang et al. 2025 Zhang et al.
2025b). Among static approaches, STA (Zhang et al. 2025a) employs 3D sliding windows with hardware-award implementation. SVG (Xi et al. 2025) uses online pattern selection to classify attention heads as spatial or temporal sparse attention.
Antial-Attention (Li al. al.) (202)) uses fixed attention mask whose sparsity grows with the query-key distance to perform spatiotemporal atten- tion with (n log n) complexity. However, these methods struggle to modeling evolving long-range dependencies, which crucial for maintaining cross-shot consistency. Moreover, the implemen- tation of fixed mask (e.g., videos (300 GB memory for one-minute video). MoGA avoids the mask-based design, making practical at long context. Another line of work adopts dynamic token routing for sparse atten- tion. VSA (Zhang al., (2025b) first obtains compressed representations of contiguous spatiotem- poral blocks, and then selects the top-k blocks for fine-grained attention. Similarly, VMoBA (Wul 2025) extends the idea of MoBA (Lu al., 2023) to video with tailored block structures and threshold-based selection. In such methods, the block size presents a trade-off between ex- pressiveness and efficiency. Smaller blocks yield more accurate coarse -reeseegriindd attention estimates but reduce efficiency. In addition, SVG2 (Yang et al.) 2025) is training- free dynamic sparse at- tention method that performs online k-means clustering over tokens during inference and selects the top-k clusters based on their centroids. It shares similar motivation with MoGA. i.c., tokens can be grouped into semantically coherent clusters. However, online clustering in SVG2 introduces additional k-means computations during the forward pass and is not straightforward to differenti- ate through. In contrast, MoGA employs trainable cluster centroids to enable simple and efficient routing with minimal computational overhead, making it suitable for end-to-end training.
requires (nn2) memory, which is prohibitive for long
5 CONCLUSION
This paper introduces MoGA, sparse attention that replaces coarse block-level scoring with pre- cise. learned group assignments via a lightweight token router. By routing tokens into coherent groups, MoGA improves attention efficiency and fidelity for very long context. Building on MoGA, we propose the video generation model that produces minute-level, multi-shot videos at 480p and 24 fps. Diversity experiments video generation demonstrate the effectiveness of our approach.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This research is supported by National Natural Science Foundation of China under Grant 623B2094. We thank the ByteDance Seedance team and Wenfeng Lin for their support.
REFERENCES
Sand. ai, Hansi Teng, Hongyu Jia, Lei Sun, Lingzhi Li, Maolin Li, Mingqiu Tang, Shuai Han, Tian- ning Zhang, W. Q. Zhang, Weifeng Luo, Xiaoyang Kang, Yuchen Sun, Yue Cao, Yunpeng Huang, Yutong Lin, Yuxin Fang, Zewei Tao, Zheng Zhang, Zhongshu Wang. Zixun Liu, Dai Shi, Guoli Su, Hanwen Sun, Hong Pan, Jie Wang, Jiexin Sheng, Min Cui, Min Hu, Ming Yan, Shucheng Yin, Siran Zhang. Tingting Liu, Xianping Yin, Xiaoyu Yang, Xin Song, Xuan Hu, Yankai Zhang, and Yuqiao Li. Magi-1: Autoregressive video generation at scale. arXiv:2505. 132/1, 2025.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song. Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-v1 technical report. 25X2::25022.1323,, 2025.
Breakthrough and Contributors. Pyscenedetect: Video scene cut detection tool, 2014.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. In NeurlPS, 2020.
Shengqu Cai, Ceyuan Yang. Lvmin Zhang, Yuwei Guo, Junfei Xiao, Ziyan Yang. Yinghao Xu, Zhenheng Yang, Alan Yuille, Leonidas Guibas, et al. Mixture of contexts for long video genera- tion. arXiv:2508.2 1058, 2025.
Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitz- mann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. In NeurIPS, 2024.
Karan Dalal, Daniel Koceja, Jiarui Xu, Yue Zhao, Shihao Han, Ka Chun Cheung, Jan Kautz, Yejin Choi, Yu Sun, and Xiaolong Wang. One-minute video generation with test-time training. In CVPR, 2025.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv:2307.08861, 2023.
DeepSeek-AL Deepseck-v3.2-exg Boosting long- cong with deepseek sparse attention, 2025.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In /CML, 2024.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. JMLR, 2022.
Yu Gao, Haoyuan Guo, Tuyen Hoang, Weilin Huang, Lu Jiang, Fangyuan Kong, Huixia Li, Jiashi Li, Liang Li, Xiaojie Li, et al. Seedance 1.0: Exploring the boundaries of video generation models. wrxiv:2506.09113. 2025.
Suyu Gc, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for 1ms. arXiv:2310.0180,, 2023.
Albert Gu and Tri Dao. Mamba: COLM, 2024.
Linear-time sequence modeling with selective state spaces. In
Yuchao Gu, Weijia Mao, and Mike Zheng Shou. Long-conter autoregressive video modeling with next-frame prediction. arXiv:2503.1932 19325, 2025.
Yuwei Guo, Ceyuan Yang, Ziyan Yang, Zhibei Ma. Zhijie Lin, Zhenheng Yang, Dahua Lin, and Lu Jiang. Long context tuning for video generation. arXiv:2503. 10589, 2025.

---

Published as a conference paper at ICLR 2026
Roberto Henschel, Levon Khachatryan, Hayk Poghosyan, Daniil Hayrapetyan, Vahram Tadevosyan, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Streamingt2v: Consistent, dynamic, and extendable long video generation from text. In CVPR, 2025.
Jiancheng Huang, Gengwei Zhang, Zequn Jie, Siyu Jiao, Yinlong Qian, Ling Chen, Yunchao Wei, and Lin Ma. M4v: Multi-modal mamba for text-to-video generation. arXiv:2506. 10915, 2025a.
Kaiyi Huang, Yukun Huang, Xintao Wang. Zinan Lin, Xuefei Ning, Pengfei Wan, Di Zhang, Yu Wang, and Xihui Liu. Filmaster: Bridging cinematic principles and generative ai for auto- mated film generation. arXiv:2506. 18899, 2025b.
Lianghua Huang. Wei Wang, Zhi-Fan Wu, Yupeng Shi, Huanzhang Dou, Chen Liang, Yutong Feng, Yu Liu, and Jingren Zhou. In-context lora for diffusion transformers. arXiv:2410.23375, 2024a.
Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. Self forcing: Bridging the train-test gap autoregressive video diffusion. In NeurIPS, 2025c.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In CVPR, 2024b.
Robert A Jacobs, Michael Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural Computation, 1991.
Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. arXiv:2309. 14509, 2023.
Jiaxiu Jiang, Wenbo Li. Jingjing Ren, Yuping Qiu, Yong Guo, Xiaogang Xu, Han Wu, and Wang- meng Zuo. Lovic: Efficient long video generation with context compression. 25di::2007.19522 2025.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv:2001.0e. 000611 2020.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. 4rXiv:2412.03603, 2024.
Xingyang Li, Muyang Li, Tianle Cai, Haocheng Xi, Shuo Yang, Yujun Lin, Lvmin Zhang, Songlin Yang. Jinbo Hu, Kelly Peng, et al. Radial attention: O(n log n) sparse attention with energy decay for long video generation. arXiv:2506. 19852, 2025.
Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, et al. Moba: Mixture of block attention for long-context llms. arXiv:2502. 13189, 2025.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniee, Vasil Khalidov, Pierre Fernande7, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv:22304.00 07/3,, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In CVPR, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Nikhila Ravi, Valentin Gabeur, Yuan- Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv:2408.003 14, 2024.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. NeurlPS, 2022a.
12

---

Published as a conference paper at ICLR 2026
Christoph Schuhmann, Romain Veneu, Romain Beaumont, Radu Kaczmarczyk, Charlie Mullis, Ashish Katta, Jenia Jitsev Coombes, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models. In NeurlPS, 2022b.
Team Seawead, Ceyuan Yang, Zhijie Lin, Yang Zhao, Shanchuan Lin, Zhibei Ma, Haoyuan Guo, Hao Chen, Lu Qi, Sen Wang, et al. Seaweed-7b: Cost-effective training of video generation foundation model. arXiv:2504.08633 2025.
Gemini Team, Rohan Anil, Sebastian Borgeaud, rean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv:2312.1 1805, 2023.
Ye Tian, Ling Yang. Haotian Yang, Yuan Gao, Yufan Deng, Xintao Wang, Zhaochen Yu, Xin Tao, Pengfei Wan, Di ZHANG, et al. Videotetris: Towards compositional text-to-video generation. In NeurlPS, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang. et al. Wan: Open and advanced large-scale video generative models. arXiv:2503.203111 2025.
Bo Wang, Haoyang Huang. Zhiying Lu, Fengyuan Liu, Guoqing Ma, Jianlong Yuan, Yuan Zhang, Nan Duan, and Daxin Jiang. Storyanchors: Generating consistent multi-scene story frames for long-form narratives. arXiv:2505.08350 2025a.
Jiahao Wang, Hualian Sheng, Sijia Cai, Weizhan Zhang, Caixia Yan, Yachuang Feng, Bing Deng, and Jieping Ye. Echoshot: Multi-shot portrait video generation. arXiv:2506.15838 2025b.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. 1xXiv:2206.066822 2022.
Jianzong Wu, Liang Hou, Haotian Yang. Xin Tao, Ye Tian, Pengfei Wan, Di Zhang, and Yunhai Tong. Vmoba: Mixture-of-blowt attention for video diffusion models. arXiv:2506.2388 2025.
Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang. Dacheng Li, et al. Sparse videogen: Accelerating video diffusion transformers with seatial-temporal sparsity. arXiv:2502.01776. 2025.
Junfei Xiao, Ceyuan Yang, Lvmin Zhang, Shengqu Cai, Yang Zhao, Yuwei Guo, Gordon Wetzstein, Maneesh Agrawala, Alan Yuille, and Lu Jiang. Captain cinema: Towards short movie generation. (80.: 2025.
Shuo Yang, Haocheng Xi, Yilong Zhao, Muyang Li, Jintao Zhang, Han Cai, Yujun Lin, Xiuyu Li, Chenfeng Xu, Kelly Peng, al. Sparse videogen2: Accelerate video generation with sparse attention via semantic-aware permutation. arXiv:2505.1887. 2025.
Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang, Xiaodong Wang, Minheng Ni, Zhengyuan Yang. Linjie Li, Shuguang Liu, Fan Yang, et al. Nuwa-xl: Diffusion over diffusion for extremely long video generation. arXiv:2303. 12346, 2023.
Tianwei Yin, Qiang Zhang, Richard Zhang, William T Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional fast autoregressive video diffusion models. In CVPR, 2025.
Jiwen Yu, Jianhong Bai, Yiran Qin, Quande Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Xi- hui Liu. Context as memory: Scene-consistent interactive long video generation with memory retrieval. arXiv:2506.03141 2025.
Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. 10.006900, 2024.
13

---

Published as a conference paper at ICLR 2026
Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang. Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware- aligned and natively trainable sparse attention. Stt1::2502.11889, 2025.
Zhihang Yuan, Hanling Zhang, Lu Pu, Xuefei Ning, Linfeng Zhang, Tianchen Zhao, Shengen Yan, Guohao Dai, and Yu Wang. Ditfastattn: Attention compression for diffusion transformer models. In NeurlPS, 2024.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. In NeurIPS, 2020.
Lvmin Zhang and Maneesh Agrawala. Packing input frame context next-frame prediction models for video generation. vrXiv:2504.126266 2025.
Peiyuan Zhang, Yongqi Chen, Runlong Su, Hangliang Ding, Ion Stoica, Zhengzhong Liu, and Hao Zhang. Fast video generation with sliding tile attention. arXiv:2502.04507, 2025a.
Peiyuan Zhang, Haofeng Huang, Yongqi Chen, Will Lin, Zhengzhong Liu, Ion Stoica, Eric P Xing, and Hao Zhang. Faster video diffusion with trainable sparse attention. arXiv:2505. 13389, 2025b.
Yifu Zhang, Hao Yang, Yuqi Zhang. Yifei Hu, Fengda Zhu, Chuang Lin, Xiaofeng Mei, Yi Jiang, Zehuan Yuan, and Bingyue Peng. Waver: Wave your way to lifelike video generation. arXiv:2508. 15761, 2025c.
Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all. arXiv:2412.20404 2024.
Wentao Zhu, Yufang Huang, Xiufeng Xie, Wenxian Liu, Jincan Deng, Debing Zhang, Zhangyang Wang, and Liu, Autoshot: short video dataset and state-of-the-art shot boundary detection. In CVPR, 2023.
Shaobin Zhuang, Kunchang Li, Xinyuan Chen, Yaohui Wang, Ziwei Liu, Yu Qiao, and Yali Wang. Vlogger: Make your dream vlog. In CVPR, 2024.
14

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1
ONE-MINUTE VIDEO OF 1,441 FRAMES
shot,
shotz
1se
30
60
90
120
150
180
210m
240gg
270
300
p30
360
390
54X
480
510
540gB
570M
600
5560
690 ch.
720
7,5
780
810
840
870
900
930
960 L
999
1020
U5000
M0
m110ph
40,
11
00
1230
290
0
1 1550p
m80
410
shotzz
Figure 9; One-minute video generated by MoGA.
ultra-long video exceeding one minute, using the MMDiT-based MoGA model (M 20). MoGA maintains strong long-range contextual consistency. The 1st and 22nd shots remain highly coherent, and fine details such as the woman's 's hairpin and earrings are preserved across shots. Moreover, even with multiple faces appearing across different shots, the model avoids identity confusion.
As shown in Fig. 9 present the generated results of MoGA
A.2 EMERGENCE OF BACKGROUND CONSISTENCY
As shown in Fig. 10 demonstrate MoGA's ability to maintain background consistency. After training on long, multi-shot videos, MoGA exhibits emergent, implicit control over consistency in both the environment and the characters. Even without explicit specification of details (e.g., the
15

---

Published as a conference paper at ICLR 2026
cabinet shape and the position of intravenous drip bottle), different shots automatically maintain coherent, temporally consistent depictions.
Shot
90A
150
210
240
270
Shots
m
390
120
-50,p
510
Shot
540m
690
Figure 10: Emergence of background consistency.
A.3 MULTI-STYLE VIDEO GENERATION
Fig. | illustrates MoGA's multi-style generation capability. MoGA not only performs strongly in realistic spaces, but also excels in stylized domains such as animation, illustration, and cinematic aesthetics. It can produce high-quality long 2D videos and long 3D videos while maintaining tempo- ral coherence, identity consistency, and scene continuity across diverse styles and camera motions.
A.4 DETAILS OF THE COMPUTATIONAL COMPLEXITY
As shown in Tab. g reports the computational cost under varying numbers of groups (M) and video duration. As the generation video duration increases, the computational complexity of STGA exhibits approximately linear growth and the computational complexity of MoGA is approximately 1 1/M of that of Full Attention.
A.5 ANALYSIS OF GROUP BALANCING Loss
As shown in Fig. 12 we validate the effectiveness the group balancing loss, which measures the balance of the router's token-to-group assignments. A higher value indicates that tokens concentrate
16

---

Published as a conference paper at ICLR 2026
Shot
Shotg
210
160g
2000
.150
199
Shot
600
6300
Shot
150g
180
210
Shot
300
338
360
290
520
Figure 11: Multi-style video generation of MoGA.
Video Duration
5s
10s
15s
20s
30s
Frames
77
157
237
317
477
Sequence length
31200
62400
93600
124800
187200
Full Attention
0.28
0.88
1.85
3.19
6.94
MoGA (M=5)
0.093
0.34
0.67
1.09
2.26
PFLOPs
MoGA (M=10)
0.065
0.25
0.48
0.78
1.56
MoGA (M=20)
0.051
0.21
0.39
0.61
1.22
Table 8: Compute (PFLOPs) versus group number and video duration on Wan2.1-1.33
a few groups, whereas lower value indicates more balanced grouping. When we include this loss during training, the metric rapidly converges to around I, reflecting globally balanced assignments. In contrast, without it, the metric increases as the router funnels tokens into a few groups to gain short-term advantages in the diffusion MSE loss. Because our goal is to separate weakly related tokens and maintain balanced grouping, the additional group balance loss is necessary to enforce the desired assignments.
17

---

Published as a conference paper at ICLR 2026
18
wil group Ialance loss
with group balance loss
12
9
30
60
90
120
150
179
Step
Figure 12: Group balancing loss curves of MoGA.
A.6 DETAILED PYTORCH PSEUDO-CODE
def moga permute (q, k, v, router_logtte T=1]: weights (router logits/T) soottmaaaaaa weights, groups torch. (weights, k=1, dim=-1) weights, groups - weights. reshapee groups.reshapet-
sort_idx torch. (groups, stable-True) groups_sorted, weights_sorted
[rroup[[sor_ioon
veights
counts torel..iiqqe_consseeuiie. groups_sorted, return_ccunts-1in
cu_seqlens torch.cat wor.h.cattcccunnst....... [[countss.eww_eercll (1),
(0)
k, glsort_idk), k[sort_idx), v /|sot_idd
return q, k, v, sort_idx, weights_sorted, eu_seqlens
def moga_repent noga_repermute (out, fort_idx weights_sorted)
out cut weights_sorted.. Nene, None]
out_ torch. zeros_iik (out)
out index add (0, sort_idt idx, out)
return out_
def Axtturc_of.grrops_aattttttt ''1
k, v, rogitt_
g, k, v: (L, H, router_logits: (I, M)
g, k, v, sort_idx, weights_scrted cu_seqlens moga_permute
5, k, v, router_Logits)
max seqien ou_seeginssmmax (item() out flash_attm aaraveentt_ out moga_repermute (out, sort_idx, weights_sorted) return out
varien ftn k, cu seglen, max sex_se
provide a minimal implementation of MoGA based on PyTorch.
A.7 EFFECT OF THE ROUTER TEMPERATURE
As shown in Fig. 13 and the quantitative analysis in Tab. we adjust the router temperature hy- perparameter and find that cross-shot similarity does not exhibit significant change, and there is obvious impact on the router's 's stability.
A.8 EFFECT OF THE GROUP BALANCING Loss WEIGHT
As shown in Fig. 14 and Tab. 10 when we increase the balancing loss weight a to 1.0, the loss does not exhibit significant fluctuations, but the cross-shot consistency slightly decreases.
18

---

Published as a conference paper at ICLR 2026
Tempercture=03. Tenperoture=10
Figure 13: Visualization results under different temperatures.
Temperature Cross-shot DINO
Cross-shot CLIP
0.7284
0.8970
0.5
0.7286
0,8946
Table 9: Results under different temperatures.
Balance loss weight=0.1
Low consistency
Bolance loss weight:1.0
Figure 14: Visualization results under different balance loss weights.
Cross-Shot DINO
Cross-Shot CLIP
0.1
0.7284
0.8970
1.0
0.7157
0.8851
Table 10: Results under different balance loss weights.
A.9 VISUALIZATIONS OF TOKEN ROUTING
We select a specific group from a router (the first group in the 14th DiT block) to examine the relationship between token assignment and the corresponding visual patches. We visualize the visual patches corresponding to this group of tokens.
we demonstrate that the router develop consistent semantic specializations for the visual concept without explicit supervision. For instance, this selection group consistently capture face-related tokens across diverse video samples.
As shown in Fig. 15
A.10 QUANTITATIVE ANALYSIS OF TOKEN ROUTING
We develop quantitative analysis tool measure whether the router assigns semantically related tokens to the same group. We treat the router's 's grouping tokens form of unsupervised segmen- tation. We then use SAM2 (Ravi et al. 2024 to obtain foreground masks for each frame as ground truth (GT), and treat the tokens aggregated by each group as predictions. For each GT mask, we match the group whose prediction mask achieves the highest loU, and use the loU as the metric.
19

---

Published as a conference paper at ICLR 2026
Shot
Shot 2
Shot
Token allccation of Group 1 Layer 14
Token allecation of Group Loyer 14
Figure 15: More visualization results of token routing.
Duration
Seq Len
Full Attention
MoGA (M-5)
Speedup (M=5)
MoGA (M=20)
Speedup (M=20)
31,200
40.06 ms
27.43 ms
1.46x
21.58 ms
1.86x
s
62,400
158.83 ms
81.15 ms
1.96x
62.24 ms
2.55x
20
124,800
650.08 ms
231.81 ms
2.80x
151.16 ms
4.30
s0
187,200
1,423.52 ms
455.57 ms
3.12x
267.24 ms
5.33x
Table
Latency comparison between Full Attention and MoGA.
We use the average IoU of different prompts as the final result. For evaluation, we use 9 scripts containing 27 prompts to generate 10-second videos. The IoU reflects the router's ability to assign tokens of the same category single group.
Train vs. Random. As shown in Tab. [5] we compare three routing methods: random assignment, randomly initialized router, and the trained router. After training, specific groups achieve substan- tially higher loU (28.6%) than both the random baseline (15.6%) and the randomly initialized router (18.5%).
Different Time Step. As shown in Tab. 6 we evaluate the loU of the router at different sampling steps during inference. The router assigns tokens suboptimally (low IoU, 17%) at the initial denois- ing step (I=999). As denoising progresses, the IoU quickly rises above 25% (t=931) and remains stable, gradually increasing 28.6%, which indicates that MoGA is robust in the semantics of token assignments throughout the denoising process.
Different Blocks. As shown in Tab. 7 evaluate the IoU of the router different DiT blocks during inference. The middle DiT blocks exhibit relatively stronger semantic grouping ability (IoU up to 31.0%), which is consistent with prior work indicating that intermediate layers of DiT can capture high-level semantic features Yu et al. 2024
A.11 RUNTIME AND MEMORY USAGE
We compare MoGA (with STGA) and full attention (FlashAttention)) in operator runtime, end-to- end training/inference time (sequence parallel=8; duration=30 s), and memory usage. All tests base on the same hardware, Ascend 910B NPU (TFLOPs roughly comparable to NVIDIA A100; 64 GB memory).
Operator Comparisons (Latency). As shown in Tab. || MoGA consistently outperforms Full Attention across different sequence lengths, with larger gains at longer durations. Speedup grows with sequence length: from 1.46x-1.86x at 5 to 3.12x-5.33x 30 s.
End-to-End Comparisons. For 30s training dataset and Wan2.1-14B, the wall-clock times are measured in Tab. 12] MoGA significantly reduces both training and inference time versus full attention, with gains at M=5 (1.72x train, 2.10x inference) and M=20 (2.24x train, 2.43x inference).
A.12 USER STUDY
We conduct a user study. Each user rates the generated videos (1-10), with rating dimensions includ- ing three aspects: video quality, consistency, and prompt following. The study covers 10s and 30s
20

---

Published as a conference paper at ICLR 2026
Full Attention
MoGA (M=5)
MoGA (M=20)
Training time (per iter)
66.87 s
38.76 s
29.84 s
Speedup vs. Full
1.72x
2.24x
Training RAM
48.1 1 GB
49.8 GB
49.6 GB
Inference time (per step)
40.21
19.06 s
16.55 s
Speedup vs. Full
2.10x
2.43
Inference RAM
33.4 GB
38.6 GB
38.6 GB
Table 12: Training and inference efficiency comparison.
Metric (1 better)
EchoShot
IC-LoRA+
MoGA
Prompt Following
6.97
5.58
8.47
Video Quality
7.11
4.53
8.05
Consistency
6.37
5.03
8.26
Table 13: Comparison across metrics.
multi-shot scripts and conducts blind comparison among MoGA, EchoShot, and IC-LoRA+Wan. As shown in Tab. [3] MoGA clearly outperforms the baselines across all metrics, achieving the high- est scores in Prompt Following (8.47), Video Quality (8.05), and Consistency (8.26). Compared with EchoShot, MoGA shows notable gains (+1.50, +0,94, +1.89 respectively), and decisively surpasses IC-LoRA+ Wan with even larger margins (+2.89 +3.73).
A.13 THE DETAILS DATA PIPELINE
We first collect publicly available long-form videos, including movies, TV series, animations, and short dramas. We then filter out videos with resolution below 720p, duration shorter than 1 minute, and videos whose VQA scores (sparse sampling) falls below the thresholds (e.g., the aesthetic threshold (Schuhmann et al., 2022a) ranges from 4 4.5 depending on the source). Next, we employ AutoShot and PySceneDeted jointly to segment the raw videos, obtaining single-shot clips with clean boundary transitions. After that, we perform denser frame sampling for each clip and compute per-clip VQA scores well as OCR results. Based on OCR, crop each clip while preserving the original aspect ratio to remove watermarks and subtitles. Clips whose cropped area less than 40% of the original whose VQA scores are below the thresholds are discarded. Fi- nally, we concatenate clips from the same source video in chronological order and use an MLLM to caption each clip, assembling multi-shot long video training samples. We thus obtain approximately 200k training samples.
A.14 LLM USAGE
Large Language Models (LLMs) were employed exclusively to assist with manuscript preparation. Their role was limited to refining language, improving readability, and enhancing the clarity of exposition. Specifically, the LLM contributed to tasks such as rephrasing sentences, correcting grammar, and smoothing the overall flow of the text.
The LLM played no part in formulating research questions, designing methodologies, conducting experiments, analyzing results. All scientific ideas, experimental designs, and analyses were conceived and executed entirely by the authors. The authors accept full responsibility for the manuscript's content, including any text edited with LLM assistance. Care was taken to ensure that LLM-derived text complies with ethical standards and does not introduce plagiarism or scientific misconduct.
21


Reviews:
[{"review_id": "aIm3KU3G9R", "reviewer": "Reviewer_Hbw2", "summary": "This paper proposes Mixture-of-Groups Attention (MoGA), a new sparse attention mechanism designed to improve computational efficiency and scalability for end-to-end long video generation with diffusion transformers.  MoGA employs a lightweight learnable token router that dynamically assigns tokens into semantically coherent groups. Using MoGA, the authors present a video generation model that can produce minute-level, multi-shot, 480p videos at 24 FPS with a context length up to 580 K tokens. Experiments across multiple baselines (Wan2.1, MMDiT, EchoShot, IC-LoRA) demonstrate improved visual quality, subject and background consistency, and substantial reductions in FLOPs.", "questions": "1. How is Mixture-of-Groups Attention fundamentally different from Mixture-of-Experts (MoE) or Mixture-of-Block Attention (MoBA/VMoBA) beyond operating tokens? Can you clarify what new insights or mechanisms MoGA introduces that are not already explored in MoBA (Lu et al., 2025) or Radial Attention (Li et al., 2025)?\n\n2.What ensures that the lightweight router produces semantically meaningful groupings instead of arbitrary clusters? Have you analyzed how stable the routing assignments are across training iterations or input perturbations? Maybe could provide quantitative evidence (e.g., token entropy or similarity distributions) to support that MoGA learns meaningful semantic partitioning?\n\n3. Have you tested MoGA on other architectures or datasets (e.g., Open-Sora, Pika, or VideoCrafter) to show generality beyond Wan/MMDiT? Do you expect MoGA to benefit smaller or lower-resolution models similarly, or is the gain limited to large-scale settings? It claims 1.7× speedup and reduced FLOPs; can you provide absolute runtime and memory usage numbers compared to full attention on the same hardware?\n\n4. Please explain on reproducibility, e.g. release of the code, or ease of reimplementation. How can the efficiency be verified without access to the internal dataset and pipeline.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The introduction of token-level routing as a “mixture-of-groups” attention mechanism is novel and intuitive. It replaces coarse block sparsity with a more fine-grained, data-driven grouping that potentially generalizes better across tasks.\n- The paper convincingly shows MoGA’s ability to reduce attention complexity from O(N^2) to approximately O(N^2/M) while preserving quality. The claimed 1.7× training/inference speedup .\n- The design is compatible with FlashAttention, sequence parallelism, and existing DiT frameworks.\n- Across metrics such as subject/background consistency, aesthetic quality, and cross-shot CLIP/DINO similarity, MoGA consistently outperforms baselines, including full-attention models.", "weaknesses": "### Major\n- My major concern is the novelty compared to the previous works. While MoGA's token router is inspired my MoE, and MoBA, it should be deeply analyzed the distinction between it and these works. The paper should clarify how fundamentally MoGA differs from the existing routing based methods, beyond the token level analysis.\n\n- The benchmarks are built on top of the models like wan2.1/MMDiT and tested on internal datasets, where the authors integrate their MoGA module into the original architectures and train/fine-tune them on their internal dataset. Therefore, it is uncertain whether MoGA’s advantages generalize to other diffusion transformer backbones or unseen datasets.  While the paper provides an anonymous demo link, it lacks details about code or data release, which is critical for an academic paper submission claiming major efficiency improvements, also reproducibility.\n\n### Others\n- It could be better if the paper could provide formal analysis of stability, or the distribution of token assignments. \"lightweight router\" would be thoretically explained.\n\n- It develops a multi-stage pipeline, which may be overly complicated (with steps like AutoShot, OCR, and LLM captioning),  should provide a experimental study on how much these steps actually improve performance, or a simpler baseline with a simplified pipeline. It would make the contribution clearer and more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WsvqDTK497", "reviewer": "Reviewer_o1S4", "summary": "This paper introduce Mixture-of-Groups Attention (MoGA), an efficient sparse attention that uses a lightweight learnable token router to precisely match tokens without blockwise estimation. By semantics-aware routing, MoGA enables effective long-range interactions.", "questions": "If the author's response is satisfactory to me, I would be willing to maintain a positive score.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "see Summary", "weaknesses": "The authors should provide between 20 to 50 video samples to better demonstrate the capabilities and limitations of their method. Additionally, for each prompt, it would be beneficial to include comparisons with other state-of-the-art methods. Ideally, there should be 3 to 5 comparison methods with corresponding videos for each prompt.\n\nThe authors should provide detailed results of the user study, including statistical analysis and user feedback. This will help in understanding how the proposed method performs in terms of user satisfaction and practical usability.\n\nThe paper only presents sub-scores from VBench, which provides a limited view of the overall performance.\n\nThe caption in Figure 3 should be more detailed. Otherwise, it is difficult to guess the meaning of the figure.\n\nWhy are there only experimental results for 1.3B in Table 2, but not for 14B? Similarly, Table 1 and Table 3 also lack these results.\n\nThe idea is very simple and clear, and it is easy to understand. However, the main difficulty and contribution may lie in the engineering implementation of the code. Since the author has not provided open-source code or code in the supplementary materials, I strongly suggest that the author provide the real Python code functions for Algorithm 1 for me to review.\n\nFigure 6 only compares the FLOPs, but I believe that the actual inference time is much more important. This conceals two issues that I am very interested in:\n1. In fact, the computational cost of Spatial-Temporal Group Attn is already quite large, and that of Mixture-of-Groups Attn is also significant. I think that under the scenario of short videos, the impact of these two designs on speed is minimal.\n2. From an engineering implementation perspective, Spatial-Temporal Group Attn and Mixture-of-Groups Attn cannot be parallelized. Therefore, they will actually be slower due to serial computation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x7j2mOxR7D", "reviewer": "Reviewer_j5yM", "summary": "This paper introduces Mixture-of-Groups Attention (MoGA), a novel and efficient sparse attention mechanism designed to overcome the computational bottleneck of generating long, high-resolution videos with diffusion transformers. Instead of relying on traditional block-wise estimation for sparsity, MoGA employs a lightweight, learnable \"token router,\" inspired by Mixture-of-Experts, to assign each token to a specific group based on its semantics. Standard self-attention is then performed independently within each group, drastically reducing computational complexity while enabling effective long-range interactions. This global attention mechanism is complemented by a local Spatial-Temporal Group Attention (STGA) to ensure local continuity, and the model is trained on a custom-built data pipeline that produces minute-long, multi-shot video samples.", "questions": "The token router is shown to learn meaningful semantic groupings in an unsupervised manner. Could the authors elaborate on the nature of these learned groups? For instance, do certain groups consistently specialize in specific visual concepts (e.g., one group for faces, another for backgrounds, another for dynamic motion), or are the groupings more abstract and context-dependent? Understanding this could provide deeper insights into the model's internal workings", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper's primary strength lies in its elegant and highly effective solution to the long-context problem. By replacing coarse block-level scoring with a precise, end-to-end token router, MoGA represents a conceptual advance over prior sparse attention methods. The approach is remarkably practical, as it is kernel-free and seamlessly integrates with existing high-performance technologies like FlashAttention and sequence parallelism. The experimental results are state-of-the-art and convincingly demonstrate the model's ability to generate coherent, minute-long, multi-shot videos, supported by comprehensive quantitative metrics and strong ablation studies that validate the design choices.", "weaknesses": "One potential point of discussion is the method's reliance on a powerful, pre-existing base model (Wan2.1) for fine-tuning, which makes it slightly difficult to isolate the gains of MoGA from the inherent capabilities of the foundation model. Additionally, the paper introduces an impressive and complex data pipeline for creating multi-shot training samples; the importance of this high-quality, specialized data to the final result is significant and could be considered a major contribution in its own right, perhaps underemphasized in the context of the attention mechanism.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ScEU13z7TS", "reviewer": "Reviewer_ANEs", "summary": "This paper introduces Mixture-of-Groups Attention (MoGA), a sparse attention mechanism designed to alleviate the quadratic complexity of Diffusion Transformers in long video generation. MoGA employs a lightweight, learnable token router to assign tokens into semantically coherent groups and performs full attention within each group, achieving fine-grained dynamic sparsification. Coupled with local spatiotemporal window attention (STGA) and shot-level textual conditioning, the proposed framework can end-to-end generate 480p, 24 fps, minute-long multi-shot videos. The authors also describe a multi-stage data curation pipeline and evaluate MoGA across single-shot, multi-shot, and ultra-long video scenarios.", "questions": "* Effectiveness of router: If the router only classify the tokens based on itself, would it be less sensitvie to global related information? Is there any detailed visusalization about router?\n* Router stability: Did you observe router collapse or gradient instability early in training? Is the group balancing coefficient α = 0.1 robust across model sizes? How do performance and sparsity change if α varies?\n* Choosing the number of groups: Since too few or too many groups hurt consistency, have you explored dynamically adjusting group counts (e.g., by sequence length or diffusion timestep)? Could this enhance performance or stability?\n* Multimodal control evaluation: Can you provide quantitative metrics (e.g., text-to-video CLIP scores, prompt coverage, human preference studies) to support the claimed shot-level textual orchestration?\n* Resource requirements: Please detail the hardware setup (GPU type/count), training time, and memory footprint for MoGA at M = 20 on minute-long videos so the community can gauge reproducibility.\n* Broader applicability: Have you experimented with MoGA on non-video long-sequence tasks (e.g., text generation, 3D scene modeling)? If not, what considerations would be necessary for such extensions?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Originality: The MoGA design brings a Mixture-of-Experts–style router into sparse attention for video generation, avoiding coarse block-level heuristics that impede prior methods. Its compatibility with FlashAttention and sequence parallelism underscores solid systems innovation.\n* Quality: Experiments cover single-shot short videos, multi-shot videos, and long-form generation, measured by VBench metrics (subject/background consistency, motion smoothness, aesthetics, etc.) plus Cross-Shot CLIP/DINO. MoGA outperforms both sparse and full-attention baselines, and the ablations (group count, STGA interplay, balancing loss) lend credibility.\n* Clarity: The manuscript is well-structured, with clear figures contrasting full, block-sparse, and MoGA attention. Pseudocode and complexity analysis clarify implementation.\n* Significance: Delivering stable 580k-token, minute-long, multi-shot videos is a notable leap for open-source systems, and MoGA’s design could generalize to other long-sequence tasks.", "weaknesses": "* Router design details: While a group balancing loss is introduced, the paper does not quantify how router collapse affects performance or explore router hyperparameters (dimension, temperature, regularization) beyond group count.\n* Compute/resource reporting: PFLOP estimates and speedup claims are provided, yet concrete wall-clock measurements (training/inference time, memory usage) on specific hardware are missing, hindering reproducibility assessments.\n* Data pipeline reproducibility: The multi-stage filtering/captioning pipeline depends on various proprietary tools (VQA/OCR/AutoShot), but code or parameter details are sparse. More statistics on filtered dataset size/quality would help.\n* Limited multimodal control evaluation: Although shot-level textual conditioning is highlighted, there is little quantitative evidence (e.g., prompt adherence metrics or user studies) demonstrating textual control effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)