MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generation
Authors: name: Weinan Jia; name: Yuning Lu; name: Mengqi Huang; name: Hualiang Wang; name: Binyuan Huang; name: Nan Chen; name: Mu Liu; name: Jidong Jiang; name: Zhendong Mao
Published: 2025-10-21 14:50:42

Abstract: Long video generation with Diffusion Transformers (DiTs) is bottlenecked by
the quadratic scaling of full attention with sequence length. Since attention
is highly redundant, outputs are dominated by a small subset of query-key
pairs. Existing sparse methods rely on blockwise coarse estimation, whose
accuracy-efficiency trade-offs are constrained by block size. This paper
introduces Mixture-of-Groups Attention (MoGA), an efficient sparse attention
that uses a lightweight, learnable token router to precisely match tokens
without blockwise estimation. Through semantic-aware routing, MoGA enables
effective long-range interactions. As a kernel-free method, MoGA integrates
seamlessly with modern attention stacks, including FlashAttention and sequence
parallelism. Building on MoGA, we develop an efficient long video generation
model that end-to-end produces minute-level, multi-shot, 480p videos at 24 fps,
with a context length of approximately 580k. Comprehensive experiments on
various video generation tasks validate the effectiveness of our approach.