MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generation
Authors: name: Weinan Jia, name: Yuning Lu, name: Mengqi Huang, name: Hualiang Wang, name: Binyuan Huang, name: Nan Chen, name: Mu Liu, name: Jidong Jiang, name: Zhendong Mao
Organization: ByteDance

Long video generation with Diffusion Transformers (DiTs) is bottlenecked by
the quadratic scaling of full attention with sequence length. Since attention
is highly redundant, outputs are dominated by a small subset of query-key
pairs. Existing sparse methods rely on blockwise coarse estimation, whose
accuracy-efficiency trade-offs are constrained by block size. This paper
introduces Mixture-of-Groups Attention (MoGA), an efficient sparse attention
that uses a lightweight, learnable token router to precisely match tokens
without blockwise estimation. Through semantic-aware routing, MoGA enables
effective long-range interactions. As a kernel-free method, MoGA integrates
seamlessly with modern attention stacks, including FlashAttention and sequence
parallelism. Building on MoGA, we develop an efficient long video generation
model that end-to-end produces minute-level, multi-shot, 480p videos at 24 fps,
with a context length of approximately 580k. Comprehensive experiments on
various video generation tasks validate the effectiveness of our approach.