Title: MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generation
Primary area: generative models
Keywords: video generation

Abstract: Long video generation with diffusion transformer is bottlenecked by the quadratic scaling of full attention with sequence length. Since attention is highly redundant, outputs are dominated by a small subset of query–key pairs. Existing sparse methods rely on block-wise coarse estimation, whose accuracy–efficiency trade-offs are constrained by block size. This paper introduce Mixture-of-Groups Attention (MoGA), an efficient sparse attention that uses a lightweight learnable token router to precisely match tokens without blockwise estimation. By semantics-aware routing, MoGA enables effective long-range interactions. As a kernel-free method, it integrates seamlessly with modern attention stacks, including FlashAttention and sequence parallelism. Built on MoGA, we develop an efficient long video generation model that end-to-end produces minute-level, multi-shot, 480p videos at 24 FPS with approximately 580K context length. Comprehensive experiments on various video generation tasks validate the effectiveness of our approach. We provide an anonymous link \url{https://anonymous.4open.science/r/MoGA} to showcase the generated videos.