Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via
  Semantic-Aware Permutation
Authors: name: Shuo Yang; name: Haocheng Xi; name: Yilong Zhao; name: Muyang Li; name: Jintao Zhang; name: Han Cai; name: Yujun Lin; name: Xiuyu Li; name: Chenfeng Xu; name: Kelly Peng; name: Jianfei Chen; name: Song Han; name: Kurt Keutzer; name: Ion Stoica
Published: 2025-05-24 21:30:29

Abstract: Diffusion Transformers (DiTs) are essential for video generation but suffer
from significant latency due to the quadratic complexity of attention. By
computing only critical tokens, sparse attention reduces computational costs
and offers a promising acceleration approach. However, we identify that
existing methods fail to approach optimal generation quality under the same
computation budget for two reasons: (1) Inaccurate critical token
identification: current methods cluster tokens based on position rather than
semantics, leading to imprecise aggregated representations. (2) Excessive
computation waste: critical tokens are scattered among non-critical ones,
leading to wasted computation on GPUs, which are optimized for processing
contiguous tokens. In this paper, we propose SVG2, a training-free framework
that maximizes identification accuracy and minimizes computation waste,
achieving a Pareto frontier trade-off between generation quality and
efficiency. The core of SVG2 is semantic-aware permutation, which clusters and
reorders tokens based on semantic similarity using k-means. This approach
ensures both a precise cluster representation, improving identification
accuracy, and a densified layout of critical tokens, enabling efficient
computation without padding. Additionally, SVG2 integrates top-p dynamic budget
control and customized kernel implementations, achieving up to 2.30x and 1.89x
speedup while maintaining a PSNR of up to 30 and 26 on HunyuanVideo and Wan
2.1, respectively.