Title: Compact Attention: Exploiting Structured Spatio-Temporal Sparsity for Fast Video Generation
Primary area: generative models
Keywords: Sparse Attention, Diffusion Models

Abstract: The quadratic computational complexity of self-attention mechanisms pose a critical challenge for transformer-based video generation in synthesizing ultra-long sequences. 
 Current sparse approaches with fixed patterns fail to fully exploit the inherent spatio-temporal redundancies in video data. Through systematic analysis of video diffusion transformers (DiT), we observed that attention matrices exhibit a stable vocabulary of structured, yet heterogeneous, sparse patterns, including a cross-shaped attention pattern and patterns' dynamic behavior over time. To fully exploit the redundancies, we propose Compact Attention, a hardware-aware acceleration framework featuring three innovations: 1) Adaptive tiling strategies that approximate diverse spatial interaction patterns via dynamic tile grouping, 2) Temporally varying windows that adjust sparsity levels based on frame proximity, and 3) a recall-driven, offline search algorithm that automatically optimizes sparse masks while preserving critical attention pathways. 
 Our method achieves up to 3x acceleration in attention computation on single-GPU setups while maintaining comparable visual quality with full-attention baselines. By grounding acceleration in the empirical discovery of fundamental attention structures, this work provides a principled approach to efficient long video generation through structured sparsity exploitation.

TL;DR: Compact Attention accelerates video generation by exploiting structured sparsity in transformer attention, achieves 2.3× speedup on models like Hunyuan while maintaining visual quality comparable to full attention.