PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and
  Quantized Attention in Visual Generation Models
Authors: name: Tianchen Zhao, name: Ke Hong, name: Xinhao Yang, name: Xuefeng Xiao, name: Huixia Li, name: Feng Ling, name: Ruiqi Xie, name: Siqi Chen, name: Hongyu Zhu, name: Yichong Zhang, name: Yu Wang

In visual generation, the quadratic complexity of attention mechanisms
results in high memory and computational costs, especially for longer token
sequences required in high-resolution image or multi-frame video generation. To
address this, prior research has explored techniques such as sparsification and
quantization. However, these techniques face significant challenges under low
density and reduced bitwidths. Through systematic analysis, we identify that
the core difficulty stems from the dispersed and irregular characteristics of
visual attention patterns. Therefore, instead of introducing specialized
sparsification and quantization design to accommodate such patterns, we propose
an alternative strategy: *reorganizing* the attention pattern to alleviate the
challenges. Inspired by the local aggregation nature of visual feature
extraction, we design a novel **Pattern-Aware token ReOrdering (PARO)**
technique, which unifies the diverse attention patterns into a
hardware-friendly block-wise pattern. This unification substantially simplifies
and enhances both sparsification and quantization. We evaluate the
performance-efficiency trade-offs of various design choices and finalize a
methodology tailored for the unified pattern. Our approach, **PAROAttention**,
achieves video and image generation with lossless metrics, and nearly identical
results from full-precision (FP) baselines, while operating at notably lower
density (~20%-30%) and bitwidth (**INT8/INT4**), achieving a **1.9x** to
**2.7x** end-to-end latency speedup.