SageAttention2++: A More Efficient Implementation of SageAttention2
Authors: name: Jintao Zhang, name: Xiaoming Xu, name: Jia Wei, name: Haofeng Huang, name: Pengle Zhang, name: Chendong Xiang, name: Jun Zhu, name: Jianfei Chen

The efficiency of attention is critical because its time complexity grows
quadratically with sequence length. SageAttention2 addresses this by utilizing
quantization to accelerate matrix multiplications (Matmul) in attention. To
further accelerate SageAttention2, we propose to utilize the faster instruction
of FP8 Matmul accumulated in FP16. The instruction is 2x faster than the FP8
Matmul used in SageAttention2. Our experiments show that SageAttention2++
achieves a 3.9x speedup over FlashAttention while maintaining the same
attention accuracy as SageAttention2. This means SageAttention2++ effectively
accelerates various models, including those for language, image, and video
generation, with negligible end-to-end metrics loss. The code will be available
at https://github.com/thu-ml/SageAttention.