SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training
Authors: Jintao Zhang
Jia wei
Haoxu Wang
Pengle Zhang
Xiaoming Xu
Haofeng Huang
Kai Jiang
Jianfei Chen
Jun Zhu
Venue: NeurIPS 2025 spotlight ()
Keywords: Attention
Quantization
Efficient Attention
GPU Kernel
Primary area: infrastructure
TL;DR: SageAttention3: Microscaling FP4 Attention for Plug-and-Play Inference Acceleration and An Exploration of 8-Bit Attention for Training.

Abstract: The efficiency of attention is important due to its quadratic time complexity. We enhance the efficiency of attention through two key contributions: First, we leverage the new $\texttt{FP4}$ Tensor Cores in Blackwell GPUs to accelerate attention computation. Our implementation achieves $\textbf{1038}$ $\texttt{TOPS}$ on $\texttt{RTX5090}$, which is a $\textbf{5}\times$ speedup over the fastest FlashAttention on $\texttt{RTX5090}$. Experiments show that our $\texttt{FP4}$ attention can accelerate inference of various models in a plug-and-play way. Second, we pioneer low-bit attention to training tasks. Existing low-bit attention works like FlashAttention3 and SageAttention focus only on inference. However, the efficiency of training large models is also important. To explore whether low-bit attention can be effectively applied to training tasks, we design an accurate and efficient $\texttt{8-bit}$ attention for both forward and backward propagation. Experiments indicate that $\texttt{8-bit}$ attention achieves lossless performance in fine-tuning tasks but exhibits slower convergence in pretraining tasks. The code is available at https://github.com/thu-ml/SageAttention.

SageAttentions Microsealing FP4 Attention for Inference and An Exploration of 8-bit Training
Jintao Zhang 12, Jia Wei Haoxu Wang', Pengle Zhang', Xiaoming Xu', Haofeng Huang', Kai Jiang', Jianfei Chen Jun Zhu
'Dept. of Comp. Sci. and Tech., Institute for AI, BNRist Center, THBI Lab, Tsinghua- Bosch Joint ML Center, Tsinghua University; Shengshu Tech., Beijing, China. {zhang- j t240mails. jianfeice, deszj ttszjhhasinghua.c eww.siinghaaedu...om edu.
Abstract
The efficiency of attention is important due to its quadratic time complexity. We en- hance the efficiency of attention through two key contributions: First, we leverage the new FP4 Tensor Cores Blackwell GPUs to accelerate attention computation. Our implementation achieves 1038 TOPS RTX5090, which is 5x speedup over the fastest FlashAttention on RTX5090. Experiments show that our FP4 atten- tion accelerate inference of various models in plug-and-play way. Second, we pioneer low-bit attention to training tasks. Existing low-bit attention works like FlashAAtention3 and SageAttention focus only on inference. However, the efficiency of training large models is also important. To explore whether low-bit attention be effectively applied training tasks, we design accurate and efficient g-bit attention for both forward and backward propagation. Experiments indicate that 8-bit attention achieves lossless performance fine-tuning tasks but exhibits slower convergence in pretraining tasks. The code is available at https: //gi:t com t/u-thuunni hatpaau..commmmemmaaaaaattttttee Saahummm/Sagetteettiioo
FlashAttention2 (End-to- End Time: 490s on RTX5090)
FashAttention2
SageAttention3
1038
5x
Kernel Speed (TOPS)
SageAttentiona [End-to-Enn Time: RTX5090)
FlashArtention2
SageAttention3
490s
3x
1648
Speedup
Figure 1: The upper left figure shows the kernel speedup on RTX5090. The other two figures show the end-to-end inference speedup of generating video using HunyuanVideo on RTX5090. Note that FlashAttention3 can only run Hopper GPUs, FlashAttention2 already the fastest on RTX5090.
Introduction
The efficiency of attention is critical for generation models, especially given their quadratic time complexity with longer sequences 2 Quantization offers effective way to
Motivation.
co-first authors,
Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

accelerate inference by utilizing low-bit Tensor Cores in GPUs 3 4 5 6 The new FP4 Tensor Cores in Blackwell GPUs deliver significantly faster performance compared to FP16 77]. We want to propose a novel FP4 attention implementation that provides plug-and-play compatibility for inference acceleration. Beyond inference, training efficiency equally important. However, no prior work has explored low-bit attention for training large models. To address this gap, we design trainable 8-bit attention to explore its feasibility in training tasks.
To the best of our knowledge, we are the first work that designs FP4 attention for inference and the first work to explore the feasibility of low-bit attention for training large models.
Challenges. There are two primary obstacles for FP4 attention and one key difficulty for 8-bit trainable attention. First, (C1) FP4 quantization suffers from severe value limitations (only 15 representable values), making both per-tensor and per-token quantization approaches inadequate for preserving model accuracy. Second, (C2) The attention map P consists primarily of small values in the range [0, 1]. When directly quantized to FP4, these values force the scaling factors into an extremely narrow dynamic range. However, hardware requires the quantization factors to be in FP8 data type. This leads to significant accuracy loss when presenting these scale factors in FP8. Third, (C3) When employing 8-bit attention during training, find that the attention map gradients are particularly vulnerable to quantization errors, resulting in accumulated errors in the input gradients.
FP4 microscaling quantization for the two matrix multiplications attention, i... QK and PV. By constraining the quantization group size to method effectively contains outlier effects within
Our Method. To address (CI), propose
1x16 (instead of per-tensor or per-channel), each block while improving FP4 quantization accuracy. To overcome (C2), we propose a two-level quantization method for P fully utilize the presentative range of the FP8 scaling factor, enhancing the quantization accuracy of P. Specifically, this approach first normalizes each token's 's range to [0, 448 6] through per-token quantization, then applies FP4 microscaling quantization for enhanced precision. To address (C3), we identify the most accuracy-sensitive matrix multiplication among the five in backpropagation and maintain accuracy in FP16.
Result. Our FP4 attention, named SageAttention3, could achieve 1038 TOPS on RTX5090, which is a 5x speedup than FlashAttention. Furthermore, demonstrate that 8-bit trainable attention, named SageBwd, could achieve lossless performance when fine-tuning base models for instruction- following tasks, but is not suitable for pretraining tasks.
Contribution. Our work makes the following key contributions:
(1) We design the first FP4 attention to accelerate inference, achieving 1000+ TOPS on RTX5090.
(2) We propose the first trainable low-bit attention, enabling accelerated training with lossless fine-tuning performance, while revealing key insights for low-bit attention in training.
FP16
FP16
FP4 FP8 Q SQ
Q
FP32
FP32
FPAMM OnlineSoftmax
P
P
Por-token Quanitzation
+
P
Smoothing
K
Micro-scaling
Quantization
K SK
6 x 448
V 8V
FP4MM
P 8p
J
Micro-scaling Quantization
FP4 FP8
Figure 2: Workflow of microscaling FP4 attention.
2 Preliminary
FlashAttention. The attention computation contains two matrix multiplications and one softmax calculation: S = QKK". PP Sof (maxx O = PV. The Q, K, V are the shape of x D, where N means the sequence length and D means the dimension of attention head. P.S are in the shape of x N. FlashAttention divides Q to blocks (Q.) } in the shape of B, x D, and divides K, V to {K,). ). (V)) in the shape of B x D. Then it uses online softmax to avoid the large memory I0 for S and P: S.y Q.K3. PsJ OnlineSoftmat tmax (S,,), O2 P.3 Vy.

---

Notation. For simplicity, we omit subscripts and use Q K, V. V.S,P, to denote the matrix blocks in FlashAttention, while retaining full subscript notation in Algorithm
and 3
Quantization. Quantization is used to accelerate Matmul by converting two matrices from high- bit to low-bit with scale factors. Take INT8 quantization for Matmul AB as example, where A and are in FP16 data type. It can be formulated: S max([Al)/127, À [A/saJ, $B = max ())x)B)))22 B2)/127, B- = [B/sB], where A, B are in INT8 and the others are in FP32. Then, AB = AB x SA x $g, which can be accelerated by the INT8 Tensor Core. The granularity of quantization is determined by the dimensions reduced by the max operation. For example, in per- token quantization, the max is computed along each row of matrix. In per-block quantization, the is computed block of matrix, which in our paper means FlashAttention block.
3
FP4 Attention for Inference Acceleration
This section presents our microscaling FP4 attention through three key components: (1) the fundamen- tal workflow for applying microscaling FP4 quantization attention in Section 3.1 (2) the two-level quantization approach for the attention map in Section 3.2 and (3) critical hardware implementation
optimization in Section
Distrbbtton
Distibutionn Quant)
Distribution (Two-levee Quant)
Quamiration mor
6.1758
Fruquancy
Abrolute Ener
0. 0.300
0.04
0.02
0
0.5
0.05 0.50
0.15
240
Blouct
Tens-leva Qunl
Direct Guant-ievel Quang
(a)
Value
(c) Volue
(d)
(e)
Figure 3: Analysis of the benefit of two-level quantization. (a) shows the distribution of P. (b) and (c) show the distribution of sp using direct quantization and two-level quantization. (d) and (e) show the error of sp and P using direct quantization and two-level quantization.
3.1 Microscaling FP4 Attention
FP4 microscaling quantization. Given matrix X E R-Nxd we quantize it to X in FP4 data type with a scale factor matrix x in FP8 data type. Specifically, is partitioned into Xú E R1x= blocks, where each 1 x n block corresponds to one scale factor 8ij- The FP4 microscaling quantization ([& sx
(xX)) and dequantization (X' 0-1 sx)) can be formulated as follows.
Quantization di 8ij max (axx(X)))6 Xi3 = [Xa//sg]
(1)
Dequantization o= X!j Sıj x Xis
(2)
Where the [-] means FP4 rounding.
FP4 microscaling quantization Matmul. Consider matrix multiplication AB, where A and are in FP16 precision. The speed of the Matmul is about 200 TOPS on RTX5090. In contrast, the speed of the FP4 microscaling Matmul is about 1600 TOPS. which is an 8x speedup. The FP4 microscaling Matmul instruction (FP4MM) takes four inputs, i.e., Å, SA, B, 8B, and the output C' equals to the Matmul result between @ (A, 84) and o (B,
= FP4MM Å, SA, B, 8B)
Attention computation. We accelerate attention computation by applying FP4 microscaling quanti- zation to both matrix multiplications: QK and PV.
Q = Φ(Q),
K SK (KT
S = =FFMMM Q sq. K,
On tma((S)
P. Sp = o(P),
V.sy (V)),
O = FPPMM sp, V.sv)
(4)

---

It is important to note that our hardware implementation builds on FlashAttention, where the matrices Q, K. P. and V in our formulation correspond to FlashAttention's tiled Q, K, P. and V blocks as described in Section Additionally, to enhance the attention aceuracy, we adopt the smoothing Q and K in SageAttention2
[4] The complete algorithm is presented in Algorithm
Data type determination. There are two choices for the FP4 data type [8 The first one is the NVFP4, which is in E2M1 data type and its quantization block size is 1 x 16 and its scale factor is in E4M3
data type. The second one the MXFP4, which is also in E2M1 data type. However, its quantization block size is 1 32 and its scale factor is in E8MO data type. We choose NVFP4 because the accuracy of NVFP4 much higher than that of MXFP4 attention quantization. Empirical results: Table [ a
shows the accuracy of MXFP4 and NVFP4 using real Q K, V across all layers of CogVideoX, Results indicate that the accuracy of NVFP4 outperforms that of MXFP4.
Algorithm 1: Implementation of the microscaling FP4 attention.
1: Input: Matrices ((FP16), (FP16), ((FP16) E R xxd block size B5, Bkv-
2: Preprocessing: K = K mean (') // Smoothing K of SageAttention.
3: Divide Q to Tm N/B, By blocks {Q.} ): divide K, and V to T. = N/ blocks (K,), {V.);
4: for i to Tm do
G = mean Q:), (sq: Q:) = pQQ q.) ; // Smoothing Q of SageAttention..
6: for in [1. T.. do
(SK, K = ((J)) (sv, V,) <(V;);
8:
S.y FP4MM Q SQ: ,K (K) + GEMV( K,) ) Smoothing Q
My m rowmax (S,,)). P., exp(S.) mma)).
4g -1 -m 11--1 rowsum
10:
SP1 rowmax P.))((488 6). P Bj SP1 SP P.j ((P (P two-level quantization
11:
Oj diag(e (.. 1.- 10 + FP4MM PP. SP2: V,,5v) x sp,
12:
end for
13:
end for diag((,,T.) O
14:
15: return 0 = {
3.2 Two-level Scaling for
Applying microscaling FP4 quantization for P presents challenge to attention accuracy. For example, Fig. 12 C shows direct quantization severely degrades output quality, producing results substantially different from full-precision outputs. Our analysis reveals that the issue occurs because microscaling NVFP4 quantization requires the scale factor to be represented in E4M3 FP8 format [2], rather than the FP32 data type typically used for scale factors. This causes accuracy loss when the scale factor is directly converted to E4M3 format. To better understand this accuracy loss, we analyze the data distribution of P and its scale factors in Fig. Since is computed using online softmax 100, the values in each microscaling block P:y fall [0, 1]. Consequently, the scale factor (scale factor max (P:) /))) ranges between O and 0.167. This narrow range leads to inefficient usage of E4M3's representable range, increasing accuracy loss. To reduce accuracy loss by fully utilizing E4M3'S range, we propose two-level quantization method for the P matrix. Specifically, we first quantize each row of P to [0, 448 x 6]. Then we apply the standard FP4 quantization & for the quantized P. The two-level quantization can be formulated follows:
SP, rowmax (P) (448 6),
P2 P/SP.: SP P2=
P P x Sp, sp,), O FPPMM(( P. 2, SP,: V sv) x Sp,
(5)
Where P2. and Sp, are in FP32 data type. SP, and sy are in FP8 data type. P2 and V are in FP4 data type.
Empirical results: As shown in Fig. 3 our two-level quantization maximizes the E4M3 range utilization for sp, thereby reducing both the numerical representation error of sp and the quantization error of P. A more formal theoretical analysis is provided in Appendix A.5 Table bb shows the accuracy of

---

two-level quantization against naive direct quantization, using real Q, K. V from layers of VideoXX Results indicate that two-level quantization boosts the accuracy.
3.3 Implementation and Optimization on Hardware
Permutation for K. Unlike FP16, the FP32 accumulator's memory layout FP4 MatMul [[]] differs from its operand A's register layout (shown in Fig. 20 and 19). Performing thread shuffles to match operand A's layout would degrade kernel performance. Our solution transforms the accumulator layout (Fig. 21) by permuting the P tile's columns. To maintain correct MatMul, we correspondingly rearrange s columns, which can be fused with the quantization kernel.
Reuse shuffle. The in-kernel scaling quantization of P requires finding the max value of 16 consecutive row elements. However, as shown in Fig-211 these 16 elements are distributed across four threads, necessitating intra-thread max reduction followed by inter-thread shuffling, significantly slowing down the kernel. We optimize this by fusing quantization with online softmax, which also computes row-wise maxima. First, we compute the max over 16 elements in S and reuse it in the subsequent softmax max-reduction. This fusion reduces redundant shuffles and max operations by 50%, yielding about 10% whole kernel speedup.
Producer warp epilogue. In conventional warp-specialized kernels, consumer warps typically handle both MatMul and store operations while producers merely load inputs, with ping-pong scheduling between consumers enabling stage overlap [12] However, register constraints make this approach infeasible for our FP4 attention kernel. Instead, we implement ping-pong scheduling between producer warps: while one producer loads inputs for the next MatMul operation, another concurrently stores outputs to global memory, with consumer warps solely responsible for transferring MatMul results from registers to shared memory. This novel design overlaps MatMul and global memory stores within register constraints, boosting throughput.
4 INT8 Attention for Training
Low-bit quantization attention works, such as FlashAttention3 and SageAttention, only for inference. In this section, we propose an INT8 attention for training, named SageBwd, which quantizes six of seven matrix multiplications in attention to INT8, achieving performance degradation in fine- tuning tasks. Besides, we implement both INT8 SageBwd and FP8 SageBud and conduct comparison experiments, proving INT8 SageBwd is superior to FP8 SageBud in Section 5.4
Algorithm 2: Forward pass of the 8-bit attention.
I: Input: FP16 matrices Q, K, V E Rxxx and block size B,, By
2: Km mean(K). K t- K- Km // Smooth-k technique.
3: Divide Q to T.. N/ B, blocks }Q divide K, and V to T, N/ blocks J.,). (V.);
4: Quantization: {sq. Q | (((().)) {sK, K.) (((KK }} (sv, V. (V(V.) ;// Per-block.
5: for
to T do
6: O, E BB DD
(0), L, E R. (0), m, B2. (0):
7: for j in [1, T., do do
S.y MM (M( K2) x sq x sk:
9:
my max rowmax )) P, exxp(S., m.y). 4y -m. 1- mus 4 rowsum (P, 1,);:
10:
exp(rowmax(S,) m., 1222/ P., P.y/sp // Per-token quantization.
11:
O,, =
00.,-1 PM u1 V,) x Sp x sv
12:
end for
13: O, stag(1,,r, 0
L. m.rr + log (1.,T.)
15: end for
16: return O= L=
5

---

4.1 Forward
There are two matrix multiplications the forward pass of attention:
= =KQ," PV
(6)
Per-token quantization for P. Following SageAttention B3, we apply smoothing and per-block INT8 quantization for the QK However, for the PV, a static per-block INT8 quantization with a static scale factor 1/127 for inaccurate B Fortunately, we find applying per-token INT8 quantization for PV and per-block INT8 quantization for V can enhance the attention accuracy. Furthermore, we eliminate the need for explicit max operations on P by reusing both global and local maximum values from the online softmax computation (Line 9 in Algorithm 2 The algorithm for the forward is shown in Algorithm 2
Given our extensive use of INT8 per-block quantization in trainable attention, we formalize the process as follows. For each FlashAttention block X, the quantization process sx. X = (X) can be formulated as:
Sx max X 127, X X/sx
Algorithm 3: Backward pass of the 8-bit attention.
I: Input: {sq. i {sK. K.). (sv. V.J. Km. O, {L,} from forward, dO E R block size B,, Bx
2: D = rowsum(dO O). divide D to Tm. N/B, blocks {D.):
for j = 1 T., do
4: for in [i. do
5:
S. = MM Qu K2) x sQ x sK: Pij = exp(S:; L,);
6:
sp, P., = v(P.). sao, d0, = v(dO.) // INT8 per- quantization.
dV, <- dV, + MM(P) dO.) x sp x sao
8:
dP,, MM(dO, v
Keep in FP16.
9:
dS.; Pú (dP,; 5j D:); sas, dS.y (dS:)) INT8 per-block quantization.
10:
dQ.- + dQ,- MM dS . K,)
Sk + rowsum(dS.). Km, // Backward for smooth-k.
11: dK, < dK, + dMS Q.) x sas x sq
12: end for
13: end for
14: return dQ, dV
4.2
Backward
There are five matrix multiplications in the backward pass of attention:
= QK dv aV=PTao dP dov aQ = ask, dK ds Q
(8)
We observe that whether applying quantizing to dov has a significant impact on the accuracy of the gradient of Q, K. This because the accuracy of dov directly determines the accuracy of dP and dS (see computational dependencies in Algorithm 3 The accuracy loss in dS will continuously accumulate errors into dQ and dK during the recurrent process along the sequence length in FlashAt- tention's 's backward pass, meaning longer sequences lead to greater error accumulation. Therefore, we maintain dOv FP16 while accelerating the other four matrix multiplications using INT8 per-block quantization. The algorithm for the forward is shown in Algorithm 3 Empirical results:
Table shows the accuracy of the dQ with and without quantization dOV! We find that the accuracy of dQ is significantly improved when keeping dOVT in FP16.
Table 1: Accuracy ablation using different quantization strategies.
(a) Different FP4 chaices
(b) Different scale strategies for P
(c) Different data types for dOV
Type
CosSim
L1j
RMSEJ
Method CosSim
L1
RMSE
Method CosSim
L1
RMSE
MXFP4
98.37%
0.294
0.994
Direct 93.32%
0.193
1.103
INT8
97.47%
0.171
2.440
NVFP4
99.52%
0.077
0.201
Two-level 99.52%
0.077
0.201
FP16
99.77%
0.039
0.692

---

RTX5090, (Heed dim 128. causal False)
RTX5090, (Head causal True)
FlashAttn
SageAtti22
PlashAtttn SageAitn1
SegeAtti2
Speea (TOPS)
xformers
SageAttn1
SageAttin3
xformers
SaggeAtti
D01
1C20
902
1000 D54
500
500
Sequence Length
Sequence Length
Figure 4: Speed comparison between SageAttention3 and Baselines (RTX5090, headim=128)
RTX5090, (Head dim 64, causel
False)
RTX3090, (Head dim 64. causal True)
Speed (TOPS)
1500
Torch
SageAttn1
FlashAttn
SegeAtti3
SagnAtti2
1500
Torch
FlshhAttn SageAttn1
SageAtti2? Saggsttind
1000
753
500
1K
2K
4K
8K
16K
32K
1K
2
4K
8K
16K
32K
Sequence Length
Sequence Length
Figure Speed comparison between SageAttention3 and Baselines (RTX5090, headim=64).
RTX4090, (Head
RTX4090,
400
Torch
FlashAttnj(udaa)
SageBwd
400
Torch
Flashhttr(Cuda))
SageBwd
Speed (TOPS)
xformers
PiachAttn/Bitann
xfomers
FlschAttin Triton)
300
300
206
200
L53
30
142
2148
100
100
55
DOOM
55
1K
2K
4K
BK
16K
32K
1K
4K
16K
3.K
Sequence Length
Sequence Length
Figure 6: Speed comparison between SageBud and Baselines (RTX4090, headim=128).
RTX4090, (Head dim 64, causal
False)
RTX4090, (Head dlm 64, causal
True]
FlashAttaiCudaa)
SageBwd
SageBwd
Speed (TOPS)
400
xformers
FlschAttn/ Witton)
xfommes
Fashhttrr/Titto)))
10 248
200 158
LSS
DCN
GO
DON
BE
Sequence Length
Sequence Length
Figure 7: Speed comparison between SageBud and Baselines (RTX4090, headim=64).
Table 2: End-to-end metrics comparison on various models.
Model
Attention
CLIPSIM 1
CLIP-T 1
VQA-a f
VQA-t t
FScore 1
Pull-Precision (16bit)
0.18665
0.9968
70.476
69.875
4,780
CogvideoX
SageAttention2 (8bit)
0.1880
0,9969
69.414
70.750
4.534
SageAttention3 (4bit)
0.1881
0.9969
69.860
70.364
4.035
Full-Precision (16bit)
0.1838
0.9993
68.998
78,891
1.4793
Hunyuan
SageAttention2 (8bit)
0.1836
0,9993
69.497
77.019
1.4741
Video
SageAttention3 (4bit)
0.1866
0,9993
70,552
75.440
1.232
Full-Precision (16bit)
0.1828
0,9990
61.9840
61,0000
L8042
Mochi
SageAttention2 (8bit)
0.1819
0.9990
61.0093
60.3732
1.7539
SageAttention3 (4bit)
0.1800
0,9993
61.863
59.429
1.649
Model
Attention
FID |
sFID |
CLIP 1
IR
Full-Precision (16bit)
162,812
146.980
31,409
0.91
Flux
SageAttention2 (8bit)
163.107
146.213
31,436
0,90
SageAttention3 (4bit)
162.121
142,839
31.450
0.94
Full-Precision (16bit)
166.421
146.379
31.93
0.93
Stable-Di
SageAttention2 (8bit)
164.986
148.557
32.01
0.93
ffusion3. 5 SageAttention3 (4bit)
166.102
145.587
32.01

---

Pretrai FinWee--du
Aineuning DROF
Fineturing
Bretuning
Br16 Llems
ar16-Bwee2 5
BF36 QW012.5
BF36 Swo12.5
3r16-Buee2
Sageded -Lame
Sageded- 0uer25
SyjcBui-Qwent
SageBhil - -wenz.5
Sage2ud- 0.er2.5
2718 -Limai2
BF36 -2j6-L1an832
BF16 -LTana32
2718 -Limm122
Sagelat-Llanal
Sayethet -lhusk2
Sagethet - 1'an922
Sagelat- Llana3
021
0.9
Shop
S1op
Stop
Stop
S359
(a)
(b)
(c)
(d)
(e)
Figure 8: Pretraining and Finetuing loss curves of BF16 and 8-bit atttention.
Table 3: 8-bit attention finetune results on Qwen2.5 5 and Llama3. 2 models.
Model
Method
DROP GSM8K
MMLU
HELLASWAG
BF16
0.521
0.733
0.569
0.905
Qwen2.5 (I.5B)
SageBwd
0.520
0.734
0.574
0.911
BF16
0.601
0.785
0.640
0.944
Qwen2.6 .5 (3B)
SageBud
0.607
0.782
0.653
0.943
BF16
0.259
0.641
0.464
0.828
Llama3.2 (IB)
SageBwd
0.268
0.637
0.458
0.823
5 Experiments
Main results. SageAttention3 is faster than FlashAttention and xformers by 5x and 11x on RTX5090, and maintains end-to-end metries across various models. Furthermore, SageBud faster than FlashAttention and xformers by 1.67x and 3x on RTX4090, and achieves no measurable degradation fine-tuning tasks.
5.1 Setup
Models and attentions. We validate the effectiveness of SageAttention3 and SageBwd across a diverse set of representative models from language, image, and video generation. Specifically,
conduct experiments on: Qwen2.5 5
1 and Llama3. [14] for text2text, CogvideoX [5)].
HunyuanVideo [16). and Mochi [17 for text2video, Flux [88, and
5 [19
for text2image. We compare our method with FlashAttention2 (20). xformers (21). SageAtten-
tion [3], and SageAttention2 [4. Please note that FlashAttention3 can only run on Hopper GPUs, so FlashAttention2 is already the fastest version for RTX5090 and RTX4090.
Datasets, metrics, and hyperparameters. For the details about the datasets, metrics, and hyperpa- rameters used, please refer to Appendix A.3
Implementation. We implement SageAttention3 using CUTLASS 22 and CUDA, and imple- ment SageBud using OpenAl Triton 23
Precition
Full Precision
SepaAttention
Figure 9: Visible examples of video generation HunyuanVi Video (left) and image generation SableeDiffusion3 (right).

---

Table 4: End-to-end speedup performance using SageAttention3 and SageBwd.
(a) Inference latency using SageAttention?:
(b) iteration training latency using SageBwd.
Model
Original
Sage1
Sage2
Sage3
Model
Original
SageBwd
CogvideoX (2B)
64 489 s
55 s 257 s
46 8
275 s
Llama (8K)
2.1 s 6.0
1.9 5.2s s
HunyuanVideo
240 8
164 s
Llama (16K)
5.2 Efficiency and Effectiveness
Kernel Speed. Fig. and show the kernel speed of SageAttention and baselines on RTX5090. We can that SageAttentions achieves 4-5x speedup FlashAttention2 and 8-11 x speedup over xformers. Fig. [nnd show the forward+backward speed of SageBwd and baselines on RTX4090. It shows that SageBud achieves 1.67x speedup at most than FlashAttention2 and higher speedup than FlashAttention2 implemented in Triton and xformers.
End-to-end metrics loss of SageAttention3. In Table on various models using SageAttention and other attention methods. The results demonstrate that SageAttention almost incurs almost end-to-end quality loss across these models.
compare the end-to-end quality metrics
End-to-end metrics loss of SageBwd. To evaluate the effectiveness of SageBwd on training tasks, we conduct two experiments. First, we fine-tune the base models of Qwen2. 5 (3B) and L1 ama3. 2 (1B) on GSMSK [24], DROP (25), MMLU [26]. and HELLASWAG [27] datasets. Fig. 8
shows the fine-tuning loss results, indicating that SageBwd perfectly aligns with BF16. Moreover, our evaluation of the fine-tuned models' answer quality across multiple test datasets (Table 3 demonstrates that
SageBwd achieves the same performance as BF16. Second, we conduct pre-training tasks on Fine Edu (28) using Llama (400M) (29] model. Fig. (a) shows the loss curve, indicating that while SageBwd achieve loss convergence, its convergence speed is relatively slow. This limitation restricts its applicability in pretraining tasks.
Visible example. Fig. visualizes some comparative examples of video generation on HunyuanVideo and image generation on Stable-diffsion3.. using SageAttention3. The results demonstrate that SageAttention3 maintains full generation quality. Additional visible examples are provided Fig. 10
13 and in the Appendix.
End-to-end speedup. Table tency improvements. The results show that SageAttention3 (Table 4 a)) achieved about 3x (HunyuanVideo) and 2.4x (CogVideoX) end-to-end inference generation speedups on RTX5090. Furthermore, SageBwd (Table 4(bb) accelerates the training of Llama (IB) by about 1.15x using 8K/16K token micro-batches on RTX4090.
and summarize end-to-end inference and training la-
5.3 Benefit of Using Both SageAttention3 and SageBwd
Table 5: Comparison between BF16 and INT8 fine-tuning followed by FP4 inference.
(a) Qwen2. 5-1. 5B results.
(b) Qwen2. 5-3B results.
Method
GSM88
MMLU 1
Method
GSM8k +
MMLU 1
BF16 Fine-tuning SageBud Fine-tuning
0.4912
0.4688
BF16 Fine-tuning
0.5860
0.6000
0.5232
0.4934
SageBwd Fine-tuning
0.5945
0.6032
We first apply SageBud during fine-tuning, followed by PageAttentions during inference. Specif- ically, we fine-tuned Qwen2.5 for 1, 1,000 steps using either BF16 or SageBud, and then evaluated inference performance using SageAttention3. The results on GSM8k and MMLU are shown in Table 5 INTS SageBwd fine-tuning followed by FP4 SageAttention3 inference achieves higher accuracy on GSM8k and MMLU, suggesting the approaches are complementary. This improvement likely because INT8 and FP4 share a more similar representable data distribution, reducing the mismatch error compared to BF16.

---

5.4 INT8 SageBwd vs FP8 SageBwd
We choose INT8 for SageBwd for two key reasons: (1) Higher gradient accuracy in attention backward. The backward of INT8 attention yields more accurate gradients for Q. K, and V compared to FP8. We evaluate all layers of CogVideoX-22 and report the error and cosine similarity of the gradients in Table 6 and Table For fairness, dOv is kept in FP16 for both methods. As shown in the results, INT8 SageBwd achieves lower LI error and higher cosine similarity than FP8 SageBwd. (2) Wider hardware support. INT8 is supported on almost all modern GPUs, including NVIDIA A100 and many non-NVIDIA devices (e.g., AMD MI250 [30]. Ascend 910B BI), while FP8 support remains limited to newer architectures. Addintionally, we fine-tune Qwen2. 5...5-1.. 5B and Qwen2. 5- for 1,000 steps using either INT8 or FP8 SageBwd (both with dOV kept in FP16 for fairness), and then inference with FP4 SageAttention3. As shown in Table 8 models fine-tuned with INT8 attention achieve higher accuracy on both GSM8K and MMLU benchmarks.
Table 6: LI error of Q, K, and V gradients.
Table 7: Cos similarity of Q, K, and V gradients.
Method
dQ
dK
dV
Method
dQ
dK
dV t
INT8 SageBwd
0.0290
0.0317
0.0423
INT8 SageBwd
0.9987
0.9993
0.9995
FP8 SageBwd
0.0696
0.0999
0.0873
FP8 SageBwd
0.9880
0.9910
0.9955
Table 8: Comparison of INT8 and FP8 SageBwd fine-tuning Qwen2.5 5 models. (b) Qwen2. 0p-B
(a) quen2.5-1.5B
Method
GSM8K f
MMLU f
Method
GSM8K
MMLU f
INT8 Fine-tuning
0.5232
0.4934
INTS Fine-turing
0.5945
0.6032 0.5907
FP8 Fine-tuning
0.5031
0.4689
FP8 Fine-tuning
0.5868
Related Work
Recent efficient attention works 22 that utilize hardware features to accelerate attention computation methods mainly include the following: FlashAttention [32] introduces tiling to reduce the GPU memory 1/O between global memory and on-chip SRAM, achieving significant speedup. FlashAtten-
tion2 [20] improves the parallelism and warp partition strategies. FlashAttention3 [33] exclusively optimizes the kernel speed on the Hopper GPUs. xformers (2 accelerates attention using dedicated
CUDA kernels. SageAttention K and Attentitnn 4 34 accelerate attention using quantiza-
tion and some novel outlier smoothing techniques. RingAttentton 335 extends Flash Attentito to
multi-GPU/Node environments. In these works, although FlashAttention3 proposes a version of FP8 attention, it has failed be applied to video generation models plug-and-play way (4) Moreover, the FP8 attention in FlashAttention, does not support the backward pass, limiting its applicability to training tasks, Additionally, numerous efficient attention variants have emerged, including linear at- tention [36] 37, 38 39 40 41 and sparse attention [42 43 44 45 46 47 48 49 [50] [1] [22 (3]]5] 54].
Although these works represent promising research directions, they are orthogonal to our work.
7 Conclusions
In this paper, we make two key contributions. Firstly, we design SageAttentions in3, the first mi- croscaling FP4 attention for inference acceleration, achieving 1038 TOPS on RTX5090, which is a 5x speedup than the fastest FlashAttcntion on RIX5090. Experiments show that SageAttentions could accelcrate various models with no end-to-end quality metrics degradation. Secondly, introduce the first trainable 8-bit attention (SageBwd) for training acceleration and explore its feasibility in training tasks. We find that the 8-bit attention could achieve lossless performance in fine-tuning tasks, but currently has some limitations in pertaining tasks.
Future Work. First, while SageBwd demonstrates faster performance than FP16 implementation, we observe a noticeable gap between its current speed and theoretical upper bounds. This gap may be caused by suboptimal Triton kernel implementations, which we plan to further optimize. Second, and more importantly, investigating the application of low-bit attention pretraining tasks presents a promising research direction worthy of exploration.
10

---

Acknowledgments
This work was supported by the NSFC Projects (Nos. 62550004, 92270001, 62376131). J.Z. is also supported by the XPlorer Prize.
References
[1] A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
[2] Jintao Zhang, Rundong Su, Chunyu Liu, Jia Wei, Ziteng Wang, Pengle Zhang, Haoxu Wang, Huiqiang Jiang. Haofeng Huang, Chendong Xiang, al. survey of efficient attention methods: Hardware-efficient sparse, compact, and linear attention.
[3] Jintao Zhang, Jia Wei, Pengle Zhang, Jianfei Chen, and Jun Zhu. Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration. In The International Conference on Learning Representations, 2025.
[4] Jintao Zhang, Haofeng Huang, Pengle Zhang, Jia Wei, Jun Zhu, and Jianfei Chen. Sageatten- tion2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization. In International Conference on Machine Learning (ICML), 2025.
[5] Jintao Zhang, Kaiwen Zheng, Kai Jiang, Haoxu Wang, Ion Stoica, Joseph E Gonzalez, Jianfei Chen, and Jun Zhu. Turbodiffusion: Accelerating video diffusion models by 100-200 times. arXiv preprint arXiv:2512. 16099,, 2025.
[6] Jianfei Chen, Yu Gai, Zhewei Yao, Michael W Mahoney, and Joseph E Gonzalez. A statistical framework for low-bitwidth training of deep neural networks. Advances in neural information processing systems, 33:883-894, 2020.
[7] NVIDIA. Nvidia rtx blackwell gpu architecture arcchiccuee http://wwwwtetttteee/ee/////ee/s http:///ttttttt.c/////////aaattt
www.wwiwww....aiaaoa.atet.aa pdf
[8] Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger, Kristof Denolf, et al. Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10553 2023.
[9] Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, Fp8 formats for deep learning. arXiv preprint arXiv:2209.0544 2022.
[10] Maxim Milakov and Natalia Gimelshein. Online normalizer calculation for softmax. arXiv preprint arXiv: /805.8807,..77 2018
[11] NVIDIA. Parallel Thread Execution ISA Version 8.7. pdf/ptx_isa_Bi 8.4. pdf 2025. Accessed: 2025-05-16.
[12] NVIDIA. Efficient gemm in cuda. https://door docs nvidia. wwt.//tlaass///ddaa//oo/// cpp/ soppefffiieett.. cenrtt_gemmmmtt. html 2025. Accessed: 2025-05-16.
[13]
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2. 5 technical report. arXiv preprint arXiv:2412. 15, 2024.
[14] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
[15] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. In The Thirteenth International Conference on Learning Representations, 2025.

---

[16] Weijie Kong, Qi Tian, Zijian Zhang, Rox Min. Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, Kathrina Wu, Qin Lin, Aladdin Wang, Andong Wang, Changlin Li, Duojun Huang, Fang Yang. Hao Tan, Hongmei Wang, Jacob Song, Jiawang Bai, Jianbing Wu, Jinbao Xue, Joey Wang, Junkun Yuan, Kai Wang, Mengyang Liu, Pengyu Li, Shuai Li, Weiyan Wang, Wenqing Yu, Xinchi Deng. Yang Li, Yanxin Long, Yi Chen, Yutao Cui, Yuanbo Peng. Zhentao Yu. Zhiyu He, Zhiyong Xu, Zixiang Zhou, Zunnan Xu, Yangyu Tao, Qinglin Lu, Songtao Liu, Daquan Zhou, Hongla Wang, Yong Yang, Di Wang, Yuhong Liu, Jie Jiang, and Caesar Zhong. Hunyuanvideo: systematic framework for large video generative models. arXiv preprint arXiv:2412.03661, 2024.
[17] Genmo Team. Mochi 1.
2024.
[18] Black Forest Labs. Flux. https://github... http:/uu...combmm.k........aaa.. fallacck-fforrt--aaabst 2023.
[19]
Stability AL. Introducing stable diffusion 3.5.
https:/sttability: ai/news/
introducing-state diffusion- 3-- 2023.
[20] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. The Twelfth International Conference Learning Representations, 2024.
[21] Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang. Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xformers: A modular and hack- able transformer modelling library. htttps://githui: http://hhw..cceeeeccooeeeeeeerra 2022.
[22] NVIDIA. CUTLASS: CUDA Templates for Linear Algebra Subroutines and Solvers. GitHub repository, 2023.
[23] Philippe Tillet, H. T. Kung, and David Cox. Triton: intermediate language and compiler for tiled neural network computations. MAPL 2019, page 10-19, New York, NY, USA, 2019, Association for Computing Machinery.
[24] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 10. 2021.
[25] Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proc. of NAACI, 2019. 2019.
[26] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representutions (ICLR), 2021.
[27] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
[28] Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest collection of educational content, 2024.
[29] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint 232. 2023.
[30] AMD. Amd instinct mi250 gpu architecture, https: Settps:/rrocns docs amd. com/en//ee wwweee/ddocceeeeeeeeeeeeeee 6..4
2/ ///ooopptulll/guuarccceeeaataeea http://gpuurrcc/m22mmaaaa m1250 html 2024. Accessed: 2025-10-21, ROCm Documenta- tion.
[31] Heng Liao, Jiajin Tu, Jing Xia, Hu Liu, Xiping Zhou, Honghui Yuan, and Yuxing Hu. Ascend: a scalable and unified architecture for ubiquitous deep neural network computing: Industry track paper. In 2021 IEEE International Symposium on Siigh-Pefformacce Computer Architecture (HPCA), pages 789-801. IEEE, 2021.
12

---

[32] Tri Dao. Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher Re. Flashattention: Fast and memory-efficintt exact attention with awaaereesss. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[33] Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. In The Tiihty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[34] Jintao Zhang, Xiaoming Xu, Jia Wei, Haofeng Huang, Pengle Zhang, Chendong Xiang, Jun Zhu, and Jianfei Chen. Sageattention2+:: A more efficient implementation of sageattention2. arXiv preprint urXiv:2505.21 2025.
[35] Hao Liu, Matei Zaharia, and Pieter Abbeel. Ringattention with blockwise transformers for near-infinite context. In The Twelfth International Conference Learning Representations, 2024.
[36] Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint vrXiv:2006.047668 2020.
[37] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy Colwell, and Adrian Weller. Rethinking attention with performers. In International Conference on Learning Representations, 2021.
[38] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEF/CVE conference on computer vision and paltern recognition, pages 1081 00819- 2022.
[39] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers mns: Fast autoregressive transformers with linear attention. In International conference on machine learming, pages 5156-5165. PMLR, 2020.
[40] Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. Lightningg attention-2: A free lunch for handling unlimited sequence lengths in large language models. arXiv preprint arXiv:2401.0440
2024.
[41] Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. arXiv preprint arXlv:2412.064444, 2024.
[42] Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. Spargeattn: Accurate sparse attention accelerating any model inference. In International Conference on Machine Learning (ICML), 2025.
[43] Jintao Zhang, Haoxu Wang, Kai Jiang, Shuo Yang, Kaiwen Zheng, Haocheng Xi, Ziteng Wang, Hongzhou Zhu, Min Zhao, Ion Stoica, et al. Sla: Beyond sparsity in diffusion transformers via fine-tunable sparse- aparse-linear attention. arXiv preprint arXiv:2509.2400 2025.
[44] Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, et al. Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity. arXiv preprint arXiv:2502.01776. 2025.
[45] Shuo Yang. Haocheng Xi, Yilong Zhao, Muyang Li. Jintao Zhang, Han Cai, Yujun Lin, Xiuyu Li, Chenfeng Xu, Kelly Peng, et al. Sparse videogen2: Accelerate video generation with sparse attention via semantic-awan aware permutation. arXiv preprint arXiv:2505.1 18875, 2025.
[46] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[47] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
13

---

[48] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024.
[49] Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. Infllm: Training- free long-context extrapolation for Ilms with an efficient context memory, In First Workshop on Long- Context Foundation Models@ @ ICML 2024, 2024.
[50] Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. In The International Conference on Learning Representations, 2024.
[51] Huiqiang Jiang, YUCHENG LI, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. MInference 1.0: Accelerating pre-filling for long-context LLMs via dynamic sparse attention. In The eiihth Annual Conference Neural Information Processing Systems, 2024.
[52] Shashanka Venkataramanan, Amir Ghodrati, Yuki M Asano, Fatih Porikli, and Amir Habib- ian. Skip-attention: Improving vision transformers by paying less attention. In The Twelfth International Conference on Learning Representations, 2024.
[53] Yizhao Gao, Zhichen Zeng. Dayou Du, Shijic Cao, Hayden Kwok-Hay So, Ting Cao. Fan Yang. and Mao Yang. Seerattention: Learning intrinsic sparse attention in your llss. arXiv preprint arXiv:2410. 13276, 2024.
[54] Tianyu Fu, Haofeng Huang, Xuefei Ning, Genghan Zhang, Boju Chen, Tianqi Wu, Hongyi Wang, Zixiao Huang, Shiyao Li, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Moa: Mixture of sparse attention for automatic large language model compression. arXiv preprint arXiv:2406. 14909, 2024.
[55] Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all. 2024.
arXiv preprint arXiv:2412.204000
[56] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th Europeun Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Purt 13, pages 740-755. Springer, 2014.
[57] Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 22139 2009-221444 22149,111119 2024.
[58] Haoning Wu, Erli Zhang. Liang Liao, Chaofeng Chen, Jingwen Hou, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin. Exploring video quality assessment user generated contents from aesthetic and technical perspectives. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 100144200150 2023.
[59] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[60] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
[61] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514-7528, 2021.
14

---

[62] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023.
[63] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference Machine Learning, pages 38087-38099, PMLR, 2023.
[64] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for Il compression and acceleration. In MLSys, 2024.
15

---

A Appendix
A.1
Visible Comparison Examples
Sage AtgeAtttnnionn
Full Precision
Figure 10: Visible examples of image generation Stable- Difflle-Dfffiinn33. 5.
Sage Attention
Full Precision
Figure 11: Visible examples of image generation on Flux.
(a) Full-Precision
(b) Two-level quantization
(c) Direct quantization
Figure 12: Visual comparison of different scale strategies for P from CogVideoX.
16

---

SageA
Full Precision
Sage AtgeeAttention
Full Precision
Figure 13: Visible examples of video generation on CogVideoX.
SageAttentions
Full Precision
AageAtteniinn
Full Precision
Figure 14: Visible examples of video generation on HunyuanVideo.
17

---

Fig.
and Fig. show additional visual comparison examples of image generation tasks. Fig. and show more visual comparison examples of video generation tasks.
10
A.2
Additional Kernel Speed Comparison
Fig.
15 and Fig. 16 show the forward kernel speed of SageBwd. Fig. and Fig. 18 show the backward kernel speed of SageBwd. SageBwd achieved 2x speed up than FlashAttention in the forward propagation. SageBwd achieved 1.2-1.6x speed than FlashAttention in the backward propagation.
RTX4090, (Head dim - 126, causal False)
RTX4090, (Head dim 128, causal Thue)
Speed (TOPS)
600
xformers
Torch
FlashAttni lriton)
FlschAtti(Cdda)
SageBwd
xformers
Tarch
FlashAtty(Cuds)
FlashAktti Triton)
SageBwd
200
152
258
200
15
LSS
1,55
157
1066
200
10
962
135
1159
157
4K
16k
Length
Sequence Length
Figure 15: Forward speed comparison between SageBwd and Baselines (RTX4090, headim=122
RTX4090, (Head dim 64, causal False)
RIX4090, [Head dim 64, causal = True)
Torch
FlschAttsiCdda)
SageBwd
Torch
FlashAttr(Cudaa)
SageBed
Speed (TOPS)
xformers
FlachAttn/ litan)
(isshatti((Tttoo))
400-
120
400
0
4K
16K
Length
Sequence Length
Figure 16: Forward speed comparison between SageBud and Baselines (RTX4090, headim=64).
RTX4090, (Head dim = 128, causal False)
RTX4090, (Head dim 128, cousal True)
Torch
Cldd))ttn(Cuda)
SageBwd
Torch
(ashhAtb(Cuda))
SageBwd
Speed (TOP5)
300
300
Flash/tty
17
140
200
2006
GDO
1K
2K
4K
BK
16K
32K
1K
2K
4K
8K
16K
32K
Sequence Length
Figure 17: Backward speed comparison between SageBwd and Baselines (RTX4090, headim=128)
RTX4090, (Head dim 64, causal False)
RTX4090, (Head dim
causal True)
500
500
Tonch
ClddhAttn(Cuda)
SageBwd
Torch
Fashhttby(Cdas)
SageBwd
Speed (TOPS)
400
Flashotty/TThtoo
200
105
226
1ST
LSS
1iee
100
DON
DON
100
1K
2K
4K
BK
16K
32K
1K
4K
8K
16K
32K
Length
Sequence
Length
Figure 18: Backward speed comparison between SageBud and Baselines (RTX4090, headim=64).
18

---

A.3 Datasets, Metrics, and Hyperparameters
Datasets. Text-to-video models evaluated using the open-sora [55]
prompt sets. Text-to-image models are assessed on COCO annotations [56]. Language models are evaluated on GSM8K (24).
DROP 25 MMLU [26]. and HELLASWAG 27 datasets.
End-to-end metrics. For text-to-text models, we use Accuracy (Acc.) and F1-Score (FI). For text-to- video models, we evaluate the quality of generated videos on five metrics: CLIPSIM and CLIP-Temp
(CLIP-T) 57 to measure
text-video alignment; (VQA-a) and (VQA-t) to assess the video
aesthetic and technical quality, respectively, and Flow-score (FScore) for temporal consistency [58]
For text-to-image models, generated images are evaluated three aspects: FID [59] and sFID 660
for fidelity evaluation, Clipscore (CLIP) 61 for text-image alignment, and ImageReward (IR) 62 for human preference.
Accuracy metrics. We use three metrics to assess the accuracy of quantized attention output O' compared to attention output in full-precision O: First, we flatten O' and into vectors in the shape of I X Then, Cosine similarity: CosSim 00' VELIOV>>O Relative LI distance: L1 = -1= - O|, Root mean square error: RMSE (/L/E) NO--
Hyperparameters. For pretraining tasks, we use a 400M model with hidden size of 1024, 20 layers, an intermediate size of 3072, and 16 attention heads. The training uses a learning rate of 1e-3 with linear decay over 1000 warmup steps, and each step processes 2M tokens. For finetuning tasks, we train for 700 steps using a learning rate of 3e-5 with linear decay and 100 warmup steps with batch size 32 on GSM8K dataset and 128 on MMLU, DROP, and HELLASWAG datasets.
(9643.00))
Figure 19: FP4 operand A register layout rows 0 and 8, thread 0-3, entries 0-15.
1200.22)
2008
1806.
1609.10 k0)
1B0k k7
TLOok
S00AA voL)
TROME 112
TROCk 4131
ISmAs. vaa)
Tainas sai
TRrvas
Figure 20: FP32 accumulator register layout rows 0 and 8, thread 0-3, entries 0-15.
TOW 43)
1206.0 v0
1206.40)
100.00
1466 V0
1804 40
T866 41
1et 40
Tlok 40
Talkk V0
19/44 V0
1004 47)
12446
10/4 471
1ape.x9
Nov4s v013
1ea1 4030
10 vi1
T1003 v11)
1101k v14)
Figure 21: Permuted FP32 accumulator register layout rows O and thread 0-3, entries 0-15.
A.4 Additional Experiments of using SageBwd
Table show Qwen2. 5 (1.5B), Qwen2. 5 (3B), and Llama3. 2 (3B) fine-tuning results four datasets with five different random seeds. The average and standard deviation show SageBwd highly consistent with BF16 across various random seeds.
19

---

Table 9: Comparison of SageBwd and BF16 performance on GSM8K and DROP across different seeds on Qwen2.5 5 (1.5B).
GSMSK
DROP
Seed
SageBwd
BF16
SageBwd
BF16
42
0.5133
0.5125
0.7316
0.7364
233
0.5027
0.5042
0.7269
0.7295
1234
0.4973
0.4973
0.7329
0.7342
5678
0.5201
0.5208
0.7340
0.7332
1
0.5049
0.5057
0.7278
0.7404
Avg Std
0.5077
0.5081
0.7307
0.7348
0.0090
0.0089
0.0032
0.0040
Table 10: Comparison SageBwd and BF16 performance on MMI.U and HellaSwag across different seeds on 5 (I.5B).
MMLU
HellaSwag
Seed
SageBwd
BF16
SageBwd
BF16
42
0.5814
0.5873
0.9089
0.9065
233
0,5746
0,5785
0,9082
0,9049
1234
0.5805
0.5836
0.9025
0.9047
5678
0.5736
0.5693
0.9112
0.9053
0.5830
0.5823
0.9058
0.9075
Avg Std
0.5786
0.5802
0.9073
0.9058
0.0043
0.0069
0.0033
0.0012
Table 11: Comparison of SageBud and BF16 performance on GSM8K and DROP across different seeds Q (3B).
GSM8K
DROP
Seed
SageBwd
BF16
SageBwd
BF16
42
0.5982
0.6232
0.7800
0.7812
233
0.5997
0.5974
0.7786
0.7812
1234
0.6156
0.6103
0.7786
0.7824
5678
0.6065
0.6012
0.7816
0.7853
0.6171
0.6073
0.7813
0.7832
Avg Std
0.6074
0.6079
0.7800
0.7827
0.0001
0.0001
0.0000
0.0000
Table 12: Comparison of SageBwd and BF16 performance on MMLU and HellaSwag across different seeds Qwen2.5 (3B).
MMLU
HellaSwag
Seed
SageBwd
BF16
SageBwd
BF16
42
0.6434
0.6425
0.9419
0.9402
233
0.6431
0.6437
0.9405
0.9402
1234
0.6492
0.6492
0.9414
0.9429
5678
0.6531
0.6400
0.9430
0.9440
0.6510
0.6454
0.9446
0.9434
Avg
0.6480
0.6442
0.9423
0.9421
Std
0.0000
0.0000
0.0000
0.0000
20

---

Table 13: Comparison of SageBud and BF16 performance on GSM8K and DROP across different seeds on Llama3. 2 (IB).
GSMSK
DROP
Seed
SageBwd
BF16
SageBwd
BF16
42
0.2722
0.2547
0.6367
0.6447
233
0.2661
0.2570
0.6456
0.6424
1234
0.2616
0.2873
0.6439
0.6352
5678
0.2684
0.2585
0.6372
0.6409
1
0.2646
0.2335
0.6393
0.6441
Avg
0.2666
0.2582
0.6405
0.6414
Std
0.0000
0.0003
0.0000
0.0000
Table 14: Comparison of SageBwd and BF16 performance on MMLU and HellaSwag across different seeds on Ll Llama3.2 (3B).
MMLU
HellaSwag
Seed
SageBwd
BF16
SageBwd
BF16
42
0.4665
0.4705
0.8230
0,8319
233
0.4646
0.4560
0.8327
0.8256
1234
0.4702
0.4757
0.8202
0.8243
5678
0.4580
0.4639
0.8232
0.8276
1
0.4666
0.4691
0.8218
0.8236
Avg
0.4652
0.4670
0.8242
0.8266
Std
0.0000
0.0000
0.0000
0.0000
A.5 Transposing V.
Performing the forward propagation of attention in full FP4 precision poses additional challenges compared to FP16. The input tensors Q, K, and V typically contiguous in the head dimensions. However, the row-major constraints on FP4 MMA for the second GEMM necessitate V to be contiguous in the sequence length dimension. Calling standalone pre-processing transpose kernel for this purpose incurs excessive overhead, particularly during inference, which is often a memory- bound situation. We address the problem by kernel fusion. For the first problem, fuse the transpose of V into the quantization kernel, thereby avoiding additional I/O overhead.
A.6 Acemulated Quantization Error Analysis.
Table 15: Layer-wise LI error analysis of SageAttention3 on CogVideoX-20 The second row shows the results by retaining the three most sensitive layers in FP16.
Method
Layer1 J
Layer10 4 Layer20
Layer30
Use SageAttention? directly Keep most sensitive layers in FP16
0.0076
0.0922
0.1
0.0571
0.0076
0.0447
0.0773
0.0429
To explore the issue of accumulated quantization error across layers, we conduct an analysis using SageAttent on3 CogVideoX-2B 2B and report the per-layer LI error in Table [5] We observe that the accumulated error generally increases with layer depth, though it occasionally decreases in deeper layers, suggesting partial error cancellation. To mitigate this drift, we apply a simple yet effective strategy: keeping the three layers with the largest observed error growth in FP16 precision. As shown in the table, this adjustment significantly reduces the overall error accumulation across layers.
21

---

A.7 Ablation of Smoothing Techniques.
Table 16: Ablation of attention accuracy with different smoothing methods on CogVideoX-2B.
Smoothing and Smoothing Q are techniques from SageAttention and SageAttention2.
Method
Cossim
L1 Error
RMSE !
None
0.915642
0.335867
0.303483
SmoothQuant
0,930125
0.267617
0.252883
Hadamard
0,941222
0,262047
0.223970
Smoothing Q
0.982848
0.115658
0,125862
Smoothing K
0.991176
0.094832
0.097668
To investigate the impact of different smoothing strategies on attention accuracy, we compare several existing techniques, including SmoothQuant [63] and Hadamard transformations, which provide per-token or per-tensor scaling control. However, we find these methods less effective our setting. SageAttention3 inherits the smoothing Q and smoothing K mechanisms introduced in SageAttentions ion..... We conduct an ablation study on all layers of CogVideoX 2B to evaluate their effects. As shown in Table T6 both smoothing Q and smoothing K yield substantially higher cosine similarity and lower reconstruction errors, demonstrating their effectiveness in stabilizing quantized attention computation.
A.8 Theoretical Speed Comparison.
Table 17: Theoretical throughput comparison between FlashAttention3 and SageAttention3 across
different GPUs
Method
B300 TOPS
B200 TOPS 1 RTX5090 TOPS f
FlashAttention3
2500
2500
209.5
FlashAttmminn3 (FP8)
5000
5000
419
SageAttention3 (FP4)
15000
10000
1676
To provide a theoretical comparison with FlashAttention3, we refer to NVIDIA's official documen- tation on throughput (TOPS) across different precisions. Since FlashAttention3 is currently only supported on H100 GPUs. direct empirical comparison is not feasible. Instead, we estimate the theoretical compute throughput of both FlashAttention3 and our SageAttention3 on GPUs that sup- port FP4 Tensor Cores (B300, B200, and RTX5090). As summarized in Table SageAttentions achieves substantially higher theoretical peak throughput, highlighting its potential for further accel- erating attention computation beyond FlashAttention-3.
A.9
FlashAttentions SageAttentions
Table 18: Speed accuracy trade-off of different attention methods.
Method
TOPS on 5090
TOPS on H100 1
Accuracy (CosSim)
FlashAutention?
214
338
100.000%
FlashAttention3 (16bit)
N/A
470
100.000%
FlashAutention3 (Sbit)
N/A
890
98.570%
Sage SageAttention1
479
518
99. 99.996%
SageAttention2 (8bit)
643
885
99.995%
SageAttentions (4bit)
1038
N/A
99.551
To illustrate the trade-off between accuracy and speed, we recorded the accuracy (Cosine similarity) of various attention methods across all layers of CogVideoX-222 along with their theoretical throughput on RTX5090 and H100 GPUs. These results summarized in the Table [8
22

---

A.10 Analysis of Two-Level Quantization.
Proof. We analyze the relative quantization error of P using both direct quantization and two-level quantization as follows:
For direct quantization, the relative quantization error, denoted as Ej is defined as:
Sp x P P
Sp, P ((P), E
(9)
PP
For two-level quantization, the first level proceeds as:
Sp (owmax(P)/(448) x 6), P2 P2=P/sp,.
The quantization emror introduced in this first step is negligible because P, P2, and 8p, all represented in FP32 format.
We focus primarily on the second-level quantization, where the relative quantization error Eg is given by:
Sp, x P P2
SP2: = o(P2). Eg
(10)
P2
The key difference between Equation 9 and
10 lies the range of the FP8 scale factor.
Let {X), denote the number of distinct representable values in the set X. Then:
in direct quantization:
Sp - 0.167, Sp E E4M3, {sp)n = 35
In two-level quantization:
0 SP, 448.0, SP, E4M3, {SP2)n 127
Since (E2MI), Jn = 8, the number of unique outputs after dequantization is: For direct quantization:
P'=sp x P. (P') 35 x 8 280
For two-level quantization:
P²= P2, (P2).
127 1016
Let (pp:) denote the interval between the two nearest quantization levels surrounding the value
E P. Then the absolute quantization error satisfies:
(p:)
1 Pil
2
The relative enror satisfies:
App))
8; <
2 X
Given that (P))a > (P'). the quantization intervals in the two-level scheme are finer:
A(P2)
A(P
P2
P
Thus, the relative quantization error satisfies:
P
P
P'-
P
P
Which leads the conclusion:
Eg < Ei
23

---

A.11 Analysis of the Benefit of Keeping dO, 3 in FP16 in SageBud.
The backward pass of SageBwd involves 5 MatMuls. The accuracy of Sij Q.K. fully addressed in SageAttentiion.. The remaining four are as follows:
(I) dP,; = d.,
(2) dQ, dQ; + K,
(3) dKj + dK, + ds Q:
(4) dV; < dV, P doj
We choose to keep (1) in FP16, while quantizing others INT8. This choice can be formally justified:
Proof. Following 64 we assume that any matrix X E Raad (e.g. Q. V, dO) satisfies:
The entries in X are mutually independent.
Xij ~ N x,,)) i.e. the distribution of each token identical.
The quantization error of matrix X denoted as:
AX = xx X,
where 8x, = (XX).
For example, consider the error in dQ. Neglecting second-order error terms, have:
AdQ (P (dOAvT AdoV AdSK dSAK.
AdQ ( from (1)
dQ
Here, dS
Po (dOVT D), where D d0 @ O. In element-wise terms (the
subscript denotes single element):
dS = Pij Σ
doa -Oa) = P Σ
dOa
j
Since V is independent of other variables, by linearity of expectation:
] =0.
ds, = E Pij Σ dOik
(Vv Σ Piepiv
Moreover, as negating V flips the sign of ds.y. the PDF of dS symmetric. Using "round-to- nearest" quantization policy, we have E[AdS) = 0. Thus
E
dQ (2)
AdSSS
dSAK 0,
while E. (dQ is generally non-zero (e.g. when distributions have non-zero means), indicating that dQ's s error is dominated by AdQ(1)
A.12 Broader Impact
This paper presents work that aims to advance the field of efficient machine learning systems. It can be used to accelerate the inference and training processes of various models. None of the negative impacts we feel must be specifically highlighted here.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Both the Abstract and the Introduction indicate the scope and contributions of this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in Introduction and Conelusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
25

---

Justification:
We provide and check the proof for theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Result can be reproduced according to the sections of Method, Experiment, and Appendix A..3 sections.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer:
[Yes]
Justification:
Source code and instructions will be provided in supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Cooeeeoodeesssseeeeeee //nips.com htte:/uulllllliiiee/e//eeeuaaasl ssiniiiiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
Please refer to the Sections Experiment and Appendex A.3
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: As listed in the Appendix, for those experiments with large errors, we provide mean and standard deviation values of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
27

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the specific computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer: [Yes]
Justification:
The authors have reviewed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
28

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not release models, generators, or datasets with a high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please refer the Experiment Section and the Appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
29

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Please refer README.md in our following supplemental material submis-
sion.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
30

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttpp/////euips..... wwwpariipssccooooeeeeeee cc/Conferencess///////eeoccc/eee for what should or should be described.
31


Reviews:
[{"review_id": "J9F1U4N6WS", "reviewer": "Reviewer_VcmM", "summary": "The paper introduce SageAttention3, with two key contributions. First, they propose FP4 attention for inference and second, they design int8 backward pass for accelerated training. For FP4 inference, the authors propose microscaling combined with two-level scaling to better use FP8 scaling factors.", "questions": "1. How does the overall memory footprint of SageAttention3 compare to other existing works for long sequences?\n2. Is there any shared implementation showing efficiency gains for both FP4 inference and int8 training together?", "limitations": "Yes. The authors indirectly address their limitations in the future work section of their work.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Exploiting low-precision FP4 acceleration is a compelling direction.\n- Hardware-centric contribution including kernel-level optimizations\n- Results prove the efficacy of the proposed method\n\nWeaknesses:\n- The two proposed contributions appears disconnected. While both broadly target low-precision methods, they employ different approaches. No experiment show the combined impact of the two methods. I think the INT8 training part is less mature and can be explored as a separate work later.\n- There are limited experiments supporting efficiency claims for INT8 training.", "was_revised": true, "final_justification": "Considering authors' rebuttal and other reviewers' feedback, I increase my score.", "rebuttal": "Dear Reviewer VcmM,  \nThank you for your valuable suggestions and questions. Below, we address each point raised.\n\n---\n**W1.** The two proposed contributions appear disconnected. While both broadly target low-precision methods, they employ different approaches. No experiment show the combined impact of the two methods. I think the INT8 training part is less mature and can be explored as a separate work later.\n\n**Response:** Thank you for your insightful and important comment. Both contributions aim to improve attention efficiency through quantization, with FP4 inference focusing on accelerating inference and INT8 attention targeting training efficiency.\n\nTo better demonstrate the connection between these two approaches, we add an experiment that combines them: first applying INT8 attention during fine-tuning, followed by FP4 attention during inference. Specifically, we fine-tuned Qwen2.5 for 1,000 steps using either BF16 or our INT8 attention, and then evaluated inference performance using our FP4 attention. The results on GSM8k and MMLU are shown below, where INT8 fine-tuning followed by FP4 inference yields higher accuracy, indicating that the two methods can be complementary, and their combination can deliver additional benefits. This improvement is likely because INT8 and FP4 share a more similar representable data distribution, reducing the mismatch error compared to BF16.\n\n**Table 1: Comparison on Qwen2.5-1.5B.**\n\n| All using FP4 Inference | Gsm8k $\\uparrow$ | Mmlu $\\uparrow$ |\n| :-----------------: | :--------------: | :-------------: |\n| BF16 Fine-tuning | 0.4912 |    0.4688    |\n| INT8 Fine-tuning | **0.5232** |  **0.4934**  |\n\n**Table 2: Comparison on Qwen2.5-3B.**\n\n| All using FP4 Inference | Gsm8k $\\uparrow$ | Mmlu $\\uparrow$ |\n| :---------------------: | :--------------: | :-------------: |\n|   BF16 Fine-tuning    |     0.5860     |    0.6000     |\n|   INT8 Fine-tuning    |   **0.5945**   |  **0.6032**   |\n\n\n---\n**W2.** There are limited experiments supporting efficiency claims for INT8 training.\n\n**Response:** We would like to clarify that we have comprehensively evaluated the kernel efficiency and end-to-end efficiency in the original paper. First, we have assessed the attention kernel speed across different sequence lengths, head dimensions, and causal masking settings in the original paper. Specifically:  \n\n- Forward pass speed of SageBwd is reported in Figures 15-16 (Appendix in the original paper). \n- Backward pass speed is detailed in Figures 17-18 (Appendix in the original paper). \n- Overall speed (forward + backward) is presented in Figures 6-7 (Main Text in the original paper).\n\nSecond, we have evaluated and reported the end-to-end training efficiency gain of SageBwd in Table 4 in the original paper.\n\nAdditionally, we appreciate the feedback regarding the need for more detailed efficiency analysis of INT8 training. To address this, we conducted additional experiments measuring the training latency (ms) breakdown of key components during training, using Llama3.1-8B and Qwen3-8B with sequence lengths of 16K and 32K. We find that:\n\n- SageAttn3 achieves about 30% speedup in attention computation compared to the original implementation.  \n- End-to-end efficiency gains increase with sequence length, as attention latency becomes dominant with longer sequence lengths.  \n- Total training latency is reduced by 13–15% across both models. \n\n**Table 3: Training latency (ms) breakdown on Llama3.1-8B and Qwen3-8B.**\n\n|Model (SequenceLength)|Method|FFN|Attention $\\downarrow$|qkvo_projection|Others|Total $\\downarrow$|\n|---|---|---|---|---|---|---|\n|Llama3.1-8B (16K)|FlashAttn2|105.97|388.30|388.88|2.03|885.18|\n|Llama3.1-8B (16K)|SageBwd|106.19|**269.47**|388.63|2.03|**766.3**|\n|Llama3.1-8B (32K)|FlashAttn2|214.01|1501.6|1504.82|4.08|3,224.5|\n|Llama3.1-8B (32K)|SageBwd|215.5|**1,036.9**|1498.09|4.09|**2,754.5**|\n|Qwen3-8B (16K)|FlashAttn2|93.37|391.07|389.82|2.03|876.29|\n|Qwen3-8B (16K)|SageBwd|93.51|**270.96**|389.02|2.02|**755.51**|\n|Qwen3-8B (32K)|FlashAttn2|184.13|1502.2|1501.8|4.08|3,192.3|\n|Qwen3-8B (32K)|SageBwd|184.9|**1037.5**|1497.7|4.08|**2,724.2**|\n\n\n\n\n---\n**Q1.** How does the overall memory footprint of SageAttention3 compare to other existing works for long sequences?\n\n**Response:** The memory footprint of Sageattention3 is the same as existing comparable methods. Specially:\n\n(1) For compute-bounded tasks such as vision transformers or LLM prefilling, the comparable method is FlashAttention. The overall memory footprint of SageAttention3 is the size of the Q, K, and V of size $N\\times d$ in FP16, saming with FlashAttention.\n\n(2) For memory-bounded LLM decoding, the overall memory footprint of SageAttention3 is the size of the Q of size $1\\times d$ in FP4 and K, V of size $N\\times d$ in FP4. Similar to other 4-bit KV cache quantization works, when applied to memory-bounded LLM decoding, SageAttention can quantize the KV cache to FP4, reducing memory usage and I/O bandwidth by 75% compared to standard FP16/BF16. \n\nIn addition, we want to emphasize that **SageAttention3, similar to FlashAttention, is primarily designed for compute-bounded tasks such as vision transformers or LLM prefilling. In these scenarios, memory consumption remains minimal and does not impact attention latency.** Specifically:\n\n- **Memory usage:** SageAttention maintains the same memory footprint as FlashAttention, i.e., the size of the Q, K, and V matrices (**very small compared to the model parameters**). This is because the large intermediate matrices (e.g., the $N \\times N$ attention scores $S = QK^\\top$ and softmax matrix $P$) are never materialized in GPU memory, due to online softmax computation.  \n- **Latency Impact:** The memory required for Q, K, and V is typically negligible compared to model parameters. Moreover, because attention is compute-bound, the I/O overhead for these matrices will be covered by computation and does not affect latency.  \n\n\n\n\n---\n**Q2.** Is there any shared implementation showing efficiency gains for both FP4 inference and int8 training together?\n\n**Response:** There are indeed shared implementation techniques that benefit both FP4 inference and INT8 training:\n\n1. **Efficiency optimizations:** Both methods leverage (1) low-bit MMA PTX instructions to accelerate matrix multiplications, (2) eliminating the need for explicit max operations in the $P$ matrix by reusing both global and local maximum values from the online softmax computation (as introduced in Section 4.1), and (3) manipulating the memory layout of the two low-bit matrices involved in matrix multiplication to meet the requirements of the fastest MMA instruction.\n2. Accuracy improvements: Both methods adopt key smoothing, where $K = K - \\text{colmean}(K)$, to improve the numerical stability and precision of the attention computation.\n\n---\n\n**If you feel your concerns have been resolved, we would greatly appreciate it if you consider raising the score. We will add the experiment and analysis in the rebuttal to our paper.**"}, {"review_id": "r35P0Jtkzj", "reviewer": "Reviewer_DSm8", "summary": "This paper studies doing forward and backward propagation for Transformer attention efficiently with low-bit precision. It proposes per-block 4-bit microscaling and a two-stage softmax quantization to compress both the query-key and attention-value matmuls. Experiments show that SageAttention3 runs 5× faster than FlashAttention while preserving end-to-end quality metrics across diverse models and SageBwd achieves lossless performance in fine-tuning\ntasks, though full-scale pre-training still faces challenges.", "questions": "1. Accumulated quantization drift across layers? Small errors per layer can compound over dozens of layers, leading to distributional shift and degraded convergence. Some discussion on analysis or mitigation to control this drift will be appreciated. \n2. The max-based scale factors seem vulnerable to outliers (sec 3.1). A single extreme outlier in a block dominates the scale and shrinks all other values toward zero. I was wondering if the authors considered any mechanism to mitigate the potential sensitivity to outliers?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**: \n\nThe proposed method introduces a principled two-level quantization and selective FP16 retention that balance accuracy and speed and leverages FP4 tensor cores with triton kernels. Validates both inference and training across diverse Transformer architectures and tasks, showing consistent efficiency improvements.\n\n**Weaknesses**:\n\n1. The optimization strategies proposed by this paper seem to be quite hardware specific, e.g., k-permutation tuned to NVIDIA Blackwell’s FP4 tensor cores. The optimizations may not map directly to other GPUs (e.g. AMD or older NVIDIA), or to TPUs.\n2. The proposed pipeline assumes unmasked rows. Real Transformer use-cases often require causal masks (e.g., autoregressive language modeling). More discussions on how to integrate mask logic into the pipeline will be appreciated. \n3. Heuristic INT8/FP16 backward mix: the proposed method only quantizes five of six backward matmuls to INT8, but keep the remaining one FP16. This choice is presented as empirical, with no systematic analysis of which matmuls are safe to quantize or why exactly dOV^T must remain in FP16. More discussions and insights into this will make the paper stronger.", "was_revised": true, "final_justification": "Post-rebuttal final justification: Thank you for your in-depth rebuttal. I appreciate the discussion and analysis on masking, quantization design choices, error drift, and smoothing methods. I will maintain the original score.", "rebuttal": "Dear Reviewer DSm8,  \nThank you for your valuable suggestions and questions. Below, we address each point raised.\n\n---\n**W1.** The optimization strategies proposed by this paper seem to be quite hardware specific, e.g., k-permutation tuned to NVIDIA Blackwell’s FP4 tensor cores. The optimizations may not map directly to other GPUs (e.g. AMD or older NVIDIA), or to TPUs.\n\n**Response:** We'd like to clarify three aspects:\n\n1. **Generality of Core Methods:**  The main techniques we propose, such as low-bit matrix quantization for faster computation, two-level quantization, removing explicit max operations for per-token quantization of the $P$ matrix, and outlier smoothing—are algorithmically general and can be adapted to different hardware platforms. \n2. **Hardware-Specific Optimizations as a Necessity:**  Achieving peak FLOPS on cutting-edge systems inevitably requires some hardware-aware optimizations. For example, FlashAttention-3 (compared to FlashAttention-2) extensively exploits Hopper-specific features such as Tensor Core asynchrony and Tensor Memory Accelerators (TMA).\n3. **Adaptable Principles:** While certain implementations (e.g., k-permutation) are architecture-dependent, the underlying principles (exploiting the tensor layout of low-bit MMA instructions) remain relevant across platforms, guiding adaptations for other hardware. \n\n---\n**W2.** The proposed pipeline assumes unmasked rows. Real Transformer use-cases often require causal masks (e.g., autoregressive language modeling). More discussions on how to integrate mask logic into the pipeline will be appreciated.\n\n**Response:** SageAttention3 already supports efficient causal masking, as evidenced by the attention speeds reported in Figures **4-5** in the original paper. We are sorry that we omitted the implementation details in our paper (like FlashAttention). The solution is: \n\n1. **Block Skipping**: Q/K blocks entirely masked by causality are excluded from memory loads. \n2. **Partial Mask Handling**: For partially masked blocks loaded in the attention kernel, we apply $-\\infty$ to the corresponding positions in the $P$ matrix to enforce causal masking. \n\nTo validate effectiveness, we add an experiment that compares the end-to-end metrics and attention speed on Llama3.2-1B model, confirming that SageAttention3 maintains accuracy and enhances the efficiency in tasks with causal masks:\n\n**Table 1: FP4 inference result on an autoregressive language model.**\n\n|Method|wikitext PPL $\\downarrow$|LAMBDA $\\uparrow$|Prefilling(64K) Latency $\\downarrow$|\n|-|-:|-:|-:|\n|BF16 Inference|9.29854|0.741|10.2145|\n|SageAttn3 Inference| 9.56827|0.726|6.09898|\n\n---\n**W3.** Heuristic INT8/FP16 backward mix: the proposed method only quantizes five of six backward matmuls to INT8, but keep the remaining one FP16. This choice is presented as empirical, with no systematic analysis of which matmuls are safe to quantize or why exactly dOV^T must remain in FP16. More discussions and insights into this will make the paper stronger.\n\n**Response:** Thank you for the insightful question. We conduct an experiment and provide a theoretical analysis to explain why dOV^T remains in FP16.\n\n**Experiment**: The backward pass involves 5 MatMuls. ${\\bf S}\\_{ij} \\gets {\\bf Q}_i\\mathbf {\\bf K}_j^\\top$, whose accuracy is fully addressed in SageAttention2. The remaining four are as follows:\n\n(1) ${\\bf dP}\\_{ij}\\gets {\\bf dO}_i {\\bf V}_j^\\top$\n\n(2) ${\\bf dQ}_i\\gets{\\bf dQ}_i+{\\bf dS}\\_{ij}{\\bf K}_j$\n\n(3) ${\\bf dK}_j\\gets{\\bf dK}_j+{\\bf dS}\\_{ij}^\\top {\\bf Q}_i$\n\n(4) ${\\bf dV}_j\\gets{\\bf dV}_j+{\\bf P}\\_{ij}^\\top {\\bf dO}_i$\n\nAmong these, the last three matmuls directly add their results to the final output. However, **(1)** involves further computation:\n$$\n{\\bf dS}\\_{ij}\\gets {\\bf P}\\_{ij}\\circ ({\\bf dP}\\_{ij}-{\\bf D}_i),\n$$\n\nwhere ${\\bf dS}\\_{ij}$ subsequently contributes to both ${\\bf dQ}_i$ and ${\\bf dK}_j$ in steps (2) and (3). Since the output of ${\\bf dO}_i {\\bf V}_j^\\top$ is used in downstream matmuls, it is reasonable to keep this computation in higher precision (FP16).\n\nEmpirical data support this conclusion. Using tensors from all layers of CogVideoX-2B, we evaluate the accuracy by quantizing each Matmul (\"-\" indicates the calculation in that row is not related to the Matmul in that column.):\n\n**Table 2: Attention gradient accuracy ablation.**\n\n|Matmul not quantized|(1)|(2)|(3)|(4)|None|\n|-|-|-|-|-|-|\n| $\\bf dQ$'s Cossim $\\uparrow$|**0.9202**|0.8152|-|-|0.8125 |\n| $\\bf dQ$'s L1 Error $\\downarrow$|**0.3308**|0.6326|-|-|0.6399 |\n| $\\bf dQ$'s RMSE Error $\\downarrow$|**5.0905**|8.5137|-|-|8.5827 |\n| $\\bf dK$'s Cossim $\\uparrow$|**0.9858**|-|0.9839|-|0.9839 |\n| $\\bf dK$'s L1 Error $\\downarrow$|**0.1826**|-|0.2003|-|0.2014 |\n| $\\bf dK$'s RMSE Error $\\downarrow$|**6.3992**|-|6.8168|-|6.8180 |\n| $\\bf dV$'s Cossim $\\uparrow$|-|-|-|**0.9982**|0.9981 |\n| $\\bf dV$'s L1 Error $\\downarrow$|-|-|-|**0.0901**|0.1220 |\n| $\\bf dV$'s RMSE Error $\\downarrow$|-|-|-|**1.1519**|1.2113 |\n\nWe can see that keeping matmul (1) in FP16 leads to significant precision improvements.\n\n**Theoretical Analysis.** Following [1], assume that any matrix ${\\bf X}\\in\\mathbb R^{n\\times d}$ (e.g. ${\\bf Q},{\\bf K},{\\bf V},{\\bf dO}$) satisfies:\n\n- The entries in ${\\bf X}$ are mutually independent.\n- ${\\bf X}\\_{ij} \\sim N(\\mu_{{\\bf X},j},\\sigma_{{\\bf X},j}^2)$, i.e. the distribution of each token is identical.\n\nThe quantization error of a matrix ${\\bf X}$ is denoted as:\n$$\n\\Delta{\\bf X} := s_{\\bf X}\\hat{\\bf X}-{\\bf X},\\quad \\text{where } s_{\\bf X},\\hat{\\bf X}=\\psi({\\bf X}).\n$$\n\nFor example, consider the error in ${\\bf dQ}$. Neglecting second-order error terms, we have:\n$$\n\\Delta{\\bf dQ}=\\underbrace{({\\bf P}\\circ({\\bf dO}\\Delta{\\bf V}^\\top+\\Delta {\\bf dO}{\\bf V}^\\top)){\\bf K}}_{\\Delta {\\bf dQ}^{(1)}\\text{ from (1)}}+\\underbrace{\\Delta{\\bf dS}{\\bf K}+{\\bf dS}\\Delta{\\bf K}}\\_{\\Delta{\\bf dQ}^{(2)}\\text{ from (2)}}\n$$\n\nHere, ${\\bf dS}={\\bf P} \\circ ({\\bf dP}-{\\bf D})={\\bf P} \\circ ({\\bf dO} {\\bf V}^\\top-{\\bf D})$, where ${\\bf D}={\\bf dO}\\odot {\\bf O}$. In element-wise terms:\n$$\n{\\bf dS}\\_{ij}={\\bf P}\\_{ij} \\sum_{k} {\\bf dO}\\_{ik} ({\\bf V}\\_{jk}-{\\bf O}\\_{ik})={\\bf P}\\_{ij} \\sum_k {\\bf dO}\\_{ik} \\left({\\bf V}\\_{jk}-\\sum_l {\\bf P}\\_{il} {\\bf V}\\_{lk}\\right).\n$$\n\nSince ${\\bf V}$ is independent from other variables, by linearity of expectation:\n$$\n\\mathbb E[{\\bf dS}\\_{ij}]=\\mathbb E\\left[{\\bf P}\\_{ij} \\sum_k {\\bf dO}\\_{ik} \\left(\\mu_{{\\bf V},k}-\\sum_l {\\bf P}\\_{il} \\mu_{{\\bf V},k}\\right)\\right]=0.\n$$\n\nMoreover, as negating ${\\bf V}$ flips the sign of ${\\bf dS}\\_{ij}$, the PDF of ${\\bf dS}\\_{ij}$ is symmetric. Using a \"round-to-nearest\" quantization policy, we have $\\mathbb{E}[\\Delta {\\bf dS}]=0$. Thus\n$$\n\\mathbb E\\left[\\Delta{\\bf dQ}^{(2)}\\right]=\\mathbb E\\left[\\Delta {\\bf dS} {\\bf K}+{\\bf dS} \\Delta {\\bf K}\\right]=0,\n$$\n\nWhile $\\mathbb{E}\\left[\\Delta {\\bf dQ}^{(1)}\\right]$ is generally non-zero (e.g., when distributions have non-zero means), indicating that ${\\bf dQ}$'s error is dominated by $\\Delta {\\bf dQ}^{(1)}$.\n\n[1] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration\n\n---\n**Q1.** Accumulated quantization drift across layers? Small errors per layer can compound over dozens of layers, leading to distributional shift and degraded convergence. Some discussion on analysis or mitigation to control this drift will be appreciated.\n\n**Response:** Thank you for raising this insightful point. We agree that quantization errors can accumulate across layers. To investigate this,\n\n1. We conduct an experiment using SageAttention3 on CogVideoX-2B and report the L1 error for each layer, as shown in the Table below. We observe that, in general, the accumulated error tends to increase with depth; however, it occasionally decreases in certain deeper layers, suggesting that some degree of error cancellation may occur. \n2. To mitigate this drift, we try a simple approach: we set the three layers exhibiting the most significant error growth to use FP16 attention. As shown in the Table below, this simple strategy effectively reduces the overall error accumulation.\n\n**Table 3: Accumulated L1 error of layers.**\n\n||Layer1 $\\downarrow$|Layer10 $\\downarrow$|Layer20 $\\downarrow$|Layer30 $\\downarrow$|\n|-|-|-|-|-|\n|Use SageAttn3 directly|0.0076|0.0922|0.1146|0.0571|\n|Keep 3 most sensitive layers to FP16|0.0076|**0.0447**|**0.0773**|**0.0429**|\n\n---\n**Q2.** The max-based scale factors seem vulnerable to outliers (sec 3.1). A single extreme outlier in a block dominates the scale and shrinks all other values toward zero. I was wondering if the authors considered any mechanism to mitigate the potential sensitivity to outliers?\n\n**Response:** We agree with your observation; however, fine-grained control of every 1×16 outlier is challenging. Unlike quantization designed for model compression, our quantization is aimed at accelerating matrix multiplication, which requires performing the computation entirely in low precision while supporting direct dequantization. This constraint limits outlier control to either per-token or per-tensor granularity. Although approaches such as SmoothQuant and Hadamard transformations enable per-token or per-tensor control, we find them ineffective in our setting. SageAttention3 adopts the smoothing K and smoothing Q introduced in SageAttention2. To ablate these choices, we conduct ablation experiment on all layers of CogVideoX-2B, and the results are as below:\n\n**Table 4: Ablation of attention accuracy with different smooth methods.**\n\n|Method|Cossim $\\uparrow$|L1 Error $\\downarrow$|RMSE $\\downarrow$|\n|-|-|-|-|\n|None|0.915642|0.33586|0.30348|\n|Smooth_Quant|0.930125|0.26761|0.25288|\n|Hadamard|0.941222|0.26207|0.22397|\n|Smoothing_Q|**0.982848**|**0.11565**|**0.12586**|\n|Smoothing_K|**0.991176**|**0.09483**|**0.09766**|\n\n---\n\n**If you feel your concerns have been resolved, we would greatly appreciate it if you consider raising the score. We will add the experiment and analysis in the rebuttal to our paper.**"}, {"review_id": "YBFV8Da4Eq", "reviewer": "Reviewer_Czmu", "summary": "The paper proposes a new low-precision attention model for LLM inference and training. A 4-bit floating point format (NVFP4) with a scaling factor is used for most of the attention parameters. To better match the dynamic range of all parameters when FP4 is used for the P parameter, two-level scaling is applied. Moreover, low-precision attention is demonstrated for the backward pass, using int8 integers for most matrix multiplications. This approach has been shown to be efficient for fine-tuning, but not suitable for training LLMs from scratch to design new pre-trained models. The new approach, combined with hardware optimizations such as fusing quantization with online softmax, results in a speedup over previous FlashAttention and Sliced Attention models.", "questions": "1- How is the value 480*6 selected for the range of P in first-level quantization?\n\n2- To better understand the speedup compared to other models, it is suggested to present a trade-off plot between speed and performance degradation for various experiments.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n1. The paper is well-written and well-organized.\n2. The int8 quantization for fine-tuning LLMs is interesting and novel.\n\nWeaknesses:\n\n1- The author is required to compare the new approach with previous work on FP4 and INT4 quantization of attention, such as [1, 2, 3, 4]. Comparisons for inference are not presented.\n[1] Lin, Yujun, et al. \"Qserve: W4a8kv4 quantization and system co-design for efficient llm serving.\" arXiv preprint arXiv:2405.04532 (2024). \n\n[2] Liu, Lian, et al. \"COMET: Towards Practical W4A4KV4 LLMs Serving.\" Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 2025.\n\n[3] Wang, Ruizhe, et al. \"Optimizing Large Language Model Training Using FP4 Quantization.\" arXiv preprint arXiv:2501.17116 (2025).\n\n[4] Frantar, Elias, et al. \"Gptq: Accurate post-training quantization for generative pre-trained transformers.\" arXiv preprint arXiv:2210.17323 (2022).\n\n2- Comparison with FlashAttention-3 is not presented. This comparison can be performed theoretically rather than by running on GPU.\n\n3- The author is suggested to explain the reason behind the significant accuracy degradations in FScore for Hunyuan Video and Mochi when comparing Sege Attention3 (4-bit) vs. Sege Attention2 (8-bit).", "was_revised": true, "final_justification": "The authors provide a comparison of the new approach with the previous FP4 design and also offer an analytical comparison with FlashAttention-3. In both cases, the new approach demonstrates better performance. As a result, I raised my score.", "rebuttal": "Dear Reviewer Czmu,  \nThank you for your valuable suggestions and questions. Below, we address each point raised.\n\n---\n**W1.** The author is required to compare the new approach with previous work on FP4 and INT4 quantization of attention, such as [1, 2, 3, 4]. Comparisons for inference are not presented. \n\n**Response:** Strictly speaking, the methods mentioned above are not directly comparable, similar to how FP8-quantized FlashAttention3 is not compared against KV-cache-only quantization methods. To be specific, these methods do not quantize $Q$, $K$, and $V$ entirely to 4-bit, nor do they perform attention computations at 4-bit. Our work is the first to leverage hardware FP4 Tensor Core to perform the entire attention computation in FP4, accelerating the attention computation beyond just reducing memory usage. In contrast, existing methods still execute attention in FP16/BF16 and focus primarily on reducing the KV-cache memory, which can save GPU memory but does not improve performance for compute-bound workloads such as ViTs or LLM pre-filling.\n\nNevertheless, to provide a more complete evaluation, we add two sets of comparison experiments. First, we compare SageAttention3 with QServe and Atom (please note that GPTQ is irrelevant here as it does not modify attention) on Qwen2.5-1B. As shown in the Table below, SageAttention3 achieves the best accuracy while also providing the fastest pre-filling speed. For decoding speed, because Atom and QServe cannot be compiled on the RTX5090, we report their theoretical decoding speedup.\n\n**Table 1: Comparison on Qwen2.5.**\n\n| Method |wikitext PPL $\\downarrow$|LAMBDA $\\uparrow$|Prefilling(64K) Latency $\\downarrow$|Theoretical decoding speedup $\\uparrow$ |\n| ---------|------------------------:|----------------:|-----------------------------------:|--------------------------------------: |\n| Qserve | 10.90296 | 0.631| 10.20257 | 4x |\n| Atom | 10.89669 | 0.631| 10.22090 | 4x |\n| SageAttn3| **9.56827** | **0.726** | **6.0989s** | 4x |\n\nSecond, we compare SageAttention3 and QServe on CogVideoX-2B. As shown in the Table below, SageAttention3 not only achieves higher accuracy but also provides substantial speedup.\n\n**Table 2: Comparison on CogVideoX.**\n\n| Method |CLIPSIM ↑|VQA-a ↑|VQA-t ↑|FScore ↑|Latency $\\downarrow$ |\n| ---------|--------:|------:|------:|-------:|-------------------: |\n| Qserve | 0.1683|19.7863|49.4831| 2.4488| 65s |\n| Atom | 0.1706|20.0896|46.9320| 2.4024| 64s |\n| SageAttn3| **0.1881** |**69.8600**|**70.3640**| **4.0350** | **27s** |\n\n\n\n\n---\n**W2.** Comparison with FlashAttention-3 is not presented. This comparison can be performed theoretically rather than by running on GPU.\n\n**Response:** Thank you for suggesting a theoretical comparison with FlashAttention3. Indeed, a direct empirical comparison is not feasible because FlashAttention-3 is only supported on H100 GPUs. Instead, we provide a theoretical comparison based on the Tensor Core TOPS for different precisions as reported in NVIDIA’s official documentation. Using these hardware specifications, we estimated the theoretical compute speed (TOPS) of FlashAttention-3 and our SageAttention3 on GPUs that support FP4 Tensor Cores (B300, B200, and RTX 5090). The results are summarized below:\n\n**Table 3: Theoretical compute speed (TOPS) comparison.**\n\n|Method|B300 TOPS $\\uparrow$|B200 TOPS $\\uparrow$|RTX5090 TOPS $\\uparrow$|\n|---|---|---|---|\n|FlashAttn3|2500|2500|209.5|\n|FlashAttn3 (FP8)|5000|5000|419|\n|SageAttn3 (FP4)|**15000**|**10000**|**1676**|\n\nThese theoretical estimates show that SageAttention3 achieves significantly higher peak throughput on supported hardware, highlighting its potential for accelerating attention beyond FlashAttention-3.\n\n\n\n\n---\n**W3.** The author is suggested to explain the reason behind the significant accuracy degradations in FScore for Hunyuan Video and Mochi when comparing SageAttention3 (4-bit) vs. SageAttention2 (8-bit).\n\n**Response:** Thank you for the valuable suggestion. The drop in FScore can be explained rather straightforwardly: FScore is highly sensitive to video quality degradation, and SageAttention3 (4-bit) inherently has lower numerical accuracy than SageAttention2 (8-bit). To better understand this effect, we performed an ablation study on the quantization precision of the $Q, K, P, V$ matrices and reported results across all layers of CogVideoX-2B, as shown in the table below. These results show that quantizing all $Q, K, P, V$ matrices to FP4 (SageAttention3) leads to noticeably higher errors compared to SageAttention2 (8-bit), which uses INT8 for $Q, K$ and FP8 for $P, V$. Lastly, we want to emphasize that although SageAttn3 slightly reduces video quality (visually negligible), it significantly boosts generation speed.\n\n**Table 4: Ablation of attention accuracy when Q K P V in different precisions.**\n\n| Method |Cossim $\\uparrow$|L1 Error $\\downarrow$|RMSE $\\downarrow$ |\n| --------------------------------------------|-----------------|---------------------|----------------- |\n| $Q,K$ in FP4 |0.997848 |0.03883 |0.04104 |\n| $P,V$ in FP4 |0.998853 |0.04044 |0.03976 |\n| $Q,K$ in INT8 |0.999958 |0.00778 |0.00986 |\n| $P,V$ in INT8 |0.999953 |0.00813 |0.01510 |\n| $Q,K,P,V$ in FP4 (SageAttn3) |**0.99551** |**0.06923** |**0.07748** |\n| $Q,K$ in INT8; $P,V$ in FP8 (SageAttn2_8bit)|**0.99995** |**0.00849** |**0.00915** |\n\n\n\n---\n**Q1.** How is the value 448*6 selected for the range of P in first-level quantization?\n\n**Response:** In standard FP4 quantization, the scale factor is typically defined as $S_{\\bf P}=\\frac{\\mathrm{max}(|\\bf P|)}{6}$, where 6 corresponds to the maximum representable value in E2M1 (FP4). However, since the values in $\\bf P$ lie in the range $(0,1]$, the resulting the range of scale factor in $[0, 1/6]$, which is significantly smaller than the range of scale factor in E4M3 (FP8), which is $[-448, 448]$.\n\nTo better utilize the FP8 range, we propose first to quantize $\\bf P$ to a bigger range:\n\n$$P=\\frac{\\bf P}{S_{\\bf P_1}}, ~~~ S_{P_1}=\\frac{\\mathrm{rowmax}(\\bf P)}{448 \\times 6}$$\n\nThis ensures that the range of $\\bf P$ is in $[0, 448 \\times 6]$, enabling the second-stage FP4 quantization to use a scale factor of: \n$$S_{\\bf P}=\\frac{\\mathrm{max}(\\bf P)}{6} \\in [0, 448]$$\n\nwhich better leverages the representational range of the E4M3 (FP8) format.\n\nIn summary, 448×6 is based on the fact that the maximum representable values for E4M3 and NVFP4 are 448 and 6, respectively.\n\n\n\n---\n**Q2.** To better understand the speedup compared to other models, it is suggested to present a trade-off plot between speed and performance degradation for various experiments.\n\n**Response:** Thank you for the suggestion. To illustrate the trade-off between accuracy and speed, we recorded the accuracy (CosSim) of various attention methods across all layers of CogVideoX-2B, along with their theoretical throughput on RTX 5090 and H100 GPUs. These results are summarized in the table below. We will update our manuscript to include this table and the corresponding trade-off plot.\n\n**Table 5: Speed and accuracy trade-off of different methods.**\n\n| Method |TOPS on 5090 $\\uparrow$|TOPS on H100 $\\uparrow$|Accuracy $\\uparrow$ |\n| -----------------------|-----------------------|-----------------------|------------------- |\n| FlashAttention2 |214 |338 |100.000% |\n| FlashAttention3 (16bit)|N/A |470 |100.000% |\n| SageAttention1 |479 |518 |99.996% |\n| SageAttention2 (8bit) |643 |885 |99.995% |\n| FlashAttention3 (8bit)|N/A |890 |98.570% |\n| SageAttention3 (4bit) |1038 |N/A |99.551% |\n\n\n\n---\n\n**If you feel your concerns have been resolved, we would greatly appreciate it if you consider raising the score. We will add the experiment and analysis in the rebuttal to our paper.**"}, {"review_id": "pLC0jqSsDa", "reviewer": "Reviewer_wcep", "summary": "This paper proposes SageAttention3, which introduces two major advances for efficient attention inference and training. First, it uses a new NVFP4 attention approach that leverages FP4 Tensor Cores on NVIDIA Blackwell GPUs for better acceleration. The method uses a two-level scaling scheme to quantize the attention map,  fully utilizing the presentative range of the FP8 scaling factor. Combined with the 1×16 group-wise microscaling NVFP4 quantization, it achieves up to 5× speedup over FlashAttention2 on RTX5090 GPUs, with no noticeable drop across large language, image, and video generation models. Second, this paper proposes SageBwd, an 8-bit trainable attention that can accelerate both the forward and backward for training. Experiments show that it can achieve lossless performance in fine-tuning tasks, but with slower convergence in pretraining. Together, these features significantly improve the efficiency of model inference and training on modern hardware.", "questions": "- For the 8-bit attention for training, why int8 and not fp8? FP8 training is already widely used in LLM training, e.g., DeepSeek-V3. If the attention operator could also use FP8 for training, it could further boost the model training efficiency, with almost all of the computation being performed in 8-bit.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- Strengths\n\t- This paper is well-written and easy to follow.\n\t- The proposed method combines new hardware features and proposes NVFP4 attention, SageAttention3, for better model efficiency. SageAttention3 can achieve impressive speedups across different model architectures and benchmarks.\n\t- Experiments show real speedups on GPUs, demonstrating the effectiveness of the proposed method.\n- Weakness\n\t- The two contributions, nvfp4 microscaling quantization for inference efficiency and int8 quantization for training efficiency, are not linked. It would be better to have another session to show how to fine-tune models with int8 and then convert them to NVFP4 for better inference efficiency. In this way, those two can be tied together.\n\t- There are some other 4-bit quantization methods like QServe[1], Atom[2], and QuaRot[3]. While they don't support 4-bit computation in attention, 4-bit memory access can still boost model performance, especially for memory-bounded scenarios. So it would be good to compare SageAttention3 with other 4-bit quantization methods to demonstrate the advantages of SageAttention3.\n\n[1] Yujun Lin, et al. QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving. MLSys, 2025.\n\n[2] Yilong Zhao, et al. Atom: Low-bit Quantization for Efficient and Accurate LLM Serving. MLSys, 2024.\n\n[3] Saleh Ashkboos, et al. QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs. NeurIPS, 2024.", "was_revised": true, "final_justification": "Most of my questions have been addressed, so I raised my rating.", "rebuttal": "Dear Reviewer wcep,  \nThank you for your valuable suggestions and questions. Below, we address each point raised.\n\n---\n**W1.** The two contributions, nvfp4 microscaling quantization for inference efficiency and int8 quantization for training efficiency, are not linked. It would be better to have another session to show how to fine-tune models with int8 and then convert them to NVFP4 for better inference efficiency. In this way, those two can be tied together.\n\n**Response:** Thank you for your insightful and important suggestion. We are very willing to add a Section to show how to fine-tune models with int8 and then convert them to NVFP4 for better efficiency and accuracy. We add an experiment that combines them: first applying INT8 attention during fine-tuning, followed by FP4 attention during inference. Specifically, we fine-tuned Qwen2.5 for 1,000 steps using either BF16 or our INT8 attention, and then evaluated inference performance using our FP4 attention. The results on GSM8k and MMLU are shown below, where INT8 fine-tuning followed by FP4 inference yields higher accuracy, indicating that the two methods can be complementary, and their combination can deliver additional benefits.\n\n**Table 1: Comparison on Qwen2.5-1.5B.**\n\n| All using FP4 Inference|Gsm8k $\\uparrow$|Mmlu $\\uparrow$ |\n| :-----------------:|:--------------:|:-------------: |\n| BF16 Fine-tuning|0.4912| 0.4688 |\n| INT8 Fine-tuning|**0.5232**| **0.4934** |\n\n**Table 2: Comparison on Qwen2.5-3B.**\n\n| All using FP4 Inference|Gsm8k $\\uparrow$|Mmlu $\\uparrow$ |\n| :---------------------:|:--------------:|:-------------: |\n| BF16 Fine-tuning | 0.5860 | 0.6000 |\n| INT8 Fine-tuning | **0.5945** | **0.6032** |\n\nThis improvement is likely because INT8 and FP4 share a more similar representable data distribution, reducing the mismatch error compared to BF16.\n\n\n---\n**W2.** There are some other 4-bit quantization methods like QServe[1], Atom[2], and QuaRot[3]. While they don't support 4-bit computation in attention, 4-bit memory access can still boost model performance, especially for memory-bounded scenarios. So it would be good to compare SageAttention3 with other 4-bit quantization methods to demonstrate the advantages of SageAttention3. (QServe, Atom, QuaRot)\n\n**Response:** Thank you for your very professional and important suggestion. Yes, as you pointed out, the methods you mentioned do not quantize $Q$, $K$, and $V$ entirely to 4-bit, nor do they perform attention computations at FP4 precision. Our work is the first to leverage hardware FP4 Tensor Cores to perform the entire attention computation in FP4, directly accelerating the computation. Strictly speaking, these approaches are not directly comparable, similar to how FP8-quantized FlashAttention3 is not compared against KV-cache-only quantization methods.\n\nTo provide a more complete evaluation, we add two sets of comparison experiments. First, we compare SageAttention3 with QServe and Atom (please note that GPTQ is irrelevant here as it does not modify attention) on Qwen2.5-1B. As shown in the Table below, SageAttention3 achieves the best accuracy while also providing the fastest pre-filling speed. For decoding speed, because Atom and QServe cannot be compiled on the RTX 5090, we report their theoretical decoding speedup.\n\n**Table 3: Comparison on Qwen2.5.**\n\n| Method |wikitext PPL $\\downarrow$|LAMBDA $\\uparrow$|Prefilling(64K) Latency $\\downarrow$|Theoretical decoding speedup $\\uparrow$ |\n| ---------|------------------------:|----------------:|-----------------------------------:|--------------------------------------: |\n| Qserve | 10.90296 | 0.631| 10.20257 | 4x |\n| Atom | 10.89669 | 0.631| 10.22090 | 4x |\n| SageAttn3| **9.56827** | **0.726**| **6.09898** | 4x |\n\nSecond, we compare SageAttention3 and QServe on CogVideoX-2B. As shown in the Table below, SageAttention3 not only achieves higher accuracy but also provides substantial speedup.\n\n**Table 4: Comparison on CogVideoX.**\n\n| Method |CLIPSIM ↑|VQA-a ↑|VQA-t ↑|FScore ↑|Latency $\\downarrow$ |\n| ---------|--------:|------:|------:|-------:|-------------------: |\n| Qserve | 0.1683|19.7863|49.4831| 2.4488| 65s |\n| Atom | 0.1706|20.0896|46.9320| 2.4024| 64s |\n| SageAttn3|**0.1881**|**69.8600**|**70.3640**|**4.0350**| **27s** |\n\n\n\n---\n**Q1.** For the 8-bit attention for training, why int8 and not fp8? FP8 training is already widely used in LLM training, e.g., DeepSeek-V3. If the attention operator could also use FP8 for training, it could further boost the model training efficiency, with almost all of the computation being performed in 8-bit.\n\n**Response:** Thank you for your highly professional and valuable question. Many works have explored INT8 training for linear layers, and FP8 has often been chosen in practice due to its better end-to-end accuracy. Our choice of INT8 for attention training is based on two key reasons:\n\n1. **Higher gradient accuracy in attention backward:** The backward pass of INT8 attention yields more accurate gradients for $Q$, $K$, and $V$. As shown in the tables below, evaluating all layers of CogVideoX-2B demonstrates that INT8 SageBwd achieves lower L1 error and higher cosine similarity than FP8 SageBwd (with $\\mathbf{dOV}^\\top$ kept in FP16 for fairness; the only difference is the data type).\n\n**Table 5: L1 Error of Q, K, V gradient.**\n\n| Method|$dQ$ L1 Error $\\downarrow$ |$dK$ L1 Error $\\downarrow$|$dV$ L1 Error $\\downarrow$ |\n| :---------:|:---------:|:---------:|------ |\n| INT8 SageBwd|**0.0290**|**0.0317**|**0.0423** |\n| FP8 SageBwd|0.0696|0.0999|0.0873 |\n\n**Table 6: Cossim accuracy of Q, K, V gradient.**\n\n| Method|$dQ$ Cossim $\\uparrow$|$dK$ Cossim $\\uparrow$|$dV$ Cossim $\\uparrow$ |\n| :-------:|:-------:|:-------:|------ |\n| INT8 SageBwd|0.9987|0.9993|0.9995 |\n| FP8 SageBwd|0.9880|0.9910|0.9955 |\n\n2. **Wider hardware support:** INT8 is supported on virtually all modern GPUs, including A100 and many non-NVIDIA GPUs (e.g., AMD MI250, Ascend 910B), while FP8 support is still limited to newer architectures.\n\nAdditionally, we compare the end-to-end performance of INT8 and FP8 trainable attention when combined with FP4 SageAttention3 for inference. Specifically, we fine-tuned Qwen2.5-1.5B and Qwen2.5-3B for 1,000 steps using either INT8 or FP8 SageBwd (both with $\\bf dOV^\\top$ kept in FP16 for fairness), then evaluated inference with FP4 SageAttention3. As shown below, models fine-tuned with INT8 attention achieve higher accuracy on both GSM8k and MMLU benchmarks:\n\n**Table 7: Comparison on Qwen2.5-1.5B.**\n\n| All Using FP4 INference|Gsm8k $\\uparrow$|Mmlu $\\uparrow$ |\n| :---------------------:|:--------------:|:-------------: |\n| INT8 Fine-tuning | **0.5232** | **0.4934** |\n| FP8 Fine-tuning | 0.5031 | 0.4689 |\n\n**Table 8: Comparison on Qwen2.5-3B.**\n\n| All Using FP4 INference|Gsm8k $\\uparrow$|Mmlu $\\uparrow$ |\n| :---------------------:|:--------------:|:-------------: |\n| INT8 Fine-tuning | **0.5945** | **0.6032** |\n| FP8 Fine-tuning |0.5868| 0.5907 |\n\nThese results indicate that INT8 attention not only offers broader hardware compatibility but also produces more accurate gradients during training, leading to better downstream performance when combined with FP4 inference.\n\n\n---\n\n**If you feel your concerns have been resolved, we would greatly appreciate it if you consider raising the score. We will add all the experiments and analysis in the rebuttal to our paper.**"}]

Author rebuttal:

Meta-review: This paper presents strong contributions in efficient FP4 inference and INT8 training. While reviewers initially raised significant concerns about the two parts feeling disconnected and lacking crucial comparisons, the authors delivered an exemplary rebuttal. They provided new experimental data that directly resolved all core issues, which successfully convinced three of the four reviewers to raise their scores. Due to the paper's technical merit and the authors' effective engagement in the discussion, the final recommendation is Accept.
Decision: Accept (spotlight)