FlatQuant: Flatness Matters for LLM Quantization
Authors: Yuxuan Sun
Ruikang Liu
Haoli Bai
Han Bao
Kang Zhao
Yuening Li
JiaxinHu
Xianzhi Yu
Lu Hou
Chun Yuan
Xin Jiang
Wulong Liu
Jun Yao
Venue: ICML 2025 poster (2025)
Keywords: flatness
post-training-quantiation
affine transformation
large language models
INT4 quantization
Primary area: deep_learning->large_language_models
TL;DR: A new post-training quantization method for large language models with SOTA W4A4KV4 performance.

Abstract: Recently, quantization has been widely used for the compression and acceleration of large language models (LLMs). Due to the outliers in LLMs, it is crucial to flatten weights and activations to minimize quantization error with equally spaced quantization points. Prior research explores various pre-quantization transformations to suppress outliers, such as per-channel scaling and Hadamard transformation. However, we observe that these transformed weights and activations can still exhibit steep and dispersed distributions. In this paper, we propose FlatQuant (Fast and Learnable Affine Transformation), a new post-training quantization approach that enhances the flatness of weights and activations. Our approach identifies optimal affine transformations for each linear layer, calibrated in hours via a lightweight objective. To reduce runtime overhead of affine transformation, we apply Kronecker product with two lightweight matrices, and fuse all operations in FlatQuant into a single kernel. Extensive experiments demonstrate that FlatQuant establishes a new state-of-the-art benchmark for quantization. For example, it achieves less than 1\% accuracy drop for W4A4 quantization on the LLaMA-3-70B model, surpassing SpinQuant by 7.5\%. Additionally, it provides up to 2.3x prefill speedup and 1.7x decoding speedup compared to the FP16 model. Code is available at: https://github.com/ruikangliu/FlatQuant.

FLATQUANT: Flatness Matters for LLM Quantization
Yuxuan Sun Ruikang Liu *2 Haoli Bai
Han Bao
Kang Zhao
Yuening Li Wulong Liu' Jun Yao
Jiaxin Hu Xianzhi Yu
Lu Hou
Chun Yuan'
Xin Jiang'
Abstract
creasing number of parameters (Achiam et al., 2023; Jiang et al., 2023; Yang et al., 2024; Dubey et al., 2024). However, the growth of model size also incurs a significant increase in computation and memory overhead. As a result, reduc- ing the computational and memory demands of LLMMs has emerged as a critical research direction, and quantization is one of the most effective solutions (Frantar al., 2022; Lin et al., 2023; Dettmers et al., 2022; Xiao et al., 2023).
Recently, quantization has been widely used for the compression and acceleration of large lan- guage models (LLMs). Due to the outliers in LLMs, it is crucial to flatten weights and activa- tions to minimize quantization error with equally spaced quantization points. Prior research ex- plores various pre-quantization transformations suppress outliers, such per-channel scaling and Hadamard transformation. However, we observe that these transformed weights and activations can still exhibit steep and dispersed distributions. In this paper, we propose FLATQUANT (Fast and Learnable Affine Transformation), a new post- training quantization approach that enhances the flatness of weights and activations. Our approach identifies optimal affine transformations for each linear layer, calibrated in hours via lightweight objective. To reduce runtime overhead of affine transformation, we apply Kronecker product with two lightweight matrices, and fuse all operations in FLATQUANT into a single kernel. Extensive experiments demonstrate that FLATQUANT es- tablishes new state-of-the-art benchmark for guantization. For example, it achieves less than 1% accuracy drop for W4A4 quantization on the LLaMA-3-70B model, surpassing SpinQuant by 7.5%. Additionally, it provides up to 2.3x prefill speedup and 1.7x decoding speedup compared to the FP16 model. Code is available at: https:/ // gi wtw.tcooooooooce.eccooooooo.toot ruw kung Fu/FlatQuntt Quaun..
Quantization decreases the memory footprint and acceler- ates the inference, by reducing the precision of model param- eters and activations. Quantization error is commonly used metric measure the performance of quantization meth- ods (Nagel al., 2020; Bai et al.. 2020; al., 2021). One key factor that affects the quantization error is the flatness of weights and activations. Intuitively, when the distribution of weights and activations is sharp and there exist multiple outspread values, quantizing them to the same quantized value usually incurs a large quantization error (Chmiel et al., 2020; Li et al., 2024). Moreover, as LLMs generate outputs layer by layer, reduced quantization error also flattens the error landscape propagated across Transformer layers.
Nevertheless, it is non-trivial to get flat distribution of weights and activations in LLMs. LLMs are known to have extreme outliers over activations (Dettmers et al., 2022; Xiao et al., 2023) and pivot tokens (Liu et al., 2024a: Sun et al., 2024). To alleviate this problem, various pre-quantization transformations are proposed mitigate the impact of out- liers (Xiao et al., 2023; Ashkboos et al., 2024; Liu et al., 2024c; Ma et al., 2024). However, revisit these transfor- mations and find them still sub-optimal in promoting flat- ness. For instance, per-channel scaling (Xiao et al., 2023; Shao et al., 2023) aims to balance the outliers between weights and activations, but it falls short of distributing out- liers over the non-outlier channels. The recent Hadamard transformation (Ashkboos et al., 2024; Lin al., 2024) attempts to solve this problem, while the individual charac- teristics of each linear layer are not considered. Moreover, the linear transformation introduced by these methods in- evitably introduces extra inference overhead that affects the overall speedup of quantization.
1. Introduction
Recent large language models (LLMs) have achieved re- markable success across a wide range of tasks with an in-
Equual contribution 'Huawei Noah's Ark Lab *Shenzhen In- ternational Graduate School, Tsinghua University "The Chinese University of Hong Kong. Correspondence Haoli Bai <bai- haoli huawei.com>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In this work, we introduce FLATQUANT, a new post-training quantization approach. The name has dual significance: it

---

FLATQUANT: Flatness Matters for LLM Quantization
stands for the proposed method (Fast and Learnable Affine Transformation) and emphasizes its goal of achieving flatter distributions of weights and activations, which are crucial for effective quantization. FLATQUANT aims to identify the optimal affine transformation for each linear layer, em- ploying a lightweight, block-wise training strategy over the calibration data. To minimize the inference overhead associ- ated with affine transformations, FLATQUANT harnesses the efficiency of Kronecker product with two lightweight matri- ces, reducing both the memory and computational demands. Our approach is compatible with various quantization tech- niques such as learnable clipping, and can be applied to various quantization settings, e.g., weight-only quantiza- tion or KV cache quantization. Additionally, as the affine transformations in FLATQUANT are memory bound, we further fuse them together with quantization into single kernel, minimizing the global memory access and kernel lunch overhead. Lastly, extensive experiments over various tasks (e.g., language modeling and question answering) and LLM families (e.g., LLaMA and Qwen) are conducted to assess FLATQUANT. Our empirical results demonstrate that FLATQUANT establishes new state-of-the-an quantization results w.r.t. both accuracy and inference latency.
on previous KV cache. Quantization is common practice to reduce the model size and inference latency. It converts the full-precision weights W E IR=x" or activations X e R**" of linear layers (i.e., Y = XW"), and optionally the KV cache to low-bit representations. For instance, b-bit weight quantization can be represented as follows:
W (W) = s (ll(()
(I)
where is the quantization step size, 11(.) is the projection
function and (t(b) {0, 1,.., 2 1} is the set of b-bit integer points. For simplicity of notation, we denote Q(-) as the general quantization function in the rest of this paper.
As recent works suggest (Xiao et al., 2023; Shao et al., 2023; Xi al., 2023), LLMs exhibit persistent outliers in activations, posing significant challenges for quantization. Various works are proposed to suppress outliers to improve the quantized LLMs. Two commonly used methods are per-channel scaling (Xiao et al., 2023; Lin et al., 2023; Wei et al., 2023) and Hadamard transformation or its variants (Xi et al., 2023; Ashkboos et al., 2024; Lin et al., 2024).
Per-channel Scaling. The input activations X of LLMs are often rich in outliers. To mitigate their impact on quan- tization, popular way is to apply thannel-wise scaling over weights and activations (Xiao et al., 2023), i.e.. Y (Xdiag(c)-)) (ddag))))))((((()))))))) (diag(c)WT)) where E R" is the channel- wise scaling factor. The scaling vector smooths the activa- tions by jointly considering the magnitudes of input activa- tions and weights, i.e. Cj = max([X,]
The contributions of this work are summarized below:
We highlight the significance of achieving flatness for LLM quantization, demonstrating that flat distributions of weights and activations facilitate quantization and reduce error propagation across Transformer layers.
max([W,)*
The scaled weights diag(e) W can be merged to eliminate the runtime computation. Additionally, Wei et al. (2023) introduces channel-wise shifting, i.e., (X z z)diag(c)-* to further mitigate the impact of outliers, and Shao et al. (2023) treats both diag(c) and z as learnable parameters.
We introduce FLATTQANT, a new post-training quan- tization method with fast and learnable affine transfor- mations optimized for each linear layer. The approach is empirically demonstrated to enhance the flatness of both weights and activations in LLMs.
Hadamard Transformation. Recent works find Hadamard matrices H E (+1,-1)axm are particularly helpful in smoothing out outliers in activations (Xi et al., 2023; Ashkboos et al., 2024; Lin et al., 2024). In contrast to per-channel scaling which only adjusts the diagonal elements in the view of matrix multiplication, Hadamard transformation rotates the channels of both activations and weights, re-distributing the outliers among all channels to effectively eliminate them. Thanks to the orthogonality of Hadamard matrices (i.e., H = I), the following equivalency holds: Y = XW = (X TT). The transformed weight WH can be similarly pre-processed offline to reduce additional runtime overhead.
Extensive experiments demonstrate that FLATQUANT sets new state-of-the-an results for quantization. To the best of our knowledge, we are the first to achieve < 1% accuracy drop with simply round-to-nearest W4A4 quantization on the LLLMA-33770B model.
We also design an efficient kernel that fuses both affine transformation and quantization, leading to 2.3x pre- fill speedup and 1.7x decoding speedup under W4A4 quantization when compared to the FP16 baseline.
2. Motivation
2.1. Preliminaries on LLM Quantization
2.2. The Flatness for Quantization
The inference of LLM typically has two stages: 1) the prefill stage, which creates a key-value cache (KV Cache) layer by layer from the input sequence; and 2) the decoding stage, where the model autoregressively generates tokens based
The Flatness of Weights and Activations. Flat tensors are intuitively easier to quantize after removing outliers, a.k.a tensors with low kurtosis (Chmiel et al., 2020; Li et al.,
2

---

FLATQUANT: Flatness Matters for LLM Quantization
Original
Per-channel Scaling
Hadamard
FlatQuant
2.0
15
35
1.8g
1.0-
Magnitude
25-
Magnitude
Magnitude
15
20
6..5-
20
5/
0.0
1000
2000
3000
4000
1000
2000
3000
4000
0
2000
4000
6000
8000
2000
4000
6000
8000
Channels
Channels
Channels
Channels
(a) W. of the 10th Transformer layer in LLaMA-3-8B.
(b) X. of the 10th Transformer
(c) W of the 30th Transformer
(d) X, of the 30th Transformer layer in LLaMA-3-7000
layer in LLaMA-3-8B.
layer in LLaMA-3-70B.
Figure 1: Distributions of weights and inputs from LLaMA-3-8B 8B and LLaMA-3-70BB sorted by the channel magnitudes (i.c., the Frobenius norm) in descending order. In Transformer layer. W. and X, denote the weight matrix and input of the output projection layer in the self-attention layer, respectively. W and Xg denote the weight and input of the gated linear layer of the feed-forward network, respectively. More visualizations can be found in Appendix D.
Per-channel Scaling
Hadamard
FlatQuant
2.0
1.0
0.20
0.8
20
1.0
0.4
0.10
0.5
0.05
0.00
10
Layer
Layer
Layer
25
Layed
Taken
100
Token
100
125
100 Token
75 Token
100
125
(a) Per-channel Scaling.
(b) Hadamard Transfrom.
(c) FLATQUANT.
(d) Stacked View.
Figure 2: The mean squared error (MSE) of quantization across Transformer layers and input sequence in LLaMA-3-8B. Figure 2a-2c plot the MSE surface of each method, while Figure 2d overlays these surfaces by dividing each MSE with that of FLATTQANT. More details and visualizations can be found in Appendix D.
2024). Figure I displays the distributions of both the origi- nal and transformed weights and activations, sorted by the channel magnitudes in descending order. The flat weights and activations with horizontal envelopes are usually pre- ferred by quantization. Compared with the original distribu- tions, pre-quantization transformations
pivot tokens (Liu et al., 2024a), which contain massive outliers (Sun et al., 2024). Both per-channel scaling and Hadamard transformation are powerless to such errors (i.e., Figure 2a-2b). Instead, FLATQUANT shows much lower error at these pivot tokens from Figure 2c. Second, the quan- tization error increases layer-wisely, but is less evident along the input sequence. According to Figure 2d, FLATQUANT is the best in controlling the error propagation, followed by Hadamard transformation and lastly the per-channel scaling.
yield flatter acti- vations (e.g., Figure 1b, 1d) but still with their limitations. Per-channel scaling flattens activations at the cost of steeper weight envelops (e.g., Figure 1a. Ic). While Hadamard transformation produces better flatness for both activations and weights than per-channel scaling, still sometimes gen- erates unsatisfactory distributions (e.g., Figure la, lb). In contrast, FLATQUANT, will be claborated in Section 3, consistently flattens both weights and activations.
3. Method
We now introduce FLATQUANT with overview in Fig- ure 3. The proposed method has dual significance: it em- ploys fast and learnable affine transformations, which also produce flat weights and activations for easy quantization.
The Flatness of Quantization Error Landscape. The quantization error inevitably propagates, and it is insight- ful to show how pre-quantization transformations mitigate this issue. We plot the wio-dimensional landscape of mean squared error (MSE) in Figure 2. First, observed that massive quantization errors occur at initial tokens, a.k.a.
3.1. Fast and Learnable Affine Transformation
We first introduce FLATQUANT for standard linear layer, and will discuss its integration with the Transformer archi-
3

---

FLATQUANT: Flatness Matters for LLM Quantization
(c)
W diag(e) c) R Ruxnog
X /dig(e) E Ru
W
Merged Weight (Quantized)
Notations
dess
Puir-wisely Merged Scaling Vector
1. Quantization Dequantizaionn
re re-distribute
To distributte
P
Affine Transformation Matrix
Pi
Lancar Layer
(b)
KV Cache
Sofhrark
LayerNor
W
LayerNorm
W
W. P.
Self-attention
Feed-forward netwerk
Figure The overall framework of FLATQUANT. (a): necessary notations of FLATQUANT; (b): the integration of FLATTQANT with a conventional LLaMA layer, where merged parameters are grouped in red, online transformation and quantization functions in blue, and merged scaling vectors in green; (c): the exemplary view of FLATQUANT applied for the down-projection layer, where the scaling vector diag(c) over X is merged to W. in practice.
tecture in Section 3.2. primary objective of FLATQUANT is to find the best affine transformation for each linear layer to quantize. Ideally, given Y X WW one can identify
2ninn 2. with the equality holds when n1 R22 vT. Moreover, the computation saving is vn/2 times with the same optimal condition. In practice, we select ni, n = arg (ain(n++2))))) s.t. ny = n and n1 S f2. For instance, the optimal configuration is (ni, n²) (64, 128) for n 8192. We find such strategy gives the best speedup without compromising performance, as shown in Figure 5. The affine transformations are therein pretty hight-weight e.g., they only take 2.61 % of the overall computational FLOPPs and 3.41MB extra memory for LLaMA-2-7B. More details of inference overhead are Appendix B.2.
the optimal invertible matrix P E m by
P arg min ((XP))(P
(2)
as studied in (Ma et al., 2024). The weights P W can be pre-computed offline akin to (Ashkboos et al., 2024). However, unlike Hadamard matrices that can be reused for all layers, it is computationally expensive to maintain individual P matrices for different linear layers. In the forward pass, this approach doubles the computational cost and memory access for matrix multiplication. Additionally, it nearly doubles the model storage requirements.
Per-channel Scaling, To enhance the ability to balance outliers between the weights and activations, FLATQUANT explicitly introduces a learnable scaling vector diag(c) E R" prior to the pre-quantization transformation, as illus- trated in Figure (c). Following (Xiao al., 2023), the scaling vector can be merged pair-wisely the preceding layer normalization or linear layers, thereby incurring no additional inference overhead.
Kronecker Product. The key of FLATQUANT is to use Kronecker product of two lightweight matrices as an ef- ficient substitution of the large affine transformation ma- trix P _=x=, Specifically, we propose to construct two lightweight matrices, i.e P P1 & Pz, where P1 E RR P2 Ra2xxa are invertible matrices in smaller sizes, and n n22 Recall the vectorization trick of the Kronecker product, i.e, vec( (PPPPPPP)))) = V VP2) for some V E R"1*"ng the matrix multiplication in Equa- tion 2 can re-written as
Learnable Clipping Thresholds. To further reduce the potential outlier after the above transformation, we com- bine learnable clipping thresholds Ows ag (0, 1) after sigmoid functions on both weight and activation for each linear layer, together with the KV cache. While previous studies (Jacob et al., 2018: Frantar et al., 2022; Ashkboos et al., 2024) demonstrate that grid search is valid to find reasonable clipping thresholds, we observe that learning the clipping thresholds yields better results. These parame- ters are layer-specific and can be jointly optimized with the linear transformation matrices P and scaling vector diag(c).
Q(XP)Q(P- (W
(3)
Q(P x X X2 P2) Q(P X1 W X2 (PZ))
where X E R X and W pmxxnix are reshaped from X and W accordingly, and x; denotes the reduction over the i-th axis. Note that both weights and activations are converted back to matrix before multiplication. This design save the memory up to n times, given that n² xftng
The Training Objective. We follow post-training quan- tization and sequentially minimize the mean squared er- ror (MSE) by quantization over small amount of calibra-
S

---

FLATQUANT: Flatness Matters for LLM Quantization
tion data (e.g., 128 randomly sampled sentences) for each Transformer block. The training objective for the 1-th Trans- former block is
residual connection of the "pre-norm" architecture would constrain all Transformer blocks share the same transfor- mation after RMSNorm. Instead, FLATQUANT preserves the LayerNorm, and allows the use of fast and learnable affine transformations in Section 3.1 after LayerNorm for different layers, thereby enhancing the expressiveness.
min - -
(4)
where Fi(-) and Fi) denote the original and the quantized Transformer block, θ = {P, C, Qu, Oty J is abbreviated for all learnable parameters within the block. The transforma- tion matrices within a Transformer block will be explained in Section 3.2. To compute the matrix inversion in Equa- tion 3 efficiently and accurately, we adopt the singular value decomposition together with automatic mixed precision. See Appendix B.1 for details. We also tried with train- ing multiple Transformer blocks together but found similar performance at higher training costs. Finally, the training progress with Equation 4 leads to flat weights and activa- tions, as will be shown in Section 4.4.
3.3. Efficient Kernel Design
We design an efficient kernel for FLATQUANT that inte- grates both affine transformations and quantization into a single operation. This design is motivated by two key fac- tors. First, P X1 X X2 P2 exhibits low computational intensity with Kronecker product of two lightweight ma- trices, making both prefilling and decoding predominantly memory- bound.. Second, the quantization is also known to be memory-bound.
To address these issues, we fuse Q(P X1 X X2 P2) into a single kernel using OpenAl Triton (Tillet et al., 2019). Specifically, we load the entire P1 R X=1 and P2 E m into SRAM. Each thread block slices tiling block X E R from X, performs the matrix multiplica- tion P,XP2, and quantizes the results on the fly. Through- out this process, all intermediate results are stored in SRAM before finally being written back to the global memory. This design thereby eliminates redundant memory accesses of intermediate results and reduces the kernel launch overhead. Finally, given the output above, we follow QuaRot (Ashk- boos et al., 2024) to adopt the CUTLASS kernel for INT4 matrix multiplication, and FlashInfer (Ye, 2023) for KV cache quantization. Further details of the kernel design are provided in the Appendix B.3.
3.2. Integration with the Transformer Architecture
We now illustrate the integration of FLATQUANT with Transformer block based on an LLaMA-like architecture. Following the conventional practices, we employ low-bit matrix multiplications for all linear layers, while keeping layer normalization layers, qrequuatization transformations, RoPE embeddings, and attention scores in FP16.
Self-Attention.
The self-attention module is equipped with four transformations [Pa. P.. Ph. P. Specifically, Pa is applied to flatten the input activation for the query, key, and value projections, while P. smooths the input ac- tivation for the output projection. Pи, and P, are used to transform the key and value cache head by head, respec- tively. Note that we only decompose Pa and P.. but leave Pa. and P. in their original shape. This is because per-head quantization already facilitates cheap transformations, given that the head size is significantly smaller than the full hid- den size. Moreover, we further fuse P. with P. to reduce overhead, inspired by QuaRot (Ashkboos et al., 2024). Our empirical results show this fusion does not result in additional loss of accuracy.
4. Experiments
4.1. Settings
Evaluation and Baselines. We primarily evaluate FLATQUANT the series of LLaMA-2 (Touvron et al., 2023) and LLaMA-3 (Dubey et al., 2024) models, and re- sults on Qwen and DeepSeek families can be found in Ap- pendix C.I. Following previous works (Shao et al., 2023; Ashkboos et al., 2024), we report the perplexity (PPL. of language generation tasks on the WikiText2 (Merity et al., 2016) and C4 (Raffel et al., 2020) datasets. For com- monsense reasoning tasks, we six zero-shot evalua- tion tasks, including ARC-Challenge, ARC-Easy (Clark et al., 2018), HellaSwag (Zellers et al., 2019), LAM- BADA (Paperno et al., 2016), PIQA (Bisk et al., 2020), and WinoGrande (Sakaguchi et al., 2021). We compare FLATQUANT against popular INT4 post-training quantiza- tion methods, including SmoothQuant (Xiao al., 2023), OmniQuant (Shao et al., 2023), AffineQuant (Ma et al., 2024), QUIK-4B (Ashkboos et al., 2023), and two recent
Feed-forward Network. The feed-forward net- work (FFN) employs two transformation matrices, i.c., Pag and Pa. Pug is applied to flatten the input of the feed-forward network after layer normalization, while P. flattens the input for the down projection layer. Both transformations are decomposed to minimize the inference overhead. Additionally, the per-channel scaling of Pa is merged into the weight of up projection layer, ensuring no additional computational overhead.
Layer Normalization. Recall that QuaRot (Ashkboos et al., 2024) and SpinQuant (Liu et al., 2024c) modify the LayerNorm to RMSNorm and merge orthogonal transforma- tions into preceding layers for efficiency. Nonetheless, the
5

---

FLATQUANT: Flatness Matters for LLM Quantization
Wikifext-2
C4
Method
W Quantizer
2-7B
2-13B
2-70B
3-8B
3-70B
2-7B
2-13B
2-70B
3-8B
3-70B
FP16
5.47
4.88
3.32
6.14
2.86
7.26
6.73
5.71
9.45
7.17
SmoothQuant
RTN
83.12
35.88
26.01
210.19
9.60
77.27
43.19
34.61
187.93
16.90
OmniQuant
RTN
14.74
12.28
21.40
16.24
AffineQuant
RTN
12,69
11.45
15.76
13.97
QuaRot
RTN
8.56
6.10
4.14
10.60
55.44
11.86
8.67
6.42
17.19
79.48
SpinQuant
RTN
6.14
5.44
3.82
7.96
7.58
9.19
8.11
6.26
13.45
15.39
FLATQUANT
RTN
5.79
5.12
3.55
6.98
3.78
7.79
7.09
5.91
11.13
7.86
QUIK-4B
GPTQ
8.87
7.78
6.91
QuaRot
GPTQ
6.10
5.40
3.79
8.16
6.60
8.32
7.54
6.12
13.38
12.87
SpinQuant
GPTQ
5.96
5.24
3.70
7.39
6.21
8.28
7.48
6.07
12.19
12.82
FLATQUANT
GPTQ
5.78
5.11
3.54
6.90
3.77
7.86
7.11
5.92
11.21
7.93
Table I: WikiText-2 and C4 perplexity of 4-bit weight & acitvation quantized LLaMA models.
state-of-the-art methods QuaRot (Ashkboos et al., 2024) and SpinQuant (Liu et al., 2024c).
quantizer consistently outperforms previous SOTA quantiza- tion methods across all major benchmarks. For the LLaMA- 2-70B model, FLATQUANT achieves a PPL score just 0.23 higher than the FP16 baseline, underscoring the effective- of our approach. For LLaMA-3-8B, FLATQUANT re- duces the PPL from 7.39 (SpinQuant) to 6.98, narrowing the gap with the FP16 baseline to 0.84. Notably, FLATQUANT with RTN exhibits performance comparable to those with GPTQ but takes significantly less calibration time. This is particularly helpful in reducing the time consumption to deploy FLATQUANT in practice. These results highlight the efficacy of our proposed learnable transformations in enhancing flatness and mitigating the impact of outliers in both weights and activations, thereby establishing new SOTA in low-bit LLM quantization.
Implementation Details. We implement FLATQUANT based on Huggingface (Wolf, 2019) and PyTorch (Paszke al., 2019). We adopt the AdamW optimizer with initial learning rate of 5e-3 and employ cosine annealing learning rate decay schedule. The learning rate for clipping thresholds is 2. FLATQUANT is trained for 15 epochs on a calibration set comprising 128 sentences from Wik each sampled with 2048 tokens. The batch size is set to The default calibration procedure costs approximately 26GB of GPU memory and about 0.9 hours for LLaMA-3- 8B single GPU. FLATTQANT is robust initialization, and we employ random affine transformation matrices the starting point. Further details about implementation and calibration time are provided in Appendix B.I.
Results on Zero-shot QA Tasks. We extend our evalu- ation to six zero-shot commonsense QA tasks, as shown in Table For a fair comparison, we reproduce QuaRot and SpinQuant with their official implementations and released checkpoints, evaluating all methods with the same version of Im-eval-harnees -hanness framework (Gao al., 2021). As can be seen, FLATQUANT significantly narrows the perfor- mance gap between quantized models and the FP16 base- line. Specifically, while the LLaMA-3 models are shown to be challenging for quantization (Huang et al., 2024), FLATQUANT perform well with an accuracy loss of 2.00% for LLaMA-3-8B and 0.94% for LLaMA-3-70B, Notably, while QuaRot with RTN largely lags behind QuaRot with GPTQ by average accuracy gap over 4%, FLATQUANT with RTN can already obtain comparable results to GPTQ.
We adopt per-channel and per-token sym- metric quantization for weights and activations, respectively. For fair comparisons with QuaRot and SpinQuant, we em- ploy both round-to- quantizer, where GPTQ uses the same calibration data for both closed-form weight updates and training. Nonetheless, our empirical results suggest that FLATQUANT with RTN is sufficient to be competitive. FLATQUANT can be also used for KV cache quantization, where group-wise asym- metric quantization with the size of 128 is applied, This matches the head dimension of LLa.M, as suggested pre- vious studies (Zhao al., 2024; Ashkboos al., 2024), to leverage the memory-boume characteristies of self-attention.
Quantization.
(RTN) and GPTQ as the weight
4.2. Main Results
Due to limited space, we leave more experimental results in Appendix C, such as exploration to other quantization set-
Results on Language Generation Tasks, Table presents the PPL results for FLATQUANT with and with- out the GPTQ weight quantizer on the WikiText-2 and C4 datasets. As can be seen, FUAATQUANT with RTN weight
https:///gichub.cm// www.cccc/ccpcccccccco..ooc...cct /QuaRot Pnttpa://githus com/ www.m//aaaaooreseerrr...oo
Spinnuant
6

---

FLATQUANT: Flatness Matters for LLM Quantization
Model
Method
Quantizer
ARC-C
ARC-E
HellaSwag
LAMBADA
PIQA
Winogrande
Avg
FPI6
46.16
74.54
75.98
73.92
79.05
69.06
69.79
QuaRot
RTN
36.60
61.41
65.07
48.06
72.20
63.06
57.73
SpinQuant
RTN
39.42
65.32
71.45
66.16
75.30
63.46
63.52
2-7B
FLATQUANT
RIN
43.26
72.05
73.64
72.04
77.26
69.53
67.96
QuaRot SpinQuant
GPTQ
42.32
68.35
72.53
65.40
76.33
65.11
65.01
GPTQ
41.72
69.28
72.90
71.28
76.17
66.06
66.23
FLAIQUANT
GPTQ
43.00
71.21
73.31
72.06
77.53
67.72
67.47
PPI6
49.15
77.44
79.39
76.73
80.47
72.14
72.55
QuaRot
RTN
42.83
69.95
73.54
65.62
77.69
67.88
66.25
SpinQuant
RTN
43.69
72.43
75.52
72.42
78.40
68.90
68.56
2-13B
FLATQUANT
RTN
48.04
76.64
77.59
76.60
79.38
70.24
71.42
QuaRot
GPTQ
45,48
73.27
76.03
69.01
79.05
70.64
68.91
SpinQuant
GPTQ
49.15
77.19
76.86
73.86
78.67
69.85
70.93
FLATQUANT
GPTQ
48.38
76.94
77.88
76.40
79.65
70.56
71.64
FP16
57.17
81.02
83.81
79.60
82.70
77.98
77.05
QuaRot
RTN
52.22
76.60
79.96
74.61
81.12
76.32
73.47
SpinQuant
RIN
55.03
79.17
81.76
78.87
81.45
74.27
75.09
2-70B
FLATQUANT
RTN
56.14
80.30
83.01
79.60
82.75
77.90
76.62
QuaRot
GPTQ
55.46
79.76
81.58
79.35
81.83
76.09
75.68
SpinQuant
GPTQ
55.38
79.04
82.57
78.75
82.37
78.22
76.06
FLATQUANT
GPTQ
56.40
80.09
82.91
80.01
82.92
76.87
76.53
FP16
53.50
77.57
79.12
75.51
80.74
72.93
73.23
QuaRot
RTN
38.65
66.54
68.82
57.20
71.82
65.04
61.34
SpinQuant
RTN
45.73
71.38
74.07
67.67
76.66
66.38
66.98
3-8B
FLATQQANT
RTN
50.00
75.80
76.80
72.91
79.16
72.69
71.23
QuaRot
GPTQ
45.73
70.83
72.97
62.70
75.35
67.17
65.79
SpinQuant
GPTQ
47.27
74.20
74.55
70.29
77.37
68.70
FLATQUANT
GPTQ
50.51
75.88
76.49
73.20
79.00
72.93
71.33
FP16
64.25
85.94
84.93
79.37
84.44
80.74
79.95
QuaRot
RTN
22.18
34.30
32.15
13.35
57.67
52.49
35.36
SpinQuant
RTN
44.03
69.07
74.57
63.34
76.99
65.98
65.66
3-70B
FLAIQUANT
RTN
62.12
84.97
83.95
78.73
84.28
80.03
79.01
QuaRot
GPTQ
49.49
74.37
77.22
71.69
78.89
71.03
70.45
SpinQuant
GPTQ
51.96
77.40
77.29
71.90
79.33
72.06
71.66
FLATQUANT
GPTQ
61.95
84.47
83.87
77.99
83.95
79.24
78.58
Table 2: Zero-shot QA task results of 4-bit weight & activation quantized LLaMA models.
tings in Appendix C.3, results on more LLM architectures in Appendix C. and MTT-bench evaluations in Appendix C.2.
FLATQUANT acheives comparable speed-up with QuaRot, thanks the Kronecker product of two lightweight matri- ces. With kemnel fusion, FLATQUANT
achieve up 2.30x speedup for prefill and 1.76x speedup for decoding under the batch size of 64, which is apparently faster than QuaRot (Ashkboos et al., 2024). Although there is still minor gap compared to the vanilla INT4 quantization, sig- nificantly enhances accuracy and facilitates the deployment of INT4 LLMs in real-world applications.
4.3. Inference Latency
All experiments of inference latency below conducted on the RTX3090 GPU. More details of the overall com- putational FLOPs, kernel profiling, and speedup gains are available in Appendix C.8.
End-to-end Speedup. Figure 4 shows the prefill and de- coding speedup of FLATQUANT across different batch sizes, with 2048 and 256 tokens for prefill and decoding, respec- tively. It can be found that even without kernel fusion,
Sizes and Perplexities. In Figure 5, we examine the impact of different decomposed matrix sizes in Equation 3 model performance and speedup.
Kronecker Product:

---

FLATQUANT: Flatness Matters for LLM Quantization
INT4
QuaRot
Flatquant w/o Kernel Fusion
FlatQuant w/ Kernel Fusion
2.00
Ubx
2.0
Speedup
1.25
1.5
1.00
0.75
0.50
0.25
0.0
0.00
32
16
Batch Size
Batch
(a) Prefill Speedup.
(b) Decoding Speedup.
Figure 4: Prefill and decoding speedup of LLaMA-2-7B model across different batch sizes. We decode 256 tokens after the prefill on sequence length of 2048.
Speedup
INT4
FlanQuant wro Kernel Fusion FotQuant wy Kamel Ruston
Qualttr
2.00
1.75
5.00
2.35
Speedup
1.50
3.55
2:30
1.25
Speedup
5.50
2.25
0.26x
0.26x
2.15
5.40
120
250
512
1024
2040
Decomposed Size
P.
P.
P,
P.
Online Transformations
Figure Prefill speedup and WikiText2 PPL results of different decomposed matrix sizes LLaMA-2-7B model. We decompose the hidden dimension 4096 into 72 and range 71 from 1 to 2048, where 1 amounts maintaining full-size transformation matrix. More details can be found in Appendix C.6.
Figure 6: Prefill speedup of LLaMA-2-7B a sequence length of 2048 under a batch size of 64 after applying differ- ent online transformations. We incorporate different online transformations sequentially to gauge their impact on the final speedup. Each point on the x-axis indicates adding a new online transformation.
As shown, the varying sizes of matrices for the Kronecker product significantly affect speedup. However, they have limited impact on the perplexity of generated text. The speedup peaks when P1 and P2 are of equal size (i.e., n1 = n2 /n 64), as predicted by our theoretical analy- sis in Section 3.1. When n2 exceeds 64, the speedup quickly decreases due irregular memory access patterns for acti- vations. These results further demonstrate FLATQUANT'S effectiveness in minimizing inference overhead while main- taining quantization accuracy with the Kronecker product.
0.17x. Meanwhile, P. results in a 0.01x slowdown, versus QuaRot's 0.1x. The rest transformations (i.e., Pa and P Pug) have an insignificant impact of less than 0.01x. Finally, can found that even without kernel fusion, the additional transformations in FLATQUANT still on par with QuaRot, thanks to the Kronecker product of two lightweight matrices.
4.4. Discussions
LT
PS
LCT
WikiText-2
C4
Avg
Overhead of Each Online Transformation. We now investigate the impact of the five online transformations (i.e., (Pa, P., Ps, P., ug: Pa)) in FLATQUANT on the over- all speedup, shown in Figure 6. Even with five per-layer transformations, FIAATQANN results in minimal 0.07x end-to-end slowdown, significantly outperforming QuaRot's 0.26x with just three Hadamard transformations. Specif- ically, FLATQUANT's Pa causes a 0.04x slowdown due large FFN intermediate sizes, compared with QuaRot's
1266.60
936.41
30.99
8.50
13.51
66.82
7.95
12.74
67.08
7.11
11.47
70.72
6.98
11.13
71.23
Table Ablation study FLATQUANNT' main components on LLaMAA3-8B..
8

---

FLATQUANT: Flatness Matters for LLM Quantization
MSE
Flatness
100
0.009
0.7
50-
0.0200
0.035
0.008
Flatness
80-
0.007
Flatness
45-
0.0150
0.0175
Flatness
120
100
0.030
M5
Flatness
0.5w
2000
0.4
0.005
6.000
0.004
25
00
1000
0.3
Epochs
Epochs
Epochs
Epochs
(a) 7 Transformer block.
(b) 15th Transformer block.
(c) 23th Transformer block.
(d) 31 Transformer block.
Figure 7: Flatness and mean squared quantization error (MSE) of different Transformer blocks in LLaMA-3-8B during FLATQUANT's training process. The metric of flatness is calculated as the sum of Euclidean distances ||d - d |l2 for all weights and activations within a Transformer block.
Ablation Study. We conduct ablation studies for FLATQUANT focusing on its main components: 1) learn- able transformation (LT); 2) per-channel scaling (PS); and 3) learnable clipping thresholds (LCT). Starting from RTN as baseline, we evaluate the impact of each component perplexity and the average accuracy on zero-shot QA tasks based on LLaMA-3-8B As shown in Table 3, enabling LT significantly enhances the accuracy of the quantized model, reducing PPL from 1266.60 to 8.50 on WikiText-2. This shows LT is capable of adaptively flattening the distribution of weights and activations. Additionally, incorporating PS and LCT further improves PPL by 0.55 and 0.84, respec- tively, demonstrating the necessity of each component to enhance the performance. Due to space limitation, we leave the more comprehensive ablation in Appendix C.4.
Due to space constraints, we provide additional experiments and discussions Appendix C.5, including the impact of calibration data, the effect of learnable clipping and mixed- precision schemes. We also analyze inference memory con- sumption in Appendix C.7. Additional visualizations of flatness and quantization error landscapes are provided in Appendix D.I and D.2, respectively.
5. Conclusions
In this study, revisit the importance of flat weights and activations for effective quantization, and find exist- ing solutions still produce steep outspread values after the pre-quantization transformation. Therefore, we introduce FLATQUANT, a novel post-training quantization method with the purpose identifying fast and learnable transforma- tions for each linear layer, to promote the flatness of weights and activations. Extensive experiments demonstrate the su- periority of FLATQUANT, e.g., with less than 1% accuracy drop for W4A4 quantization on the LLaMA-3-70B. Our efficient kernel fusion integrates the affine transformation and quantization, bringing up to 2.3x and 1.7x speedup over FP16 inference at the prefill and decoding stages, respec- tively. We hope this work advances the practical application of low-bit quantization for LLMs.
FLATQUANT Leads Flatness. To further analyze how FLATQUANT promotes flatness, we quantitatively evaluate the flatness of weights and activations by analyzing their channel-wise magnitude distributions. Specifically, each distribution is represented as a one -ie-dimenioona vector d, as illustrated in Figure 1. We measure flatness with the mean squared error (MSE) between the observed distribu- tion and an idealized perfectly flat distribution d', The flat distribution d' is defined such that all channels pos- sess equal magnitudes and the same norm d, i.e., d' |dde 1N, where is the number of channels and 1y is an -i-dimnnioonll vector with all entries equal to one. The Euclidean distance ||d-d' ||2 thus serves as our flatness metric, where smaller values indicate distributions closer to uniformity across all channels. Figure 7, we visualize the evolution of flatness and the training objective (Equation 4) across different Transformer blocks of LLaMA-3-88 dur- ing training. With the decreasing training loss, the channel distributions become increasingly flat. This indicates that FLATQUANT learns better transformations to obtain a flatter distribution which ultimately contributes to smaller quanti- zation error, i.e., flatness matters for L.M quantization.
Impact Statement
VN
This paper presents work aimed at advancing the field of Machine Learning by improving the efficiency of LLMs through enhanced quantization techniques. These improve- ments have the potential lower operational costs and en- ergy consumption, enabling broader access to AI, fostering more equitable use of advancee technologies. However, it is important to recognize that quantization does not address inherent societal biases in training data. Careful consider- ation and responsible deployment of quantized models are necessary to mitigate ethical risks and ensure their fair use.

---

FLATQUANT: Flatness Matters for LLM Quantization
References
Duanmu, H., Yuan, Z., Li, X., Duan, J., Zhang, X., and Lin, D. Skvq: Sliding- wwindwww key and value cache quantization for large language models. arXiv preprint arXiv:2405.06211 2024.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08772 2023.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang. A., Fan, A., et al. The llama 3 herd of models. arXiv preprint mIXiv2447.21788. 2024.
Ashkboos, S., Markov, I., Frantar, E., Zhong, T., Wang, X., Ren. J., Hoefler, T., and Alistarh, D. Towards end-to- end 4-bit inference on generative large language models. arXiv preprint arXiv:2310.09259, 2023.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Optq: Accurate quantization for generative pre-trained trans- formers. In The Eleventh International Conference on Learning Representations. 2022.
Ashkboos, S., Mohtashami,, A., Croci, M. L., Li, B., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. Quarot: Outlier-free 4-bit inference in rotated Ilms, arXiv preprint arXiv:2404.00455. 2024.
Gao, L., Tow, J., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., McDonell, K., Muennighoff, N., et al. A framework for few-shot language model evaluation. Version vO. 0.1. Sept, 10:8-9, 2021.
Bai, H., Zhang, W., Hou, L.., Shang, Jin, Jiang, X., Liu, Q., Lyu, M., and King, I. Binarybert: Pushing the limit of bert quantization. arXiv preprint 15571 2020.
Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., and Gholami, A. Kvquant: Towards 10 million context length Ilm in- ference with kv cache quantization. arXiv preprint arXiv:2401. 18079, 2024.
Bai, H., Hou, L., Shang, L., Jiang, X., King, I., and Lyu, M. R. Towards efficient post-training quantization pre- trained language models. Advances in neural information processing systems, 35:1405-11 14114418188,,, 2022.
Huang, W., Ma, X., Qin, H., Zheng, X., Lv, C., Chen, H., Luo, J., Qi, X., Liu, X., and Magno, M. How good are low-bit quantized Ilama3 models? an empirical study. arXiv preprint Aiv:22404.14444 2024.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reason- ing about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. Quantization and training of neural networks for efficient integer- arithmetic-only inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2704-2713, 2018.
Chee, J.. Cai, Y., Kuleshov, V., and De Sa, C. M. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024.
Chen, X., Bai, H., Yuan, T., Liu, R., Zhao, K., Yu, X., Hou, L., Guan, T., He, Y., and Yuan, C. A simple linear patch revives layer-pruned large language models. arXiv preprint arXiv:2505.246800 2025.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. I., Bressand, F., Lengyel, G., Lample, G., Saulnier, I.., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Chmiel, B., Banner, R., Shomron, G., Nahshan, Y., Bron- stein, A., Weiser, U., et al. Robust quantization: One model to rule them all. Advances in neural information processing systems, 33:5308 001885317,2200 2020.
Kim, S., Hooper, C. R. C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. Squeezellm: Dense-and-sparse quantization. In International Conference on Machine Learning, pp. 33901-23922 PMLR, 2024.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: Ntt.:1803305557 2018
Li, S., Ning, X., Wang. L., Liu, T., Shi, X., Yan, S., Dai, G., Yang, H., and Wang, Y. Evaluating quantized large lan- guage models. arXiv preprint arXiv:2402. 18158, 2024.
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Gpt3. int8 Q: 8-bit matrix multiplication for transform- ers at scale. Advances in Neural Information Processing Systems, 15:30318-30333 2022.
Li, Y., Gong, R., Tan, X., Yang, Y., Hu, P., Zhang, Q., Yu, F., Wang. W., and Gu, S. Breeq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint arXiv:2102.05446 2021.
10

---

FLATQUANT: Flatness Matters for LLM Quantization
Lin, J., Tang, J., Tang. H., Yang, S., Chen, W.-M., Wang. W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-aware weight quantization for IIm compres- sion and acceleration. arXiv preprint arXiv:2306.00975 2023.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z.., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Lin, Y., Tang, H., Yang, S., Zhang, Z., Xiao, G., Gan, C., and Han, S. Qserve: W4a8kv4 quantization and sys- tem co-design for efficient Ilm serving. arXiv preprint arXiv:2405.00 045320. 2024.
Raffel, C.,, Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W. and Liu, P. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-555 2020.
Liu, R., Bai, H., Lin, H., Li, Y., Gao, H., Xu, Z., Hou, L., Yao, J., and Yuan, Intactkv: Improving large language model quantization by keeping pivot tokens intact. arXiv preprint arXiv:2403.01241 2024a.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9))99-106, 2021.
Liu, R., Sun, Y., Zhang. M., Bai, H., Yu, X., Yu, T., Yuan, C.,, and Hou, L. Quantization hurts reasoning? empirical study on quantized reasoning models. arXiv preprint arXiv:2504.0455 008223, 2025.
Shao, W., Chen, M., Zhang, Z., Xu, P., Zhao, L.., Li, Z., Zhang, K., Gao, P., Qiao, Y., and Luo, P. Omniquant: Omnidirectionally calibrated quantization for large lan- guage models. arXiv preprint arXiv:2308. www.:2388.11377.. 13137, 2023.
Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. Kivi: A tniig-ffree asym- metric 2bit quantization for ky cache. arXiv preprint 2XXv:2402.00250. 2024b.
Shen, S., Dong, Z., Ye, J., Ma, L.., Yao, Z., Gholami, A., Mahoney, M. W., and Keutzer, K. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pp. 8815 8821, 2020.
Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Kr- ishnamoorthi, R., Chandra, V., Tian, Y., and Blankevoort, T. Spinquant-IIm quantization with learned rotations. arXiv preprint arXiv:2405.16400 2024c.
Sun, M., Liu, Z., Bair, A., and Kolter, Z. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.1169) 2023.
Ma, X., Fang, G., and Wang, X. LIm-pruner: On the struc- tural pruning of large language models. Advances in neural information processing systems, 36:21702-21720, 2023.
Sun, M., Chen, X., Kolter, J. Z., and Liu, Z. Massive activations in large language models. arXiv preprint arXiv:240..1766, 2024.
Tillet, P., Kung, H.-T., and Cox, D. Triton: an inter- mediate language and compiler for tiled neural net- work computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pp. 10-19, 2019.
Ma, Y., Li, H., Zheng, X., Ling, F., Xiao, X., Wang, R., Wen, S., Chao, F., and Ji, R. Affinequant: Affine transformation quantization for large language models. arXiv preprint arXiv:2403.12544, 2024.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In International Conference on Learning Representations, 2016.
Touvron. H., Martin, L., Stone, K., Albert, P., Almahairi. A.. Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhos- ale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Nagel, M., Amjad, R. A., Van Baalen, M., Louizos, C., and Blankevoort, T. Up down? adaptive rounding for post-training quantization. In International Conference on Machine Learning, pp. 7197-7206. PMLR, 2020.
Tseng, A., Chee, J., Sun, Q. Kuleshov, V., and De Sa, C. Quip#: Even better Ilm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.04396 2024.
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N.-
Q. Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The lambada dataset: Word predic- tion requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguisties, pp. 1525 1534, 2016.
Wei, X., Zhang, Y., Zhang. X., Gong, R., Zhang, S., Zhang, Q., Yu, F., and Liu, X. Outlier suppression: Pushing the limit of low-bit transformer language models. Advances in Neural Information Processing Systems, 35:17402- 17414, 2022.

---

FLATQUANT: Flatness Matters for LLM Quantization
Wei, X., Zhang, Y., Li, Y., Zhang, X., Gong, R., Guo, J., and Liu, X. Outlier suppression+: Accurate quantization of large language models by equivalent and effective shift- ing and scaling. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Wolf, T. Huggingface's transformers: State-of-the- art natural language processing. arXiv preprint arXiv: 191..3777.,,7 2019.
Xi, H., Li, C., Chen, J., and Zhu, Training transform- with 4-bit integers. Advances in Neural Information Processing Systems, 16:49146-49166 2023.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference Machine Learning. pp. 38087-38099. PMIR, 2023.
Yang. A.. Yang, B., Hui, B., Zheng, B.. Yu, B., Zhou, C., Li C., Li, C., Liu, D., Huang, F, et Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024.
Ye, Z. Flashinfer: Kernel library for Ilm serving. 2023. URL https: https:///ithbb com/ flashinfer-ai/ flashinfer.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sen- tence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791- 4800, 2019.
Zhang, Y., Bai, H., Lin, H., Zhao, J., Hou, L., and Cannis- traci, C. ald-and-play: efficient post-training prun- ing method for large language models. In The Twelfth International Conference Learning Representations, 2024.
Zhao, Y., Lin, C.-Y., Zhu, K., Ye, Z.,, Chen, I.., Zheng, S., Ceze, L., Krishnamurthy, A., Chen, T., and Kasikci, B. Atom: Low-bit quantization for efficient and accurate Ilm serving. Proceedings of Machine Learning and Systems, 6:196-209 2024.
12

---

FLATQUANT: Flatness Matters for LLM Quantization
A. Related Work
Quantization for Large Language Models. Quantization crucial technique for reducing memory footprint and accelerating inference by employing fewer bits for storage and computation. Unlike pruning (Ma et al., 2023; Sun et al., 2023; Zhang al.. 2024; Chen al., 2025), quantization does not alter the network architecture and is usually more competitive in performance under the same compression ratio. Different from the previous pretrained language models (Shen et al., 2020; Bai et al., 2020; 2022). LLMs are shown to exhibit outliers in activation and massive outliers in pivot tokens (Wei et al., 2022; Dettmers et al., 2022; Liu et al., 2024a; Sun et al., 2024), which can severely degrade quantization accuracy, especially on on complex reasoning tasks (Liu et al., 2025). To eliminate the negative impact of outliers, pre-quantizatinn transformations have been widely adopted in weight- quantization (Xiao et al., 2023; Wei et al., 2023; Shao et al., 2023; Ma et al., 2024; Ashkboos et al., 2024; Liu et al., 2024c) as well as in fully quantized training (Xi et al., 2023). Additionally, several weight-only quantization methods (Lin et al., 2023; Chee et al., 2024; Tseng et al., 2024) incorporate pre-quanizzation transformations. Searched or learnable clipping thresholds for weights activations (Lin et al., 2023; Duanmu al., 2024; Ashkboos et al., 2024; Liu al., 2024c; Shao et al., 2023) are also explored to eliminate outliers.
Per-channel Scaling Transformation. SmoothQuant (Xiao et al., 2023) employs per-channel scaling to shift the challenge of quantization from activations to weights in weight-activation quantization. Building on this, Wei et al. (2023) additionally introduces wise shifting, while OmniQuant (Shao et al., 2023) utilizes a differentiable approach to learn optimal scaling and shifting parameters. However, the scaling-based methods can negatively impact weight quantization and struggle in low-bit settings, such as W4A4 quantization.
Hadamard and Orthogonal Transformation. Recent research (Xi al., 2023; Tseng et al., 2024; Ashkboos et al., 2024) has shown that the Hadamard transformation is effective in eliminating outliers and lowering the quantization error by redistributing outliers across all channels through matrix multiplication. QuaRot (Ashkboos et al., 2024) is the first to apply Hadamard transformation in the LLM W4A4 PTQ setting, while SpinQuant (Liu et al., 2024c) exploits learnable orthogonal matrices with model-level loss to further alleviate outliers.
Affine Transformation. Considering that per-channel scaling corresponds to the diagonal elements of the affine trans- formation matrix, AffineQuant (Ma et al., 2024) proposes learning the equivalent affine transformation. However, their approach focuses learning full-size diagonally dominant matrices and employs gradual mask optimization method, which may hinder the full potential of affine transformation in reducing quantization loss. Moreover, due to the formidable overhead associated with full-size matrix multiplication, Affineuuant can only apply affine transformation to small fraction of linear layers. In contrast, we employ fast and learnable affine transformations without these limitations, leading to substantial accuracy improvements and practical speedup.
Pre-quantization Transformations in Other Quantization Tasks. Inspired by SmoothQuant, AWQ (Lin et al., 2023) introduces activation- -aware per-channel scaling to reduce quantization errors in weight-only quantization. QUIP (Chee al., 2024) and its extension, QUIP# (Tseng et al., 2024), leverage random rotation matrices or Hadamard transformations to enhance incoherence in weight-only quantization. In fully quantized training task, (Xi al., 2023) propose to utilize block-diagonal transformation consisting of Hadamard matrices to reduce the quantization error.
B. Implementation Details
B.I. Matrix Inversion and Training Cost
A critical aspect to implement FLATQUANT is the computation of the inverse affine transformation matrix P 1, As discussed below, we use singular value decomposition (SVD) and automatic mixed precision to train FLATQUANT, enjoying both training stability and efficiency.
Direct Inversion and FP32 Training. One straightforward approach is to use the inverse function provided by PyTorch. However, we find that the precision of this inverse function at FP16 is insufficient. Specifically, PP-1 does not closely approximate I. The off-diagonal elements are on the order of 1 x 10-3, which negatively impacts FLATQUANT's performance during the early stages of training. Therefore, a simple solution is to conduct training in FP32 without Automatic Mixed Precision (AMP) to maintain precision. However, this inevitably increases training time and more GPU memory consumption.
13

---

FLATQUANT: Flatness Matters for LLM Quantization
SVD and AMP Training. To further reduce resource requirements during calibration, we propose to employ singular value decomposition for the affine transformation. For any real matrix P, we decompose it as P UEVT, where U and V are orthogonal matrices, and Σ is diagonal matrix. This formulation allows us to easily compute P V offering a more computationally efficient method for obtaining the inverse. Notably, this approach reduces the off- off-diagonal elements of PP-1 to the order of x 10-6 at FP16 precision, enabling us to utilize AMP during calibration. With AMP, we can achieve a 50% reduction in training time and memory usage while maintaining nearly lossless accuracy in most cases, For the orthogonal matrices U and V, we employ the Cayley parameterization provided by PyTorch
Comparison of the Two Training Recipes. We compare the two training recipes in Table 4. As shown, FP32 training requires more than twice the time of AMP training and necessitates 1.28x more GPU memory under the same setting, while the performance remains relatively close. Thus, our default choice is the SVD approach combined with AMP training. However, we observe that in certain models extremely low-bit scenarios, numerical errors may occur within the AMP framework. In such cases, full-precision training becomes necessary.
Training Recipe
WikiText-2 PPL
C4
QA Acc
Memory
Time
Inverse
6.95
11.04
71.35
35384MiB
2.2 hours
FP32
SVD
9.96
11.07
71.24
35360MiB
2.2 hours
Inverse
7.00
11.17
70.57
27624MiB
0.9 hours
AMP
SVD
6.98
11.13
71.23
27554MiB
0.9 hours
Table 4: Comparison of different training recipes for FLATQUANT on the LLMMA-3-8B.
Calibration Time. We further present the calibration time required by FLATQUANT for the LLaMA family in Table 5. Compared to SpinQuant (Liu et al., 2024c) and QAT methods, FLATQUANT requires significantly fewer computational resources and less training time, while delivering superior performance. For weight-only quantization, only transformations related to the linear weights are introduced, resulting in a shorter calibration time compared to weight-activation quantization. Moreover, as discussed in Section 4.2, FLATQUANT does not need to be combined with GPTQ to achieve optimal performance, further reducing the calibration overhead.
LLaMA
2-7B
2-13B
2-70B
3-8B
3-70B
weight-activation
1.15 hours
1.55 hours
6.15 hours
0.90 hours
5.94 hours
weight-only
0.67 hours
1.01 hours
5.00 hours
0.70 hours
4.89 hours
Table 5: Calibration time for LLaMA models. The reported times correspond to training on 128 segments of 2048 tokens
over 15 epochs with a batch size 4, using a single GPU.
B.2. Overhead Analysis of Affine Transformations
Total FLOPs of Online Transtormations. (1) Self-Attention. The self-attention module has three online transformations, i.e., (Pa, P.. Pr). Suppose the hidden dimension ha and intermediate dimension h; of LLM can be perfectly decomposed into Vha x Vha and Vhu x Vh,, respectively, then the total FLOPPs of (Pa. P., P., is Abshay/hi ha + 2bshha + 4bsh o. where is the batch size, s is the sequence length, and a is the number of attention heads. (2) forward module has two online transformations, i.e.. (P (Pag. Pa). The total FLOPs of aug. Pa} is 4bshav/h + 4bsh; VTg. In summary, the total FLOPs of the online transformations in a Transformer block amounts to 8bshav ha + 2bsh + 4bsh3/a a 4bsh; / /T. In LLaMA-2-7B (i.e., ha 4096, h; 11008 and 32), the FLOPS of online transformations only account for about 2.61% 5 of those of the FP16 model when reaches 2048.
Network. The feed-
hhttps://pytorchh.t org. stable
http:///a/a/....aao/////aa/////r /ttgg//eeeettrrrtttteee/o nn. ut parametri zat ons. orthogonal.
html
14

---

FLATQUANT: Flatness Matters for LLM Quantization
Tensers HBM
Tensors SRAM
Kernels executed on SRAM
For-loop slacing within thread block
Vig
Y0g
f12
11
X
P
P
P
HIBM
HBM
P
X
122
Piyy
Tig
F1
X
X
P2
X
P2
11g
SRAM
SRAM
Quantiao
Quantize
SRAM
Gather Quantize
SRAM
CUTLASS INT4
CUTLASS INT4 Kernel
CUTLASS INT4 Kemel
(a) Default Design.
(b) Corner Case
(c) Corner Case 2.
Figure 8: The visualization of the kernel fusion in FLATQUANT based the computation within thread block. The design holds mainly for (a), where both transformations and quantization are fused together. For completeness, we also revise the design for corner cases in (b) and (c), when the SRAM is not large enough to hold the intermediate results.
Memory Consumption of Online Transformations. We compute the parameter count of each online transformation below: (I) Pa: 2 (2) Po: a²: (3) Pи: (ha/a)?; (4) Pug: 2(vha)?; (5) Pa: 2(Vh,)². The total parameter count in one Transformer block is 4ha + 2h, + a² + (ha/a)². The additional memory consumption during inference is 2(4ha + 2h, + a² + (ha/a)²) bytes, which only consumes about 3.41MB extra memory space for LLaMA-2-7B.
B.3. Detailed Design of Kernel Fusion
To avoid redundant memory access and improve computational efficiency, we attempt to fuse Q(P x X X2 P2) into single kernel, followed by the INT4 CUTLASS kernel to multiply the 4-bit quantized weights and activations. In most cases, the shared memory per thread block is sufficient to hold the source matrices P1, P2. X, and their intermediate results X as visualized in Figure 8a. Nonetheless, there are corner cases when the shared memory insufficient to hold all necessary tensors (e.g., n > 28762 with n1, n2 > 128 on the NVIDIA RTX 3090). We thus revise our design for the two cases, as shown in Figure 8b and Figure 8c, respectively, To distinguish these scenarios more clearly, we have the following equations:
Default Design:
(n1 ma + 2
122) 2 < (n2 * n2 + 2 k * 122) * 2 < m
(5)
Corner Case 1:
(tn, n1 + nı * n2 + tR, * m22) # < m
(n2 n2
tn, * n22) #2 < m
(6)
Corner Case 2:
(m bng bmg 12 n1 122 2 <
(n1 bng bmg 12 + nn n22) 2 <
(7)
where m is the shared memory size per thread block, tn, the tiling size of non-reduction dimension of P1, ba, is the tiling size of reduction dimension of Pi, bng is the tiling size of reduction dimension of Pa and 2 refers to two bytes to hold tensors in float16. Below we review the designs for the two corner cases respectively.
Corner Case When both and n are excessively large, is suggested prevent from loading the entire P1 and X into SRAM. We manage this by tiling the non-reduction dimension of P1 into fn, slices. This strategy enables to integrate P, XPP,, into one kernel, with P, representing slice of P1 on the non-reduction dimension. Subsequently, we invoke separate fused kernel for quantization, computing the quantization scale and scaling the input.
Corner Case 2. When both n and n2 are extremely large, P,, X and P2 cannot be loaded into SRAM together. To handle this, we first compute X = P. X, where each thread block slicing the non-reduction dimension of P1 and with the tiling shape bng- The output X written back the global memory, and the SRAM memory is thus released. Next, we
15

---

FLATQUANT: Flatness Matters for LLM Quantization
slice the non-reduction dimension of X and P2 with tiling size Uug and compute the matrix multiplication, followed by quantizing the result on the fly.
Kernel Profiling. We enumerate popular hidden sizes in the series of LLaMA models, and provide the detailed profiling results of FL.ATQUANT'S online transformation with and without kernel fusion in Table 6. Note that the SRAM can hold all of these shapes with the default design on the NVIDIA RTX 3090. It can be found that kernel fusion achieves significant speedup across various hidden dimensions and batch sizes, e.g.. 1.5x-3x prefill speedup and 1.2x-4x decoding speedup, respectively. We also selectively test the two corner cases with the hidden size of 28762, both of which bring considerably 2.3x speedup.
without Kernel Fusion
with Kernel Fusion
Speedup
Hidden Dimension
Batch Size
Prefill Time (ms)
Decode Time (ms)
Prefill Time (ms)
Decode Time (ms)
Prefill
Decode
1
0.1956
0,0184
0.0625
0.0082
3.13x
2.25x
2
0.3809
0,0195
0.1116
0.0072
3,41x
2.71x
4
0.7199
0,0212
0.2120
0.0082
3.40x
2.59x
4096
8
1.4019
0,0236
0,4188
0.0082
3.35x
2.88x
16
2.7628
0.0307
0.8417
0.0073
3.28x
4.20x
32
5.5101
0.0317
1.7091
0.0082
3.22x
3.87x
64
10.9752
3.4898
0.0082
3.14x
4.00x
0,2519
0.0195
0.1321
0.0113
1.91x
1.73x
2
0.4915
0,0205
0.2570
0.0113
1.91x
1.82x
4
0,9073
0,0225
0.5161
0.0113
1.76x
2.00x
5120
8
1.7582
0,0266
1.0363
0,0113
1.70x
2.36x
16
3,4748
0,0338
2,0480
0,0121
1.70x
2.80x
6.9079
0.0358
4.1313
0.0123
1.67x
2.92x
64
13.8619
0.0379
8,2033
0.0123
1.69x
3.08x
1
0.3845
0.0195
0.1608
0.0132
2.39x
1.48x
2
0,7393
0,0205
0,3092
0.0132
2.39x
1.55x
1,4433 4
0,0205
0.6257
0.0123
2.31x
1.67x
8192
8
2.8529
0.0215
1.2411
0.0133
2.30x
1.62x
5.6668 16
0.0225
2.4904
0.0133
2.28x
1.69x
32
11.3183
0.0246
4,9418
0,0133
2.29x
1.85x
64
22.6714
0.0297
9.8459
0.0143
2.30x
2.07x
1
0.6154
0.0215
0,3830
0.0173
1.61x
1.24x
2
1.2032
0.0225
0.7547
0.0173
1.59x
1.30x
2,3654 4
0.0223
1,5032
0.0164
1.57x
1.36x
11008
8
4.7570
0,0236
2.9983
0.0174
1,59x
1.35x
9,4536 16
0,0256
6.0099
0.0184
1.57x
1.39x
32
18.9102
0,0287
12.04 04.044
0.0195
1.57x
1.47x
64
38,2700
0.0379
24,0000
0.0248
1.59x
1.53x
0.7260
0.0225
0.4444
0.01
1.63x
1.22x
2
1.4203
0.0236
0,8653
0.0184
1.64x
1.28x
4
2,8088
0.0246
1.7254
0.0184
1.63x
1.33x
13824
8
5.6228
0.0247
3,4273
0.0195
1.64x
1.27x
16
11.2297
0,0266
6.8726
0.0195
1.63x
1.37x
32
22.4302
0.0319
13.7216
0.0205
1.63x
1.56x
64
45.4374
0.0471
27.4698
0.0275
1.65x
1.72x
0.6932
0.0215
0.4178
0.0184
1.66x
1.17x
2
1,3466
0.0225
0.8233
0.0184
1.64x
1.22x
4
2.6557
0.0236
1.6507
0.0184
1.61x
1.28x
14336
8
5.2910
0.0246
3.2922
0.0195
1.61x
1.26x
16
10.5185
0.0257
6.5966
0.0195
1.59x
1.32x
32
20.9249
0,0317
13.0601
0.0205
1.60x
1.55x
64
42.7981
0.0461
25.9308
0.0266
1.65x
1.73x
Table Prefill and decoding speedup kernel fusion across different hidden dimensions and batch sizes. The sequence length is 2048 for prefill and for decoding. The default kernel design holds for all the above settings.
16

---

FLATQUANT: Flatness Matters for LLM Quantization
C. Additional Experiments
C.1. Results on Other LLM Architectures
Aside from the pre-trained LLaMA models, we also investigate the quantization 1---A-.------nsttteee representative of the instruction iutiionn-tnednIe LLM. The perplexity language modeling and the aceuracy of QA tasks for LLaMA-3. --Stttr shown in Table 7, where FLATQUANT again outperforms QuaRot by large margin.
Results on LLaMA-3.1-BB-Iniiie performance
WikiText-2
C4
ARC-C
ARC-E
HellaSwag
LAMBADA
PIQA
Winogrande
Avg
FP16
7.22
11.38
55.20
79.67
79.20
73.14
81.12
73.80
73.69
QuaRot
9.25
15.13
45.39
73.15
73,45
66.41
76.01
66.61
66.84
FLATQUANT
7.97
12.99
52.90
79.25
76.68
70.79
79.49
73.09
72.03
Table 7: Evaluation results of FLATTQANT L.LMA-3.1-8B-Imm-t
Results on Qwen-2.5-Instruct. In addition the series of LLaMA models, we also validate FLATQUANT on Qwen-2.5 Instruct, including both the 7B and 32B models. The results language modeling and QA benchmarks are summarized in Table 8. For the 7B model, FLATQUANT achieved a slightly lower average performance compared to the FP16 baseline, with an average score of 68.62. For the 32B model, TLATQUAANaISS demonstrates competitive performance, achieving an average score of 74.89 (e.g., merely 0.21% % drop), which is slightly lower than the FP16 baseline but higher than the QuaRot.
Model
Method
Quantizer
WikiText-2
C4
ARC-C
ARC-E
HellaSwag
LAMBADA
PIQA
Winogrande
Avg
FP16
8.36
14.37
51.37
75.80
79.57
67.61
80.20
69.93
70.75
7B
FLATQUANT
RTN
13.94
$1.71
77.69
78.42
57.46
76.93
68.62
FP16
5.32
10.45
58.62
77.02
85.25
75.14
81.39
73.16
32B
QuaRot
RTN
6.95
74.37
80.41
78.45
67.72
QuaRot
GPTQ
6.54
11.65
56.06
76.52
81.83
71.26
78.78
69.06
72.25
FLATQUANT
RTN
5.80
10.86
58.62
78.58
83.72
75.26
80.74
72.45
74.89
Table 8: Evaluation results of FLATQUANT on -.ntntruttttt models.
Results on DeepSeek-V3- V3-Base and DeepSeek-R1. We further scale the evaluation of FLATQUANT to DeepSeek V3-Base and DeepSeek-RI, both of which large-scale Mixture-of-Expeer (MoE) models of 671B parameters. Table 9 presents the results under 4-bit weight and activation quantization. It can be found that FLATQUANT maintains strong performance across both LLMs, demonstrating its applicability beyond standard dense LLMs.
Model
Quantization
C-Eval
MMLU
AIME2224
FP8
90.10
87.10
DeepSeek V3-Base
FLATQUANT-WANL
89.59
86.32
FP8
79,8
DeepSeek RI
FLATQUANT-WAA
73.3
Table Evaluation results of FLATQUANT on LLaMA-3, 1-1---3.--BB---tteee
C.2. Results on MT-Bench
Aside from language modeling and question answering, we also evaluate FLATTQANT on MT-Beench with LLaMA-3.1 1-BBB Instruct model in Table 10. We use GPT-4o as the evaluator to justify the ability multi-turn conversation. It can be found that while FLATQUANT trails behind the FP16 baseline coding and STEM, it consistently outperforms QuaRot with GPTQ across all categories, narrowing the gap between the quantized model and the FP16 baseline. Notably, for math problems, FLATTQANT matches the FP16 baseline's score, exceeding QuaRot by 1.9 points.
17

---

FLATQUANT: Flatness Matters for LLM Quantization
Method
Writing
Roleplay
Reasoning
Math
Coding
Extraction
STEM
Humanities
Avg
FP16
8.17
8.10
5.05
7.00
6.10
8.67
8.50
8.91
7.60
QuaRot
7.20
6.90
3.90
5.30
4.05
6.70
6.05
7.80
5.99
FLATQUANT
7.95
7.35
4.70
7.20
4.80
7.60
7.20
8.70
6.94
Table 10: MT-Bench results of 4-bit weight & activation quantized LLaMA-3.1-8B -nnssruuuttttt model.
C.3. Extension More Quantization Settings
Weight-only Quantization While our primary analysis focuses on joint full weight-activatiin--veeess quantization schemes (Section 4.2), FLATQUANT demonstrates notable versatility across different quantization paradigms. Table 11 presents the results of the weight-only quantization compared several state-of-the-art baselines in bits and 3 bits. We adopt per-channel symmetric quantization for weights, maintaining consistency with full quantization scheme's weight processing methodology. FLATQUANT again obtains leading accuracy compared with leading baselines. Specifically, it outperforms RTN, GTPQ, and AWQ, while also slightly surpassing GPTQ with ger-group quantization (group size 128). Meanwhile, FLATQUANT performs comparably to QuIP. Additionally, we report results for FLATQUANT when combined with GPTQ, where GPTQ utilizes the same calibration data for both closed-form weight updates and training. The findings remain consistent with the results joint full setttt-accivatioovvvaatee quantization schemes.
WikiText-2 PPL
C4 PPL
LLaMA-3-8B
W4A16
W3A16
W4A16
W3A16
FP16
6.14
9.45
RTN
8.70
2.2E3
14.00
5.6E3
GPTQ
7.00
13.00
11.80
45,90
GPTQ-g128
6.50
8,20
10.40
13.70
AWQ
7.10
12.80
10.10
16.80
QuIP
6.50
7.50
11.10
11.30
FLATQUANT-RTN
6.54
7.78
10.17
12.64
FLATQUANT-GPTQ
6.48
7.52
10.28
12.91
Table 11: WikiText-2 and C4 perplexity of weight-only quantizationon on I.LMMA3-8B model.
K bits
V bits
WikiText-2
C4
ARC-C
ARC-E
HellaSwag
LAMBADA
PIQA
Winogrande
Avg
16
16
6.14
9.45
53.50
77.57
79.12
75.51
80.74
72.93
73.23
6.20
9.56
52.82
78.20
79.13
75.32
80.47
72.77
73.12
4
6.25
9.66
52.90
77.65
79.00
75.10
80.79
73.48
73.15
4
2
6,60
10.33
49.32
74.37
77.88
72.77
79.22
72.69
71.04
6.35
9.91
52.05
77.95
78.41
73.94
79.71
73.48
72.59
6.41
10.03
52.47
76.85
78.25
74.02
79.98
72.61
72.36
3
2
6,84
10.83
47.44
73.91
77.18
70.37
78.73
71.19
69.80
7.70
13.36
49.15
74.62
74.74
63.65
77.58
68.67
68.07
2
7.79
13.44
46.67
71.63
74.17
63.05
77.48
68.51
66.92
8.93
16.13
42,92
68.60
71.54
55.58
75.30
64.40
63.06
Table 12: Different bits for KV cache quantization on the LLaMA-3-8B model.
KV Cache Quantization. To further evaluate its versatility, we apply FLATQUANT to KV cache only quantization. In this setting, we retain high precision for the rest of the model (including weights and activations) and apply the group-wise asymmetric quantization (with a group size of 128) to keys and values. Table 12 presents the results of KV cache quantization using various bit-widths on the LLaMA-3-8 model. Consistent with previous studies (Hooper et al., 2024; Liu et al., 2024b; Ashkboos et al., 2024), we observe that keys are more sensitive quantization than values. Furthermore, Table 13
18

---

FLATQUANT: Flatness Matters for LLM Quantization
K bits
bits
LLaMA-2-7B
LLaMA-2-138
Methods
16
16
5.47
4.88
5.51
4.91
QuaRot
3
3
5.68
5.02
2
2
9.23
7.07
5.50
4.91
FLATQUANT
3
3
5.61
5.00
2
2
6.66
5.69
Table 13: WikiText-2 perplexity of LLaMA-2 models with different bits of KV cache quantization.
compares FLATQUANT with QuaRot for KV cache quantization on LLaMA-2-7B and LLaMAA2-13B models. As shown, FLATQUANT delivers superior performance in most cases, particularly for lower-bit (2-3 bits). When both keys and values are quantized bits, FLATQUANT outperforms QuaRot by 2.57 perplexity for the 7B model.
Extreme Low-bit Quantization. We quantize the LLM extreme low-bit representations (e.g., INT3) to investigate the limitations of quantization. The results in Table 14 show that FLATQUANT still keeps most of the model's abilities in the 3-bit setting, whereas QuaRot struggles under such extreme low-bit conditions. Nevertheless, 4-bit quantization remains a better balance between inference resource efficiency and acceptable performance degradation for now.
LLaMA3-BB
WikiText-2
C4
ARC-C
ARC-E
HellaSwag
LAMBADA
PIQA
Winogrande
Avg
FP16
6.14
9.45
53.50
77.57
79.12
75.51
80.74
72.93
73.23
DuaRor-W4A4KK
8.16
13.38
45.73
70.83
72.97
62.70
75.35
67.17
65.79
PLATQUANT-WAAKKV
6.98
11.13
50.00
75.80
76.80
72.91
79.16
72.69
71.23
BnaRot-WAARKV3
686.54
630.89
25.34
28.41
28.07
0.78
50.71
48.70
30.33
FLATQUANT-W3ARKV3
10.82
19.03
35.41
63.26
65.30
52.49
73.56
60.69
58.45
Table 14: Extreme low bit quantization results on LLAMA-3-8B models.
Train One and Get More. Remarkably, we demonstrate that the affine transformations learned from weight-activation quantization can be directly applied to other quantization settings, such as weight-only KV cache quantization, with surprisingly strong performance. The associated results presented in Table 15, For instance, the results labeled "W4" are comparable to those in Table 11 that specifically trained for weight-only quantization. This significantly saves time when applying FLATQUANT to different quantization settings, as only one set of transformation matrices is saved.
A4
KV4
WikiTeext2 PPL
C4 PPL
QA Acc
6.14
9.45
73.23
6.56
10.25
72.92
6.49
10.13
72.20
6.23
9.61
73.43
6.98
11.13
71.23
Table 15: Extending the affine transformations trained under W4A4KV4 to different quantization settings on LLaMA-3-8B model. QA Acc is the average accuray of the six QA tasks in
C.4. Detailed Ablation Study
To better disentangle the contributions of affine transformations and clipping thresholds, we present the full ablation results in Table 16, as an extension of the analysis in Table 3. The results highlight the effectiveness of each component in FLATQUANT, with learnalle transformations (LT) playing central role. Moreover, when built upon LT, other components further enhance the overall performance of FUATQUANT.
19

---

FLATQUANT: Flatness Matters for LLM Quantization
LT
PS
LCT
WikiText-2
ARC-C
ARC-E
HellaSwag
LAMBADA
PIQA
Winogrande
Avg.
1266.60
936.41
25.26
28.62
27.04
1.26
51.80
51.93
30.99
NaN
NaN
22.70
25.08
25.04
0,00
49.51
49.57
28.65
1 1149.08
1490.08
22.95
29.29
27.35
0.60
52.99
50.83
30.67
8197.96
4654.07
25.43
25.72
25.96
0.02
50.49
48.86
29.41
8.50
13.51
44.97
71.38
73.17
67.05
76.88
67.48
66.82
7.95
12.74
44.20
71.89
74.21
68.72
77.15
66.30
67.08
7.11
11.47
49.32
76.14
76.30
72.17
78.89
71.51
70.72
6.98
11.13
50.00
75.80
76.80
72.91
79.16
72.69
71.23
Table 16: Ablation study of FLATQUANT's main components on LLaMA-3-8B.
C.5. Additional Discussions
Calibration Set. Since FLATQUANT employs a gradient-based method to optimize transformations for increased flatness, one reasonable concern is whether FLATQUANT might overfit the calibration set. To assess its generalization ability. we conducted an ablation study using different calibration datasets: WikiText-2, C4, and Pile. As shown in Table 17, FLATQUANT maintains stable performance across all datasets. For example, when calibrated on different datasets, FL.ATQUANT exhibits similar performance on WikiText-2, with PPL. ranging from 6.98 to 7.04. On the C4 dataset, results equally consistent, with PPLs between 11.05 and 11.13. Furthermore, QA accuracy remains within narrow range (71.04% 71.23%), suggesting that FLATQUANT generalizes well across different calibration datasets. This robustness is attributed to FL.TTQUNNT, focus on learning an equivalent affine transformation with minimal quantization loss, rather than altering the model's weights. Nevertheless, it is reasonable to assume that the diversity of calibration data can further enhance the performance of our method.
Calibration set
WikiTexx-2
C4
ARC-C
ARC-E
HellaSwag
LAMBADA
PIQA
Winogrande
Avg
WikiText2
6.98
11.13
50.00
75.80
76.80
72.91
79.16
72.69
71.23
7.04
11.05
50.34
75.38
76.7
73.28
78.67
71.82
71.04
Pile
7.04
11.08
51.11
77.36
76.63
72.37
78.94
70.56
71.16
Table 17: Ablation study FLATQUANT's calibration set on LLaMA-3-80 model.
Effect of Clipping. While weight clipping has been widely adopted in LLM quantization, activation clipping remains relatively underexplored. Prior works (Ashkboos et al., 2024; Liu et al., 2024c) have shown that activation clipping alone yields limited benefits, primarily due to the presence of severe outliers. In contrast, our method demonstrates that learnable clipping thresholds (LCT), when applied after transformation, yields significant improvements. As shown in Table 18, applying LCT before transformation, similar to the RTN--ttyle approach, yields only marginal gains. This observation is consistent with prior findings (Dettmers et al., 2022), suggesting that early clipping fails to effectively suppress activation outliers. In our method, the affine transformation redistributes outliers across channels, enabling the subsequent LCT step to more effectively clip larger portion of extreme values. Crucially, the inverse transformation retains the ability recover the original scale of informative signals, thereby preserving model quality after quantization. For comparison, also report results using QuaRot-style clipping strategy (with thresholds of 0.9 for activations and 0.95 for KV cache values). Overall, these results highlight that the integration of learnable transformations and adaptive clipping significantly enhances the performance of weight-activation quantization.
LLaMA3-8B
WikiText-2
C4
ARC-C
ARC-E
HellaSwag
LAMBADA
PIQA
Winogrande
Avg
FP16
6.14
9.45
$3.50
77.57
79.12
75.51
80.74
72.93
73.23
7.95
12.74
44.20
71.89
74.21
68.72
77.15
66.30
67.08
LCT hefore Transformatinn
7.37
11.86
48.72
76.18
75.1
66.65
77.91
67.17
68.62
QuaRot-style Fixed Threshold
7.25
11.62
48.21
75.29
75.66
71.32
78.73
70.01
69.87
LCT after Transformation
6.98
11.13
50.00
75.80
76.80
72.91
79.16
72.69
71.23
Table 18: The effect of Learnable Clipping Thresholds.
20

---

FLATQUANT: Flatness Matters for LLM Quantization
Effect of Mixed-preee Mixed-precksion Schemes. To further evaluate the practicality of FlatQuant, evaluate its performance under mixed-precision quantization scheme. Specifically, we explore how integrating FlatQuant into layer-wise heterogeneous bit-width setting can improve accuracy while retaining high inference speedup. As shown in Table 19, we apply W8A8 quantization to the top 5 important Transformer layers and all down- nown-projection layers, based on their relative importance (Kim et al., 2024). This configuration substantially mitigates the degradation observed in uniform low-bit settings. These results indicate that FlatQuant can be effectively combined with mixed- muxed-precision strategies, enhancing its applicability in real-world deployments.
WikiText-2
C4
ARC-C
ARC-E
HellaSwag
LAMBADA
PIQA
Winogrande
Avg
FP16
6.14
9.45
53.50
77.57
79.12
75.51
80.74
72.93
73.23
FLATQUANT
6.98
11.13
50.00
75.80
76.80
72.91
79.16
72.69
71.23
down.proj_80nl Bbits
6.73
10.62
50.00
77.78
77.49
73.96
79.54
70.56
71.55
Top5_ Shits
6.80
10.82
49.23
76.56
76.54
73.51
79.71
73.95
71.58
Top5 & nown.proj.8hhr
6.61
10.43
51.11
77.74
77.47
74.69
79.92
72.14
72.18
Table 19; The effect of mixed-precision quantization with selectively higher-bit layers in FlatQuant.
C.6. Experiment Details of Figure 5
In Figure 5, we present the prefill speedup and WikiText2 PPL. results of different decomposed matrix sizes on LLaMA-2-7B model. We decompose the hidden dimension 4096 into n1 x n2 and range n1 from to 2048, where n1 = amounts to maintaining a full-size transformation matrix. The intermediate dimension 11008 decomposed into 64 x 172 as done in FUATQUANT. For PPL evaluation, we only quantize the last Transformer block and learn the affine transformations within it. For speedup evaluation, we do not leverage the online transformation kernel in Section 3.3 and implement online transformations with naive matrix multiplication in PyTorch.
C.7. Analysis of Inference Memory
To further validate the efficiency of FLAATQUANT, provide additional results its inference-time memory consumption. These experiments complement our theoretical analysis in Appendix B.2, and empirically confirm that the online affine transformations introduced in FlatQuant incur negligible memory overhead.
Table 20 reports the peak memory usage during single-token decoding on a single Transformer layer the LLaMA-2-7B model, under varying KV cache lengths, with batch size set to I. We compare standard FP16 inference, INT4 quantization, and our method, FlatQuant.
shown, FlatQuant matches the memory efficiency of INT4 quantization across all sequence lengths, achieving a consistent memory reduction of over 3.3x compared FP16. These results indicate that FlatQuant preserves the low memory footprint of INT4 quantization, while introducing additional memory cost from its online processing.
Sequence Length
FP16 (GB)
INT4 (GB)
FlatQuant (GB)
Saving Factor
256
0.393
0.110
0.110
3.58
512
0.399
0.112
0.112
3.56x
1024
0.411
0.118
0.118
3.48
2048
0.434
0.130
0.130
3.35x
Table 20:
Peak memory usage for decoding a single token on one Transformation layer of LLaMA-2-7B model with KV caches of different lengths and batch size
C.8. Additional Analysis of Inference Latency
Baseline. We implement and report the latency results of INT4 quantization and QuaRot with QuaRot's official code These baselines share the same quantization settings with FLATQUANT described in Section 4.1 for fair comparison.
/https://gt thub. com spc1 Rot
21

---

FLATQUANT: Flatness Matters for LLM Quantization
End-to-end Speedup. We decode 256 tokens after the prefill sequence length of 2048 and provide the prefill and decoding speedup of FLATQQUATT in Figure 9 and Figure 10. FLATQUAATT achieves prefill speedup of 2.30x and decoding speedup of 1.76x under the batch size of 64, with only 0.07x speedup loss compared to the naive INT4 quantization for both prefill and decoding. Note that when the batch size is smaller than 16, quantization overhead outweighs the benefits brought by KV cache memory reduction for the decoding stage, resulting in less than 1x speedup for both INT4 quantization and FLATQUANT. However, since the decoding speedup shows good scalability with the batch size, we can gain practical decoding speedup simply by employing a large batch size.
INT4
QuaRot
FlatQuant w/o Kernel Fusion
FlatQuant w/ Kernel Fusion
2.0
2.11x
38x
2.24
2.044
4.0M
2.20c
Speedup
1.5
0.0
16
32
64
Batch Size
Figure Prefill speedup of LLaMA-2-7B on a sequence length of 2048.
INT4
QuaRot
FlatQuant w/o Kernel Fusion
FlatQuant w/ Kernel Fusion
2.00
1.50
Speedup
1.00
1:01x
0.75
6:1
0.50
0.00
64
Batch Size
Figure 10: Decoding speedup LLaMA-2-78 model. We decode 256 tokens after the prefill sequence length of 2048.
Speedup across Sequence Lengths. In addition to the results presented in Figure we further evaluate the runtime speedup of FlatQuant under varying sequence lengths to better characterize its practical efficiency. Table 21 and Table 22 provide detailed prefill and decoding speedups on LLaMA-3-8B, under different input and KV cache lengths. For the prefill stage with batch size 1, FLATQUANT achieves consistent speedups across sequence lengths, achieving 2. 12x at length 2048 and 1.80x at length 16384, comparable to INT4 and outperforming QuaRot. In the decoding stage with batch size 64, FLATQUANT consistently surpasses QuaRot across all KV cache lengths and closely approaches the efficiency of INT4 quantization. These results demonstrate that FLATQUANT maintains its low-overhead advantage across a wide range of generation scenarios, including both short and long contexts.
C.9. Comparison with AffineQuant
AffineQuant also proposed learning an equivalent affine transformation to reduce quantization error. As shown Table 1, AffineQuant and FLATQUANT exhibit markedly different performance. To further clarify why FLATQUANT outperforms Affineuuant, how their differences affect quantization performance, and what factors matter most in quantization, we
22

---

FLATQUANT: Flatness Matters for LLM Quantization
Prefill Length
INT4
QuaRot
FlatQuant
2048
2.16x
1.97x
2.12x
4096
2.06x
1.90x
2.04x
8192
1.94x
1.79x
1.92x
16384
1.83x
1.72x
1.80x
Table 21: Prefill Speedup on LLaMA-3-8B Compared to FP16 for Different Input Sequence Lengths at Batch Size 1.
KV Cache Length
INT4
QuaRot
FlatQuant
256
1.38x
1.09x
1.24x
512
1.62x
1.38x
1.56x
1024
1.70x
1.61
1.63
2048
1.78x
1.72x
1.76x
Table 22: Decoding Speedup on LLaMA-3-8B Compared to FP16 for Different KV Cache Lengths at Batch Size 64.
provide additional analysis below.
Improved Expressivity. Although FLATQUANT adopts Kronecker product with two lightweight matrices, its expressivity remains competitive. shown in Figure the decomposed transformation achieves accuracy comparable full-size affine transformation, suggesting that such structural constraint does not lead to loss in functional capacity. In contrast, AffineQuant enforces strictly diagonally dominant transformations ensure invertibility, which may inadvertently reduce expressivity and make the transformation behave similarly to per-channel scaling (see Figure 7 in the AffineQuant paper).
Applicability to All Linear Layers. FLATQUANT offers broader applicability across the model architecture. Thanks to its lightweight Kronecker structure, it can be efficiently applied to all linear layers with negligible overhead. In contrast, Affineuuant directly learns a full-size affine matrices, which are only feasible to apply to output projection layers due to their high computational cost. For other layers, AffineQuant reverts to simpler per-channel scaling. These practical limitations further constrain its overall effectiveness, whereas FLATQUANT maintains unified and expressive transformation across the entire model.
23

---

FLATQUANT: Flatness Matters for LLM Quantization
D. Additional Visualizations
D.I. More Visualizations Weight and Activation Distributions
Experiment Details. We visualize the distribution of weights and activations after different transformations, including per-channel scaling SmoothQuant (Xiao al., 2023), Hadamard transformation in QuaRot (Ashkboos et al., 2024), and affine transformation in FLATQUANT. We compute the per-channel Frobenius norm quantify the channel magnitude. We randomly sample from the C4 (Raffel al., 2020) dataset collect activation statistics.
Visualizations the LLaMA Models. We visualize the distribution envelopes of both original and transformed weights and activations on the LLaMA models in Figure 11-15. It can be observed that neither per-channel scaling nor Hadamard transformation can fully smooth out outlier channels to produce flatness, still leaving outlier channels, especially on activations. On the other hand, the affine transformation learned by FLATQUANT can effectively produce flatter distributions for both weights and activations which are easier to quantize.
D.2. More Visualizations of Quantization Error Landscapes
Experiment Details. We randomly sample 128 samples from the C4 (Raffel et al., 2020) dataset and compute their average mean squared error for visualization. For per-channel scaling, we follow SmoothQuant (Xiao et al., 2023) and only perform per-channel scaling for the inputs of the self-attention and feed-forward modules. For the Hadamard transformation, replace the affine transformation in FUATQUANT with fixed Hadamard transformation. The quantization settings the same as those described in Section 4.1.
Visualizations the LLaMA Models. We visualize the quantization error landscapes of LLaMA models in Figure 2 and Figure 18-21. With the affine transformation to smooth outliers, FLATQUANT can effectively suppress the quantization errors at pivot tokens and case the quantization error propagation, leading to flatter quantization error landscape compared with per-channel scaling and Hadamard transformation.
E. Limitations
In this study, present FLATQUANT, but there are certain limitations to acknowledge. First, the full potential of 4-bit quantization has not been thoroughly explored. While follow the previous studies to build the calibration set and demonstrate that FLATQUANT is robust across various data sources, the optimal selection of calibration sets remains an open question. Additionally, our focus has primarily been on the INT4 data type, and we have not examined the integration of FLATTQANT with newer data types, such as MXFP4, which may offer advantages over INT4, Addressing these aspects represents promising avenues for future research.
Original
Per-channel Scaling
Hadamard
FlatQuant
Magnia
5.5%
2.0-
Magnitude
25
Magnitu
22.5
Magnitude
60
20
3.0
5.0
0.5
255
10
2000
2000
Channels
Channels
Channels
Channels
(a) W. of the thth Transformer layer LLaMA-2-7B.
(b) X. of the thth Transformer layer in LLaMA-2-7B.
(c)
W, of the 30th Transformer layer in LLaMA-2-7B.
(d) X, of the 3hth Transformer layer LLaMA-2-7B.
Figure 11: Distributions of weights and inputs from LLaMA-2-7B, sorted by the channel magnitudes (i.e., the Frobenius norm) in descending order.
24

---

FLATQUANT: Flatness Matters for LLM Quantization
Original
Per-channel Scaling
Hadamard
FlatQuant
42.5
Magnitude
Ma Magnitude
Wagnitude
Magning
7:5
5.0
25
1000
2000
2000
4000
5000
2.
1000
2000
3000 4000 Channels
5000
1000
2000 3000
4000
5000
1000
2000 2000 4000 5000 Channels
Channels
Channels
(a) W. the 10th Transformer layer in LLaMA-2-13B.
W of the 30 Transformer (b) X. the 10th Transformer layer in LLaMA-2-13B.
(c)
(d) X of the 30th Transformer layer in LLaMA-2-1360
layer in LLaMA-2-138B
Figure 12: Distributions of weights and inputs from LLaMA-2-13BB sorted by the channel magnitudes (i.c., the Frobenius norm) in descending order.
3.9
Magnitudes
Magnitude
Magnitude
Magnitude
15
5.0-
10
0.5-
Channels
Channels
Channels
Channels
(a) W. of the 10th Transformer layer in 2L-BA-2-70B.
(b) X. of the 10th Transformer
(c)
W of the 30 Transformer
(d) X, of the 30 Transformer layer in LLaMA-2
layer in LLaMA-2-70B.
layer in LLaMA-2-70B.
Figure 13: Distributions of weights and inputs from LLaMA-2- 2-70B, sorted by the channel magnitudes (i.e., the Frobenius norm) in descending order.
120
1.8%
1.0-
Magnitude
Magnituck nituded
Magnitude
0.8-
60-
g.s..
20.4-
0.2
2000
2000
3000
Channels
Channels
Channels
Channels
(a) W. of the thth Transformer layer LLaMA-3-8B.
(b) X. of the thth Transformer layer LLaMA-3-8B.
(c) W, of the 30 Transformer layer in LLaMA-3-8B.
(d) X, of the 30 Transformer layer LLaMA-3-8B.
Figure 14: Distributions of weights and inputs from LLaMA-3-8BB sorted by the channel magnitudes (i.e., the Frobenius norm) in descending order.
35
1.50
3.00
Magnitude
Magnitude
10
Magnitude
10
0.50
Channels
Channels
Channels
Channels
(a) W. of the thth Transformer layer in LLaMA-3-70B.
(b) X. of the thth Transformer
(c)
W, of the 30 Transformer
(d) X, of the th Transformer
layer in LLaMA-3-70B.
layer in LLaMA-3-70B.
layer in LLaMA-3-70B.
Figure 15: Distributions of weights and inputs from I.LMA-3-70R, sorted by the channel magnitudes (i.e., the Frobenius norm) in descending order.
25

---

FLATQUANT: Flatness Matters for LLM Quantization
Original
Per-channel Scaling
Hadamard
FlatQuant
Magnitude
Magnitude
100
60
Mag Mlagiitud iigintudde
Magnitude
175
40
50
0
1000
2000
3000
4000
1000
2000
3000
4000
1000
2000 Channels
2000
4000
1000
2000
3000
4000
Channels
Channels
Channels
W the thth Transformer layer in LLaMA-3-8B.
(b) X, the 3hth Transformer layer in LLaMA-3-8B.
(c) of the 30 Transformer layer in LLaMA-3-8B.
X₃ of the 30th Transformer (d) layer in LLaMA-3-8B.
200
350
8/
Magnitude
Magnitudo
175
Magnitude
22.5
Magnitude
342
10.0
330
25
2.5
10-
2000
1000
2000
2000
4000
0.0
5000
10000
15000
0-0
5000
10000
15000
Channels
Channels
Channels
Channels
(e) W. of the thth Transformer layer LLaMA-3-SB.
(1) X. of the 30th Transformer layer in LLaMA-3-8B.
(g)
Wa of the 30th Transformer
(h) X., of the 30 Transformer layer LLaMA-3-8B,
layer in LLaMA-3-8B.
Figure 16: Distributions of weights and inputs from LLaMA-3-8B, sorted by the channel magnitudes (i.e., the Frobenius norm) in descending order.
20
170
170
22.5
Magnitude
hlognitude
Magnitude
Maggitudee
160
20
20
5.0
0.0
6
2000
4:000
6000
5000
2000
40000
8000
8000
2000
4000 Channels
6000
8000
2000
4000
6000
R000
Channels
Channels
Channels
W, the thth Transformer layer in LLaMA-3-700 7-BB.
(b) X, the thth Transformer layer in LLaMA-3- -----B.
(c) W. of the 30th Transformer layer in LLaMA-3-70B.
Xx of the 30th Transformer layer LLaMA-3-708
3.00
Magnitude
Magnitude
Magnitude
Magnitude
1.00
160
10
20
0.50
0.00
2000
4,000
5000
5000
2000
4000
8000
8000
100000
20000
30000
10000
20000
30000
Channels
Channels
Channels
Channels
(e) W. th Transformer layer in LLaMA-3-70B.
(1) X. of the 30th Transformer layer in LLaMA-3-70B.
(g) Wa of the 30th Transformer
(h) X., of the 30 Transformer layer in LLaMA-3-70B.
layer in LLaMA-3-70B.
Figure 17: Distributions of weights and inputs from I.LaMA-3-70BB sorted by the channel magnitudes (i.e., the Frobenius norm) in descending order.
26

---

FLATQUANT: Flatness Matters for LLM Quantization
Per-channel Scaling
Hadamard
FlatQuant
25
0.8
80
0.4
20
0.0
10
10 Layer
10 Layer
10 Layer
15
19 Token
50
100
125
30
20
25
50 Token
75
100
125
20
25
50 Token
19
100
125
20
25
15 Token
50
100
125
30
25
Layer
20
(a) Per-channel Scaling.
(h) Hadamard Transform.
(c) FLATQUANT.
(d) Stacked View.
Figure 18: The MSE of quantization across Transformer layers and input sequence LLaMA-2-7B. Figure 18a-18c plot the MSE surface of each method, while Figure 18d overlays these surfaces by dividing each MSE with that of FLATQUANT.
3.
2.5
2.0
0.2
20
0.5
10
10
10
20 Layer
20
Layer
Layer
20
30 Layet
25
Token
40
Taken
100
40
Taken
100
40
Token
(a) Per-channel Scaling.
(b) Hadamard Transform.
(c) FLAIQUANT.
(d) Stacked View.
Figure 19: The MSE of quantization across Transformer layers and input sequence in LLaMA-2-13B. Figure 19a-19c plot the MSE surface of each method, while Figure 19d overlays these surfaces by dividing each MSE with that of FLATQUANT.
50
10
80
2.0
1.0
0.5
20
0.0
20
20 Layer
25
15 Token
50
100
125
50
40 Layer
25
50 Token
15
100
125
60
25
50 Taken
75
100
125
40 Layerr
25
50 75 100 Taken
125
Layer
(a) Per-channel Scaling.
(b) Hadamard Transform.
(c) FLAIQUANT.
(d) Stacked View.
Figure 20: The MSE of quantization across Transformer layers and input sequence in LLaMA-2-70B. Figure 20a-20c plot the MSE surface of each method, while Figure 20d overlays these surfaces by dividing each MSE with that of FLAT
15.0
0.25
600
12.5
0.20
500
300
6.10
200
0.05
100
0.0
0. 0.00
20
20
20
40 Layer
Layer
Layer
60
100 Token
Token
100
125
80
Token
100
125
80
loken
100
80
(a) Per-channel Scaling.
(b) Hadamard Transform.
(c) FLATQUANT.
(d) Stacked View.
Figure 21: The MSE of quantization across Transformer layers and input sequence in LLaMA-3-70B Figure 21a-21c plot the MSE surface of each method, while Figure 21d overlays these surfaces by dividing each MSE with that of FLATQUANT.
27


Reviews:
[{"review_id": "lFVbMHV3tJ", "reviewer": "Reviewer_re4r", "summary": "This paper presents FLATQUANT, a novel post-training quantization framework that enhances the flatness of weight and activation distributions in large language models (LLMs) through learnable affine transformations. FLATQUANT introduces layer-wise affine transformations trained on calibration data to reshape distributions, coupled with Kronecker decomposition to reduce parameter overhead. The method outperforms existing techniques (e.g., QuaRot, SpinQuant) across perplexity, QA accuracy, and inference speed benchmarks, particularly excelling in W4A4 settings. Results on LLaMA-3 models (7B–70B) demonstrate robustness, with kernel fusion further enhancing practical deployment viability.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**. *The authors do not provide a comparative assessment of GPU memory consumption...*\n\n\n**A1**. Thanks for the helpful advice. Below, we provide more results on the memory consumption. The experiment results are consistent with our theoretical analysis in Appendix B.2 Line 736-739, demonstrating the minimal memory overhead brought by the online transformations in FlatQuant.\n\n* **Peak memory usage for decoding a single token** on one Transformation layer of LLaMA-2-7B model with KV caches of different lengths and batch size 1.\n\n|**Sequence Length**|**FP16 (GB)**|**INT4 (GB)**|**FlatQuant (GB)**|**Saving Factor**|\n|:----|:----|:----|:----|:----|\n|256|0.393|0.110|0.110|3.58x|\n|512|0.399|0.112|0.112|3.56x|\n|1024|0.411|0.118|0.118|3.48x|\n|2048|0.434|0.130|0.130|3.35x|\n\n\n---\n**Q2**. *The paper lacks exploration of potential biases introduced by Kronecker decomposition...*\n\n\n**A2**. Thanks for the thoughtful feedback. As shown in Figure 5, experiments demonstrate that Kronecker decomposition preserves expressivity, achieving accuracy comparable to full-size transformations. We leave theoretical justifications as future work.\n\n\n---\n**Q3**. ...*A more targeted ablation would clarify the relative importance of affine transformations versus clipping thresholds.*\n\n\n**A3**. Thanks for the constructive advice. Below we provide more results and discussions to demonstrate the effectiveness of LT:\n\n* **Full Ablation Table**. We provide full ablation results as a complement to Table 3, which clearly showcases the effectiveness of each part in FlatQuant, especially LT.\n\n|**LT**|**PS**|**LCT**|**WikiText-2**|**C4**|**ARC-C**|**ARC-E**|**HellaSwag**|**LAMBADA**|**PIQA**|**Winogrande**|**Avg.**|\n|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|\n||||1266.60|936.41|25.26|28.62|27.04|1.26|51.80|51.93|30.99|\n||√||NaN|NaN|22.70|25.08|25.04|0.00|49.51|49.57|28.65|\n|||√|1149.08|1490.08|22.95|29.29|27.35|0.60|52.99|50.83|30.67|\n||√|√|8197.96|4654.07|25.43|25.72|25.96|0.02|50.49|48.86|29.41|\n|√|||8.50|13.51|44.97|71.38|73.17|67.05|76.88|67.48|66.82|\n|√|√||7.95|12.74|44.20|71.89|74.21|68.72|77.15|66.30|67.08|\n|√||√|7.11|11.47|49.32|76.14|76.30|72.17|78.89|71.51|70.72|\n|√|√|√|6.98|11.13|50.00|75.80|76.80|72.91|79.16|72.69|71.23|\n\n* **Comparison between FlatQuant's affine transformation and SpinQuant's orthogonal transformation**. For both methods, we use asymmetric activation quantization following SpinQuant and do not activate weight or activation clipping. For FlatQuant, we also do not use PS. It can be seen that the affine transformations in FlatQuant are more effective in easing outliers, leading to higher accuracy especially when RTN is used.\n\n|**LLaMA3-8B**|**WikiText-2**|**C4**|**ARC-C**|**ARC-E**|**HellaSwag**|**LAMBADA**|**PIQA**|**Winogrande**|**Avg.**|\n|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|\n|FP16|6.14|9.45|53.50|77.57|79.12|75.51|80.74|72.93|73.23|\n|SpinQuant-RTN|41.15|63.89|24.15|38.09|45.12|20.80|59.47|55.01|40.44|\n|FlatQuant-LT-RTN|**8.05**|**12.85**|46.08|71.30|73.74|67.15|76.28|68.59|**67.19**|\n|SpinQuant-GPTQ|7.95|13.44|44.80|73.36|73.79|66.99|76.01|68.11|67.18|\n|FlatQuant-LT-GPTQ|**7.66**|**12.58**|46.16|73.57|74.83|67.55|76.01|67.32|**67.57**|\n\n\n---\n**Q4**. *The authors treat each layer's quantization independently, … , potentially overlooking cascading effects.*\n\n\n**A4**. Thanks for this valuable question. We tried to minimize the MSE loss between ground truth and outputs given quantized inputs instead of given full-precision inputs (as in Equation 4) to mitigate cascading effects. However, results show that full-precision inputs lead to better performance. To learn the mechanism behind, we think this is similar to the \"teacher forcing\" in training RNNs (i.e., use the ground truth instead of auto-regressive output for training at each step). Similar effects were also verified to be helpful in model quantization [1].\n\n|**LLaMA-3-8B**|**WikiText-2**|**C4**|**ARC-C**|**ARC-E**|**HellaSwag**|**LAMBADA**|**PIQA**|**Winogrande**|**Avg.**|\n|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|\n|FP16|6.14|9.45|53.50|77.57|79.12|75.51|80.74|72.93|73.23|\n|FlatQuant|6.98|11.13|50.00|75.80|76.80|72.91|79.16|72.69|71.23|\n|+quant_inps|7.14|11.51|50.68|76.64|75.77|70.25|80.41|68.51|70.38|\n\n[1] Bai, Haoli, et al. \"Towards efficient post-training quantization of pre-trained language models.\" Advances in neural information processing systems 35 (2022): 1405-1418.\n\n\n---\n**Q5**. *The paper would benefit from a clearer explanation of how the online transformations are initialized,....*\n\n**A5**. Please see A7 to Reviewer pwvD.\n\n\n---\n**Q6**. *The hyperlinks throughout the paper appear to be non-functional,...*\n\n\n**A6**. We have double-checked the manuscript and the hyperlinks function well on our side. We are sorry for the inconvenience, and maybe a different PDF browser could resolve your issue.", "claims_and_evidence": "The authors claim that flatness of weights and activations is crucial for quantization effectiveness, and their method produces flatter distributions than previous approaches. This claim is well-supported by the visualizations in Figure 1, which clearly show the distribution envelopes after different transformations.\n\nThe claim that FLATQUANT outperforms previous SOTA methods is convincingly supported by extensive experiments across multiple benchmarks and model scales. For instance, Table 1 and Table 2 show consistent improvements over SpinQuant and QuaRot in perplexity metrics and QA tasks respectively.", "methods_and_evaluation_criteria": "The proposed methods align well with the problem at hand. The use of Kronecker decomposition to reduce computational overhead is particularly clever, as it maintains the benefits of affine transformations while minimizing inference costs.\n\nThe evaluation criteria are comprehensive, including perplexity on language modeling tasks (WikiText-2, C4) and accuracy on zero-shot QA tasks. The authors also properly evaluate both accuracy and speed metrics, providing a holistic view of the method's practical utility. The comparison against state-of-the-art baselines using the same evaluation framework (lm-eval-harness) ensures fair comparisons.", "theoretical_claims": "The paper makes minimal purely theoretical claims, focusing instead on empirical findings. The explanation of how Kronecker decomposition reduces computational complexity (line 205-213) appears sound, showing that memory requirements are reduced by a factor of approximately n/2 when the dimensions are balanced (n1 ≈ n2 ≈ √n)..", "experimental_designs_or_analyses": "The experimental design is thorough and well-executed. The authors test their method across multiple model sizes (7B to 70B parameters) and across different quantization settings (weight-only, weight-activation, KV cache). The additional experiments in the appendix further bolster confidence in the results.\n\nThe kernel fusion experiments (Figure 4 and Table 6) are particularly well-designed, showing how the theoretical speedups translate to practical benefits across different batch sizes and hidden dimensions.", "relation_to_broader_scientific_literature": "The authors provide a comprehensive discussion of related work, clearly positioning FLATQUANT relative to existing methods like SmoothQuant, OmniQuant, AffineQuant, QuaRot, and SpinQuant. This paper builds upon the insight from previous work that outliers in LLMs pose challenges for quantization, but extends beyond prior approaches by learning model-specific transformations rather than relying on fixed transformations like Hadamard.", "essential_references_not_discussed": "The authors discuss in depth all relevant work related to the key contributions of this paper.", "other_strengths_and_weaknesses": "**Strengths:**\n\n1. The kernel fusion implementation (Section 3.3, Appendix B.3) is particularly impressive, showing how theoretical ideas can be translated into practical performance gains. \n2. Figure 2 provides an insightful visualization of quantization error propagation across layers and tokens, helping explain why flatter distributions matter. \n3. The method's effectiveness across different model architectures (shown in the supplementary material) speaks to its generalizability.\n\n**Weaknesses:**\n1. The authors do not provide a comparative assessment of GPU memory consumption during inference relative to baseline methods. This omission is particularly significant for edge deployment scenarios where memory constraints are often the primary limiting factor.\n2. The paper lacks exploration of potential biases introduced by Kronecker decomposition and how these biases might affect quantization error propagation through the network. Understanding this relationship would strengthen the theoretical foundation of the approach.\n3. While FLATQUANT incorporates LCT and shows improvements over SpinQuant, the paper doesn't adequately isolate whether this specific component is the primary driver of performance gains. A more targeted ablation would clarify the relative importance of affine transformations versus clipping thresholds.\n4. The authors treat each layer's quantization independently, but don't thoroughly examine how quantization decisions in earlier layers affect the optimal transformations for later layers, potentially overlooking cascading effects.", "other_comments_or_suggestions": "1. The paper would benefit from a clearer explanation of how the online transformations are initialized, as initialization strategies can significantly impact convergence and final performance.\n2. The hyperlinks throughout the paper appear to be non-functional, which hinders navigation between sections and references. Authors should consider revising their citation formatting to ensure proper link functionality.", "questions_for_authors": "Please refer to Strengths And Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MqcW5i9szD", "reviewer": "Reviewer_7f4H", "summary": "This paper proposes FlatQuant for post-training quantization with learnable linear transformation to remove outliers of weights and activations. The authors propose the use of Kronecker decomposition to reduce the inference overhead. Extensive experiments demonstrate the effectiveness of the proposed approach on W4A4 quantization.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**. *Regarding techniques for suppressing outliers..., the authors overlooked the use of infinity-norm regularization for reducing the weight range.*\n\n\n**A1**. Thanks for the recommendation regarding related works. We will add the discussions in the revised manuscript.\n\n\n---\n**Q2**. *The results are primarily empirical, lacking sufficient theoretical justification for the effectiveness of the Kronecker decomposition-based linear transformation. Specifically, it remains unclear why this approach outperforms rotation-based transformations. A deeper theoretical analysis would strengthen the claims.*\n\n\n**A2**. Thanks for the advice. Compared with rotational transformations, FlatQuant relaxes the optimization space from Stiefel manifold to the general linear space, which has more potential to suppress outliers. We leave more detailed theoretical analyses in future work.\n\n\n---\n**Q3**. *It is counterintuitive that FlatQuant outperforms AffineQuant significantly...*\n\n**A3**. In the following, we discuss the superiority of FlatQuant over AffineQuant in more detail as a complement to Line 631-637:\n\n* **Improved Expressivity**. As discussed in Figure 5, we find that the Kronecker decomposed transformations deliver comparable accuracy with the full-size transformation, which demonstrates its expressivity. In contrast, AffineQuant optimizes strictly diagonally dominant transformation to ensure invertibility, which may restrict its expressivity and make the transformation more like a per-channel scaling (see Figure 7 in the AffineQuant paper).\n\n* **Applicability to All Linear Layers**. FlatQuant applies Kronecker-decomposed linear transformations to all linear layers with minimal overhead. In contrast, AffineQuant directly learns a full-size transformation, and can only apply it to the output projection linear layer for WA quantization so that the transformation can be merged into preceding linear layer to avoid the formidable overhead. The other linear layers can only use per-channel scaling.\n\n\n---\n**Q4**. *The contributions are somewhat incremental, extending existing work on block-wise quantization and linear transformations for smoothing weights and activation values.*\n\n\n**A4.** We elaborate on the key differences between FlatQuant and previous works below:\n\n* **Linear Transformations v.s. Orthogonal Transformations.** While previous works (e.g. QuaRot) explore orthogonal transformations to smooth outliers, we find linear transformations can achieve better flatness, delivering superior quantization accuracy as shown in Table 1-2.\n\n* **Kronecker Decomposed Matrices with Kernel Fusion.** While linear transformations suffer from high computation and memory consumption, we mitigate it with Kronecker decomposition along with kernel fusion, ensuring practical speedup with minimal overhead.\n\n* **Learnable Clipping After Transformations.** As detailed in Appendix C.4, a key difference of learnable clipping in FlatQuant is that it is applied after the pre-quantization transformations. This can help avoid damaging critical outliers during activation clipping.\n\n\n---\n**Q5**. *What is the role of Kronecker decomposition-based transformation in mitigating outliers? Can we achieve similar effects by imposing alternative low-complexity structures, such as low-rank decomposition?*\n\n\n**A5**. The learnable transformation to remove outliers needs to satisfy 1) **invertible**, so that the transformation is computational invariance (similar to QuaRot); 2) **lightweight** so that little inference overhead is introduced. Kronecker decomposition is a valid approach to satisfy both conditions, where the invertibility is ensured from (Line 661-663).  However, low-rank decomposition does not satisfy 1) invertibility. We leave the exploration of other alternatives in the future.\n\n\n---\n**Q6**. *I wonder how FlatQuant performs on 2-bit weight quantization.*\n\n\n**A6**. Thanks for the question. In the following, we experiment with 2-bit asymmetric per-channel weight-only quantization and compare FlatQuant against two strong uniform weight-only quantization methods GPTQ and QuIP[1]. For GPTQ, we use activation reorder and grid search the best weight clipping thresholds.\n\n|**LLaMA-3-8B**|**WikiText-2**|**C4**|**ARC-C**|**ARC-E**|**HellaSwag**|**LAMBADA**|**PIQA**|**Winogrande**|**Avg.**|\n|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|\n|FP16|6.14|9.45|53.50|77.57|79.12|75.51|80.74|72.93|73.23|\n|GPTQ|**26.76**|247.27|22.70|31.86|35.37|5.20|54.08|51.14|33.39|\n|QuIP|59.96|139.70|23.98|28.24|30.82|1.59|52.18|50.28|31.18|\n|FlatQuant|31.27|**132.02**|**24.49**|**41.04**|**39.42**|**14.28**|**59.03**|**53.75**|**38.67**|\n\n[1] Chee, Jerry, et al. \"Quip: 2-bit quantization of large language models with guarantees.\" Advances in Neural Information Processing Systems 36 (2023): 4396-4429.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "The authors show the effectiveness of the algorithm through extensive experiments.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Solid experimental results on LLM quantization.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "Regarding techniques for suppressing outliers to enhance quantized LLMs, in addition to per-channel scaling and linear transformations, the authors overlooked the use of infinity-norm regularization for reducing the weight range, e.g., [1,2].\n\n[1] Zhang et al., MagR: Weight Magnitude Reduction for Enhancing Post-Training Quantization.\n\n[2] Kundu et al., R2 Loss: Range Restriction Loss for Model Compression and Quantization.", "other_strengths_and_weaknesses": "Strengths: \n\n1. Proposed Kronecker decomposition to mitigate the computational and memory costs for the additional linear transformation.\n\n2. Demonstrated the effectiveness of the proposed method through extensive experiments.\n\nWeakness:\n\n1. The contributions are somewhat incremental, extending existing work on block-wise quantization and linear transformations for smoothing weights and activation values. \n\n2. The results are primarily empirical, lacking sufficient theoretical justification for the effectiveness of the Kronecker decomposition-based linear transformation. Specifically, it remains unclear why this approach outperforms rotation-based transformations. A deeper theoretical analysis would strengthen the claims.\n\n3. The reported results in Table 1 raise concerns about plausibility. It is counterintuitive that FlatQuant outperforms AffineQuant significantly, given that AffineQuant employs a more general affine transformation, while FlatQuant imposes constraints through low-complexity Kronecker decomposition, trading expressivity for efficiency. Justification is needed to explain why the reduced expressivity yields the better performance.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. What is the role of Kronecker decomposition-based transformation in mitigating outliers? Can we achieve similar effects by imposing alternative low-complexity structures, such as low-rank decomposition?\n\n2. I wonder how FlatQuant performs on 2-bit weight quantization.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dlS0aGCvce", "reviewer": "Reviewer_pwvD", "summary": "This paper presents FLATQUANT, a post-training quantization approach for large language models (LLMs) that focuses on improving the \"flatness\" of weight and activation distributions to enhance quantization performance. The authors introduce a novel method employing fast and learnable affine transformations tailored to each linear layer in a Transformer model. The approach uses Kronecker decomposition to reduce computational overhead and incorporates kernel fusion for efficient implementation. The paper demonstrates state-of-the-art quantization accuracy with minimal performance degradation, even in challenging W4A4 (4-bit weights and activations) scenarios for large models like LLaMA-3-70B, while providing significant speedups compared to FP16 inference.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**. ...*a clearer quantitative comparison of what constitutes the previous SOTA.*\n\n\n**A1**. We consider QuaRot and SpinQuant as previous SOTA in Table 1-2, and compare with the latest DuQuant in A1 to Reviewer x5Xi.\n\n\n---\n**Q2**. *The notion of \"flatness\" is somewhat intuitive but would benefit from a more rigorous definition or metric.*\n\n\n**A2**. Thanks for the suggestion. We have provided a quantitative definition of flatness in Appendix D.1 Line 1106-1113, and visualized the associated flatness score in Figure 7 and Figure 11.2. We will move this part to the main text.\n\n\n---\n**Q3**. *The claim that \"FLATQUANT with RTN is sufficient to be competitive\" is intriguing but lacks thorough theoretical explanation.*\n\n\n**A3**. Thanks for pointing this out. We suspect the affine transformations alone make the distributions friendly to RTN, where the flatness is quantitatively measured in Appendix D.1 and Figure 7. We decide to leave theoretical explanations in future works.\n\n\n---\n**Q4**. *The paper would benefit from a more detailed explanation of why learnable clipping after transformation is more effective than before transformation (as shown in Table 17).*\n\n\n**A4**. Thanks for the question. This can be attributed to the preservation of important outlier channels.\n\n* **\"LCT before Transformation\"** consistently performs worse, as it directly clips important outliers in LLMs.\n\n* **\"LCT after Transformation\"** performs significantly better because the affine transformations redistribute outliers across channels. Even after clipping, the inverse transformation can effectively recover the original outliers.\n\nWe will further clarify this issue in our revision.\n\n\n---\n**Q5**. *Some minor typos and grammatical issues exist.*\n\n\n**A5**. Thanks for the attentive feedback. We will fix these issues in the revised version.\n\n\n---\n**Q6**. *How would FLATQUANT perform when incorporated into more advanced quantization schemes like mixed-precision quantization, where different layers use different bit-widths? This could affect my evaluation as it might demonstrate broader applicability.*\n\n\n**A6**. Thanks for raising this interesting question. Below we explore some orthogonal mixed-precision strategies to enhance FlatQuant.\n\n* **Weight Importance Score Analysis**. We follow SqueezeLLM[1] to use Fisher information for the estimation of weight importance, and simply sum over the scores to estimate the overall importance of different parts in LLMs, including different linear layer types and Transformer layers.\n\n|**Self-attention**|$\\mathbf W_q$|$\\mathbf W_k$|$\\mathbf W_v$|$\\mathbf W_o$|\n|:----|:----|:----|:----|:----|\n|**Score**|44.86|11.01|1139.08|365.17|\n\n|**Feed-forward network**|$\\mathbf W_g$|$\\mathbf W_u$|$\\mathbf W_d$|\n|:----|:----|:----|:----|\n|**Score**|177.20|401.03|2468.97|\n\n|**Top 5 Transformer Layer Index**|**1**|**0**|**2**|**3**|**31**|\n|:----|:----|:----|:----|:----|:----|\n|**Score**|63684.88|18075.19|6740.75|5780.25|5297.59|\n\n* **Mixed-precision Results**. According to the overall importance, we use W8A8 for the top 5 Transformer layers as well as all the down projection layers, which proves to be an effective way to improve FlatQuant.\n\n|**LLaMA-3-8B**|**WikiText-2**|**C4**|**ARC-C**|**ARC-E**|**HellaSwag**|**LAMBADA**|**PIQA**|**Winogrande**|**Avg.**|\n|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|\n|FP16|6.14|9.45|53.50|77.57|79.12|75.51|80.74|72.93|73.23|\n|FlatQuant|6.98|11.13|50.00|75.80|76.80|72.91|79.16|72.69|71.23|\n|+down_proj_8bits|6.73|10.62|50.00|77.78|77.49|73.96|79.54|70.56|71.55|\n|+Top5_8bits|6.80|10.82|49.23|76.56|76.54|73.51|79.71|73.95|71.58|\n|+Top5&down_proj_8bits|6.61|10.43|51.11|77.74|77.47|74.69|79.92|72.14|72.18|\n\n[1] Kim, Sehoon, et al. \"Squeezellm: Dense-and-sparse quantization.\" arXiv preprint arXiv:2306.07629 (2023).\n\n\n---\n**Q7**. *Have you analyzed how different initialization strategies for the affine transformation matrices affect convergence speed or final performance?*\n\n\n**A7.** Thanks for the thoughtful feedback. We initialize the transformations with random orthogonal matrices by default and find FlatQuant is generally robust to initialization strategies.\n\n|**Init**|**WikiText-2**|**C4**|**ARC-C**|**ARC-E**|**HellaSwag**|**LAMBADA**|**PIQA**|**Winogrande**|**Avg.**|\n|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|\n|Identity Matrix|6.98|11.15|49.49|77.10|76.82|74.21|80.09|70.32|71.34|\n|Random Orthogonal Matrix|6.98|11.13|50.00|75.80|76.80|72.91|79.16|72.69|71.23|\n\n\n---\n**Q8**. *How would FLATQUANT perform in even more extreme scenarios, such as W3A3 quantization? Some results are shown in Appendix Table 14, but a more detailed analysis of limitations would be valuable.*\n\n\n**A8**. Thanks for the advice. As shown in Table 14, although FlatQuant achieves notable improvements over QuaRot, it still suffers significant accuracy degradation. W3A3 needs both improved algorithms and hardware support, which still remains an open question.", "claims_and_evidence": "The authors make several claims that are generally well-supported by experimental evidence:\n\nPage 1, lines 51-52: The claim that FLATQUANT achieves less than 1% accuracy drop for W4A4 quantization on LLaMA-3-70B is supported by the comprehensive evaluation in Table 2, which shows a minimal drop from 79.95% to 79.01% on average across multiple benchmarks.\nPage 2, paragraph 2: The claim that flatter distributions are easier to quantize is well-substantiated through multiple visualizations (Figure 1) and analyses of quantization error landscapes (Figure 2).\nPage 3, paragraph 3: The authors claim that Kronecker decomposition reduces memory and computational demands, which is validated by the quantitative analysis showing 2.61% computational overhead and 0.11MB memory overhead (page 4).", "methods_and_evaluation_criteria": "The methods are generally appropriate and well-designed:\n\nThe selection of benchmark tasks (WikiText2, C4, and six zero-shot QA tasks) provides good coverage for evaluating language modeling capabilities and task performance.\nThe comparisons with state-of-the-art baselines like SmoothQuant, OmniQuant, QuaRot, and SpinQuant offer appropriate context for assessing FLATQUANT's contributions.\nThe ablation studies effectively isolate the contributions of different components (learnable transformations, per-channel scaling, and learnable clipping thresholds).\nHowever, there are methodological limitations:\n\nPage 4, paragraph 3: The calibration procedure using only 128 sentences from WikiText-2 seems limited, though the authors do later show in Table 16 (Appendix) that performance remains consistent across different calibration datasets.\nPage 4, line 230-234: The selection of n1 and n2 for Kronecker decomposition is presented as a simple optimization, but the exploration of these parameters is limited to one model size.", "theoretical_claims": "The paper's theoretical claims are generally sound and verified:\n\nThe idea that flatness matters for quantization is well-established in prior literature, as the authors acknowledge. Their contribution is showing how to better optimize for this objective.\nThe Kronecker decomposition approach (page 3) is mathematically correct and the efficiency claims are validated by experiments.\nI did not identify any issues with the mathematical derivations provided, though the theoretical justifications for why the training process leads to flatter distributions could be more formally developed.", "experimental_designs_or_analyses": "The experimental analyses are thorough and well-executed:\n\nPage 5, Tables 1-2: The comprehensive evaluation across multiple models (7B to 70B parameters) and datasets provides strong evidence for FLATQUANT's effectiveness.\nPage 7, Figure 4-6: The detailed analysis of inference latency demonstrates the practical benefits of FLATQUANT.\nPage 8, Figure 7: The correlation between training progress and increasing flatness offers a nice validation of the method's working principle.\nSome experimental limitations exist:\n\nPage 7, Figure 5: While the impact of different decomposition sizes is analyzed, this analysis is limited to a single model size (LLaMA-2-7B).\nIt would be helpful to see more detailed performance profiling across different hardware platforms beyond the RTX3090 GPU mentioned on page 5, line 267.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\nPage 3, section 3.1: The Kronecker decomposition approach is a clever solution to the computational overhead problem that plagues prior affine transformation methods.\nPage 3, lines 173-179: The training objective for each Transformer block individually simplifies the optimization problem without apparent degradation in performance.\nPage 4, section 3.3: The kernel fusion approach is well-designed and practically important for achieving the reported speedups.\nWeaknesses:\n\nPage 1, paragraph 1: While the paper claims to establish \"new state-of-the-art\" results, it would be strengthened by a clearer quantitative comparison of what constitutes the previous SOTA.\nSection 2.2: The notion of \"flatness\" is somewhat intuitive but would benefit from a more rigorous definition or metric.\nPage 6, lines 364-368: The claim that \"FLATQUANT with RTN is sufficient to be competitive\" is intriguing but lacks thorough theoretical explanation.", "other_comments_or_suggestions": "The paper would benefit from a more detailed explanation of why learnable clipping after transformation is more effective than before transformation (as shown in Table 17).\nThe visualization in Figure 1 effectively illustrates the concept of flatness, but could be improved by adding a metric that quantifies the degree of flatness.\nSome minor typos and grammatical issues exist (e.g., \"is powerless to such errors\" on page 3).", "questions_for_authors": "How would FLATQUANT perform when incorporated into more advanced quantization schemes like mixed-precision quantization, where different layers use different bit-widths? This could affect my evaluation as it might demonstrate broader applicability.\nThe paper mentions that FLATQUANT is robust to initialization (page 4, lines 261-262). Have you analyzed how different initialization strategies for the affine transformation matrices affect convergence speed or final performance?\nThe paper shows impressive results on LLaMA-3-70B with W4A4 quantization. How would FLATQUANT perform in even more extreme scenarios, such as W3A3 quantization? Some results are shown in Appendix Table 14, but a more detailed analysis of limitations would be valuable.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e3loeKtRyx", "reviewer": "Reviewer_x5Xi", "summary": "This work proposes an approach of learning scaling and rotation transformation to mitigate the problem of outliers and varying dynamic range of weights and activations in LLMs. The learnable rotation transformation is parameterized as a Kronecker product of 2 matrices. Parameters of the transformation are trained to optimize the block reconstruction objective.  The method proposed in validated on several LLM models for W4A4 quantization.\n\n### Update after rebuttal\n\n I believe than contribution and experimental results are sufficient for acceptance of this work.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**. *For an exhaustive comparison I would suggest adding a few more state-of-the-art methods [1, 2]. The former provides other strategy for learning rotation and the latter accounts for outliers tokens in KV cache.*\n\n*[1] Lin H. et al. Duquant: Distributing outliers via dual transformation makes stronger quantized llms //Advances in Neural Information Processing Systems. – 2025. – Т. 37. – С. 87766-87800.*\n\n*[2] Chen, Mengzhao, et al. \"Prefixquant: Static quantization beats dynamic through prefixed outliers in llms.\" arXiv preprint arXiv:2410.05265 (2024).*\n\n\n**A1**. Thanks for the recommendation regarding the recent works. We will add them in the revised version. PrefixQuant is mainly experimented with static quantization and is a technology orthogonal to our method, we leave the integration of PrefixQuant with FlatQuant as the future work. Below we conduct a detailed comparison between FlatQuant and DuQuant.\n\n* **Comparison with DuQuant in Accuracy.** FlatQuant significantly outperforms DuQuant across multiple benchmarks.\n\n|**LLaMA3-8B**|**WikiText-2**|**C4**|**ARC-C**|**ARC-E**|**HellaSwag**|**LAMBADA**|**PIQA**|**Winogrande**|**Avg.**|\n|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|\n|FP16|6.14|9.45|53.50|77.57|79.12|75.51|80.74|72.93|73.23|\n|DuQuant|8.13|12.91|44.80|71.30|73.00|68.04|75.73|69.46|67.05|\n|FlatQuant|6.98|11.13|50.00|75.80|76.80|72.91|79.16|72.69|71.23|\n\n* **Comparison with DuQuant in Latency.** DuQuant has the same number of online transformations (each consisting of two matrix multiplications and one channel permutation) as FlatQuant (each consisting of a Kronecker-decomposed matrix multiplication). It can be seen that FlatQuant achieves comparable speedup without kernel fusion and even faster speedup with kernel fusion.\n\n|**Batch Size**|**1**|**4**|**16**|\n|:----|:----|:----|:----|\n|DuQuant|1.95x|2.03x|2.08x|\n|FlatQuant w/o Kernel Fusion|1.94x|2.02x|2.10x|\n|FlatQuant w/ Kernel Fusion|2.12x|2.21x|2.27x|\n\n\n---\n**Q2**. *What are the speed-ups on newer generation of GPUs (say GPU 4090) or high-end GPU on prefill and decode? Would you expect them to be lower or higher?*\n\n\n**A2**. Thanks for the valuable suggestion. Currently, we haven't implemented INT4 kernels on other GPUs, but we can approximately estimate the speedup. In the following, we estimate the speedup for a single linear layer with hidden dimension $d$ on A100 GPU. The speedup for attention layers can be estimated likewise.\n\n* **A100 Specs**. For A100, we have $\\text{Perf} _{\\text{BF16}}=312\\text{ TFLOPS},\\text{Perf} _{\\text{INT4}}=1248\\text{ TFLOPS},\\text{Bandwidth}=1555\\text{ GB/s}$\n\n* **Prefill Speedup**. Assume $s=2048$ and $d=4096$ where $s$ is the number of tokens in the prefill. Since in the prefill stage, the linear layer is mostly compute-bound, while the online quantization and transformations are memory-bound, we can estimate the prefill speedup accordingly:\n\n$\n\\text{FLOPs} _{\\text{Linear}}=2sd^2,\n\\text{Mem} _{\\text{Quant+Kron}}=2.5sd+4d\n$\n\n\n$\n\\text{Prefill Speedup}\\approx \\frac{\\text{FLOPs} _{\\text{Linear}}/\\text{Perf} _{\\text{BF16}}}{\\text{FLOPs} _{\\text{Linear}}/\\text{Perf} _{\\text{INT4}}+\\text{Mem} _{\\text{Quant+Kron}}/\\text{Bandwidth}}\\approx 3.21\n$\n\n* **Decoding Speedup**. Similarly, the decoding speedup can be estimated with\n\n$\n\\text{Mem} _{\\text{BF16}}=2d^2+4d,\\text{Mem} _{\\text{INT4}}=d^2/2+2.5d,\\text{Mem} _{\\text{Quant+Kron}}=6.5d\n$\n\n$\n\\text{Decoding Speedup}\\approx \\frac{\\text{Mem}_ {\\text{BF16}}}{\\text{Mem}_ {\\text{INT4}}+\\text{Mem}_ {\\text{Quant+Kron}}} \\approx 3.98\n$\n\n---\n**Q3**. *In addition, it could be insightful to measure speed-ups with other sequence lengths.*\n\n\n**A3**. Thanks for the helpful advice. In the following, we provide more speedup results with other sequence lengths as a complement to Figure 4:\n\n* **Prefill** speedup with batch size 1.\n\n|**Prefill Length**|**INT4**|**QuaRot**|**FlatQuant**|\n|:----|:----|:----|:----|\n|2048|2.16x|1.97x|2.12x|\n|4096|2.06x|1.90x|2.04x|\n|8192|1.94x|1.79x|1.92x|\n|16384|1.83x|1.72x|1.80x|\n\n* **Decoding** speedup with batch size 64.\n\n|**KV Cache Length**|**INT4**|**QuaRot**|**FlatQuant**|\n|:----|:----|:----|:----|\n|256|1.38x|1.09x|1.24x|\n|512|1.62x|1.38x|1.56x|\n|1024|1.70x|1.61x|1.63x|\n|2048|1.78x|1.72x|1.76x|", "claims_and_evidence": "The authors claim to achieve new state-of-the-art for W4A4 and the evidence provided does support this statement. Performance drop for W4A4 quantization of Llama-2, Llama-3 models relative to baseline is much smaller than the one produced by baselines (even quite strong, such as QuaRot and SpinQuant). \n\nIn addition, FlatQuant comes with certain inference speed-up, and the paper provides an analysis of overhead induces by rotation transformations, making the provided results plausible.", "methods_and_evaluation_criteria": "The evaluation protocol adopted is standard for research on LLM compression.", "theoretical_claims": "This work provides primarily a practical contribution. The theoretical motivation of the proposed approach is sound.", "experimental_designs_or_analyses": "Overall, experimental protocol and choice of baselines is sensible. \nHowever, for an exhaustive comparison I would suggest adding a few more state-of-the-art methods [1, 2]. The former provides other strategy for learning rotation and the latter accounts for outliers tokens in KV cache. \n\nAn ablation on impact of each component in the proposed method is provided as well as analysis of computational overhead caused by rotations.\n\n---\n\nReferences\n\n[1] Lin H. et al. Duquant: Distributing outliers via dual transformation makes stronger quantized llms //Advances in Neural Information Processing Systems. – 2025. – Т. 37. – С. 87766-87800.\n\n[2] Chen, Mengzhao, et al. \"Prefixquant: Static quantization beats dynamic through prefixed outliers in llms.\" arXiv preprint arXiv:2410.05265 (2024).", "relation_to_broader_scientific_literature": "This work can be regarded as a new approach for learning scale and rotation transformation to simplify the problem of quantization.", "essential_references_not_discussed": "Two recent papers with strong performance are not discussed. I would suggest adding them to related work section.\n\n---\n\nReferences\n\n[1] Lin H. et al. Duquant: Distributing outliers via dual transformation makes stronger quantized llms //Advances in Neural Information Processing Systems. – 2025. – Т. 37. – С. 87766-87800.\n\n[2] Chen, Mengzhao, et al. \"Prefixquant: Static quantization beats dynamic through prefixed outliers in llms.\" arXiv preprint arXiv:2410.05265 (2024).", "other_strengths_and_weaknesses": "-", "other_comments_or_suggestions": "-", "questions_for_authors": "What are the speed-ups on newer generation of GPUs (say GPU 4090) or high-end GPU on prefill and decode? Would you expect them to be lower or higher? In addition, it could be insightful to measure speed-ups with other sequence lengths.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents FlatQuant, a novel post-training quantization method for large language models (LLMs) that effectively enhances the flatness of weight and activation distributions through learnable affine transformations combined with Kronecker decomposition. Reviewers found the paper's contributions impactful, highlighting strong empirical results, comprehensive evaluations across model scales, and practical inference speedups as notable strengths. I also think the kernel fusion results showing the practical speedup is a nice contribution.

Reviewers unanimously recommended acceptance after the rebuttal, appreciating the extensive experimental validation, clarity of presentation, and innovative approach to managing outliers in quantization. 

However, several suggestions remain for improving the final manuscript:

- Clearly present memory usage comparisons in the main text.

- Include detailed ablation results from reviewer re4r to further illustrate the role and effectiveness of the learnable transformations.

- Further clarify in the main text why FlatQuant outperforms AffineQuant.

- Clearly integrate and discuss mixed-precision quantization findings.

- Include comparison with DuQuant within the main experimental results section.

Overall, the paper significantly advances the state-of-the-art in LLM quantization and offers practical benefits with demonstrable inference improvements. Based on the reviewers' feedback and the thorough responses provided by the authors, my recommendation is to accept this submission.
Decision: Accept (poster)