SpinQuant: LLM Quantization with Learned Rotations
Authors: Zechun Liu
Changsheng Zhao
Igor Fedorov
Bilge Soran
Dhruv Choudhary
Raghuraman Krishnamoorthi
Vikas Chandra
Yuandong Tian
Tijmen Blankevoort
Venue: ICLR 2025 Poster (2025)
Keywords: LLM
quantization
outlier reduction
rotation
Primary area: foundation or frontier models, including LLMs

Abstract: Post-training quantization (PTQ) techniques applied to weights, activations, and the KV cache greatly reduce memory usage, latency, and power consumption of Large Language Models (LLMs), but may lead to large quantization errors when outliers are present. Rotating activation or weight matrices helps remove outliers and benefits quantization. In this work, we identify a collection of applicable rotation parameterizations that lead to identical outputs in full-precision Transformer architectures while enhancing quantization accuracy. In addition, we find that some random rotations lead to much better quantization than others, with an up to 13 points difference in downstream zero-shot reasoning performance. As a result, we propose SpinQuant, a novel approach that incorporates learned rotation matrices for optimal quantized network accuracy. With 4-bit quantization of weight, activation, and KV-cache, SpinQuant narrows the accuracy gap on zero-shot reasoning tasks with full precision to merely 2.9 points on the LLaMA-2 7B model, surpassing LLM-QAT by 19.1 points and SmoothQuant by 25.0 points. Furthermore, SpinQuant also outperforms concurrent work QuaRot, which applies random rotations to remove outliers. In particular, for LLaMA-3 8B models that are hard to quantize, SpinQuant reduces the gap to full precision by up to 45.1% relative to QuaRot. Code is available at https://github.com/facebookresearch/SpinQuant.

Published as a conference paper at ICLR 2025
SP INQUANT: LLM QUANTIZATION WITH LEARNED ROTATIONS
Zechun Liu" Changsheng Zhao'
Igor Fedorov
Bilge Soran
Dhruv Choudhary Tijmen Blankevoort
Raghuraman Krishnamoorthi
Vikas Chandra
Yuandong Tian
Meta
ABSTRACT
Post-training quantization (PTQ) techniques applied to weights, activations, and the KV cache greatly reduce memory usage, latency, and power consumption of Large Language Models (LLMs), but may lead to large quantization errors when outliers are present. Rotating activation or weight matrices helps remove outliers and benefits quantization. In this work, we identify collection of applicable rota- tion parameterizations that lead to identical outputs in full-precision Transformer architectures while enhancing quantization accuracy. In addition, find that some random rotations lead to much better quantization than others, with an up to 13 points difference downstream zero-shot reasoning performance. As result, we propose SpinQuant, novel approach that incorporates learned rotation matrices for optimal quantized network accuracy. With b-b quantization of weight, activa- tion, and KV-cache, SpinQuant narrows the accuracy gap on zero-shot reasoning tasks with full precision to merely 2.9 points on the L.LMMA22 model, surpassing LLM-QAT by 19.1 points and SmoothQuant by 25.0 points. Furthermore, Spin- Quant also outperforms concurrent work QuaRot, which applies random rotations to remove outliers. In particular, for LLaMA-3 8B models that are hard to quantize, SpinQuant reduces the gap to full precision by up to 45.1 1% relative to QuaRot. Code is available at gighhu..b..... com httmm//aaccbooeeearccc//a//
INTRODUCTION
Large Language models (LLMs) have demonstrated impressive performance across many disciplines. SoTA open source models (e.g., LLaMA (Tourron et al. 2023b), Mistral (Jiang al. (023). ctc)
and proprictary LLMs (e.g., GPT (Achiam et al. 2202)), Gemini Team et al. 20223). 3). ctc) have been used in general purpose chatting assistants, medical diagnosticians (Thirunavukarasu al. 2023).
computer game content generators (Cox and Ooil 2023), coding co-pilots (Roziere et al. 2023). ) and much more.
To serve such a high demand, the inference cost becomes real issue. Many effective techniques have been developed. Post-training Quantization (PTQ), as one effective category of techniques, quantizes the weights (or activations) into low-precision and thus reduces the memory usage and may significantly improve latency. This is not only important for server-side inference, but also for on-device scenarios with small-sized LLMs (Liu et al. 2024 AI@ Meta 2024).
When applying quantization, outliers remain an open challenge because they stretch the quantization range, leaving fewer effective bits available for the majority of values. Prior research mitigates this challenge by trading quantization difficulty between weights and activations (Xiao et al., 2022] Lin et al. 2023) or employing mixed-precision focus on a new angle: multiplying the weight matrix with a rotation matrix to reduce outliers and enhance quantizability. Inspired by (Elhage et al. (023) and SliceGPT (Ashkboos al. 2023a), we leverage the property of rotational invariance to construct rotation matrices in pairs from identity mapping, which can be integrated into nearby weights without affecting the overall network outputs. By applying these random rotations, we produce distribution of weight or activation entries that is less,er-less, facilitating easy quantization.
handle outliers (Zhao et al. (202)). In this work,
Equal contribution. Correspondence Zechun Liu <zechunliu @ meta.com>>.

---

Published as a conference paper at ICLR 2025
Multi-Head Sclf-Attention
Feed-forward Network
Embodding W:
Rg
R2
R2
Output Whead
X Kev
x
Reduce outliers inside the block
(a)
Reduce outliees inside the block
Rg
RoPE
Soitmax
Way
RoPe
R4
Wdown
R2
Jance
Swash
H
R2
(b)
(c)
Mer Mergeable
Onle rotations
K-V cache quantirtton
Activation
Merge and quantize weights
Rotated
Original activation
rotations
Quantixaton
R₂
activation
Figure 1: Overall diagram of rotation. (a) The residual stream can be rotated in the transformer network, resulting numerically equivalent floating point networks before and after rotation. The rotated activations exhibit fewer outliers and are casier to quantize. (h) & (c) The rotation matrix can he integrated with the corresponding weight matrices and we further define R2, Rs, and R& for reducing outliers inside the block.
In addition using random rotation, which statistically works well, we find that the performance of quantized network could vary a lot with different rotation matrices. For example, the downstream averaged accuracy on zero-shor shot reasoning tasks may change up to points with different rotations. As a result, we propose Spin that integrates and optimizes the rotation matrix to minimize the final loss of the quantized network, with fixed weight parameters, by employing the Cayley SGD (Li et al. 2020), a proficient technique for optimizing orthonormal matrices. This optimization does not alter the full full-precision network output but refines the intermediate activations and weights, making them more cruutization-frienddl
In SpinQuant, we introduce two rotation strategies tailored for different complexity levels: SpinQuantro had and SpinQuanthad- Here, had refers to hadamard rotation matrix. In SpinQuant had, as depicted in Figure [b)). we implement shorteut rotation (R1) and W.- W, pair rotation (R2). which
be directly absorbed into the respective weight matrices. During inference, the original weights are simply replaced with the rotated quantized weights, eliminating the need for modification in the forward pass. Conversely, in Spi SpinQuante nnQaantad designed for scenarios with low-bit quantization of KV cache or activations (c.g.. 4-bit), we further incorporate online Hadamard rotation matrices (Ra, R&) address activation outliers inside MLP block and KV cache.
To rigorously assess the effectiveness of SpinQuant, we executed comprehensive experiments across seven leading Large Language Models (LLMs), including LLaMA- 2(Tovrro al., 2023b) models (7B/13B/70B), LLaMA- ((I Metal 2024) models (IB/3B/8B), and the Mistral (Jiang et al. 2023 7B model. The key contributions this study are summarized as follows:
We introduce SpinQuant, the first method that employs learned rotations to mitigate outliers in weight and activation distributions, boosting the performance of quantized LLMs.
We reveal that random rotations introduce substantial variance in quantized network perfor- mance. We propose optimizing rotation matrices within Stiefel manifold, directly minimiz- ing the final loss of rotated quantized network. Ablation studies validate that our learned rotations consistently outperform random rotations, with improvements up to 16.2 points.
SpinQuant no had merges rotation matrices into pre-trained weights without altering the network architecture, significantly narrowing the W4A8KV8 quantization performance gap from 12.1 1.6 on the Mistral-7B model in zero-shot commonsense reasoning tasks. Noteworthily, SpinQuant,, had W4A8 quantization achieves comparable performance state-of-the-art weight only quantization methods like QulP# (Tseng et al. 2024 and OminiQuant (Shao et al. 2023) on LLaMA-2.
SpinQuanthod had attains an average accuracy of 64.0 in extreme W4A4KV4 quantization settings LLaMA-2 7B. This represents mere 2.9 point gap from the full- aell-precision net- work, a substantial improvement over the previous LLM-QAT (Liu et al. 2023c) approach, which exhibited a 22.0 point gap under identical precision conditions.
2

---

Published as a conference paper at ICLR 2025
MHSA RIM
Rocaaed MHSH irSur irput
PPN Inpux
Roused FPN inpor
(8)
(b)
(C)
(d)
Figure 2: Activation distribution in LLaMA-2 7B model before and after rotation. Outliers exist in particular channels before rotation. Since channel-wist quantization is not supported in most hardware, outlier removal using rotation enables accurate token-wise tensor-wise quantization.
Berfore recation
4.31 weig quandizaonn oroor
Before rossion
1500
rocation
M Mr
44t actentton qUammtato
AMM
Aller
2002
roseton
After rocatio
6.26
02
(a)
(b)
(c)
Figure 3:
Outlier measurement and quantization error across input activation and weights in the five layers
that take inputs from the residual (O/K/V/Up/Gaat-ppr/ projeecto)))))ooin of cach block in the LLaMA-2 7B model. (a) After rotation, kurtosis of activation distributions significantly reduced to approximately three across all layers. Quantization reduced after rolation in both (b) activations and weights.
2 MOTIVATION
Quantization reduces the precision of weights (and/or activations) in neural network in order to save memory and lower the latency. The quantization process can be formulated as:
XQ al XR 3 +
where a max((Xal)
min(min asymmetric quantization. Here Xę is quantized tensor and XR is real-valued FP16 tensor. N number of bits. For Large language models (LLMs), the presence of outliers extends the range of weight/activation values and increases the reconstruction errors for normal values (Dettmers et al.
22-1-1 3 symmetric quantization or a max(Xx))min(Xx) 2N
2022
Liu et al. 2023b Yelysei Bondarenko 2023) (Figures (a)&(c)).
2.1
OUTLIER REDUCTION
There exist many ways to mitigate the effect outliers (Xiao al. 2022, Dettmers al. 2022). In this paper, we propose to use optimized rotation to reduce outliers. Intuitively, random rotation matrix statistically blends large and small weights together into well-behaved distribution with fewer outliers (Elhage et al. 2023), and thus is easier to quantize.
Figure (a) illustrates the measurement the Kurtosis x of the activations before and after rotation. quantifies the "tailedness" of real-valued random variable's probability distribution. A larger indicates more outliers, while x 3 suggests a Gaussian-like distribution. Figure 3 (a), the activation distribution in the transformer contains numerous outliers, with of many layers exceeding 200. However, after multiplying these activations with a random rotation matrix, the k across all layers becomes approximately 3, indicating more Gaussian-shaped distribution that is easier to quantize. This is corroborated by Figure (b), where the quantization error of the activation tensor significantly decreases after rotation.

---

Published as a conference paper at ICLR 2025
Zero shot Reasoning Accuracy (Avg.)
60
Random
Hadamard
Cayley optimized
Figure 4: The performance distributions of W4A4 quantized LLaMA-2 7B under different random rotations, using network-level parameterization (Sec. 3.1 We compare the distributions using random floating ppinnt rotations, random Hadamard matrices, and optimized rotation matrices with Cayley optimization (Sec. (.2) Despite that Hadamard matrices mostly perform better than random rotations, both random groups demonstrate large variance. contrast, by optimizing the rotation matrix with Cayley optimization (i.e., SpinQuant), the performance is improved significantly and the variance becomes much smaller.
2.2 RANDOM ROTATIONS PRODUCE LARGE VARIANCE
Interestingly, while statistically random rotation leads to better quantization, not all random rotations give the same quantization outcome. To show this, we tested the zero-shot average accuracy of the rotated version of LLaMA-2 7B, quantized 4-bit weight and 4-bit activation, under 100 randomized trials. As shown in Figure 4 the performance variance is substantial, with the best random rotation matrix outperforming the worst by 13 points. Random Hadamard matrices outperform random rotation matrices, in consistent with the findings in (Teng et al. 2024) that Hadamard matrices yield tighter bounds on weight maximal value. However, even random Hadamard rotation matrices exhibit non-negligible variance in final performance, as large as 6 points.
Given the huge variance across multiple trials of rotations, natural question arises: Is it possible to optimize the rotation to maximize the benefit of quantization? We affirmatively answer this question by presenting a viable framework with quantization-oriene rotation learning that consistently achieves high accuracy across 7 models and 4 low-bit quantization settings.
3
METHOD
In this section, we introduce SpinQuant, framework that integrates and optimizes rotations in LLMs targeting at quantization loss. We start with defining rotation parameterization of popular LLM architectures, which includes two mergeable rotation matrices (R1, R2) that produce rotationally invariant full-precision network, and two online Hadamard rotation (R3, R4) to further reduce the outliers for extreme activation and KV-cache quantization. Then, we present how optimize these rotation matrices on Stiefel manifold with target loss.
3.1 ROTATION PARAMETERIZATION
Rotating activations in residual As shown in F a). rotate the activations in the residual path by multiplying the embedding output X with a random rotation matrix (R1). This rotation removes outliers and eases the quantization of the input activations the fully-connected layers that read from the residual. To maintain numerical invariance, we reverse the rotation of the activation by
A Hadamard matrix H is special type of rotation matrix, where the entries of the matrix are solely +vnn Given Hadamard matrix H, generate 2n different random Hadamard matrices by multiplying with S, diagonal matrix with elements randomly chosen from (-1,1).
4

---

Published as a conference paper at ICLR 2025
multiplying it with R (= (=R R network, which contains non-linearity. When the quantization not present, the full-precision network remains intact no matter which rotation is applied The rotation matrices be merged into corresponding weight matrices, as illustrated in Figures &T))& After absorption, no new parameters are introduced in the network. We can modity Rı freely without impacting the floating-pon network's 's accuracy or parameter count.
prior to its passage through the attention block and corwafo
Rotating activations in the attention block As depicted in Figure [b) in the attention block, we propose to rotate the value matrix by multiplying R2, and the activations to out-projection layer by R3 head-wisely. Rg has the shape of (Dhead. Dhead) and can be independently chosen across layers. The numerical in- varianeee is illustrated in Figure [] these two rotations can be offset in full-precision network since there are no operators between R2 and R2 Meanwhile, it can improve quantization for value cache and input activations to out-projection layer without introducing any new parameters in the network.
OuE
Artn
x
W
R
W
Figure 5: Rotation equivalence in
Self-Attention.
We denote the method with only R₁ and R2 inserted and optimized as Spi nQuantin had, which readily achieve significant accuracy improvement than previous quantization methods, and closing the gap between W4A8 quantized LLMs and their full-precision counterparts to 0.1 2.5 points on zero-shot commonsense reasoning averaged aceuracy.
Additional unabsorbed rotations To further enhance outlier suppression for lower-bit (e.g. 4-bit) activation quantization, we incorporate Hadamard matrix multiplication (R, in Figure ([e)) inside the feed-forward block, reducing the outliers in the input to the down projection layer, similar to (Tseng al., 2024 Ashkboos et al., 2023b). Hadamard rotation can be computed with fast hadamard transform and introduce marginal overhead to the inference latency. Similarly, Hadamard matrix (R3 in Figure [b)) can be inserted when low-bit KV cache quantization is required. We denote the resulting method, equipped with all rotations, as SpinQuanthod Next, we demonstrate how to jointly optimize these rotations.
3.2 Cayley-OPTIMIZED ROTATION
As illustrated in Figure I we have determined that the incorporation of four rotation matrices (R1, R2, Rs. RA) can improve quantization performance while preserving numerical consistency in full-precision network. Given that R3 and R4 are online rotation operations. meaning they cannot be absorbed into the weight matrix, we retain them Hadamard matrices. This is because online Hadamard transforms can be efficiently implemented without significant overhead. We then define the optimization objective as identifying the optimal rotation matrix R₁ and Rg that minimizes the final loss of the quantized network:
arg min Lo(R1, R2 W.X) REM
(2)
Here, M represents the Stiefel manifold i.e., the set of all orthonormal matrices. Lo(+) denotes the task loss, such as cross-entropy, on the calibration set. It function of (R1. R2), given the fixed pretrained weights W and the input tensor X and with the quantization function Q in the network. To optimize the rotation matrix on the Stiefel manifold, we employ the Cayley SGD method (Li et al., 2020), which is an efficient optimization algorithm on the Stiefel manifold. More specifically, in each iteration, the update of the rotation R is parameterized as the following:
R' = =R()) (1- (
(I
Y R
"In pre-norm LLM like LLaMA Touvron et al. 2023a we can convert a transformer network into rotation-invariant network by incorporating the RMSNorm scale parameters into the weight matrix right after the RMSNorm layer (Ashkboos et al. 2023a
5

---

Published as a conference paper at ICLR 2025
where AR(Y) = (I- (1- gY) is the Cayley Transform of skew matrix Y (i.e.,
Y = -Y). ). is computed from a projection & of the gradient := VRCQ of the loss function:
Y Y=G-
C==GRT RRR GR
It can be shown that AR(Y) is always orthonormal and thus R' is guaranteed to be orthonormal (R'T R' 1) if R is orthonormal. While Eqn. requires matrix inverse, the new rotation matrix It' can be computed via efficient fixed point iteration (Li et al. 2020). Overall, the approach maintains the property of orthonormality with only ~2 times the computation time per iteration compared to a naive SGD algorithm.
We apply the Cayley SGD method to solve Eqn. 2 for { R2), while the underlying weight parameters in the network remain frozen. { R2/ / count for only ~0.66% of the weight size and constrained to be orthonormal. Consequently, the underlying floating-poins network remains unchanged, and the rotation only influences the quantization performance.
By employing Cayley optimization to update the rotation for 100 iterations on an 800-sample WikiText2 calibration dataset, we obtain rotation matrix that outperforms the best random matrix and random Hadamard matrix in 100 random seeds, shown in Figure The Cayley-optimized rotation exhibits minimal variance when initiated from different random seeds. The rotation matrices are initialized with random Hadamard matrices for optimization and our ablation study in Section 4.3.3 demonstrates that the optimized rotation is robust to random rotation initialization as well.
4 EXPERIMENTS
We conduct experiments on the LLLMA-22 (Touvron et al. 2023b) models (7B/13B/70B), LLaMA- (AI@ Metal 2024) models (IB/3B/8B) and Mistral (Jiang al.] 2023) 7B model. Our evalu- ation of the proposed SpinQuant was carried out on eight zero-shot commonsense reasoning tasks. These tasks include BoolQ (Clark et al., 2019). PIQA (Bisk et al. 2020). SIQA (Sap et al.. 2019), HellaSwag (Zellers et al. 2019), WinoGrande (Sakaguchi et al. (202)). ARC-easy and ARC- challenge (Clark et al.) 2018), and OBQA (Mihaylov et al. 2018). Additionally, we also report the perplexity score Wiki testset (Merity et al. 2016) for our evaluation.
4.1 EXPERIMENTAL SETTINGS
We employ Cayley SGD Li et al. 2020) to optimize the rotation matrix, R1 and R2, both initialized as a random Hadamard matrix, while maintaining all network weights constant. R1 is the residual rotation, shaped as (Droken, Dtoken). R2 is head-wise rotation in each attention block, shaped as (Dread; Dread) and is separately learned in each layer. The learning rate starts at 1.5 and linearly decays to O. We utilize 800 samples from WikiText-2 optimize rotation for 100 iterations. It takes only 13 18 30 minutes for LLaMA-3 IB/ / 3B/ 8B, respectively, and ~ 25 / 30 minutes for LLaMA-2 7B/ / 13B, respectively, For LLaMA-2 70B, it takes 3.5 hours and for Mistral-7B it takes 16 minutes.
In the main results, we optimize the rotation with respect to the activation quantized network. where the weights remain bit After rotation is learned. we apply GPTQ on the rotated weights (Frantar al. 2022), for which we adhere the standard GPTQ settings by using 128 samples from WikiText- 2 with a sequence length of 2048 as the calibration set for GPTQ quantization. In the main table, we present the results SpinQuant with GPTQ, and in the ablation study, while we also show the results of employing simple round-to-nearest (RTN) quantization in the ablation study.
4.2 MAIN RESULTS
We present two rotation schemes scenarios. In Table [] we use seven models and four most commonly used bit-width settings to provide guideline which rotation scheme should be chosen in practice.
had and SpinQuanthad accommodate different
Recap SpinQuant had uses learned rotation R₁ and R2 only, which can be merged into corre- sponding model weights during inference time after the rotation learned. Using Spi nQuantro had

---

Published as a conference paper at ICLR 2025
Table Comparison the perplexity score on WikiTex12 and averaged accuracy on eight Zero-shot Common Sense Reasoning tasks. Results for SmoothQuant (Xiao et al. (022). LLM-QAT (Liu et al 2023c), GPTQ (Fran
tar et al. 2022 were obtained using their publicly released codebase. While OmniQuant (Shao et al. 2023).
AWQ (Lin et al. 2023). and QulP# (Tseng et al.
2024 results were quoted from their papers. Full results
the Appendix.
[LaMA-22
7B
LL-L-- LLMA2 70B
LLAMA-32 LB,ML.
3B
LLaMA-3 8B
Mistral-BB
BBits
Methud
Dishor
Wiki
10
Wiki 10-shoi Wiki
Dishot
Wiki
10-sho
Wiki
O-shot
Wiki
Dahoo Wiki
(WLA-VV)
Avg.(t)
(1)
(Avg(()
(Q) (tvg())
())
Arget)
(1)
(vg
(1)
Avg.(t)
(
Avg(())
(
16-16-16
FloatingPuimt
66.9
5.5
68.3
5.0
1. 72.9
3.3
56.9
13.4
63.9
10.7
69.6
6.1
71.0
5.4
RIN
62.4
51,3"
-!- 68.6
5.0
55.4
20.7
58.6
29.0
3.2
30.3
6.8
SmoothQuant
58.9
7.5
63.6
6.1
70.6
4.1
47.1
1e2
55.6
3e2
61.0
10.7
LLM-QAT
64.8
14.5
53.2
AWQ (w4)
6.2
5.1
4-8-16
OmniQuant (w4)
5.7
5.0
3.5
QuIP# (w-4
50
3.4
GPTQ
64.9
20.2
65,2
5.9
71.7
4.3
55.0
17.3
58.7
25.2
64.5
7.2
$1.7
8.6
Spinqueii... And
65.7
72.1
56.0
Spinguanted Boed
65.7
5.7
68.1
5.0
72.7
3.5
56.5
14.4
63.2
11.5
68.4
6.5
69.9
5.5
62.5
$7.6
6.7
68.4
5.0
55.7
20.7
58.4
28.8
65.3
8.2
58.9
6.7
SmoethQuant
58.8
7.5
63.4
6.1
70.5
4.1
47.1
1e2
55.5
3e2
60.9
10.7
LLM-QAT
64.6
53.1
39.3
7.6
4-8-8
GPTO
64.8
20.2
65.3
5,9
71.6
4.3
54.8
17.3
58.7
24.1
64.6
7.2
51.7
8.6
Sp Spinquantes
65.8
5.8
68.1
5.1
72.2
3.7
55.7
15.3
61.8
11.7
68.6
6.7
69.4
5.7
Spinguantaed
65.8
72.7
RĀN
35'6"
2:3
53.3
7:3
35.1
"285"
-1122
12"
42.T
45.9
2:22
41.4
422
SmaothQuant
41.8
3e2
44,9
34,5
57.7
$7.1
37.9
2c3
43.6
4e2
40.3
9e2
4-4-16
LLM-QAT
GPTQ
36.8
47,8
12.9
34,3
4c3
35.5
41.6
42.0
62.1
46.9
37.6
44.9
42.9
spinguenton.
$7.0
9.2
61.8
7.2
61.0
7.3
44.8
48.4
$2.9
22.4
$1.9
18.6
$2.7
13
Spinguanttaed
64.1
5,9
67.2
5.2
71.0
3.8
53.5
15.3
61.0
11.1
65.8
7.1
68.4
5.7
RIN
37.1
2:3
'3.33"
7:3
35.0
"285"
40.6
22"
5c3
95.1
3:2
31.4
722
Smacthhuant
39.0
7e2
40.5
56.6
55.9
10.5
36.5
263
40.0
6c2
38.7
2:3
4-4-4
LLM-QAT
44,9
14.9
35.0
4c3
41.5
76.2
45.9
42.0
43.2
52.5
GPTQ
36.8
35.6
41.6
4c2
2c2
2c2
spinguanton And
56.0
9.2
60.7
7.1
62.0
7.4
45.3
47.7
52.9
22.4
52.6
18.6
52.4
13.7
QuinQuent
64.0
5,9
66.9
5,3
71.2
3.8
53.4
15,9
60.5
11.4
65.5
7.3
68.6
5.8
only needs to replace the original model weights with the rotated model weights, necessitating no modification to the forward pass nor any additional kernel support. While Spinguanthed comprises both learned rotations (R1, R2) and the online Hadamard rotations (Rg, R4). During inference time, R3 and R4 can be computed with fast Hadamard kernel (Tseng al., (024) and we show in Sec. 4.5 the online Hadamard rotation only introduces ~8% of the network latency overhead.
As shown in Table in the scenarios where weights are quantized 4-bit and activations are quantized to 8-bit, using SpinQuant,o had can readily achieve good performance. For exam- ple, SpinQuant had enhances the 4-8-8 quantized Mistral 7B by 10.5 points. In 1lama3-8B, SpinQuantno had achieves more than 4.1 point improvements compared to GPTQ (Frantar et al., 2022) on 4-8-16 setting, and leaving the gap to full-precision network to only 1.0 point. In these settings with activations not extremely quantized, using Spin SpinQuant had is a viable solution, and adding additional online Hadamard rotation yields marginal benefit.
In contrast, when activations quantized 4 bits, the accuracy drops significantly and most previous methods fail to produce meaningful results. Spinquant had bridge the gap by up to 20 points. In 4-4-4 quantized LLaMA-2 models, SpinQuant.o hed significantly surpasses LLM-QAT (Liu et al. (2023c), by 11.1 points on 7B model and outperforms SmoothQuan (Xiao et al. 2023) by 20.2 on the 13B model, thereby reducing the gap the corresponding full full-precision network from 22.0/ / 27.8 points to 10.9 / 7.6 points respectively. Still, the gap to the full full-precision network is this scenario, SpinQuanthed can further improve the accuracy by more than points and close the gap to the respective FP network to 2-4 points. In 4-4-4 quantized I.LaMA-2 7B/13B/70B models, Spi nQuanthad leaves only 19/1. accuracy gap to the corresponding full-precision network, significantly surpassing the previous SoTA methods by 19.1/16.4/15.3 points, respectively.
nog-neglig
In addition, compared to the state-of-the-art weight-only quantization methods, OmniQuant (Shao) et al. 2023), AWQ (Lin et al. 2023) and QulP# (Tseng et al., 2024), SpinQuant achieves similar evaluation perplexity on Wiki dataset with 4-bit weights and 8-bit activations, and without using advance vector quantization technique. These results show SpinQuant is suitable for various scenarios and achieves state-of-the-art performance.
7

---

Published as a conference paper at ICLR 2025
Table 2: Compared to Hadamard rotation, SpinQuant learned rotation consistently outperform by significant margin. Results are averaged accuracy on eight sero-shot CommonSense Reasoning tasks.
LLaMA-3.2 3B
LLaMA-3 8B
Mistral-7B
4-4-16
4-4-
4-4-16
4-4-4
4-4-16
4-4-4
Random Hadumard R
49.8
49.6
49.5
50.0
51.4
51.5
spinquantwo Ria.)
12
12.9
41
52.6
12.7
12.4
Random Hadumaird R (1.2..4)
35.
38.4
64.2
65.9
52.7
52,4
Sp peoinquantaom aad (11.2.3.4)
(1.0
(0.5
65.8
65.5
68.4
68.6
Table 3: Ablation study on compatibility with GPTQ Frantar et al. 2022 on I.L.MA2-BB model.
#Bits AKV)
Task
Cayley on 4-4-KV
Cayley 16-4-KV
4-4-16
0-shot Avg.
Wiki
61.0 =1.
6.7 20.07
64.1 20.4
5.9 $0.00
4-44
O-shor Avg.
60.9 20.6
64.0 20.3
Wiki
6.8 20.15
5,9
4.3 ABLATION STUDIES
4.3.1 LEARNED ROTATION VS RANDOM ROTATION
In Table 2 we contrast the use of random Hadamard rotations with SpinQuant's optimized rotations. Employing learned rotations, whether under R1,2 settings or R1,2334 settings, consistently enhances accuracy across various models and bit-width configurations. Notably, in the quantization of Mistral-7B, Spin aepinnuantiaan secures an improvement exceeding 15.7 points over using random Hadamard rotations. Given that rotation optimization incurs minimal time cost (only 30 minutes for smaller models and up to 3.5 hours for 70B model) we advocate for the adoption of optimized rotations for precise quantization of LLMs.
4.3.2 COMPATIBILITY WITH GPTQ
In the context where both weights and activations are quantized, we observed that the learned rotations tend to adapt effectively to both weight and activation quantization. Given that GPTQ significantly helps mitigate the errors due to weight quantization, but leaves activation quantization untouched, we elect to optimize the rotation matrices with respect to network where only activations are quantized. This approach allows the rotation to more efficiently manage the activation quantization error while leaving the weight quantization error to be addressed by GPTQ. As shown in Table 3 this modification resulted in superior performance in both W4A4 and W4A4KV4 settings in the LLaMA-2 7B model, which is the configuration we choose to utilize throughout the rest of this paper.
4.3.3 ROTATION TYPE
In Table we evaluate the impact of random orthogonal floating-poii rotation matrices and random Hadamard matrices on quantization accuracy, utilizing round-to- -o-neesst quantization for our analysis. Prior to optimization, the Hadamard matrices yield a better-quantized network performance compared to floating-pott poatingg--ontttto rotation matrices. However, after optimization, the initial choice of rotation, whether floating-pon point Hadamard, becomes less significant. This is likely due to the loss-aware rotation optimization's 's ability to locate an optimal local minima that effectively minimizes quantization error, thereby enhancing robustness varying types of rotation initialization.
4.3.4 COMPARISON WITH QUAROT
Compared QuaRot (Ashkboos et al., 2023b), which exhibits significant accuracy variances in quantized networks networks-experieen experiencing drops of 28.1 1 and 33.2 points when quantizing a 70B model with round-to-nearest methods to W4A4 and W4A4KV4 degradation stems from inherent noise in using random rotation matrix that introduce high variance and compromise robustness. In contrast, SpinQuanthod consistently maintains high accuracy across various configurations, achieving improvements of 2.0 to 28.6 points over QuaRot (Table 5). while utilizing fewer online Hadamard matrices (two per block in Spinquant had versus four per block in QuaRot).
Furthermore, the integration of R₂ in SpinQuant effectively reduces in-block outliers, thereby enabling SpinQuant,, had to deliver optimal performance in W4A8 settings. SpinQuant,,. had can be achieved by simply substituting the model weights with rotated weights, making it more
8

---

Published as a conference paper at ICLR 2025
Table 4: Floating ponnt((PP rotation vs Hadamard rotation on a LLaMA-2 7B model.
#Bits (W-A-KV)
No Cayley RTN Hadamard
Cayley RIN
Task
FP init.
Hadamard init.
4-16-16
O-shot" Avg.(t)
62.5 10.8
62.4 +1.0
64.9 20.4
64.6 20.3 5.5 10.01
Wiki(()
6.7 10.12
6.9 20.45
5.5 10:01
4-4-16
0-shot Avg.(t)
Wiki(()
15.9 14.04
49.4 +2.8
59.0 +1.0
8.2 20.73
61.6 +0.4
6.2 20.00
61,8 +0.4
6.1 20.0
4-4.4
O-shot Avg.(t)
48.3 12.7
58.7 11.0
61.5 10.8
61.5 10.3
Wiki(()
18.2
14:35
8.2
20.36
6.3 40.00
6.2 20.00
Table 5: Comparison with QuaRot (Ashkboos et al. 2023b
LLaMA-3 8B (FP: 69.6, 6.1)
LLaMA-3 70B (FP: 74,5,
4-4-16
4-4.4
4.4-16
4.4.4
0-shot Avg.(t)
Wiki D-shot"
Wiki
0-shot
Wiki
0-shot
Wiki
(J)
Avg.(t)
(1)
Avg-(t)
(1)
Avg.(t)
(1)
QuaRot+RTN
59.5
10.4
58.6
10.9
41.5
91.2
41.3
92.4
Spinguant hou RTN
64.6
7.7
64.1
7.8
70.1
4.1
70.1
4.1
Spi GPTO
63.8
7.9
63.3
8.0
65.4
20.4
65.1
20.2
65.8
7.1
65.5
7.3
69.5
5.5
69.3
5.5
straightforward and efficient approach compared QuaRot, which requires modifying the model architecture and special kernel support.
4.4 ILLUSTRATIVE ANALYSIS OF THE ROTATION EFFICACY
The rationale behind rotating network weights and activations can be elucidated through a straightfor- ward example. Consider activation (X) represented as 2D vector, where one entry I1 consistently receives higher magnitude activations than 222 (as depicted in Figure 6 [)). Quantizing these compo- nents together typically results in quantization range dominated by 21, thereby compromising the precision for 2+2.
From information entropy standpoint, expanding each axis to fully utilize the available quantization range maximizes the representational capacity of each axis. Thus, matrix rotation emerges as an intuitive solution. In a 2D scenario, rotating the axis by 45 equalizes the value representation range across axes (illustrated in Figure b()).. Assuming the network as a black box without knowledge of the exact activation distribution, uniformly rotating all axes by the maximal degree (45% in 2D) can optimize distribution evenness across each axis. partially explaining why Hadamard rotation often outperforms random rotation matrices.
Taking this further, if the activation distribution is known, treating the network as a white box during quantization allows for the identification of more optimal rotations than Hadamard. For instance, in 3D scenario depicted in ( where xi 's magnitude is four times that of and 2z, rotating the distribution by 45 along 33 and 22 redistributes the maximum values from [2.0.5 0.5, 0.5] to [1, 1, 1.414). However, even more optimal rotation strategies may exist, and learning the rotation can help pinpoint the most effective rotation for a given distribution.
This opens up intriguing rescarch avenues, such determining if, given activation distribution with known outlier axes and magnitudes, closed -fom solution for the optimal rotation matrix that evenly distributes magnitude across different axes can be derived. Additionally, it raises the question
of
Figure 6: An illustration how rotation helps reduce outliers and maximize quantization range utilization.
9

---

Published as a conference paper at ICLR 2025
Table 6: Real-time and-to-end speed measurement of 3 on MacBook MI Pro CPU.
Method
w.)
Decoding speed
FloatingPoint
16-16
177. ms/token
Sp nQuannt... luad
4-8
58.88 ms/token
SpinQuant kad
4-8
63,90 ms/token
of whether this theoretically calculated rotation yields the best quantization performance. We leave this question to future research.
4.5 SPEED MEASUREMENT
We conduct an end-to-ent speed measurement of the LLaMA-3 8B model with W16A16 and W4A8 configurations on MacBook MI Pro CPU (OS version 14.5). The results in T that 4-bit quantization yields ~33 speedup compared to the 16-bit model. Comparing Spi quuanthad to SpinQuantno had, online Hadamard processing introduced modest 8% increase in latency. Therefore, is trade-off between using SpinQuantro had without online Hadamard for its simpleness using Spi nuuant with online Hadamard rotations for higher accuracy in lower- bit activation quantization. Detailed GPU latency results are provided in the Appendix.
5 RELATED WORK
Quantization Neural network quantization has been demonstrated as effective tool for model size compression and storage reduction (Nagel et al.) 2020 Krishnamoorthi 2001 Nagel et al. 2019 Li et al. (021). However, in large language models (LLMMs), quantization presents unique challenges due to the presence of numerous outliers. These outliers dominate the quantization range, leaving only few effective bits for the majority of values. Various strategies have been proposed to address the difficulties in LLM quantization. These include separating outliers and using mixed precision (Dettmers et al. 2022 van Baalen et al.) 2023] Kim al., 2023; Huang et al. 2024
Egiazarian et al. (024)) employing Hessian methods to mitigate quantization difficutty (Frantar
al. [al2). trading outliers between weights and activations (Xiao et al. [0022] Lin et al. [003]
Liu et al. (202b) utilizing weight equalization (Nagel et al. 2019). outlier suppression (Wei et al.
2022; 2023). channel reassembly (Liu et al.) 2023a) and even suggesting architectural modifications
to handle outliers during pre- pre-training( Yelysei Bondarenko 2023). Recently two QulP papers (Chec) cct al. 2024 Tseng et al. 2024) introduce the incoherence processing using random rotation matrices and applying vector quantization the weights for compression. This does introduce extra overhead and imposes some constraints on the devices the LLM is deployed to in the availability of vector quantization kernels.
Optimization in orthonormal space The optimization of rotation matrices is carried out within the Stiefel Manifold (James) 1976), which encompasses all orthonormal matrices. Optimization while staying on this manifold can be done by egg., parameterizing a skew-symmetric matrix and applying the Cayley transformation on top (Nishimori and Akaho 2005). using matrix exponential (Absil and Malick 2012 Lezcano- Casado and Martinez- Rubio 2019). However, these methods rely on expensive inverse or matrix exponental functions that are applied every iteration. Instead, we follow the more efficient method named Cayley SGD (Li et al.] (020). which can be applied to optimize a rotation matrix for arbitrary loss functions efficiently. Cayley SGD relies on an iterative approximation of the Cayley Transform that is conducted solely with matrix multiplications.
CONCLUSIONS
In this paper, we present Spi nQuant, novel quantization technique that utilizes learned rotation to effectively bridge the performance gap between full precision and 4-bit weight, activation, and kv-cache quantization. At its core, SpinQuant leverages the rotation invariance property of LLM models to insert rotation matrices that diminish outliers in the weights and intermediate activations while maintaining the network's 's full-precision output numerically identical. Additionally, SpinQuant incorporates Cayley SGD for optimizing rotation matrices, resulting in improved and robust quantization outcomes. Importantly, SpinQuant is compatible with more advanced weight quantization techniques (e.g., GPTQ) and demonstrates oate-of-the-art performance.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
We extend our gratitude Scott Wolchok and Chen Lai for their crucial contributions to latency measurement on the MacBook MI Pro CPU, and to Sijia Chen, Geonhwa Jeong and Jiecao Yu for their expert support in GPU GaU'latncyy assessment.
REFERENCES
P-A Absil and Jéröme Malick. Projection-like retractions matrix manifolds, SIAM Journal on Optimization, 22(1):135-158, 2012.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
AT@Meta. Ilama model card, 2024. URL httpss///githhbb..///tttaaa----a main/MODEL CARD md
a/b3 ob/
Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. The Twelfth International Conference on Learning Representations, 2023a.
Salch Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoetler, and James Hensman. Quarot: Outlier-fre 4-bit inference in rotated llms. 2023b.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantization large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.1004: 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni. Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc. the ai2 reasoning challenge. arXiv preprint arXiv:1803.05455 2018.
Samuel Rhys Cox and Wei Tsang Ooi. Conversational interactions with npes in Ilm llm-driven gaming: Guidelines from content analysis of player feedback. In International Workshop on Chatbot Research and Design, pages 167-184. Springer, 2023.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int8(): 8-bit matrix multiplication for transformers scale. 2022.
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Flias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization. arXiv preprint orXiv:2401.06 18, 2024.
Nelson Elhage, Robert Lasenby, and Christopher Olah. Privileged bases in the transformer residual stream. Transformer Circuits Thread, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022.
Wei Huang, Haotong Qin, Yangdong Liu. Yawei Li. Xianglong Liu. Luca Benini, Michele Magno, and Xiaojuan Qi. Slim-Ilm: Salience-drive mixed-precision quantization for large language models. arXiv preprint drXiv:2405.44917. 2024.
Ioan Mackenzie James. The topology of Stiefel manifolds, volume 24, Cambridge University Press, 1976.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, al. Mistral 7b. arXiv preprint arXiv:2310.066: 2023.
Schoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: aad-s-and- quantization. arXiv preprint XxXx::3306.07622 2023.
11

---

Published as a conference paper at ICLR 2025
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint wxxiv:806.088 08340, 2018.
Mario Cercano-Casado and David Bartinez-Rubio Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group. 2019.
Jun Li, Fuxin, and Sinisa Todorovic. Efficient riemannian optimization on the stiefel manifold via the cayley transform. arXiv preprint 2xXiv:2002.0113. 2020.
Yuhang Li, Ruihao Gong. Xu Tan, Yang Yang. Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang. and Shi Gu. Breeq: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations (ICLR), 2021.
Ji Lin, Jiaming Tang, Haotian Tang. Shang Yang, Xingyu Dang, and Song Han, Awq: Activation-aware weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.0099 2023.
Jing Liu, Ruihao Gong. Xiuying Wei, Zhiwei Dong, Jianfei Cai, and Bohan Zhuang. Qllm: Accurate and efficient low-bitwidth quantization for large language models. arXiv preprint arXiv:2310.0804), 2023a.
bang Liu, Zechun Liu, Xijie Huang, Pingcheng Dong, and Kwang- Cheng. Lim-fp4: 4-bit floating- point quantized transformers. arXiv preprint arXiv:2310. 16836, 2023b.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad. Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Lim-qat: Data-free quantization aware training for large language models. arXiv preprint 2r05v 17888, 2023c.
Zechun Liu, Changsheng Zhao, Forrest landola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Emic Chang, Yangyang Shi, Raghuraman Krishnamoorthi, et al. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases. arXiv preprint arXiv:2402. 14905, 2024.
Stephen Merity, Caiming Xiong. James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07844 2016.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv: 1809.02789, 2018.
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Drta-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1325-1334, 2019.
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? Adaptive rounding for post-training quantization. In International Conference on Machine Learning (ICML). 2020.
Yasunori Nishimori and Shotaro Akaho. Learning algorithms utilizing quasi-geodesic flows on the stiefel manifold. Nearocomputing, 67:106-135, 2005.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li. and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, ((())5485-5551) 2020.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. /2950, 2023.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communications of the ACM, 64(9):99-106, 1069 2021.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:/904009) 9904.0928. 2019.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao. Yu Qiao, and Ping Luo. Omniquant: Omnidinectionally calibrated quantization for large language models. arXiv preprint arXiv:2308. 13137, 2023.
Gemini Team, Rohan Sebastian Borgeaud, Yonghui Wu, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: a family of highly capable multimodal models. arxiv preprint arXiv:2312. 1805, 2023.
12

---

Published as a conference paper at ICLR 2025
Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez. Ting Fang Tan, and Daniel Shu Wei Ting. Large language models medicine. Nature medicine, 1909)11930, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Llama: Open and efficient foundation language models. arXiv preprint Arv::23302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09888 2023b.
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quipf: Even better Ilm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.04336, 2024.
Mart van Baalen, Markus Nagel Andrey Kuzmin, Peter Couperus. Cedric Bastoul, Eric Mahurin, Tijmen Blankevoort, and Paul Whatmough. Gptvq: The blessing of dimensionality for Ilm quantization. 2023.
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. arXiv preprint WwXiv:2209.13322 2022.
Xiuying Wei. Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outler suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling, arXiv preprint arXiv:2304.099 09145, 2023.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In CVPR, 2022.
Guangxuan Xiao, Ji Lin, Mickael Scznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post cest-training quantization for large language models. International Conference on Machine Learning, pages 38087 38099. PMLR, 2023.
Tijmen Blankevoort Yelysei Bondarenko, Markus Nagel. Quantizable transformers: Removing outliers by helping attention heads do nothing. 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk. Farhadi, Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: 1905.07830 2019.
Yilong Zhao, Chien Yu Lin, Kan Zhu, Zihao Yc, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate Ilm serving. arXiv preprint arXiv:2310. 19102, 2023.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX / SUPPLEMENTAL MATERIAL
A.1 COMPLETE RESULTS OF MAIN RESULT TABLE
In Tables 7 and we show the complete results of Table 1 We compare the accuracy on eight
zero-shot commonsense reasoning tasks including ARC-easy, ARC-challenge (Clark et al. 2018) 8) BoolQ (Clark al. 2019), PIQA (Bisk et al. 2020), SIQA (Sap et al. 2019). HellaS wag Zellers al.) 2019). OBQA (ihhaylov et al. 2018), and WinoGrande (Sakaguchi al. 2021 well as the perplexity score on WikiText2 testset (Merity et al. 2016). We compare our results with previous
works including SmoothQuant Xiao et al. 2022), LLM-QAT et al. 2023c). GPTQ (Frantar et al.,
2022).
OmniQuant (Shao et al. 2023), QulP# (Tseng et al. 2024))
A.2 RESULTS ON 3-BIT WEIGHT QUANTIZATION
We present the 3-bit weight and 8-bit activation quantization results across seven models in Table I0 Our method, SpinQuant, successfully reduces the gap to the full full-precision network from the previous 9.0 28.0 points 1.2 5.3 points, demonstrating its effectiveness for low-bit quantization.
A.3 Cayley OPTIMIZATION CHOICE
In Table II we evaluate the impact of varying the number of samples and iterations used in Cay- ley optimization. Given the limited trainable parameters the rotation matrix and its constraint optimization nature, minimal calibration data and iterations sufficient to optimize the rotation for better quantization. The findings indicate that rotation optimization is resilient to modifications in the number of samples. Even though we used 800 samples in our experiments, reducing this to 128 samples does not lead to significant change in the perplexity. Furthermore, we examined the optimal number of iterations and found that the wiki perplexity ceases decrease and stabilizes at 100 iterations. Consequently, we chose to use 100 iterations in all our experiments.
A.4 QUANTIZATION CHOICE
We conduct an ablation study on symmetric vs asymmetric quantization and whether to clip the min-max ranges or not during activation and KV-cache quantization. The results in Table show that for both activation quantization and V-ccchhe quantization, asymmetric quantization outperforms symmetric quantization. In the clip settings, we the activation clipping ratio to 0.9 and the KV-cache clipping ratio to 0.95 as suggested in the previous works (Zhao et al. (023). However, the results show that clipping the range or not does not impact the final result significantly. Therefore we opt for no clipping, i.e, using the min-max quantization for activation and KV cache quantization across our experiments due its simplicity.
A.5 CALIBRATION DATA CHOICE
To assess the robustness of Spin Quant with respect calibration data used in rotation optimization we use C4 dataset (Raffel et al., 2020) as calibration data and performe experiments on the LLaMA-2 7B model. The results in Table 13 reflect that using C4 datasets yields consistent results with utilizing the Wiki dataset, showing that SpinQuant is robust to calibration data choice
A.6 LATENCY MEASUREMENT ON GPU
In light of the available Tensor cores in NVIDIA's Hopper (H100) architecture, we provide the whole network end-to-end speed test result of W-fp8-A-fps quantization H100 GPU, both with and without Hadamard transformations. Specifically, we utilize FPS GEMM from the FBGEMM repo which incorporates dequantization via epilogue fusion. We also leverage the Tensor Core-based Hadamard transform kernel to minimize the overhead of the online Hadamard transform. The end-
fp8 gemm.py http://iwwoyytoooooaoaaaoaaaaaaa
hadamard transform.epp
14

---

Published as a conference paper at ICLR 2025
Table 7: Complete comparison of the perplexity score on WikiTex12 and averaged accuracy on Zero-shot
Common Sense Reasoning tasks on LLaMA-2.
Model
#Bits W.A-KV
Method
ARC-
(1)
ARC-c BoolQ
(t)
(1)
PIOA
(1)
SIQA
HellaS.
OBQA (1)
WinoG.
(f)
Arg.
Wikiz
16-16-16
Foll Precision
75.0
50.8
77.3
78.9
48.5
76.0
59.3
69.5
RTN
70.9
44.3
73.5
76.8
46.0
70.3
51.8
65.9
62.4
7.9
SmoothQuant
L.M QAT
73.6
49.0
72.4
78.2
47.8
74.0
56.1
67.7
64.8
AWQ (w-4)
6.2
OmniQuant (w4)
5.7
QulPa (w4)
GPTQ
73.7
47.5
77.7
46.4
69.3
Spinquant Aud
73.6
49,4
79.0
47.8
68
Spi nwu
74.0
50.1
74.4
78.9
47.6
74.8
56.7
68.9
65.7
RTN
71.1
44.3
76.8
45.8
70.3
52.3
65.8
62.5
7.9
SmoothQuant
76.3
I.L. QAT
73.5
48.3
72.4
78,1
47.4
74.0
55.3
68.0
64.6
114
7B
GPTQ
73.7
48.0
74.2
78.
46.6
73.9
55.1
68.5
64.8
20.2
Spinquant
25.1
49.8
74.7
78.2
47.8
75.0
$7.6
67.7
65.8
5.8
Sp1 nwu
73.4
49.6
76.0
78.4
47.7
74.6
56.2
70.3
5.7
RTN
26.6
22.1
44.3
50.9
38.9
26.2
26.6
494
35.6
2.1672
SmoothQant
37.8
27.1
$1.9
59.4
40.2
34.1
31.6
52.4
41.8
254.5
4-4-16
L.M QAT
46.2
32.4
61.8
62.0
41.3
47.6
36.1
54.7
47.8
12.9
GPTQ
27.6
24.9
50.7
38.6
28.3
49.9
8,949.0
SpinQuant Aual
72.6
Spinguant/jui
72.1
47.5
74.4
77.0
47.3
73.2
54.4
66.9
64.1
599
RTN
27.1
24.4
44.8
51.4
39.4
26.7
33.0
50.0
37.1
2.382.5
L.M. QAT
42.0
27.7
59.5
58.9
41.0
43.1
33.5
$3.3
44.9
14.9
GPTQ
27.6
23.6
47.8
51.0
38.7
27.0
28.5
50.3
36.8
9.253.1
Spinnuaant
61.8
39.1
64.8
71.6
44.5
65.0
41.4
60.0
56.0
9.2
SpinQuant
72.6
47.5
73.9
77.0
47.2
73.0
54.1
66.9
64.0
5.9
16-16-16
Full Precision
5144
50.5
72.5
RTN
6..
3979
66..7
74.6
36.2
36.7
45.3
6T5
37.5
6.7
SmoothQuant
71.7
46.3
72.0
78.2
47.3
51.2
69.2
63.6
6.1
LLM-QAT
75.3
49.7
79.0
80.0
50.3
77.4
56.3
71.6
67.5
14.5
4-8-16
AWQ (w4)
5.1
OmniQuant (w-4
5.0
QulP# (w4)
GPTQ
74.1
Spinguuantan And
76.5
52.0
81.5
80.0
49.9
78.8
5A.8
72.4
68.2
51
Spinguant
76.2
50.6
80.1
80,1
49.8
78.5
58.0
71.7
68.1
5.0
RTN
62
30.T
74.55
36.
46.2
37.6
6.7
SmeothQuant
73.3
45.3
71.9
78.8
47.6
72.7
49.6
67.7
6.1
LLM-QAT
75.0
48.8
79.2
80.3
50.7
77.7
56.1
72.3
67.5
14.2
13B
GPTQ
74.1
48.8
75.1
78.1
48.8
74.1
53.6
69.5
65.3
5.9
Spinguant
76.8
52.1
80.8
80.5
49.9
55.8
70.6
SpinQuant
51.2
71.5
5.1
RTN
26.0
26.0"
39.7
26.6
"2R
35.
35.5
72/67
SmmelthQuant
45.2
27.1
55.4
62.5
40.5
44.3
33.4
50.8
44,9
34,5
4-4-16
LLM-QAT
26.0
23.7
37.8
49,2
39.5
23.8
48.
1,889.9
GPTQ
Spingusntee And
68.5
43.0
72.1
75,4
48.5
71.2
51.0
64.6
61.8
72
Spinguantaad
75.9
50.8
78.1
79.5
49
77.5
58.2
70.8
67.2
$2
RTN
24.1"
38.7
3906
25.8
29.2
29.6
35.5
7,2N
SuncothQuant
36.9
24.8
49.4
57.2
39.6
33.3
31.2
$1.7
40.5
56.6
4-4-4
LLM-QAT GPTQ
26.3
24.6
37.8
48.8
39.3
26.3
26.8
504
35.0
3.777.5
26.6
24.1
37.9
48.8
38.9
26.1
29.1
50.1
35.2
5,237.1
Spinguant... Aual
67.1
39.7
72.5
74.7
47.4
47.8
65.3
SpinQuant
75.7
50.5
79.3
79.5
69.9
16-16-16
Full Precision
82.8
50.8
59.0
80.6
78.2
80.8
46.9
73.3
SmoothQuant OmniQuant (w:4
79.4
57.3
82.4
82.0
50,3
81.5
56.2
75.9
70.6
4.1
3.5
4.8-16
QulP# (w-4)
3.4
GPTQ
#2.4
82.6
50.3
82.
77.9
Spiinuuant.o. Aual
50.3
78.1
Spinguant.we
80.2
59.9
82.5
50.4
60.1
79.3
72.7
RTN
78.
55.9"
NN
BT.
47.3
76.7
56.0
755
68.4
300
SmoothQuant
70B
GPTQ
79.6
60.3
82.4
82.2
49,9
82.2
77.3
71.6
Spinguant.e. And
80.4
60.3
84.4
81.8
49.8
82.8
59.1
79.0
72.2
37
Spinnuuant
80.4
59.7
$5.2
82.6
50.4
83.8
59.9
79.8
72.7
3.5
RTN
23.2
48.
37,0
25.6
50.5
35
SmoothQuant
69.5
71.7
29.0
66.6
73.1
45.1
67.
39.4
S7.7
57.1
4-4-16
GPTQ
25.3
25.8
45.7
50.1
36.4
25.8
24.6
50.0
35.5
2e6
Spinguant...
66.8
42.4
72.9
74.0
46.7
73.2
48.2
63.9
61.0
7.3
Spinnuanttaur
78.4
N2.7
81.4
50.2
5R.5
77.0
RTN
23.3
24.5
23.7
30.2
36.7
26.6
"A
39.
35.0
23
SmanthQuant
68.1
31.9
65.8
72.0
43,5
64.2
38.2
63.1
55.9
10.5
4-4.4
GPTQ
26.1
25.2
45.7
49.5
36.8
26.0
25.4
50.1
35.6
le6
Spingusntas And
Sp spinguant
57.6
82.1
81.7
77.3
15

---

Published as a conference paper at ICLR 2025
Table 8: Complete omparison of the perplexity score on WikiText2 and averaged accuracy on Zero-shot
Common Sense Reasoning tasks on LLaMA-3.
Model
#Bits
Method
ARC-
(f)
ARC- BoolQ
(1)
(1)
PIQA
(1)
SIQA
(1)
HellaS.
(1)
OBQA
(*)
WinG.
(1)
Avg. (1)
Wiki2
(1)
16-16-16
Full Procision
65.2
75.3
60.7
40.2
60.9
56.9
13.4
RTN
62.4
39.7
66.3
72.1
44.6
56.6
a1.
58.6
53.2
11.7
SmmotthQuant
4-8-16
L.LMQQAT
GPTQ
61.7
38.5
65.6
71.4
43.9
56.4
44.1
58.7
35.0
17.3
SpinQuantso had
60.9
39.5
65.9
73:2
46.1
57.7
44.3
00.3
36.0
15.3
Sp Sprintuatteed
60.8
39.8
66.5
73.9
44.7
59.0
46.9
60.8
56.5
14.4
RTN
SuanthQuant
48.2
31.5
59
65.4
41.7
47.2
31.5
52.0
47.1
108.6
4-8-8
LLM-QAI
60.0
36.7
62.2
73.1
43.0
$7.0
37.7
53.2
53.1
21.0
GPTQ
61.7
38.0
65.4
71.4
43.5
56.1
45.3
$7.0
54.8
17.3
SpinQuanter
61.2
40.6
64.9
12.7
45.1
58.3
43.2
59.9
55.7
15.3
SpinQuanthed
59.1
RTN
37.8
28.3
$1.2
56.4
40.0
$5.9
28.5
51.
42
1375
SmmotthQuant
32.3
26.4
46.3
54.7
39.7
28.7
27.0
48.0
37,9
2,0278
4-4-16
QAT
39.3
28.5
55.6
58.9
40.9
32.7
28.1
52.0
42.0
62.1
SpinCuant.e had
44.8
5p Reiinuanntee
59.3
37.1
64.6
69.9
44.4
55.4
412
56.0
$3.5
15.3
KTN
37.6
27.6
49.5
56.4
40.7
35.1
27.6
51.3
40.6
1604
SmmothhQuant
4-4-4
LLM-QAI GPTQ
37.7
26.7
55.7
40.6
32.0
31.3
50.5
41.5
37.8
29.0
53.9
$6.8
39.9
34.7
29.7
51.3
41.6
1246
SpinQuanter
45.4
30.7
59.2
60.7
41.4
40.8
32.6
51.4
45.3
47.7
Spi nQuant
59.4
39.4
64.4
68.9
43.4
54.6
41.4
55.9
$3.4
15.9
16-16-16
Full Precision
68.9
47.6
79.0
76.0
52.1
71.0
50.2
66.6
63.9
RTN
602
42.6
70.9
12.6
49.7
66.2
23.6
627
58.6
290
SumnothSuant
59.8
40.7
59.2
73.8
46.9
65.5
40.7
S8.5
$5.6
288.
4-8-16
LLM-QAT GPTQ
64.7
46.1
74.1
75.4
49.3
69.9
45.3
614
60.8
41.1
60.8
41.4
71.9
73.6
47.7
65.9
43.4
65.0
S8.7
25.2
Spinguant.o hod
Sp Anountttd
63.2
RTN
60.2
41.
71.5
75.1
.9.6
66.2
316
633
5&2
13
SmoothQuant
59.5
39.3
57.9
65.3
41.9
60.1
55.5
2813
4-8-8
MATT
76.
GPTQ
42.0
72.5
72.7
47.9
66.3
43.4
63.6
58.7
24.1
SpinQuant.o
65.2
45.7
76.
75.8
48.7
49.
47.9
65.5
618
11.7
3B
SpinQuantAsd
67.2
46.4
78.4
76.5
$1.0
69.5
$0.6
66.0
63.2
11.2
41.4
4.11
Smmooth:uumt
43.6
30.5
$2.8
$8.0
37.7
33.1
52.9
43.6
372.3
4-4-16
LLM-QAI
47.3
30.9
64.8
63.8
42.4
43.2
35.9
$1.1
46,9
37.6
GPTQ
42.0
30.0
44,8
60.1
41.2
44.7
34.0
50.5
43.4
264.
SpinQuant...
$4.6
37.7
65.7
66.7
43.3
56.3
41.8
56.9
$2.9
22.4
Couant
66.3
43.9
74.2
75.0
67.2
47.1
65.5
61.0
T1.1
RTN
R..
26.9
ACS
56.3
39.9
4000
31.T
523
"A2"
799T
Smooth-Qantt
36.4
26.2
50.4
55.8
39.0
30.3
30.2
52.
40.0
553.2
4.4.4
M.T.QQAT
44.4
29.7
61.5
62.0
42.3
41.2
33.8
514
45.9
42.0
38.2
SpinQuanten and
56.8
52.9
22.4
SpinQuantaef
66.0
43.2
76.4
74.6
47.0
67.7
45.1
64.2
60.5
11.4
16-16-16
Full Precision
77.6
$7.7
83.3
80.7
48.7
79.6
55.8
73.7
69.6
6.1
RTN
73.2
76.3
77.1"
75.3
34,3
63.3
8.2
SmoothQuant
70.8
45.8
61.0
LLM-QAI
77.6
50.6
81.2
79.0
47.5
76.0
53.5
72.4
67.2
7.7
GPTQ
71.5
46.8
76.1
76.6
47.9
73.9
$2.1
70.7
64.5
7.2
Sp SpinQuant.e.
77.8
55.4
80L6
79.9
48.9
77.5
55.5
73.3
68.6
6.7
SpinQuantass
76.5
54.0
81.5
79.6
48.6
78.1
56.4
72.4
68.4
6,5
RTN
73.7
76.5
77.1
75.5
30.8
734
65.
SunnothSuaatt
66.6
41.8
73.2
74.1
45.9
71.1
48.2
66.5
60.9
10.7
4-8-8
L.M-QAT
77.2
50.6
81.5
79.3
47.7
76.3
$2.0
70.6
66.9
7.6
GPTQ
71.5
46.9
76.6
76.2
48.5
73.7
$2.1
71.0
64.6
7.2
SpinQuante... had
77.2
56.1
Sp SpinQuantAes E
77.6
55.5
72.0
68.8
KTN
42.7
29.5
$4.0
$7.8
39.9
41.2
36.9
49.4
43,9
24166
SmoothQuant
36.3
26.3
50.6
31.4
30.6
52.9
40.3
867.5
4.4.66
LLM-QAI
GPTQ
27.6
40.8
58.5
41.7
31.9
32.0
53.1
40.6
187.9
Sp nQBanter and
56.5
35.3
$3.3
68.0
44.5
$9.9
37.5
59.7
$1.9
18.6
SpinQuantaud
75
50.9
78.9
77.5
47.2
75.9
$2.9
68.5
65.8
7.1
RTN
39.3
27.3
34.6
5T.T"
316
51.3"
"..1
2609""
Smmmith-uaml
49.6
53.1
28.8
29.6
49.6
38.7
1 1,530.50
4-44
LIM-QAT
40.5
26.6
52.7
$9.9
42.3
37.5
33.6
52.7
43.2
$2.5
GPTQ
40.6
26.5
40.9
58.0
41.5
31,9
33.0
51.8
40.5
195.8
SpinQuantes,
58.4
37.1
$4.7
67.7
43.4
60.1
41.2
51.9
52.6
18.6
SpinQuant.ad
75.1
51.2
77.2
77.3
75.2
54.1
65.5
to-end speed test results of LLaMA-3 70B detailed in Table 14 When implemented meticulously. SpinQuant with Hadamard rotation sees marginal difference in the latency compared to without Hadamard rotation.
16

---

Published as a conference paper at ICLR 2025
Table 9: Complete comparison of the perplexity score on WikiTex12 and averaged accuracy on Zero-shot
Common Sense Reasoning tasks on Mistral-7B-v0.3.
#Bits W-A-KV
Method
ARC-e
ARC-c
BoolQ
PIQA
SIQA
HellaS.
OBQA
WinoG.
Avg. (*)
WikiZ
(f)
(1)
(1)
(7)
(t)
(t)
(t)
(f)
(4)
16-16-16
Full Precision
81.0
57.9
84,2
82.1
48.2
80.8
59.6
73.8
71.0
5,4
RTN GPTQ
53.4
49.0
78.4
67.6
45.6
59.7
54.3
66.3
59.3
6.8
4-8-16
38.4
41.4
74.7
59.8
42.3
45.5
50.6
61.
51.7
8.6
SpinQuant.n
75.4
55.7
81.9
80.3
48.2
78.1
57.8
72.7
68.3
5.7
SpinQuant
78.9
55.9
82.7
81.9
48.5
80.0
58.4
72.7
69.9
5.5
RTN
52.9
48.7
78.5
67.3
45.5
59.4
52.7
66.4
58.9
6.7"
4-8-8
GPTQ
38.7
40.6
74,8
58.9
42.5
45.8
$1.0
61.3
51.7
8.6
spinquant... And
76.7
54.5
82.2
80.3
50.3
78.6
59.0
73.4
69.4
5.7
SpinQuant
80.1
56.9
83.9
81.5
48.6
79.9
57.2
73.0
70.2
5.5
RTN
39.9
24.7"
30.0
51.8
39.7
34.7
33.8
50.4
41.4
149.
4-4-6
GPTQ
27.1
57.3
35.6
31.4
40.4
260.8
spinQuant.w.
55.2
34.6
67.9
70.8
41.9
50.8
44.7
56.0
52.7
13.4
SpinQuant
76.5
53.3
80.7
80.7
48.2
78.6
57.8
71.2
68.4
5.7
RIN GPTQ
36.9
26.7
51.2"
58.1
40.3
34.4
28.7
51.7
41.4
443.5
4-4-4
40.4
28.5
43.6
57.4
39.2
35.2
33.8
52.1
41.3
249.9
spinquant... And
55.4
33.3
68.5
71.4
42.4
50.9
41.0
56.3
52.4
13.7
Sp SpinQuanthed
77.3
52.5
80.2
80.3
48.9
79.2
58.4
72.3
68.6
5.8
Table 10: 3-bit weight 8-bit activation quantization results WikiTex12 and Zero-shot Common Sense
Reasoning tasks.
#Bits W-A-KV
Method
ARC-e
(1)
ARC-e
(f)
BoolQ (*)
PIQA
(f)
SIQA (f)
HellaS.
(1)
OBQA
(f)
WinoG.
(*)
Avg. (f)
Wiki2 (4)
Fuli Precision
75.0
50.8
77.3
78.9
48.5
76.0
59.3
69.5
66.9
5,5
RIN
12.6
30.6
34.6
58.0
18.
20.3
39.
36.7
0551
LLaMA-2 7B
SmoothQuant
26.4
26.5
39.2
48.8
39.4
26.0
25.8
49.2
35.1
275,935.2
LLM-QÄT
44.0
29.5
64.4
63.3
42.2
52.7
32.6
52.3
47.6
15.2
GPTQ
63.8
40.2
67.3
73.1
43.3
63.5
46.9
65.5
57.9
14.6
Sp moiinummmtaa had
Full Preciston
75.3
51.4
79.8
80.4
50.5
56.8
72.5
68.3
RTN
30.3
24.6
488
33.8
39.8
20T0
25.44
39.5
376
167.8
LLaMA-2 13B
SmanthQuant
26.1
25.5
37.8
49.0
39.4
26.1
26.4
49.5
35.0
8,979.3
20.7
27.9
GPTQ
56.5
34.5
63.3
68.9
44.2
39,8
56.3
$1.2
10.8
Sp nouant
75.9
52.4
76.6
78.4
49.3
74.6
56.2
70.6
66.7
5.4
Full Piecision
802
66.5
85.
82.8
50'
84.3
59.0
80.6
72.9
3,3
RTN
30TD"
39.5
400
10.3
331.2
374
LLaMA-2 70B
SmoothQuant
26.9
22.7
38.4
49.0
38.6
25.6
25.2
$2.0
34,8
6.682.0
GPTQ
72.5
49.3
72.1
76.7
46.3
69.9
$1.8
72.2
63.9
9.0
SpinQuant.en And
79.4
58.7
84.4
81.6
50.5
82.3
58.3
78.6
71.7
3.8
Full Procision
65.2
38.7
69.5
75.3
44.8
60.7
40.2
60.9
56.9
13.4
RIN
32.6
2830"
34.8
55.7
34.2
29.7
47.8
40.2
2,097,66
3LaMA-3 1B
SmoothQuant
28.8
24.0
40.4
$1.6
37.8
25.9
28.2
48.0
35.6
58.367.5
LLM-QÄT
47.0
30.4
60.3
62.8
41.6
39.9
33,6
51.8
45.9
46.9
GPTQ
30.4
61.4
33.0
Spi Beinguant.er
58.8
36.4
63.7
68.7
44.2
$1.5
38.1
56.5
$2.2
17.2
Fall Precision
68.9
41.6
79..6
76.6
$2.1
710
50.2
66.6
63.9
10.77
RTN
40.1
29.5
48.8
39.3
41.6
46.0
34.4
534
44.1
1,178.9
LLaMA-3 3B
SmoothQuant
27.8
21.6
38.4
50.2
38.0
25.4
26.0
504
34.7
17,409.2
LLM-QAT
32.1
29.4
55.7
53,3
39.7
29.5
50.4
41.5
26.2
GPTQ
48.4
33.0
65.5
63.6
41.7
57.8
38.7
57.8
50.8
176.3
SpinQuant
61.8
41.4
78.2
73.0
47.
63.3
41.0
62.8
58.6
13.7
Pall Pisiiion
ST.
833
3O.7
48.
79.6
"53.8
75T
60.
6.
RTN
40.9
25.3
368
39.7
35.T
"31.4"
19622
LLaMA-3 8B
SmoothQuant
27.4
24.9
38.3
50.9
37.9
25.7
29.8
49.8
35.6
179.664.5
LLM-QAT
35.9
28.0
54.3
58.5
39.8
31.7
27.7
50.9
40.8
14,9
GPTQ
50.8
34.5
65.6
64.0
42.4
55.1
37.3
61.5
$1.4
944
Spa nouamt And
50.3
79.6
50.6
70.9
Full Procioion
81.0
57.9
84.2
82.1
48.
80.8
59.6
73.8
71.0
RTN GPTQ
28.2
261
62.2
53.
38.
28.0
335.9
28.3
30.3
1671
31.9
32.7
63.8
54.8
40.0
31.0
36.9
52.2
42.9
29.3
Spa moinnnuammmm. And
59.4
Table 11: Ablation study on Number of training samples and iterations in Cayley SGD optimization, using
LLaMA-2 7B.
#Bits
Task
Training sample
Training itcrations
(W-A-KV)
128
800
50
100
200
444
Wiki ())
6.2 20.03
6.2 +0.08
6.6 +0.02
6.4 #0.02
6.3 20.03
6.2 =0.03
6.2 +0.05
17

---

LLaMA 3.2 1B and Published as a conference paper at ICLR 2025
Table 12: Ablation symmetric and asymmetric quantization range clipping options on LLaMA-2 7B.
#Bits
RTN
GPTQ
(W-A-KV)
K asym
K clip
clip
Zero-shot Avg. (1)
Wiki (4)
Zero-shot Avg. (1)
Wiki (L)
4-4-16
61.2 20.6
63.3 10.4
4-4-66
61.8 20.4
6.1
64.0
5.9
4-4-16
/
/
62.1 #0.6
6.0
64.0 #0.4
5.9
4-4-4
x
x
61.4 #0.5
63.7 #0.4
4-4-4
x
61.5 20.8
6.2
63.7 103
5.9
4-44
61.5 #0.3
63.7 #0.2
Table 13:
Ablation study on calibration data choice using LLaMA-2 7B.
Calibration
#Bits
ARC-e ARC-e
BoolQ
PIQA
SIQA
HellaS,
OBQA
WinoG.
Avg.
Wiki2
Data
(W-A-KV)
(1)
(1)
(t)
(1)
(1)
(t)
(t)
(1)
(1)
(U)
Wiki2
4-4-16
72.1
47.5
74.4
77.
47.3
73.2
54.4
66.9
64.1
5.9
Wiki2
4-4-4
72.6
47.5
73.9
77.0
47.2
73.0
54.1
66.9
64.0
5.9
C4
4-4-16
-2.55
47.3
74.8
77.6
47.7
73.2
55.4
66.2
64.3
C4
4-4-4
72.5
47.9
74
78.4
46.7
73.1
55.5
66.4
64.3
A.7 OPTIMIZATION TIME
we show the comparison of optimization time between GPTQ and SpinQuant. SpinQuant requires a scale of optimization time similar to that of GPTQ. The additional optimization time required by SpinQuant is worthwhile considering the substantial improvements it offers over GPTQ.
In Table 15
A.8 ABLATION STUDY ON RTN GPTQ
SpinQuant is fully compatible with both GPTQ and naive RTN. To isolate the contributions of GPTQ and rotation overall performance, we present results for SpinQuant combined with RTN in the W4A4KV V quantization scenario in Table 16. Our analysis indicates that the primary accuracy gains are attributed to the incorporation of learned rotations, which enhances accuracy by 6.5 20.9 percentage points over previous methods (including GPTQ). The subsequent integration of GPTQ further boosts performance by up to 2.3 percentage points.
A.9 WEIGHT-ONLA QUANTIZATION
We also include a comparison of SpinQuant performance under weight-only quantization in Table 17 The weight-only results show that SpinQuant consistently achieves higher accuracy than AWQ and other previous work.
A. FEW-SHOT RESULTS ON INSTRUCTION-FINETTO MODELS
We further conduct experiments applying SpinQuant to
3B models in Table 18 We present the results for few-shot learning scenarios. SpinQuant W4A8 quantized models demonstrate significant improvements in 5-shot accuracy on the MMLU benchmark and 1-shot rouge score on the TLDR9 summarization benchmark. It significantly closed the gap to the BF16 baseline.
B ANALYSIS
B. GRADIENT ANALYSIS
On the one hand, have shown that the class of LLMs interested in rotation invariant, ie:. the full-precision model output does not change regardless of what R is. On the other hand, we are claiming that some R are better than others for quantized LLM and that better R can be learned with backpropagation on equation (2). To reconcile these seemingly conflicting claims, we inspect the gradient of the output of a single linear, W. and activations, X, which are both rotated and quantized:
Ei (Q(WR' (Q((X))) = Σ (WR am (R -Q((X))),, (
+ Q(WR R R Xmm
(5)
2Rmm
18

---

Published as a conference paper at ICLR 2025
Table 14:
Real-time end-to end-to-end speed measurement LLaMA-3 70B NVIDIA's Hopper (H100) GPU. TIFT (Time to First Token) and TTIT (Time to Iterative Token) are performance metrics to measure the pre-filling speed and decoding speed, respectively.
Spin Quant without hadamard SpinQuant with hadamand
TIFT (ms)
TTTT (ms)
TTFT (ms)
TTIT (ms)
BS=1 T=4096
153 153.58
9.85
158. 158.25
10.15
BS=8 4096
1205.41
10.6
1243.48
10.94
3S=32 T=4096
5008.25
13.83
5147.59
14.2
Table 15: Optimization time comparison between GPTQ and SpinQuant.
llama3 IB
Ilamaš 3B
Ilama3 8B
Ilama2 7B
Ilama2 13B Mistral
SpinQuant
13 min
18 min
25 min
30 min
16 min
GPTQ
min
13 min
min
min 25 min
12 min
We see that equation
is non-zero general, which validates our approach of using backpropagation to learn
reduces to 0 when quantization is not present, which validates the claim that it only makes sense to learn Rt for quantized models
demonstrates that two components move the gradient with respect to R away from O: 1) differences in quantized and unquantized rotated weights; differences in quantized and unquantized rotated activations
B.2
LOSS ANALYSIS
End und-to-end SNIt
SNR
SNKRR 1) SNIKR
0.6
16.2
0.4
2
16.0
0.0
R=R
10.2
R-R_T
15.6
-0.4
150
200
Iteration
Reration
Activation Index
(a)
(b)
Figure 7: Training curves for LLaMA-2 7B with 4-bit weights and 4-bit activations in wiki2 train set. (a) End- to-end quantization SNR. Ro and Rr denote randomly initialized rotation and learned rotation after T 200 itcrations; (b) Activation quantization. SNR for layer attention out projection; (c) Improvement activation quantization SNR after optimization of R for each layer.
While Sec. 4 shows that learning R yields significant benefits zero-shot reasoning tasks, in this section we shed some light on why our method is able to achieve accuracy gains. Intuitively, we expect the end-to-end signal (quantization) noise ratio (SNR) improve result of learning R. In other words, learning R should bring the quantized model output closer to the floating point model output. As Table 19 shows,
observe an SNR improvement of 3.8 dB when introducing
a random R into LLaMA A-2 7B with weights/activations quantized to bits, and then an additional 5.9dB improvement after learning R. all measured on the WikiText2 (Merity al.) 2016) test set. Figure shows that the batch-level training set SNR during R training progressively improves as expected, as well as the layer-level SNR for a particular layer in Figure 7b| Digging bit deeper, 7c shows the layer-level SNR improvement for each layer as result of training R. We see that, perhaps counter-intuitively. layer-level SNR improves significantly for a few layers, but does not change much for most layers, and even gets worse for one of the layers. We hypothesize that: 1) certain layers have a disproportionate impact on model output or have a disproportionately low quantization SNR without rotation; 2) The process of optimizing R rotates the residual stream basis such as to prioritize improving the SNR such layers, possibly at the cost of hurting less important layers.
19

---

Published as a conference paper at ICLR 2025
Table 16: Ablation study on SpinQuant combined with RTN GPTQ the W4A4KV16 quantization scenario.
3LaMA-3
8B
LLaMA-2
LLaMA-2
13B
LLaMA-2
708
Method
Zero-shot
Wiki
Zero-shot
Wiki
Zero-shot
Zero-shot
Wiki
Full-precision
69.6
6.1
66.9
5.5
68.3
72.9
3.3
RTN
43.9
2c2
35.6
263
35.3
7e3
35.1
2e5
SmoothQuant
40.3
9c2
41.8
3c2
44.9
34.5
57.7
57.1
L.M-QAT
GPTQ
40.6
2c2
36.8
9c3
35.5
2c6
9p1 Aud (RTN)
64.6
7.7
61.8
6.1
65.8
5.4
71.1
3.9
SpinQuantAed (GPTQ)
65.8
7,1
64.1
5.9
67.2
5.2
71.0
3.8
Table 17: A comparison of SpinQuant performance under 4-bit weight-only quantization.
LLaMA-2 7B
LLaMA-2
LLaMA-2 70B
Method
Zero-shot Avg. (?)
Wiki pol(()
Zero-shot Avg.t)
Wiki ppl({)
Zero-shot Avg.ts
Wiki ppl()).
Pull-precision
66.9
5,5
68.3
.50
72.9
3.
RTN
63.6
7.2
57.9
6.4
69.2
4.6
SmoathQuant
59.1
7.5
63.3
70.2
4.1
GPTQ
64,5
11.3
64.7
71.9
AWQ
Spi Rouant
Table 18: Results of applying SpinQuant instruction-finetunee LLaMA 3.2 IB and 3B models.
laama3,2 IB
llama3.2 3B
BF16
Vamilla RTN
SpinQuant
BF16
Vanilla RTN
SpinQuant
MMLU (S-shot)
49.3
43.4
47.3
63.4
60.5
62.0
TLDR9 (test, I-shot rougel.)
16.8
14.9
16.7
19.0
19.1
19.2
Table 19: Average end-to-end signal to quantization noise ratio (dB) for LLaMA-2 7B with weights and activations quantized to 4 bits on wiki2 test set
Randomly initialized Learned
-299
C DISTRIBUTION VISUALIZATIONS BEFORE AND AFTER ROTATION
We present visualizations the activation distributions before and after rotation in Figures and 9 respectively. Similarly, the weight distributions before and after rotation are depicted in Figures 10
and I1 Overall, after rotation, the extreme values are attenuated, and the distribution exhibits no noteworthy outliers across the token dimension. Additionally, we make an interesting observation: in several activation layers, the first token displays substantial values in multiple channels. After rotation, this outlier is distributed across all channels of the first token. Although per-token activation quantization can readily manage this distribution, investigating the source of these outliers and reducing them prior applying SpinQuant might further enhance quantization accuracy, which could be a potential future research direction.
20

---

Published as a conference paper at ICLR 2025
Myees.0 seft potne_p
20cee Lamaapoi
lpprrst woo wtm.tta pro
lyyera 10 seft strviLsn
Rgges LO K.DRR
lyyss 50.5 AL
beyera aud jatoo som
koyass Stlas auloy, prg
pio)
byyers
Sy down.rn proj
byene 10 mip down_rne proj
loyors 20 down_rtd pro]
byerr S0 mii cown _ordl
www.Corlo-uu....o 2015
layuts www.oooooooo......
kypara SEL SMAL
Figure 8: Magnitude of the input activations of linear layer in (1st. th 21st and 31 blocks in LLaMA-2 7B model before rotation.
21

---

Published as a conference paper at ICLR 2025
laper self poo
20cee Lim.a.od
Myeer 30 seft stnviLsn
lyyes 50.5 AOBLLL
Beyera and jatrorys yom
koyass yalno, pod
layens
Jayers...mm down_pro
byyen 10 mip down, proj
loyors unlp down_srd
byeers S0 www.coom.mmmm.mmmmmmmmmmmmm _ordl
legurs D0 www.orrggge.......
kyyasa StL UAAA pndi
Figure 9: Magnitude of the input activations of linear layer in (1st. th 21st and 31st) blocks in LLaMA-2 7B model after rotation.
22

---

Published as a conference paper at ICLR 2025
lyper self ppo_
Sperr.20.oo Limagppd
Myeera 10 sett sErvi.sn
Rager LE K.DRR)
leyera aud aatror pom
koyass atla yalnoy, prg
pio)
O down
byyen 10 mip down.ree proj
loyors 20 downdssnnn proe
bayers SO riip cown
layuts www.ooowoooo......
kapara Stl gait prij
Figure 10: Magnitude of the weights of linear layer in (1st, 11th 21 and 31st] blocks in LLaMA-2 7B before rotation.
23

---

Published as a conference paper at ICLR 2025
lypes set poo
20cee Lamaapoi
lpprrst www. aam
Myeer 30 oetf stnviLsn
layors 50 LO
Beyera and wwwooootoo pom
koyars walroy p013
pioj
ayers.O.r down_rro
byyen 10 mip down proj
loyors 20tr down_oton
bayers S0 miip cowm pood
legurs D0u www.oorgge........
kapara SEL gait pra
Figure 11: Magnitude of the weights of linear layer in (1st, 11th 21 and 31st] blocks in LLaMA-2 7B after rotation.
24


Reviews:
[{"review_id": "GLlLxhJ3BH", "reviewer": "Reviewer_6v9t", "summary": "The paper presents a learned rotation based method, namely $\\texttt{SpinQuant}$ to mitigate outliers in weight and activation distributions, boosting the performance of quantized LLMs. In specific, the authors  proposes optimizing rotation matrices within Stiefel manifold, directly minimizing the final loss of rotated quantized network. The authors further propose two variants, $\\texttt{SpinQuant}$_easy and $\\texttt{SpinQuant}$_hard without and with change of", "questions": "1. It would be interesting to see the performance of such dynamic quantization ($\\texttt{SpinQuant}$) with static quantization alternative [1]. \n\n2. Is the R3, R4 needed in case someone uses block reconstruction of [2]? \n\n3. Also, how does $\\texttt{SpinQuant}$ compare w.r.t these alternative methods?\n\n4. Why the choice of ``optimizing these rotation matrices on Stiefel manifold\", is not fully clear.\n\n[1] PREFIXQUANT: STATIC QUANTIZATION BEATS DYNAMIC THROUGH PREFIXED OUTLIERS IN LLMS, arxiv 2024\n\n[2] TESSERAQ: ULTRA LOW-BIT LLM POST-TRAINING QUANTIZATION WITH BLOCK RECONSTRUCTION, arxiv 2024.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The idea of learned rotation in  the context of LLM activation outlier reduction seems interesting.\n\nThe authors have done thorough analysis to justify the need of learned rotation \n\n$\\texttt{SpinQuant}$ can outperform baseline GPTQ by significant margin!", "weaknesses": "1. The performance of $\\texttt{SpinQuant}$ is similar or worse than QuIP in most of the cases. This further raises concerns on the learned rotation method as QuIP does not rely on any activation rotation method.\n\n2. The calibration discussion (thorough) is missing. As the method needs to learn R1,R2, R3 and R4, such calibration and/or fine-tuning overhead is a concern. \n\n3. How the learning gets affected for activation rotation, when there is extremely low calibration data?\n\n4. The can be other alternatives to learn the R1-4, why did the author chose Cayley SGD method (Li et al., 2020).\n\n5. Why table 6 shows results with $\\texttt{SpinQuant}$_hard even for W4A8? \n\n6. What speedup benefit we may have with $\\texttt{SpinQuant}$_hard and $\\texttt{SpinQuant}$_easy with W4A4?\n\n7. The contribution is primarily centered around rotation to get rid of activation outliers and smoothen out while for the quantization part the authors heavily rely on GPTQ (or RTN). The baseline GPTQ may often be considered a calibration heavy that require Hessian information. I have concerns on the calibration overhead of the method as well as the novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QQ8x2y9uko", "reviewer": "Reviewer_Jf43", "summary": "This work investigates post-training quantization (PTQ) for LLMs, applied to weights, activations, and KV cache. It addresses the large quantization errors issue from outliers by introducing the learned rotation matrices into the network architecture. Through experiment validations, the proposed framework shows effectiveness in extreme W4A4KV4 quantization settings on representative networks.", "questions": "Although the rotation method is effective, the readers may be interested in the theoretical foundations of this methodology.\n\nPlease provide analysis on the overhead introduced by the rotation matrices into the inference pipeline vs the benefits.\n\nThe reviewer would like to know the training complexity of the proposed method comparing with other quantization approaches.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "+ The paper pursues the rotation method for post training LLM quantization, which demonstrates convincing performance.\n\n+ The authors conducted thorough investigations about random rotations to propose quantization-oriented rotation learning.\n\n+ The proposed framework develops strategies for the challenging activation and KV cache quantization, besides weight quantization.", "weaknesses": "- While the rotation matrix method is effective in mitigating outliers in LLM quantization, it is better if more rigorous explanations/proofs are provided for the rationale behind, besides empirical results.\n\n- Introducing rotation matrices into the inference pipeline may lead to overhead in computing. Could the authors provide analysis the overhead vs benefit from quantization?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "udEeHjceYd", "reviewer": "Reviewer_U4Hi", "summary": "The paper presents SpinQuant, a quantization technique specifically designed to reduce the impact of outliers in large language models (LLMs) through learned rotations. SpinQuant introduces learned rotation matrices that can be integrated into pre-trained weights, which aids in preserving accuracy in low-bit quantization scenarios. The proposed method includes two modes: SpinQuant-easy and SpinQuant-hard—the former for more straightforward quantization with mergeable weights, and the latter for complex scenarios needing enhanced outlier suppression with non-mergeable weights. The authors validate SpinQuant across multiple models (e.g., LLaMA-2, LLaMA-3, and Mistral) using a comprehensive set of zero-shot reasoning benchmarks.", "questions": "- Have the authors considered testing alternative calibration datasets beyond WikiText2? It would be interesting to see if different calibration sets impact SpinQuant’s generalization, especially in domain-specific applications (e.g., code). While the paper describes the use of WikiText2 for calibration, further analysis on its impact and comparison with alternative calibration data could strengthen the understanding of its generalization.\n- it would be interesting to know if SpinQuant has been tested on models from other domains, such as Vision Transformers (ViTs) or vision-language models, to further assess its versatility across different types of architectures.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- By employing learned rotations, SpinQuant offers a new approach to minimizing outliers.\n- The SpinQuant-easy and SpinQuant-hard modes offer flexibility, making the approach adaptable to different computational constraints and accuracy requirements through mergeable and non-mergeable weights.\n- SpinQuant shows compatibility with methods like GPTQ, allowing it to integrate into established quantization pipelines.", "weaknesses": "- The benchmarks and model selections, such as the LLaMA series, seem to focus on favorable cases for SpinQuant. Testing on a broader range of architectures (e.g., Gemma2) would provide a more thorough evaluation. Especially Gemma2 series which exhibit distinct activation characteristics—could provide a more comprehensive assessment of SpinQuant's generalizability and robustness across diverse LLM types.\n- Although the authors note the optimization time (up to 3.5 hours for larger models), it remains unclear how this time impacts real-world deployment, especially on low-resource devices.\n- From an outlier-aware quantization perspective, as demonstrated by methods like AWQ, SpinQuant should ideally exhibit strong performance even in weight-only quantization scenarios. However, even after examining the appendix, there is no evaluation comparing its performance under weight-only quantization. It would be beneficial to include a comparison of SpinQuant’s performance in weight-only quantization settings to provide a more comprehensive view of its robustness and effectiveness relative to other quantization techniques.\n- While the paper presents promising results on zero-shot reasoning tasks, it does not evaluate SpinQuant's performance in fine-tuning or few-shot learning scenarios. Including such evaluations would strengthen the claims about SpinQuant’s overall effectiveness and versatility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wCAzUpsBu9", "reviewer": "Reviewer_tcZR", "summary": "This paper introduces SpinQuant, a PTQ method that applies optimized rotations to reduce quantization errors from outliers, improving LLM efficiency without compromising accuracy. It achieves near full-precision performance, narrowing the accuracy gap to 2.9 points on zero-shot tasks in LLaMA-2 7B, outperforming previous works.", "questions": "- Could you provide a comparison of the computation needed for the quantization? Going beyond comparing accuracies on benchmarks alone would be helpful.\n- the number of rotation matrices is determined empirically. Can the authors investigate this further and explain in more detail how this can be validated better?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- paper is well-written\n- specifically, the illustrative example in 4.4 is very helpful. Yet, it would be sensible to move this one more to the front of the paper\n- experiments are extensive across models, settings and methods", "weaknesses": "- The current rotation is not optimal, as the authors state. It would be a very valuable contribution to investigate optimal rotations further.\n- a more detailed discussion on model size vs rotation-based quantization is needed. Larger models are impacted less by quantization compared to smaller models, why?\n- currently, no effort to publish code or quantized models given, limiting reproducibility and usability for the community.\n- the tradeoff discussion between easy and hard is too shallow. Please elaborate on which cases are useful to use one or the other.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3GCMZPaIsa", "reviewer": "Reviewer_brBn", "summary": "This paper introduces SpinQuant, a novel approach to mitigate outliers during Large Language Model (LLM) quantization by leveraging learned rotation matrices. The paper presents two rotation strategies: 1) an \"easy mode\" for W4A8KV8 quantization where rotation matrices can be absorbed without impacting inference, and 2) a \"hard mode\" incorporating two additional online rotation matrices for more aggressive quantization scenarios (e.g., W4A4KV4). Through evaluation across various LLMs and reasoning tasks, SpinQuant demonstrates that its learned rotation matrices outperform random rotation approaches.", "questions": "- Regarding Formula (1), is the clip operation missing?\n- For Figure 1, there are two R3 after the `RoPE`, should one of them be the inverse of R3?\n- Could you clarify which rotation type is being demonstrated in Figures 9 and 10?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Proposed a learned version of rotation matrices to address variance issues inherent in random rotation approaches, with demonstrated flexibility across different quantization scenarios.\n- The paper is well-structured, presenting clear methodology and comprehensive diagrams that effectively illustrate the proposed approach.", "weaknesses": "- Limited evaluation scope on complex tasks benchmarks: The evaluation would be more compelling with results from more challenging benchmark tasks, such as [MMLU](https://arxiv.org/abs/2009.03300) and [MMLU-Pro](https://arxiv.org/abs/2406.01574).\n- Incomplete theoretical analysis, while Section 4.4 demonstrates Hadamard rotation's superiority over random rotation, it lacks a thorough comparative analysis between learned and random rotation matrices. The significant 10+ point improvement observed on Mistral-7B using hard mode could serve as a good case study to demonstrate the benefits of learned rotation matrices.\n- Limited performance benchmarking: The end-to-end speed evaluation is confined to MacBook M1 hardware, with only basic FP8 quantization latency testing on GPU rather than comprehensive benchmarking across the paper's proposed quantization scenarios. Given recent research highlighting potential performance impacts of online rotation transformations (e.g., [QServe]( https://arxiv.org/abs/2405.04532), Part V Section C), more extensive benchmarking on modern GPU architectures would strengthen the paper's practical implications. Key speedup metrics could include time to first token (TTFT) and tokens per second (TPS).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)