SpinQuant: LLM Quantization with Learned Rotations

## Abstract

Abstract Post-training quantization (PTQ) techniques applied to weights, activations, and the KV cache greatly reduce memory usage, latency, and power consumption of Large Language Models (LLMs), but may lead to large quantization errors when outliers are present. Recent findings suggest that rotating activation or weight matrices helps remove outliers and benefits quantization. In this work, we identify a collection of applicable rotation parameterizations that lead to identical outputs in full-precision Transformer architectures, and find that some random rotations lead to much better quantization than others, with an up to 13 points difference in downstream zero-shot reasoning performance. As a result, we propose SpinQuant that optimizes (or learns ) the rotation matrices with Cayley optimization on a small validation set. With 4-bit quantization of weight, activation, and KV-cache, SpinQuant narrows the accuracy gap on zero-shot reasoning tasks with full precision to merely 2.9 points on the LLaMA-2 7B model, surpassing LLM-QAT by 19.1 points and SmoothQuant by 25.0 points. SpinQuant also outperforms concurrent work QuaRot, which applies random rotations to remove outliers. In particular, for LLaMA-2 7B/LLaMA-3 8B models that are hard to quantize, SpinQuant reduces the gap to full precision by 30.2%/34.1% relative to QuaRot.

## 1 Introduction

Large Language models (LLMs) have demonstrated impressive performance across many disciplines. SoTA open source models (e.g., LLaMA [], Mistral [], etc) and proprietary LLMs (e.g., GPT [], Gemini[], etc) have been used in general purpose chatting assistants, medical diagnosticians [], computer game content generators [], coding co-pilots [], and much more.

To serve such a high demand, the inference cost becomes a real issue. Many effective techniques have been developed. Post-training Quantization (PTQ), as one effective category of techniques, quantizes the weights (or activations) into low-precision and thus reduces the memory usage and may significantly improve latency. This is not only important for server-side inference, but also for on-device scenarios with small-sized LLMs [].

When applying quantization, outliers remain an open challenge because they stretch the quantization range, leaving fewer effective bits available for the majority of values. Prior research mitigates this challenge by trading quantization difficulty between weights and activations [, ] or employing mixed-precision to handle outliers []. Recent studies take a different angle by demonstrating an interesting property: Multiplying the weight matrix with a random rotation can effectively reduce outliers and enhance quantizability [, ]. Intuitively, due to the statistical property of random rotation, such a transform yields a outlier-less distribution of resulting weight or activation entries []. Since the rotation matrices can be constructed in pairs from identity mapping, and get integrated into nearby weights without changing the overall network outputs, a property known as *rotational invariance*[], the transformed weights (or activations) can be quantized with lower reconstruction error, without additional inference overhead.

While statistically any random rotation works well, in this paper, we find that the performance of quantized network *varies a lot* with different rotation matrices. For example, the downstream averaged accuracy on zero-shot reasoning tasks may change up to 13 points with different rotations. As a result, we propose SpinQuant that *optimizes* the rotation matrix to minimize the final loss of the quantized network, with fixed weight parameters, by employing the Cayley SGD[], a proficient technique for optimizing orthonormal matrices. This optimization does not alter the full-precision network output but refines the intermediate activations and weights, making them more quantization-friendly.

In SpinQuant, we also extend the concept of outlier reduction for single matrices [, ] to a comprehensive network-level perspective. We identify four different places that are rotationally invariant in most prevalent LLMs (e.g., OPT [], LLaMA []), as depicted in Figure [1](#S1.F1). This constitutes the optimization space of SpinQuant.

Experimental results demonstrate that SpinQuant significantly improves accuracy compared to random rotation matrices. With 4-bit quantization of weights, activations, and KV cache, SpinQuant achieves an average accuracy of 64.0 on zero-shot commonsense reasoning tasks for LLaMA-2 7B. This marks a gap of just 2.9 points from the full-precision network, considerably better than previous LLM-QAT [] with a gap as large as 22.0 points despite using the same precision. We also demonstrate results on LLaMA-3, showing 17.7 points accuracy improvement than SmoothQuant [] on 70B model with 4-bit weights, activations, and KV cache quantization, narrowing the gap to full-precision to only 4.4 points. In terms of speed, it only takes 100 iterations (and 1.3 hours) to optimize the rotation matrices on 800 WikiText2[] calibration data for a LLaMA-2 7B model on a single A100 node.

Figure: Figure 1: Overall diagram of rotation. (a) The residual stream can be rotated in the transformer network, resulting in numerically equivalent floating point networks before and after rotation. The rotated activations exhibit fewer outliers and are easier to quantize. (b) & (c) The rotation matrix can be integrated with the corresponding weight matrices and we further define $R_{2}$, $R_{3}$, and $R_{4}$ for reducing outliers inside the block.
Refer to caption: /html/2405.16406/assets/x1.png

## 2 Motivation and Preliminaries

Figure: Figure 2: Activation distribution in LLaMA-2 7B model before and after rotation. Outliers exist in particular channels before rotation. Since channel-wise quantization is not supported in most hardware, outlier removal using rotation enables accurate token-wise or tensor-wise quantization.
Refer to caption: /html/2405.16406/assets/x2.png

Figure: Figure 3: Outlier measurement and quantization error across input activation and weights in the five layers that take inputs from the residual (Q/K/V/Up/Gate-projection) of each block in the LLaMA-2 7B model. (a) After rotation, kurtosis of activation distributions is significantly reduced to approximately three across all layers. Quantization error is reduced after rotation in both (b) activations and (c) weights.
Refer to caption: /html/2405.16406/assets/x3.png

Quantization reduces the precision of weights (and/or activations) in a neural network in order to save memory and lower the latency. For Large language models (LLMs), the presence of outliers extends the range of weight/activation values and increases the reconstruction errors for normal values [, , ] (Figures [2](#S2.F2) (a)&(c)).

### 2.1 Outlier Reduction via Random Rotation

There exist many ways to mitigate the effect of outliers [, ]. In this paper, we focus on the technique that uses random rotation to reduce outliers, which outperforms previous approaches. Intuitively, a random rotation matrix statistically blends large and small weights together into a well-behaved distribution with fewer outliers, and thus is easier to quantize. Theoretically, QuIP[, ] proves that multiplication of random orthonormal matrix leads to high incoherence (i.e., lower maximal entry of a matrix compared to its norm) with high probability and thus fewer outliers.

Figure [3](#S2.F3) (a) illustrates the measurement of the Kurtosis $\kappa$ of the activations before and after rotation. $\kappa$ quantifies the “tailedness” of a real-valued random variable’s probability distribution. A larger $\kappa$ indicates more outliers, while $\kappa\approx 3$ suggests a Gaussian-like distribution.
In Figure [3](#S2.F3) (a), the activation distribution in the transformer contains numerous outliers, with $\kappa$ of many layers exceeding 200. However, after multiplying these activations with a random rotation matrix, the $\kappa$ across all layers becomes approximately 3, indicating a more Gaussian-shaped distribution that is easier to quantize. This is corroborated by Figure [3](#S2.F3) (b), where the quantization error of the activation tensor significantly decreases after rotation.

Figure: Figure 4: The performance distributions under different random rotations on LLaMA-2 7B, using network-level parameterization (Sec. [3.1](#S3.SS1)). We compare the distributions using random floating-point rotations, random Hadamard matrices, and optimized rotation matrices with Cayley optimization (Sec. [3.2](#S3.SS2)). Despite that Hadamard matrices mostly perform better than random rotations, both random groups demonstrate large variance. In contrast, by optimizing the rotation matrix with Cayley optimization (i.e., SpinQuant), the performance is improved significantly and the variance becomes much smaller.
Refer to caption: /html/2405.16406/assets/x4.png

### 2.2 Random rotations produce large variance

Interestingly, while statistically random rotation leads to better quantization, not all random rotations give the same quantization outcome. To show this, we tested the zero-shot average accuracy of the rotated version of LLaMA-2 7B, quantized to 4-bit weight and 4-bit activation, under 100 randomized trials. As shown in Figure [4](#S2.F4), the performance variance is substantial, with the best random matrix outperforming the worst by 13 points. Random Hadamard matrices (^1^11A Hadamard matrix $H$ is a special type of rotation matrix, where the entries of the matrix are solely $\pm\sqrt{n}$. Given a Hadamard matrix $H$, we can generate $2^{n}$ different random Hadamard matrices by multiplying with $S$, a diagonal matrix with elements $s_{i}$ randomly chosen from $\{-1,1\}$.) outperform random matrices, in consistent with the findings in [] that Hadamard matrices yield tighter bounds on weight quantization error. However, even random Hadamard rotation matrices exhibit a non-negligible variance in final performance, as large as 6 points.

Given the huge variance across multiple trials of rotations, a natural question arises:

Is it possible to *optimize* the rotation to maximize the benefit of quantization?

As the main contribution of this work, we show that not only it is possible to do so, but also such an optimization procedure leads to a much better quantized network. For LLaMA-2 7B, it reduces the gap between the 4-bit quantization of weights, activations, and KV caches and its 16-bit precision version to merely 2.9 points. It also works well for LLaMA-3 70B, whose performance deteriorates under existing quantization techniques [], shrinking the 4-bit quantized network accuracy gap to full-precision from previous SoTA 9.4 points to 4.4 points.

## 3 Method

In this section, we introduce our rotation parameterization of popular LLM architectures, which covers a broad search space to optimize. Such a parameterization leads to identity network output without quantization. We then introduce Cayley optimization to optimize these rotations for better downstream performance under quantization.

### 3.1 Rotation parameterization

Rotating activations in residual
As shown in Figure [1](#S1.F1)(a), we rotate the activations in the residual path by multiplying the embedding output $X$ with a random rotation matrix ($R_{1}$). This rotation removes outliers and eases the quantization of the input activations to the fully-connected layers that read from the residual. To maintain numerical invariance, we reverse the rotation of the activation by multiplying it with $R_{1}^{T}$ ($=R_{1}^{-1}$) prior to its passage through the attention block and feed-forward network, which contains non-linearity. When the quantization is not present, the full-precision network remains intact no matter which rotation is applied.(^2^22In a pre-norm LLM like LLaMA [], we can convert a transformer network into a rotation-invariant network by incorporating the RMSNorm scale parameters $\alpha$ into the weight matrix right after the RMSNorm layer [].)
The rotation matrices can be absorbed into corresponding weight matrices, as illustrated in Figures [1](#S1.F1) (b) and (c). After absorption, no new parameters are introduced in the network. We can now modify $R_{1}$ freely without impacting the floating-point network’s accuracy or parameter count.

Rotating activations in the attention block
As depicted in Figure [1](#S1.F1) (b), in the attention block, we propose to rotate the value matrix by multiplying $R_{2}$, and the activations to out-projection layer by $R_{2}^{T}$ head-wisely. $R_{2}$ has the shape of ($D_{head},D_{head}$) and can be independently chosen across layers. The numerical in-variance is illustrated in Figure [5](#S3.F5), these two rotations can be offset in a full-precision network since there are no operators between $R_{2}$ and $R_{2}^{T}$. Meanwhile, it can improve quantization for value cache and input to linear layer ($W_{o}$) without introducing any new parameters in the network.

Figure: Figure 5: Rotation equivalence in Multi-Head Self-Attention.
Refer to caption: /html/2405.16406/assets/x5.png

Additional unabsorbed rotations
To further address the presence of outliers in KV-cache, we adopt a recent approach [], which adds a Hadamard matrix ($R_{3}$ in Figure [1](#S1.F1)(b)) on the query and key outputs. Due to the existence of the RoPE operation [], this rotation cannot be absorbed into the $W_{q}$ and $W_{k}$ matrices, as shown in Figure [6](#S3.F6) (a). As a result, the Hadamard rotation multiplication is used to rotate the query and key during inference, since it is fast to compute.
Similarly, we added a Hadamard matrix multiplication ($R_{4}$ in Figure [1](#S1.F1)(c)) inside the feed-forward layer, reducing the outliers in the input to the down projection layer [].

Figure: Figure 6: Application of rotation pair to (a) query and key (b) down project layer input.
Refer to caption: /html/2405.16406/assets/x6.png

After inserting rotation into LLM, the difficulty of network quantization is significantly alleviated due to outlier elimination. As shown in Figure [3](#S2.F3) (b), activation quantization error drops sharply from as high as 0.7 to around 0.15. In addition, weight quantization also becomes easier by absorbing the rotation matrix into the weight matrix, as shown in Figure [3](#S2.F3) (c), as depicted in Figure [3](#S2.F3) (c), remarking rotation as a beneficial technique that simultaneously helps both weight and activation quantization with a single implementation.

### 3.2 Cayley -optimized rotation

As illustrated in Figure [1](#S1.F1), we have determined that the incorporation of four rotation matrices ($R_{1}$, $R_{2}$, $R_{3}$, $R_{4}$) can improve quantization performance while preserving numerical consistency in a full-precision network. Given that $R_{3}$ and $R_{4}$ are online rotation operations, meaning they cannot be absorbed into the weight matrix, we retain them as Hadamard matrices. This is because online Hadamard transforms can be efficiently implemented without significant overhead. We then define the optimization objective as identifying the optimal rotation matrix $R_{1}$ and $R_{2}$ that minimizes the final loss of the quantized network:

$$ $\displaystyle\operatorname*{arg\,min}_{R\in\mathcal{M}}\mathcal{L}_{Q}(R_{1},R_{2}\mid W,X)$ (1) $$

Here, $\mathcal{M}$ represents the Stiefel manifold i.e., the set of all orthonormal matrices. $\mathcal{L}_{Q}(\cdot)$ denotes the task loss, such as cross-entropy, on the calibration set. It is a function of {$R_{1}$, $R_{2}$}, given the fixed pretrained weights $W$ and the input tensor $X$ and with the quantization function $Q$ in the network. To optimize the rotation matrix on the Stiefel manifold, we employ the Cayley SGD method [], which is an efficient optimization algorithm on the Stiefel manifold. More specifically, in each iteration, the update of the rotation $R$ is parameterized as the following:

$$ $R^{\prime}=\Delta R(Y)R:=\left(I-\frac{\alpha}{2}Y\right)^{-1}\left(I+\frac{\alpha}{2}Y\right)R$ (2) $$

where $\Delta R(Y):=(I-\frac{\alpha}{2}Y)^{-1}(I+\frac{\alpha}{2}Y)$ is the *Cayley Transform* of a skew-symmetric matrix $Y$ (i.e., $Y^{\top}=-Y$). $Y$ is computed from a projection $\hat{G}$ of the gradient $G:=\nabla_{R}\mathcal{L}_{Q}$ of the loss function:

$$ $Y=\hat{G}-\hat{G}^{\top},\quad\quad\hat{G}:=GR^{\top}-\frac{1}{2}RR^{\top}GR^{\top}$ (3) $$

It can be shown that $\Delta R(Y)$ is always orthonormal and thus $R^{\prime}$ is guaranteed to be orthonormal ($R^{\prime\top}R^{\prime}=I$) if $R$ is orthonormal. While Eqn. [2](#S3.E2) requires a matrix inverse, the new rotation matrix $R^{\prime}$ can be computed via an efficient fixed point iteration []. Overall, the approach maintains the property of orthonormality with only $\sim$2 times the computation time per iteration compared to a naive SGD algorithm.

We apply the Cayley SGD method to solve Eqn. ([1](#S3.E1)) for $\{R_{1},R_{2}\}$, while the underlying weight parameters in the network remain frozen. $\{R_{1},R_{2}\}$ count for only 0.26% of the weight size and is constrained to be orthonormal. Consequently, the underlying floating-point network remains unchanged, and the rotation only influences the quantization performance.

By employing Cayley optimization to update the rotation for 100 iterations on an 800-sample WikiText2 calibration dataset, we obtain a rotation matrix that outperforms the best random matrix and random Hadamard matrix in 100 random seeds, shown in Figure [4](#S2.F4). The Cayley-optimized rotation exhibits minimal variance when initiated from different random seeds. The rotation matrices are initialized with random Hadamard matrices for optimization in our experiments and our ablation study in Section [4.3.3](#S4.SS3.SSS3) demonstrates that the optimized rotation is robust to random rotation initialization as well.

## 4 Experiments

We conduct experiments on the LLaMA-2[] models (7B/13B/70B) and the LLaMA-3[] models (8B/70B). Our evaluation of the proposed SpinQuant was carried out on eight zero-shot commonsense reasoning tasks. These tasks include BoolQ[], PIQA[], SIQA[], HellaSwag[], WinoGrande[], ARC-easy and ARC-challenge[], and OBQA []. Additionally, we also report the perplexity score on WikiText2 testset [] for our evaluation.

### 4.1 Experimental settings

We employ Cayley SGD [] to optimize the rotation matrix, $R_{1}$ and $R_{2}$, both initialized as a random Hadamard matrix, while maintaining all network weights constant. $R_{1}$ is the residual rotation, shaped as ($D_{token},D_{token}$). $R_{2}$ is head-wise rotation in each attention block, shaped as ($D_{head},D_{head}$) and is separately learned in each layer. The learning rate starts at 1.5 and linearly decays to 0. We utilize 800 samples from WikiText-2 to optimize rotation for 100 iterations. It takes $\sim$1.39 hours for LLaMA-3 8B, $\sim$1.25 hours for the LLaMA-2 7B, $\sim$2.36 hours for LLaMA-2 13B, and $\sim$12 hours for 70B models on 8 A100 GPUs.

The first results we show employ simple round-to-nearest (RTN) quantization for the weights. The activation and key-value cache use asymmetric min-max dynamic quantization, with per-token activation quantization and group size 128 for the key-value quantization. Weight quantization employs per-channel symmetric quantization, and we set the quantization ranges via a linear search of the mean-squared error loss between quantized full-precision weights, following common practice [, , ]. Besides rounding-to-nearest, we also show that our method is compatible with GPTQ [], for which we adhere to the standard GPTQ settings by using 128 samples from WikiText-2 with a sequence length of 2048 as the calibration set for GPTQ quantization.

### 4.2 Main results

**Table 1: Comparison of the perplexity score on WikiText2 and averaged accuracy on Zero-shot Common Sense Reasoning tasks. 0-shot${}^{4}$ employs ARC-easy, ARC-challenge, PIQA, and WinoGrande tasks, while 0-shot${}^{8}$ adds BoolQ, SIQA, HellaSwag, and OBQA tasks. Results for SmoothQuant[], LLM-QAT[], GPTQ [], and QuaRot [] were obtained using their publicly released codebase. While OmniQuant [], AQLM [], AWQ [], QuIP [], and QuIP# [] results were quoted from their papers. SpinQuant* and QuaRot* represent using RTN quantization, while SpinQuant and QuaRot denote using GPTQ weight quantization. Mean scores for SpinQuant, GPTQ, and QuaRot are reported from six trials. Full results, including error bars, are in the Appendix.**
|  |  | LLaMA-3 8B | LLaMA-3 70B | LLaMA-2 7B | LLaMA-2 13B | LLaMA-2 70B |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| \cdashline3-15<br>#Bits | Method | 0-shot${}^{8}$ | Wiki | 0-shot${}^{8}$ | Wiki | 0-shot${}^{4}$ | 0-shot${}^{8}$ | Wiki | 0-shot${}^{4}$ | 0-shot${}^{8}$ | Wiki | 0-shot${}^{4}$ | 0-shot${}^{8}$ | Wiki |
| W-A-KV |  | Avg.($\uparrow$) | ($\downarrow$) | Avg.($\uparrow$) | ($\downarrow$) | Avg.($\uparrow$) | Avg.($\uparrow$) | ($\downarrow$) | Avg.($\uparrow$) | Avg.($\uparrow$) | ($\downarrow$) | Avg.($\uparrow$) | Avg.($\uparrow$) | ($\downarrow$) |
| 16-16-16 | FloatingPoint | 69.6 | 6.1 | 74.5 | 2.8 | 68.6 | 66.9 | 5.5 | 69.9 | 68.3 | 5.0 | 76.0 | 72.9 | 3.3 |
| \hdashline<br>4-16-16 | RTN | 65.4 | 7.8 | 35.5 | 1e5 | 65.3 | 63.6 | 7.2 | 59.9 | 57.9 | 6.4 | 72.3 | 69.2 | 4.6 |
|  | SmoothQuant | 61.0 | 10.7 | 66.9 | 12.0 | 62.3 | 59.1 | 7.5 | 66.1 | 63.3 | 6.1 | 73.6 | 70.2 | 4.1 |
|  | LLM-QAT | 67.7 | 7.1 | – | – | 67.1 | 64.9 | 5.9 | – | – | – | – | – | – |
|  | OmniQuant | – | – | – | – | 62.5 | – | 5.7 | 64.9 | – | 5.0 | 71.1 | – | 3.5 |
|  | QuIP | – | – | – | – | – | – | – | 66.7 | – | – | 69.4 | – | – |
|  | AQLM | – | – | – | – | 63.6 | – | – | 66.3 | – | – | 71.9 | – | – |
|  | QuIP# | – | – | – | – | 63.9 | – | – | 67.1 | – | – | 71.8 | – | – |
|  | AWQ | – | – | – | – | – | – | 6.2 | – | – | 5.1 | – | – | – |
|  | GPTQ | 66.5 | 7.2 | 35.7 | 1e5 | 66.6 | 64.5 | 11.3 | 67.0 | 64.7 | 5.6 | 75.0 | 71.9 | 3.9 |
|  | QuaRot* | 66.2 | 7.5 | 57.2 | 41.6 | 64.5 | 62.4 | 6.9 | 69.3 | 67.1 | 5.5 | 74.6 | 71.8 | 3.7 |
|  | QuaRot | 68.4 | 6.4 | 70.3 | 7.9 | 67.6 | 65.8 | 5.6 | 70.3 | 68.3 | 5.0 | 75.2 | 72.2 | 3.5 |
|  | SpinQuant* | 67.6 | 6.5 | 71.4 | 3.9 | 66.7 | 64.6 | 5.5 | 69.6 | 67.4 | 4.9 | 74.9 | 72.2 | 3.5 |
|  | SpinQuant | 68.5 | 6.4 | 71.6 | 4.8 | 67.7 | 65.9 | 5.6 | 70.5 | 68.5 | 5.0 | 75.4 | 72.6 | 3.5 |
| \hdashline<br>4-4-16 | RTN | 38.5 | 9e2 | 35.6 | 1e5 | 37.3 | 35.6 | 2e3 | 37.9 | 35.3 | 7e3 | 37.2 | 35.1 | 2e5 |
|  | SmoothQuant | 40.3 | 8e2 | 55.3 | 18.0 | 44.2 | 41.8 | 2e2 | 46.4 | 44.9 | 34.5 | 46.8 | 64.6 | 57.1 |
|  | LLM-QAT | 44.9 | 42.9 | – | – | 48.8 | 47.8 | 12.9 | – | – | – | – | – | – |
|  | OmniQuant | – | – | – | – | – | – | 14.3 | – | – | 12.3 | – | – | – |
|  | GPTQ | 37.0 | 9e2 | 35.3 | 1e5 | 38.3 | 36.8 | 8e3 | 37.7 | 35.3 | 5e3 | 37.8 | 35.5 | 2e6 |
|  | QuaRot* | 59.5 | 10.4 | 41.5 | 91.2 | 60.9 | 59.0 | 8.2 | 66.9 | 64.8 | 6.1 | 72.6 | 69.7 | 4.2 |
|  | QuaRot | 63.8 | 7.9 | 65.4 | 20.4 | 65.6 | 63.5 | 6.1 | 68.5 | 66.7 | 5.4 | 72.9 | 70.4 | 3.9 |
|  | SpinQuant* | 64.6 | 7.7 | 70.1 | 4.1 | 63.6 | 61.8 | 6.1 | 67.7 | 65.8 | 5.4 | 73.5 | 71.1 | 3.9 |
|  | SpinQuant | 65.8 | 7.1 | 69.5 | 5.5 | 65.9 | 64.1 | 5.9 | 69.3 | 67.2 | 5.2 | 73.5 | 71.0 | 3.8 |
| \hdashline<br>4-4-4 | RTN | 38.2 | 1e3 | 35.2 | 1e5 | 38.2 | 37.1 | 2e3 | 37.2 | 35.4 | 7e3 | 37.4 | 35.0 | 2e5 |
|  | SmoothQuant | 38.7 | 1e3 | 52.4 | 22.1 | 40.1 | 39.0 | 6e2 | 42.7 | 40.5 | 56.6 | 58.8 | 55.9 | 10.5 |
|  | LLM-QAT | 43.2 | 52.5 | – | – | 45.5 | 44.9 | 14.9 | – | – | – | – | – | – |
|  | GPTQ | 37.1 | 1e3 | 35.1 | 1e5 | 38.1 | 36.8 | 9e3 | 37.4 | 35.2 | 5e3 | 37.8 | 35.6 | 1e6 |
|  | QuaRot* | 58.6 | 10.9 | 41.3 | 92.4 | 60.5 | 58.7 | 8.2 | 66.5 | 64.4 | 6.2 | 72.2 | 69.5 | 4.2 |
|  | QuaRot | 63.3 | 8.0 | 65.1 | 20.2 | 64.4 | 62.5 | 6.4 | 68.1 | 66.2 | 5.4 | 73.0 | 70.3 | 3.9 |
|  | SpinQuant* | 64.1 | 7.8 | 70.1 | 4.1 | 63.2 | 61.5 | 6.2 | 67.7 | 65.5 | 5.4 | 73.2 | 70.5 | 3.9 |
|  | SpinQuant | 65.2 | 7.3 | 69.3 | 5.5 | 66.0 | 64.0 | 5.9 | 68.9 | 66.9 | 5.3 | 73.7 | 71.2 | 3.8 |

### 4.3 Ablation studies

#### 4.3.1 Compatibility with GPTQ

**Table 2: Compatibility with GPTQ.**
| #Bits${}_{\text{(W-A-KV)}}$ | Task | Cayley on 4-4-KV | Cayley on 16-4-KV |
| --- | --- | --- | --- |
| 4-4-16 | 0-shot${}^{8}$ Avg. | 61.0 ${}_{\pm 1.0}$ | 64.1 ${}_{\pm 0.4}$ |
| Wiki | 6.7 ${}_{\pm 0.07}$ | 5.9 ${}_{\pm 0.00}$ |  |
| 4-4-4 | 0-shot${}^{8}$ Avg. | 60.9 ${}_{\pm 0.6}$ | 64.0 ${}_{\pm 0.3}$ |
| Wiki | 6.8 ${}_{\pm 0.15}$ | 5.9 ${}_{\pm 0.01}$ |  |

In the context where both weights and activations are quantized, we observed that the Cayley-optimized rotations tend to adapt effectively to both weight and activation quantization. Given that GPTQ significantly helps mitigate the errors due to weight quantization, but leaves activation quantization untouched, we elect to optimize the rotation matrices with respect to a network where only activations are quantized. This approach allows the rotation to more efficiently manage the activation quantization error, while leaving the weight quantization error to be addressed by GPTQ. As shown in Table [2](#S4.T2), this modification resulted in superior performance in both W4A4 and W4A4KV4 settings in the LLaMA-2 7B model, which is the configuration we have chosen to utilize throughout the rest of this paper.

#### 4.3.2 Impact of each rotation

**Table 3: Impact of individual rotation matrices ($R_{1}$, $R_{2}$, $R_{3}$, $R_{4}$) to quantized model perplexity on WikiText2 and average accuracy on eight zero-shot common sense reasoning tasks. Refer to Figure [1](#S1.F1) for $R_{1}$ (residual rotation), $R_{2}$ (MHSA rotation), $R_{3}$ (query/key rotation), and $R_{4}$ (down projection layer input rotation).**
|  | W4A16KV16 | W4A4KV16 | W4A4KV4 |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | 0-shot${}^{8}$ | Wiki | 0-shot${}^{8}$ | Wiki | 0-shot${}^{8}$ | Wiki |
| Rotation | Avg.($\uparrow$) | ($\downarrow$) | Avg.($\uparrow$) | ($\downarrow$) | Avg.($\uparrow$) | ($\downarrow$) |
| no rotation | 63.6 | 7.2 | 35.6 | 2167.2 | 37.1 | 2382.5 |
| $R_{1}$ | 60.2${}_{\pm 0.5}$ | 6.5${}_{\pm 0.0}$ | 52.3${}_{\pm 0.7}$ | 9.6${}_{\pm 0.0}$ | 51.8${}_{\pm 0.5}$ | 9.9${}_{\pm 0.1}$ |
| $R_{1}$ + $R_{4}$ | 64.7${}_{\pm 0.3}$ | 5.5${}_{\pm 0.0}$ | 61.6${}_{\pm 0.6}$ | 6.2${}_{\pm 0.04}$ | 60.6${}_{\pm 0.6}$ | 6.4${}_{\pm 0.1}$ |
| $R_{1}$ + $R_{2}$ + $R_{4}$ | 64.6${}_{\pm 0.2}$ | 5.5${}_{\pm 0.0}$ | 62.0${}_{\pm 0.6}$ | 6.2${}_{\pm 0.0}$ | 61.0${}_{\pm 0.4}$ | 6.3${}_{\pm 0.0}$ |
| $R_{1}$ + $R_{2}$ + $R_{3}$ + $R_{4}$ | 64.6${}_{\pm 0.3}$ | 5.5${}_{\pm 0.0}$ | 61.8${}_{\pm 0.4}$ | 6.1${}_{\pm 0.0}$ | 61.5${}_{\pm 0.3}$ | 6.2${}_{\pm 0.0}$ |

We performed an ablation study to assess the influence of each rotation matrix, employing round-to-nearest quantization for our evaluation. Cayley optimization was used to optimize $R_{1}$ and $R_{2}$, while $R_{3}$ and $R_{4}$ were maintained as Hadamard matrices across all configurations. As indicated in Table [3](#S4.T3), the inclusion of additional rotations generally enhances the accuracy of quantization, particularly in the case of activation quantization. In LLaMA models, the most significant outliers are found prior to the down projection layer. Consequently, implementing the online Hadamard rotation ($R_{4}$) before these layers can most effectively improve accuracy. The introduction of $R_{2}$ further boosts the 4-bit activation quantization by $\sim$0.4 points. Interestingly, applying initial rotations in the residual ($R_{1}$) results in a 3-5 point drop in weight quantization accuracy. However, this negative impact is counterbalanced by the incorporation of $R_{4}$, effectively restoring the accuracy.

#### 4.3.3 Rotation type

**Table 4: Floating-point(FP) rotation vs Hadamard rotation**
| #Bits |  | No Cayley + RTN | Cayley + RTN |  |  |
| --- | --- | --- | --- | --- | --- |
| (W-A-KV) | Task | FP | Hadamard | FP init. | Hadamard init. |
| 4-16-16 | 0-shot${}^{8}$ Avg.($\uparrow$) | 62.5 ${}_{\pm 0.8}$ | 62.4 ${}_{\pm 1.0}$ | 64.9 ${}_{\pm 0.4}$ | 64.6 ${}_{\pm 0.3}$ |
| Wiki($\downarrow$) | 6.7 ${}_{\pm 0.12}$ | 6.9 ${}_{\pm 0.45}$ | 5.5 ${}_{\pm 0.01}$ | 5.5 ${}_{\pm 0.01}$ |  |
| 4-4-16 | 0-shot${}^{8}$ Avg.($\uparrow$) | 49.4 ${}_{\pm 2.8}$ | 59.0 ${}_{\pm 1.0}$ | 61.6 ${}_{\pm 0.4}$ | 61.8 ${}_{\pm 0.4}$ |
| Wiki($\downarrow$) | 15.9 ${}_{\pm 4.04}$ | 8.2 ${}_{\pm 0.73}$ | 6.2 ${}_{\pm 0.06}$ | 6.1 ${}_{\pm 0.03}$ |  |
| 4-4-4 | 0-shot${}^{8}$ Avg.($\uparrow$) | 48.3 ${}_{\pm 2.7}$ | 58.7 ${}_{\pm 1.0}$ | 61.5 ${}_{\pm 0.8}$ | 61.5 ${}_{\pm 0.3}$ |
| Wiki($\downarrow$) | 18.2 ${}_{\pm 4.35}$ | 8.2 ${}_{\pm 0.36}$ | 6.3 ${}_{\pm 0.08}$ | 6.2 ${}_{\pm 0.03}$ |  |

In Table [4](#S4.T4), we evaluate the impact of random orthogonal floating-point rotation matrices and random Hadamard matrices on quantization accuracy, utilizing round-to-nearest quantization for our analysis. Prior to the application of Cayley optimization, the Hadamard matrices yield better quantized network performance compared to floating-point rotation matrices.
However, after Cayley optimization, the initial choice of rotation, whether floating-point or Hadamard, becomes less significant. This is likely due to the Cayley optimization’s ability to locate an optimal local minima that effectively minimizes quantization error, thereby enhancing robustness to varying types of rotation initialization.

## 5 Related Work

Quantization
Neural network quantization has been demonstrated as an effective tool for model size compression and storage reduction [, , , ].
However, in large language models (LLMs), quantization presents unique challenges due to the presence of numerous outliers. These outliers dominate the quantization range, leaving only a few effective bits for the majority of values. Various strategies have been proposed to address the difficulties in LLM quantization. These include separating outliers and using mixed precision [, , ], employing Hessian-based methods to mitigate quantization difficulty [], trading outliers between weights and activations [, , ] utilizing weight equalization [], and even suggesting architectural modifications to handle outliers during pre-training[]. Recently two QuIP papers [, ] introduce the incoherence processing using random rotation matrices and applying vector quantization on the weights for compression. This does introduce extra overhead and imposes some constraints on the devices the LLM is deployed to in the availability of vector quantization kernels. Ashkboos et al. [] recently released their method QuaRot.
Our work was done concurrently with theirs. The authors also introduce random rotation matrices in a similar fashion as our paper. However, there are some notable differences. (1) Instead of relying on random matrices, we learn the rotation matrices. As seen in section [2.2](#S2.SS2), if accidentally randomized to a poor random matrix choice could be quite detrimental to performance. (2) We introduce less extra parameters in our method with the rotation matrix inside of each block as in section [3.1](#S3.SS1). (3) Our pipeline gives significantly better performance compared to QuaRot, as demonstrated in Section [4](#S4).

Optimization in orthonormal space
The optimization of rotation matrices is carried out within the Stiefel Manifold [], which encompasses all orthonormal matrices. Optimization while staying on this manifold can be done by e.g., parameterizing a skew-symmetric matrix and applying the Cayley transformation on top of it [], or using a matrix exponential [, ]. However, these methods rely on expensive inverse or matrix-exponential functions that are applied every iteration. Instead, we follow the more efficient method named Cayley SGD [], which can be applied to optimize a rotation matrix $R$ for arbitrary loss functions efficiently. Cayley SGD relies on an iterative approximation of the Cayley Transform that is conducted solely with matrix multiplications.

## 6 Conclusions

In this paper, we present SpinQuant, a novel quantization technique that effectively bridges the performance gap between full precision and 4-bit weight, activation, and kv-cache quantization, for the LLaMA-2 7B model to within a mere 2.9 points. At its core, SpinQuant leverages the rotation invariance property of LLM models to insert rotation matrices that diminish outliers in the weights and intermediate activations while maintaining the network’s full-precision output numerically identical. Additionally, SpinQuant incorporates Cayley SGD for optimizing rotation matrices to boost quantization performance further. Importantly, our method is compatible with more advanced weight quantization techniques (e.g., GPTQ) and demonstrates state-of-the-art performance.

## 7 Limitations and Broader Impacts

In this study, we introduce a rotation-based quantization technique designed for precise low-bit quantization of LLMs. This method has the potential to reduce energy consumption during LLM inference. We evaluated our model’s performance in an academic context. However, real-world tasks may involve different training data and activation distributions, suggesting that the model’s generalizability to real-world scenarios warrants further investigation.

## Appendix A Appendix / supplemental material

### A.1 Complete results of main result table

In Tables [A.1](#A1.SS1) and  [A.1](#A1.SS1), we show the complete results of Table [4.2](#S4.SS2). We compare the accuracy on eight zero-shot commonsense reasoning tasks including ARC-easy, ARC-challenge [], BoolQ [], PIQA [], SIQA [], HellaSwag [], OBQA [], and WinoGrande []. We compare our results with previous works including SmoothQuant[], LLM-QAT[], GPTQ [], OmniQuant [], AQLM [], ATOM [], AWQ [], QuIP [], QuIP# [], and QuaRot [].

**Table 5: Complete omparison of the perplexity score on WikiText2 and averaged accuracy on Zero-shot Common Sense Reasoning tasks on LLaMA-3. We reported the mean and standard deviation across six trails for SpinQuant as well as our reproduced results of GPTQ and QuaRot.**
| Model | #Bits | Method | ARC-e | ARC-c | BoolQ | PIQA | SIQA | HellaS. | OBQA | WinoG. | Avg. | Wiki2 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| W-A-KV | ($\uparrow$) | ($\uparrow$) | ($\uparrow$) | ($\uparrow$) | ($\uparrow$) | ($\uparrow$) | ($\uparrow$) | ($\uparrow$) | ($\uparrow$) | ($\downarrow$) |  |  |
| 8B | 16-16-16 | Full Precision | 77.6 | 57.7 | 83.3 | 80.7 | 48.7 | 79.6 | 55.8 | 73.7 | 69.6 | 6.1 |
| \cdashline2-13 | 4-16-16 | RTN | 74.7 | 49.0 | 73.0 | 77.0 | 47.6 | 76.6 | 53.4 | 71.4 | 65.4 | 7.8 |
|  |  | SmoothQuant | 67.6 | 41.3 | 72.6 | 74.4 | 46.7 | 70.6 | 48.0 | 67.0 | 61.0 | 10.7 |
|  |  | LLM-QAT | 77.1 | 53.0 | 82.4 | 79.0 | 48.1 | 76.6 | 54.4 | 71.3 | 67.7 | 7.1 |
|  |  | GPTQ | 73.5 ${}_{\pm 1.9}$ | 50.2 ${}_{\pm 1.0}$ | 79.7 ${}_{\pm 1.5}$ | 77.9 ${}_{\pm 0.7}$ | 48.8 ${}_{\pm 0.3}$ | 76.4 ${}_{\pm 0.2}$ | 52.6 ${}_{\pm 0.9}$ | 72.6 ${}_{\pm 1.0}$ | 66.5 ${}_{\pm 0.6}$ | 7.2 ${}_{\pm 0.02}$ |
|  |  | QuaRot* | 73.8 ${}_{\pm 1.4}$ | 51.4 ${}_{\pm 0.9}$ | 80.0 ${}_{\pm 1.7}$ | 77.7 ${}_{\pm 1.1}$ | 47.8 ${}_{\pm 0.8}$ | 75.9 ${}_{\pm 0.4}$ | 52.1 ${}_{\pm 0.8}$ | 71.1 ${}_{\pm 0.7}$ | 66.2 ${}_{\pm 0.6}$ | 7.5 ${}_{\pm 0.02}$ |
|  |  | QuaRot | 77.0 ${}_{\pm 0.8}$ | 55.2 ${}_{\pm 0.8}$ | 82.2 ${}_{\pm 0.6}$ | 79.5 ${}_{\pm 0.4}$ | 48.6 ${}_{\pm 0.5}$ | 78.3 ${}_{\pm 0.3}$ | 54.4 ${}_{\pm 0.9}$ | 72.1 ${}_{\pm 0.8}$ | 68.4 ${}_{\pm 0.2}$ | 6.4 ${}_{\pm 0.01}$ |
|  |  | SpinQuant* | 76.5 ${}_{\pm 1.1}$ | 54.8 ${}_{\pm 1.5}$ | 79.8 ${}_{\pm 2.1}$ | 79.0 ${}_{\pm 0.8}$ | 47.6 ${}_{\pm 1.0}$ | 78.0 ${}_{\pm 0.3}$ | 53.3 ${}_{\pm 1.1}$ | 71.6 ${}_{\pm 0.8}$ | 67.6 ${}_{\pm 0.6}$ | 6.5 ${}_{\pm 0.01}$ |
|  |  | SpinQuant | 77.6 ${}_{\pm 0.7}$ | 55.5 ${}_{\pm 0.9}$ | 81.4 ${}_{\pm 1.3}$ | 79.4 ${}_{\pm 0.2}$ | 48.2 ${}_{\pm 0.4}$ | 78.4 ${}_{\pm 0.2}$ | 55.1 ${}_{\pm 1.3}$ | 72.4 ${}_{\pm 1.0}$ | 68.5 ${}_{\pm 0.2}$ | 6.4 ${}_{\pm 0.01}$ |
| \cdashline2-13 | 4-4-16 | RTN | 31.8 | 27.6 | 47.2 | 53.8 | 39.7 | 30.8 | 28.2 | 48.9 | 38.5 | 923.9 |
|  |  | SmoothQuant | 36.3 | 26.3 | 50.6 | 54.1 | 40.3 | 31.4 | 30.6 | 52.9 | 40.3 | 867.5 |
|  |  | LLM-QAT | 44.1 | 29.7 | 58.0 | 61.5 | 42.1 | 39.9 | 33.0 | 51.3 | 44.9 | 42.9 |
|  |  | GPTQ | 31.4 ${}_{\pm 0.9}$ | 24.7 ${}_{\pm 1.4}$ | 42.5 ${}_{\pm 1.3}$ | 52.7 ${}_{\pm 1.0}$ | 39.1 ${}_{\pm 0.9}$ | 27.8 ${}_{\pm 0.3}$ | 27.3 ${}_{\pm 2.5}$ | 50.7 ${}_{\pm 1.1}$ | 37.0 ${}_{\pm 0.7}$ | 955.9 |
|  |  | QuaRot* | 66.0 ${}_{\pm 1.2}$ | 42.5 ${}_{\pm 1.0}$ | 70.5 ${}_{\pm 2.0}$ | 72.5 ${}_{\pm 0.6}$ | 45.4 ${}_{\pm 0.9}$ | 68.6 ${}_{\pm 0.9}$ | 46.7 ${}_{\pm 2.1}$ | 63.5 ${}_{\pm 1.7}$ | 59.5 ${}_{\pm 0.6}$ | 10.4 ${}_{\pm 0.26}$ |
|  |  | QuaRot | 72.4 ${}_{\pm 1.1}$ | 48.0 ${}_{\pm 1.1}$ | 75.8 ${}_{\pm 1.4}$ | 75.9 ${}_{\pm 0.6}$ | 47.1 ${}_{\pm 0.8}$ | 73.7 ${}_{\pm 0.6}$ | 51.0 ${}_{\pm 1.8}$ | 66.7 ${}_{\pm 1.2}$ | 63.8 ${}_{\pm 0.5}$ | 7.9 ${}_{\pm 0.04}$ |
|  |  | SpinQuant* | 74.1 ${}_{\pm 1.6}$ | 49.7 ${}_{\pm 1.7}$ | 75.8 ${}_{\pm 3.2}$ | 77.0 ${}_{\pm 0.6}$ | 46.4 ${}_{\pm 0.9}$ | 74.7 ${}_{\pm 0.4}$ | 52.0 ${}_{\pm 1.9}$ | 67.1 ${}_{\pm 1.0}$ | 64.6 ${}_{\pm 0.8}$ | 7.7 ${}_{\pm 0.05}$ |
|  |  | SpinQuant | 75.0 ${}_{\pm 1.0}$ | 50.9 ${}_{\pm 1.2}$ | 78.9 ${}_{\pm 0.6}$ | 77.5 ${}_{\pm 0.7}$ | 47.2 ${}_{\pm 0.6}$ | 75.9 ${}_{\pm 0.4}$ | 52.9 ${}_{\pm 1.6}$ | 68.5 ${}_{\pm 1.0}$ | 65.8 ${}_{\pm 0.2}$ | 7.1 ${}_{\pm 0.02}$ |
| \cdashline2-13 | 4-4-4 | RTN | 31.9 | 26.1 | 46.2 | 52.3 | 39.9 | 29.9 | 28.6 | 51.0 | 38.2 | 1,118.5 |
|  |  | SmoothQuant | 33.5 | 25.1 | 49.6 | 53.1 | 40.3 | 28.8 | 29.6 | 49.6 | 38.7 | 1,530.5 |
|  |  | LLM-QAT | 40.5 | 26.6 | 52.7 | 59.9 | 42.3 | 37.5 | 33.6 | 52.7 | 43.2 | 52.5 |
|  |  | GPTQ | 31.0 ${}_{\pm 0.9}$ | 24.9 ${}_{\pm 1.0}$ | 41.9 ${}_{\pm 1.1}$ | 52.8 ${}_{\pm 0.6}$ | 38.4 ${}_{\pm 0.3}$ | 27.9 ${}_{\pm 0.4}$ | 28.2 ${}_{\pm 2.0}$ | 51.4 ${}_{\pm 1.1}$ | 37.1 ${}_{\pm 0.3}$ | 1,071.7 |
|  |  | QuaRot* | 65.9 ${}_{\pm 3.0}$ | 41.3 ${}_{\pm 2.2}$ | 69.5 ${}_{\pm 2.3}$ | 71.9 ${}_{\pm 0.9}$ | 44.8 ${}_{\pm 1.1}$ | 67.2 ${}_{\pm 1.6}$ | 46.5 ${}_{\pm 1.8}$ | 61.9 ${}_{\pm 1.5}$ | 58.6 ${}_{\pm 0.8}$ | 10.9 ${}_{\pm 0.26}$ |
|  |  | QuaRot | 71.6 ${}_{\pm 0.9}$ | 48.0 ${}_{\pm 1.2}$ | 74.9 ${}_{\pm 1.8}$ | 75.1 ${}_{\pm 0.5}$ | 46.8 ${}_{\pm 0.9}$ | 73.1 ${}_{\pm 0.7}$ | 50.4 ${}_{\pm 1.0}$ | 66.1 ${}_{\pm 1.4}$ | 63.3 ${}_{\pm 0.3}$ | 8.0 ${}_{\pm 0.05}$ |
|  |  | SpinQuant* | 72.6 ${}_{\pm 1.4}$ | 49.5 ${}_{\pm 2.3}$ | 74.8 ${}_{\pm 6.3}$ | 76.6 ${}_{\pm 0.8}$ | 46.4 ${}_{\pm 0.4}$ | 74.3 ${}_{\pm 1.0}$ | 50.6 ${}_{\pm 3.1}$ | 67.9 ${}_{\pm 1.0}$ | 64.1 ${}_{\pm 1.7}$ | 7.8 ${}_{\pm 0.05}$ |
|  |  | SpinQuant | 74.4 ${}_{\pm 1.3}$ | 50.4 ${}_{\pm 1.7}$ | 77.7 ${}_{\pm 1.6}$ | 76.9 ${}_{\pm 0.6}$ | 47.2 ${}_{\pm 0.5}$ | 75.5 ${}_{\pm 0.2}$ | 52.0 ${}_{\pm 1.1}$ | 67.2 ${}_{\pm 1.4}$ | 65.2 ${}_{\pm 0.6}$ | 7.3 ${}_{\pm 0.02}$ |
| 70B | 16-16-16 | Full Precision | 80.6 | 64.5 | 87.4 | 83.7 | 51.7 | 85.3 | 62.0 | 80.5 | 74.5 | 2.8 |
| \cdashline2-13 | 4-16-16 | RTN | 27.3 | 27.2 | 37.8 | 51.0 | 39.1 | 25.6 | 26.2 | 49.8 | 35.5 | 1e5 |
|  |  | SmoothQuant | 76.7 | 45.5 | 80.6 | 81.2 | 48.7 | 81.1 | 46.2 | 75.5 | 66.9 | 12.0 |
|  |  | GPTQ | 28.2 ± 1.3 | 24.7 ± 1.6 | 37.9 ± 0.1 | 50.7 ± 0.8 | 39.0 ± 0.6 | 26.8 ± 1.7 | 27.7 ± 3.6 | 50.5 ± 0.6 | 35.7 ± 0.6 | 1e5 |
|  |  | QuaRot* | 65.9 ± 1.4 | 44.3 ± 2.7 | 67.0 ± 4.9 | 75.2 ± 2.0 | 44.1 ± 2.1 | 59.8 ± 8.6 | 42.5 ± 3.8 | 58.5 ± 2.9 | 57.2 ± 2.7 | 41.6 ± 16.76 |
|  |  | QuaRot | 74.4 ± 0.7 | 58.6 ± 2.6 | 86.4 ± 0.5 | 83.8 ± 0.4 | 51.9 ± 0.4 | 83.7 ± 0.1 | 47.7 ± 2.2 | 76.1 ± 0.6 | 70.3 ± 0.7 | 7.9 ± 0.21 |
|  |  | SpinQuant* | 78.5 ± 2.2 | 57.9 ± 1.5 | 84.5 ± 1.0 | 82.3 ± 0.6 | 50.3 ± 0.6 | 82.6 ± 0.4 | 57.4 ± 5.9 | 77.5 ± 1.9 | 71.4 ± 1.5 | 3.9 ± 0.47 |
|  |  | SpinQuant | 78.0 ± 2.7 | 59.1 ± 0.9 | 85.2 ± 1.1 | 82.8 ± 1.0 | 50.6 ± 0.6 | 83.5 ± 0.2 | 54.8 ± 6.6 | 78.5 ± 1.6 | 71.6 ± 1.1 | 4.8 ± 1.97 |
| \cdashline2-13 | 4-4-16 | RTN | 27.6 | 27.0 | 38.2 | 50.1 | 38.5 | 26.0 | 28.4 | 49.1 | 35.6 | 1e5 |
|  |  | SmoothQuant | 59.5 | 35.7 | 62.4 | 70.3 | 44.3 | 61.7 | 44.2 | 63.9 | 55.3 | 18.0 |
|  |  | GPTQ | 27.0 ± 0.4 | 26.1 ± 1.5 | 39.1 ± 1.2 | 50.4 ± 0.4 | 38.9 ± 0.4 | 25.7 ± 0.2 | 25.7 ± 1.7 | 49.6 ± 0.9 | 35.3 ± 0.2 | 1e5 |
|  |  | QuaRot* | 40.8 ± 4.8 | 26.2 ± 2.8 | 52.4 ± 3.8 | 58.4 ± 3.5 | 40.0 ± 0.9 | 33.8 ± 3.9 | 30.0 ± 3.6 | 50.2 ± 1.7 | 41.5 ± 2.5 | 91.2 ± 24.05 |
|  |  | QuaRot | 72.4 ± 1.5 | 52.2 ± 1.6 | 78.5 ± 2.4 | 78.9 ± 0.8 | 49.0 ± 1.1 | 78.5 ± 0.9 | 45.2 ± 1.9 | 68.2 ± 3.0 | 65.4 ± 1.3 | 20.4 ± 3.23 |
|  |  | SpinQuant* | 77.2 ± 0.9 | 55.9 ± 1.1 | 81.7 ± 1.7 | 80.9 ± 0.6 | 49.0 ± 0.5 | 80.9 ± 0.4 | 58.7 ± 1.9 | 76.2 ± 0.8 | 70.1 ± 0.4 | 4.1 ± 0.02 |
|  |  | SpinQuant | 76.7 ± 1.9 | 55.6 ± 2.1 | 82.3 ± 1.3 | 80.6 ± 1.1 | 49.8 ± 0.9 | 81.0 ± 1.6 | 55.4 ± 6.3 | 74.5 ± 3.9 | 69.5 ± 2.1 | 5.5 ± 2.56 |
| \cdashline2-13 | 4-4-4 | RTN | 27.0 | 24.1 | 38.5 | 50.4 | 38.8 | 25.8 | 25.2 | 51.8 | 35.2 | 1e5 |
|  |  | SmoothQuant | 55.0 | 34.9 | 62.2 | 66.8 | 43.1 | 59.4 | 39.8 | 58.0 | 52.4 | 22.1 |
|  |  | GPTQ | 27.1 ± 0.3 | 24.8 ± 1.7 | 38.8 ± 0.7 | 50.8 ± 0.6 | 39.0 ± 0.4 | 25.5 ± 0.2 | 24.8 ± 2.9 | 49.8 ± 0.6 | 35.1 ± 0.2 | 1e5 |
|  |  | QuaRot* | 40.5 ± 4.9 | 25.7 ± 3.7 | 50.9 ± 4.5 | 57.7 ± 3.3 | 39.9 ± 0.8 | 33.9 ± 3.9 | 31.0 ± 2.6 | 51.0 ± 1.6 | 41.3 ± 2.6 | 92.4 ± 24.18 |
|  |  | QuaRot | 72.3 ± 1.9 | 51.6 ± 1.0 | 77.5 ± 2.0 | 78.9 ± 1.1 | 49.0 ± 1.0 | 78.2 ± 0.9 | 45.5 ± 1.7 | 67.8 ± 2.6 | 65.1 ± 1.1 | 20.2 ± 3.12 |
|  |  | SpinQuant* | 77.3 ± 1.0 | 56.0 ± 1.1 | 81.8 ± 1.7 | 80.8 ± 0.4 | 49.3 ± 0.5 | 80.9 ± 0.3 | 58.7 ± 0.7 | 76.4 ± 0.3 | 70.1 ± 0.5 | 4.1 ± 0.01 |
|  |  | SpinQuant | 76.8 ± 2.1 | 55.8 ± 2.6 | 82.2 ± 1.7 | 81.0 ± 0.8 | 49.5 ± 1.0 | 81.2 ± 1.3 | 53.8 ± 6.3 | 74.2 ± 3.8 | 69.3 ± 2.2 | 5.5 ± 2.59 |

### A.2 Cayley optimization choice

**Table 7: Ablation study on Number of training samples and iterations in Cayley SGD optimization.**
| #Bits | Task | # Training sample | # Training iterations |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| (W-A-KV) | 128 | 800 | 10 | 25 | 50 | 100 | 200 |  |
| 4-4-4 | Wiki ($\downarrow$) | 6.2 ${}_{\pm 0.03}$ | 6.2 ${}_{\pm 0.03}$ | 6.6 ${}_{\pm 0.02}$ | 6.4 ${}_{\pm 0.02}$ | 6.3 ${}_{\pm 0.03}$ | 6.2 ${}_{\pm 0.03}$ | 6.2 ${}_{\pm 0.05}$ |

In Table [7](#A1.T7), we evaluate the impact of varying the number of samples and iterations used in Cayley optimization. Given the relatively small number of trainable parameters in the rotation matrix compared to the original weight parameters, and considering it as a constraint optimization, we only need a minimal amount of calibration data and iterations to enhance the rotation for improved quantization. The findings indicate that rotation optimization is resilient to modifications in the number of samples. Even though we used 800 samples in our experiments, reducing this to 128 samples does not lead to a significant change in the perplexity. Furthermore, we examined the optimal number of iterations and found that the wiki perplexity ceases to decrease and stabilizes at 100 iterations. Consequently, we chose to use 100 iterations in all our experiments.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. It would be interesting to see the performance of such dynamic quantization ($\texttt{SpinQuant}$) with static quantization alternative [1]. 

2. Is the R3, R4 needed in case someone uses block reconstruction of [2]? 

3. Also, how does $\texttt{SpinQuant}$ compare w.r.t these alternative methods?

4. Why the choice of ``optimizing these rotation matrices on Stiefel manifold", is not fully clear.

[1] PREFIXQUANT: STATIC QUANTIZATION BEATS DYNAMIC THROUGH PREFIXED OUTLIERS IN LLMS, arxiv 2024

[2] TESSERAQ: ULTRA LOW-BIT LLM POST-TRAINING QUANTIZATION WITH BLOCK RECONSTRUCTION, arxiv 2024.
soundness: 2
strengths: The idea of learned rotation in  the context of LLM activation outlier reduction seems interesting.

The authors have done thorough analysis to justify the need of learned rotation 

$\texttt{SpinQuant}$ can outperform baseline GPTQ by significant margin!
summary: The paper presents a learned rotation based method, namely $\texttt{SpinQuant}$ to mitigate outliers in weight and activation distributions, boosting the performance of quantized LLMs. In specific, the authors  proposes optimizing rotation matrices within Stiefel manifold, directly minimizing the final loss of rotated quantized network. The authors further propose two variants, $\texttt{SpinQuant}$_easy and $\texttt{SpinQuant}$_hard without and with change of
weaknesses: 1. The performance of $\texttt{SpinQuant}$ is similar or worse than QuIP in most of the cases. This further raises concerns on the learned rotation method as QuIP does not rely on any activation rotation method.

2. The calibration discussion (thorough) is missing. As the method needs to learn R1,R2, R3 and R4, such calibration and/or fine-tuning overhead is a concern. 

3. How the learning gets affected for activation rotation, when there is extremely low calibration data?

4. The can be other alternatives to learn the R1-4, why did the author chose Cayley SGD method (Li et al., 2020).

5. Why table 6 shows results with $\texttt{SpinQuant}$_hard even for W4A8? 

6. What speedup benefit we may have with $\texttt{SpinQuant}$_hard and $\texttt{SpinQuant}$_easy with W4A4?

7. The contribution is primarily centered around rotation to get rid of activation outliers and smoothen out while for the quantization part the authors heavily rely on GPTQ (or RTN). The baseline GPTQ may often be considered a calibration heavy that require Hessian information. I have concerns on the calibration overhead of the method as well as the novelty.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Although the rotation method is effective, the readers may be interested in the theoretical foundations of this methodology.

Please provide analysis on the overhead introduced by the rotation matrices into the inference pipeline vs the benefits.

The reviewer would like to know the training complexity of the proposed method comparing with other quantization approaches.
soundness: 4
strengths: + The paper pursues the rotation method for post training LLM quantization, which demonstrates convincing performance.

+ The authors conducted thorough investigations about random rotations to propose quantization-oriented rotation learning.

+ The proposed framework develops strategies for the challenging activation and KV cache quantization, besides weight quantization.
summary: This work investigates post-training quantization (PTQ) for LLMs, applied to weights, activations, and KV cache. It addresses the large quantization errors issue from outliers by introducing the learned rotation matrices into the network architecture. Through experiment validations, the proposed framework shows effectiveness in extreme W4A4KV4 quantization settings on representative networks.
weaknesses: - While the rotation matrix method is effective in mitigating outliers in LLM quantization, it is better if more rigorous explanations/proofs are provided for the rationale behind, besides empirical results.

- Introducing rotation matrices into the inference pipeline may lead to overhead in computing. Could the authors provide analysis the overhead vs benefit from quantization?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Have the authors considered testing alternative calibration datasets beyond WikiText2? It would be interesting to see if different calibration sets impact SpinQuant’s generalization, especially in domain-specific applications (e.g., code). While the paper describes the use of WikiText2 for calibration, further analysis on its impact and comparison with alternative calibration data could strengthen the understanding of its generalization.
- it would be interesting to know if SpinQuant has been tested on models from other domains, such as Vision Transformers (ViTs) or vision-language models, to further assess its versatility across different types of architectures.
soundness: 2
strengths: - By employing learned rotations, SpinQuant offers a new approach to minimizing outliers.
- The SpinQuant-easy and SpinQuant-hard modes offer flexibility, making the approach adaptable to different computational constraints and accuracy requirements through mergeable and non-mergeable weights.
- SpinQuant shows compatibility with methods like GPTQ, allowing it to integrate into established quantization pipelines.
summary: The paper presents SpinQuant, a quantization technique specifically designed to reduce the impact of outliers in large language models (LLMs) through learned rotations. SpinQuant introduces learned rotation matrices that can be integrated into pre-trained weights, which aids in preserving accuracy in low-bit quantization scenarios. The proposed method includes two modes: SpinQuant-easy and SpinQuant-hard—the former for more straightforward quantization with mergeable weights, and the latter for complex scenarios needing enhanced outlier suppression with non-mergeable weights. The authors validate SpinQuant across multiple models (e.g., LLaMA-2, LLaMA-3, and Mistral) using a comprehensive set of zero-shot reasoning benchmarks.
weaknesses: - The benchmarks and model selections, such as the LLaMA series, seem to focus on favorable cases for SpinQuant. Testing on a broader range of architectures (e.g., Gemma2) would provide a more thorough evaluation. Especially Gemma2 series which exhibit distinct activation characteristics—could provide a more comprehensive assessment of SpinQuant's generalizability and robustness across diverse LLM types.
- Although the authors note the optimization time (up to 3.5 hours for larger models), it remains unclear how this time impacts real-world deployment, especially on low-resource devices.
- From an outlier-aware quantization perspective, as demonstrated by methods like AWQ, SpinQuant should ideally exhibit strong performance even in weight-only quantization scenarios. However, even after examining the appendix, there is no evaluation comparing its performance under weight-only quantization. It would be beneficial to include a comparison of SpinQuant’s performance in weight-only quantization settings to provide a more comprehensive view of its robustness and effectiveness relative to other quantization techniques.
- While the paper presents promising results on zero-shot reasoning tasks, it does not evaluate SpinQuant's performance in fine-tuning or few-shot learning scenarios. Including such evaluations would strengthen the claims about SpinQuant’s overall effectiveness and versatility.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: - Could you provide a comparison of the computation needed for the quantization? Going beyond comparing accuracies on benchmarks alone would be helpful.
- the number of rotation matrices is determined empirically. Can the authors investigate this further and explain in more detail how this can be validated better?
soundness: 3
strengths: - paper is well-written
- specifically, the illustrative example in 4.4 is very helpful. Yet, it would be sensible to move this one more to the front of the paper
- experiments are extensive across models, settings and methods
summary: This paper introduces SpinQuant, a PTQ method that applies optimized rotations to reduce quantization errors from outliers, improving LLM efficiency without compromising accuracy. It achieves near full-precision performance, narrowing the accuracy gap to 2.9 points on zero-shot tasks in LLaMA-2 7B, outperforming previous works.
weaknesses: - The current rotation is not optimal, as the authors state. It would be a very valuable contribution to investigate optimal rotations further.
- a more detailed discussion on model size vs rotation-based quantization is needed. Larger models are impacted less by quantization compared to smaller models, why?
- currently, no effort to publish code or quantized models given, limiting reproducibility and usability for the community.
- the tradeoff discussion between easy and hard is too shallow. Please elaborate on which cases are useful to use one or the other.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Regarding Formula (1), is the clip operation missing?
- For Figure 1, there are two R3 after the `RoPE`, should one of them be the inverse of R3?
- Could you clarify which rotation type is being demonstrated in Figures 9 and 10?
soundness: 3
strengths: - Proposed a learned version of rotation matrices to address variance issues inherent in random rotation approaches, with demonstrated flexibility across different quantization scenarios.
- The paper is well-structured, presenting clear methodology and comprehensive diagrams that effectively illustrate the proposed approach.
summary: This paper introduces SpinQuant, a novel approach to mitigate outliers during Large Language Model (LLM) quantization by leveraging learned rotation matrices. The paper presents two rotation strategies: 1) an "easy mode" for W4A8KV8 quantization where rotation matrices can be absorbed without impacting inference, and 2) a "hard mode" incorporating two additional online rotation matrices for more aggressive quantization scenarios (e.g., W4A4KV4). Through evaluation across various LLMs and reasoning tasks, SpinQuant demonstrates that its learned rotation matrices outperform random rotation approaches.
weaknesses: - Limited evaluation scope on complex tasks benchmarks: The evaluation would be more compelling with results from more challenging benchmark tasks, such as [MMLU](https://arxiv.org/abs/2009.03300) and [MMLU-Pro](https://arxiv.org/abs/2406.01574).
- Incomplete theoretical analysis, while Section 4.4 demonstrates Hadamard rotation's superiority over random rotation, it lacks a thorough comparative analysis between learned and random rotation matrices. The significant 10+ point improvement observed on Mistral-7B using hard mode could serve as a good case study to demonstrate the benefits of learned rotation matrices.
- Limited performance benchmarking: The end-to-end speed evaluation is confined to MacBook M1 hardware, with only basic FP8 quantization latency testing on GPU rather than comprehensive benchmarking across the paper's proposed quantization scenarios. Given recent research highlighting potential performance impacts of online rotation transformations (e.g., [QServe]( https://arxiv.org/abs/2405.04532), Part V Section C), more extensive benchmarking on modern GPU architectures would strengthen the paper's practical implications. Key speedup metrics could include time to first token (TTFT) and tokens per second (TPS).

Meta-review: The paper introduces SpinQuant, a new method to learn rotation matrices for the purpose of improving quantized network accuracy by reducing errors from outliers in weight and activation distributions. Reviewers agree that the paper is well structured, clearly written, and that it presents a valuable contribution. The empirical performance of the method was well received. The raised weaknesses of the paper were its somewhat incremental novelty, given prior work with rotation ideas, the desire for more detailed corroboration in terms of more extensive benchmarks and e.g. compute time, as well as more theoretical analysis. The authors have added several clarifications, extensions and revisions to address a good amount of open concerns. The reviewers who engaged in discussion have acknowledged that these have improved the paper and have raised their recommendations to accept. In regards to the concern on incremental novelty, the AC  agrees with some reviewers that a thorough embedding in the literature should not be to the detriment of a paper. Overall, the AC believes that the advantages of the paper outweigh the remaining need for improvements, especially because the presentation of the paper is good. The AC thus recommends to accept the paper.

Decision: Accept (Poster)