QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
Authors: Saleh Ashkboos
Amirkeivan Mohtashami
Maximilian L. Croci
Bo Li
Pashmina Cameron
Martin Jaggi
Dan Alistarh
Torsten Hoefler
James Hensman
Venue: NeurIPS 2024 poster (2024)
Keywords: quantization
efficient inference
large language models
Primary area: natural_language_processing
TL;DR: We present a quantization scheme that enables 4-bit quantization of all weights, activations and KV-caches of LLMs during inference

Abstract: We introduce QuaRot, a new Quantization scheme based on Rotations, which is able to quantize LLMs end-to-end, including all weights, activations, and KV cache in 4 bits. QuaRot rotates LLMs in a way that removes outliers from the hidden state without changing the output, making quantization easier. This computational invariance is applied to the hidden state (residual) of the LLM, as well as to the activations of the feed-forward components, aspects of the attention mechanism, and to the KV cache. The result is a quantized model where all matrix multiplications are performed in 4 bits, without any channels identified for retention in higher precision. Our 4-bit quantized LLAMA2-70B model has losses of at most 0.47 WikiText-2 perplexity and retains 99% of the zero-shot performance. We also show that QuaRot can provide lossless 6 and 8 bit LLAMA-2 models without any calibration data using round-to-nearest quantization. Code is available at github.com/spcl/QuaRot.

QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
Saleh Ashkboos
Amirkeivan Mohtashami
ETH Zurich
EPFL
saleh. ashbboosfk www.ff.ehhccc.mmm ch
ch
Maximilian L. Croci Microsoft Research
Bo Li ETH Zurich
Pashmina Cameron
Microsoft pcanerootmiccrooff..coo com
mcrociin croooft.oooooomm
boliboGethz.ch
Martin Jaggi EPFL
Dan Alistarh
IST Austria NeuralMagic
martin. jartin.jaggiieppee seaggieepl..c
dan alistarh@ist. ac.
Torsten Hoefler
James Hensman
ETH Zurich
Microsoft Research aamesshennmmanmicccosssseoee
horsten.hoeflergge
Abstract
We introduce QuaRot. new Quantization scheme based Rorations, which is able to quantize LLMs end-to-end, including all weights, activations, and KV cache in 4 bits. QuaRot rotates LLMs in way that removes outliers from the hidden state without changing the output, making quantization easier. This computational invariance is applied to the hidden state (residual) of the LLM, well as to the acti- vations of the feed-forward components, aspects of the attention mechanism, and to the KV cache. The result is quantized model where all matrix multiplications are performed in bits, without any channels identified for retention in higher precision. Our 4-bit quantized LLAMA2-701 model has losses of most 0.47 WikiText-2 per- plexity and retains 99% of the zero-shot performance. We also show that QuaRot can provide lossless and 8 bit LLAMA-2 models without any calibration data using round-to-nearest quantization. Code is available at github... com/ cwm/spcl///ccccccccccccccccccccc Sapcl/QuaRor
Introduction
Large language models (LLMs) have become increasingly important due to their countless applica- tions. However, using these models in practice, known inference, requires significant amount of computation, memory, and energy, specifically during the prefill phase, in which the model is supposed to process large prompts and cache them in each layer. Quantization is among the most important techniques to improve both memory and compute issues by keeping the data types at lower precision during the forward pass.
As the prefill stage is known be compute-bound [Ashkboos al., 2023], joint quantization aims reduce the precision of parameters and KV cache (which results in lower memory usage) as well as inputs (known as activations) and compute the forward pass in low precision. However, quantizing the activations is hard as they have large outlier elements (see Figure for an illustrative example) with much larger values, making activation quantization more difficult than weight quantization, especially for the 4-bit case. Previous work relies on using a calibration set to characterize the outlier features and keeping them higher precision for inference [Zhao al., 2023, Ashkboos al., 2023].
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Before QuaRot
With QuaRot
0.4
Min/Max
Activation value
1/99 Percentile
0.2
25/75 Percentile
-0.2
-0.4
0
1,000
2,000
3,000
4,000
0
1,000
2,000
3,000
4,000
Hidden dimension index
Hidden dimension index
Figure 1: The distributions activations at the input to the FFN block in LLAMA2-7B model, the tenth layer. Left: using the default configuration as downloaded from Hugging Face. Right: after processing using QuaRot. The processed distribution has no outliers, leading to superior quantization.
In this work, we address the issue of outlier features by rotating the inputs of the model using random- ized Hadamard transformations. We do this using the computational invariance idea [Ashkboos et al., 2024] and fuse Hadamard transformations into the weight matrices, resulting in an equivalent network without outlier features. This enables the weights, activations, and KV caches to be quantized to 4 bits with minimal accuracy drop. Our main contributions are:
We show that randomized Hadamard transformations can be applied the weight matrices without additional model modifications. In turn, this completely eliminates outlier features and makes the activations easy to quantize, without changing the output of the model. This can be seen as an extension of the computational invariance idea, proposed in SliceGPT [Ashkboos et al., 2024] in the context of structured pruning.
We extend this approach to apply online Hadamard transformations to the attention module to remove outlier features in keys and values, enabling the KV cache to be quantized.
Using the above modifications, QuaRot enables 4-bit LLM inference by quantizing all weights, activations, and KV caches using integer quantization. We provide efficient kernel support for QuaRot: on -LAMA2-70B model, QuaRot achieves up to x.33 prefill speedups (on a batch size 64 with 2048 sequence length), and 3.89 memory saving during the decoding stage, with at most 0.47 WikiText-2 perplexity loss. QuaRot preserves 99% of the accuracy of zero-shot tasks and we show that our 6 and 8-bit quantization is lossless with simple cound-to-nearest quantization.
2 Related Work
The majority of quantization schemes focus compressing LLMs by using weight-only quantizationn [Frantar et al., 2022, Dettmers et al., 2023, Lin et al., 2023, Egiazarian et al., 2024, Tseng et al., 2024]. These methods downcast each weight into low- precision representation and upcast it before the actual computation. The main computation is still performed in high precision. Several works show that, unlike weights, quantizing the activations is hard due to the outlier features [Wei al., 2022, Dettmers et al., 2022, Xiao et al., 2023). For 8-bit case, L.M. L..M.nn88) [Dettmers et al., 2022] identifies the outlier features during inference and keeps them in 16 bits which results in poor performance. SmoothQuant [Xiao et al., 2023] normalizes the features using some scaling factors from calibration set, solving the issue for the 8-bit case the cost of introducing extra hyper-parameters. For 4-bit quantization, recent studies identify the outlier features offline and keep them in high precision. Atom [Zhao et al., 2023] developed a complex kernel for mixed-precision MatMul in the presence of outliers while QUIK [Ashkboos et al., 2023] keeps the down down-projection layer in 8 bits.
Two weight-only quantization methods, QulP [Chee et al., 2024] and QulP# ['Tseg et al., 2024] have previously considered improving quantization by applying rotations. Chee et [2024] introduced the idea of incoherence processing which applies rotation matrices to the left and right of each weight

---

matrix, as well as the Hessian, which is used in minimizing the ceigtt-quantization objective. Xi al. [2023] uses similar idea during training, using exact Hadamard transformations for each linear layer in the forward pass.
Finally, KV cache quantization is another line of research that aims to compress the cached keys and values during the generation phase. This is crucial for large batch size and long-context length generation as the KV cache will be the main memory bottleneck in such problems. Sheng al. [2023] quantizes the KV cache using 4-bit group-wise quantization. KVQuant [Hooper et al., 2024] pushes this limit to 3-bit quantization and KIVI [Liu et al., 2024] shows promising results on 2-bit KV cache quantization. Such methods show that outliers also exist in the keys, and apply a set of complex ideas (like feature-wise quantization, non-uniform representation, and keeping high precision outliers) to recover the accuracy of a quantized KV cache.
In this work we also adopt the Hadamard transform to improve quantization of weights through incoherence processing, Instead of undoing the Hadamard transform during the forward pass, we adopt the computational invariance theorem from SliceGPT [Ashkboos et al., 2024] to fuse the transformations into the weights where possible. Instead of requiring two Hadamard transforms per weight-matrix in the forward pass, QuaRot requires just 1 Hadamard transforms per transformer layer. Computational invariance also means that the activations are prooherence-processee enabling them to be effectively quantized. We also apply similar technique to the attention block and quantize the KV cache in 4 bits with minimal accuracy loss.
3 Background
Here we introduce some mathematical concepts and notation that are necessary for QuaRot.
3.1 Orthogonal, Rotation and Hadamard Matrices
An orthogonal matrix Q is square matrix such that QQ T = I. In this work, we consider only real orthogonal matrices. A rotation matrix an orthogonal matrix. A Hadamard matrix is an orthogonal matrix with entries drawing from +11. A Walsh-Hadamard matrix is a square matrix of size 2n, with
1
1
H
and H2n H2 X H2n-1.
-1
These identities give rise to the Walsh-Hadaman transform, which computes the matrix-vector product Hæ in (d)) operations.
For matrix sizes that are not 2", the existence of a Hadamard matrix is not guaranteed. A useful list of known Hadamard matrices is made available by Sloane [2024]. Where we require a Hadamard matrix of size d & 2", we factorize d = m,, where is the size of known Hadamard matrix. Then we use a Kronecker construction H = H2n & Hm- This allows computation of Haæ Oddmm + n)) operations.
Following Tseng et al. [2024] make of randomized Hadamard matrices where convenient. Let be vector containing random draws from (+1,-1), and H H diag(s). It is straightforward see that H is also orthogonal matrix.
3.2 Incoherence Processing
The idea of incoherence processing was introduced by [Chee et al., 2024] in the context of weight normalization for weight-only LLM quantization. We define a weight matrix W be p-incoherent
max (W) S ||| || /m
(2)
where max is the element-wise max of the matrix, and mn is the number of elements. A weight matrix that has high incoherence is hard to quantize: the largest element is an outlier relative to the magnitude of the average element. Chee al. [2024] showed that multiplying a weight matrix on the left and right by an orthogonal matrix can reduce the incoherence, making matrices easier to quantize. In this work we adopt a similar technique, multiplying weight matrices by orthogonal matrices to improve incoherence, though we add fewer operations to the forward pass. Importantly, we additionally apply incoherence processing to the activations, enabling improved weight and activation quantization. Figure shows the effect of applying incoherence processing the activations of LLAMA-2.
3

---

FPN
RMSNorm
gre
Wasan
diag(a)
Wap
Figure 2: The gated feed- serd-forward network used in most LMs, including the pre-positioned RMSNorm. The input signal is divided by its norm, and re-scaled by parameters a. Two linear blocks, Wop and W gate are applied. The activation function a is applied to the gated signal, and the two signals are element-wiss multiplied together. The final linear block Waown down produces the output signal Y. Before quantization, different operations are performed either in single (32 bit) or half (16 bit) precision.
3.3 Transformer structures
Large Language Models are neural networks with repeating attention and feed-forward layers. We introduce our notation through Figures and 5, which show the construction of these blocks. We assume that the construction of the network is "pre-norm", in that each block is preceded by LayerNorm or RMSNorm operation. We also assume that the feed-forward network uses gated urchitecture, as in LLAMA-2, though our methodology straightforwardly applied MLP architectures also.
3.4 Computational Invariance
The computational invariance theorem [Ashkboos al., 2024, Theorem 1] states that the weights and between-block activations in transformer can be transformed using an orthogonal matrix with no change to the model output. Here we sketch the main idea. If Wi, a weight matrix that appears on the left of transformer block (i.e., W gate W up in Figure 2, or W. k, W W, in Figure 5) then we can multiply on the left by an orthogonal matrix Q, and cancel out this effect by multiplying the output matrix (W down: W out) by Q This applies despite the fact that RMSNorm is applied between the two blocks, so long as no re- re-scaling happens in the RMSNorm block (and in practice, we absorb any re-scaling into adjacent weight matrices first). Conceptually, this is because RMSNorm divides the activations by their norm, and applying a rotation Q to the activations does not affect the norm. We have the commutation property
RMSNorm X) = R2MSNor(XQ'" QQ.
where we assume here that RMSNorm applied to each row of the activations X as x; + x; / This means that multiplying an output matrix by Q" T makes the linear layer output XQ T, which is normalized and then passed into the next block whose input weight matrix is now QW, and so this linear layer outputs the original activations without modification.
Method
QuaRot consists of two stages. In the first stage, the model weights are manipulated (in full precision), and additional Hadamard operations inserted into the model's forward pass. In the second stage, the weights are quantized using some existing method, and quantization operations are added to the forward pass enable on-line quantization of the activations (and caches). By default, GPTQ [Frantar et al., 2022] for quantizing weights, whilst activations are quantized on-the-fly using simple round-to-nearest scheme. Figures and show updated block diagrams for the forward pass with QuaRot modifications, including updated weight matrices, inserted blocks and the bit-width of weights and activations.
Weight Modification. We first make use of computational invariance to multiply each weight matrix by orthogonal matrix. To enable this, the linear parts of LayerNorm or RMSNorm are fused into adjacent weight matrices. Figure shows how the herdffo block of a transformer is modified by removing the scaling operation from RMSNorm (diag (<) and absorbing into the
Stage la:

---

FEN
INT4
FP16
RMSNorm
(a)
INT4
hachomard
quantize
PP16
FP16
quantize
INT4
INT4
HW aWam
YQ
XQ
INT4
Q (a/
FP16
Figure 3: QuaRot applied to MLaMa-ttyle FFN. The RMSNorm scaling (xx) has been absorbed into the weight matrices ((x) is diagonal matrix with RMSNorm parameters). The hidden state X has been rotated by Q, which is canceled out by the absorption of Q into the first two weight matrices. All weights are stored INT4, and all activations immediately before the weights are also quantized to INT4. The result of the matmul between the INT4 weights and activations on TensorCore is INT32, which we immediately cast (and scale) to FP16 which is the default precision of the model. Whilst the signal is still in FP16, we perform a single on-the-fly fly Hadamard transform before quantizing and computing a (modified) down-proj. which results rotated output YQ.
subsequent weight matrices. We select a randomized Hadamard matrix with size that matches the hidden dimension of the model and pre- post-multiply each weight matrix. In Figures and 6 this matrix is denoted Q. For example the key- projection weight matrix Wa is modified as
We < Q (ong((:)<:::
and similarly for other weight matrices. Matrices that appear on the output side of a block are post-multipled by Q.
This weight modification does not affect the output of the model (assuming sufficient precision)
as per the computational invariance theorem [Ashkboos et al., 2024]. We note that the modified weights resemble the modifications used in QulP# [Tseng et al., 2024], reducing the incoherence of the weights, though our modification does not require any additional processing at run-time. Additionally, the activation matrix passed between blocks of the transformer is also incoherence processed, becoming X + XQ. Figure I shows the result of this processing: we see that the processed activations no longer contain any outliers.
Stage 1b: Rotate FFN activations. With the above weight- veight-modiffiations in place, we have multiplied many weight matrices on one side by Hadamard matrix and the activations have been changed. It remains to improve the quantization of the activations within each block, which we achieve by inserting on-line Hadamard operations.
We first insert a Hadamard operation into the feed-forward network, before the down-projection matrix. This operation is performed in full precision, and implemented using fast kernel following Tseng et al. [2024]. This operation is implicitly reversed by fusing a Hadamard matrix into the nown-projectio matrix of the network: W down HW down- Combined with the global matrix Q, this means that the down- wwwneproecction matrix now becomes HW Q (see Figure 3).
Stage Ie: Attention Value Projection. Next, we apply additional Hadamard operation each attention block. This modification is partially on-line, and partially fused into the weight matrices as will now detail.
First, note that in the computation of attention, the W, and W Wo matrices are implicitly multiplied together within each head. To see this, note that the attention computation consists of
Y
(P Vi) (Pn, Vn W
(5)
"
Σ (x ((()))
(6)
where Pj, is a sequence-length sized square matrix computed by softmaxing keys and values, and VA (h) is the value matrix for one head. This presents opportunity to perform additional
5

---

processing on W and W out using Hadamard matrix Hd, which matches the dimension of each head:
W
H de.
w out Ha, dr W (h) out
Substituting these modifications into equation (6), we see that the computed result of attention remains unchanged. Since the weights for each head are concatenated in the weight representation, we can equivalently perform a single Kronecker structured multiplication:
(8) This transformation has now been applied head-wing wead-wise the weight matrices, and results in computed activations (emitted by the block multi-head attention) rotated head-wise also. To complete a "full" Hadamard operation on the attention-activations, activations, sharing the transform across heads, we make use of the identity
We (I @ Hd,),
W out < Hd, Wout. out.
H xd. (I @ HA, (H,,, I)
which holds when the number of heads n/, and the dimension of each head d, are both powers of 2. Since we have already applied (I @ Hd, ) to both W. and W it remains apply (H&, @ I) to W Wout, which results in a complete transformation of W out f HW out. and to insert a block into the forward pass that computes Z <- Z(H,,, & I) where Z is the attention activation. This block is denoted Hadamard heads in Figure 6 and can be computed efficiently using a reshape to deal with the Kronecker structure, and a Walsh-Hadamarr transform on the reshaped data.
Stage 1d: Key Rotation. Using the method above, we can successfully quantize the value vectors. However, key vectors in the attention module are also known to suffer from outliers [Hooper et al., 2024, Liu et al., 2024]. Similar to above, we can use Hadamard rotation to alleviate this issue, allowing us to have a fully quantized KV cache. First note that the attention scores Pl,...,Po PA are computed as:
Q + Pos XW 9) = concat [Pos( Q1). Pos Quan Pos
(10)
K= + Pos( X W k) = concat(Pos( (K) K),,. ..... Pos (Kn,))
(11)
Py + Softmax( a Pos(Qr) Pos(
M),
(12)
where a is the Softmax scale usually set viz. M is the attention mask (e.g., causal), and Pos denotes the positional embedding. Previously, positional embedding was only added before the first layer to the input, in which case Pos is an identity function. However, recent methods such as RoPE [Su et al., 2021] add position information directly to the key and query vectors.
We can now observe the same interaction between Q and K as we observed between W. and W out- However, the existence of Pos prevents us from directly fusing the Hadamard matrix into W, and W.. Therefore, we use online head-wise Hadamard rotation to rotate both the queries and keys. As a result, the computation of query and key matrices is altered as follows:
Q <- Pos(XW,))) (I &9 Ha,) = concat Q1 Ha,, . Qm,, H K < X & Ha, = concat[Post K2 H d. Pos(Ko K,,,, Ha,
(13)
(14)
Since both queries and keys are rotated, the final attention scores PI,...,BU Pn remain unchanged. We note that an alternative to the above process is caching the keys before applying the positional encoding. This approach (called Pre-RoPE Caching [Hooper et al., 2024J) needs the inverse rotation to be applied online before applying the positional encoding but removes the need to rotate the query vector. It also adds the overhead of rotating the keys and values for every query. Given that at the time of decoding there is single query vector and many cached key vectors, we use Post-RoPE caching. This helps us to apply a Hadamard transformation on a single token at each decoding step.
Overall, our modifications to the forward pass, including the insertion of special Hadamard blocks and adjustments to the weights do not change the forward pass of the model. The effect is that the activations between blocks have been multiplied by a Hadamard matrix, and the activations within blocks are processed on-line using Hadamard transforms in a way that undone by corresponding weight matrix modifications. We are now ready to quantize the weights and activations.
Stage 2a: Weight Quantization. We apply GPTQ [Frantar et al., 2022] to quantize the weights of the network. We note that after the above forward-pass modifications, any quantization method could be applied. In subsequent sections, we show that a simple round-to-nearest (RTN) scheme can be applied instead of GPTQ, at the cost of some accuracy.

---

Stage 2b: Online Quantization Operations. With the weights quantized, we are ready to apply operations to the forward pass that quantize the activations. Following PyTorch implementation, we leave the computation of RMSNorm (without scaling) in FP32. We quantize the input of the linear layers using symmetric per-token (rows of the input matrix). During symmetric quantization, the row scales are computed by dividing the maximum absolute value of each token by 7 (largest representable number in INT4). We then divide each row to its corresponding scale and round the result to its nearest integer. The dequantization is also done by casting the INT32 output of GEMM into FP16, multiply the corresponding scale for the row (from input scales) and column (from weight scales).
Stage 2c: Quantized Attention. Attention is significantly memory bound for longer sequences and larger batch sizes. Having rotated both keys and values, we can successfully quantize the cache into low bit-width. This reduces the number of IO operations needed. We keep the queries in FPI6 and use online softmax calculation similar to Flash Attention [Dao et al., 2022]. After a segment of the KV vectors are loaded from the memory, we dequantize and compute the dot product in FP
5 Experimental Validation
Setup. We implement QuaRot using Hugging Face [Wolf al., 2019] on top of the Py framework [Paszke et al., 2019]. To quantize the inputs, we use per-token symmetric quantization (a single scale for every row) with a constant clipping ratio of 0.9 in all our experiments. We quantize the KV caches using asymmetric quantization with a group size 128 with a constant clipping ratio of 0.95. For weight quantization, we use round-to-nearest (RTN) and GPTQ [Frantar et al., 2022] with per-column (also known as per-channel) symmetric quantization, where we extract the clipping ratio using linear search over the squared error. We use 128 samples from WikiText-2 [Merity et al., 2016] training set with 2048 sequence length as the calibration set during GPTQ quantization. On a single NVIDIA A 100 GPU, modifying LLAMA2-70B with QuaRot takes minutes and quantizing the model with GPTQ takes a further 2 hours. We present LLAMA-3 results in Appendix A.8.
Models, Tasks, and GPUs. We evaluate QuaRot on the LLAMA-2 family [Touvron al., 2023] on both language generation and zero-shot tasks. We implement our low-level CUDA kernel to perform 4-bit matrix-multiplication using the CUTLASS [NVIDIA, 2023] library. We the FlashInfeer (Ye, 2023] library for implementing our KV cache quantization. As we target consumer-type GPUs, we evaluate all the performance experiments on NVIDIA RTX 3090 GPUs.
5.1 Accuracy Results
Language Generation Tasks. First, we evaluate the accuracy of QuaRot on the language generation task. Table I shows the perplexity of LLAMA-2 models on WikiText-2 when we quantize the weights using GPTQ. We compare against 4-bit SmoothQuant [Xiao et al., 2023] and OmniQuant [Shao et al., 2023]. We also include the QUIK [Ashkboos et al., 2023] results when they keep all the layers (including down- ) in 4 bits. QuaRot outperforms all previous work with at most 0.63 perplexity loss (0.47 on LLAMA2-70B model) without any re-training (as in OmniQuant) nor higher precision outlier features and asymmetric quantization (as in QUIK). We also apply group-wiss quantization to compare against Atom [Zhao et al., 2023] on the same number of groups for weight and activations. In this setting, QuaRot doesn't 't need to keep any higher precision features and related operations (like re-ordering). QuaRot outperforms Atom with 0. perplexity points in the 7B model. On the 13B model, we get the same perplexity number as Atom.
Zero-Shot Tasks. Next, we focus on evaluating QuaRot on six important zero-shot tasks: PIQA [Bisk et al., 2020], WinoGrande [Sakaguchi et al., 2021], HellaSwag [Zellers et al., 2019]. LAMBADA (OpenAI) [Radford et al., 2019], and Are (Easy and Challenge) [Clark et al., 2018]. We use the LM Evaluation Harness [Gao et al.. 2021] with default parameters for our experiments. Table 2 shows the accuracy of our scheme on the above tasks as well as the average score. On LLAMA-2 family, QuaRot preserves the accuracy with at most 4. 18% average score loss (1.09% for 70B model).
5.2 Performance Analysis
We implement QuaRot using CUDA/12.1 on top of PyTorch and use CUTLASS for performing INT-4 matrix multiplication on TensorCore (where the results will be saved in an INT32 accumulator). In this section, we evaluate the performance of our kernels for both prefill and decoding steps on NVIDIA RTX 3090 GPU, We provide all our experiments single transformer block as the whole
7

---

Table WikiText-2 perplexity results on 4-bit quantization of LLAMA-2 models with 2048 sequence length. We extract the results for SmoothQuant and OmniQuant results of [Shao et al., 2023]. 128G shows the group-wise quantization with group size 128.Her, we quantize all weights, activations, and caches in 4-bits in QuaRot.
Weight Quantization
#Outlier Features
LLAMA-2
Method
7B
13B
70B
Baseline
5.47
4.88
3.32
SmoothQuant
RTN
83.12
35.88
OmniQuant
14.26
12.30
QUIK-4B
GPTQ
256
8.87
7.78
QuaRot
GPTQ
0
6.10
5.40
3.79
Atom-128C QuaRot 128G
128
6.03
5.26
GPTQ-122 128G
5.93
5.26
3.61
Table 2: Zero-shot accuracy LLAMA-2 models with 4-bit (A4W4KV4) QuaRot on PIQA (PQ), WinoGrande (WG), HellaSwag (HS), Are-Easy (A-e), Are- rre-Challenge (A-c), and LAMBADA (LA).
Model
Method
PQ
WG
HS
A-e
A-c
LA
Avg.
FP16
79.11
69.06
75.99
74.58
46.25
73.90
LLAMA2-7B
QuaRot
76.77
63.77
72.16
69.87
40.87
70.39
65.64
FP16
80.47
72.22
79.39
77.48
49.23
76.75
72.59
LLAMA2-13B
QuaRot
78.89
70.24
76.37
72.98
46.59
73.67
69.79
FP16
82.70
77.98
83.84
80.98
57.34
79.58
77.07
LLAMA2-70B
QuaRot
82.43
76.24
81.82
80.43
56.23
78.73
75.98
Time-to-fisstookkn (prafill) Sequence Length
7B
7BB
Memory Saving Fastor
3.8%x
3,80s
3.7ts
3.72x
3.7%.
3.30x
2,00s
16
64
256
512
1024
2048
4096
Batch Size
Sequence Length
Figure 4: Performance of the QuaRot kernel on a single transformer block of LLAMA-2 models using NVIDIA RTX 3090 GPU. Left: For the speedup results, we evaluate using sequence length 2048 with different batch sizes. Right: Peak memory saving during decoding of 50 tokens with different prefill sequence lengths using batch size 16.
model does not fit on our GPU cluster for large batch sizes. We provide more performance analysis of our kernels (as well as complete results) Appendix A.10.
Prefill Stage Performance Increases. For the compute-bount prefill stage, we present the speedups of using QuaRot 2048 sequence length with different batch sizes in Figure Left. On LLAMA2-7B model, we get 1.97x-2. 16x speedup over the FP16 implementation using our QuaRot kernel. The speedup increases with batch sizes the computation will become bottleneck in larger batch sizes. on LLAMAA2700 model, we get up to 3.33x speedup. Note that our performance results could be improved by optimizing kemels (e.g., fusing the quantization operations into the MatMul).
Decoding Stages Memory Saving. Finally, we evaluate the memory improvement which is the main bottleneck of the decoding stage. Figure Right shows the peak memory saving on LLAMA-2 models. We provide results for LLAMA2-7B and LLAMA2-70B models. In both models, we get at least 3.6 3.63x peak memory saving compared to FP16 case during the decoding stage. Note that the KV cache is larger in LLAMA2-78 model as the LLAMA2-708 uses grouped-query attention [Ainslie et al., 2023]. In the LLAMA2-7B model, the memory saving increases with the sequence length, resulting up to 3.75x memory saving. on LLAMA2-70B model, we get 3.89x savings in almost all the cases. We expect these values to be larger for the whole model (instead of just the single layer

---

Table WikiText-2 Perplexity and zero-shot aceuracy QuaRot on the LLAMA-2 family using and 8-bits with Gound-to-Nearee (RTN) weights and activation quantization. For zero-shot tasks, we use PIQA (PQ), WinoGrande (WG), HellaSwag (HS), Are-Easy (A-e), Are-Challenge (A-c), and
LAMBADA (LA). We quantize all weights, activations, and caches.
Model
Method
Precision
PPL
PQ
WG 1
HS
A-e
A-c
LA 1
Avg. t
Baseline
FP16
5.47
79.11
69.06
75.99
74.58
46.25
73.90
69.82
7B
INT4
8.37
72.09
60.69
65.40
58.88
35.24
57.27
58.26
QuaRot-RTN
INT8
5.50
78.94
68.67
75.80
74.79
45.39
74.33
69.65
Baseline
FP16
3.32
82.70
77.98
83.84
80.98
57.34
79.58
77.07
70B
INT4
4.14
80.69
75.14
79.63
77.57
51.71
77.02
73.63
QuaRot-RIN
INT8
3.33
82.97
77.98
83.67
80.77
58.11
79.53
77.17
Table WikiText-2 perplexity 4-bit QuaRot with various group-sizes on LLAMA-2 models. We GPTQ during the weight quantization. In all cases, keep the KV cache group-size to 128 (same as the head dimension). 128G shows the group-wise quantization with 128 group size.
LLAMA-2
Method
7B
13B
70B
Baseline
5.47
4.88
3.32
QuaRot
6.10
5.40
3.79
QuaRot-256G
5.98
5.28
3.63
QuaRot- 128G
5.93
5.26
3.61
QuaRot-64G
5.88
5.25
3.58
here) since as the number of layers increases the effect of constant size objects memory becomes much less significant.
5.3 Ablation Studies
To evaluate different aspects of QuaRot, we evaluate the use of Round-to-Nearest Weight Quanti- zation, Group-wise Quantization (with different group sizes), and KV cache Quantization with different bit- combinations (Appendix A.3). In addition, we investigate the role of applying Hadamard transformation on the Weight-only Quantization schemes (Appendix A.4) as well as using Random Orthogonal Matrices (Appendix A.5) instead of Hadamard matrices. Finally, we evaluate the accuracy our quantized models when we apply FP16 Hadamard Transformation (Appendix A.7).
Round-to-Nearest Weight Quantization. GPTQ is our default choice for weight quantization in QuaRot. Here, we study the role of quantizing the weights using Round-to-Nearest (RTN). Table shows that applying RTN weight quantization fully maintains the FP16 model accuracy in bits. We note that RTN does not need any calibration set or hyper-parameter during the quantization. Comparing Table 3 and
conclude that in 4 bits, the gap between QuaRot-RTN and QuaRot- GPTQ decreases when the model size is increased (2.27 on LLAMA2-78 and 0.34 on LLAMA2-70B showing that GPTQ better option in smaller models. For more detailed results see Appendix A.6.
Group-wise Quantization. Table shows the accuracy of applying QuaRot with various group-sizes for the activations and weights. The results show clear trade-off between the accuracy and the group-sizes: smaller group-sizes give better accuracy (but require more bits to store scales for each group and more complex matrix multiplication kernels).
6 Conclusion
We introduce QuaRot: a method which uses Hadamard matrices to eliminate outliers in the activations and KV cache of pre-trained L.MMs, enabling end-to-end 4-bit quantization for the first time (to the best of our knowledge). Quantizing LLAMA2-700 70B to 4 bits with QuaRot maintains 99% of the

---

downstream task performance of the FP16 baseline, with 2.16x speedup on RTX 3090 GPUs during the prefill stage (and up to 3.39x memory saving during the decoding stage). Quantizing all LLAMA-2 models to 6 and 8 bits is lossless.
Opportunities to build QuaRot include quantizing the residuals and extending the method to mixture--o-exxper architectures. terms of hardware, end-to-end INT4 inference with QuaRot could be exploited give similar speedups as that of the recently announced NVIDIA B200 GPU architecture, while being much cheaper to implement compared to the floating point (FP4) format.
References
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Amit Garg, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell Hewett, Jamie Huynh, Mojan Javaheripi. Xin Jin, Piero Kauffiman, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang, Weishung Liu, Eric Lin, Zeqi Lin, Piyush Madan, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Xia Song, Masahiro Tanaka, Xin Wang, Rachel Ward, Guanhua Wang. Philipp Witte, Michael Wyatt, Can Xu, Jiahang Xu, Sonali Yadav, Fan Yang. Ziyi Yang. Donghan Yu, Chengruidong Zhang, Cyril Zhang. Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally your phone, 2024.
Joshua Ainslie, James Lee- Thorp. Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized qulti-query transformer models from multi-head checkpoints. arXiv preprint arXivv2305. 13245, 2023.
Saleh Ashkboos, Iia Markov, Elias Frantar, Tingxuan Zhong, Xincheng Wang, Jie Ren, Torsten Hoefler, and Dan Alistarh. Towards end-to-end 4-bit inference on generative large language models. arXiv preprint arXiv:2310.0923. 2023.
Saleh Ashkboos, Maximilian Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint 4rX1v:2401.15024, 2024.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourthh AAAI Conference on Artificial Intelligence, 2020.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning chal- lenge. ArXiv, abs/1 0as/1803.0545,, 2018. URI. https: //api. semanticscholar wrg//orrpsssaaoooooooooooo 3922816.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with 10- D-waerensss. In Advances in Neural Information Processing Systems, 2022.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 Q) 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35: 30318 -03322, 2022.
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashk- boos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized represen- tation for near-lossless Ilm weight compression. arXiv preprint 2rXiv:2306.03077 2023,
10

---

Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression large language models via additive quantization. arXiv preprint arXiv:2401.06//8 18, 2024.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers. arXiv preprini urXiv:2210.1732.. 2022.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. framework for few-shot language model evaluation. Version vO. 0.1. Sept, 2021.
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kyquant: Towards 10 million context length Ilm inference with kv cache quantization. arXiv preprini arXiv:2401.18077 2024.
Lin, Jiaming Tang. Haotian Tang, Shang Yang. Xingyu Dang, and Song Han. Awq: Activation- aware weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.00978 2023.
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu, Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750, 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
NVIDIA. Nvidia cutlass library, 2023. URL htt.::ttt//:tthhieeeee httpt/ttbb..omm/iiDI/I//caaassss http///////////////oo//////o////
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et PyTorch: An imperative style, nigh-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Ommidirectionally calibrated quantization for large language models. arXiv preprint 2rXiv:2308.13133 2023.
Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, lon Stoica, and Ce Zhang. Flexgen: High- tigh-throughhut generative inference of large language models with single gpu. In International Conference on Machine Learning, pages 31094-31116. PMLR, 2023.
Neil J A Sloane. A library of hadamard matrices, 2024. URL http://nnelsslane.ccom com/ hadamard/.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. CoRR, abs/2104.09864, 2021. URL https://arxiv..or org/ abs/ 2104.09864
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra,
11

---

Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quipi: Even better Ilm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.0439% 2024.
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang. Ruihao Gong, Shanghang Zhang, Qi Zhang. Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. Advances in Neural Information Processing Systems, 35:17402-17414, 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et Huggingface's transformers: State-of-the-aar natural language processing. arXiv preprini arXiv: 2019.
Haocheng Xi, Changhao Li, Jianfei Chen, and Jun Zhu. Training transformers with 4-bit integers. Advances in Neural Information Processing Systems, 36:491 19949144-44168,, 2023.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087-38099, PMLR, 2023.
Zihao Ye. FlashInfer: Kernel Library for LLM Serving. https: //ttth////ttub.... / com/fllahhinfer-ai/ flashinfer, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv: 1905.0780. 2019..
Yilong Zhao, Chien- Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate Ilm serving, arXiv preprint arXiv:2310.191 2023.
12

---

A Appendix
A.1 QuaRot on the Attention Module
Figure shows the original attention module in large language models with RoPE. The input of the attention module is already rotated using the randomized Hadamard matrix Q (see Section 4) and in the first step, we fuse the inverse of such matrices into the input linear layers of the attention. In the next step, fuse the exact Hadamard matrices on each block of the columns (proportional to each head) on the V_ V_projection layer make sure that the Values will rotated at the output of that layer. In the next step, we apply exact Hadamard transformations on the Keys and Queries and quantize the KV after RoPE operation (note that Keys and Queries Hadmard transformations will be canceled during the attention operation). Finally, apply another Hadamard transformation between heads before Out _ut__rooee projection layer and fuse the inverse into the weights. Figure 6 shows the result of applying QuaRot on the attention module.
Attention
W
RoPE
Multi-head Attention
RMSNorn
X
diag(a:)
Wi
Wowr
Y
KV-cache
W.
Figure 5: Flow diagram of a self-attention block as used in most LMs, including the pre-positioned R.MNNrm. Solid arrows represent flow during training, prefill and inference of each token. Dashed arrows show access to and from the KV cache, used at generation-time. The RoPE block computes relative positional embeddings.
Attention
INT4
Q (a)
FP16
RoPE
Multi-head
Attention
RMSNom
INT4
FP16
quantize
INT4
Q (a)
FP16
hadamard
hadamard heads
quantize
INT4
FP16
dequantize
NT4
HW Q
YQ
INT4
hadamard
quantize
KV-cache
FP16
Q (a) W.
INT4
Figure 6: QuaRot applied an attention component. The RMSNorm scaling a absorbed into the input weight matrices, and the hidden state has been rotated by Q in the same way as for the FFN block (see previous figure). Colored labels show the bit-width of each flow, and dashed lines show the flow to/from the KV cache.
A.2 Clipping Ratio Ablation
We use the clipping ratio for both weights and activations during the quantization. During the weight quantization, we apply linear search over the MSE error to extract the best clipping ratio for each
13

---

column of the weight matrix. However, this is not possible as we quantize the inputs on the fly during the inference and we need use constant clipping ratio for such quantization. We conclude that using 0.95 and 0.9 are suitable during asymmetric (KV cache) and symmetrie (inputs) quantization which matches the finding from [Zhao et al., 2023].
Table 5:
Wiki'Text perplexity of LLAMA2-7B with different clipping ratio. To study the effect of various clipping ratios, keep the rest of the model in full precision.
1.0
0.95
0.9
0.85
Input Quantization 5.938
5.910
5.828
5.850
KV Cache Quantization
5.513
5,510
5.517
5.532
A.3 KV Cache Quantization Ablation
We keep the rest of the model (including weights and activations) in high precision and apply our group-wise asymmetric quantization (with group-size 128) with various precision to keys and values. Table 6 shows the results of using various precision during KV cache quantization. The results show negligible (at most 0.21) perplexity degradation up to 3-bit KV cache (0.07 for LLAMA2-705 model). In addition, by comparing the 3 and 4-bit quantization, we can see that compared to the values, keys are more sensitive to quantization as keeping the keys in 4-bits and values in 3-bits has 0.03 perplexity loss (0.18 for 3-bit keys and 4-bit values) on the LLAMA2-7B model. This matches the previous study on KV cache quantization [Hooper et al., 2024, Liu et al., 2024]. The results show that using 3-bit V-cchhes results better accuracy (5.68 on LLAMA2-7B model) compared to keeping the keys in 4-bits and quantizing the values using 2-bits (with 5.75 perplexity on LLAMA2-78 7B model).
Table 6: 2 perplexity with various KV cache precision using QuaRot.
K bits
V bits
LLAMA-2
7B
13B
70B
16
16
5.47
4.88
3.32
4
4
5.51
4.91
3.33
4
5.54
4.93
3.35
4
5.09
3.43
3
4
5.65
5.01
3.38
3
3
5.68
5.02
3.39
3
5.93
5.21
3.48
2
8.06
6.42
3.89
2
3
8.18
6.50
3.92
2
2
9.23
7.07
4.13
A.4 Weight-only Quantization Ablation
QuaRot improves the quality of quantized models by removing the outlier features during the Hadamard transformations. As we fuse the Hadamard matrices into the weights, we study the role of these transformations for weight-only quantization (we keep the rest the data-types in FP16). Table 7 shows the WikiText-2 perplexity results with asymmetric quantization. Using GPTQ quantization, QuaRot improves the perplexity by up to 2.65 in bits. In addition, applying QuaRot improves quality more in lower precision (2-3 bits) in all models. QuaRot also improves the RTN quantization up to 0.24 perplexity points. GPTQ still has a lower perplexity in 2-3 bits. However, applying QuaRot improves the quality of GPTQ in 2 bits to non-trin con-trivial value (5.6 the LLAMA2- 70B model).
14

---

Table Weight-only quantization results on WikiText-2 LLAMA-2 models. We use asymmetric per-column quantization and keep the inputs and KV cache in FPI6. We show the perplexity results >10 by Inf. We show the failed GPTQ experiments using NaN.
LLAMA-2
Method
7B
13B
70B
Baseline
5.47
4.88
3.32
A16W4
A16W3
A16W2
A16W4
A16W3
A16W2
A16W4
A16W3
A16W2
RTN
6.99
Inf
Inf
6.32
Inf
Inf
4.45
42.11
Inf
GPTQ
8.25
NaN
NaN
5.65
9.51
Inf
3.87
5.91
25.30
QuaRot-RIT
6.76
Inf
Inf
5.48
48.89
Inf
3.66
5.25
Inf
QuaRot-GPTQ
5.60
6.09
22.07
5.00
5.37
10.41
3.41
3.72
5.60
A.5 Random Orthogonal Matrices Ablation
QuaRot fuses Hadamard transformations into weight matrices to eliminate outliers, However, due to the computational invariance property in LLMs, any orthogonal matrix can be fused the model and we only need to apply an online 1 Hadamard transformations in each layer (see Section 4). Here, we study the use random orthogonal matrices in QuaRot. We start with a uniformly random matrix and apply QR decomposition to make it orthogonal before fusing it into the weights.
Table 8: WikiText-2 perplexity of 4-bit QuaRot on LLAMA-2 models with different orthogonal matrices.
LLAMA-2
Method
7B
13B
70B
Baseline
5.47
4.88
3.32
QuaRot (Random)
7.45
5.84
4.07
QuaRot (Hadamard)
6.10
5.40
3.79
Table 8 shows the results of applying random orthogonal matrices on LLAMA-2 models. Random orthogonal matrices not good random Hadamard transformations and we have up 1.35 perplexity gap on LLAMA2-7B However, as the model size increases, the gap decreases, resulting in perplexity change of 0.28 in the LLAMA2-70B model. Note that using the above matrices does not change the computation as we still use fast Hadamard kernel for the down-projection and out-projection layers.
A.6 Round-to-Nearest Weight Quantization: Detailed Results
Table 9 shows the detailed results of QuaRot with GPTQ and round-to-nearest (RTN) weight quanti- zation for both 6 and 8 bits various tasks for LLAMA-2 models.
A.7 FP16 Hadamard Transformation Ablation
We use FP32 online Hadamard transformation across all our experiments. Table 10 shows the results of using FP16 Hadamard transformation during the inference (for down-prow fown-projection and out-projection layers). On 1- model, the results show <0.1 perplexity change on WikiText-2 and <0.6% averaged accuracy change on the zero-shot tasks, which we consider as noise. On LLAMA2-133 model, different Hadamard precisions have the same perplexities with 0.07% difference in the averaged zero-shot results. We conclude that the model will be changed using different Hadamard precision.
15

---

Table WikiText-2 Perplexity and zero-shot accuracy of QuaRot on the LLAMA-2 family using 4, and 8-bits with GPTQ and RTN weight quantization and RTN activation quantization. For zero-shot tasks, we use PIQA (PQ), WinoGrande (WG), HellaSwag (HS), Are-Easy (A-e), Are-Challenge (A-c), and LAMBADA (he).The Precision column shows the bitwidth for all inputs, weights, and
KV-caches.
Model
Method
Precision
PPL
PQ
WG 1
HS
A-e
A-c t
LA 1
Avg. f
Baseline
FP16
5.47
79.11
69.06
75.99
74.58
46.25
73.90
69.82
INT4
8.37
72.09
60.69
65.40
58.88
35.24
57.27
58,26
QuaRot-RTN
INT6
5.56
78.73
67.80
75.92
74.16
46.08
73.86
69.42
7B
INT8
5.50
78.94
68.67
75.80
74.79
45.39
74.33
69.65
INT4
6.10
76.77
63.77
72.16
69.87
40.87
70.39
65.64
QuaRot-GPTQ
INT6
5.52
78.45
69.46
75.60
74.45
46.50
74.19
69.77
INT8
5.50
78.94
68.90
75.79
74.66
46.16
74.44
69.81
Baseline
FP16
4.88
80.47
72.22
79.39
77.48
49.23
76.75
72.59
INT4
6.09
77.37
67.32
73.11
70.83
43.69
70.66
67.16
QuaRot-RTN
INT6
4,95
79.65
72.22
79.10
77.27
50.34
76.75
72.56
13B
INT8
4.90
80.52
71.59
79.38
77.31
49.32
76.63
72.46
INT4
5.40
78.89
70.24
76.37
72.98
46.59
73.67
69.79
QuaRot-GPTQ
INT6
4.92
79.98
72.69
79.17
77.78
49.74
76.27
72.60
INT8
4.90
80.36
71.98
79.38
77.31
49.15
76.79
72.49
Baseline
FP16
3,32
82.70
77.98
83.84
80.98
57.34
79.58
77.07
INT4
4.14
80.69
75.14
79.63
77.57
51.71
77.02
73.63
QuaRot-RTN
INT6
3.36
83.24
77.90
83.47
80.93
58.28
79.41
77.21
70B
INT8
3.33
82.97
77.98
83.67
80.77
58.11
79.53
77.17
INT4
3.79
82.43
76.24
81.82
80.43
56.23
78.73
75.98
QuaRot-GPTQ
INT6
3.35
82.
77.66
83.63
80.89
57.08
79.70
77.02
INT8
3.33
83.13
78.06
83.72
80.85
58.19
79.72
77.28
Table 10: Ablation on the precision of online Hadamard transformations for QuaRot. WikiText-2 perplexity well zero-shot tasks, explained in Section 5.3.
We use
Hadamard Precision
Model
Method
PPL
PQ f
WG f
HS f
A-e
A-e f
LA
Avg. f
Baseline
5.47
79.11
69.06
75.99
74.58
46.25
73.90
69.82
7B
FP32
6.10
76.77
63.77
72.16
69.87
40.87
70.39
65.64
QuaRot
FP16
6.08
76.99
66.46
72.59
69.07
41.21
70.59
66.21
Baseline
4.88
80.47
72.22
79.39
77.48
49.23
76.75
72.59
13B
FP32
5,40
78.89
70.24
76.37
72.98
46.59
73.67
69.79
QuaRot
FP16
5.40
77.69
70.09
75.75
73.95
47.61
73.22
69.72
A.8 LLAMA-3 Results
In this section, we show the accuracy of applying QuaRot for quantizing the LLAMA3-8B and LLAMA3- models. Table shows the WikiText-2 perplexity of quantizing the LLAMA-3 models with QuaRot using 4-bit quantization. Compared to Table 1, we conclude that LLAMA-3 is more sensitive quantization as we can see higher gap between the quantized and FP16 models. Table 12 shows the accuracy results of those models on zero-shot tasks.
16

---

Table 11: WikiText-2 perplexity results on 4-bit quantization of LLAMA-3 models with 2048 sequence length. 128G shows the group-wise quantization with group size 128.
Weight Quantization
#Outlier Features
LLAMA-3
Method
8B
70B
Baseline
6.14
2.86
QuaRot
GPTQ
8.16
6.66
QuaRot-12833
GPTQ-128G
7.36
5.51
Table 12: Zero-shot accuracy of LLAMA-3 models with 4-bit QuaRot on PIQA (PQ), WinoGrande (WG), HellaSwag (HS), Arc-Easy (A-e), Arc-Challenge (A-c), and LAMBADA (LA).
Model
Method
PQ
WG
HS
A-e
A-c
LA
Avg.
FP16
80.74
72.77
79.06
77.82
53.33
75.63
73.22
LLAMA3-8B
QuaRot
75.14
65.82
72.94
68.01
43.34
65.81
65.18
FPI6
84.66
80.51
84.89
85.86
64.25
79.47
79.94
LLAMA3-70B
QuaRot
78.07
69.30
77.33
73.44
47.53
69.57
69.21
A.9 Phi-3-mini-4k-intreee Results
In this section, we show the accuracy of applying QuaRot for quantizing the Phi-3-mini-4k 4k-iittruct model [Abdin et al., 2024). Table 13 shows the accuracy results of the model in terms of perplexity and on zero-shot tasks.
Table 13: WikiText-2 Perplexity and zero-shot accuracy of QuaRot on the Phi-3-mini-4k-insse model (revision = ff07dc01) using 4, and 8-bits with GPTQ and RTN weight quantization and RTN activation quantization. For zero-shot tasks, we use PIQA (PQ), WinoGrande (WG), HellaSwag (HS),
Arc-Easy (A-e), Are-Challenge (A-c), and LAMBADA (LA).
Model
Method
Precision
PPL |
PQ
WG f
HS 1
A-e
A-e
LA
Avg. 1
Baseline
FP16
6.35
80.47
73.72
78.45
80.13
57.51
68.37
73.11
INT4
11.69
68.39
58.64
60.60
65.87
39.25
43.99
56.
QuaRot-RTN
INT6
6.78
79.54
73.01
77.46
79.21
55.12
67.53
71.98
Phi-3-mini
INT8
6.58
79.71
74.11
78.63
80.47
56.66
68.56
73.02
INT4
7.85
75.35
67.88
72.95
72.98
48.12
60.78
66.34
QuaRot-GPTQ
INT6
6.63
79.54
72.69
78.50
79.42
56.74
68.85
72.67
INT8
6.58
80.25
74.19
78.54
80.35
57.08
68.64
73.18
A.10 Performance Analysis
We implement the attention mechanism using three routines: Init: During the prefill stage, this routine initializes the cache from all the key and value vectors the prefill. The attention output during prefill is computed directly using Flash Attention [Dao et al., 2022] since we already have access to dequantized keys and values. 2) Append: During decoding, this routine is called first to quantize the current keys and values and append them to the cache. 3) Decode: Finally, this routine is called during decoding with the current query vector. The routine computes the attention output using quantized implementation of flash attention which can load the quantized cache and compute the final value vector.
4-bit Linear and Attention Layers. We benchmark our 4-bit linear layer which involves 4-bit matrix multiplication. For given input of FP16, the layer optionally computes the Hadamard operation, then calls quantization kernel to quantize and save the input in a sub-byte format. the next step, the quantized weights and input are passed to the CUTLASS 4-bit GEMM kernel. Finally, the output dequantized and cast back to FP16. Figure shows the speedup of our 4-bit layer for different layer sizes where the layer sizes match the FFN linear layer sizes in LLAMA-2 models.
17

---

FP16
12.45
INT4
10
IIITT4+ Had.
Runtime (ms)
5
3.98
2.57
2.88 2.91
1.06 1.14
0.75 0.8
0
7B
13B
70B
Figure 7: Performance of 16-bit and 4-bit linear layer for 2048 sequence lengths with and without online Hadamard transformation on a NVIDIA RTX 3090 GPU, averaged over 1000 runs. The matrix sizes correspond the linear layer sizes in LLAMA-2 FFN blocks (i.e. W Woww). Here the batch size is 1, but the performance ratio holds for larger batches (see Table 14).
Our 4-bit linear layer gets 3.2x speedup relative to FP16 in the LLAMA2-7B model, and 4.3x on the LLAMA2-70B model. These numbers for batch size of 1, we find that scaling is approximately linear with batch size: more results in Table 14. We include the runtime with and without Hadamard operations, W.. up and W gate do not require Hadamard transforms, whilst W down does. We see that the Hadamard transform adds very little overhead to the forward pass at most 7% overhead.
We also compare the speed of performing append and decode routines for a single token given cache of size 2047. This is equivalent the cost of decoding the 2048-th token in a sequence. The comparison between the speed of FP16 and INT4 for different batch sizes and layer sizes is reported in Table 15. For the layer size used LLAMA2-7B. our 4-bit implementation gets up to 1.72x improvement in speed for the larger batch sizes (e.g. from 16 onwards). The 4-bit cache is slower than FP16 for smaller batch sizes (e.g. up to 8). Note that this intuitive as the main benefit of the 4-bit cache is reducing the I/O cost. A speed up is only visible if this reduction is more significant than the quantization overhead which happens for either larger batch sizes or longer sequences.
Table shows the results of benchmarking our 4-bit linear layer. The layer sizes are extracted based the linear layer sizes in LLAMA-2 models (for out- projerreetionn and down-projections). We apply both FP16 and FP32 Hadamard transformations and show the runtime on NVIDIA RTX GPU using 2048 sequence lengths. Table 15 shows the results of decoding single token in the attention layer when we apply KV-cache quantization. We extract the size of the attention layer based on the LLAMA-2 models.
18

---

Layer Size
Batch Size
FP16
INT4
INT4 + FP32 Had
INT4 FP16 Had
1
1.043
0.370
0.409
0,403
2
1.902
0.696
0.790
0.789
4
3.715
1.361
1,522
1.529
4096x4096
8
7.200
2.675
2.999
3.011
16
14.508
5.357
5,973
5.976
32
29.029
10.641
11,900
11.911
1.418
0.464
0.552
0.547
2
2.918
0.937
1.100
1.097
4
5.852
1.888
2.206
2,207
5120x5120
8
11.465
3.809
4.428
4.422
16
22.807
7.547
8.755
8.759
32
45.312
15.019
17.417
17.440
1
3.696
0.997
1.084
1.083
2
7.191
1.944
2.099
2.099
4
14.236
3.918
4.208
4.207
8192x8192
8
28.508
7.944
8.460
8.415
16
57.814
15.793
16.859
16.871
32
115.462
31.693
33.780
33.791
1
2.569
0.749
0.798
0.801
2
5.027
1.478
1.555
1.558
4
9.752
2.990
3.140
3.144
11008x4096
8
19.696
6.031
6,296
6.306
16
38.883
11.978
12.503
12.527
32
78.320
23.874
24.935
24.974
1
3,983
1,063
1.142
1.139
2
7.869
2.148
2.291
2,293
4
15.410
4.340
4.616
4.614
13824x5120
8
30.761
8.719
9.231
9,240
16
61.203
17.318
18.345
18.343
32
122.926
34.816
36.953
36.940
1
12.450
2.881
2.911
2.911
2
25.391
5.828
5.892
5.896
4
50.742
11.938
11.947
11.976
28672x8192
8
101.290
24.186
24.202
24.216
16
202.909
48.238
48.325
48.356
32
406.344
96.761
97.044
96.892
Table 14: Performance of 4-bit linear layer for 2048 sequence lengths with and without online
Hadamard transformation NVIDIA RTX 3090 GPU. The matrix sizes correspond to the linear layer sizes in LLAMA-2 models. We averaged over 100 runs and report the numbers milliseconds.
19

---

head num head dim
Batch Size
FP16
INT4
INT4 FP32 Had
INT4 + FP16 Had
0.713
1.033
1.163
1.117
0.723
1.035
1.168
1.122
32x128
0.781
1.033
1.168
1.118
8
0.984
1.042
1,173
1.126
16
1.348
1.018
1.153
1.102
32
2.098
1.168
1.247
1.216
0.712
1.026
1,157
1.106
0.726
1.035
1.173
1.121
40x128
0.831
1.038
1,166
1.115
8
1.065
1.048
1.181
1.128
16
1.525
1.021
1,153
1.102
32
2.480
1.244
1.320
1.287
0.715
1.028
1.160
1.108
0.780
1.034
1.171
1.117
64x128
0.984
1.034
1.171
1.120
1.361
1.048
1.182
1.130
16
2.071
1.147
1.223
1.192
32
3.563
1.566
1.645
1.612
Table 15: Performance of decoding single token with 4-bit KV cache for the attention layer for 2048 sequence lengths with and without online Hadamard transformation on an NVIDIA RTX 3090 GPU. We evaluate generating the last token when the 2047 tokens already cached in the attention. We extract the number of heads (head num) and their dimensions (head dim) based on different LLAMA-2 models. We averaged over 100 runs report the numbers in milliseconds.
Tables 16 and 17 show the detailed speedups and memory saving of single transformer block for QuaRot on LLAMA2-78 7B model using NVIDIA RTX 3090 GPU.
Model
Batch Size
Speedup
1
1.97 x
4
2.06
LLAMA2-7B
16
2.11x
32
2.14x
64
2.16x
1
3.16x
4
3.27
LLAMAA-70B
16
3.32x
32
3.33
Table 16: Time-to- time-to-first-tokee first-token (prefill) speedup of each transformation block of LLAMA-2 models in QuaRot (over the FP16 model) NVIDIA RTX 3090 GPU. We use 2048 sequence lengths with different batch sizes.
20

---

Batch Size
Sequence Length
Baseline (GB)
QuaRot (GB)
Saving
Model
Factor
256
0.392GB
0.108GB
3.63
512
0.396GB
0,108GB
3.66x
1024
0.404GB
0.110GB
3.66
2048
0.419GB
0.114GB
3.67
4096
0.451GB
0.125GB
3.60
LLAMA2-7B
256
0.464GB
0.128GB
3.63
512
0.528GB
0.144GB
3.66x
16
1024
0.655GB
0.177GB
3.70
2048
0,908GB
0.244GB
3,72x
4096
1.416GB
0.378GB
3.75x
256
1.60SGB
0.409GB
3,92x
512
1.606GB
0.409GB
3,92x
1
1024
1.608GB
0.410GB
3,92
2048
1.612GB
0.411GB
3.92x
4096
1.620GB
0.413GB
3.92x
LLAMA2-70B
256
1.626GB
0.418GB
3,89
512
1.642GB
0.422GB
3.89
16
1024
1.674GB
0.430GB
3.89
2048
1.738GB
0.447GB
3.89
4096
1.865GB
0.480GB
x.89
Table 17: Peak Memory usage (in GB) for decoding a single token on single transformation block of LLAMA-2 models with K caches of different lengths and with different batch size.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect paper's contributions and scope?
Answer: [Yes]
Justification: We provide all the results for supporting our claims for both abstract and introduction section in the experiment section (see Section 5).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide the next steps of our work in the Conclusion section (see Section
6).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer: [NA]
Justification: We do not provide any theoretical results and we cited all related works (like SliceGPT Ashkboos et al. [2024]) in the main text.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide all the codes and experimental settings for our results (see Section 5).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
23

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We use public models and datasets in our experiments with clear instructions to reproduce the main results of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.c cc/
for more details.
While we encourage the release of code and data, understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (httpa: gudddde/oodeeuummiiii CCodeSubmiiiec ssi asionPooicyy for more details.
/ //nips. wttpsssccccccccccccccccceccccl www.de//////eeeeeee/oooccoo.ee
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All parameters are presented in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: As we use large models (with at least 7B parameters), different experiments do not have too different outputs. We do not repeat the experiments they costly as well.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All details are presented in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:/ httppppp////uips....... http://uurris.ccppubliiee wwwpp/ubliitttiicccuiieeeeeeeeed
Answer: [Yes]
Justification:
The paper conforms the Neurips CoE.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is about LLM inference acceleration and it does not directly have any specific societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
25

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not provide any new model or changing the models to behave in a new way and it does add new ability the already existing models so it does not have any risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We develop our code using publicly available libraries, models, and datasets. We submit our assets using CC-BY 4.0 license.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwperswith thode.. cwwccoo//////// datastsssss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All the assets documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not provide any crowdsourcing experiments and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "e6ykErlkq1", "reviewer": "Reviewer_rEuG", "summary": "The paper introduces QuaRot, a novel quantization approach for Large Language Models (LLMs) that utilizes Hadamard transformations to address the challenge of outlier features in activations, weights, and KV caches. By incorporating these transformations, QuaRot enables the entire model, including activations and caches, to be quantized to 4 bits without significant loss in performance. This method improves both the computational efficiency and memory usage during model inference, preserving up to 99% of zero-shot performance in tests with the LLAMA2-70B model.", "questions": "How to extend your method if the model adopts different normalization methods, like layer normalization or batch normalization. Also, why INT4 but not INT2 or INT1 as QuaRot can handle the latter two too?\n\nWhy the RMSNorm is not quantized? In the case of normalization layer quantization, how much the acceleration performance of the model and its performance on the datasets will be affected?\n\nFurthermore, are the biases in the linear layers and the residual connections quantized to INT4?\n\nPlease specify your meaning of online Hadamard transform on line 38. \n\nThe consideration of the Hadamard transformations increases the difficulty of implementing the quantization kernel. Please explain more about how the quantization kernel is implemented based on the CUTLASS library to achieve real speedup?\n\nIn Fig. 3, (\\alpha) should be diag(\\alpha)? Also, line 122, the Q^T on output matrix becomes Q in Fig. 3?", "limitations": "The INT4 QuaRot still incurs significant drops from INT8 and FP16.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "QuaRot addresses quantization of both weights, activations and KV caches, making the approach practical in the real-world scenarios.\n\nThe application of randomized Hadamard transformations helps in effectively managing outlier features, allowing for lower bit representation without performance degradation.\n\nThe method achieves substantial efficiency improvements, evidenced by up to 3.33× speedups in prefill operations and significant memory savings during the decoding stage.", "weaknesses": "Although the method sounds good in terms of the computational invariance, the experimental results are not impressive to some extent. Based on Table 2, there is a clear margin between the FP16 baseline and the proposed QuaRot on 7B, 13B and 70B settings.\n\nThe distinction between the proposed randomized Hadamard transformations and the Hadamard quantization method in Xi et al “Training Transformers with 4-bit Integers” should be elaborated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your comments!\n\n>The distinction between the proposed randomized Hadamard transformations and the Hadamard quantization method in Xi et al “Training Transformers with 4-bit Integers” should be elaborated.\n\nWe acknowledged the above work in our paper (lines 63-64) and described the fact that they have used the Hadamard transformations during training. We have edited the manuscript to emphasize that they use “exact” Hadamard (not randomized Hadamard) transformation.\n\n>How to extend your method if the model adopts different normalization methods, like layer normalization or batch normalization. \n\nThe main difference between LlamaRMSNorm (LRMSN) and LayerNorm (LN) is that the former does not have mean subtraction. \nHowever, as shown in [1] (equation 1), mean subtraction could be written as another linear transformation where we can fuse into the previous linear layer and replace LN with our RMSNorms. To our knowledge, Batch-Normalization is not used in any modern LLMs (which is the main focus of our work).\n\n>Also, why INT4 but not INT2 or INT1 as QuaRot can handle the latter two too?\n\nQuaRot could be applied with any precision (for example, we presented weight-only results for lower precision like INT3 in the submission -> Table 7). However, we focus on INT4 due to hardware support, which allows us to get direct speedups. We should note that Quantizing weights to fewer than 4 bits is possible (Table 7), but this does not always translate to a speedup due to lack of hardware support in all scenarios. In practice, it means using 4 bits but still sacrificing quality unless custom packing/unpacking support is implemented to reap the benefit.\n\n>Why the RMSNorm is not quantized?\n\nWe keep the RMSNorm in high precision to avoid numerical instability (we need to calculate the l_2 norm of the input vectors and divide the input by it). However, keeping those modules in high precision does not meaningfully affect the end-to-end FLOP reduction as they are <0.4% of the total FLOPs in the baseline model [2].\n\n>Are the biases in the linear layers and the residual connections quantized to INT4?\n\nIn LLaMa models, the linear layers do not have biases. However, biases could be fused into the matmul kernel to be applied just after dequantization. We also do not quantize the shortcuts as we focus on the computational bottlenecks of the model (which are linear layers). \n\n>Please specify your meaning of online Hadamard transform on line 38.\n\nThroughout the paper, we make the distinction between “fused” Hadamard transforms (i.e where the weight is modified as WH) and “not fused” ones, which we call “online” as they must be executed independently.\n\n>Please explain more about how the quantization kernel is implemented based on the CUTLASS library to achieve real speedup?\n\nRight now, we use separate kernels for this. We  Apply Hadamard -> Quantize -> MatMul -> DeQuantize. The quantization part is done on PyTorch.  However, the first two operations could be fused into a single kernel. We described these details in the Appendix A.10.\n\n>In Fig. 3, (\\alpha) should be diag(\\alpha)? Also, line 122, the Q^T on output matrix becomes Q in Fig. 3?\n\nThanks for spotting that, we have amended the manuscript. For the Q^T, we apply exact Hadamard transformation from the left and call it by H (instead of Q) so the Q in the down-proj layer of Fig. 3 is the randomized Hadamard transform we apply to the output of that linear layer (to form YQ).\n\n\n**References**:\n\n[1] https://arxiv.org/pdf/2401.15024\n\n[2] https://huggingface.co/spaces/MrYXJ/calculate-model-flops", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sti9TNi6HP", "reviewer": "Reviewer_QCed", "summary": "The authors provide a framework for W4A4KV4 quantization of LLMs, leading to computing and peak memory improvement while maintaining good performance on language generation and zero-shot tasks. They achieve this by introducing randomised Hadamard transformations at both weight and activations of transformer blocks. They effectively build on the ideas of incoherence processing and computational invariance that have shown that applying orthogonal transformation to weight matrices in LLMs allows them to be quantized with smaller errors while ensuring the network output is unchanged.", "questions": "- Figure 3. What does the (α) mean in the box? Is it supposed to be a diagonal?\n- Have you considered listing all the quantization bitwidths/granularities weights/activations, etc., mentioned in section 5 in a table? See Table 2 of SmootQuant.\n- What operation does the Hadamard block do in Figs. 3&6. Based on my calculations, it is a post: $Y = X H^T$ . Please add more information about what these blocks do for each graph. \n- Are all results in Table 1 for all methods with W4A4KV4 and the same granularity everywhere?\n- Is the perplexity in Table 1 for the test or validation set?\n- Line 280 explicitly mentions that this is about a single transformer block rather than saying it later in the paragraph. One needs to look in the appendix or the brackets below.\n- Figure 4: Please mention that these are the averages of 100 runs as you do in the appendix. What is the STD of these runs?\n- Table 9: Are weight and activation quantized to the same bitwidth? What about KV cache? Please be more explicit about the bit width choices per tensor in all tables and figures.\n- Appendix A.6: why is this ablation study there? What is the take-aways?", "limitations": "The authors recognise the limitations of the method and provide a roadmap of improvements in the conclusion section.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* Achieving good accuracy in W4A4KV4 LLM quantization is a challenging problem. It is commendable that the authors achieve good accuracy in zero-shot and language generation tasks in such an aggressive quantization regime with significant compute and peak memory savings. \n* The paper is very well written, with a concise and clear background section and readable graphs (although some colour-coding of the boxes would be helpful).\n* Applying the online Hadamard transformation to all activations is an interesting idea that improves the quantizabilty of activation and KV cache. Given the complexity of rotational embeddings, the authors have also taken great care in consistently applying these transformations.\n* The authors have conducted very good and extended quantization ablation studies. In the current landscape of numerous quantization hacks and choices for LLMs, this research is crucial in understanding the effect of each choice.", "weaknesses": "* The motivation for using Hadamard matrices as orthogonal matrices is not well-established. Given that incoherence processing and computation invariance have established that orthogonal matrices can be used to improve quantization/pruning, I would like to see a more coherent and concise motivation for using Hadamard matrices. I can tell from section 3.1 that they are computationally efficient, but this is mentioned as a side note rather than the reason for using them. Building on this, it is still unclear to me from sections 3 & 4 if Hadamard matrices are the authors' contribution or taken from previous LLM quantization work.\n* The comparison to other relevant work lacks details. In Table 1, the authors should take better care to ensure that the quantization granularity of weights, activations, and cache is coherent between their method and the other benchmarks. For example, for a fair comparison, they should use SmoothQuant-O1 (see Table 2 of the paper). Even if the authors use the correct quantization scheme, they should detail any differences between their method and the competing ones either in section 5.1 or the appendix. LLM quantization methods are becoming increasingly complex. Therefore, it is important that there is transparency when comparing perplexity to other methods.\n* Why not include the same benchmark methods from Table 1 to Table 2?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your comments and encouragement. We agree that we can improve the readability of our results and include more details in our Tables. We have included these details in our manuscript.\n\n>Figure 3. What does the (α) mean in the box? Is it supposed to be a diagonal?\n\nYes, this means the diagonal matrix. \n\n>What operation does the Hadamard block do in Figs. 3&6. Based on my calculatio…\n\nThis is an online (non-fused) Hadamard, as described in Section 4.\n\n>Are all results in Table 1 for all methods with W4A4KV4 and the same granularity everywhere?\n\nYes.\n\n>Is the perplexity in Table 1 for the test or validation set?\n\nThe PPL in Table 1 is computed over the test  set.\n\n>Figure 4: Please mention that these are the averages of 100 runs as you do in the appendix. What is the STD of these runs?\n\nThe std of the runs are <0.1 in all cases. \n\n>Table 9: Are weight and activation quantized to the same bitwidth? What about KV cache? Please be more explicit about the bit width choices per tensor in all tables and figures.\n\nYes. all the weights, activations, and KV-caches are in the same precision in Table 9.\n\n>Appendix A.6: why is this ablation study there? What is the take-aways?\n\nThe main take-away from this section is that in 8- and 6-bits RTN performs essentially as well as GPTQ when we use orthogonal transformations in QuaRot.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cKS3oCsS9d", "reviewer": "Reviewer_hgDj", "summary": "This paper applies the random Hadamard transform (RHT) in strategic places in the GPT architecture to improve the quality of weight and activation quantized models. The RHT is a fully invertible (up to machine epsilon) transformation that effectively concentrates matrices. The authors claim that applying the RHT to activations reduces the effect activation outliers have on activation quantization performance. The authors show that their method can be used for INT4 weight and activation quantization without significant degradation, which enables higher throughput inference in compute-bound scenarios.", "questions": "1. What is the comptuational cost of each of the steps in QuaRot? The matrix multiplication is the dominating factor, but the Hadamard transformation is not completely free. If you used the RHT as presented in QuIP#, the RHT can actually be quite slow for \"bad\" matrix dimensions where the non-power-of-2 component is quite large (eg 172 in Llama 2 7B). How would you solve this in practice? \n2. The RHT requires \"mixing\" activations across the input dimension, making it costly to use with tensor parallelism since you need to send lots of data across a potentially slow bus. Have you run any experiments on the effectiveness of QuaRot when the RHT applied \"per slice\" (eg 1/8 of the activation dimension for 8 GPUs)? How well does the RHT work here?\n3. RE the uniform scalar quantizer comment above, FP4 would probably work better for quantizing a Gaussian. Can you run an experiment with FP4 activations and weights and see if the quality improves?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The empirical results appear to be pretty good. Quantizing both the weights and activations to INT4 using QuaRot results in minimal degradation and enables up to a >3x speedup during prefill.", "weaknesses": "My main concern with this paper is that it combines methods from existing works (RHT from QuIP#, GPTQ from GPTQ, \"Comptuational Invariance\" from SliceGPT) instead of introducing something actually novel and new. Those works independently showed that their respective ideas worked, so it is not surprising that combining them also works. Fusing the Hadamard transformation into the weight matrix and rotating the activations is also not something \"new\", as this is actually how QuIP# does inference (it is cheaper to rotate the activations than the weights in small-batch settings). \n\nRegarding the actual method, QuaRot essentially does GPTQ + Incoherence Processing on the weight matrix, which QuIP showed was optimal within a certain class of rounding methods. However, to the best of my knowledge, there is no optimality result for doing Incoherence Processing + RTN, which is what QuaRot does on the activations. In fact, the distortion for quantizing a Gaussian (the result of the RHT) with a uniform scalar quantizer (INT4) is actually pretty bad relative to the distortion for quantizing other common distributions with a uniform scalar quantizer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your comments.\n\n> this paper is that it combines methods from existing works\n\nHere, we explain our contributions and the main differences between QuaRot and the existing work.\n\n1. SliceGPT: SliceGPT focuses on compressing LLMs by slicing the weights. Although we use the same principle (computational invariance) from SliceGPT, we apply RHT as an orthogonal transformation which is different from what they use  (Principal components) and especially suited to quantization because it makes the input distribution more uniform when it has some outliers (Fig. 1). In addition, SliceGPT needs to undo the transformations in the shortcuts as they use PCA over the input of each linear layer and QuaRot uses a single RHT for the whole model which eliminates the need for shortcut transformations.\n\n2. GPTQ and QuIP#: GPTQ and QuIP# focus on weight-only quantization while we focus on quantizing weights, activations, and KV-caches.  Quantizing weights only has the practical impact of having a smaller footprint for the model. Quantizing activations additionally saves working space at inference time. Quantizing KV-cache has a huge practical advantage of much higher token throughput (especially in long-context generation). Although we also use RHT (similar to QuIP#), we remove the overhead of applying such transformations (and their inverse) by fusing the RHT into the adjacent layers and applying it after RoPE for KV-cache quantization. \n\n3. Practical usage: We developed a set of CUDA kernels for our work and got up to 3.33x speedups and 3.89x memory saving over the FP16 version. Compared to the previous 4-bit quantization work [1, 2], we have simpler kernels as we do not need complex quantization schemes after applying RHT.\n\n>as this is actually how QuIP# does inference\n\nThere are a number of important differences between QuIP# and QuaRot which we summarize here:\nQuIP# applies the Haramard transformations before quantization for each weight matrix in the model. During the inference, the inverse of such transformations should be applied to undo the transformation which adds overhead in each linear module. However, QuaRot does not need such extra transformations as we fuse the Hadamards into the previous layer (Stage 1a. of the Method section). This enables us to 1) apply fewer Hadamard transformations (only 1 in the MLP and only a single intra-head in the attention) during the inference and 2) remove the input outliers so we can quantize both weights and inputs to 4-bits. \n\n\n>What is the comptuational cost of each of the steps in QuaRot?\n\nThis is one of the advantages of QuaRot over existing work. In the first step, QuaRot fuses the layer-norm and replaces them with RMSNorm (which preserves any orthogonal transformations on the input). This helps us to completely remove the overhead of applying RHT by fusing it into the previous linear weights. This guarantees that the input of each linear layer is already on the (randomized) Hadamard space. We undo the input Hadamard by fusing its inverse to the weight of the linear layer so we will have XQQW = XW. In total, for every decoder block, we need to apply a single Hadamard transformation in the MLP (after non-linearity) and only a single intra-head in the attention (before Our-proj). For such transformations, we use a fast Hadamard kernel which leads to minimal overhead (Table 15).\n\n>Have you run any experiments on the effectiveness of QuaRot when the RHT applied \"per slice\"?\n\nWe did not apply the tensor-parallel as we focus on LLM inference where TP is a less common setting in that case. However, consider an input matrix of size `BxH` where B is the `batch-size*sequence_len` and `H` is the hidden dimension of the model. If the matrix is split across the rows which is suitable for RowMajor matrices (for example, we have 8 matrices of size `B/8xH`), the Hadamard of size `H` could be applied on each chunk independently. In the case of splitting the columns, one can apply smaller Hadamard matrices on each submatrix of `BxH/8` independently and quantize each block separately. \n\n>FP4 would probably work better for quantizing a…\n\nWe ran the following experiment for LLaMa2 models with W4A4KV4 just now, comparing INT4 vs MXFP4 with group-size 128. The MXFP4 format [3] is an alternative to the standard FP4 format which shares an additional common scale within the group, so should perform slightly better than FP4 at a cost of an extra 1/16th of a bit per weight (see [3], Section 5.1 for details). The results show that MXFP4 performs worse than INT4 across the LlaMa2 model sizes. This perhaps surprising result implies the post RHT distributions are more uniform than purely Gaussian.  \n\n\n| Model |  INT4 | MXFP4|\n|----|------|------|\n|7B| 5.93|6.52|\n|13B| 5.26 |5.71|\n|70B| 3.61 |3.82|\n\n**References**:\n\n[1] https://arxiv.org/abs/2310.19102\n\n[2] https://arxiv.org/abs/2310.09259\n\n[3] https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BzRELbCrR3", "reviewer": "Reviewer_tVLH", "summary": "The paper focusses on improving post-training-quantization for LLMs. Specifically, they combine insights into incoherence processing from the two #QUIP papers and rotational invariance from SliceGPT to improve accuracy after quantization. The insight is that LLMs are, or can be made, invariant to rotations on the residual, which can then be exploited by multiplying the residual with random rotation matrices (and it's inverse), which approximately reduce outliers in the activations and weights. A part of this method comes at the cost of no extra overhead, as the weight matrices are absorbed into the layers. The paper also introduces a specific form of rotation matrix, the Hadamard matrix (which was also used in QUIP# in several other parts of the network. These Hadamard matrices cannot be absorbed, but can be efficiently executed with the Walsh-Hadamard matrix multiplication algorithm, incurring only little overhead. \nThe paper does extensive experiments on the LLamaV2 model showing the quantization is significantly improved after the rotation operations. The authors also combine the method with other methods that improve performance, such as GPTQ and grouped-quantization, showing that the method can be combined with other quantization-improvement methods. \nThe authors also show actual speed-ups from their 4bit kernels.", "questions": "Is it possible to put more comparisons with other methods into the tables?\n\nIn Table 7, you show that the GPTQ algorithm for 7B gives worse performance than RTN. I also validated that this happens in your codebase. How is it possible that GPTQ gives a worse performance than RTN? Theoretically, it should reduce the MSE for each layer, and never really give a worse performance. Is your implementation correct? \n\nEditorial notes: \n- 140 consider linking computational invariance to the relevant section", "limitations": "They have adequately addressed the limitations.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The idea of combining SliceGPT and QUIP was out there hanging in the ether, as the two papers compliment each other very neatly. I might expect more authors in this field to do similar work. Regardless, I believe this work is novel at this point.\nThe impact of this work is potentially quite large, as it allows LLMs to significantly improve in speed by having lower-bit-width quantization much more available. On top of this, the algorithms are pretty easy to apply. The paper reads as a full state-of-the-art quantization pipeline that can be applied on LLMs in the PTQ setting. I can see this being applied in many practical LLM settings.\nThe paper is well-written and easy to follow. It does not claim anything it doesn't do, and is factual without scruples.\nThe experiment section is solid, and the ablations in the appendices good. The code has been released, and I have been able to verify some of the numbers from the authors. The authors went the extra mile showing not only their theoretical gains, but also speed-ups in practice.", "weaknesses": "- Emoji's in titles should get desk-rejected ;)\n- Table 2 does not really compare to anything. There's plenty of papers that do 4 bits, you have the original GPTQ paper, LLM-QAT, ATOM, OmniQuant, SmoothQuant, simple RTN. it's pretty tough to contextualize the results here; but zero-shot is quite important for a proper evaluation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments.\n\n> Emoji's in titles should get desk-rejected ;)\n\nThe emoji is there just because we wanted to give a hint on how QuaRot should be pronounced :-)\n\n> Table 2 does not really compare to anything. Is it possible to put more comparisons with other methods into the tables?\n\nWe compared the WikiText results with other papers in Table 1. However, zero-shot experiments are presented to show that our scheme works well over various tasks. We will evaluate other schemes on the same set of tasks and include them in the next version.\n\n> In Table 7, you show that the GPTQ algorithm for 7B gives worse performance than RTN. \n\nWe agree that this is surprising. This may be because of using GPTQ with default parameters and not tuning the parameters in the algorithm (some of them are mentioned here: https://github.com/IST-DASLab/gptq?tab=readme-ov-file#new-features).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Hello Reviewers, we appreciate you taking the time to read and evaluate our paper. It is great to see that you liked our paper and found it impactful. 

We would like to summarise the updates we made to the paper and extra experiments we have done to address your concerns:

1. **Changes to the paper**: We included all the comments in the main text and updated the manuscript with a blue color in our draft. These updates primarily concern clarifying experiment setups and fixing typos.

2. **New results with new data type**: We have added a set of experiments with the MXFP4 format which is a similar format to FP4. Our results show that applying the randomized Hadamard transformations makes the input distribution more uniform than Gaussian, which is suitable for integer quantization (see our reply to the reviewer hgDj)

3. **New results with Mixtral**: we ran QuaRot on Mixtral 8x7B v0.1, a mixture of expert model, to show that QuaRot also applies to this architecture. We ran weight-only (W4A16) and 4-bit weights and 4- and 6-bit activations and performed within 1.5% of the baseline model on the LM eval benchmarks using QuaRot, highlighting the method’s validity on this different architecture.  

| Format          | PPL  | Avg acc |
|-----------------|------|---------|
| FP16 (baseline) | 3.84 | 77.63%  |
| W4A16KV16 128G  | 3.95 | 77.51%  |
| W4A6KV16 16G    | 3.92 | 77.41%  |
| W4A4KV16 16G    | 4.18 | 76.33%  |


4. **Answering other questions**: We also addressed all questions about the paper in different sections.

Thanks again for your time. We look forward to discussing and answering follow-up questions and comments.

Meta-review: This paper presents QuaRot, a framework for quantization of large language models (LLMs) using Hadamard transformations applied to weights and activations. QuaRot is able to quantize the entire model, including activations and caches, in 4 bits without significant loss in performance. The approach builds on concepts from existing works, such as incoherence processing and computational invariance, to achieve significant improvements in computational efficiency and memory usage while maintaining strong performance in language generation and zero-shot tasks. Reviewers acknowledged the method's effectiveness, particularly in improving quantization without major performance loss. However, there were concerns about the novelty of the contribution, as the paper primarily integrates methods from previous research. Additionally, some aspects, such as the motivation for using Hadamard matrices and the comparison to other quantization techniques, were found to be insufficiently detailed. Most of them have been addressed during the rebuttal period. Overall, the practical benefits and solid experimental results make this paper a valuable contribution to the field of LLM quantization. I recommend accepting the paper.
Decision: Accept (poster)