I am planning to research quantization for small-area NPUs. Please appropriately draft the content for the following task.

Writing Assignment 2: Research Topic Overview
Objective:
To provide students with an opportunity to present a comprehensive overview of their chosen research topic, including a visual representation (figure), a table of related works, and an overview of existing literature. The assignment will be formatted using the Elsevier Article (elsarticle) LaTeX template.
Requirements:
1.	Length: Minimum of two pages (excluding references).
2.	Content:
o	Introduction to the research topic.
o	Background and importance of the research topic.
o	Detailed discussion on the topic-related work.
o	At least one figure relevant to the research topic.
o	At least one table comparing at least 5 related works.
3.	Formatting: Use the Elsevier Article (elsarticle) LaTeX template.
4.	Submission: Submit both the PDF and the LaTeX source files (Overleaf project zip).
Instructions:
1.	Introduction: Start with a brief introduction to the research topic, outlining its importance and relevance in the current academic or practical context.
2.	Background and Significance: Provide background information on the topic, discussing the current state of research, and existing gaps in the research topic.
3.	Discussion: Offer a detailed explanation of key concepts, theories, or methodologies relevant to your research.
4.	Figure: Include at least one figure that visually represents an aspect of your research. This could be a diagram, graph, or illustration that helps in understanding your topic.
5.	Table: Include a table that compares at least 5 related works. The table should summarize key aspects such as objectives, methodologies, results, and conclusions of these works.
6.	References: List all the references cited in your document in the appropriate format.
Elsevier Article (elsarticle) LaTeX Template:
1.	Download the template from the Elsevier website or Overleaf.
2.	Ensure your document follows the formatting and style guidelines provided in the template.
Submission:
1.	Compile your LaTeX document to generate a PDF.
2.	Download the source files from Overleaf as a zip file.
3.	Submit both the PDF and the Overleaf project zip file.
Evaluation Criteria:
•	Content Quality: Depth and clarity of the discussion on the research topic.
•	Figure and Table: Relevance and clarity of the included figure and table.
•	Comparative Analysis: Effectiveness of the table in comparing related works.
•	Formatting and Presentation: Adherence to the Elsevier Article (elsarticle) template guidelines.
•	References: Proper citation and listing of all referenced works [numbering style].
Tips:
•	Ensure your figure is clear and properly labeled.
•	Use the table to highlight similarities and differences among the related works.
•	Proofread your document to avoid grammatical and typographical errors.
•	Review the Elsevier Article (elsarticle) template guidelines thoroughly to ensure compliance.
This assignment will help you develop skills in academic writing, LaTeX formatting, and introducing your research work. Good luck!

QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
Saleh Ashkboos
ETH Zurich
<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
Amirkeivan Mohtashami
EPFL
<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
Maximilian L. Croci
Microsoft Research
<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
Bo Li
ETH Zurich
<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
Martin Jaggi
EPFL
<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
Dan Alistarh
IST Austria & NeuralMagic
<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
Torsten Hoefler
ETH Zurich
<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
James Hensman
Microsoft Research
<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
Abstract
We introduce QuaRot, a new Quantization scheme based on Rotations, which
is able to quantize LLMs end-to-end, including all weights, activations, and
KV cache in 4 bits. QuaRot rotates LLMs in a way that removes outliers from
the hidden state without changing the output, making quantization easier. This
computational invariance is applied to the hidden state (residual) of the LLM, as
well as to the activations of the feed-forward components, aspects of the attention
mechanism and to the KV cache. The result is a quantized model where all
matrix multiplications are performed in 4-bits, without any channels identified
for retention in higher precision. Our quantized LLAMA2-70B model has losses of
at most 0.29 WikiText-2 perplexity and retains 99% of the zero-shot performance.
Code is available at: https://github.com/spcl/QuaRot.
1 Introduction
Large language models (LLMs) have become increasingly important due to their countless applications.
However, their inference requires a significant amount of computations, memory, and energy.
LLM inference has two main phases: during the prefill phase, the prompt is processed and cached in
each layer, and then in the decoding phase the output tokens are generated one-by-one. The prefill
phase is usually compute-bound while the decoding phase is known to be memory-bound.
Quantization is among the most important techniques to solve both memory and compute issues
during LLM inference. Joint quantization aims to reduce the precision of parameters and KV cache
(which results in lower memory usage) as well as inputs (known as activations) and compute the
forward pass in low precision. However, quantizing the activations is hard as they have large outlier
elements (see Figure 1 for an illustrative example) with much larger values, making activation
quantization more difficult than weight quantization, especially for the 4-bit case. Previous work
relies on characterizing outlier features using a calibration set and keeping them in higher precision
during the inference.
Preprint. Under review.
arXiv:2404.00456v1 [cs.LG] 30 Mar 2024
Figure 1: The distributions of activations at the input to the FFN block in LLAMA2-7B model, in the
tenth layer. Left: using the default configuration as downloaded from Hugging Face. Right: after
processing using QuaRot. The processed distribution has no outliers, leading to superior quantization.
In this work, we address the issue of outlier features by rotating the inputs of the model using randomized
Hadamard transformations. We do this using the computational invariance idea [Ashkboos et al.,
2024] and fuse Hadamard transformations into the weight matrices, resulting in an equivalent network
without outlier features. This enables the weights, activations, and KV caches to be quantized to 4
bits with minimal accuracy drop. Our main contributions are:
• Using computational invariance from SliceGPT [Ashkboos et al., 2024], we apply randomized
Hadamard transformations to the weight matrices without changing the model. This
completely eliminates the outlier features and makes the activations easy to quantize without
changing the output of the model.
• We also apply online Hadamard transformations to the attention module to remove outlier
features in keys and values, enabling the KV cache to be quantized.
• Using the above modifications, QuaRot enables 4-bit LLM inference by quantizing all the
weights, activations, and KV caches using integer quantization. On LLAMA2-7B model,
QuaRot achieves up to 2.16x prefill speedups, and 3.39x memory saving during the decoding
stage, with at most 0.63 WikiText-2 perplexity loss. QuaRot preserves 99% of the accuracy
of zero-shot tasks. We show that our 6 and 8-bit quantization (weights and activations) is
lossless.
2 Related Work
The majority of quantization schemes focus on compressing LLMs by using weight-only quantization,
[Frantar et al., 2022, Dettmers et al., 2023, Lin et al., 2023, Egiazarian et al., 2024, Tseng et al., 2024].
These methods downcast each weight into a low precision representation and upcast it before the
actual computation. The main computation is still performed in high precision. Several works show
that, unlike weights, quantizing the activations is hard due to the outlier features [Wei et al., 2022,
Dettmers et al., 2022, Xiao et al., 2023]. For 8-bit quantization, LLM.int8() [Dettmers et al., 2022]
identifies the outlier features during the inference and keeps them in 16-bits which results in poor
performance. SmoothQuant [Xiao et al., 2023] normalizes the features using some scaling factors
from a calibration set, solving the issue for the 8-bit case. For 4-bit quantization, recent studies
identify and keep the outlier features in high precision offline. Atom [Zhao et al., 2023] keeps 128
outliers in 8-bits and quantizes the inputs using group-wise quantization (through a complex kernel).
QUIK [Ashkboos et al., 2023] keeps 256 outlier features with per-token quantization but at the cost
of keeping the down-projection layer in 8-bits.
SliceGPT [Ashkboos et al., 2024] is a structured pruning method that produces a model with a smaller
hidden dimension. The method relies on computational invariance which means that the hidden
state of a transformer model can be computed in any transformed coordinate system, by multiplying
2
input (output) weight matrices of each block on the left (right) by an orthogonal matrix. In this
work, we apply computational invariance to LLMs to enable more effective quantization: by applying
Hadamard orthogonal matrices to each weight matrix, we change the coordinate system of the hidden
state, eliminating outliers. We extend this idea further than SliceGPT, applying coordinate transforms
to activations within blocks, and to the KV cache.
Two weight-only quantization methods, QuIP [Chee et al., 2024] and QuIP# [Tseng et al., 2024] have
previously considered improving quantization by applying rotations. [Chee et al., 2024] introduced
the idea of incoherence processing which applies rotation matrices to the left and right of each weight
matrix, as well as the Hessian, which is used in minimizing the weight-quantization objective.
Incoherence processing replaces the weight matrix W with ˜W = UWV, where U and V are
orthogonal matrices. This suppresses outliers in the weights, making quantization more effective. In
the forward pass of the quantized network, the effects of U and V must be undone. To make this fast,
QuIP used Kronecker-structured orthogonal matrices which allows these additional computations to
be computed quickly. Building on this, QuIP# introduced the use of Hadamard matrices instead of
Kronecker-structured matrices. This enables both improved quantization (through better incoherence)
and a faster forward pass since the Hadamard transform can be computed in O(n log n) addition
operations.
Finally, KV cache quantization is another line of research where the goal is to compress the cached
keys and values during the generation task. This is crucial for large batch size and long-context length
generation as the KV cache will be the main memory bottleneck in such problems. [Sheng et al.,
2023] quantizes the KV cache using 4-bit group-wise quantization. KVQuant [Hooper et al., 2024]
pushes this limit to 3-bit quantization and KIVI [Liu et al., 2024] shows promising results on 2-bit
KV cache quantization. Such methods show that outliers also exist in the keys, and apply a set of
complex ideas (like feature-wise quantization, non-uniform representation, keeping high precision
outliers) to recover the accuracy of a quantized KV cache.
In this work we also adopt the Hadamard transform to improve quantization of weights through
incoherence processing. Instead of undoing the Hadamard transform during the forward pass, we
adopt the computational invariance theorem from SliceGPT to fuse the transformations into the
weights where possible. Instead of requiring two Hadamard transforms per weight-matrix in the
forward pass, QuaRot requires just 11
2 Hadamard transforms per transformer layer. Computational
invariance also means that the activations are incoherence-processed, enabling them to be effectively
quantized. We also apply a similar technique to the attention block and quantize the KV cache in
4-bit with minimal accuracy loss.
3 Background
Here we introduce some background mathematical concepts and notation that are necessary for
QuaRot.
3.1 Orthogonal, Rotation and Hadamard Matrices
An orthogonal matrix Q is a square matrix such that QQ⊤ = I. In this work, we consider only real
orthogonal matrices. A rotation matrix is an orthogonal matrix with |Q| = 1.
A Hadamard matrix is an orthogonal matrix with entries proportional to {+1,−1}. A Walsh-
Hadamard matrix is a square matrix of size 2n, with
H2 = √1
2

1 1
1 −1

and H2n = H2 ⊗H2n−1 . (1)
These identities give rise to the Walsh-Hadamard transform, which computes the matrix-vector
product Hx in O(d log2(d)) operations.
For matrix sizes that are not 2n, the existence of a Hadamard matrix is not guaranteed. A useful list
of known Hadamard matrices is available at [Sloane, 2024]. Where we require a Hadamard matrix
of size d ̸= 2n, we factorize d = 2nm, where m is the size of a known Hadamard matrix. Then we
use a Kronecker construction Hd = H2n ⊗Hm. This allows computation of Hdx in O(d(m + n))
operations.
3
X x
∥x∥ diag(α)
Wgate
Wup
σ
× Wdown Y
RMSNorm
FFN
Figure 2: The gated feed-forward network used in most LMs, including the pre-positioned RMSNorm.
The input signal is divided by its norm, and re-scaled by parameters α. Two linear blocks, Wup and
Wgate are applied. The activation function σ is applied to the gated signal, and the two signals are
element-wise multiplied together. The final linear blockWdown produces the output signal Y. Before
quantization, the operations are performed in single (32 bit) or half (16 bit) precision.
Following [Tseng et al., 2024] we make use of randomized Hadamard matrices where convenient. Let
s be a vector containing random draws from {+1,−1}, and ˜H = Hdiag(s). It is straightforward to
see that ˜H is also an orthogonal matrix.
3.2 Incoherence Processing
The idea of incoherence processing was introduced by [Chee et al., 2024] in the context of weight
normalization. We define a weight matrixWto be μ-incoherent if
max
􀀀
W

≤ μ∥W∥F /
√
mn (2)
where max is the element-wise max of the matrix, and mn is the number of elements. A weight
matrix that has high incoherence is hard to quantize: the largest element is an outlier relative to the
magnitude of the average element. [Chee et al., 2024] showed that multiplying a weight matrix on the
left and right by an orthogonal matrix can reduce the incoherence, making matrices easier to quantize.
In this work we adopt a similar technique, multiplying weight matrices by orthogonal matrices to
improve incoherence, though we add fewer operations to the forward pass, and additionally apply
incoherence processing to the activations, enabling improved weight and activation quantization.
Figure 1 shows the effect of applying incoherence processing to the activations of LLAMA-2 .
3.3 Transformer structures
Large Language Models are neural networks with repeating attention and feed-forward layers.
We introduce our notation through Figures 2 and 3, which show the construction of these blocks.
We assume that the construction of the network is “pre-norm”, in that each block is preceded
by a LayerNorm or RMSNorm operation. We also assume that the feed-forward network uses a
gated architecture, as in LLAMA-2 , though our methodology is straightforwardly applied to MLP
architectures also.
3.4 Computational Invariance
The computational invariance theorem [Theorem 1, [Ashkboos et al., 2024]] states that the weights
and between-block activations in a transformer can be transformed using an orthogonal matrix with
no change to the model output. Here we sketch the main idea. IfWin is a weight matrix that appears
on the left of a transformer block (i.e. Wgate,Wup in Figure 2, orWk,Wq,Wv in Figure 3) then
we can multiply on the left by an orthogonal matrix Q, and cancel out this effect by multiplying the
output matrix (Wdown,Wout) by Q⊤. This applies despite the fact that RMSNorm is applied between
the two blocks, so long as no re-scaling happens in the RMSNorm block (and in practise, we absorb
any re-scaling into adjacent weight matrices first). Conceptually, this is because RMSNorm divides
the activations by their norm, and applying a rotation Q to the activations does not affect the norm.
4
X x
∥x∥ diag(α)
Wq
Wv
Wk
RoPE Multi-head
Attention
Wout Y
KV-cache
RMSNorm
Attention
Figure 3: Flow diagram of a self-attention block as used in most LMs, including the pre-positioned
RMSNorm. Solid arrows represent flow during training, prefill and inference of each token. Dashed
arrows show access to and from the KV cache, used at generation-time. The RoPE block computes
relative positional embeddings.
We have the commutation property
RMSNorm(X) = RMSNorm(XQ⊤)Q, (3)
where we assume here that RMSNorm applied to each row of the activations X as xi ← xi/∥xi∥.
This means that multiplying an output matrix by Q⊤ makes the linear layer output XQ⊤, which is
normalized and then passed into the next block whose input weight matrix is nowQW, and so this
linear layer outputs the original activations without modification.
4 Method
QuaRot consists of two stages. In the first stage, the model weights are manipulated (in full
precision), and two additional Hadamard operations are inserted into the model’s forward pass. In the
second stage, the weights are quantized using some existing method, and quantization operations
are added to the forward pass to enable on-line quantization of the activations. By default, we use
GPTQ for quantizing weights, whilst activations are quantized on-the-fly using a simple round-tonearest
scheme. Figures 4 and 5 show updated block diagrams for the forward pass with QuaRot
modifications, including updated weight matrices, inserted blocks and the bit-width of weights and
activations.
Stage 1a: Weight Modification. We first make use of computational invariance to multiply each
weight matrix by an orthogonal matrix. To enable this, the linear parts of LayerNorm or RMSNorm
are fused into adjacent weight matrices. Figure 4 shows how the feed-forward block of a transformer
is modified by removing the scaling operation from RMSNorm (diag(α)) and absorbing into the
subsequent weight matrices. We select a randomized Hadamard matrix with size that matches the
hidden dimension of the model and pre- or post-multiply each weight matrix. In Figures 4 and 5 this
matrix is denoted Q. For example the key-projection weight matrixWk is modified as
Wk ← Q⊤diag(α)Wk , (4)
and similarly for other weight matrices. Matrices that appear on the output side of a block are
post-multipled by Q.
This weight modification does not affect the output of the model (assuming sufficient precision)
as per the computational invariance theorem [Ashkboos et al., 2024]. We note that the modified
weights resemble the modifications used in QUiP# [Tseng et al., 2024], reducing the incoherence
of the weights, though our modification does not require any additional processing at run-time.
Additionally, the activation matrix passed between blocks of the transformer is also incoherence
processed, becoming X ← XQ. Figure 1 shows the result of this processing: we see that the
processed activations no longer contain any outliers.
5
XQ
FP16
x
∥x∥
quantize
Q⊤(α)Wgate
INT4
Q⊤(α)Wup
INT4
σ
×
hadamard
quantize
HWdownQ
INT4
YQ
FP16
INT4
FP16
FP16
INT4
RMSNorm
FFN
Figure 4: QuaRot applied to a FFN component. The RMSNorm scaling (α) has been absorbed
into the weight matrices. The hidden state X has been rotated by Q, which is cancelled out by
the absorption of Q⊤ into the first two weight matrices. All weights are stored in INT4, and all
activations immediately before the weights are also quantized to INT4. The result of the matmul
between the INT4 weights and INT4 activations on a TensorCore is INT32, which we immediately
cast (and scale) to FP16. Whilst the signal is still in FP16, we perform a single on-the-fly Hadamard
transform before quantizing and computing a (modified) down-proj, which results in a rotated output
YQ.
XQ
FP16
x
∥x∥
quantize
Q⊤(α)Wq
INT4
Q⊤(α)WvHhead
INT4
Q⊤(α)Wk
INT4
RoPE Multi-head
Attention
hadamard heads
quantize
HWoutQ
INT4
YQ
FP16
KV-cache
INT4 hadamard
quantize
dequantize
INT4 hadamard
FP16
FP16
FP16
INT4
RMSNorm
Attention
Figure 5: QuaRot applied to an attention component. The RMSNorm scaling α is absorbed into the
input weight matrices, and the hidden state has been rotated by Q in the same way as for the FFN
block (see previous figure). Colored labels show the bit-width of each flow, and dashed lines show
the flow to/from the KV cache.
Stage 1b: Rotate FFN activations. With the above weight-modifications in place, we have
multiplied many weight matrices on one side by a Hadamard matrix, and changed the computed
activations between blocks of the transformer. It still remains to improve the quantize-ability of the
activations within each block, which we achieve by inserting on-line Hadamard operations.
We first insert a Hadamard operation into the feed-forward network, before the down-projection
matrix. This operation is performed in full precision, and implemented using a fast kernel following
[Tseng et al., 2024]. This operation is implicitly reversed by fusing a Hadamard matrix into the
down-projection matrix of the network: Wdown ← HWdown. Combined with the global matrix Q,
this means that the down-projection matrix now becomesHWdownQ (see Figure 4).
Stage 1c: Attention Value Projection. Next, we apply an additional Hadamard operation to each
attention block. This modification is partially on-line, and partially fused into the weight matrices as
we will now detail.
6
First, note that in the computation of attention, theWv andWout matrices are implicitly multiplied
together within each head. To see this, note that the attention computation consists of
Y = concat[(P1V1) . . . (PnhVnh)]Wout (5)
=
XH
h=1
PhXW(h)
v W(h)
out (6)
where Ph is a sequence-length sized square matrix computed by softmaxing keys and values, and
Vh = XW(h)
v is the value matrix for one head. This presents an opportunity to perform additional
processing onWv andWout using a Hadamard matrix Hdh which matches the dimension of each
head:
W(h)
v ←W(h)
v Hdh, W(h)
out ← HdhW(h)
out . (7)
Substituting these modifications into equation 6, we see that the computed result of attention remains
unchanged. Since the weights for each head are concatenated in the weight representation, we can
equivalently perform a single Kronecker structured multiplication:
Wv ←Wv(I ⊗Hdh), Wout ← (I ⊗Hdh)Wout . (8)
This transformation has now been applied head-wise to the weight matrices, and results in computed
activations (emitted by the block multi-head attention) rotated head-wise also. To complete a “full”
Hadamard operation on the attention-activations, sharing the transform across heads, we make use of
the identity
Hnh×dh = (I ⊗Hdh)(Hnh ⊗ I) (9)
which holds when the number of heads nh and the dimension of each head dh are both powers of
2. Since we have already applied (I ⊗ Hdh) to bothWv andWout, it remains to apply (Hdh ⊗ I)
toWout, which results in a complete transformation ofWout ← HWout, and to insert a block into
the forward pass that computes Z ← Z(Hnh ⊗ I) where Z is the attention activation. This block is
denoted Hadamard heads in Figure 5 and can be computed efficiently using a reshape to deal with
the Kronecker structure, and a Walsh-Hadamard transform on the reshaped data.
Stage 1d: Key Rotation. Using the method above, we can successfully quantize the value vectors.
However, key vectors in the attention module are also known to suffer from outliers []. Similar to
above, we can use a Hadamard rotation to alleviate this issue, allowing us to have a fully quantized
KV-cache. First note that the attention scores P1, . . . ,Ph are computed as:
Q := Pos(XWq) =: concat[Pos(Q1), . . . ,Pos(Qnh)] (10)
K := Pos(XWk) =: concat[Pos(K1), . . . ,Pos(Knh)] (11)
Ph := Softmax(α Pos(Qh)Pos(KTh
) ⊙M) , (12)
where α is the Softmax scale usually set to √1
dh
, M is the attention mask (e.g. causal), and Pos
denotes the positional embedding. Previously, positional embedding was only added before the first
layer to the input, in which case Pos is an identity function. However, recent methods such as RoPE
[Su et al., 2021] add position information directly to the key and query vectors.
We can now observe the same interaction between Q and K as we observed betweenWv andWout.
However, the existence of Pos prevents us from directly fusing the Hadamard matrix intoWq and
Wk. Therefore, we use online head-wise Hadamard rotation to rotate both the queries and keys. As a
result, the computation of query and key matrices is altered as follows:
Q ← Pos(XWq)(I ⊗Hdh) = concat[Pos(Q1)Hdh, . . . ,Pos(Qnh)Hdh] (13)
K ← Pos(XWk)(I ⊗Hdh) = concat[Pos(K1)Hdh, . . . ,Pos(Knh)Hdh] . (14)
Since both queries and keys are rotated, the final attention scores P1, . . . ,Ph remains unchanged.
We note that an alternative to the above process is caching the keys before applying the positional
encoding. This approach (called Pre-RoPE Caching [Hooper et al., 2024]) needs the inverse rotation
to be applied online before applying the positional encoding but removes the need to rotate the query
vector. It also adds the overhead of rotating the keys and values for every query. Given that at the
7
time of decoding there is a single query vector and many cached key vectors, we decided to use
Post-RoPE caching.
Overall, our modifications to the forward pass, including the insertion of special Hadamard blocks
and adjustments to the weights do not change the forward pass of the model. The effect is that the
activations between blocks have been multiplied by a Hadamard matrix, and the activations within
blocks are processed on-line using Hadamard transforms in a way that is undone by corresponding
weight matrix modifications. We are now ready to quantize the weights and activations.
Stage 2a: Weight Quantization. We apply GPTQ [Frantar et al., 2022] to quantize the weights of
the network. We note that after the above forward-pass modifications, any quantization method could
be applied. In subsequent sections, we show that a simple round-to-nearest (RTN) scheme can be
applied instead of GPTQ, at the cost of some accuracy.
Stage 2b: Online Quantization Operations. With the weights quantized, we are ready to apply
operations to the forward pass that quantize the activations. We deliberately leave the computation of
RMSNorm (without scaling) in FP32. We quantize the input of the linear layers using symmetric
per-token (rows of the input matrix). During symmetric quantization, the row scales are computed by
dividing the maximum absolute value of each token by 7. We then divide each row to its corresponding
scale and round the result to its nearest integer. The dequantization is also done by casting the INT32
output of GEMM into FP16, multiply the corresponding scale for the row (from input scales) and
column (from weight scales).
Stage 2c: Quantized Attention. Attention is significantly memory bound for longer sequences and
larger batch sizes. Having rotated both keys and values, we can successfully quantize the cache into
low bit-width. This reduces the number of IO operations needed. We keep the queries in FP16 and
use online softmax calculation similar to Flash Attention [Dao et al., 2022]. After a segment of the
key and value vectors are loaded from the memory, we dequantize them and compute the dot product
in FP16.
5 Experimental Validation
Setup. We implement QuaRot using Hugging Face [Wolf et al., 2019] on top of the PyTorch
framework [Paszke et al., 2019]. To quantize the inputs, we use per-token symmetric quantization (a
single scale for every row) with a constant clipping ratio of 0.9 in all our experiments. We quantize
the KV caches using asymmetric quantization with a group size 128 with a constant clipping ratio
of 0.95. For weight quantization, we use round-to-nearest (RTN) and GPTQ [Frantar et al., 2022]
with per-column (also known as per-channel) symmetric quantization, where we extract the clipping
ratio using a linear search over the squared error. We use 128 samples from WikiText-2 [Merity et al.,
2016] training set with 2048 sequence length as the calibration set during GPTQ quantization. On a
single NVIDIA A100 GPU, modifying LLAMA2-70B with QuaRot takes 5 minutes and quantizing
the model with GPTQ takes a further 2 hours.
Models, Tasks, and GPUs. We evaluate QuaRot on the LLAMA-2 family [Touvron et al., 2023] on
both language generation and zero-shot tasks. We implement our low-level CUDA kernel to perform
4-bit matrix-multiplication using the CUTLASS [NVIDIA, 2023] library. We use the FlashInfer [Ye,
2023] library for implementing our KV cache quantization. As we target consumer-type GPUs, we
evaluate all the performance experiments on NVIDIA RTX 3090 GPUs.
5.1 Accuracy Results
Language Generation Tasks. First, we evaluate the accuracy of QuaRot on the language generation
task. Table 1 shows the perplexity of LLAMA-2 models on WikiText-2 when we quantize the weights
using GPTQ. We compare against 4-bit SmoothQuant [Xiao et al., 2023] and OmniQuant [Shao
et al., 2023]. We also include the QUIK [Ashkboos et al., 2023] results when they keep all the
layers (including down-projection) in 4 bits. QuaRot outperforms all previous work with at most
0.63 perplexity loss (0.47 on LLAMA2-70B model). QuaRot does not need re-training steps (as in
OmniQuant) nor higher precision outlier features and asymmetric quantization (as in QUIK). We also
apply group-wise quantization to compare against Atom [Zhao et al., 2023] on the same number of
8
groups for weight and activations. In this setting, QuaRot doesn’t need to keep any higher precision
features and related operations (like re-ordering). QuaRot outperforms Atom with 0.1 perplexity
points in the 7B model. On the 13B model, we get the same perplexity number as Atom. Our results
demonstrate the effectiveness of applying the Hadamard transformation in suppressing the outliers on
the generation task.
Table 1: WikiText-2 perplexity results on 4-bit quantization of LLAMA-2 models with 2048 sequence
length. We extract the results for SmoothQuant and OmniQuant results from [Shao et al., 2023].
128G shows the group-wise quantization with group size 128.
Method Weight #Outlier LLAMA-2
Quantization Features 7B 13B 70B
Baseline - - 5.47 4.88 3.32
SmoothQuant RTN 0 83.12 35.88 -
OmniQuant RTN 0 14.26 12.30 -
QUIK-4B GPTQ 256 8.87 7.78 6.91
QuaRot GPTQ 0 6.10 5.40 3.79
Atom-128G GPTQ-128G 128 6.03 5.26 -
QuaRot-128G 0 5.93 5.26 3.61
Zero-Shot Tasks. Next, we focus on evaluating QuaRot on six important zero-shot tasks: PIQA
[Bisk et al., 2020],WinoGrande [Sakaguchi et al., 2021], HellaSwag [Zellers et al., 2019], LAMBADA
(OpenAI) [Radford et al., 2019], and Arc (Easy and Challenge) [Clark et al., 2018]. We use the LM
Evaluation Harness [Gao et al., 2021] with default parameters for our experiments. Table 2 shows
the accuracy of our scheme on the above tasks as well as the average score. QuaRot preserves the
accuracy with at most 4.18% average score loss (1.09% for 70B model).
Table 2: Zero-shot accuracy of LLAMA-2 models with 4-bit QuaRot on PIQA (PQ), WinoGrande
(WG), HellaSwag (HS), Arc-Easy (A-e), Arc-Challenge (A-c), and LAMBADA (LA).
Model Method PQ WG HS A-e A-c LA Avg.
7B FP16 79.11 69.06 75.99 74.58 46.25 73.90 69.82
QuaRot 76.77 63.77 72.16 69.87 40.87 70.39 65.64
13B FP16 80.47 72.22 79.39 77.48 49.23 76.75 72.59
QuaRot 78.89 70.24 76.37 72.98 46.59 73.67 69.79
70B FP16 82.70 77.98 83.84 80.98 57.34 79.58 77.07
QuaRot 82.43 76.24 81.82 80.43 56.23 78.73 75.98
5.2 Performance Analysis
We implement QuaRot using CUDA/12.1 on top of PyTorch and use CUTLASS for performing INT-4
matrix multiplication on TensorCore (where the results will be saved in an INT32 accumulator). We
implement the attention mechanism using three routines: 1) Init: During the prefill stage, this routine
initializes the cache from all the key and value vectors in the prefill. The attention output during
prefill is computed directly using Flash Attention [Dao et al., 2022] since we already have access to
dequantized keys and values. 2) Append: During decoding, this routine is called first to quantize
the current keys and values and append them to the cache. 3) Decode: Finally, this routine is called
during decoding with the current query vector. The routine computes the attention output using a
quantized implementation of flash attention which can load the quantized cache and compute the
final value vector. In this section, we evaluate the performance of our kernels separately. Then, we
evaluate the prefill and decoding stage performances.
4-bit Linear and Attention Layers. First, we benchmark our 4-bit linear layer which involves
4-bit matrix multiplication. For a given input of FP16, the layer optionally computes the Hadamard
operation, then calls the quantization kernel to quantize and save the input in a sub-byte format. In the
9
7B 13B 70B
0
5
10
2.57
3.98
12.45
0.75 1.06
2.88
0.8 1.14
2.91
Runtime (ms)
FP16
INT4
INT4 + Had.
Figure 6: Performance of 16-bit and 4-bit linear layer for 2048 sequence lengths with and without
online Hadamard transformation on a NVIDIA RTX 3090 GPU, averaged over 1000 runs. The matrix
sizes correspond to the linear layer sizes in LLAMA-2 FFN blocks (i.e. Wdown). Here the batch size
is 1, but the performance ratio holds for larger batches: see Appendix A.2 for detailed results.
next step, the quantized weights and input are passed to the CUTLASS 4-bit GEMM kernel. Finally,
the output is dequantized and cast back to FP16. Figure 6 shows the speedup of our 4-bit layer for
different layer sizes where the layer sizes match the FFN linear layer sizes in LLAMA-2 models.
Our 4-bit linear layer gets 3.2x speedup relative to FP16 in LLAMA2-7B model, and 4.3x on the
LLAMA2-70B model. These numbers are for a batch size of 1, we find that scaling is approximately
linear with batch size: more results in Table 12. We include the runtime with and without Hadamard
operations, asWup andWgate do not require Hadamard transforms, whilstWdown does. We see that
the Hadamard transform adds very little overhead to the forward pass at most 7% overhead.
We also compare the speed of performing append and decode routines for a single token given a
cache of size 2047. This is equivalent to the cost of decoding the 2048-th token in a sequence. The
comparison between the speed of FP16 and INT4 for different batch sizes and layer sizes is reported
in Table 13. For the layer size used in LLAMA2-7B , our 4-bit implementation gets up to 1.72x
improvement in speed for the larger batch sizes (e.g. from 16 onwards). The 4-bit cache is slower
than FP16 for smaller batch sizes (e.g. up to 8). Note that this is intuitive as the main benefit of the
4-bit cache is reducing the I/O cost. A speed up is only visible if this reduction is more significant
than the quantization overhead which happens for either larger batch sizes or longer sequences.
Prefill and Decoding Stages Performance increases. Finally, we evaluate the improvements in
memory and computation from using QuaRot at different phases of inference for the LLAMA2-7B
model. We divide our evaluations into two separate parts: First, we evaluate the speedup of QuaRot
during the prefill (time-to-first-token) phase which is usually compute-bound. Table 3 shows the
speedup of QuaRot during prefill with various batch sizes. QuaRot achieves up to 2.16x speedup
on LLAMA2-7B . This is less than the 3.2x achieved by the linear layers alone since the majority of
attention computations is still done in FP16 or FP32 even though the smaller cache size reduces IO
demand. We expect further improvement by specialized attention and more kernel optimizations. We
note that QuaRot’s prefill also includes quantizing the KV cache which is needed for the decoding
stage.
In addition to the speedup gained during pre-fill, we observed significant reductions in memory usage
up to 3.71x during decoding with QuaRot. This improvement comes from both quantizing the KV
cache as well as quantizing the model’s weights. To demonstrate the reduced memory usage, we
present the peak memory during one step of decoding (after pre-fill has finished) in Table 4. In all
cases, we observe a significant reduction in memory usage. This reduction is especially important on
consumer hardware, such as on an nVidia RTX 3090 GPU where using QuaRot allows inference at
2048 sequence length which would be impossible otherwise.
10
Batch Size Speedup
1 1.97x
4 2.06x
16 2.11x
64 2.16x
Table 3: Time-to-first-token (prefill)
speedup of each transformation block
of LLAMA2-7B model in QuaRot
(over the FP16 model) on NVIDIA
RTX 3090 GPU. We use 2048 sequence
lengths with different batch
sizes.
Batch Sequence Baseline QuaRot Saving
Size Length (GB) (GB) Factor
1
256 12.75 3.66 3.48x
512 12.88 3.69 3.49x
1024 13.13 3.76 3.49x
2048 13.64 3.89 3.50x
4096 14.72 4.22 3.49x
16
256 14.66 4.16 3.52x
512 16.69 4.69 3.56x
1024 20.76 5.76 3.60x
2048 28.88 7.88 3.66x
4096 45.20 12.19 3.71x
Table 4: Peak Memory usage (in GB) for decoding a single
token using LLAMA2-7B model with KV caches of different
lengths and with different batch size.
5.3 Ablation Studies
To evaluate different aspects of QuaRot, we investigate the role of applying Hadamard transformation
on the Weight-only Quantization schemes as well as using Random Orthogonal Matrices instead
of Hadamard matrices. We also evaluate the accuracy of our quantized models when we apply FP16
Hadamard Transformation. At the quantization level, we evaluate the use of Round-to-Nearest
Weight Quantization and Group-wise Quantization (with different group sizes). Finally, we
evaluate using different precision for KV cache Quantization.
Weight-only Quantization. QuaRot improves the quality of quantized models by removing the
outlier features during the Hadamard transformations. As we fuse the Hadamard matrices into
the weights, we study the role of these transformations for weight-only quantization (we keep the
rest of the data-types in FP16). Table 5 shows the WikiText-2 perplexity results with asymmetric
quantization. Using GPTQ quantization, QuaRot improves the perplexity by up to 2.65 in 4 bits.
In addition, applying QuaRot improves the quality more in lower precision (2-3 bits) in all models.
QuaRot also improves the RTN quantization up to 0.24 perplexity points. GPTQ still has a lower
perplexity in 2-3 bits. However, applying QuaRot improves the quality of GPTQ in 2 bits to a
non-trivial value (5.6 on the LLAMA2-70B model).
Table 5: Weight-only quantization results on WikiText-2 on LLAMA-2 models. We use asymmetric
per-column quantization and keep the inputs and KV cache in FP16. We show the perplexity results
>100 by Inf. We show the failed GPTQ experiments using NaN.
Method LLAMA-2
7B 13B 70B
Baseline 5.47 4.88 3.32
A16W4 A16W3 A16W2 A16W4 A16W3 A16W2 A16W4 A16W3 A16W2
RTN 6.99 Inf Inf 6.32 Inf Inf 4.45 42.11 Inf
GPTQ 8.25 NaN NaN 5.65 9.51 Inf 3.87 5.91 25.30
QuaRot-RTN 6.76 Inf Inf 5.48 48.89 Inf 3.66 5.25 Inf
QuaRot-GPTQ 5.60 6.09 22.07 5.00 5.37 10.41 3.41 3.72 5.60
11
Random Orthogonal Matrices. QuaRot fuses
Hadamard transformations into weight matrices to
eliminate outliers. However, due to the computational
invariance property in LLMs, any orthogonal
matrix can be fused to the model and we only need
to apply an online 11
2 Hadamard transformations
in each layer (see Section 4). Here, we study the
use of random orthogonal matrices in QuaRot. We
start with a uniformly random matrix and apply
QR decomposition to make it orthogonal before
fusing it into the weights.
Table 6: WikiText-2 perplexity of 4-bit QuaRot
on LLAMA-2 models with different orthogonal
matrices.
Method LLAMA-2
7B 13B 70B
Baseline 5.47 4.88 3.32
QuaRot (Random) 7.45 5.84 4.07
QuaRot (Hadamard) 6.10 5.40 3.79
Table 6 shows the results of applying random orthogonal matrices on LLAMA-2 models. Random
orthogonal matrices are not as good as random Hadamard transformations and we have up 1.35
perplexity gap on LLAMA2-7B . However, as the model size increases, the gap decreases, resulting
in a perplexity change of 0.28 in the LLAMA2-70B model. Note that using the above matrices does
not change the computation as we still use a fast Hadamard kernel for the down-projection and
out-projection layers.
FP16 Hadamard Transformation. We use FP32 online Hadamard transformation across all
our experiments. Table 7 shows the results of using FP16 Hadamard transformation during the
inference (for down-projection and out-projection layers). On LLAMA2-7B model, the results show
<0.1 perplexity change on WikiText-2 and <0.6% averaged accuracy change on the zero-shot tasks,
which we consider as noise. On LLAMA2-13B model, different Hadamard precisions have the same
perplexities with 0.07% difference in the averaged zero-shot results. We conclude that the model will
not be changed using different Hadamard precision.
Table 7: Ablation on the precision of online Hadamard transformations for QuaRot. We useWikiText-
2 perplexity as well as zero-shot tasks, explained in 5.3.
Model Method Hadamard PPL ↓ PQ ↑ WG ↑ HS ↑ A-e ↑ A-c ↑ LA ↑ Avg. ↑ Precision
7B
Baseline - 5.47 79.11 69.06 75.99 74.58 46.25 73.90 69.82
QuaRot FP32 6.10 76.77 63.77 72.16 69.87 40.87 70.39 65.64
FP16 6.08 76.99 66.46 72.59 69.07 41.21 70.59 66.21
13B
Baseline - 4.88 80.47 72.22 79.39 77.48 49.23 76.75 72.59
QuaRot FP32 5.40 78.89 70.24 76.37 72.98 46.59 73.67 69.79
FP16 5.40 77.69 70.09 75.75 73.95 47.61 73.22 69.72
Round-to-NearestWeight Quantization. GPTQ is our default choice for weight quantization in
QuaRot. Here, we study the role of quantizing the weights using Round-to-Nearest (RTN). Table
8 shows that applying GPTQ and RTN during the weight quantization maintains 99% and 98%
respectively of the FP16 model accuracy in both 6 and 8 bits. In 4 bits, the gap between QuaRot-RTN
and QuaRot-GPTQ decreases when the model size is increased (2.27 on LLAMA2-7B and 0.34 on
LLAMA2-70B ) showing that GPTQ is a better option in smaller models.
12
Table 8: WikiText-2 Perplexity and zero-shot accuracy of QuaRot on the LLAMA-2 family using 4, 6
and 8-bits with GPTQ and RTN weight quantization and RTN activation quantization. For zero-shot
tasks, we use PIQA (PQ), WinoGrande (WG), HellaSwag (HS), Arc-Easy (A-e), Arc-Challenge
(A-c), and LAMBADA (LA).
Model Method Precision PPL ↓ PQ ↑ WG ↑ HS ↑ A-e ↑ A-c ↑ LA ↑ Avg. ↑
7B
Baseline FP16 5.47 79.11 69.06 75.99 74.58 46.25 73.90 69.82
QuaRot-RTN
INT4 8.37 72.09 60.69 65.40 58.88 35.24 57.27 58.26
INT6 5.56 78.73 67.80 75.92 74.16 46.08 73.86 69.42
INT8 5.50 78.94 68.67 75.80 74.79 45.39 74.33 69.65
QuaRot-GPTQ
INT4 6.10 76.77 63.77 72.16 69.87 40.87 70.39 65.64
INT6 5.52 78.45 69.46 75.60 74.45 46.50 74.19 69.77
INT8 5.50 78.94 68.90 75.79 74.66 46.16 74.44 69.81
13B
Baseline FP16 4.88 80.47 72.22 79.39 77.48 49.23 76.75 72.59
QuaRot-RTN
INT4 6.09 77.37 67.32 73.11 70.83 43.69 70.66 67.16
INT6 4.95 79.65 72.22 79.10 77.27 50.34 76.75 72.56
INT8 4.90 80.52 71.59 79.38 77.31 49.32 76.63 72.46
QuaRot-GPTQ
INT4 5.40 78.89 70.24 76.37 72.98 46.59 73.67 69.79
INT6 4.92 79.98 72.69 79.17 77.78 49.74 76.27 72.60
INT8 4.90 80.36 71.98 79.38 77.31 49.15 76.79 72.49
70B
Baseline FP16 3.32 82.70 77.98 83.84 80.98 57.34 79.58 77.07
QuaRot-RTN
INT4 4.14 80.69 75.14 79.63 77.57 51.71 77.02 73.63
INT6 3.36 83.24 77.90 83.47 80.93 58.28 79.41 77.21
INT8 3.33 82.97 77.98 83.67 80.77 58.11 79.53 77.17
QuaRot-GPTQ
INT4 3.79 82.43 76.24 81.82 80.43 56.23 78.73 75.98
INT6 3.35 82.13 77.66 83.63 80.89 57.08 79.70 77.02
INT8 3.33 83.13 78.06 83.72 80.85 58.19 79.72 77.28
Group-wise Quantization. Table 9 shows
the accuracy of applying QuaRot with various
group-sizes for the activations and weights.
The results show a clear trade-off between the
accuracy and the group-sizes: smaller group-sizes
give better accuracy (but require more bits to
store scales for each group and more complex
matrix-multiplication kernels).
KV Cache Quantization. We keep the rest of
the model (including weights and activations) in
high precision and apply our group-wise asymmetric
quantization (with group-size 128) with various
precision to keys and values. Table 10 shows the
results of using various precision during KV cache
quantization. The results show a negligible (at
most 0.21) perplexity degradation up to 3-bit KV
cache (0.07 for LLAMA2-70B model). In addition,
by comparing the 3 and 4-bit quantization,
we can see that compared to the values, keys are
more sensitive to quantization as keeping the keys
in 4-bits and values in 3-bits has 0.03 perplexity
loss (0.18 for 3-bit keys and 4-bit values) on the
LLAMA2-7B model. This matches the previous
study on KV cache quantization [Hooper et al.,
2024, Liu et al., 2024]. The results show that using
3-bit KV-caches results in a better accuracy
(5.68 on LLAMA2-7B model) compared to keeping
the keys in 4-bits and quantizing the values
using 2-bits (with 5.75 perplexity on LLAMA2-7B
model).
Table 9: WikiText-2 perplexity of 4-bit QuaRot
with various group-sizes on LLAMA-2 models.
We use GPTQ during the weight quantization. In
all cases, we keep the KV cache group-size to
128 (which is the same as the head dimension).
Method LLAMA-2
7B 13B 70B
Baseline 5.47 4.88 3.32
QuaRot 6.10 5.40 3.79
QuaRot-256G 5.98 5.28 3.63
QuaRot-128G 5.93 5.26 3.61
QuaRot-64G 5.88 5.25 3.58
Table 10: WikiText-2 perplexity with various KV
cache precision using QuaRot.
K bits V bits LLAMA-2
7B 13B 70B
16 16 5.47 4.88 3.32
4 4 5.51 4.91 3.33
4 3 5.54 4.93 3.35
4 2 5.75 5.09 3.43
3 4 5.65 5.01 3.38
3 3 5.68 5.02 3.39
3 2 5.93 5.21 3.48
2 4 8.06 6.42 3.89
2 3 8.18 6.50 3.92
2 2 9.23 7.07 4.13
13
6 Conclusion
We introduce QuaRot: a method which uses Hadamard matrices to eliminate outliers in the activations
and KV cache of pre-trained LLMs, enabling end-to-end 4-bit quantization for the first time (to
the best of our knowledge). Quantizing LLAMA2-70B to 4 bits with QuaRot maintains 99% of the
downstream task performance of the FP16 baseline, with a 2.16x speedup on RTX 3090 GPUs during
the prefill stage (and up to 3.39x memory saving during the decoding stage). Quantizing all LLAMA-2
models to 6- and 8-bits is lossless.
Opportunities to build on QuaRot include quantizing the residuals and extending the method to
mixture-of-experts architectures. In terms of hardware, end-to-end INT4 inference with QuaRot
could be exploited to give similar speedups as that of the recently announced NVIDIA B200 GPU
architecture, while being much cheaper to implement compared to the floating point (FP4) format.
References
Saleh Ashkboos, Ilia Markov, Elias Frantar, Tingxuan Zhong, Xincheng Wang, Jie Ren, Torsten
Hoefler, and Dan Alistarh. Towards end-to-end 4-bit inference on generative large language models.
arXiv preprint arXiv:2310.09259, 2023.
Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James
Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv
preprint arXiv:2401.15024, 2024.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning
about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial
Intelligence, 2020.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantization of
large language models with guarantees. Advances in Neural Information Processing Systems, 36,
2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and
Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.
ArXiv, abs/1803.05457, 2018. URL https://api.semanticscholar.org/CorpusID:
3922816.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and
memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing
Systems, 2022.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix
multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:
30318–30332, 2022.
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos,
Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized representation
for near-lossless llm weight compression. arXiv preprint arXiv:2306.03078, 2023.
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan
Alistarh. Extreme compression of large language models via additive quantization. arXiv preprint
arXiv:2401.06118, 2024.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training
quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence
Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot
language model evaluation. Version v0. 0.1. Sept, 2021.
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao,
Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length llm inference with
kv cache quantization. arXiv preprint arXiv:2401.18079, 2024.
14
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activationaware
weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978,
2023.
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi
Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint
arXiv:2402.02750, 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture
models, 2016.
NVIDIA. Nvidia cutlass library, 2023. URL https://github.com/NVIDIA/cutlass/.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor
Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style,
high-performance deep learning library. Advances in neural information processing systems, 32,
2019.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language
models are unsupervised multitask learners. 2019.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An
adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106,
2021.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang,
Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large
language models. arXiv preprint arXiv:2308.13137, 2023.
Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang,
Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large
language models with a single gpu. In International Conference on Machine Learning, pages
31094–31116. PMLR, 2023.
Neil J A Sloane. A library of hadamard matrices, 2024. URL http://neilsloane.com/
hadamard/.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer
with rotary position embedding. CoRR, abs/2104.09864, 2021. URL https://arxiv.org/abs/
2104.09864.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay
Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian
Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu,
Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn,
Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel
Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee,
Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra,
Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, <PRESIDIO_ANONYMIZED_PERSON>,
Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh
Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen
Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic,
Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models,
2023.
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#:
Even better llm quantization with hadamard incoherence and lattice codebooks. arXiv preprint
arXiv:2402.04396, 2024.
XiuyingWei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei
Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language
models. Advances in Neural Information Processing Systems, 35:17402–17414, 2022.
15
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi,
Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface’s transformers:
State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant:
Accurate and efficient post-training quantization for large language models. In International
Conference on Machine Learning, pages 38087–38099. PMLR, 2023.
Zihao Ye. FlashInfer: Kernel Library for LLM Serving. https://github.com/flashinfer-ai/
flashinfer, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine
really finish your sentence? arXiv preprint arXiv:1905.07830, 2019.
Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind
Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and
accurate llm serving. arXiv preprint arXiv:2310.19102, 2023.
16
A Appendix
A.1 Clipping Ratio
We use the clipping ratio for both weights and activations during the quantization. During the weight
quantization, we apply a linear search over the MSE error to extract the best clipping ratio for each
column of the weight matrix. However, this is not possible as we quantize the inputs on the fly during
the inference and we need to use a constant clipping ratio for such quantization. We conclude that
using 0.95 and 0.9 are suitable during asymmetric (KV cache) and symmetric (inputs) quantization
which matches the finding from [Zhao et al., 2023].
Table 11: WikiText perplexity of LLAMA2-7B with different clipping ratio. To study the effect of
various clipping ratios, we keep the rest of the model in full precision.
1.0 0.95 0.9 0.85
Input Quantization 5.938 5.910 5.828 5.850
KV Cache Quantization 5.513 5.510 5.517 5.532
A.2 Performance Analysis: Detailed Results
Table 12 shows the results of benchmarking our 4-bit linear layer. The layer sizes are extracted
based on the linear layer sizes in LLAMA-2 models (for out-projection and down-projections). We
apply both FP16 and FP32 Hadamard transformations and show the runtime on NVIDIA RTX GPU
using 2048 sequence lengths. Table 13 shows the results of decoding a single token in the attention
layer when we apply KV-cache quantization. We extract the size of the attention layer based on the
LLAMA-2 models.
17
Layer Size Batch Size FP16 INT4 INT4 + FP32 Had INT4 + FP16 Had
4096x4096
1 1.043 0.370 0.409 0.403
2 1.902 0.696 0.790 0.789
4 3.715 1.361 1.522 1.529
8 7.200 2.675 2.999 3.011
16 14.508 5.357 5.973 5.976
32 29.029 10.641 11.900 11.911
5120x5120
1 1.418 0.464 0.552 0.547
2 2.918 0.937 1.100 1.097
4 5.852 1.888 2.206 2.207
8 11.465 3.809 4.428 4.422
16 22.807 7.547 8.755 8.759
32 45.312 15.019 17.417 17.440
8192x8192
1 3.696 0.997 1.084 1.083
2 7.191 1.944 2.099 2.099
4 14.236 3.918 4.208 4.207
8 28.508 7.944 8.460 8.415
16 57.814 15.793 16.859 16.871
32 115.462 31.693 33.780 33.791
11008x4096
1 2.569 0.749 0.798 0.801
2 5.027 1.478 1.555 1.558
4 9.752 2.990 3.140 3.144
8 19.696 6.031 6.296 6.306
16 38.883 11.978 12.503 12.527
32 78.320 23.874 24.935 24.974
13824x5120
1 3.983 1.063 1.142 1.139
2 7.869 2.148 2.291 2.293
4 15.410 4.340 4.616 4.614
8 30.761 8.719 9.231 9.240
16 61.203 17.318 18.345 18.343
32 122.926 34.816 36.953 36.940
28672x8192
1 12.450 2.881 2.911 2.911
2 25.391 5.828 5.892 5.896
4 50.742 11.938 11.947 11.976
8 101.290 24.186 24.202 24.216
16 202.909 48.238 48.325 48.356
32 406.344 96.761 97.044 96.892
Table 12: Performance of 4-bit linear layer for 2048 sequence lengths with and without online
Hadamard transformation on a NVIDIA RTX 3090 GPU. The matrix sizes correspond to the linear
layer sizes in LLAMA-2 models. We averaged over 100 runs and report the numbers in milliseconds.
18
head_num x head_dim Batch Size FP16 INT4 INT4 + FP32 Had INT4 + FP16 Had
32x128
1 0.713 1.033 1.163 1.117
2 0.723 1.035 1.168 1.122
4 0.781 1.033 1.168 1.118
8 0.984 1.042 1.173 1.126
16 1.348 1.018 1.153 1.102
32 2.098 1.168 1.247 1.216
40x128
1 0.712 1.026 1.157 1.106
2 0.726 1.035 1.173 1.121
4 0.831 1.038 1.166 1.115
8 1.065 1.048 1.181 1.128
16 1.525 1.021 1.153 1.102
32 2.480 1.244 1.320 1.287
64x128
1 0.715 1.028 1.160 1.108
2 0.780 1.034 1.171 1.117
4 0.984 1.034 1.171 1.120
8 1.361 1.048 1.182 1.130
16 2.071 1.147 1.223 1.192
32 3.563 1.566 1.645 1.612
Table 13: Performance of decoding a single token with 4-bit KV cache for the attention layer for
2048 sequence lengths with and without online Hadamard transformation on an NVIDIA RTX 3090
GPU. We evaluate generating the last token when the 2047 tokens are already cached in the attention.
We extract the number of heads (head_num) and their dimensions (head_dim) based on different
LLAMA-2 models. We averaged over 100 runs to report the numbers in milliseconds.
19