Pyramid Vector Quantization for LLMs

## Abstract

Abstract Recent works on compression of large language models (LLM) using quantization considered reparameterizing the architecture such that weights are distributed on the sphere. This demonstratively improves the ability to quantize by increasing the mathematical notion of coherence, resulting in fewer weight outliers without affecting the network output. In this work, we aim to further exploit this spherical geometry of the weights when performing quantization by considering Pyramid Vector Quantization (PVQ) for large language models. Arranging points evenly on the sphere is notoriously difficult, especially in high dimensions, and in case approximate solutions exists, representing points explicitly in a codebook is typically not feasible due to its additional memory cost. Instead, PVQ uses a fixed integer lattice on the sphere by projecting points onto the 1-sphere, which allows for efficient encoding and decoding without requiring an explicit codebook in memory. To obtain a practical algorithm, we propose to combine PVQ with scale quantization for which we derive theoretically optimal quantizations, under empirically verified assumptions. Further, we extend pyramid vector quantization to use Hessian information to minimize quantization error under expected feature activations, instead of only relying on weight magnitudes. Experimentally, we achieves state-of-the-art quantization performance with pareto-optimal trade-off between performance and bits per weight and bits per activation, compared to compared methods. On weight-only, we find that we can quantize a Llama-3 70B model to 3.25 bits per weight and retain 98% accuracy on downstream tasks.

## 1 Introduction

Quantization enables compression of large language models (LLMs) by reducing the number of bits per weight required to represent weights. Weight outliers can make quantization difficult, as they cause weight distributions to not match the implicit evenly distributed grids used in many quantization methods. To overcome this, recent works have proposed to reparameterize architectures by rotating weights in a way that leaves the network as a function unchanged (Ashkboos et al., ; Chee et al., ).

In this work, we aim to exploit the spherical geometry in weights when performing quantization by considering Pyramid Vector Quantization (PVQ) (Fischer, ) for LLMs. In PVQ, weights are quantized on a hyper-pyramidal lattice that allows efficient encoding and decoding without having to explicitly represent a codebook in memory. By projecting the lattice onto the hyper-sphere, a quantization grid is obtained that accurately approximates a uniform grid on the spherical domain.

PVQ has been been very successful in well-known audio (Valin et al., ) and video codecs (Daede et al., ). We demonstrate that the same algorithm allows practical quantization of large language models, by proposing a group-wise quantization scheme and further extending PVQ to use Hessian information accounting for curvature in the loss. We also propose a scheme to quantize the normalized scales (gains) of each group according to theoretically derived quantiles, which we verified to closely match the empirical weight distributions of pretrained LLMs in practice. Our contribution extends beyond Liguori (), who considered PVQ on weights of small neural networks.

Experimentally, we find that our proposed PVQ quantization scheme outperforms the state-of-the-art in terms of bits per weight and bits per activation. We do not only perform simulated quantization, but also provide kernels that allow hardware accelerated encoding and decoding of PVQ. We achieve state-of-the-art quantization on the most prominent Llama-3, Phi-3 and Mistral architectures in terms of performance against bits per weight (BPW). In particular, we demonstrate 3.25 bit weight quantization at a negligible 1-3% drop in performance, as measured in accuracy on downstream tasks.

## 2 Background

Before discussing our approach on using pyramid vector quantization to quantize LLMs, we provide an overview of vector quantization, spherical geometry of LLM weights, and describe classic PVQ.

### 2.1 Quantization

Quantization is a compression technique for machine learning models by storing weights (${\bm{W}}$) or activations (${\bm{x}}$) in a some chosen lower bit representation, such as lower precision floats or scaled integers ($\widehat{\bm{W}}$). A common conversion is to minimize a second-order layer-wise proxy loss (Nagel et al., ; Frantar et al., ),

$$ $\displaystyle L(\widehat{{\bm{W}}})=\mathbb{E}_{{\bm{x}}}\left[||{\bm{W}}{\bm{ x}}-\widehat{{\bm{W}}}{\bm{x}}||_{2}\right]=\mathbb{E}_{{\bm{x}}}\left[ \operatorname{Tr}\left(({\bm{W}}-\widehat{{\bm{W}}}){\bm{H}}({\bm{W}}-\widehat {{\bm{W}}}\right)\right]$ (1) $$

where the layer-wise Hessian ${\bm{H}}=\mathbb{E}_{{\bm{x}}}\left[{\bm{x}}{\bm{x}}^{T}\right]=\sum_{n=1}^{N}
{\bm{x}}_{n}{\bm{x}}_{n}^{T}$ is typically empirically approximated using a calibration dataset $\{{\bm{x}}_{n}\}_{n=1}^{N}$. The objective is optimal in the sense that it minimizes the layer-wise output at each layer, but a crude approximation with respect to the actual training loss. Some recent works have proposed to also use gradients to improve the approximation to the true training objective (van der Ouderaa et al., ). Although it would be interesting to combine ideas presented in this paper with gradients, we stick to a layer-wise loss Equation 1 for simplicity and because this yields a faster quantization method that does not require backpropagation.

### 2.2 Vector quantization

Instead of individually quantizing weights, as done in scalar quantization, vector quantization aims to simultaneously quantize multiple weights. It can be shown that, even for completely independent Gaussian sources, this typically results in much higher theoretical signal-to-noise ratios, leading recent works to consider vector quantization for LLMs (van Baalen et al., ; Liu et al., ). Yet, vector quantization is not widely adopted in practice because of two practical problems. Firstly, naive vector quantization requires constructing an explicit codebook using clustering (such as K-means), quickly becoming infeasibly large for higher number of dimensions. Secondly, quantization requires an expensive search over this explicit codebook, which can not practically be used on-the-fly on activations. This limits VQ to weights-only quantisation. Pyramid vector quantization is a vector quantization method that solves these issues simultaneously by being codebook-free and search-free, unlike standard vector quantization. It uses an implicit codebook that does not have to be instantiated in memory to perform quantization, and can encode and decode vectors without having to search it. As a result, it can be applied on-the-fly and not only on the neural network weights, but also to the activations during inference.

### 2.3 Weights on the sphere

Instead of viewing weight vectors in Euclidean $D$-space ${\bm{w}}\in\mathbb{R}^{D}$, they can be interpreted as scaled points ${\bm{w}}=s{\bm{v}}$ on the unit sphere ${\bm{v}}\in\Omega_{D}$, with $\Omega_{D}=\{{\bm{v}}=(v_{1},v_{2},\ldots,v_{D})\in\mathbb{R}^{D}:\|{\bm{v}}\|
_{2}=1\}$. We refer to this spherical decomposition as the shape ${\bm{v}}=\frac{{\bm{w}}}{\|{\bm{w}}\|_{2}}$ and the gain $s=\|{\bm{w}}\|_{2}$ of a vector. Recent works that use such a spherical perspective on LLM weights have offered new insights in properties of the training dynamics and guide algorithmic improvements. For example, (Kosson et al., ) noted that LLM weights under weight decay or popular deep learning optimizers converge to an equilibrium on the sphere, theoretically predicting the magnitude of the gain after training. Some recent works have shown that only updating shape components can be beneficial in low-rank adaptation Liu et al. () and training itself (Loshchilov et al., ). Yet, weights are not always uniformly distributed making quantization hard and giving rise to outliers. We use recent work on coherence processing that reparameterize weights on the sphere in a way that reduces outliers without functionally changing the output of the network. We then extend existing quantization approaches by considering pyramid vector quantization to construct a spherical quantization code.

#### Hadamard coherence processing

Recent works have shown that by considering rotations of the weight matrices ${\bm{W}}\in\mathbb{R}^{R\times C}$, it is possible to improve weight coherence Chee et al. (), thereby diminishing the number of outliers without altering the network’s output,
${\bm{W}}={\bm{U}}{\bm{W}}{\bm{V}}$
where ${\bm{U}}\in\mathbb{R}^{R\times R}$ and ${\bm{V}}\in\mathbb{R}^{C\times C}$ are both orthogonal matrices. To perform a forward pass this reparameterization must be undone, $\widehat{W}={\bm{V}}^{T}\widehat{{\bm{W}}}{\bm{U}}^{T}$. To make this efficient, Chee et al. () proposed to use random Hadamard matrices for ${\bm{U}}$ and ${\bm{V}}$.

Representing weights on the sphere through incoherence processing has shown to improve existing quantization methods by preventing outliers in weight distributions. Yet, these methods still quantize weights to Euclidean grids and do not exploit the spherical geometry of underlying weight distributions. We explore pyramid vector quantization to construct a quantization grid that is tailored to the spherical geometry by being approximately uniform on the sphere.

#### Absorbing rotation matrices

In most cases, the additional rotation matrices do not result in a memory or compute overhead during inference since rotation matrices can be absorbed into weight matrices: depending on the placement of the rotation matrix in the architecture, it can be left- or right- multiplied with an adjacent weight matrix. This principle was used in QuaRot (Ashkboos et al., ), to which we refer for a more detailed description of this for attention and fully-connected layers found in most common LLM architectures including specifics around absorbing rotation matrices in the presence of activation functions and normalization layers.

### 2.4 Classic Pyramid Vector Quantization

Figure: Figure 1: Illustration of the PVQ integer lattice in $d=3$ dimensions with increasing pulses $k$ from 1 to 6. Points on the pyramid $P_{3,k}$ are projected onto the sphere $S_{3,k}$.
Refer to caption: x1.png

To quantize points on the sphere, we would like to construct spherical code, a finite subset on the unit sphere $S\subset\Omega_{D}$. Packing a set of points on the surface of a sphere such that their distance is maximised is a notoriously hard problem in mathematics, famously dating back to the Dutch botanist Tammes (). Just like sphere packing, optimal spherical codes are not generally known, with the exception of some specific dimensions, similar to the E8 packing in 8 dimensions exploited in (Tseng et al., ). Even though good but sub-optimal spherical codes exist (Conway & Sloane, ), there is not always an efficient method to enumerate the packing without requiring an explicit codebook, making use for quantization costly. PVQ (Fischer, ) provides a solution to both of these issues simultaneously, allowing good spherical codes to be constructed in arbitrary dimension that can be efficiently encoded and decoded without having to maintain an explicit codebook in memory. This is achieved by projecting an integer lattice on the L1 ball $P_{D,K}$ onto the hypersphere $S_{D,K}$. We now provide a brief overview of the classic PVQ algorithm.

#### The integer pyramid lattice

Formally, the integer lattice of PVQ on the $D$-dimensional hyper-sphere is obtained by starting from a set of points on the L1 ball of radius $K$, and projecting the set of points onto the sphere. We denote the set of integer points on the L1 ball as $P_{D,K}$,

$$ $\displaystyle P_{D,K}=\left\{{\bm{p}}\in\mathbb{Z}^{D}:||{\bm{p}}||_{1}=\sum_{ d=1}^{D}|{\bm{p}}_{d}|=K\right\}$ (2) $$

To obtain our spherical code $S_{D,K}$, we can project the points $P_{D,K}$ onto the sphere, by normalizing,

$$ $\displaystyle S_{D,K}=\left\{\frac{{\bm{p}}}{||{\bm{p}}||_{2}}:{\bm{p}}\in P_{ D,K}\right\}\,.$ (3) $$

The number of codes $N(D,K)=|S_{D,K}|=|P_{D,K}|$ can be written as

$$ $\displaystyle N(D,K)=2D\cdot{}_{2}F_{1}(1-D,1-K,2;2)$ (4) $$

where
${}_{2}F_{1}(a,b,c;z)$ is the hypergeometric function (Terriberry, ), and for specific $D,K\in\mathbb{Z}$ can be computed through the following recurrence relation,

$$ $\displaystyle N(D,K)=N(D-1,K)+N(D,K-1)+N(D,K)$ (5) $$

where we define $N(D,0)=1$ for all $D\geq 0$ and $N(0,K)=0$ for all $K\geq 1$.

### 2.5 Subroutines of classic PVQ

#### Quantizing the shape

To quantizing a vector ${\bm{w}}\in\mathbb{R}^{D}$, we map it to the closest point on the pyramid $P_{D,K}$ using an iterative procedure that projects the vector ${\bm{w}}_{0}={\bm{w}}$ onto the L1 ball of radius $K$, and round it to the closest integer,

$$ $\displaystyle{\bm{w}}_{t+1}=\text{quantize_step}_{K}({\bm{w}}_{t})=\text{ round}\left(\frac{K}{||{\bm{w}}_{t}||_{1}}{\bm{w}}_{t}\right)$ (6) $$

After this step, we check whether the norm satisfies $||{\bm{w}}||_{1}=K$. If this is the case, we’re done. If not, we either decrease by 1 or increase by 1, the element in ${\bm{w}}_{t}$ with the biggest deviation $|{\bm{w}}_{i}|-K$. We then call $\text{quantize_step}_{K}(\cdot)$ again, and repeat this process until convergence, which should happen within at least $T<D$ steps. The resulting vector lies on the pyramid ${\bm{w}}_{T}\in P_{D,K}\subset\mathbb{Z}^{D}$, meaning it is integer-valued and the absolute values sum to 1. We provide pseudocode of the quantization algorithm in Algorithm 2 in Section B.2 to be fully self-contained.

Figure: Figure 2: Illustration of points on pyramid $P_{2,7}$, their projections onto the sphere $S_{2,7}$ and codes in $C_{2,7}$.
Refer to caption: x2.png

#### Pyramid encoding

We can encode points on the pyramid ${\bm{p}}\in P_{D,K}$ to integer codes $c\in[1,\ldots,N(D,K)-1]$ using an algorithm, which avoids having to build an explicit table in memory to represent the codebook. The original PVQ paper (Fischer, ) describes an encoding scheme, of which we provide pseudocode Algorithm 2 in Section B.2, for the purpose of being self-contained. The algorithm provides a bijective mapping from the points of the pyramid $P_{D,K}$ to integer indices $C_{D,K}$, providing a compact representation that allow vectors to be efficiently stored in as few bits as possible. In Figure 2, we provide an example illustration of points on $P_{2,7}$ and indices $C_{2,7}=[0,27]$. The table can be precomputed a table of $N(d,k)$ for $0\leq d\leq D$ and $0\leq k\leq K$ and reused to avoid recomputing the same quantity.

#### Pyramid decoding

We decode the integer codes $c\in C$ of PVQ back to the associated vector ${\bm{p}}\in P_{D,K}$ by performing a decoding algorithm which performs the inverse operation of the encoding algorithm above. In Section B.3, we provide a corrected version of the decoding algorithm described in the original PVQ paper (Fischer, ). The original paper contains an error and misses a line after setting $x_{i}\leftarrow-j$, resulting in wrong decodings when ${\bm{p}}$ contains negative values, except for when the last value is negative (which happens to be the case for the example given in the original paper).

## 3 PVQ for LLM compression

Before we present our overall algorithm to perform PVQ to LLMs in Section 3.3, we discuss the motivations and practical benefits of PVQ for LLM quantization in Section 3.1 and analyse the theoretical signal-to-noise ratio of PVQ Section 3.2.

### 3.1 Why use PVQ for LLM quantization?

PVQ offers several practical advantages over competing methods. Firstly, PVQ is a vector quantization method which means it can achieve higher signal-to-noise ratios than scalar quantization methods that quantize weights independently. Secondly, PVQ is codebook-free, which means that it does not require an explicit codebook to be constructed in memory. This makes the approach more memory efficient, but more importantly, the implicit codebook size can reach far beyond the memory that would have been required with an explicit codebook. To illustrate, explicitly storing a codebook that quantizes 16 bit precision vectors of a groupsize of 128 to 4 bit per weight would require approximately $2.7\cdot 10^{154}$ bytes, exceeding the estimated information capacity of the observable universe. With PVQ, we can use implicit codebooks of this size because encoding and decoding are done by an algorithm, not a table lookup. Thirdly, PVQ is search-free, which means that vectors can be encoded without having to search over a codebook. This is significant apart from the computational benefits, because it allows for on-the-fly quantization of activations and opens the door to quantization at train time. Lastly, the desired bits per weight after quantization can also be fractions (e.g. $b_{\text{shape}}=3.5$) and are not limited to integers. The ability to choose the groupsize, and bits for the shape make PVQ highly flexible and can be chosen such that the most optimal trade-off between compression and performance is achieved. We find that PVQ outperforms competitive quantization methods in terms of weight-only and activation quantization.

In this work, our focus is on post-training quantization of weights and activations. Some current works quantize LLMs during training (e.g. Ma et al., ). We anticipate that PVQ could be used during training because of the advantages outlined above, though we leave a thorough investigation and comparisons to future work.

Figure: Figure 3: Signal-to-noise-ratio of weight and activation quantization methods on a Gaussian source. PVQ achieves similar high signal-to-noise as state-of-the-art E8 weight-only vector quantization. As PVQ is search-free and codebook-free, it can practically be used to quantize both weight and activations.
Refer to caption: x3.png

### 3.2 Signal-to-noise on ideal Gaussian source

To get an idea of how effective different quantization algorithms are, we start by comparing their performance on ideal Gaussian source data. We measure the signal-to-noise ratio by averaging the mean squared error between true and quantized signal over 1000 samples in Figure 3. We find that PVQ achieves high signal-to-noise ratios on Gaussian data equivalent to E8, a state-of-the-art quantization method that exploits geometric structure. PVQ achieves especially superior signal-to-noise ratios for activation quantization, in which case we compare to RTN without search as a baseline that is suitable to activation quantization also.

### 3.3 Pyramid vector quantization for LLMs

This section describes the overall method for quantizing an LLM using PVQ.

#### Step 0. Choose the desired BPW.

In PVQ, the trade-off between performance and the effective bits per weight (BPW) can be controlled through the groupsize $D$, shape bits $b_{\text{shape}}$ and gain bits $b_{\text{gain}}$:

$$ $\displaystyle\text{BPW}=b_{\text{shape}}+b_{\text{gain}}/G$ (7) $$

The average shape bits $b_{\text{shape}}$ are not required to be integer, and can be a fractional (e.g. $b_{\text{shape}}=3.5$).

#### Step 1. Coherence processing.

Before we begin quantizing the weights, we perform coherence processing using efficient Hadamard rotation matrices proposed in (Chee et al., ), which can be fused into the architecture as described in QuaRot(Ashkboos et al., ).

#### Step 2. Determine the number of pulses K 𝐾 K italic_K .

To determine the number of pulses $K$ such that the number of bits required to encode PVQ vectors remains within the desired maximum, we find the largest $K$ by increasing it such that it still satisfies $\left\lceil\log_{2}(N(D,K))/D\right\rceil<b_{\text{shape}}$. Here, $N(D,K)$ is computed using the recursive algorithm outlined in Section 2.4. In practice, the number $N(D,K)$ can exceed regular integer types and may require arbitrary precision integers.

#### Step 3. Quantizing the shape.

We individually quantize all weight matrices in the LLM that make up the key, query, value and fully-connected matrices. We can write the normalized weight matrix ${\bm{W}}=\begin{bmatrix}{\bm{W}}_{1}&{\bm{W}}_{2}&\ldots&{\bm{W}}_{G}\end{
bmatrix}\in\mathbb{R}^{N\times GD}$ in terms of $G$ groups with a groupsize $D$, where $N$ denotes the number of features. Given a value of $K$, we quantize a group ${\bm{W}}_{g}\in\mathbb{R}^{N\times D}$ by repeatedly performing $\text{quantize_step}_{K}$ on row vectors of ${\bm{W}}_{g}$ until convergence and increase or decrease the largest deviation $|{\bm{w}}_{i}|-K$, as described in Section 2.4. This results in a quantized shape matrix $\widehat{{\bm{W}}}_{g}$ on the pyramid $P_{D,K}$, which means that the elements are rounded integers and absolute values of rows sum to 1. The operation can be parallelised over features and groups.

#### Step 4. Computing the gain.

For each quantized row vector $\widehat{{\bm{w}}}\in\mathbb{R}^{D}$ in a group, we can find the optimal scale parameter $s\in\mathbb{R}$ that minimizes the squared error to the original weight matrix ${\bm{w}}$ in closed-form $s=\widehat{{\bm{w}}}^{T}\widehat{{\bm{w}}}/\widehat{{\bm{w}}}^{T}{\bm{w}}$. Repeating this operation across features and groups results in a gain matrix ${\bm{S}}\in\mathbb{R}^{N\times G}$.
We refer to the quantized $\widehat{{\bm{W}}}$ as the ‘shape’ and the scales ${\bm{S}}$ as the ‘gain’.

#### Step 5. Quantizing the gain (optional)

For small groupsizes, we may also consider quantizing the gain using the approach discussed in Section 3.4. For each row ${\bm{s}}\in\mathbb{R}^{G}$ in ${\bm{S}}$, this entails quantizing normalized elements $\lfloor\text{CDF}(s_{i}^{2}/||{\bm{s}}||_{2}^{2})\cdot 2^{b}\rfloor$ using the CDF of the $\text{Beta}(D/2,D(G-1)/2)$ distribution. The normalizing constant $||{\bm{s}}||_{2}^{2}$ needs to be stored for dequantization, but because it is shared across groups its contribution to the total bits per weight $<0.01$ is negligibly small.

#### Step 6. Correcting the quantization error.

After quantizing each group of weights $\widehat{{\bm{W}}}_{g}$, we can perform a partial update to the remaining weights ${\bm{W}}_{>g}:=\begin{bmatrix}{\bm{W}}_{g+1}&{\bm{W}}_{g+2}\ldots{\bm{W}}_{G}
\end{bmatrix}\in\mathbb{R}^{N\times(G-g)D}$ to ‘compensate’ the error of the quantization ${\bm{W}}_{g}$ that best minimizes our proxy loss $L$ of Equation 1. To compute the update, we use the transpose of the inverse Cholesky decomposition ${\bm{L}}^{-T}$ of the Hessian ${\bm{H}}={\bm{L}}{\bm{L}}^{T}$.

$$ $\displaystyle{\bm{W}}_{>g}\leftarrow({\bm{W}}_{g}-{\bm{s}}_{g}^{T}..\text{( think about most elegant notation here)}...\widehat{{\bm{W}}}_{g})...{\bm{L}}^ {-T}....$ (8) $$

The correction was originally proposed in the context of scalar quantization in GPTQ (Frantar et al., ), but is also applicable in pyramid vector quantization, as we demonstrate.

#### Step 7. Encoding the shape.

To encode a quantized weight $\widehat{{\bm{W}}}\in\mathbb{R}^{N\times GD}$ (e.g. 32 bit float for $\mathbb{R}$) into the actual low bit integer representation $\overline{{\bm{W}}}\in\mathbb{N}^{N\times G}$, sometimes referred to as the ‘code’, we can use the PVQ encoding algorithm described in Algorithm 2. In practice, the amount of bits required to represent a code may grow beyond the maximum integer precision of many languages (e.g. when $\lceil\log_{2}(N(D,K))\rceil>128$ bits, it can not be represented in a 128 bit integer). We implemented a hardware kernels for encoding and decoding using custom arithmetic precision arithmetic in CUDA/C++.

### 3.4 Gain quantization

#### Normalized gains follow the Beta distribution.

Smaller groupsizes lead to a lower quantization error, but lead to an increase in the number of bits required to store gain parameters. This is because the gain ${\bm{S}}\in\mathbb{R}^{N\times G}$ grows linearly with the number of groups $G$, and therefore inversely proportional to the chosen groupsize $D$. To overcome this issue, we propose a theoretically and empirically motivated scheme to quantize gain parameters. In Theorem 3.1, we note that row vectors of normalised gains follow a Beta distribution $\text{Beta}(\frac{D}{2},\frac{D(G-1)}{2})$ of which the coefficients depend on the groupsize $D$ and the number of groups $G$. We empirically confirm that our theory meets practice, as we find that the Beta distribution matches normalized weight distributions of pretrained LLM models, after performing the rotation described in Section 2.3, as shown for selection of layers in a pretrained LLM in Figure 4. To exploit the observation that gains are Beta distributed, we propose to use the quantiles of this distribution to quantize gains.

Figure: Figure 4: Effect of different shape and gain bits on effective PVQ quantization grid. The distribution automatically matches the theoretical Beta weight distribution.
Refer to caption: x4.png

###### Theorem 3.1

Let ${\bm{w}}\in\mathbb{R}^{GD}$ be a normally distributed vector that can be grouped in $G$ equally sized vectors ${\bm{w}}=\begin{bmatrix}{\bm{v}}_{1}&{\bm{v}}_{2}&\cdots&{\bm{v}}_{G}\end{bmatrix}$ where each of the vectors ${\bm{v}}_{g}$ has the same dimensionality equal to the groupsize ${\bm{v}}_{g}\in\mathbb{R}^{D}$. Then the normalised radius (the ‘gain’) of each group $s_{h}={\bm{v}}_{g}^{T}{\bm{v}}_{g}/||{\bm{w}}||_{2}^{2}$ follows the $s_{g}\sim\text{Beta}(\frac{D}{2},\frac{G(D-1)}{2})$ distribution. (Proof in Section A.1)

Figure: Figure 5: Theoretical Beta distribution of Theorem 3.1 closely match gains of rotated weights in trained LLMs. Here demonstrated for empirical weight distributions of a pretrained Llama-v3 8B.
Refer to caption: x5.png

Figure: Figure 6: Quantiles of Beta(2, 6).
Refer to caption: x6.png

###### Theorem 3.1

#### Quantizing gains using Beta quantiles.

The observation that normalized gains are Beta distributed is important, as it suggests that gains can effectively be quantized with the quantiles of this Beta distribution, without having to introduce additional hyper-parameters. Effectively, $b$ bit quantization using quantiles amounts to quantizing on a regular uniform grid of $2^{b}$ points after first transforming elements through a change-of-variables given by the CDF of the Beta distribution and dequantizing using the inverse CDF (see Figure 6):

$$ $\displaystyle\text{quantize}(x)$ $\displaystyle=\lfloor\text{CDF}(x)\cdot 2^{b}\rfloor$ (9) $\displaystyle\text{dequantize}(x)$ $\displaystyle=\text{PPF}((x+0.5)/2^{b})$ (10) $$

where $\lfloor\cdot\rfloor$ denotes the floor function, and $\text{CDF}/\text{PPF}$ denote the cumulative density function of the Beta distribution and its inverse, the percentile point function.

## 4 Results

### 4.1 Weight-only quantization (without gain quantization)

Since most prior work focuses on weight-only quantization, we begin by comparing the performance of weight-only PVQ quantization with common weight-only quantization baselines. Following prior work (Frantar et al., ; Ashkboos et al., ), we measure test perplexity (PPL) and average accuracy on a range of zero-shot downstream tasks after quantizing common open-source LLM models of Phi, Mixtral and Llama families. We compare groupsizes in [16, 32, 64, 128, 256], keep the gain in 16 bit, and use shape bits in [3, 3.5, 4, 4.5, 5, 5.5, 6, 7, 8] for PVQ and [3, 4, 5, 6, 7, 8] for other methods. PVQ is the only quantization method that supports non-integer shape bits.

Figure: Figure 7: Weight only quantization. Test perplexities (PPL) after quantizing with different methods at group size settings (within connected set) and various shape bits (between connected sets).
Refer to caption: x7.png

**Table 1: Weight-only quantization in sub-4 bits. Post-quantization test perplexity (PPL) and average zero-shot (Avg. Acc) performance. PVQ yields the highest performance after quantization. Details and additional results in Section E.1. nan^∗ indicates GPTQ fails due to non-psd Hessian.**
|  |  |  |  |  |  | Phi-3-mini-4k | Mixtral-8x7B | Llama-3-8B | Llama-3-70B |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method |  | Hessian | Spherical | Groupsize | BPW | PPL ($\downarrow$) | Avg. Acc ($\uparrow$) | PPL ($\downarrow$) | Avg. Acc ($\uparrow$) | PPL ($\downarrow$) | Avg. Acc ($\uparrow$) | PPL ($\downarrow$) | Avg. Acc ($\uparrow$) |
| Original |  |  |  |  | 16 | 6.01 | 0.72 | 3.84 | 0.78 | 6.13 | 0.73 | 2.85 | 0.80 |
| RTN |  |  |  | 128 | 3.125 | 19.03 | 0.53 | 8.95 | 0.66 | 29.41 | 0.41 | 487.94 | 0.45 |
| GPTQ |  | ✓ |  | 128 | 3.125 | 7.36 | 0.65 | 8.40 | 0.52 | 17.77 | 0.40 | nan^∗ | nan^∗ |
| QuaRot |  | ✓ | ✓ | 128 | 3.125 | 7.17 | 0.67 | 4.29 | 0.76 | 7.62 | 0.69 | 5.14 | 0.76 |
| PVQ |  | ✓ | ✓ | 128 | 3.125 | 6.85 | 0.68 | 4.20 | 0.77 | 7.01 | 0.72 | 4.82 | 0.77 |

### 4.2 Weight-only quantization (shape and gain quantization)

A benefit of PVQ is that the number of bits for the shape and bits for the gains can be chosen flexibly, even to not-integer ratios. To evaluate the effect of different gain bits, we repeat the same experiment as before, but rather than varying the groupsize, we fix the groupsize to 16 and vary the amount of bits used for the gain.

Figure: Figure 8: Shape and gain quantization. Test perplexities (PPL) and test accuracies (Avg. Acc ) at different bits for gains (within connected sets) and various shape bits (between sets).
Refer to caption: x8.png

**Table 2: Quantizing shape and gain. We compare post-training perplexity (PPL) and average zero-shot performance (Avg. Acc). PVQ yields the highest performance after quantization. Details and additional results in Section E.2.**
|  |  |  |  |  |  | Phi-3-mini-4k | Mixtral-8x7B | Llama-3-8B | Llama-3-70B |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method |  | Groupsize | Hessian | Spherical | BPW | PPL ($\downarrow$) | Avg. Acc ($\uparrow$) | PPL ($\downarrow$) | Avg. Acc ($\uparrow$) | PPL ($\downarrow$) | Avg. Acc ($\uparrow$) | PPL ($\downarrow$) | Avg. Acc ($\uparrow$) |
| Original |  |  |  |  | 16.00 | 6.01 | 0.72 | 3.84 | 0.78 | 6.13 | 0.73 | 2.85 | 0.80 |
| PVQ [2.5 bit shapes, 16 bit gains] |  | 16 | ✓ | ✓ | 3.50 | 7.52 | 0.67 | 4.42 | 0.76 | 8.04 | 0.68 | 5.92 | 0.74 |
| PVQ [3 bit shapes, 4 bit gains] |  | 16 | ✓ | ✓ | 3.25 | 6.85 | 0.69 | 4.22 | 0.76 | 7.14 | 0.70 | 4.51 | 0.78 |

### 4.3 Weights and Activation quantization

As PVQ is codebook-free and search-free, it can
be applied to not only the weights but also the activations during inference. In Figure 13 we evaluate the final test perplexity for different settings of effective bits per weight and bits per activations. In Table 3, we compare the resulting test perplexities to other methods for quantizing weight and activation. We also include quantization methods that are not codebook-free, by relying on round-to-nearest (RTN) to quantize activations in real-time during inference. We find that PVQ obtains state-of-the-art performance across the considered LLM architectures, and that this holds generally across different settings of bit rates.

Figure: Figure 9: Weights and activations. Comparing test perplexity at different bits per weight and bits per activations. From minimal compression (top right) to high levels of compression (bottom left).
Refer to caption: x9.png

**Table 3: Quantizing weight and activation in 4 bits. We compare perplexity (PPL) and average zero-shot performance (Avg. Acc) after quantizing different open source LLM models using various post-training quantization methods. PVQ yields the highest performance after quantization. Additional results in Section E.3.**
| Method |  |  |  |  |  |  | Phi-3-mini-4k | Mixtral-8x7B | Llama-3-8B |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Weights | Activations |  | Groupsize | Hessian | Spherical | BPW | BPA | PPL ($\downarrow$) | Avg. Acc ($\uparrow$) | PPL ($\downarrow$) | Avg. Acc ($\uparrow$) | PPL ($\downarrow$) | Avg. Acc ($\uparrow$) |
| Original | - |  |  |  |  | 16 | 16 | 6.01 | 0.72 | 3.84 | 0.78 | 6.13 | 0.73 |
| GPTQ | RTN |  | 128 | ✓ |  | 4.125 | 4.125 | 8.36 | 0.63 | 7.41 | 0.59 | 4747.68 | 0.36 |
| QuaRot | RTN |  | 128 | ✓ | ✓ | 4.125 | 4.125 | 7.48 | 0.67 | 4.43 | 0.75 | 7.34 | 0.70 |
| PVQ | RTN |  | 128 | ✓ | ✓ | 4.125 | 4.125 | 7.37 | 0.66 | 4.40 | 0.76 | 7.16 | 0.70 |
| PVQ | PVQ |  | 128 | ✓ | ✓ | 4.125 | 4.125 | 6.94 | 0.68 | 4.57 | 0.75 | 6.89 | 0.71 |

### 4.4 Downstream zero-shot tasks

In Table 4, we provide results on downstream zero-shot tasks split out per task. We report weight-only PVQ with both shape and gain quantization on a Llama-3-8B model. Additional results including other LLM models can be found in Section E.4.

**Table 4: Performance on downstream tasks. We compare performance on zero-shot downstream tasks after quantizing weights using different weight quantization methods.**
|  |  |  |  |  |  | Llama-3-8B |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method |  | Groupsize | Hessian | Spherical | BPW | PPL $\downarrow$ | PQ $\uparrow$ | WG $\uparrow$ | HS $\uparrow$ | A-e $\uparrow$ | A-c $\uparrow$ | LA $\uparrow$ | Avg. $\uparrow$ |
| Original |  |  |  |  | 16.00 | 6.01 | 0.81 | 0.73 | 0.78 | 0.79 | 0.57 | 0.65 | 0.72 |
| RTN |  |  |  |  | 4.00 | 8.39 | 0.77 | 0.68 | 0.74 | 0.76 | 0.53 | 0.55 | 0.67 |
| GPTQ |  | 16 | ✓ |  | 4.00 | 6.82 | 0.79 | 0.71 | 0.74 | 0.79 | 0.57 | 0.63 | 0.70 |
| QuaRot |  | 16 | ✓ | ✓ | 4.00 | 6.66 | 0.78 | 0.72 | 0.73 | 0.78 | 0.55 | 0.64 | 0.70 |
| PVQ [3.0 bit shapes, 4 bit gains] |  | 16 | ✓ | ✓ | 3.25 | 6.85 | 0.79 | 0.72 | 0.74 | 0.77 | 0.54 | 0.60 | 0.69 |
| PVQ [3.5 bit shapes, 6 bit gains] |  | 16 | ✓ | ✓ | 3.88 | 6.33 | 0.81 | 0.72 | 0.76 | 0.80 | 0.57 | 0.64 | 0.72 |

### 4.5 Optimizing PVQ for CUDA

For large $D$ and $K$, PVQ results in large precision integer codes $c$, far surpassing the native 32-bit integer operations on CUDA. As highlighted in Section 3.3, even 128-bit operations introduced with CUDA 11.5 are not sufficient. We therefore implement the CUDA kernels for PVQ on custom subroutines for arbitrary precision integer arithmetic relying on PTX instructions using the carry-forward registry (CC.CF) for multi-word integer addition, subtraction and bit-shifting. We use a word-minor memory layout to ensure that the memory access can be coalesced. Table 5 presents the time and I/O complexity for our implementations.

For quantization, encoding and decoding we parallelize the work across the batch dimension. Further optimizations are possible, especially for small batch sizes $B$. In particular, the inner loops of each algorithm (Appendix B) can be further parallelized by first sorting or computing the cumulative sum of $|x_{i}|$ respectively using a CUDA optimized reduction (Harris et al., ).

The recurrent formulation for the size table ($N\left[D,K\right]$) quickly becomes prohibitive as its naive implementation requires $O(D\cdot K\cdot W)$ serial operations, each requiring overlapping reads and writes. Instead, we reformulate the algorithm to perform $K$ operations in parallel using a CUDA optimized Hillis-Steele type scan reduction (Hillis & Steele, ) which accounts for the $\log(K)$ factor in Table 5, computing each row in parallel (Section C.1). This has the further advantage of only having to read from global memory for synchronization following (Xiao & Feng, ), otherwise the threads keep the values in registry and exclusively writes to global memory.

**Table 5: The time and I/O complexity on the HBM for our CUDA optimized PVQ kernels, where $B$ is the batch size and $W$ the number of words comprising each integer. All functions have $O(W)$ space complexity. We highlight ordered I/O operations as this more efficient use of the memory bandwidth as they can be coalesced into fewer operations as opposed to random ones.**
| Function | Time complexity | I/O complexity |
| --- | --- | --- |
| Quantization | $O(B\cdot D)$ | $O(B\cdot D)$ |
| Encoding / Decoding | $O(B\cdot D\cdot W)$ | $O(B\cdot D\cdot W)$ reads, $O(B\cdot D)$ ordered writes |
| Size table generation | $O(K\cdot\log(K)\cdot D\cdot W)$ | $O(K\cdot D\cdot W)$ ordered writes |

## 5 Conclusion

This work explored pyramid vector quantization (PVQ) for quantization of weights and activations in large language models (LLMs). PVQ is a vector quantization method that allows high signal-to-noise ratios without having to build an explicit codebook or perform search. This results in state-of-the-art quantization performance in terms of the most favourable performance to bits-per-weight trade-off, and is amenable to quantization of activation. This has direct practical benefit for post-training model compression, but also opens the door towards quantization at train time. We propose to quantize LLMs using an implicit PVQ codebook on the unit sphere, which can be flexibly configured for codesize and dimensions. On top, we propose a theoretically and empirically motivated way to also quantize gains also enabling small groupsizes in practice. Lastly, we incorporate Hessian information throughout the process to minimize feature error due to quantization. This yields a novel and highly parallelisable algorithm for LLM weights and activations. We demonstrate state-of-the-art quantization performance in terms of superior performance after quantizing pre-trained models, on both weight-only and weight and activation quantization.

## Appendix A Mathematical details

### A.1 Proof that gains are Beta distributed

Assume we have weights ( $w_{1},w_{2},\ldots,w_{N}$) that are normally distributed: $w_{i}\sim\mathcal{N}(0,\sigma^{2})$. The sum of the squares of these weights for a subset of size ( D ) is chi-squared distributed $u=w_{1}^{2}+w_{2}^{2}+\ldots+w_{G}^{2}\sim\sigma^{2}\chi^{2}_{B}$. Now, consider two independent chi-squared distributed variables: $a\sim\sigma^{2}\chi^{2}_{A},b\sim\sigma^{2}\chi^{2}_{B}$. It can be shown that the ratio of these two variables follows a Beta distribution: $\frac{a}{a+b}\sim\text{Beta}\left(\frac{A}{2},\frac{B}{2}\right)$. This result is independent of the scale parameter ( $\sigma^{2}$ ). To normalize the weights in a group, we consider the ratio: $\frac{w_{1}^{2}+w_{2}^{2}+\ldots+w_{D}^{2}}{w_{1}2+w_{2}^{2}+\ldots+w_{N}^{2}}$, where $D<N$. This can be rewritten as: $\frac{w_{1}^{2}+w_{2}^{2}+\ldots+w_{D}^{2}}{(w_{1}2+w_{2}^{2}+\ldots+w_{D}^{2}
)+(w_{D+1}^{2}+\ldots+w_{N}^{2})}$. Given that $(w_{1}^{2}+w_{2}^{2}+\ldots+w_{D}^{2})\sim\sigma^{2}\chi^{2}_{D}$ and $(w_{D+1}^{2}+\ldots+w_{D}^{2})\sim\sigma^{2}\chi^{2}_{N-D}$, the ratio follows a Beta distribution: $\frac{w_{1}^{2}+w_{2}^{2}+\ldots+w_{B}^{2}}{w_{1}2+w_{2}^{2}+\ldots+w_{N}^{2}}
\sim\text{Beta}\left(\frac{D}{2},\frac{N-D}{2}\right)$. Thus, the normalized weights in a group follow a Beta distribution.

## Appendix B Classic PVQ subroutines

The quantization, encoding and decoding algorithms of classic PVQ are provided below. They are equivalent to the algorithms originally proposed in (Fischer, ), and included to be self-contained. We also fixed a small bug in the original description of the decoding algorithm.

### B.1 Algorithm 1: PVQ Quantization

Figure: Algorithm 1 PVQ quantization: ${\bm{v}}\mapsto\widehat{{\bm{v}}},\hskip 10.00002pt\mathbb{R}^{D}\to\mathbb{Z} ^{D}$.

### B.2 Algorithm 2: PVQ Encoding

Figure: Algorithm 2 PVQ encoding: ${\bm{p}}\mapsto c,\hskip 10.00002pt\mathbb{Z}^{D}\to[1,N(D,K)]$.

### B.3 Algorithm 3: PVQ Decoding

Figure: Algorithm 3 PVQ decoding: $c\mapsto{\bm{p}},\hskip 10.00002pt[1,N(D,K)]\to\mathbb{Z}^{D}$.

The decode algorithm Algorithm 3 is akin to that described in the original PVQ paper (Fischer, ) and provided to be self-contained. It also fixes a missing line 23, setting $x_{i}\leftarrow-j$. Not including this line results in wrongly decoded vectors when ${\bm{p}}$ contains negative values, except for when the last value is negative (which is why the example provided in the original paper does not fail).

## Appendix C Implementation details

### C.1 Parallelization using CUDA kernels

Here, we provide additional details for the CUDA implementation described in Section 4.5.

The quantization, encoding and decoding operations described in Appendix B can all be parallelized across the batch dimension. To utilize this, we implementing these operations in custom CUDA kernels. To achieve the time complexities of Table 5 we must re-write the summation in the encoding operation (Algorithm 2, line 7). This is done by in addition to the size table $N$ computing an additional cumulative sum

$$ $V(d,k)=\sum_{i=1}^{k}N(d,i)\quad\forall d=0,\dots,D\,\text{ and }k=0,\dots,K\,.$ (11) $$

Using V, the expression becomes

$$ $\displaystyle c\leftarrow c$ $\displaystyle+N(d-1,k)$ $\displaystyle+2\left(V(d-1,k-1)-V(d-1,k-|x_{i}|)\right)$ $\displaystyle+\frac{1-\text{sgn}(x_{i})}{2}N(d-1,k-|x_{i}|)\,,$ $$

which has constant time and I/O complexity.

Furthermore, the size table function can be parallelized across the $K$ axis, by recognizing that the contributions $N(D-1,K)+N(D,K-1)$ to $N(D,K)$ from Equation 5 correspond to adding the cumulative sum $V(D-1,k)$ to each $k$ at row $D$ which may be done efficiently in CUDA (Harris et al., ). The pseudocode for each thread is shown in 4. Notice how the threads only write to global memory, except for any synchornization reads.

Figure: Algorithm 4 Size table computation.

### C.2 Gain quantization

For scale quantization, we pre-compute a list of 10000 points using scipy.stats.beta.cdf onto GPU. We then directly index neighbouring points on this list which we interpolate linearly to obtain quantized values. To dequantize, we use scipy.stats.beta.ppf to construct an explicit codebook on GPU, which we can index in parallel to perform dequantization.

## Appendix D Empirical gain histograms

To assess how well our theory of theoretical Section 3.4 matches the empircial weight distribuitons of pretrained LLM models, we compare the empirical weight histograms of all layers of a pretrained LLM model with the expected $\text{Beta}\left(\frac{D}{2},\frac{D(G-1)}{2}\right)$ distribution. We consider the weights of a pretrained Llama-v2-7b after coherence processing and provide the histograms of all weight matrices in the model below. We find that the Beta distribution closely matches the empirical weight distributions in practice.

|  |  |  |  |
| --- | --- | --- | --- |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |

|  |  |  |  |
| --- | --- | --- | --- |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |

|  |  |  |  |
| --- | --- | --- | --- |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |

|  |  |  |  |
| --- | --- | --- | --- |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |

## Appendix E Additional results

### E.1 Additional weight-only experiments (shape only)

**Table 6: Performance on downstream tasks. We compare performance on zero-shot downstream tasks after quantizing weights using different weight quantization methods.**
|  |  |  |  |  |  | Phi-3 |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method |  | Groupsize | Hessian | Spherical | BPW | PPL $\uparrow$ | PQ $\uparrow$ | WG $\uparrow$ | HS $\uparrow$ | A-e $\uparrow$ | A-c $\uparrow$ | LA $\uparrow$ | Avg. $\uparrow$ |
| Original |  |  |  |  | 16.000 | 6.01 | 0.81 | 0.73 | 0.78 | 0.79 | 0.57 | 0.65 | 0.72 |
| RTN |  |  |  |  | 3.125 | 19.03 | 0.72 | 0.58 | 0.62 | 0.60 | 0.41 | 0.25 | 0.53 |
| GPTQ |  | 128 | ✓ |  | 3.125 | 7.36 | 0.77 | 0.66 | 0.69 | 0.74 | 0.51 | 0.53 | 0.65 |
| QuaRot |  | 128 | ✓ | ✓ | 3.125 | 7.17 | 0.77 | 0.69 | 0.71 | 0.74 | 0.50 | 0.62 | 0.67 |
| PVQ [3 bit shapes, 16 bit gains] |  | 128 | ✓ | ✓ | 3.125 | 6.85 | 0.79 | 0.71 | 0.72 | 0.76 | 0.51 | 0.62 | 0.68 |

**Table 7: Performance on downstream tasks. We compare performance on zero-shot downstream tasks after quantizing weights using different weight quantization methods.**
|  |  |  |  |  |  | Mixtral 8x7B |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method |  | Groupsize | Hessian | Spherical | BPW | PPL $\uparrow$ | PQ $\uparrow$ | WG $\uparrow$ | HS $\uparrow$ | A-e $\uparrow$ | A-c $\uparrow$ | LA $\uparrow$ | Avg. $\uparrow$ |
| Original |  |  |  |  | 16.000 | 3.84 | 0.84 | 0.76 | 0.84 | 0.83 | 0.60 | 0.78 | 0.78 |
| RTN |  |  |  |  | 3.125 | 8.95 | 0.78 | 0.66 | 0.69 | 0.71 | 0.45 | 0.64 | 0.66 |
| GPTQ |  | 128 | ✓ |  | 3.125 | 8.40 | 0.69 | 0.60 | 0.47 | 0.47 | 0.30 | 0.60 | 0.52 |
| QuaRot |  | 128 | ✓ | ✓ | 3.125 | 4.29 | 0.82 | 0.76 | 0.83 | 0.82 | 0.58 | 0.78 | 0.76 |
| PVQ [3 bit shapes, 16 bit gains] |  | 128 | ✓ | ✓ | 3.125 | 4.20 | 0.83 | 0.76 | 0.82 | 0.81 | 0.58 | 0.79 | 0.77 |

**Table 8: Performance on downstream tasks. We compare performance on zero-shot downstream tasks after quantizing weights using different weight quantization methods.**
|  |  |  |  |  |  | Llama-3-8B |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method |  | Groupsize | Hessian | Spherical | BPW | PPL $\uparrow$ | PQ $\uparrow$ | WG $\uparrow$ | HS $\uparrow$ | A-e $\uparrow$ | A-c $\uparrow$ | LA $\uparrow$ | Avg. $\uparrow$ |
| Original |  |  |  |  | 16.000 | 6.13 | 0.81 | 0.73 | 0.79 | 0.78 | 0.53 | 0.76 | 0.73 |
| RTN |  |  |  |  | 3.125 | 29.41 | 0.64 | 0.55 | 0.42 | 0.41 | 0.25 | 0.22 | 0.41 |
| GPTQ |  | 128 | ✓ |  | 3.125 | 17.77 | 0.63 | 0.59 | 0.35 | 0.43 | 0.26 | 0.17 | 0.40 |
| QuaRot |  | 128 | ✓ | ✓ | 3.125 | 7.62 | 0.77 | 0.71 | 0.73 | 0.75 | 0.46 | 0.71 | 0.69 |
| PVQ [3 bit shapes, 16 bit gains] |  | 128 | ✓ | ✓ | 3.125 | 7.01 | 0.80 | 0.73 | 0.76 | 0.78 | 0.50 | 0.75 | 0.72 |

### E.2 Additional quantizing shape and gain experiments

Figure: Figure 10: Shape and gain quantization. Test perplexities (PPL) with different quantization methods at various bit levels for gains (within connected sets) and for shape bits (between connected sets).
Refer to caption: x234.png

Figure: Figure 11: Shape and gain quantization. Average accuracies (Avg. Acc.) after quantizing with different methods at different bits for gains (within connected set) and various shape bits (between connected sets).
Refer to caption: x235.png

### E.3 Additional weight and activations experiments

Figure: Figure 12: Weights and activations. Comparing test perplexity at different bits per weight and bits per activations. From minimal compression (top right) to high levels of compression (bottom left).
Refer to caption: x236.png

Figure: Figure 13: Weights and activations. Comparing average accuracy at different bits per weight and bits per activations. From minimal compression (top right) to high levels of compression (bottom left).
Refer to caption: x237.png

### E.4 Additional Zero-shot downstream task experiments

**Table 9: Performance on downstream tasks after quantizing different open source LLM models using various post-training quantization methods. PVQ yields the highest performance after quantization.**
|  |  |  |  |  |  | Phi-3 |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method |  | Groupsize | Hessian | Spherical | BPW | PPL $\uparrow$ | PQ $\uparrow$ | WG $\uparrow$ | HS $\uparrow$ | A-e $\uparrow$ | A-c $\uparrow$ | LA $\uparrow$ | Avg. $\uparrow$ |
| Original |  |  |  |  | 16.00 | 6.01 | 0.81 | 0.73 | 0.78 | 0.79 | 0.57 | 0.65 | 0.72 |
| RTN |  |  |  |  | 4.00 | 8.39 | 0.77 | 0.68 | 0.74 | 0.76 | 0.53 | 0.55 | 0.67 |
| GPTQ |  | 16 | ✓ |  | 4.00 | 6.82 | 0.79 | 0.71 | 0.74 | 0.79 | 0.57 | 0.63 | 0.70 |
| QuaRot |  | 16 | ✓ | ✓ | 4.00 | 6.66 | 0.78 | 0.72 | 0.73 | 0.78 | 0.55 | 0.64 | 0.70 |
| PVQ [3.0 bit shapes, 4 bit gains] |  | 16 | ✓ | ✓ | 3.25 | 6.85 | 0.79 | 0.72 | 0.74 | 0.77 | 0.54 | 0.60 | 0.69 |
| PVQ [3.5 bit shapes, 6 bit gains] |  | 16 | ✓ | ✓ | 3.88 | 6.33 | 0.81 | 0.72 | 0.76 | 0.80 | 0.57 | 0.64 | 0.72 |

**Table 10: Performance on downstream tasks after quantizing different open source LLM models using various post-training quantization methods. PVQ yields the highest performance after quantization.**
|  |  |  |  |  |  | Mixtral 8x7B |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method |  | Groupsize | Hessian | Spherical | BPW | PPL $\uparrow$ | PQ $\uparrow$ | WG $\uparrow$ | HS $\uparrow$ | A-e $\uparrow$ | A-c $\uparrow$ | LA $\uparrow$ | Avg. $\uparrow$ |
| Mixtral 8x7B<br>Original |  |  |  |  | 16.00 | 3.84 | 0.84 | 0.76 | 0.84 | 0.83 | 0.60 | 0.78 | 0.78 |
| RTN |  |  |  |  | 4.00 | 4.74 | 0.83 | 0.76 | 0.81 | 0.80 | 0.55 | 0.75 | 0.75 |
| GPTQ |  | 16 | ✓ |  | 4.00 | 4.53 | 0.81 | 0.76 | 0.78 | 0.79 | 0.55 | 0.76 | 0.74 |
| QuaRot |  | 16 | ✓ | ✓ | 4.00 | 4.13 | 0.83 | 0.76 | 0.83 | 0.83 | 0.59 | 0.79 | 0.77 |
| PVQ [3.0 bit shapes, 4 bit gains] |  | 16 | ✓ | ✓ | 3.25 | 4.22 | 0.83 | 0.76 | 0.83 | 0.81 | 0.57 | 0.78 | 0.76 |
| PVQ [3.5 bit shapes, 6 bit gains] |  | 16 | ✓ | ✓ | 3.88 | 3.99 | 0.84 | 0.76 | 0.84 | 0.82 | 0.60 | 0.78 | 0.77 |

**Table 11: Performance on downstream tasks. We compare after quantizing different open source LLM models using various post-training quantization methods. PVQ yields the highest performance after quantization.**
|  |  |  |  |  |  | Llama-3-8B |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method |  | Groupsize | Hessian | Spherical | BPW | PPL $\uparrow$ | PQ $\uparrow$ | WG $\uparrow$ | HS $\uparrow$ | A-e $\uparrow$ | A-c $\uparrow$ | LA $\uparrow$ | Avg. $\uparrow$ |
| Original |  |  |  |  | 16.00 | 6.13 | 0.81 | 0.73 | 0.79 | 0.78 | 0.53 | 0.76 | 0.73 |
| RTN |  |  |  |  | 4.00 | 8.17 | 0.78 | 0.71 | 0.74 | 0.70 | 0.45 | 0.70 | 0.68 |
| GPTQ |  | 16 | ✓ |  | 4.00 | 1415.44 | 0.66 | 0.58 | 0.36 | 0.57 | 0.34 | 0.03 | 0.42 |
| QuaRot |  | 16 | ✓ | ✓ | 4.00 | 7.10 | 0.79 | 0.73 | 0.76 | 0.76 | 0.51 | 0.75 | 0.72 |
| PVQ [3.0 bit shapes, 4 bit gains] |  | 16 | ✓ | ✓ | 3.25 | 7.14 | 0.78 | 0.72 | 0.76 | 0.74 | 0.49 | 0.74 | 0.70 |
| PVQ [3.5 bit shapes, 6 bit gains] |  | 16 | ✓ | ✓ | 3.88 | 6.53 | 0.80 | 0.73 | 0.77 | 0.76 | 0.51 | 0.75 | 0.72 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 1
questions: What is the superiority of your  quantization scheme over the renowned  1.58   llm  quatization scheme?
soundness: 3
strengths: -
summary: The paper addresses the problem of quantization in llms. They introduce PVQ which is a  codebook-free and search-free vector  quantization scheme .
weaknesses: -please provide proper  referencing for  “coherence processing” on page 2. 
 - on page 3, the   \hat{W} is not well introduced. how is it differen from W ? also \hat{W} should be bold since it is a matrix.
- move the legend in fig 2
- introduce all sets and params before their first appearance: S_D,k , P_D,K not introduced in  section 2.4. sets are better be  distinguishable from matrices  , you could use  caligraphic S and P for  sets. 

define “G”  in eq 7.  

- scalars better be represented by small letter. W for number of  words can be mistaken with W for weights. 

-please address the following  typos/grammatical errors:

“we achieves”, “

“compared to
compared methods”

“maximised”

“, making use for quantization costly.”

“The table can be precomputed a table of N”

“quantised” in fig 6

“On top” , “ to also quantize gains also enabling”

- this is not undestandable:  “to which we refer for a more detailed description of this for attention and fullyconnected layers found in most” - L1 ball should be $l_1$ - E8 ,RTN are not properly intriduced. 

-Figure 3 seems to be quantization on a Gaussian source not a neural network. provide mean and variance of the distributon of the Gaussian.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: The background and motivation need improvement. First, I suggest motivating VQ over scalar quantization which is already widely used in LLMs. Next, you can make the case that PVQ is an interesting VQ technique: It can be implemented in high-dimension due to an implicit codebook (like some other VQ techniques) and is also used in some applications. It appears to me that PVQ provides useful encoding under an L1 metric, whereas other methods would be better under, say, the more common L2. Analyzing the performance of PVQ may still be a solid contribution, but the discussion must be more comprehensive.  

Please explain more about the E8 method you provided as "state of the art". Can you compare performance to this method?
Please elaborate more on the other competing methods. Could you also show the performance of 4-bit scalar quantization, say using: https://huggingface.co/docs/bitsandbytes/en/reference/nn/linear4bit

It is well known, hence should be mentioned, that in high dimensions the information in the amplitude ('gain') is negligible compared to the information in the direction ('shape'). For example, see
[4] Kipnis, Alon, and Galen Reeves. "Gaussian approximation of quantization error for estimation from compressed data." IEEE Transactions on Information Theory 67, no. 8 (2021): 5562-5579.

The quantizer you used for the vector's magnitude is not optimized for the Beta distribution in any known sense. The authors either implement the optimal quantization method (under, say, mean squared error) or justify their approach.

Concerning Theorem 1, the authors should discuss how their result relates to or builds upon existing works like [2].
soundness: 2
strengths: The topic of quantizing LLM's is timely and important. Quite a few commercial tools offer bit-level compression of LLMs and it appears that theoretical treatment of the topic is lagging.

Analyzing and discussing PVQ in the context of LLMs are solid contributions because the method is used in some applications. The PVQ description and illustrations are clear and engaging. 

Performance comparison to other methods is generally relevant although I have comments below.
summary: The paper proposes to use a known vector quantization (VQ) method called Pyramid VQ (PVQ) to compress a large language model (LLM) at the bit level. The goal is to maximize the tradeoff between LLM's performance and the number of bits per weight. 

The paper describes the process of applying PVQ to LLMs, provides performance evaluations and comparisons to other VQ methods, and provides a result about the distribution of a normal random vector projection onto a sphere.
weaknesses: The discussion concerning the advantages of PVQ over other quantization techniques is misleading because the advantage of an implicit codebook generally exists in all practical VQ techniques.  See standard references in the topic like:

[1] Gray, Robert M., and David L. Neuhoff. "Quantization." IEEE Transactions on Information Theory 44, no. 6 (1998): 2325-2383.

Additionally, there is no discussion on the motivation for using VQ compared to scalar quantization in LLMs. Here I'd expect the authors to mention some property of the distribution of attention weights (say) like dependency that makes VQ particularly effective. Other competing methods such as RTN, QuaRot, and QGPT mentioned in the paper are also not fully explained and it is unclear what are the benefits of the proposed method and whether they can be truly attributed to PVQ or the detailed implementation. The coherence processing is perhaps attractive from a practical perspective but it ``hides'' structure in the signal that is ideally exploited by the compressor (not to be confused with `dithering' in scalar quantization that has a different effect). 

Numerical evaluations are only against three methods. Because encoding and decoding are not straightforward, I believe that you should also compare runtime information. Indeed, one can easily think about an elaborated lossy weight compression procedure that would certainly outperform all methods but spend considerable time for encoding and decoding (again, see discussion in [1]). There is no comparison to the popular scalar quantization methods (such as the one described in https://huggingface.co/docs/bitsandbytes/en/reference/nn/linear4bit), no comparison to the E8 method that the paper references. There is no explanation of what are the zero-shot tasks. PPL values appear small -- perhaps you mean log(PPL)? 

In Section 3.2 the authors say that E8 is a ``state-of-the-art'' method that exploits ``geometric structure''. Both terms in parentheses need explanation. 

The theoretical result (Theorem 1) appears to be a reformulation of well-known properties of the distribution of projection of Gaussian vectors. I've found via a quick search the reference: 
[2] Frankl, Peter, and Hiroshi Maehara. "Some geometric applications of the beta distribution." Annals of the Institute of Statistical Mathematics 42 (1990): 463-474.
which explains the main point and includes references to earlier works. 

For the amplitude ('gain'), the authors first argue that it makes sense to use a scalar quantizer optimized for the Beta distribution (Theorem 1). However, the quantizer they describe is obtained via quantile transform of a Beta-distributed random variable and is not optimal in general. See the discussion about optimal scalar quantization form [1] and from
[3] Lloyd, Stuart. "Least squares quantization in PCM." IEEE Transactions on Information Theory 28, no. 2 (1982): 129-137.

In Section 2.5, it is said the encoder maps a vector to its **closest point** on the pyramid, but the mapping procedure to follow does not necessarily lead to the closest point in the Eecleadian sense.  

In Section 3.1, the fact that decoding is done by an ``algorithm'' is meaningless unless you can say something about the runtime of this algorithm. 


There are several writing issues: 
Unclear what is "infeasibly large". 
The 0.01 in Step 5 is misplaced.
Eq. (8) has a weird text.
Typo in Theorem 1 (s_h should probably be s_g)
PPF in Eq. (10) is undefined.
I suggest the following notational changes: 'gain' -> 'amplitude', 'shape' -> 'direction', 'SNR' -> 'QSNR' (Read: signal to quantization-noise ratio).

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 1
questions: 1. What is the targetted application?
2. Why is the lattice-based approach not compared carefully?
3. Why is the trellis-based approach not compared?
4. Is scalar quantization compared? Is it significantly worse? How about scalar quantization plus entropy coding?
soundness: 2
strengths: The idea of using spherical code to compress weight and activation may be worth studying. If it can be shown convincingly that this approach indeed outperforms others, then it can make a worthwhile contribution to the area of LLM quantization/compression.
summary: The paper proposes a quantization method based on using representative points on a sphere to compress LLM weights and activation. The authors claim that it provides a significant improvement over existing art, and it is "codebook free" and "search free".
weaknesses: I found the paper poorly written, lacking clarity, and missing some important baseline comparisons to be convincing.

1. Quantization can be used for different purposes in LLMs. It is done for compressing LLMs (either for storage and transportation across platforms, or for reducing memory usage by reusing weights), but it can also be done for approximate computation. Depending on the targetted usage scenario, the evaluation needs to be designed more carefully. It was never made clear what scenario this work was aiming at. I assume the authors are targeting LLM compression for storage and transportation across platforms since bits were used in the evaluation. This assumption, however, leads to another issue. In data compression, after quantization, an entropy coding step (e.g. Huffman or LZ) is usually taken. The authors seem to have completely ignored this step. It is not clear to me why this is not considered since it is considered common practice.

2. The claim of codebook-free and search-free are dubious. The authors appear to take the naive lookup table implementation as the literal interpretation of "codebook" and "search". However, a structured set of representation points, like the proposed pyramid vectors, is still a codebook. Some codebooks have more structure than others, for example, the simple scalar-quantizer is very structured, but a randomly generated codebook may be the least structured. Representation points obtained by K-mean are not well-structured, but it can also be forced to take certain structures if needed. "Search-free" is also not an accurate term. Even for a completely unstructured codebook, there are approximation methods to simplify the "search" step, for example, using a sum of component-wise differences as an estimate to prune away most candidates. I found using the claim of advantage ambiguous in terms of these two "codebook-free" and "search-free" 

3. The authors did not compare the approach to standard approaches of "lattice quantization", "lattice quantization+entropy coding", and even "scalar quantization" and "scala quantization+entropy coding". Lattice does not require a lookup table, and has efficient enumeration techniques. In Fig. 3, we see that E_8 lattice is already performing better than the proposed approach. However, the authors did not include E_8 and other lattices in all the other comparisons. In fact, A^*_8 may be even better than E_8 in dimension-8. The lattice-based quantization has been known for over 40 years to perform well, and there are very efficient algorithms that avoid "search". See for example: (a) Conway, John, and Neil Sloane. "Fast quantizing and decoding and algorithms for lattice quantizers and codes." IEEE Transactions on Information Theory 28.2 (1982): 227-232. (b) Conway, J., and N. Sloane. "A fast encoding method for lattice codes and quantizers." IEEE Transactions on Information Theory 29.6 (1983): 820-824. I found the lack of comparison with these well-known techniques unacceptable.

4. Another quantization approach is trellis-based quantization, and it is known to also perform well and reduce computation. The authors should consider providing a comparison. 

5. The paper is quite poorly written. A few extreme examples are as follows 1) There is still an unremoved comment in equation (8). 2) Page 4, below (6), it should be "at most" T<D steps. 3) RTN appears in many figures in Section 3, but was not introduced until Section 4. 4) equation (1) is incorrectly/inaccurately written. H is already the Hessian with the expectation over x, then the last expression in (1) should not need to take expectation over x. 

6. I do not believe the claim of "the only method with non-integer shape bits" holds water. The codebook size is not chosen arbitrarily, since it is determined by N(D,K). Therefore, though the number of bits is not necessarily an integer, the number of codewords can still only take discrete values. 

7. The example given in 3.1 is quite ridiculous. In practice, no one would store a quantization table in this manner. Even for an unstructured codebook, only the representative points will be stored, and then the quantization procedure will find one representative point in the codebook through some algorithm, either more direct or searching.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: See the Weaknesses.
soundness: 3
strengths: A classic pyramid vector quantization (PVQ) method is well-combined with conventional tricks (e.g., coherence processing, quantization error compensation, Hessian) for the quantization of LLMs.
summary: In this paper, the authors propose a new post-training quantization algorithm for the quantization of LLMs.
The key idea is to apply a classic pyramid vector quantization (PVQ).
To validate the efficacy of the proposed method, the authors conduct experiments and provide kernels that allow hardware-accelerated encoding and decoding of PVQ.
weaknesses: 1. The authors need to improve the presentation of this paper. It seems that the authors just employed classic encoding and decoding algorithms in the pyramid vector quantization (PVQ) field. Although it is true, the authors need to explain the intuition and meaning of those algorithms. In the current version, the authors just mentioned that they used classic PVQ algorithms, and their pseudocode is provided in the Appendix. Furthermore, the detailed experimental setup is missing. How did the authors construct a calibration dataset? How many data samples are used for the quantization? What is the test dataset? 

2. The authors claimed that one key benefit of PVQ is that decoding is search-free while conventional vector quantization algorithms rely on look-up-table-based search. However, this claim is not justified. Why is this a benefit? Is the PVQ decoding process faster than look-up-table-based decoding? Also, while the authors analyze the time complexity of PVQ encoding/decoding in Table 5, actual processing time has not been measured, and thus the reviewer cannot judge whether the proposed algorithm is efficient.

3. The most crucial concern is that while the authors consider vector quantization, all the compared algorithms (GPTQ and QuaRot) are uniform quantization methods. It is natural that vector quantization methods exhibit better performance, but uniform quantization methods are much more hardware-friendly. The authors need to compare the proposed method with existing **vector quantization** methods [1, 2, 3, 4, 5].

[1] GPTVQ: The Blessing of Dimensionality for LLM Quantization, arXiv 2024

[2] Extreme Compression of Large Language Models via Additive Quantization, ICML 2024

[3] QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks, ICML 2024

[4] QTIP: Quantization with Trellises and Incoherence Processing, arXiv 2024

[5] VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models, arXiv 2024


---‐--‐-------------------------
It might be my final chance to adjust the score. 

While the authors have mentioned that they achieved similar performance with state-of-the-art methods, I cannot see any results regarding this claim.

Please upload the results about apple-to-apple comparison.

If the authors have uploaded results, I would kindly ask AC to consider my score as 6.

If not, my score is fixed to 5.

Also, while I agree some feedbacks of Reviewer 1xev, I honestly think some comments are very tough to address because the major topic is deep learning, not coding theory.

I hope AC to consider my opinions and other reviewers' comments.

Meta-review: **Summary:**
This paper proposes application of pyramid vector quantization (PVQ) to compression of weights and activations in large language models (LLMs).

**Strengths:**
This paper discusses an important and timely subject of compressing LLMs, and the proposal based on PVQ is interesting.

**Weaknesses:**
- Most reviewers criticized lack of comparison of the proposal with existing vector quantization methods.
- Most reviewers evaluated quality of the presentation of this paper, in its current form, negatively. 

**Reasons:**
Besides the lack of comparison with existing vector quantization methods, I am concerned the most with the quality of the presentation, which would require much more careful proofreading to have the manuscript to get accepted. Below is a relatively long list of points which I think would require revision.

Additional points:
- Page 1, line 22: we achieve(s); (p → P)areto-optimal
- Page 1, lines 23-4: compared to (compared) methods
- Page 1, line 38: has been (been)
- Page 2, lines 4, 45-6; page 3, line 10: pyramid vector quantization → PVQ
- Page 2, line 8: in (a) some chosen
- Page 2, lines 7-9: While the authors argue quantization of weights $W$ and activations $x$, they only introduce $\hat{W}$ and not $\hat{x}$.
- Page 2, line 9: A common conve(rs → nt)ion
- Page 2, line 12: The expectation $\mathbb{E}_{x}[xx^T]$ should never be equal to the empirical mean. One would also need the prefactor $1/N$.
- Page 2, line 14: the (layer-wise output → output error?) at each layer
- Page 2, line 17: "Equation" can be removed.
- Page 2, line 26: higher (number of) dimensions
- Page 2, line 36: "shape" might be better termed as "direction".
- Page 2, lines 38-9: (Kosson et al., 2023) → Kosson et al. (2023)
- Page 2, line 43: that reparameterize(s)
- Page 3, line 2: Chee et al. (2024) → (Chee et al., 2024)
- Page 3, lines 3, 5: The formulas $W=UWV$ and $\hat{W}=V^T\hat{W}U^T$ do not make sense.
- Page 4, line 6: To quantiz(ing → e)
- Page 4, line 14: sum to (1 → $K$)
- Page 4, line 17: $c\in[1,\ldots,N(D,K)-1]$ → $c\in[1,N(D,K)]$
- Page 4, line 25: that allow(s)
- Page 4, line 27: $C_{2,7}=[0,27]$ → $C_{2,7}=[1,28]$; Figure 2 should be corrected accordingly.
- Page 4, lines 27-30: I did not understand the sentence "The table can be precomputed a table ...". It might have been meant "The table of $N(d,k)$ for ... can be precomputed and reused ...".
- Page 5, line 4: (in) Section 3.2.
- Page 6, lines 20-1: "until convergence" should be moved just after "$|w_i|-K$"; One should refer not to Section 2.4 but to Section 2.5.
- Page 6, line 26: The formula for $s$ should be $s=\hat{w}^Tw/\|\hat{w}\|^2$.
- Page 6, line 30; Section 3.4: As Reviewer BTWr questioned, as opposed the proposal by the authors here, it is known that the quantization that is equispaced in the CDF values is not optimal in terms of the mean squared quantization error. See, e.g., Zador, IEEE Trans. Info. Theory, volume 28, pages 139-149, 1982.
- Page 6, equation (8): Please think about most elegant notation here, as is written.
- Page 6, lines 46-7: custom (arithmetic → arbitrary) precision arithmetic
- Page 7, line 7: coefficients → parameters
- Page 7, Theorem 3.1: Why do not the authors write $s_h=\|v_g\|^2/\|w\|^2$? The second parameter of the beta distribution should be $\frac{D(G-1)}{2}$, not $\frac{G(D-1)}{2}$. The statement of this theorem is well known: See, e.g., Ross, *A First Course in Probability*, 10th ed., Pearson, 2020, where Example 3b in Section 6.3.2 states that a sum of iid zero-mean Gaussians is chi-squared and thus gamma, and where Example 7c in Section 6.7 states that $X/(X+Y)$ for independent gammas $X,Y$ is beta.
- Page 7, Figure 5 caption: closely match(es)

Decision: Reject