QTIP: Quantization with Trellises and Incoherence Processing
Authors: Albert Tseng
Qingyao Sun
David Hou
Christopher De Sa
Venue: NeurIPS 2024 spotlight (2024)
Keywords: quantization
llms
trellises
fast inference
post training quantization
trellis coded quantization
model compression
computed codes
Primary area: other
TL;DR: We present the first tractable ultra-high dimensional quantizer for LLM PTQ that supports fast inference, enabling in state of the art quantization quality and inference speed.

Abstract: Post-training quantization (PTQ) reduces the memory footprint of LLMs by quantizing weights to low-precision datatypes.
Since LLM inference is usually memory-bound, PTQ methods can improve inference throughput.
Recent state-of-the-art PTQ approaches use vector quantization (VQ) to quantize multiple weights at once, which improves information utilization through better shaping.
However, VQ requires a codebook with size exponential in the dimension.
This limits current VQ-based PTQ works to low VQ dimensions ($\le 8$) that in turn limit quantization quality.
Here, we introduce QTIP, which instead uses trellis coded quantization (TCQ) to achieve ultra-high-dimensional quantization. 
TCQ uses a stateful decoder that separates the codebook size from the bitrate and effective dimension. 
QTIP introduces a spectrum of lookup-only to computed lookup-free trellis codes designed for a hardware-efficient "bitshift" trellis structure; these codes achieve state-of-the-art results in both quantization quality and inference speed.

QTI:: Quantization with Trellises and Incoherence Processing
Albert Tseng Cornell University
Qingyao Sun Cornell University gs2340corne11 edu
David Hou
dhou@alumni caltech. edu
albertêcs. cornell.edu A.bert@cs.cornell..oo
Christopher De
Comelll University cdesa@cs. cornell. edu
Abstract
Post-training quantization (PTQ) reduces the memory footprint of LLMs by quan- tizing weights to low-precision datatypes. Since LLM inference is usually memory- bound, PTQ methods can improve inference throughput. Recent state-of-the-an PTQ approaches use vector quantization (VQ) to quantize multiple weights once, which improves information utilization through better shaping. However, VQ re- quires codebook with size exponential in the dimension. This limits current VQ- based PTQ works to low VQ dimensions (S 8) that in turn limit quantization qual- ity. Here, we introduce QTIP, which instead uses trellis coded quantization (TCQ) to achieve separates the codebook size from the bitrate and effective dimension. QTIP intro- duces spectrum of lookup-only to computed lookup-frre trellis codes designed for a hardware-efficient "bitshift" trellis structure; these codes achieve state-of-the- art results in both quantization quality and inference speed.
quantization. TCQ uses a stateful decoder that
Introduction
Large language models (LLMs) have accelerated advancements in fields ranging from natural language processing 34 to scientific modeling (28). However, the largest LLMs have hundreds of billions of parameters that can take over a terabyte of memory to load in half prefisin:;;;;;; this size poses significant challenges for the practical deployment of LLMs [33 17 (2). For example, small- batch autoregressive decoding, a common form of inference for LLMs, is memory bound [35). Even on a modern datacenter GPU with ~ 3TB/s memory bandwidth, large LLM (> 200GB) can only be directly run at < 20 tokens per second and may require multiple devices 35 One way to accelerate inference is by compressing LLMs. This directly reduces the memory footprint of the model and increases the theoretical maximum inference throughput on any given machine.
One form of compression, weight-only post-training quantization (PTQ), quantizes trained model weights to lower precision datatypes 99| 35 5. The latest state-of-the-an weight-only PTQ methods, QuiP# and AQLM, use vector quantization (VQ) to achieve high-quality 2-bit models [15 In VQ. vector & E Rd is quantized one of kd vectors in Rd that form a codebook C E R xd higher vector dimension d allows for better codebook shaping and packing density, improving information utilization [19]. However. unstructured VQ requires exponential time and space in both the bitrate and dimension. limiting its practicality. During quantization, VQ costs O(2kd d) time to perform nearest- neighbor rounding to and during inference, C' must in hardware cache for fast lookups. This exponential scaling limits how high d can be and thus the advantages of VQ over scalar quantization.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Llama 2 Scaling
QTIP
QTIP Bit
Wikitex22 Perp wxiiy (ctr. 4096
6.0
QTP 3
"Bitshift" Trellis Quantizer
QUIP Bit
QuiP# Bit
AQLM Bit
5.0
FP16 @ Bit
4.5
Incoherent W
3.5
Ultra-High Dimensional Quantization
BlockLDLQ
10'
Total Madel Size (GB)
Figure 1: QTIP performs ultra-high dimensional (> 100) quantization by using Trellis Coded Quanti- zation, which has linear cost in dimension. This enables QTIP to outperform Vector Quantization- based approaches (QuIP#, AQLM) that limited to low dimensions. With QTIP, bit models scale better than theoretically optimal 4 bit models.
To address this limitation, we propose QTIP, which uses trellis-coded quantization (TCQ) to enable tractable dinre-iigh-dmeensionalll (> 100) quantization and improve quantization quality over prior VQ-based approaches. the simplest scalar form of TCQ, a length-T sequence S is statefully quantized using trellis directed graph G with 2 nodes, each with 2* incoming and outgoing edges and scalar value (24] The reconstructed sequence Š corresponds the node values of length-T" walk on G, and quantization finds the walk that minimizes some distortion metric on S and S. Since neighboring entries in S are connected by one of 2 edges, we only need to store which edge an entry came from, which takes i bits. For additive distortion metrics such as squared error, the optimal S can found with the Viterbi algorithm, which runs (22 time [3] 24] This means that the cost of quantization is independent of the bitrate & and linear in the sequence dimension T, cnabling tractable high dimensional quantization.
However, TCQ is not free. During inference, vanilla TCQ requires storing both G and the size 2 x V node value codebook, which can be too large to fit in cache. TCQQ-quatized sequences also cannot generally decoded in parallel, as /th elment of Š could depend on up to the first tk encoded bits. In QTIP, we solve these issues by introducing series of fast compute-based Gaussian codes designed for the hardware-efficient "bitshift trellis." Specifically, the bitshift trellis supports parallel decoding, does not require storing G. and our compute-bad baseu codes climinate needing to store a large node value codebook. This enables high-quality quantization of Gaussian sources while supporting fast inference, and we adopt incoherence processing with the random Hadamard transform ensure that LLM weights are approximately i.i.d Gaussian distributed. Altogether, QTIP
1. Achieves a state-of-the -hrt combination of weight-only LLM PTQ quality and fast inference through hardware -ffireefffcie trellis and codebook design.
2. Introduces multiple novel hardware-efficient (< instructions per weight) compute-based random Gaussian codes for TCQ on i.i.d. Gaussian sources.
2 Background and Related Works
We focus on weight-only post-training quantization (PTQ) of LLMs in this work; other model- compression approaches include quantization-awat training (QAT) and pruning. These methods are not strictly orthogonal to each other, as one could both prune and quantize a model. Since QTIP is a weight-only PTQ method, the rest of this section focuses on this area. Most current of-e-oof-the-rr PTQ methods round to minimize the per-layer proxy loss from Nagel et al. (27.
(() = E. [I(W- W)rr|]] = .r ((W W)H(WWWWW - w)7)
(1)

---

Here, W- E Rmxn is the quantized weight matrix, E IR" is an input activation, and H E, [xx" R=x" interpreted as proxy Hessian matrix. This objective is defined per-layer, making it tractable for very large models. However, minimizing it is difficult due to the non- differentiable nature of quantization. Instead many works have proposed algorithms such as Hessian- based adaptive rounding, alternating optimization, and even coordinate descent to approximately minimize the proxy error 1 35
2.1 Incoherence Processing
The effectiveness of these methods depends on properties of W. For example, many works have observed that weight and activation outliers cause poor quantization quality [TO 20 29]. In QulP, Chee et al. [ proposed that incoherence was important for quantifying this effect.
Definition 2.1 (Chee et al. [5). A Hessian H. R is p-incoherent if its eigendecomposition
H = QAQ" has max,,j Qiyl max,,j le_ Qeyl I p/vn. A weight matrix W E Rm is p incoherent if m
[Wayl = max,y le; Weyl S pl|lWl /mn.
Essentially, incoherence means the weights and important rounding directions (Hessian eigenvectors) are not too large in any direction, aiding quantization. To make W; Il incoherent (small μ)) one can perform incoherence processing (IP) by conjugating W, H with random orthogonal matrices U, V W < UWVT, H t- VHVT. QuIP# introduced IP with the random Hadamard transformation (RHT), which performs W <- S., S,, V.. H <- V,,S, HS, V?" where V& is h: x h: Hadamard matrix and S₂ is a length & random sign vector. The RHT achieves, with probability 2 1 - 6, Pw = 2 1og(4mm/6)). meaning that W's entries are approximately independently Gaussian distributed, which can aid quantization [35 3.. We choose build on incoherence processing here because the independent Gaussian-like weights it produces are suitable inputs for trellis coding [23)
2.2 Vector Quantization (VQ) for LLM PTQ
b-bit VQ quantizes d dimensional vector S to one of 2kd d-dimensional vectors that form codebook E R22dxxd xd M. Since C is an unstructured collection of arbitrary vectors, VQ enables better shaping and packing density than scalar product quantization (SPQ), where each entry in S is quantized independently [19]. However, this also comes at the cost of exponential time quantization and exponential space inference: finding the nearest neighbor in C' requires ((2kd,)) time, and storing C' requires ( d) space. The current crop of state-of-the-art LLM PTQ methods, QuIP# and AQLM, both use VQ to achieve high-quality 2-bit models. Since the shaping advantage of VQ comes from high dimensionality, both QulP# and AQLM attempt to maximize dimensionality. AQLM's uses a large 8D codebook (1MiB) that does not fit in L1 cache. QulP# uses an 8D compressible codebook based on the Eg lattice, which is highly symmetric. This codebook is compressible by 256x and barely fits LI cache. In either case, the VQ dimension effectively hardware-limited to < 8, motivating methods that enable even higher-dimenssonal quantization.
2.3 Trellis-Coded Quantization (TCQ)
TCQ was first proposed by Marcellin and Fischer (24 to apply the benefits of trellis coded modulation, a conceptually dual problem, to quantization. Define a (L, k, V) trellis G as directed graph with 24 nodes, each of which has 2 incoming and outgoing edges and a value E IR V: these values form a codebook C' E 2 xV To quantize a length-T sequence S E R each contiguous length-V subsequence of S is assigned to a node E G, with the restriction that the assigned nodes form walk. The reconstruction Š of S is then given by concatenating node values in the walk. When V = 1, this setup describes Marcellin and Fischer [24]'s original scalar TCQ. When V > 1, this describes TCVQ, which applies TCQ to vectors 12 37].
Finding the optimal S under an additive distortion metric can be done with the Viterbi algorithm in O(2-T) time. This is linear in sequence length, enabling ultra-high dimensional quantization. For exposition, we briefly describe the Viterbi algorithm here. Concretely, if we want to quantize a T- length scalar sequence reinterpreted sequence of vectors $1, 82, ..., STyV R. using trellis
3

---

code with graph G and codebook C, this corresponds to solving the optimization problem
T/V
minimize
Σ 1C,,
l over 21, 22........ Tryv the vertex sequence of a walk on graph G.
This optimization problem can be solved exactly with dynamic programming via the value function
V2(+)) min Σ |IC.: - xil? #2,42...... Fy the vertex sequence of a walk on G and xt = x
using the update rule
V.(0)
min ( (a.
s;1l². -
This Viterbi approach clearly takes time linear in I'' and in the number of edges of G; with a few simple optimizations this can be brought to O22 In comparison, crute-force-searching all possible 2 which is what we would need to do for an unstructured k-bit T-dimensional codebook- would take time proportional to 2LT/V The ability to tractably find the closest representable vector in IRT", even for large T, is in some sense the "main benefit" of trellis coding. For i.i.d sources, as L increases, TCQ efficiently approaches the infinite-length distortion- the attainable distortion of a k-bit quantizer [19] As shown in Table [ when quantizing
DR. which lower bounds i.i.d.
Gaussian with k = 2, the scalar Lloyd-Max quantizer attains 0. 18 MSE, :ulP:''s 8D ESP codebook 0.089 MSE, our (QTIP) 256D L 16 TCQ quantizer 0.069 MSE, and DR 0.063 21 25 35 8)
3 QTIP
Quantizing with TCQ requires storing both the codebook (24 x V) and trellis structure (24 x 2 during inference. These components are too large for fast inference when L ズ 12, which is necessary for high quality. Furthermore, for a generic trellis, recovering the state (and so the decoded value) at step tth requires a graph walk using the first kt bits: this prevents parallel decoding. QTIP solves these problems with novel combination of incoherence processing, hardware-efficien "bitshift trellis," " and fast compute-based random Gaussian codes. Incoherence processing makes W approximatelly ii.d Gaussian, which reduces quantization to Gaussian source coding. The bitshift trellis removes needing to store the trellis structure during decoding and also enables parallel decoding. Finally, the fast compute-base
random Gaussian codes remove the need to store the full codebook, completing the equation for fast inference. On the quality side, the fast random Gaussian codes enable the simple bitshift trellis to match complicated trellises and achieve state-of-the-a quantization quality.
The main focus of QTIP is what to quantize with (i.e. TCQ) and not how to quantize (e.g. adaptive rounding or descent methods). The general construction of QTIP can be used as drop-in replacement for VQ in any rounding framework. In the following sections, we first describe the "bitshift" trellis (Section 3.D. Then, we describe a series of fast compute- based codes for i.idd Gaussian sources, aligning with different types of hardware (Sections B.I. and 3.1.2). Finally, we give approximation for the tail- bitiniing trellis problem, which lets us more efficiently load weights in hardware (Section 3.2).
3.1 "Bitshift" Trellis and Codebook Design
The bitshift trellis was introduced by Mao and Gray 23 as part of the "random permutation trellis coder" (RPTC). In the bitshift trellis, node has an edge to node j 3e E Z, 0 S c < 2kV s.t. j = -12kV mod 2 + Essentially, the top L - kV bits of equal the bottom L - kV bits of i. This means that the first group of V weights depends only on the bits at positions [1, 2, .......7. L). the second only on bit positions {&V + 1, + 2, ..... kV + L). and in general the 7th on bit positions [(-1))AV + 1,...,, (t - 1)&V + L}. During inference, obtaining the next compressed group of V weights in sequence only requires bitshifting by &V bits, which is supported hardware. Furthermore, since each group of V weights only depends on a contiguous window of
virtually all
L bits in S, decoding can be parallelized. Figure shows a simple (L = 2, k = 1, V = 1) bitshift trellis. Note that edges only exist between nodes that overlap by 1 bit, and storing the quantized length 6 Š indeed only requires 6 bits (plus the initial state).
Quantizing an i. source with the bitshift trellis is nontrivial because neighboring groups of weights sharing many bits can potentially lead to strong correlations (Figure B LL). The RPTC permutes

---

Unquantized S:
0.54
0.03
0.72
0.19
0.26
0.89
00
01
0.5
00
00
00
00
00
00
Codebook
Trellis Walks
0.1
01
D1
01
01
Minimize
Squared
0.3
11
11
11
11
Error
0.8
10
10
10
10
10
10
Trellis
Quantized $:
0.5
0.1
0.8
0.1
0.3
0.8
Nodes 0, 1, and 3 have code values 0.5, 0.1, 0.8, and 0.3, respectively. Each node only transition the 2 2 nodes that share their top kV 1 bit with its bottom L kV $ also tail- Iillbiiiing so the last L kV 1 bits can be dropped to give Š 001011.
Figure 2: A bitshift trellis code with L - 2, k: = 1,
bit. In this example, $ can be stored as 0010110.
Table I: QTIP's compute-based codes (1 (1MAD, 3INST, HYB) achieve similar distortion rates pure-lookup random Gaussian trellis code (RPTC) when quantizing an i.i.d Gaussian source to 2 bits. All TCQ methods (L 16) outperform SQ and VQ and significantly closer the infinite-length distortion rate DR. which lower bounds the distortion a k-bit quantizer can attain.
SQ LLOYD-MAX
VQ
TCQ
2D TCQ
QUANT.
QUIP# ESP
IMAD
3INST
RPTC
HYB
RPTC
DR
DIM.
256
256
256
256
256
MSE.
0.118
0.089
0.069
0.069
0.068
0.071
0.069
0.063
the codebook to decorrelate neighboring weight groups (Figure RR) storing the codebook or storing and applying the permutation, both of which are prohibitively expensive during decoding. Instead, QTIP introduces a series of compute- based codes to produce a psuedorandom code, which has the same decorrelating effect and admits fast inference. To match approximately i.i.d. Gaussian RHT- CHT-transformed matrices, these codes produce psuedorandom approximate Gaussians in as few as instructions per weight (see Table I and Figure 3}. To the best of our knowledge, these code constructions alone are novel and are the first to propose lookup- free Gaussian trellis code.
223 RR))
However, this requires
3.1.1 Lookup-Free Computed Codes
Here, present two pure-computed lookup-free codes that produce pseudorandom approximately Gaussian number from a L bit word, enabling fast decoding on cache-limited hardware. These codes avoid strong correlations and can be implemented in < hardware instructions per weight on NVIDIA GPUs. We present two codes here to illustrate that multiple such codes are possible: in practice lookup-free code can be designed we want to run on.
the instructions available whatever hardware
Algorithm (IMAD) first runs linear congruential generator (LCG) to produce pseudorandom 32-bit word 32. This requires 2 instructions (MAD and &). It then sums the 32-bit word as four bit unsigned integers; this sum is approximately Gaussian distributed. This requires instruction (vabsdiff4) Finally, this sum must be scaled and shifted (another MAD). Although there are only 20 representable values when 10, this does not empirically affect quantization quality. 1MAD requires choosing a and b to avoid strong correlations; we set a 34038481 and b 76625530 (Figure LC).
Algorithm (3INST) also first runs an LCG to produce random 32-bit word X. Then, it XORs the bottom 16 bits of X with the mantissa bits, bottom two exponent bits, and sign bit of magic FP16 number m to produce an FP16 number my. then repeats this with the top 16 bits of to produce m2 and returns m m2- This entire process can be implemented in 3 ALU instructions with MAD for the LCG, 1op3 to mask and XOR with a packed duplicated m, and then summing mi and m2-
As there is currently no instruction on NVIDIA GPUs that sums the top and bottom half of 32-bit word as 165,, this requires extra data movement instruction "split" the 32-bit word into 16-bit registers.
5

---

Figure 3: Set of representable neighboring values in a bitshift trellis with = 16, 2, = for (far left) code with strong correlations, (left center) [ ("3INST") and (far right) a random Gaussian code. Note that while IMAD has minor correlations, both 1MAD and 3INST are close random Gaussian, resulting in good quantization quality.
(right center) algorithm
Algorithm Computed Gaussian Code "IMAD"
input L-bit left- seft-padded integer z, uint32 a, b.
< (ar + b) mod 23 {run LCG to get uniform random z)
(sum x as four 8-bit unsigned integers, this is approximately Gaussian]
(x & 255) + ((x >> 8) & 255) ((x: >> 16) & 255) ((x >> 24) 255)
x <- (x 510)/1411 147.8
output Pseudorandom approximate Gaussian s.
mi + m2 is approximately distributed by the sum of two mirrored exponential distributions, which is close to Gaussian. Like with Algorithm [ a, b, and m must be chosen to avoid correlations; we used 89226354, 64248484, m 0.922 (Figure right).
3.1.2 Hybrid Lookup-Compute Codes
Here, we describe a hybrid computed-lookup code that computes a pseudorandom (or hashed) index into a 2D vector codebook (V = 2). This code is tailored for modern GPUs, which have enough cache for small in in-memory LUT one benefit of using such a LUT over a purely computed codebook is that LUT can be fine-tur the-tuned after quantization. Algorithm first performs the hash X + X + X to mix the lower order and upper order bits of X [18] Then, takes bits (14 Q+11+ 14 indexed) as an index into 2 x 2 LUT to get two 16-bit floats. (The reason why we chose 2D codebook here is that shared memory on NVIDIA GPUs is accessed in 32-bit-word elements, and each such word can contain two 16-bit floats.) Finally, it XORs bit 15 of X to flip the sign of the second entry of the codebook vector. Algorithm Eerorittm3aan can be implemented with MAD, bitshift, mask, and 1op3, giving an amortized 2 instructions per weight. This effectively assigns L bit word to one of 2Q+1 2D vectors, each of which can be fine-tuned to improve quality. Algorithm cann also be implemented to XOR bit 31 alongside bit 15 (this is free in the 1op3) to give effectively 2R+2-sized codebook, which can improve quantization quality. We only realized this after running all the experiments, so the numbers in this paper use the "one sign flip" version of Algorithm 3 In QTIP, initialize the LUT using K-means on an empirical 2D i.i.d. Gaussian distribution.
3.2 Tail-Biting Trellises
Directly quantizing a length-7 sequence to a (L, k: V) trellis results in a total kT L kV bits since the starting state takes an additional I, - kV bits to store. If we run inference on a machine with w-bit words where w| kT', we must read an extra LL- w (L kV) wasted bits per sequence. For common w (e.g. 32), setting L AV + w makes the Viterbi algorithm intractable. One way to solve this is by enforcing that the start and end state share L - &V bits, i.e. the trellis is tail-biting 4. Exactly solving the tail- tail-biting trellis problem via dynamic programming takes time quadratic in the state space (2*), making this problem intractable for reasonable L 2 12 [30] However, since RHT-processed weights are approximately i.i.d., simple algorithms can be effective for approximately solving the tail-biting problem. We propose Algorithm 4 which first rotates the sequence by T/2, quantizes it, and then extracts the overlap between the rotated start and end states. It then requantizes the original sequence with this overlap as the tail-biting overlap. This only requires two Viterbi calls

---

Algorithm 2 Computed Gaussian Code "3INST"
input L-bit 0 left-padded integer I; uint32 a, b, float16
I < (ar + b) mod 23 (run LCG to get uniform random x)
(modify sign, mantissa, and bottom 2 exponent bits of m and sum, this is approximately Gaussian]
m < reinterpret uint32) << 16 + reinterpret(m, wint32)
< (r & < reinterpret (xr & 2 1, foaoo16 + reinterpret((tt >> 16) & 216 1, f,oo1161)
XOR
output Pseudorandom approximate Gaussian œ.
Algorithm 3 Hybrid Computed-Lookuu 2D Gaussian Code "HYB"
input L-bit left-padded integer x, codebook E R²a (V==), f r-
mod 232 (calculate hash)
o E 12 + C( >> (15 - Q)) & 2 1] {lookup from symmetric codebook)
t XOR (a & (1 << 15)) (apply sign flip)
output Pseudorandom approximate Gaussian vector e.
shows that in practice, Algorithm[ccn find close-to-optimal tail-biting sequences in total. Table Shows while being significantly cheaper to run than other tail-biting approximation algorithms [30]
Experiments
Here, we present experiments quantizing the Llama family of models with QTIP models offer strong performance across a wide range of sizes, allowing us to compare how different quantization methods perform and scale. We primarily compare QTIP against QulP# and AQLM. For Llama 1, we include GPTVQ-2D instead of AQLM since AQLM does not publish Llama numbers [36.. GPTVQ-2D performs 2D VQ inside GPTQ and offers strong performance. These methods outperform scalar quantization methods including GPTQ, AWQ, and OmniQuant; comparisons to those methods can be found in QulP+/ and AQLM 20 14 29 M. We mainly focus on the hybrid code (Section 4.2) since it is tailored for modern GPUs, and present a full suite of results for it. For the computed codes (Section 4.1. we present results for Llama 2.
33 34 26
These
Since the proxy error is not an additive distortion metric, we cannot minimize by quantizing W as one sequence. Instead, for all experiments, we use QTIP as quantizer in QulP=#'s BlockL.DLQ.. which allows us simultaneously achieve high dimensionality and low proxy error (35) Specifically, we quantize block of T,, x Ty weights dimensions of W, respectively. Since BlockLDLQ only specifies feedback along the input dimension, this is equivalent to BlockLDLQ with g Ty, but vector dimension of T.TY >> Ty. This has the benefit of limiting the effect of g in BlockLDLQ's error bound gmp²² ortrr(" (tt(H2// / /n while achieving a high dimension for TCQ. Algorithm 5 in the Appendix describes this in more detail.
sequence, where T- and T, span the output and input
4.1 Lookup-Free Computed Codes
Here, we use IMAD and 3INST with L 16, V 1, - T, 16. Setting T2 T, 16 enables using a 16 x 16 MMA tile per trellis sequence to perform matrix multiplication during inference. 16 16 MMA tiles form the basis of many types of "AI hardware," making fast decoding relatively simple 66. We do not perform fine-tuning since the codes themselves are not tunable, but these codes are fully compatible with QuIP##-style fine-tuning (recall that QulP#'s 's codebook is also not tunable).
Table 3 shows that both 1 IMAD and 3INST significantly outperform QulP# without fine-tuning (AQLM does not have numbers without fine-tuning). Even 4 bits, where all methods are close to lossless, QTIP results in significant improvements. Notably, the computed-code QTIP variants without fine-tuning outperforms both QulP # and AQLM with fine-tuning on almost all models and sizes, showing that fine-tuning is not a silver bullet.
4.2 Hybrid Lookup-Computedd Codes
7

---

Algorithm 4 Tail-biting Trellis Approx.
Table 2:
Quantizing 4K T 256 i.i.d Gaus- sian seqs. with a tail-biting (12, k. trellis.
input Sequence IR", (L, k, V) Trellis G. S' + Rotate S to the right by [T/2]
k
Alg.
MSE
Optimal MSE
S Viterbi(S", G)
0.2803
0.2798
0 L RV bit overlap of (T/22 $ [T/2]+1
2
0.0733
0.0733
$ < Viterbi(S, G) with start/end overlap 0
3
0.0198
0.0198
output Tail biting $
4
0,0055
0.0055
Table 3: Wikitext2 and C4 perplexity (4), ctx. 4096, QTIP with pure-computed codes. Even without fine-tuning, pure-computed QTIP outperforms QulP#/ and AQLM, both of which use fine-tuning, at almost all models sizes.
4 BIT No FT
24 BIT
3 BIT No FT
R33 BIT
2 BIT No FT
R/2 BIT
FP16
IMAD
SINST
QUIP4
QUIPA
AQLM
IMAD
SINST
QUIP4
QUIP
AQLM
IMAD
SINST
QUIP4
QUIPA
AQLM
W2
5.12
5.17
5.17
5.22
5.19
5.21
5.38
5.40
5.60
5.41
5.38
7.05
6.82
8.22
6.19
6.14
2-7
C4
6.63
6.71
6.71
6.79
6.75
6.75
6.99
7.01
7.34
7.04
7.01
9.14
8.96
11.0
8.16
8.09
W2
4.57
4.62
4.62
4.65
4.63
4.64
4.74
4.74
4.90
4.78
4.78
5.59
5.52
6.06
5.35
5.33
2-13
C4
6.05
6.10
6.10
6.15
6.13
6.14
6.28
6.28
6.50
6.35
6.33
7.46
7.39
8.07
7.20
7.19
W2
3.12
3,16
3.16
3.18
3.18
3.19
3.27
3.27
3.41
3.35
3.36
3.87
3.90
4.16
3.91
3.83
2-70
C4
4.97
5.00
5.00
5.02
5.02
5.03
5.09
5.09
5.20
5.15
5.17
5.70
5.69
6.01
5.71
5.62
Here, we use the hybrid lookup-computed code with
Table 4: Batch size 1 decoding throughput
== 16, V = 2, T. Ty 16, Q 9. Setting Q 9 gives a 2KiB codebook, which fits in L1 cache even after duplication for bank conflicts (32x) on modern GPUs. This codebook is diffcrentiable, so we can fine- tune it: evaluate this, we fine-tune using QulP#'s 's methodology, tuning both the codebook entries and the as- as-yet-unquantize weights in blockwise fashion. Table shows Sshows the perplexity of quantized Llama I and 2 models. In all cases, QTIP outperforms the other vector quantization-basse methods. Even at 3 and 4 bits, where QulP# and AQLM are close lossless, QTIP roughly halves the perplexity gap. These results also show the importance of dimensionality. Note that the 3- and 4-bit Llama 70B numbers here match those in 3 Since Table juees pure- computed code without fine-tuning, fine-tuning has no effect in these regimes and the improvement over *ulP# is purely from dimensionality.
on a RTX6000 Ada (960GB/s mem. B BW).
METHOD BITS 2-7B Tok/s 2-70B Tok/s
FP16
16
55.9
00M
AQLM
81.5
8.78
QUIP#
186
22.2
QTIP
188
23.5
QTIP
161
19.1
QTIP
140
16.3
Tsshos zeroshot results computed with LM Eval, which are slightly random; QTIP generally matches exceeds QulP# and AQLM on these tasks (15). Table 7 contains results Llama 3. Like other works, we have observed that Llama (especially 70B base) is harder to quantize than Llama [16] Since the contribution and focus of this work is what round with (TCQ) and not how to round (BlockI.DLLQ we only compare against the proximal baseline QulP#, which uses BlockLDLQ with VQ. QTIP significantly improves upon QuIP## all model sizes and bitrates, once again showing the dimensionality ad advantage of TCQ over VQ. Table 8 shows results for Llama 3.1 instruct-tuned models, including Llama 3. 405B. At all sizes, QTIP achieves strong results. Notably, QTIP is able to match or exceed PV-Tuning, recent quantization method that focuses on better fine- tuning algorithms (22]. However, PV-Tuning is based off of AQLM and inherits its slow inference speed, making it significantly slower than QTIP Finally, Table 3.2 instruct-tuned models to 4 bits. Since the embedding layers are very large relative to the decoder layers for small Llama 3 models (Rs 500 750MB), quantizing the decoder layers to fewer than 4 bits does not make a significant difference on the final model size. Here, QTIP is still able to achieve mcaningful end-to-en compression rate (2.5-3X) without degrading the final model.
results for quantizing Llama

---

Table 5: Wikitex12 and C4 perplexity (4), QTIP with the hybrid-computed code. QTIP enables high- dimensional quantization and outperforms state-of-the-ard vector quantization approaches.
CTX. 2048, X = GPTVQ. 0.13
CTX. 4096, x AQLM, Y Y 0
WIKTEXT2
C4
WIKITEXT2
C4
METHOD
BITS
1-7
1-13
1-30
1-65
1-7
1-13
1-30
1 1-65
2-7
2-13 2-70
2-7
2-13
2-70
FP16
16.0
5.68
5.09
4.10
3.53
7.04
6.61
5.98
5.62
5.12
4.57
3.12
6.63
6.05
4.97
X
4+Y
5.94
5.20
4.18
3.64
5.21
4.65
3.19
6.75
6.14
5.03
QUIP#
4.00
5.76
5.17
4.18
3.60
7.18
6.67
6.03
5.66
5.19
4.63
3.18
6.75
6.13
5.02
QTIP
4.00
5.15
4.15
3.58
7.13
6.65
5.64
5.17
4.61
3.16
6.69
6.09
5.00
X
3+Y
6.32
5.31
4.38
3.79
5.38
4.78
3.36
7.01
6.33
5.17
QUIP#
3.00
5.98
5.31
4.36
3.70
7.39
6.83
6.17
5.77
5.41
4.78
3.35
7.04
6.35
5.15
QTIP
3.00
5.85
5.24
4.26
3.68
7.26
6.74
6.09
5.71
5.28
4.69
3.26
6.87
6.22
5.08
X
2+Y
9.64
6.58
5.63
4.91
6.14
5.33
3.83
8.09
7.19
5.62
QUIP#
2.00
6.86
5.97
4.36
8.36
7.48
6.71
6.19
6.19
5.35
8.16
7.20
5.71
QTIP
2.00
6.52
5.80
4.83
4.21
7.99
7.31
6.56
6.08
5.86
5.11
3.70
7.73
6.85
5.48
Table 6: Zeroshot accuracy (t), QTIP with the hybrid-computed code.
2-70
2-13
2-7
MTHD.
BITS ARCC ARCE
PIQA
WINO
BITS ARCC ARCE PIQA
WINO
BITS
ARCC ARCE PIQA WINO
FP16
16
51.1
77.7
81.1
77.0
16
45.6
73.3
73.5
69.6
16
40.0
69.3
78.5
67.3
AQLM
4.14
50.7
77.3
81.5
76.5
3.94
44.8
73.3
78.4
69.9
4.04
41.0
70.2
78.2
67.3
QUIP#
50.5
77.7
81.4
77.3
43.6
71.3
78.7
69.6
40.4
68.6
78.5
67.4
QTIP
50.0
77.8
81.3
76.9
4
44.8
73.6
78.9
69.9
4
40.0
68.9
78.4
67.1
AQLM
3.01
50.3
78.0
80.7
75.3
3.03
42.8
72.9
78.5
68.8
3.04
38.5
66.8
77.3
65.4
QUIP#
50.9
77.6
81.4
76.1
3
44.0
72.5
78.4
69.1
39.2
68.4
77.3
66.5
QTIP
50.3
78.2
80.6
77.0
3
44.0
72.8
78.0
69.5
38.9
68.1
78.1
66.9
AQLM
2.07
47.9
77.7
80.4
75.9
1.97
38.8
69.3
75.9
68.8
2.02
32.8
63.7
74.8
65.7
QUIP#
47.6
77.1
79.5
74.6
2
39.6
69.0
77.3
67.4
2
35.2
65.3
75.4
64.9
QTIP
2
48.0
76.3
80.2
75.1
2
41.4
70.8
77.3
67.6
2
35.7
65.6
75.9
64.7
4.3 Inference Speed
Table 4 shows the batch size 1 inference speed of QTIP, QulP#, and AQLM Llama 2 7B and 70B with matrix fusion. Here, the design choices of QTIP and QulP# become apparent. Whereas large to fit in cache and thus prevents fast inference, both QTIP and QulP# achieve significant speedups over FP16. Furthermore, while it is impressive that both QulP# and QTIP are 2x faster than AQLM, is even more impressive that QTIP able match QulP#'s throughput with an effective dimension size of 256, or 32x larger than QulP#'s. This means that the improved quantization quality of QTIP comes with additional inference-tine cost. Although our empirical throughput numbers were timed on NVIDIA GPUs, QTIP can be fast on a broad class of accelerators due to its flexibility. QTIP only requires generating pseudorandom Gaussian efficiently. and can work on devices with no cache well as devices with lookup hardware. For example, if we were using a ARMv8 CPU, we could use the vqtbl4q_us NEON intrinsic to look 16 indices in 64-entry codebook. This would let use 6 bit ID codebook with the HYB code (Q=6, V=1). Quantizing Llama 7B to 2 bits with this setup and w/out fine-tuning gives 6.89 Wikitext2 perplexity essentially the same state-of-the quality 3INST.
AQLM uses codebook that
5 Conclusion
We present QTIP, a weight-only post-training quantization algorithm that achieves state-of of- art results through the use of trellis-coded quantization (TCQ). TCQ enables tractable altra-high dimensional quantization, significantly reducing quantization distortion over vector quantization (VQ). However, naive TCQ does not admit fast inference due to sequential bottlenecks during decoding and needing store large codebook. QTIP solves this problem through novel combination of incoherence processing, the hardware-efficient bitshift trellis, and fast computed codes. Specifically, QTIP introduces series of compute- based pscudorandom Gaussian codes that, when used in

---

Table 7: QTIP vs. QulP#, Llama 3 (ctx. 8192 for perplexity). Although Hessian-based rounding generally underperforms on Llama 3, the focus of this work is on what quantize with (TCQ vs. VQ).
Here, the high-dimensionality of TCQ in QTIP improves over the low-dimensional VQ in QulP#.
3-70 PPL (1)
3-70 ZEROSHOT ACC (1)
3-8 PPL (L)
3-8 ZEROSHOT ACC (t)
MTHD.
BITS
C4
ARCC
ARCE
BooLQ
PIQA
WING
W2
ARCC
ARCE
BOOLQ
PIQA
WINO
BF16 16.0
2.59
5.78
60.5
86.9
85.3
82.4
80.3
5.54
7.10
50.2
80.1
81.0
79.7
72.9
QUIP= 4.00
2.99
5.96
35.0
67.3
84.7
71.9
76.7
5.81
7.32
50.2
79.7
81.3
79.7
73.1
QTIP
4.00
2.75
5.83
56.1
83.9
85.8
81.3
80.6
5.67
7.20
50.2
79.6
79.5
79.4
73.4
QUIP# 3.00
3.59
6.18
31.1
36.6
85.7
58.8
76.4
6.27
7.71
46.4
77.4
79.9
77.9
72.9
QTIP 3.00
3.18
5.98
48.6
77.8
85.0
77.8
79.7
6.01
7.48
49.2
79.3
80.0
79.2
74.5
QUIP# 2.00
7.46
18.3
32.2
82.1
54.7
68.9
9.06
39.2
72.9
76.6
75.6
68.2
QTIP 2.00
4.97
6.80
28.0
35.2
83.6
57.1
72.6
7.33
8.62
44.2
75.2
76.7
77.6
70.7
Table Llama 3.1 instruct-tuned model results (ctx. 8192 perplexity). QTIP performs well at all
model sizes and generally outperforms PV-Tuning, recent quantization method that focuses on fine- tuning algorithms. The zeroshot results in this table use LMM Eval 0.4.4 and the "standard" versions of each task instead of the Meta versions in 226].
PPL. (1)
ZEROSHOT (1)
BITS
W2
ARCC
ARCE HSWAG
PIQA
META "FP8"
16 ATTN. /8 MLP
1.70
61.6
81.4
67.1
83.8
QTIP
1.79
61.3
66.7
84.2
3.1 405B INST.
QTIP
2.05
61.5
81.4
66.8
83.5
QTIP
2
3.29
60.7
81.1
65.4
82.2
BF16
T6
3.52
56.7
75.6
61.5
82.8
QTIP
3.73
56.3
75.8
61.4
83.0
3.1 70B INST.
QTIP
4.12
55.1
75.1
60.8
82.6
QTIP
2
5.08
54.4
72.6
59.4
82.5
PV-TUNING
2.01
5.70
52.7
72.2
60.2
82.6
BF16
16
6.50
51.6
77.8
57.7
80.0
QTIP
6.61
50.7
78.0
57.5
80.1
3.1 8B INST.
QTIP
6.80
50.4
77.7
56.9
79.3
QTIP
2
7.82
45.1
75.6
54.5
79.0
PV-TUNING
2.07
8.45
46.2
75.4
54.4
78.7
Table 9: Llama 3.2 instruct-tuned results when quantizing to 4 bits (ctx. 8192 for perplexity). Even on extremely small models, QTIP is still able to achieve meaningful compression without sacrificing quality. This table uses the same LM Eval setup as Table g
(1)
ZEROSHOT (t)
SIZE (GB)
W2
ARCC
ARCE
HSWAG
PIQA
BF16
9.58
43.3
74.3
52.2
75.7
3B
QTIP
9.77
43.5
74.3
51.9
75.1
BF16
2.4
11.57
36.0
68.5
45.2
74.2
IB
QTIP
0.97
11.93
34.8
68.4
44.5
73.3
conjunction with the bitshift trellis and incoherence processing, simultancously achieves state-of- the-art PTQ quality and fast inference. QTIP improves quantization quality at all tested bitrates over the latest VQ-based PTQ methods, QuIP# and AQLM, further pushing the boundary LLM PTQ. QTIP's codes use as few as 2 instructions per weight during decoding, enabling matrix-vector multiplication to run at over 80% of peak memory bandwidth on modern GPUs. Altogether, our results indicate that high dimensional quantization is necessary for high high-quality compression, and QTIP is the first LLM PTQ method to scale to ultra-high dimensions while supporting fast inference.
A Acknowledgements
C.D. was supported by NSF-2046760 CAREER. We thank Together AI for compute resources.
10

---

References
[1] An algorithm for vector quantizer design. IEEE Transactions on communications, 28(1):84
1980.
[2] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malar- tic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series of open language models, 2023.
[3] Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated 1lms, 2024.
[4] A.R. Calderbank, G.D. Forney, and A. Vardy. Minimal tail- trellises: the golay code and more. IEEE Transactions Information Theory, 45(5):1435-1455 1999. doi: 10.1109/18. 771145.
[5] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. QulP: 2-bit quantization of large language models with guarantees. Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL httpps://pennreviez http::///ppppeeereeeeeeee//eeeer
[6] Jack Choquette, Wishwesh Gandhi, Olivier Giroux, Nick Stam, and Ronny Krashinsky. Nvidia a100 tensor gpu: Performance and innovation. IEEE Micro, 41(2):29-35, 2021. doi: 20.110900..2220..6001
[7] Together Computer. Redpajama: An open source recipe reproduce llama training dataset, 2023. URL /ttpptt:s///gthtutt wwwetttwww.oottteeeeeeeeeeeeeeee www.r//ggeeeeerooourreeeeeeeeeee
ama Data
[8] Thomas M. Cover and Joy A. Thomas. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, USA, 2006. ISBN
047 144144.
[9] Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 7750-7774, PMLR, 23-29 Jul 2023. URL https://prroceddnngs mlr wrw.pprssoo00 2et.222///tttms22233a html
[10] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int8(): 8-bit matrix multiplication for transformers at scale, 2022.
[11] Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization, 2024.
[12] T.R. Fischer, M.W. Marcellin, and M. Wang. Trellis-coded vector quantization. IEEE Transactions on Information Theory, 37(6):1511-1566 1991. doi: 10.1 1091119/1..11316.
[13] G.D. Forney. The viterbi algorithm. Proceedings of the IEEE, 61(3):268-27) 1973. doi: 00.1109/ROC.1193.9903
[14] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023. URL httttss//opeereeiiiee net/forum 7id tcbBPnfwxx
[15] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL 10256836
h, Haonan Li, Kyle McDonell, Niklas
org/ records/

---

[16] Wei Huang, Xudong Ma, Haotong Qin, Xingyu Zheng, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, and Michele Magno. How good are low-bit quantized Ilama3 models? an empirical study, 2024.
[17] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024.
[18] Alexander Klimov and Adi Shamir. new class of invertible mappings. In Cryptographic Hardware and Embedded Systems-CHES 2002: 4th International Workshop Redwood Shores, CA, USA, August 13-15, 2002 Revised Papers 4, pages 470-483. Springer, 2003.
[19]
Victoria Kostina and Sergio Verdu. Fixed-length lossy compression in the finite blocklength regime, IEEE Transactions on Information Theory, 58(6):330--3338, 2012. doi: 10.1109/TIT. 2012.2186786.
[20] Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration, 2023.
[21] Stuart Lloyd. Least squares quantization in pem. IEEE transactions on information theory, 28 (2):129-137, 1982.
[22] Vladimir Malinovskii, Denis Mazur, Ivan Ilin, Denis Kuznedelev, Konstantin Burlachenko, Kai Yi, Dan Alistarh, and Peter Richtarik. Pv-tuning: Beyond straight-through estimation for extreme Ilm compression, 2024.
http://axxvvv..r.o//.. /bw//////5..... 2405 14852
[23] Mark Z. Mao and Robert M. Gray. Stationary and trellis encoding for iid sources and simulation. In 2010 Data Compression Conference, pages 3-12, 2010. doi: 10.1 20.00
[24] M.W. Marcellin and T.R. Fischer. Trellis coded quantization of memoryless and gauss-markoy
sources. IEEE Transactions on Communications, 38(1):82-93, 1990. doi: 10. 10.1109/26.46532.
[25] J. Max. Quantizing for minimum distortion. IRE Transactions on Information Theory, 6(1):
7-12, 1960. doi: 10. 10.110001..119001157777
[26] meta llama. Ilama3. https://githin sttps://gicuub..oo//mttaaa
com/meta. aeaaalaaaaaaaaaaa 2024.
[27] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? Adaptive rounding for post-training quantization. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learning Research, pages 7197-7206. PMLR, 13-18 Jul 2020. URL htttps://proceeednngg mlr. prwps/resssiii hatpe//v119//ageee0. hhtml
[28] Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Callum Birch-Sykes, Michael Wornow, Aman Patel, Clayton Rabideau, Stefano Massaroli, Yoshua Bengio, Stefano Ermon, Stephen A. Baccus, and Chris Ré. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. 2023.
[29] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quan- tization for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL httphs://ppeereeiiceee wwwp:oounntdddddeeoooooom wwd.rrrwmnmddddwwhhhoooooo
[30] Y. Shao. Rose, Shu Lin, and Mare P. C. Fossorier. Tail biting trellis representation of codes: Decoding and construction. 1999. URL.
3727533
[31] Neil Sloane. Hadamard Matrices - neilsloane.com. http::/neilsllonn. com/hadamarda [Aceessed 02-02-2024].
12

---

[32] W. E. Thomson. A Modified Congruence Method of Generating Pseudo-random, Numbers. The Computer Journal, 1(2):83-83 83, 01 1958. ISSN 0010-4620. doi: 10.1093/comjnn/1.222 2..... URL
https://doi. http:://dooooo//1///// http://.oorrr...../////////////o
[33] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
[34]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini. Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang. Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
[35] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quipit: Even better IIm quantization with hadamard incoherence and lattice codebooks, 2024.
[36] Mart van Baalen, Andrey Kuzmin, Markus Nagel, Peter Couperus, Cedric Bastoul, Eric Mahurin, Tijmen Blankevoort, and Paul Whatmough. Gptvq: The blessing of dimensionality Ilm quantization. arXiv preprint arXiv:2402.15319 2024.
[37] H.S. Wang and N. Moayeri. Trellis coded vector quantization. Communications, 1073):1273 1992. doi: 10. 156629.
IEEE Transactions on
13

---

A
Appendix
A.1 Additional Results
A.1.1 Ablations on Trellis Size
Table 10 shows an ablation on for quantizing Llama 2 7B with K 2, 1, the bitshift trellis, pure-lookup codebook, and no fine-tuning. trellis and codebook in the same amount of cache as the HYB code (2KiB). L = 10 is the largest achievable if only had store the codebook. As expected, increasing L improves quality, Table I0 also shows very little difference between an equal-sized LUT codebook and QTIP's codes, meaning that QTIP isn't sacrificing quality for speed. However, an equal-sized LUT would need 10x more cache than the latest GPUs have, making the bitshift trellis and compute- based codes necessary to achieve both quality and speed. Table [T] shows an ablation on V with L 12 and 16, K 2, and the same settings Table 10 Increasing V generally decreases quality, but this can be recovered with larger L. hard to measure V"s impact on decoding speed since this is highly implementation and hardware dependent, so V is more of user-chosen hyperparameter.
is the largest achievable if we had to store the
Table 10: Ablation on L when quantizing Llama 7B to 2 bits (K = 2 and V = 1).
L
Trellis Size
CB size
total size
W2
C4
QuIP# 8
8Kb
8Kb
8.22
11.0
8.19 Kb
4.10 Kb
12.29 Kb
7.83
10.3
10
40.96 Kb
16.38 Kb
57.34 Kb
7.49
9.67
12
196.61 Kb
65.54 Kb
262.14 Kb
6.97
9.21
16
4.19 Mb
1.05 Mb
5.24 Mb
6.83
8.92
16
Bitshift
3INST
0Kb
6.82
8.96
Table 11: Ablation on V when quantizing Llama 2 7B
bits (K 2).
Codebook
V
W2
C4
LUT
6.97
9.21
LUT
12
2
7.09
9.24
LUT
4
7.55
9.88
LUT
16
1
6.83
8.92
LUT
16
2
6.79
8.97
QTIP HYB (no FT)
16
2
6.83
8.97
LUT
4
6.92
9.07
A.1.2
Zeroshot Results
Table 12:
Zeroshot results for the IMAD code.
Bits
AreC (acc)
ArcE (acc)
BoolQ (acc)
PiQA (acc)
Wino (acc)
2-7
16
39.9
69.3
71.1
78.4 .
67.2
2-7
4
39,0
69.4
72.0
78.4
67.9
2-7
38.8
68.0
68.2
77.6
68.4
2-7
2
32.1
63.5
66.3
73.3
62.7
2-13
45.6
73.3
69.1
78.7
69.7
2-13
4
45.6
72.9
68.1
78.7
70.3
2-13
42.2
71.0
69.9
78.6
69.8
2-13
2
38.5
71.5
71.4
75.9
68.9
2-70
16
51.2
77.7
76.7
81.1
76.9
2-70
4
51.1
77.8
75.2
81.5
77.0
2-70
50.8
77.8
77.9
80.7
76.3
2-70
2
49.3
77.7
83.3
80.4
75.7
14

---

Table 13: Zeroshot results for the 3INST code.
Bits
AreC (acc)
ArcE (acc)
BoolQ (acc)
PiQA (acc)
Wino (acc)
2-7
16
39,9
69.3
71.1
78.4
67.2
2-7
4
40.2
68.5
70.3
78.0
67.7
2-7
3
40.2
68.6
73.0
77.5
65.4
2-7
2
32.9
61.9
65.5
74.5
65.0
2-13
16
45.6
73.3
69.1
78.7
69.7
2-13
4
45.4
72.7
67.9
78.5
69.9
2-13
3
44.5
72.6
70.1
78.5
69.4
2-13
2
38.7
68.2
63.6
75.6
68.7
2-70
51.2
77.7
76.7
81.1
76.9
2-70
4
50.3
77.9
77.3
80.7
76.5
2-70
3
50.9
78.3
78.8
81.1
77.5
2-70
2
48.0
76.5
76.7
80.1
77.6
Table 14: Llama Zeroshot results for the Hybrid code
Bits
ArcC (acc)
ArcE (acc)
BoolQ (acc)
PiQA (acc)
Wino (acc)
1-7
16
38.2
67.4
73.1
78.4
67.0
1-7
4
38.8
67.1
74.2
78.3
67.1
1-7
3
37.0
65.7
74.1
77.7
67.3
1-7
2
35.3
64.9
72.9
76.1
65.4
1-13
16
43.9
74.6
68.5
78.8
70.1
1-13
4
43.4
73.7
68.2
79.1
70.1
1-13
3
42.2
74.2
68.0
78.7
70.5
1-13
2
39.7
72.1
66.6
77.6
68.9
1-30
16
46.7
75.4
68.4
81.0
72.6
1-30
4
46.7
75,4
69.9
81.0
73.3
1-30
3
47.8
75.0
70.0
80.4
73.6
1-30
2
44.0
72.7
72.8
78.7
71.7
1-65
47.0
75.3
82.3
81.5
77.2
1-65
4
46.8
74.5
82.8
81.4
76.6
1-65
3
46.8
75.3
83.0
81.3
75.9
1-65
2
44.4
74.2
83.1
80.4
75.7
A.1.3
Lookup-Only Codes
Table 15; Wikitext2 and C4 perplexity ()), ctx. 4096, QTIP with size 2 LUT codebook. This codebook is too large (32KB) for current GPU LI caches, but could fit on near-future hardware.
4 Bit
~3 Bit
~2 Bit
FP16
QTIP
QulP#
AQLM
QTIP
QulP#
AQLM
QTIP
QulP# AQLM
W2
5.12
5.16
5.19
5.21
5.30
5.41
5.46
5.89
6.19
6.64
2-7
C4
6.63
6.68
6.75
6.75
6.86
7.04
7.08
7.78
8.16
8.56
W2
3.12
3.15
3.18
3.19
3.26
3.35
3.36
3.77
3.91
3.94
2-70
C4
4.97
4.99
5.02
5.03
5.07
5.15
5.17
5.55
5.71
5.72
Here, we use pure-lookup code ~ NO0, 1) with L -4, 32, T, = 8, and QulP#''s fine-tuning scheme. These parameters show what performance QTIP could achieve we did not care about fast inference today. Specifically, a pure-lookup codebook is tunable, and setting T, reduces the BlockLDLL group size while maintaining high dimensionality (256). This codebook uses 32KB; this only fits in GPU L1 cache with bank conflicts. Setting T, 32, Ty corresponds using a larger MMA tile size than current GPUs allow for. The largest tile size usually 16 the T2 dimension, meaning that 32 x trellis needs two tiles. Thankfully, hardware required serve
15

---

Table 16: Wikitex12 and C4 zeroshot accuracy (t), QTIP with a size 2 LUT codebook. This
codebook is too large (32KB) for current GPU LI caches, but could fit on near-future hardware.
Bits
ArcC (acc)
ArcE (acc)
BoolQ (acc)
PiQA (acc)
Wino (acc)
2-7
16
40.0
69.3
71.0
78.5
67.3
2-7
40.3
69.2
73.0
78.1
67.5
2-7
3
39.1
69.3
69.6
77.8
66.3
2-7
37.0
64.6
67.2
75.6
66.9
2-70
16
51.1
77.7
76.6
81.1
77.0
2-70
4
50.1
77.5
76.4
81.3
77.3
2-70
3
50.6
77.9
78.0
81.1
76.1
2-70
2
47.1
76.9
79.5
80.1
76.3
such model quickly is likely only few years away, as these parameters are only slightly outside of what today's 's hardware is capable
Table 15 shows that QTIP outperforms both QulP# and AQLM at all compression ratios, with bit QTTP achieving similar quality bit AQLM. While it is not fair compare this QTIP setup with QulP#, since QulP# was designed for fast inference, we note that AQLM's VQ codebook uses 2 x 2 1 MiB. This is 32 times larger than the QTIP codebook here, and would require 32 MiB of LI cache to read from without bank conflicts. Not only is this orders of magnitude larger than current L1 caches (256KB the H100), it is larger than many L2 caches!
A.1.4 Decoding Speed on Different GPUs
Table 17: Decoding speed on different Ampere and Lovelace GPUs.
GPU Model
Model
2- tok/s
3-bit tok/s
4-bit tok/s
FP16 tok/s
RTX 3090
2-7
127
119
109
52.5
RTX 3090
2-70
15.3
OOM
0OM
OOM
RTX A6000 Ampere
2-7
116
106
95
43.5
RTX A6000 Ampere
2-70
15.0
13.1
11.7
OOM
RTX 6000 Ada
2-7
188
161
140
55.9
RTX 6000 Ada
2-70
23.5
19.1
16.3
OOM
A.2 QTIP with BlockLDLQ
Here, we detail how we use TCQ within BlockLDLQ to produce our experimental setup. Essentially, QTIP is used as a high dimensional T.T, quantizer within BlockLDLQ and drop-in replacement for vector quantization in BlockLDLQ. The regular blockLDLQ step Q(W + (W W))) exactly the same, and the only difference in how rounds. Instead of rounding each row of x W (W- W)A A independently, it groups T. rows into a block round m/T, high-dimensional sequences.
A.3
Implementation Details
A.3.1 Code
Our code is available at
A.3.2
Hessian Generation
Hessian matrices were generated with 6144 sequences of length 2048 for Llama 1, 6144 sequences of length 2048 for Llama 2, 4096 sequences of 8192 for Llama 3, and 4096 sequences of 8192 for Ilama 3.1 except for 405B, which only used 2048 sequences due to time constraints. All sequences were sampled from the RedPajama dataset [
16

---

Algorithm 5 QTIP with BlockLDLQ
input We E R H R Tx, Ty L, k, V, code
W + Om, LDL - Ty-block LDL decomposition H
A L-I
for j & {n/TT - 1, n/y 2,.,, 0 do
<- W W.I,,,-(++17, (W :JTT: -y. Σ - x.reshape(m./TT Ty) Viterbi(ar, (I, k, V) bitshift trellis, C) (row-wise)
(T,, < E.reshape(m, Tyy)
end for
output Quantized W.
A.3.3 Hadamard Matrices
We use Hadamard matrices from Neil Sloane's website [1
A.3.4 Perplexity
We the OPTQ "Wikitext2" and "C4" test splits to calculate perplexity [1]
A.3.5
Fine Tuning
We use the same fine-tuning hyperparemters as QulP#, except that we do not perform matrix fusion 33).
This results in having to quantize 7 matrices (q. k, o, up, gate, down) instead of 4 (qkv, upgate, down). We quantize and fine tune in the following order: (v, q, k, o, up gate, down).
A.4 Example Generations
The following are some sample generations from Llama 70B (not instruction tuned) quantized to bits with QTIP, truncated to 256 tokens.
Prompt: It is truth universally acknowledged
It a truth universally acknowledged that single man in possession of good fortune must be in want of wife. However little known the feelings or views of such man may be on his first entering a neighbourhood. this truth is so well fixed in the minds of the surrounding residents, that he is considered the rightful property of some one or other of their daughters. "My dear Mr. Bennet," said his lady to him one day, "have you heard that Netherfield Park is let at last?" "But is." returned she; "for Mrs. Long has just been here, and she told me that Netherfield taken by young man large fortune from the north of England; that he came down on Monday in chaise and four to the place, and he agreed with Mr. Morris immediately; that he to take possession before Michaelmas, and that he is now in town in search of wife." "I see occasion for that. You and the girls may go. or you may send them by themselves, which do you prefer? Only let me know what you meaning do, for shall certainly not be home. You
QTIP:
much delighted with it that
17

---

Prompt:
Call me Ishamel
Call me Ishmael. Some years ago ago never mind how long precisely- having little or no money in my purse, and nothing particular interest me shore, thought I would sail about little and see the watery part of the world. It is a way have of driving off the spleen, and regulating the circulation. Whenever find myself growing grim about the mouth; whenever it is a damp, drizzly November in my soul; whenever I find myself involuntarily pausing before coffin warehouses, and bringing up the rear of every funeral meet; and especially whenever my hypochondriacal palate is out of sorts, and am desperately dull, and the dumps, and have such an overpowering sense of fatality that I find myself involuntarily gravitating toward Cemeteries and especially toward those open ones which are occasionally checred by the gleam the moon, I then take to the ship. There is nothing surprising in this,
QTIP:
bonfire, or cigar, or a pipe, or a lantern, or a candle, or a torch, or
The following is sample generation from Llama 3.1 405B Instruct quantized to bits with QTIP
Prompt: Give proof for why the square root of 2 is irrational:
QTIP: The proof that the square root of is irrational is follows: Assume, for the sake of contradiction, that v2 rational. This means it can be expressed the ratio of two integers, a and b, where b is non-zero:
V2 a/b
can further assume that a and b are not both even (i.e., they do not share a common factor of 2), because if they were, we could divide both by and obtain a smaller representation of /2. Squaring both sides of the equation, we get:
= a²² //2
Multiplying both sides by 6², we get:
2b = a²
This means that a² is even, because is equal to 2 times an integer (b²). Therefore, a must also be even, because the square of an odd number is always odd. Since is even, write 2c, where e is an integer. Substituting this into the equation above, we get:
26 (2c)²
Expanding the right-hand side, we get:
2b 2c²
Dividing both sides by 2, we get:
1/2 2c
This means that 6² is even, and therefore b is also even. However, this contradicts our assumption that a and are not both even. Therefore, our original assumption that /2 is rational must be false, and /2 is actually irrational. This proof is often attributed to the ancient Greek mathematician Euclid, and is still considered one of the most elegant and simple proofs of the irrationality of /2.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The main text describes limitations of the work that the authors were able perceive at submission time.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data provided not)?
Answer: [Yes]
Justification: This paper's method is well documented and can be easily reproduced. The code will be made publicly available at a later date as well.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
20

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code will be made publicly available a later date.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
gablit//uuides//odeeSuiiiiss Cosooooeeaamiissiooooooorryy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ "pipsscc/puullc//uuu /Ctee//guuddeeeeedSuuuummbbbba ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperpa- rameters, how they were chosen, type optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Yes, details are listed in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: It is standard practice in LLM quantization papers to not report error bars metrics.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
21

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See the appendix for details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We are not aware of any violations of the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper concerns foundational research on LLM quantization. There not direct path to any negative applications.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All such instances properly cited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "xh6oC0VRbi", "reviewer": "Reviewer_3PHK", "summary": "This work presents a new method of weight-only post-training quantization (PTQ) that uses trellis-coded quantization (TCQ) to achieve ultra-high-dimensional quantization. Although TCQ was introduced by Mao and Gray, QTIP transfers it into the LLM space and introduces new algorithms to make the method hardware efficient, thereby enabling fast inference.", "questions": "1. In the paper Line 236, you mentioned that QTIP can be fine-tuned. Did you try it? Did it provide any performance boost?\n2. Can you please add the formula to calculate the average bit? While it can be deduced from the details given, having the formula would make readability clearer.\n3. There are misspellings in line 46 and line 128: tk/kt , this should be corrected.\n4. Is the formula in line 125 for the trellis structure correct? If we look at line 101, one of these cases seems to be a misspelling.\n5. The numbers in Table 5 are not properly highlighted, especially where other methods are winning or matching.\n6. The inference speed reported for AQLM is very odd. The AQLM paper claims around a 1.3x speed-up over FP16, but in your case, it is much slower. I suppose this may happen due to other library overheads like transformers, Pytorch and Python. Did you try the code with compiled CUDA graphs? Please take a look at [this example](https://colab.research.google.com/github/Vahe1994/AQLM/blob/main/notebooks/aqlm_cuda_graph.ipynb) from there repository.\n\nOther:\n1. Line 70 begins with \"The effectiveness of these methods.\" It is a new section, and because of this, it is not clear which methods you are referring to.\n2. Cite the linear congruential generator (LCG) paper.\n3. In Line 240, should be mentioned that  AQLM doing block fine-tuning. Quip# doing full model fine-tuning. Full model fine-tuning also can be applied at AQLM, and the numbers will much Quip#. See Table 4 https://arxiv.org/pdf/2401.06118.", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of using TCQ for LLM is pretty interesting and, to the best of my knowledge, new.\n2. The paper is well-written, with explanatory figures and detailed explanations of the algorithms.\n3. Paper includes a comprehensive amount of experiments.\n4. The authors  provide extensive details to reproduce paper and promised to make the code public.", "weaknesses": "1. The ablation study is almost non-existent. For example, there is no analysis of the inference speed dependency on each introduced trick.Another thing you can do is experiment - perplexity (ppl) without the tricks, using L, k, and V values that still fit in the L1 cache, alongside the introduced algorithms.\n2. The L, K, and V parameters were not explored. It would be beneficial to see different V values compared at the same bit width.\n3. The paper contains a lot of notation and symbols that are either described once briefly or assumed to be known (e.g., see kd in line 27 line 128). This made reading the paper hard for me, as I had to constantly refer back to previous pages to understand what each symbol represents. For example, see line 229. This issue persists throughout the paper. Maybe it would be good to soft reintroduce symbols for time to time.\n4. To be frank, the performance gains for me are not that impressive, except perhaps for LLama-3 at 2 bits.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Ablations and fast inference (W1 and W2)\nThe main components of QTIP that enable fast inference are the bitshift trellis and compute-based codes. To understand why these are necessary, let us look at the L, K, and V parameters in trellis coding. L is the trellis size, K is the bitrate, and V is how many weights we quantize per step. Larger trellises improve quality, and increasing V can increase decoding throughput by amortizing ops. However, a larger V enforces more structure on the search space and may decrease quality. When L = KV, TCQ becomes K bit V dim VQ, bridging the two.\n\nThe bitshift trellis lets us avoid storing the trellis, which requires $2^{L+KV}L$ bits ($2^L$ nodes each with $2^{KV}$ edges). The bitshift trellis also enables parallel decoding, which is important since modern accelerators are highly parallel. The compute-based codes dissociate L from the amount of cache needed. While a pure-lookup codebook would take $16 \\* 2^L V$ bits of cache, scaling exponentially with L, the compute-based codes use a constant amount or none at all. Note that this means we could have used L > 16 in the paper to achieve higher quality at the cost of *encoding* time, since L does not affect decoding speed.\n\nTable A shows an ablation on L for quantizing Llama 2 7B with K=2, V=1, the bitshift trellis, a pure-lookup codebook, and no fine-tuning. L=8 is the largest L achievable if we had to store the trellis *and* codebook in the same amount of cache as the HYB code (2KiB). L=10 is the largest L achievable if we only had to store the codebook. As expected, increasing L improves quality. Table A also shows very little difference between an equal-sized LUT codebook and QTIP’s codes, meaning that QTIP isn't sacrificing quality for speed. However, an equal-sized LUT would need >10X more cache than the latest GPUs have, making the bitshift trellis and compute-based codes necessary to achieve both quality and speed.\n\nTable B shows an ablation on V with L=12 and 16, K=2, and the same settings as Table A. Increasing V generally decreases quality, but this can be recovered with a larger L. It is hard to measure V's impact on decoding speed since this is highly implementation and hardware dependent, so V is more of a user-chosen hyperparameter. \n### Table A\n|   L   | Trellis Size  |    CB size   |  total size  |  W2  |  C4  |\n|:-----:|:-------------:|:------------:|:------------:|:----:|:----:|\n| QuIP# |       -       |      8Kb     |      8Kb     | 8.22 | 11.0 |\n|   8   |    8.19 Kb    |    4.10 Kb   | **12.29 Kb** | 7.83 | 10.3 |\n|   10  |    40.96 Kb   | **16.38 Kb** |   57.34 Kb   | 7.49 | 9.67 |\n|   12  |   196.61 Kb   |   65.54 Kb   |   262.14 Kb  | 6.97 | 9.21 |\n|   16  |    4.19 Mb    |    1.05 Mb   |    5.24 Mb   | 6.83 | 8.92 |\n|   16  |    Bitshift   |     3INST    |      0Kb     | 6.82 | 8.96 |\n\n### Table B\n|      Codebook     |  L | V |  W2  |  C4  |\n|:-----------------:|:--:|:-:|:----:|:----:|\n|        LUT        | 12 | 1 | 6.97 | 9.21 |\n|        LUT        | 12 | 2 | 7.09 | 9.24 |\n|        LUT        | 12 | 4 | 7.55 | 9.88 |\n|        LUT        | 16 | 1 | 6.83 | 8.92 |\n|        LUT        | 16 | 2 | 6.79 | 8.97 |\n| QTIP HYB (no FT)  | 16 | 2 | 6.83 | 8.97 |\n|        LUT        | 16 | 4 | 6.92 | 9.07 |\n\n## Notation (W3)\nWe will update the camera ready to improve readability.\n\n## Performance Gains (W4)\nThe easiest way to see QTIP’s improvements are at 4 bits and in larger models, where fine-tuning has less impact (Table 3). Tables 3 and 4 show that at 4 bits, all 3 of the QTIP formulations significantly reduce the quantization error over QuIP# and AQLM, which is impressive given how well these methods do at 4 bits. Table 3 also shows that for most model sizes and bitrates, QTIP *without fine tuning* matches or exceeds QuIP# *with fine tuning*, showing the importance of using a good quantizer. Finally, these quality gains are essentially “free” over QuIP#, since QTIP offers the same fast inference speeds.\n\n## Fine-tuning QTIP (Q1)\nYes, the HYB code has a small codebook that can be fine-tuned. Table 4 uses fine-tuning with the small codebook included in the set of tunable parameters. Tuning the codebook helps slightly, but a better fine-tuning algorithm could probably do better.\n\n## Bit Accounting (Q2)\nQTIP has two sources of overhead beyond the quantized weights: sign vectors used for the random Hadamard transform (RHT), and the codebook in the HYB code (3INST and 1MAD do not have codebooks). For a nxm matrix, the sign vectors take up (n+m) bits. For the HYB code, the codebook takes up 2^Q x V x 16 bits (see Section 3.1.2 for more details). We used Q=9, but you can go down to Q~6 without noticeable degradation. At Q=9, the codebook uses 2KiB. Amortized over n*m entries, the codebook and sign vectors take up <0.01 additional bits. We will add these details to the appendix.\n\n## tK vs Kt (Q3)\ntK and Kt = t times K\n## L125 (Q4)\n125 should say $2^L \\times 2^{kV}$\n## Highlighting (Q5)\nWe will re-highlight the numbers to be more consistent.\n\n## AQLM inference speed (Q6)\nThe QuIP# authors recently released a more accurate generation timing script. We have re-timed all the methods on an RTX 6000 Ada. The numbers below are the average throughput for decoding 1024 tokens over 8 trials; we will update the paper with this. \n\n|    Method   |    2-7B    | 2-70B |\n|:-----------:|:----------:|:-----:|\n|     FP16    | 55.9 tok/s |  OOM  |\n|  AQLM 2 Bit |    81.5    |  8.78 |\n| QuIP# 2 Bit |     186    |  22.2 |\n|  QTIP 2 Bit |     188    |  23.5 |\n\n## L70 (O1)\n“These methods” refers to those listed on L66-68. We will clarify this.\n\n## LCG citation (O2)\nWe will update the paper with this.\n\n## AQLM fine-tuning (O3)\nThank you for pointing this out, we will update the paper with these numbers. We note that this does not change our analysis, which is that QTIP achieves better performance than both QuIP# and AQLM with the same general fine-tuning scheme, while preserving QuIP#'s fast inference.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V2uNXS107q", "reviewer": "Reviewer_1CRX", "summary": "This paper introduces trellis coded quantization (TCQ) into large language model (LLM) quantization, achieving ultra-high-dimensional quantization with less inference burden compared to traditional vector quantization (VQ) methods. The main innovations are a hardware-efficient \"bitshift\" trellis structure and fast compute-based Gaussian codes, enabling high-quality quantization and fast inference.", "questions": "Please see the weaknesses for details.", "limitations": "The paper lacks inference speed testing on more bit levels and additional devices.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The motivation to address the drawbacks of VQ is clear and intuitive.\n2. The proposed \"bitshift trellis\" and \"compute-based random Gaussian codes\" are innovative and improve the inference efficiency of TCQ.\n3. The performance is impressive, significantly enhancing 2-bit performance even without fine-tuning and optimizing quantization bits to 3-bit.\n4. The method significantly improves inference speed.", "weaknesses": "This is a solid paper introducing a novel and robust quantization format. Inference efficiency is crucial for new quantization formats, but this paper only presents the inference speed on the RTX 4090 with 2-bit quantization. Providing inference speeds for more bits (e.g., 3-bit, 4-bit) and more devices (e.g., RTX 3090, A100) could enhance the paper further.\n\nIt is possible that existing quantization computation kernels achieve speedup on the RTX 4090 but not on the A100. Such phenomena are normal, and adapting a new kernel to different devices is challenging. However, I encourage the authors to report inference speeds on various devices. Even if results are not favorable, an in-depth analysis with potential solutions would provide a more comprehensive understanding without affecting the paper's contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Updated Inference Speeds on More Devices and More Bitrates\n\nBelow are throughput numbers for decoding 1024 tokens on the RTX 3090, RTX A6000 Ampere, and RTX 6000 Ada, averaged over 8 runs. The kernels were not re-tuned for each device, so they could be made faster. These numbers were run using an updated inference speed script from the QuIP# codebase that is more accurate than the numbers in the paper. In all cases where the FP16 model fits on the device, QTIP is faster than FP16. \n\n|     GPU Model    | Model  | 2-bit tok/s | 3-bit tok/s | 4-bit tok/s | FP16 tok/s|\n|:----------------:|:------:|:-----------:|:-----------:|:-----------:|:----:|\n|     RTX 3090     |  2-7b  |     127     |     119     |     109     | 52.5 |\n|     RTX 3090     |  2-70b |     15.3    |     OOM     |     OOM     |  OOM |\n| RTX A6000 Ampere |  2-7b  |     116     |     106     |      95     | 43.5 |\n| RTX A6000 Ampere |  2-70b |     15.0    |     13.1    |     11.7    |  OOM |\n|   RTX 6000 Ada   |  2-7b  |     188     |     161     |     140     | 55.9 |\n|   RTX 6000 Ada   |  2-70b |     23.5    |     19.1    |     16.3    |  OOM |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yjuX670XWc", "reviewer": "Reviewer_F569", "summary": "This paper proposes QTIP, a new method for efficient post training quantization of LLMs. QTIP is a vector quantization method inspired from Quip# [1], but with no limitation in the dimension of the codebook. The work’s fundamental contributions are: 1) Propose to adapt trellis quantization to the compression of LLM pre-trained matrices. 2) Design efficient trellis and codebook in dimension (or 'sequence length') $=256$. 3) Integrate the trellis quantization into the Quip# [1] optimization process to fine-tune the pre-trained weights. The numerical experimental results of the proposed method outperform AQLM [2] and Quip# [1] on the C4 and Wikitext2 datasets.\n\n---\n\n[1] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks, 2024.\n\n[2] Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization, 2024.", "questions": "1) Table.1 shows the superiority of QTIP over scalar quantization and Quip# [1]. How does it translate to the case of 'real' pre-trained LLM matrices? Can you show some results similar to [2], to identify which matrix presents the lowest quantization error ? This would enable us to decouple the gains provided by QTIP, and understand better which part (the quantization process itself, or the fine-tuning machinery) is key for sota performances.\n2) Is 'hyb' codebook unique? Or do you have a different (fine-tunable) codebook for each LLM block?\n3) Do you think QTIP fine-tuning step is (computationally) competitive with respect to (low-rank) adapters methods such as LoRA [3] and LQ-LoRA [2] ? Or QTIP is only tailored for LLM compression ?\n\n\n---\n\n[1] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks, 2024.\n\n[2] Guo, H., Greengard, P., Xing, E., and Kim, Y. (2023). Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. In The Twelfth International Conference on Learning Representations.\n\n[3] Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. (2021). Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.", "limitations": "A comparison of inference speedups is provided in section 4.3.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The authors proposed an interesting description of the limitations (in terms of codebook dimension) related to standard vector quantization, and a sound introduction to trellis quantization.\n2) This paper presents two innovative lookup-free code for (non-fine-tunable) trellis weight quantization: '1mad' and '3inst'. These methods obtain sota performances with respect to other non-fine-tuned approaches, and are efficiently implemented without the need to store the codebooks.\n3) The author also propose an hybrid (fine-tunable) trellis weight quantization: 'hyb'. Though 'hyb' requires to store the codebook, it allows one to fine-tune the codewords, and thus enable QTIP to compete with fine-tuned sota approaches [1,2].\n\n---\n\n[1] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks, 2024.\n\n[2] Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization, 2024.", "weaknesses": "1) QTIP is mainly based on Quip# [1]: 'we use QTIP as a quantizer in QuIP#’s BlockLDLQ' (line.224). The code is not provided, but the authors explain that the whole quantization and fine-tuning process is based on Quip# [1]. Hence the contribution is more to the field of signal quantization in general, and LLM compression is an application.\n2) QTIP natively reuses the incoherence processing from [1]. But this pre-processing step may erase the native correlation patterns that pre-trained LLM matrices bear [2,3]. No discussion about this process (and the blockLDLQ) is provided.\n3) QTIP (with fine-tuning) update the pre-trained weights 'in a blockwise fashion' (line.247). This step is computationally intensive regarding the size of the matrices (and the optimizer states related to it), and it may impact the efficiency of the quantization process.\n4) Very few details are given regarding the experiments design (which calibration dataset is used for fine-tuning 'hyb' codewords?), what may affect the results reproducibility (in addition the code is not publicly available).\n\n---\n\n[1] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks, 2024.\n\n[2] Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. (2023). Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629.\n\n[3] Guo, H., Greengard, P., Xing, E., and Kim, Y. (2023). Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. In The Twelfth International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## QTIP's contribution (W1) \nThe focus of QTIP is on *what to quantize with* (e.g. VQ, TCQ), and not *how to quantize* (e.g. GPTQ, fine-tuning). Choosing a good quantizer is hard since LLM weight matrices have outliers and small-batch LLM inference is memory bound, necessitating fast decoding. While the field of signal processing has known for a while that TCQ achieves lower distortion than VQ, there is little to no existing work on designing trellis quantizers for fast parallel decoding. This is the “missing piece of the puzzle” needed to make TCQ work for LLM compression, which QTIP addresses. \n\nOur experiments used QuIP#’s BlockLDLQ framework for two main reasons. First, QuIP# is a state-of-the-art vector quantization framework that supports fast inference, making it a good baseline to show the benefits of TCQ. Second, LDLQ variants are optimal among adaptive rounding methods that use linear feedback from the Hessian. By using BlockLDLQ, we can be reasonably certain that the empirical improvements are from TCQ and not spurious interactions between the quantizer and a subpar rounding algorithm. Finally, we note that QTIP could be used as a drop-in replacement for VQ in other rounding algorithms. QTIP’s orthogonality to the actual rounding method further broadens QTIP’s practicality.\n\n## Incoherence Processing (W2)\n\nBoth QuIP and QuIP# detail how incoherence processing affects the Hessian and how we can use incoherence to bound the proxy error tr(WHW.T). We refer the reviewer to QuIP and QuIP# to see how incoherence processing can benefit quantization, even when using Hessian information. \n\nQTIP uses the random Hadamard transform (RHT) to perform incoherence processing. The RHT is fully invertible up to numerical error, meaning that the Hessian matrices do not lose information from incoherence processing bar catastrophic imprecision. We ran the quantization algorithm in fp64 and did not observe differences vs. fp32, so numerical error is not a dominating factor in the quantization step.\n\n## Cost of Fine-Tuning (W3)\n\nQTIP’s focus is on the quantizer itself and not fine-tuning. Whether the end user chooses to fine-tune or not is a “deployment choice,” and in either case QTIP offers improvements over SOTA vector quantization methods. Table 3 shows that *even without any fine-tuning*, QTIP usually outperforms SOTA vector quantization based methods *that use fine-tuning*, meaning that fine-tuning isn't strictly necessary to make QTIP SOTA. Table 4 shows that QTIP still gives significant improvements after fine-tuning. In fact, QTIP holds up where fine-tuning “fails.” QTIP almost halves the 4 bit perplexity gap vs. QuIP#, while fine-tuning has little effect at 4 bits. Finally, since QTIP is orthogonal to fine-tuning, we expect that new fine-tuning algorithms will compose with QTIP to produce even better quantized models.\n\n## Implementation Details (W4)\nMost of these details, including datasets, are available in the Appendix. We will add additional information in an updated manuscript and the code will be made available at a later date.\n\n## Quantization Error (Q1)\nTable 1 shows the distortion for an i.i.d. Gaussian sample. If we re-run Table 1 with actual weight matrices and incoherence processing, then we get very similar results since incoherence processing produces approximately i.i.d. Gaussian weights.\n\n|           Source           |  1MAD MSE | 3INST MSE |  HYB MSE |\n|:--------------------------:|:-----:|:-----:|:-----:|\n| i.i.d. Gaussian (Table 1)  | 0.069 | 0.069 | 0.071 |\n|    Llama 2 7B 0_v + IncP   | 0.069 | 0.069 | 0.070 |\n\nHowever, LLM quantization algorithms usually minimize the expected activation error (proxy error, Eq 1) instead of MSE (Table 1). Using a calibration set of 25 million tokens from RedPajama, scalar quantization achieves a relative error of 0.073, VQ (QuIP# E8P) 0.060, and TCQ (QTIP 3INST) 0.045, when quantizing the 10_down layer of Llama 2 70B to 2 bits without fine-tuning. Like in Table 1, QTIP reduces the proxy error over SQ and VQ.\n\n## HYB Uniqueness (Q2)\nIn our experiments, we use a different codebook per linear layer. Since this codebook is very small (1K entries) relative to the matrix size (>>1M entries), it adds < 0.01 bits per weight. If a per-layer codebook is not possible, a shared codebook should also work fine. 1MAD and 3INST use the same codebook per layer and also achieve SOTA performance. This is due to incoherence processing producing approximately i.i.d Gaussian weights, so the codebooks are all essentially Gaussian.\n\n## LLM Quantization Fine-Tuning vs. LoRA-Style Methods (Q3)\n\nTo clarify, QTIP’s focus is on the quantizer, not the rounding or fine-tuning algorithm. Table 4 used QuIP#’s tune-as-you-go fine-tuning method. This method, and other similar ones like AQLM’s fine-tuning algorithm, tune the quantized model to recover the original model. In contrast, LoRA and LQ-LoRA are aimed at fine-tuning to new downstream tasks. These are two fundamentally different problems, so it is difficult to compare methods across problem areas. It may be possible to adapt methods to solve both problems, but that is beyond the scope of this work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JE6gecNbNl", "reviewer": "Reviewer_ZTfk", "summary": "The paper introduces Trellis-based compression for quantization. Essentially, the method applies incoherence processing on the weight matrices, turning the approximately into Gaussians, such that the best compression method available to compression roughly Gaussian data can be applied to it. In this case, the authors choose a Trellis-based compression scheme.\nThey also introduce a specific Trellis setup with little overhead, for the setting where Gaussian data is the name of the game, making the method feasible in practice. According to the authors, this approach is novel. \nThe authors show that there is a slight benefit of this compression method over other codebooking methods, such as GPTVQ, AQLM and QUIP#, at next to no extra overhead in terms of the implementation. \n\nAs a note for the other reviewers and authors - I have no idea about trelisses, or the relevant literature. I am only knowledgeable about neural network quantization. So the veracity of some of the claims, like a novel Gaussian specific Trellis, I cannot comment on.", "questions": "Is section 24-32 entirely correct? Technically you have 2^kd vectors, but nobody ever fills out the whole space with the codebook. So it’s always a subset \\subset R^{2^kd} x d you are selecting for your codebook C. So it’s not really fair to say VQ requires exponential time and space… if you keep your codebook size similar, it does not require exponential time and space. Similarly, the O(2^{kd}d ) complexity is a bit misleading. You can also say the lookup is log(|C|) depending on the size of the codebook you chose. I am not being festicious here, vector quantization methods often take a fixed set of vectors \n\nIs the analysis is 116 sensible? Why would one ever have an unstructured k-bit codebook? There is always a way to add structure. \n\nTable 1 - Is this done keeping the overhead of each method the same? I can always increase the dimensionality of VQ to get lower values right? This comparison should only make sense if the compute/storage overhead is the same.  \n\nHow does this Trellis thing actually work? I'm trying to follow figure 2. I'm assuming the firs thing you need to store is the starting value 00. And then you follow the up-and-down errors each costing you a bit, so the red arrows should give you down, down, up, down down... So how do we end up with 0010110? Wouldn't it be 00 | 00 1 00 or 00 | 11 0 11? \nAlso - what happens in your code if the next value cannot be transitioned to? In your example, if 0.1 follows 0.3, you're out of luck... does it then give you the closest value 0.3?\n\nDo you have a simple explanation for why Trellis-based coding would work better than normal codebooks? Just 'dimensionality' doesn't sit well with me, because that doesn't really explain anything concrete. \n\n\nIf I get more confidence from the authors on the speed of the kernels and the overhead of this method, with a discussion about non-gpu devices, especially given the very small gains they achieve - I am happy to increase my rating.", "limitations": "No pain points here", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper is very well-written, and quite easy to follow. It also feels like it's high-quality, well-researched, and the authors clearly know this field and their Trellis field very well. \n- I have not seen this Trellis-based compression scheme in the deep learning literature before, so to my knowledge it is novel\n- I can believe this compression scheme is better than normal VQ-based methods. Once you've turned your data into Gaussians, anything that compresses Gaussian distributed data better is likely to work.\n- The paper actually implemented the idea on GPU kernels, and show that the overhead is negligible. Essentially saying the improvements are free.", "weaknesses": "- The results are not TOO significant. In terms of perplexity, we're talking 0.1 ppl compared to QUIP# here. And the zero-shot results are always noisy, so tough to say what's faster.\n- Because the compression is likely theoretically slightly more accurate, what remains is the inference-speed of this method. Now the authors, to their credit, do address this with their GPU implementation. But, because the improvements in perplexity are so little, for fair comparison we have to be absolutely sure that the latency for the kernel is the same. Otherwise, we might as well increase the codebook size for QUIP# a little and get a better trade-off, or have one layer in a slightly higher bit-width or whatever. The Speed analysis in Table 7 does not totally satisfy my need for a thorough analysis on this. Since this is so important for the paper, I would like to see an analysis for the kernels themselves, analyze the roofline points for the band-width/compute trade-offs (does your method do more compute than QUIP#?), and have this done for more bit-widths. I believe the authors have to be more thorough on this, else I would not recommend anyone implementing the paper, and the trellis compression is relegated to being a fun curiosity instead of a practical method.\n- We don't all run things on GPUs - there are devices, with hardware accelerators, CPUs etc. What does hypothetical overhead look for this method compared to simple codebook look-ups? Some hardware has actual Sillicon for codebook look-ups, or specific instructions for it like in ARM CPUs. Would your method in these cases also still be faster? Since quantization is not only a 'speed-things-up-on-gpu' topic, I would like to see a discussion on this in the paper.\n\nEditorial Notes\n5 - […] PTQ approaches have converged. I don’t think convergence has really happened, so papers do this, some don’t. \n24-32 What is k? it’s undefined in the intro\n160 psuedorandom -> pseudorandom", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## QTIP’s significance (W1)\n\nTables 3 and 4 show QTIP’s strengths. Table 3 indicates that QTIP *w/out fine-tuning* consistently beats VQ-based approaches *w/ fine-tuning*, showing that there is significant value in better quantizers. QTIP also succeeds where fine-tuning “fails.” Fine-tuning does not reliably improve QuIP# at 4 bits, but QTIP consistently outperforms QuIP# there. This is even more impressive since 4 bit QuIP# is almost lossless. Table 4 shows that even after fine-tuning, QTIP still consistently improves over QuIP#. At all bitrates, QTIP can reduce QuIP#’s perplexity gap by >25%, an impressive feat.\n\n## Inference speed analysis (W2)\n\nDecoding QuIP#’s E8P takes roughly 3 instructions per weight. This is similar to QTIP’s codes. However, E8P reads 8 weights at once while QTIP reads fewer (2 for HYB, none for 3INST/1MAD). The achievable decoding speed for both methods is dependent on hardware properties (**see comments for more details**). However, QTIP’s strength lies in its flexibility. The general QTIP framework just requires efficiently generating a pseudorandom Gaussian. The paper gives 3 very different code constructions as examples, two of which don’t even require lookups. In contrast, E8P is very rigid and specially constructed. QuIP# wouldn’t work on devices with little cache, and still scales exponentially w.r.t. dimension and bitrate. For example, even a 3 bit codebook using E8P’s construction wouldn’t fit in cache on modern GPUs. QuIP# uses residual quantization to scale, but RQ is suboptimal vs. a single quantizer.\n\n## Other hardware (e.g. ARM) (W3)\n\nDifferent hardware has different properties and instructions, but we believe that QTIP can be fast on a broad class of accelerators due to its flexibility. QTIP only requires generating a pseudorandom Gaussian efficiently, and can work on devices with no cache as well as devices with lookup hardware. The paper mainly targets Nvidia GPUs due to their popularity. That said, we answer this question from two perspectives:\n\n**Can look-up instructions replace TCQ?** Generally no, if you want the same quantized model quality. The look-up instructions in most architectures are limited. For example, ARMv8 NEON’s vqtbl4q_u8 intrinsic looks up 16 idxs in a 64-entry codebook. QuIP#’s E8P has 256 8D entries. We know that QTIP outperforms QuIP#, and vqtbl4q_u8 can’t even handle QuIP#. Reducing codebook size would reduce quality, so only using look-up instructions aren’t enough for QTIP’s quality.\n\n**Can look-up instructions work with TCQ?** Yes, this is essentially the HYB code. In the ARM example, we can use a 6 bit 1D codebook with HYB (Q=6, V=1). Quantizing Llama 2 7B to 2 bits with this setup and w/out fine-tuning gives 6.89 Wikitext2 perplexity – essentially the same as 3INST. We could implement this by packing the 6 bit codebook into registers and using vqtbl4q_u8.\n\n## Editorial Notes\n\n[5] To the best of our knowledge, AQLM and QuIP# are the highest quality LLM PTQ methods currently available. Both perform vector quantization. [24-32] K is the bitrate; we will fix this.\n\n## L24-32, exponential space and time (Q1)\n\nWe’re not sure what you mean here. VQ with an unstructured codebook takes $O(2^{kd}d)$ space and time since closest vector search is NP-hard (https://cseweb.ucsd.edu/~daniele/papers/CVPP.pdf). “Exponential” refers to how the space and time complexity of VQ scale with dim and bitrate, not whether it is possible to perform VQ fast. In practice, people do use structured codebooks to reduce cost by a constant factor. This is what QuIP# uses to do 8D VQ for roughly the cost of 4D VQ (256X reduction). However, this doesn’t solve exponential scaling – it just makes VQ cheaper.\n\n## L116, discussion on structured search (Q2)\n\nSearching over an unstructured k bit T dim codebook requires $O(2^{kT})$ time. You’re right that adding structure can make this tractable. For example, product quantization (PQ) with a group size of g makes the codebook the outer product of a g-dim codebook, reducing the runtime to $O(T2^{kg}/g)$. However, this is no longer simply VQ. TCQ is another way to add structure by making the codebook entries fixed-length walks on a trellis. The tradeoff to adding structure is increased distortion. The rate-distortion limit lower-bounds the distortion for an unstructured k bit quantizer. Table 1 shows that TCQ gets much closer to this limit than vector/scalar PQ, making it a better way to add structure.\n\n## Table 1 (Q3)\nTable 1 computes distortion w/out considering speed. Inf-dim VQ would achieve the rate-distortion limit. 8D VQ with E8P is the limit of what we can do for fast inference on current GPUs, so the fact that QTIP can reduce the rate-distortion gap by >2/3 while using fewer resources makes it a much better quantizer.\n\n## Figure 2 (Q4) \n\nIn the bitshift trellis, the last L-kV bits of a state are the first L-kV bits of the next state, so we only need to store the kV bit “delta” for each entry in the input sequence. 0010110 represents *node indices* 00, 01, 10, 01, 11, and 10. Your coding scheme stores which edge to take, but would require knowing the graph structure to decode. The bitshift trellis bakes the structure into the numbering scheme, enabling fast decoding. Regarding transitioning to the next value, this is why we want a large trellis (large L) and randomized codebook, since that increases the probability we can transition to arbitrary values. Naive TCQ has exponential space complexity in L, so we need QTIP’s compute-based codes that dissociate L from space to get high quantization quality on today’s hardware.\n\n## Why does TCQ work? (Q5)\n\nMao and Gray’s RPTC paper (https://ieeexplore.ieee.org/document/5895067/) has theory on TCQ. In short, TCQ satisfies the four necessary conditions of an optimal quantizer. Intuitively, TCQ performs better because it is stateful, letting you “switch” between which size $2^{KV}$ part of the larger size $2^L$ space you are searching in at every step.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their detailed reviews. As multiple reviewers noted, QTIP is well motivated and novel (1CRX, 3PHK, ZTfk), and simultaneously achieves strong quality and fast inference (1CRX, F569, ZTfk). Below, we have written individual responses to reviewers. We have also run a number of new experiments. These include ablations on L and V (3PHK), more accurate timing numbers with an updated script (3PHK), 3 and 4 bit inference kernels (1CRX), and timing on different GPUs (1CRX).

**To satisfy the character count, we have abbreviated the weaknesses/questions we are responding to. "WX" referrs to weakness X and "QY" question Y according to the order in the review.**

Meta-review: Application of TCQ for quantizing trained weights results in strong performance and fast inference, as the reviewers mostly agreed. TCQ is a well-established practice in the communicaiton field and this paper is mainly about borrowing it to model qualitization in ML; neverthless, bitshift trellis and compute-based random Gaussian codes are important new features and the value is significant. Ablation requests were complied to the satsifaction of the reviewer. Studies on larger bit width and variety of devices were also added, which made the work more convincing.
Decision: Accept (spotlight)