Title: QTIP: Quantization with Trellises and Incoherence Processing
Venue: NeurIPS 2024
Accepted: yes
Keywords: quantization

llms

trellises

fast inference

post training quantization

trellis coded quantization

model compression

computed codes
Abstract: Post-training quantization (PTQ) reduces the memory footprint of LLMs by quantizing weights to low-precision datatypes.
Since LLM inference is usually memory-bound, PTQ methods can improve inference throughput.
Recent state-of-the-art PTQ approaches use vector quantization (VQ) to quantize multiple weights at once, which improves information utilization through better shaping.
However, VQ requires a codebook with size exponential in the dimension.
This limits current VQ-based PTQ works to low VQ dimensions ($\le 8$) that in turn limit quantization quality.
Here, we introduce QTIP, which instead uses trellis coded quantization (TCQ) to achieve ultra-high-dimensional quantization. 
TCQ uses a stateful decoder that separates the codebook size from the bitrate and effective dimension. 
QTIP introduces a spectrum of lookup-only to computed lookup-free trellis codes designed for a hardware-efficient "bitshift" trellis structure; these codes achieve state-of-the-art results in both quantization quality and inference speed.

QTIP: Quantization with Trellises and Incoherence Processing

Abstract

Post-training quantization (PTQ) reduces the memory footprint of LLMs by quantizing weights to low-precision datatypes. Since LLM inference is usually memorybound, PTQ methods can improve inference throughput. Recent state-of-the-art PTQ approaches use vector quantization (VQ) to quantize multiple weights at once, which improves information utilization through better shaping. However, VQ requires a codebook with size exponential in the dimension. This limits current VQbased PTQ works to low VQ dimensions ( ≤ 8 ) that in turn limit quantization quality. Here, we introduce QTIP, which instead uses trellis coded quantization (TCQ) to achieve ultra-high-dimensional quantization. TCQ uses a stateful decoder that separates the codebook size from the bitrate and effective dimension. QTIP introduces a spectrum of lookup-only to computed lookup-free trellis codes designed for a hardware-efficient 'bitshift' trellis structure; these codes achieve state-of-theart results in both quantization quality and inference speed.

1 Introduction

Large language models (LLMs) have accelerated advancements in fields ranging from natural language processing [34] to scientific modeling [28]. However, the largest LLMs have hundreds of billions of parameters that can take over a terabyte of memory to load in half-precision; this size poses significant challenges for the practical deployment of LLMs [33, 17, 2]. For example, smallbatch autoregressive decoding, a common form of inference for LLMs, is memory bound [35]. Even on a modern datacenter GPU with ≈ 3 TB/s memory bandwidth, a large LLM ( > 200 GB) can only be directly run at < 20 tokens per second and may require multiple devices [35]. One way to accelerate inference is by compressing LLMs. This directly reduces the memory footprint of the model and increases the theoretical maximum inference throughput on any given machine.

One form of compression, weight-only post-training quantization (PTQ), quantizes trained model weights to lower precision datatypes [9, 35, 5]. The latest state-of-the-art weight-only PTQ methods, QuIP # and AQLM, use vector quantization (VQ) to achieve high-quality 2-bit models [35, 11]. In VQ, a vector x ∈ R d is quantized to one of 2 kd vectors in R d that form a codebook C ∈ R 2 kd × d . Ahigher vector dimension d allows for better codebook shaping and packing density, improving information utilization [19]. However, unstructured VQ requires exponential time and space in both the bitrate and dimension, limiting its practicality. During quantization, VQ costs O (2 kd d ) time to perform nearestneighbor rounding to C , and during inference, C must fit in hardware cache for fast lookups. This exponential scaling limits how high d can be and thus the advantages of VQ over scalar quantization.

Incoherent W

T

y

T

x

00

01

11

10

00

01

11

10

00

01

11

10

00

01

11

10

'Bitshift' Trellis Quantizer

Ultra-High Dimensional

Quantization

QTIP

BlockLDLQ

10 1

3.0

3.5

4.0

4.5

5.0

5.5

6.0

Wikitext2 Perplexity (ctx. 4096)

Llama 2 Scaling

QuIP# 2 Bit

AQLM 2 Bit

FP16 @ 4 Bit

To address this limitation, we propose QTIP, which uses trellis-coded quantization (TCQ) to enable tractable ultra-high-dimensional ( > 100 ) quantization and improve quantization quality over prior VQ-based approaches. In the simplest scalar form of TCQ, a lengthT sequence S is statefully quantized using a trellis - a directed graph G with 2 L nodes, each with 2 k incoming and outgoing edges and a scalar value [24]. The reconstructed sequence ˆ S corresponds to the node values of a lengthT walk on G , and quantization finds the walk that minimizes some distortion metric on S and ˆ S . Since neighboring entries in ˆ S are connected by one of 2 k edges, we only need to store which edge an entry came from, which takes k bits. For additive distortion metrics such as squared error, the optimal ˆ S can be found with the Viterbi algorithm, which runs in O (2 L T ) time [13, 24]. This means that the cost of quantization is independent of the bitrate k and linear in the sequence dimension T , enabling tractable high dimensional quantization.

However, TCQ is not free. During inference, vanilla TCQ requires storing both G and the size 2 L × V node value codebook, which can be too large to fit in cache. TCQ-quantized sequences also cannot generally be decoded in parallel, as t th elment of ˆ S could depend on up to the first tk encoded bits. In QTIP, we solve these issues by introducing a series of fast compute-based Gaussian codes designed for the hardware-efficient 'bitshift trellis.' Specifically, the bitshift trellis supports parallel decoding, does not require storing G , and our compute-based codes eliminate needing to store a large node value codebook. This enables high-quality quantization of Gaussian sources while supporting fast inference, and we adopt incoherence processing with the random Hadamard transform to ensure that LLM weights are approximately i.i.d Gaussian distributed. Altogether, QTIP

Achieves a state-of-the-art combination of weight-only LLM PTQ quality and fast inference through hardware-efficient trellis and codebook design.

Introduces multiple novel hardware-efficient ( ≤ 4 instructions per weight) compute-based random Gaussian codes for TCQ on i.i.d. Gaussian sources.

2 Background and Related Works

We focus on weight-only post-training quantization (PTQ) of LLMs in this work; other modelcompression approaches include quantization-aware training (QAT) and pruning. These methods are not strictly orthogonal to each other, as one could both prune and quantize a model. Since QTIP is a weight-only PTQ method, the rest of this section focuses on this area. Most current state-of-the-art PTQ methods round to minimize the per-layer proxy loss from Nagel et al. [27].

Here, ˆ W ∈ R m × n is the quantized weight matrix, x ∈ R n is an input activation, and H = E x [ xx T ] ∈ R n × n is interpreted as a proxy Hessian matrix. This objective is defined per-layer , making it tractable for very large models. However, minimizing it is difficult due to the nondifferentiable nature of quantization. Instead many works have proposed algorithms such as Hessianbased adaptive rounding, alternating optimization, and even coordinate descent to approximately minimize the proxy error [11, 5, 35, 14].

2.1 Incoherence Processing

The effectiveness of these methods depends on properties of W . For example, many works have observed that weight and activation outliers cause poor quantization quality [10, 20, 29]. In QuIP, Chee et al. [5] proposed that incoherence was important for quantifying this effect.

Definition 2.1 (Chee et al. [5]) . A Hessian H ∈ R n × n is µ -incoherent if its eigendecomposition H = Q Λ Q T has max i,j | Q ij | = max i,j | e T i Qe j | ≤ µ/ √ n . A weight matrix W ∈ R m × n is µ -incoherent if max i,j | W ij | = max i,j | e T i We j | ≤ µ ∥ W ∥ F / √ mn .

Essentially, incoherence means the weights and important rounding directions (Hessian eigenvectors) are not too large in any direction, aiding quantization. To make W,H incoherent (small µ ), one can perform incoherence processing (IP) by conjugating W,H with random orthogonal matrices U, V : ˜ W ← UWV T , ˜ H ← V HV T . QuIP # introduced IP with the random Hadamard transformation (RHT), which performs ˜ W ← V m S m WS n V T n , ˜ H ← V n S n HS n V T n where V k is a k × k Hadamard matrix and S k is a length k random sign vector. The RHT achieves, with probability ≥ 1 -δ , µ ˜ W = 2 log(4 mn/δ ) , meaning that ˜ W 's entries are approximately independently Gaussian distributed, which can aid quantization [35, 3]. We choose to build on incoherence processing here because the independent Gaussian-like weights it produces are suitable inputs for trellis coding [23].

2.2 Vector Quantization (VQ) for LLM PTQ

k -bit VQ quantizes a d dimensional vector S to one of 2 kd d -dimensional vectors that form a codebook C ∈ R 2 kd × d [1]. Since C is an unstructured collection of arbitrary vectors, VQ enables better shaping and packing density than scalar product quantization (SPQ), where each entry in S is quantized independently [19]. However, this also comes at the cost of exponential time quantization and exponential space inference: finding the nearest neighbor in C requires O (2 kd d ) time, and storing C requires O (2 kd d ) space. The current crop of state-of-the-art LLM PTQ methods, QuIP # and AQLM, both use VQ to achieve high-quality 2-bit models. Since the shaping advantage of VQ comes from high dimensionality, both QuIP # and AQLM attempt to maximize dimensionality. AQLM's uses a large 8D codebook (1MiB) that does not fit in L1 cache. QuIP # uses an 8D compressible codebook based on the E 8 lattice, which is highly symmetric. This codebook is compressible by 256 × and barely fits in L1 cache. In either case, the VQ dimension is effectively hardware-limited to ≤ 8 , motivating methods that enable even higher-dimensional quantization.

2.3 Trellis-Coded Quantization (TCQ)

TCQwas first proposed by Marcellin and Fischer [24] to apply the benefits of trellis coded modulation , a conceptually dual problem, to quantization. Define a ( L, k, V ) trellis G as a directed graph with 2 L nodes, each of which has 2 kV incoming and outgoing edges and a value ∈ R V ; these values form a codebook C ∈ R 2 L × V . To quantize a lengthT sequence S ∈ R T , each contiguous lengthV subsequence of S is assigned to a node ∈ G , with the restriction that the assigned nodes form a walk. The reconstruction ˆ S of S is then given by concatenating node values in the walk. When V = 1 , this setup describes Marcellin and Fischer [24]'s original scalar TCQ. When V > 1 , this describes TCVQ, which applies TCQ to vectors [12, 37].

Finding the optimal ˆ S under an additive distortion metric can be done with the Viterbi algorithm in O (2 L T ) time. This is linear in sequence length, enabling ultra-high dimensional quantization. For exposition, we briefly describe the Viterbi algorithm here. Concretely, if we want to quantize a T -length scalar sequence reinterpreted as a sequence of vectors s 1 , s 2 , . . . , s T/V ∈ R V using a trellis

code with graph G and codebook C , this corresponds to solving the optimization problem

This optimization problem can be solved exactly with dynamic programming via the value function

using the update rule

This Viterbi approach clearly takes time linear in T and in the number of edges of G ; with a few simple optimizations this can be brought to O (2 L T ) . In comparison, brute-force-searching all possible 2 kT codes-which is what we would need to do for an unstructured k -bit T -dimensional codebookwould take time proportional to 2 LT/V . The ability to tractably find the closest representable vector in R T , even for large T , is in some sense the 'main benefit' of trellis coding. For i.i.d sources, as L increases, TCQ efficiently approaches the infinite-length distortion-rate D R , which lower bounds the attainable distortion of a k -bit quantizer [19]. As shown in Table 1, when quantizing an i.i.d. Gaussian with k = 2 , the scalar Lloyd-Max quantizer attains 0.118 MSE, QuIP # 's 8D E8P codebook 0.089 MSE, our (QTIP) 256D L = 16 TCQ quantizer 0.069 MSE, and D R = 0 . 063 [21, 25, 35, 8].

3 QTIP

Quantizing with TCQ requires storing both the codebook ( 2 L × V ) and trellis structure ( 2 L × 2 kV ) during inference. These components are too large for fast inference when L ⪆ 12 , which is necessary for high quality. Furthermore, for a generic trellis, recovering the state (and so the decoded value) at step t th requires a graph walk using the first kt bits: this prevents parallel decoding. QTIP solves these problems with a novel combination of incoherence processing, a hardware-efficient 'bitshift trellis,' and fast compute-based random Gaussian codes. Incoherence processing makes W approximatelly i.i.d Gaussian, which reduces quantization to Gaussian source coding. The bitshift trellis removes needing to store the trellis structure during decoding and also enables parallel decoding. Finally, the fast compute-based random Gaussian codes remove the need to store the full codebook, completing the equation for fast inference. On the quality side, the fast random Gaussian codes enable the simple bitshift trellis to match complicated trellises and achieve state-of-the-art quantization quality.

The main focus of QTIP is on what to quantize with (i.e. TCQ) and not how to quantize (e.g. adaptive rounding or descent methods). The general construction of QTIP can be used as a drop-in replacement for VQ in any rounding framework. In the following sections, we first describe the 'bitshift' trellis (Section 3.1). Then, we describe a series of fast compute-based codes for i.i.d Gaussian sources, aligning with different types of hardware (Sections 3.1.1 and 3.1.2). Finally, we give an approximation for the tail-biting trellis problem, which lets us more efficiently load weights in hardware (Section 3.2).

3.1 'Bitshift' Trellis and Codebook Design

The bitshift trellis was introduced by Mao and Gray [23] as part of the 'random permutation trellis coder' (RPTC). In the bitshift trellis, node i has an edge to node j if ∃ c ∈ Z , 0 ≤ c < 2 kV , s.t. j = ( i 2 kV mod 2 L ) + c . Essentially, the top L -kV bits of j equal the bottom L -kV bits of i . This means that the first group of V weights depends only on the bits at positions { 1 , 2 , . . . , L } , the second only on bit positions { kV +1 , kV +2 , . . . , kV + L } , and in general the t th on bit positions { ( t -1) kV +1 , . . . , ( t -1) kV + L } . During inference, obtaining the next compressed group of V weights in a sequence only requires bitshifting by kV bits, which is supported on virtually all hardware. Furthermore, since each group of V weights only depends on a contiguous window of L bits in ˆ S , decoding can be parallelized. Figure 2 shows a simple ( L = 2 , k = 1 , V = 1) bitshift trellis. Note that edges only exist between nodes that overlap by 1 bit, and storing the quantized length 6 ˆ S indeed only requires 6 bits (plus the initial state).

Quantizing an i.i.d. source with the bitshift trellis is nontrivial because neighboring groups of weights sharing many bits can potentially lead to strong correlations (Figure 3 LL). The RPTC permutes

Codebook

00

01

11

10

00

01

11

10

0.5

0.1

0.3

0.8

00

01

11

10

00

01

11

10

00

01

11

10

00

01

11

10

0.54

0.03

0.72

0.19

0.26

0.89

0.5

0.1

0.8

0.1

0.3

0.8

Unquantized S:

Quantized Ŝ:

Minimize

Squared

Error

00

01

11

10

Trellis G

the codebook to decorrelate neighboring weight groups (Figure 3 RR) [23]. However, this requires storing the codebook or storing and applying the permutation, both of which are prohibitively expensive during decoding. Instead, QTIP introduces a series of compute-based codes to produce a psuedorandom code, which has the same decorrelating effect and admits fast inference. To match approximately i.i.d. Gaussian RHT-transformed matrices, these codes produce psuedorandom approximate Gaussians in as few as 2 instructions per weight (see Table 1 and Figure 3). To the best of our knowledge, these code constructions alone are novel and we are the first to propose a lookupfree Gaussian trellis code.

3.1.1 Lookup-Free Computed Codes

Here, we present two pure-computed lookup-free codes that produce a pseudorandom approximately Gaussian number from a L bit word, enabling fast decoding on cache-limited hardware. These codes avoid strong correlations and can be implemented in ≤ 4 hardware instructions per weight on NVIDIA GPUs. We present two codes here to illustrate that multiple such codes are possible: in practice a lookup-free code can be designed to use the instructions available on whatever hardware we want to run on.

Algorithm 1 (1MAD) first runs a linear congruential generator (LCG) to produce a pseudorandom 32-bit word [32]. This requires 2 instructions ( MAD and & ). It then sums the 32-bit word as four 8bit unsigned integers; this sum is approximately Gaussian distributed. This requires 1 instruction ( vabsdiff4 ). Finally, this sum must be scaled and shifted (another MAD ). Although there are only 2 10 representable values even when L > 10 , this does not empirically affect quantization quality. 1MAD requires choosing a and b to avoid strong correlations; we set a = 34038481 and b = 76625530 (Figure 3 LC).

Algorithm 2 (3INST) also first runs an LCG to produce a random 32-bit word X . Then, it XORs the bottom 16 bits of X with the mantissa bits, bottom two exponent bits, and sign bit of a magic FP16 number m to produce an FP16 number m 1 . It then repeats this with the top 16 bits of X to produce m 2 and returns m 1 + m 2 . This entire process can be implemented in 3 ALU instructions 1 with a MAD for the LCG, a lop3 to mask and XOR with a packed duplicated m , and then summing m 1 and m 2 .

3

3

3

4

4

2

2

2

2

1

1

1

0

0

0

0

1

1

1

2

-3

M

际

45

咖

A

86

F.8"

2

3

2

-3

4

2

4

3

2

-1

0

1

2

3

3

-2

-1

0

1

2

3

=4

2

0

2

4

=4

2

0

2

4

Algorithm 1 Computed Gaussian Code '1MAD'

m 1 + m 2 is approximately distributed by the sum of two mirrored exponential distributions, which is close to Gaussian. Like with Algorithm 1, a, b, and m must be chosen to avoid correlations; we used a = 89226354 , b = 64248484 , m = 0 . 922 (Figure 3 right).

3.1.2 Hybrid Lookup-Computed Codes

Here, we describe a hybrid computed-lookup code that computes a pseudorandom (or hashed) index into a 2D vector codebook ( V = 2 ). This code is tailored for modern GPUs, which have enough cache for a small in-memory LUT-one benefit of using such a LUT over a purely computed codebook is that a LUT can be fine-tuned after quantization. Algorithm 3 first performs the hash X ← X 2 + X to mix the lower order and upper order bits of X [18]. Then, it takes bits (14 -Q +1) -14 (0 indexed) as an index into a 2 Q × 2 LUT to get two 16-bit floats. (The reason why we chose a 2D codebook here is that shared memory on NVIDIA GPUs is accessed in 32-bit-word elements, and each such word can contain two 16-bit floats.) Finally, it XORs bit 15 of X to flip the sign of the second entry of the codebook vector. Algorithm 3 can be implemented with MAD , bitshift, mask, and lop3 , giving an amortized 2 instructions per weight. This effectively assigns a L bit word to one of 2 Q +1 2D vectors, each of which can be fine-tuned to improve quality. Algorithm 3 can also be implemented to XOR bit 31 alongside bit 15 (this is free in the lop3 ) to give an effectively 2 Q +2 -sized codebook, which can improve quantization quality. We only realized this after running all the experiments, so the numbers in this paper use the 'one sign flip' version of Algorithm 3. In QTIP, we initialize the LUT using K-means on an empirical 2D i.i.d. Gaussian distribution.

3.2 Tail-Biting Trellises

Directly quantizing a lengthT sequence to a ( L, k, V ) trellis results in a total of kT + L -kV bits since the starting state takes an additional L -kV bits to store. If we run inference on a machine with w -bit words where w | kT , we must read an extra ⌈ L -kV w ⌉ w -( L -kV ) wasted bits per sequence. For common w (e.g. 32), setting L = kV + w makes the Viterbi algorithm intractable. One way to solve this is by enforcing that the start and end state share L -kV bits, i.e. the trellis is tail-biting [4]. Exactly solving the tail-biting trellis problem via dynamic programming takes time quadratic in the state space ( 2 L ), making this problem intractable for reasonable L ≥ 12 [30]. However, since RHT-processed weights are approximately i.i.d., simple algorithms can be effective for approximately solving the tail-biting problem. We propose Algorithm 4, which first rotates the sequence by T/ 2 , quantizes it, and then extracts the overlap between the rotated start and end states. It then requantizes the original sequence with this overlap as the tail-biting overlap. This only requires two Viterbi calls

Algorithm 2 Computed Gaussian Code '3INST'

Algorithm 3 Hybrid Computed-Lookup 2D Gaussian Code 'HYB'

in total. Table 2 shows that in practice, Algorithm 4 can find close-to-optimal tail-biting sequences while being significantly cheaper to run than other tail-biting approximation algorithms [30].

4 Experiments

Here, we present experiments quantizing the Llama family of models with QTIP [33, 34, 26]. These models offer strong performance across a wide range of sizes, allowing us to compare how different quantization methods perform and scale. We primarily compare QTIP against QuIP # and AQLM. For Llama 1, we include GPTVQ-2D instead of AQLM since AQLM does not publish Llama 1 numbers [36]. GPTVQ-2D performs 2D VQ inside GPTQ and offers strong performance. These methods outperform scalar quantization methods including GPTQ, AWQ, and OmniQuant; comparisons to those methods can be found in QuIP # and AQLM [20, 14, 29, 35, 11]. We mainly focus on the hybrid code (Section 4.2) since it is tailored for modern GPUs, and present a full suite of results for it. For the computed codes (Section 4.1), we present results for Llama 2.

Since the proxy error is not an additive distortion metric, we cannot minimize it by quantizing W as one sequence. Instead, for all experiments, we use QTIP as a quantizer in QuIP # 's BlockLDLQ, which allows us to simultaneously achieve high dimensionality and low proxy error [35]. Specifically, we quantize a block of T x × T y weights as a sequence, where T x and T y span the output and input dimensions of W , respectively. Since BlockLDLQ only specifies feedback along the input dimension, this is equivalent to BlockLDLQ with g = T y but a vector dimension of T x T y ≫ T y . This has the benefit of limiting the effect of g in BlockLDLQ's error bound gmµ 2 σ 2 tr ( H 1 / 2 ) 2 /n while achieving a high dimension for TCQ. Algorithm 5 in the Appendix describes this in more detail.

4.1 Lookup-Free Computed Codes

Here, we use 1MAD and 3INST with L = 16 , V = 1 , T x = T y = 16 . Setting T x = T y = 16 enables using a 16 × 16 MMAtile per trellis sequence to perform matrix multiplication during inference. 16 × 16 MMAtiles form the basis of many types of 'AI hardware,' making fast decoding relatively simple [6]. We do not perform fine-tuning since the codes themselves are not tunable, but these codes are fully compatible with QuIP # -style fine-tuning (recall that QuIP # 's codebook is also not tunable). Table 3 shows that both 1MAD and 3INST significantly outperform QuIP # without fine-tuning (AQLM does not have numbers without fine-tuning). Even at 4 bits, where all methods are close to lossless, QTIP results in significant improvements. Notably, the computed-code QTIP variants without fine-tuning outperforms both QuIP # and AQLM with fine-tuning on almost all models and sizes, showing that fine-tuning is not a silver bullet.

4.2 Hybrid Lookup-Computed Codes

Here, we use the hybrid lookup-computed code with L = 16 , V = 2 , T x = T y = 16 , Q = 9 . Setting Q = 9 gives a 2KiB codebook, which fits in L1 cache even after duplication for bank conflicts (32 × ) on modern GPUs. This codebook is differentiable, so we can finetune it: to evaluate this, we fine-tune using QuIP # 's methodology, tuning both the codebook entries and the as-yet-unquantized weights in a blockwise fashion. Table 5 shows the perplexity of quantized Llama 1 and 2 models. In all cases, QTIP outperforms the other vector quantization-based methods. Even at 3 and 4 bits, where QuIP # and AQLM are close to lossless,

QTIP roughly halves the perplexity gap. These results also show the importance of dimensionality. Note that the 3- and 4-bit Llama 2 70B numbers here match those in 3. Since Table 3 uses a purecomputed code without fine-tuning , fine-tuning has no effect in these regimes and the improvement over QuIP # is purely from dimensionality.

Table 6 shows zeroshot results computed with LM Eval, which are slightly random; QTIP generally matches or exceeds QuIP # and AQLM on these tasks [15]. Table 7 contains results on Llama 3. Like other works, we have observed that Llama 3 (especially 70B base) is harder to quantize than Llama 2 [16]. Since the contribution and focus of this work is what to round with (TCQ) and not how to round (BlockLDLQ), we only compare against the proximal baseline QuIP # , which uses BlockLDLQ with VQ. QTIP significantly improves upon QuIP # at all model sizes and bitrates, once again showing the dimensionality advantage of TCQ over VQ. Table 8 shows results for Llama 3.1 instruct-tuned models, including Llama 3.1 405B. At all sizes, QTIP achieves strong results. Notably, QTIP is able to match or exceed PV-Tuning, a recent quantization method that focuses on better finetuning algorithms [22]. However, PV-Tuning is based off of AQLM and inherits its slow inference speed, making it significantly slower than QTIP. Finally, Table 9 shows results for quantizing Llama 3.2 instruct-tuned models to 4 bits. Since the embedding layers are very large relative to the decoder layers for small Llama 3 models ( ≈ 500 -750 MB), quantizing the decoder layers to fewer than 4 bits does not make a significant difference on the final model size. Here, QTIP is still able to achieve a meaningful end-to-end compression rate (2.5-3X) without degrading the final model.

4.3 Inference Speed

Table 4 shows the batch size 1 inference speed of QTIP, QuIP # , and AQLM on Llama 2 7B and 70B with matrix fusion. Here, the design choices of QTIP and QuIP # become apparent. Whereas AQLM uses a codebook that is too large to fit in cache and thus prevents fast inference, both QTIP and QuIP # achieve significant speedups over FP16. Furthermore, while it is impressive that both QuIP # and QTIP are > 2 × faster than AQLM, it is even more impressive that QTIP is able to match QuIP # 's throughput with an effective dimension size of 256, or 32 × larger than QuIP # 's. This means that the improved quantization quality of QTIP comes with no additional inference-time cost . Although our empirical throughput numbers were timed on NVIDIA GPUs, QTIP can be fast on a broad class of accelerators due to its flexibility. QTIP only requires generating a pseudorandom Gaussian efficiently, and can work on devices with no cache as well as devices with lookup hardware. For example, if we were using a ARMv8 CPU, we could use the vqtbl4q_u8 NEON intrinsic to look up 16 indices in a 64-entry codebook. This would let us use a 6 bit 1D codebook with the HYB code (Q=6, V=1). Quantizing Llama 2 7B to 2 bits with this setup and w/out fine-tuning gives 6.89 Wikitext2 perplexity - essentially the same state-of-the-art quality as 3INST.

5 Conclusion

We present QTIP, a weight-only post-training quantization algorithm that achieves state-of-theart results through the use of trellis-coded quantization (TCQ). TCQ enables tractable ultra-high dimensional quantization, significantly reducing quantization distortion over vector quantization (VQ). However, naive TCQ does not admit fast inference due to sequential bottlenecks during decoding and needing to store a large codebook. QTIP solves this problem through a novel combination of incoherence processing, the hardware-efficient bitshift trellis, and fast computed codes. Specifically, QTIP introduces a series of compute-based pseudorandom Gaussian codes that, when used in

conjunction with the bitshift trellis and incoherence processing, simultaneously achieves state-ofthe-art PTQ quality and fast inference. QTIP improves quantization quality at all tested bitrates over the latest VQ-based PTQ methods, QuIP # and AQLM, further pushing the boundary of LLM PTQ. QTIP's codes use as few as 2 instructions per weight during decoding, enabling matrix-vector multiplication to run at over 80% of peak memory bandwidth on modern GPUs. Altogether, our results indicate that high dimensional quantization is necessary for high-quality compression, and QTIP is the first LLM PTQ method to scale to ultra-high dimensions while supporting fast inference.

References

An algorithm for vector quantizer design. IEEE Transactions on communications, 28(1):84-95, 1980.

Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series of open language models, 2023.

Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms, 2024.

A.R. Calderbank, G.D. Forney, and A. Vardy. Minimal tail-biting trellises: the golay code and more. IEEE Transactions on Information Theory, 45(5):1435-1455, 1999. doi: 10.1109/18. 771145.

Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. QuIP: 2-bit quantization of large language models with guarantees. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=xrk9g5vcXR .

Jack Choquette, Wishwesh Gandhi, Olivier Giroux, Nick Stam, and Ronny Krashinsky. Nvidia a100 tensor core gpu: Performance and innovation. IEEE Micro, 41(2):29-35, 2021. doi: 10.1109/MM.2021.3061394.

Together Computer. Redpajama: An open source recipe to reproduce llama training dataset, 2023. URL https://github.com/togethercomputer/RedPajama-Data .

Thomas M. Cover and Joy A. Thomas. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, USA, 2006. ISBN 0471241954.

Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 7750-7774. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr.press/v202/dettmers23a. html .

Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int8(): 8-bit matrix multiplication for transformers at scale, 2022.

Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization, 2024.

T.R. Fischer, M.W. Marcellin, and M. Wang. Trellis-coded vector quantization. IEEE Transactions on Information Theory, 37(6):1551-1566, 1991. doi: 10.1109/18.104316.

G.D. Forney. The viterbi algorithm. Proceedings of the IEEE, 61(3):268-278, 1973. doi: 10.1109/PROC.1973.9030.

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=tcbBPnfwxS .

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL https://zenodo.org/records/ 10256836 .

Wei Huang, Xudong Ma, Haotong Qin, Xingyu Zheng, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, and Michele Magno. How good are low-bit quantized llama3 models? an empirical study, 2024.

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024.

Alexander Klimov and Adi Shamir. A new class of invertible mappings. In Cryptographic Hardware and Embedded Systems-CHES 2002: 4th International Workshop Redwood Shores, CA, USA, August 13-15, 2002 Revised Papers 4, pages 470-483. Springer, 2003.

Victoria Kostina and Sergio Verdu. Fixed-length lossy compression in the finite blocklength regime. IEEE Transactions on Information Theory, 58(6):3309-3338, 2012. doi: 10.1109/TIT. 2012.2186786.

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for llm compression and acceleration, 2023.

Stuart Lloyd. Least squares quantization in pcm. IEEE transactions on information theory, 28 (2):129-137, 1982.

Vladimir Malinovskii, Denis Mazur, Ivan Ilin, Denis Kuznedelev, Konstantin Burlachenko, Kai Yi, Dan Alistarh, and Peter Richtarik. Pv-tuning: Beyond straight-through estimation for extreme llm compression, 2024. URL https://arxiv.org/abs/2405.14852 .

Mark Z. Mao and Robert M. Gray. Stationary and trellis encoding for iid sources and simulation. In 2010 Data Compression Conference, pages 3-12, 2010. doi: 10.1109/DCC.2010.8.

M.W. Marcellin and T.R. Fischer. Trellis coded quantization of memoryless and gauss-markov sources. IEEE Transactions on Communications, 38(1):82-93, 1990. doi: 10.1109/26.46532.

J. Max. Quantizing for minimum distortion. IRE Transactions on Information Theory, 6(1): 7-12, 1960. doi: 10.1109/TIT.1960.1057548.

meta llama. llama3. https://github.com/meta-llama/llama3 , 2024.

Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? Adaptive rounding for post-training quantization. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 7197-7206. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/nagel20a.html .

Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Callum Birch-Sykes, Michael Wornow, Aman Patel, Clayton Rabideau, Stefano Massaroli, Yoshua Bengio, Stefano Ermon, Stephen A. Baccus, and Chris Ré. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. 2023.

Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=8Wuvhh0LYW .

Y. Shao.Rose, Shu Lin, and Marc P. C. Fossorier. Tail biting trellis representation of codes: Decoding and construction. 1999. URL https://api.semanticscholar.org/CorpusID: 3727533 .

Neil Sloane. Hadamard Matrices - neilsloane.com. http://neilsloane.com/hadamard/ . [Accessed 02-02-2024].

W. E. Thomson. A Modified Congruence Method of Generating Pseudo-random Numbers. The Computer Journal, 1(2):83-83, 01 1958. ISSN 0010-4620. doi: 10.1093/comjnl/1.2.83. URL https://doi.org/10.1093/comjnl/1.2.83 .

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.

Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks, 2024.

Mart van Baalen, Andrey Kuzmin, Markus Nagel, Peter Couperus, Cedric Bastoul, Eric Mahurin, Tijmen Blankevoort, and Paul Whatmough. Gptvq: The blessing of dimensionality in llm quantization. arXiv preprint arXiv:2402.15319, 2024.

H.S. Wang and N. Moayeri. Trellis coded vector quantization. IEEE Transactions on Communications, 40(8):1273-1276, 1992. doi: 10.1109/26.156629.

A Appendix

A.1 Additional Results

A.1.1 Ablations on Trellis Size

Table 10 shows an ablation on L for quantizing Llama 2 7B with K = 2 , V = 1 , the bitshift trellis, a pure-lookup codebook, and no fine-tuning. L = 8 is the largest L achievable if we had to store the trellis and codebook in the same amount of cache as the HYB code (2KiB). L = 10 is the largest L achievable if we only had to store the codebook. As expected, increasing L improves quality. Table 10 also shows very little difference between an equal-sized LUT codebook and QTIP's codes, meaning that QTIP isn't sacrificing quality for speed. However, an equal-sized LUT would need > 10 × more cache than the latest GPUs have, making the bitshift trellis and compute-based codes necessary to achieve both quality and speed. Table 11 shows an ablation on V with L = 12 and 16, K = 2 , and the same settings as Table 10. Increasing V generally decreases quality, but this can be recovered with a larger L . It is hard to measure V 's impact on decoding speed since this is highly implementation and hardware dependent, so V is more of a user-chosen hyperparameter.

A.1.2 Zeroshot Results

A.1.3 Lookup-Only Codes

Here, we use a pure-lookup code ∼ N (0 , 1) with L = 14 , V = 1 , T x = 32 , T y = 8 , and QuIP # 's fine-tuning scheme. These parameters show what performance QTIP could achieve if we did not care about fast inference today . Specifically, a pure-lookup codebook is tunable, and setting T y = 8 reduces the BlockLDLQ group size while maintaining high dimensionality (256). This codebook uses 32KB; this only fits in GPU L1 cache with bank conflicts. Setting T x = 32 , T y = 8 corresponds to using a larger MMA tile size than current GPUs allow for. The largest tile size is usually 16 in the T x dimension, meaning that a 32 × 8 trellis needs two tiles. Thankfully, hardware required to serve

such a model quickly is likely only a few years away, as these parameters are only slightly outside of what today's hardware is capable of.

Table 15 shows that QTIP outperforms both QuIP # and AQLM at all compression ratios, with 3 bit QTIP achieving similar quality as 4 bit AQLM. While it is not fair to compare this QTIP setup with QuIP # , since QuIP # was designed for fast inference, we note that AQLM's VQ codebook uses 2 16 × 8 × 2 = 1 MiB. This is 32 times larger than the QTIP codebook here, and would require 32 MiB of L1 cache to read from without bank conflicts. Not only is this orders of magnitude larger than current L1 caches (256KB on the H100), it is even larger than many L2 caches!

A.1.4 Decoding Speed on Different GPUs

A.2 QTIP with BlockLDLQ

Here, we detail how we use TCQ within BlockLDLQ to produce our experimental setup. Essentially, QTIP is used as a high dimensional T x T y quantizer within BlockLDLQ and is a drop-in replacement for vector quantization in BlockLDLQ. The regular blockLDLQ step Q ( W +( W -ˆ W ) A ) is exactly the same, and the only difference is in how Q rounds. Instead of rounding each row of x = W +( W -ˆ W ) A independently, it groups T x rows into a block to round as m/T x high-dimensional sequences.

A.3 Implementation Details

A.3.1 Code

Our code is available at https://github.com/Cornell-RelaxML/qtip .

A.3.2 Hessian Generation

Hessian matrices were generated with 6144 sequences of length 2048 for Llama 1, 6144 sequences of length 2048 for Llama 2, 4096 sequences of 8192 for Llama 3, and 4096 sequences of 8192 for Llama 3.1 except for 405B, which only used 2048 sequences due to time constraints. All sequences were sampled from the RedPajama dataset [7].

Algorithm 5 QTIP with BlockLDLQ

A.3.3 Hadamard Matrices

We use Hadamard matrices from Neil Sloane's website [31].

A.3.4 Perplexity

We use the OPTQ 'Wikitext2' and 'C4' test splits to calculate perplexity [14].

A.3.5 Fine Tuning

We use the same fine-tuning hyperparemters as QuIP # , except that we do not perform matrix fusion [35]. This results in having to quantize 7 matrices (q, k, v, o, up, gate, down) instead of 4 (qkv, o, upgate, down). We quantize and fine tune in the following order: (v, q, k, o, up gate, down).

A.4 Example Generations

The following are some sample generations from Llama 2 70B (not instruction tuned) quantized to 2 bits with QTIP, truncated to 256 tokens.

Prompt: It is a truth universally acknowledged

QTIP: It is a truth universally acknowledged that a single man in possession of a good fortune must be in want of a wife. However little known the feelings or views of such a man may be on his first entering a neighbourhood, this truth is so well fixed in the minds of the surrounding residents, that he is considered the rightful property of some one or other of their daughters. 'My dear Mr. Bennet,' said his lady to him one day, 'have you heard that Netherfield Park is let at last?' 'But it is,' returned she; 'for Mrs. Long has just been here, and she told me that Netherfield is taken by a young man of large fortune from the north of England; that he came down on Monday in a chaise and four to see the place, and was so much delighted with it that he agreed with Mr. Morris immediately; that he is to take possession before Michaelmas, and that he is now in town in search of a wife.' 'I see no occasion for that. You and the girls may go, or you may send them by themselves, which do you prefer? Only let me know what you are meaning to do, for I shall certainly not be at home. You

Prompt:

Call me Ishamel

QTIP: Call me Ishmael. Some years ago-never mind how long precisely-having little or no money in my purse, and nothing particular to interest me on shore, I thought I would sail about a little and see the watery part of the world. It is a way I have of driving off the spleen, and regulating the circulation. Whenever I find myself growing grim about the mouth; whenever it is a damp, drizzly November in my soul; whenever I find myself involuntarily pausing before coffin warehouses, and bringing up the rear of every funeral I meet; and especially whenever my hypochondriacal palate is out of sorts, and I am desperately dull, and in the dumps, and have such an overpowering sense of fatality that I find myself involuntarily gravitating toward Cemeteries-and especially toward those open ones which are occasionally cheered by the gleam of a bonfire, or a cigar, or a pipe, or a lantern, or a candle, or a torch, or the moon, I then take to the ship. There is nothing surprising in this,

The following is a sample generation from Llama 3.1 405B Instruct quantized to 2 bits with QTIP.

Prompt: Give a proof for why the square root of 2 is irrational:

QTIP: The proof that the square root of 2 is irrational is as follows: Assume, for the sake of contradiction, that √ 2 is rational. This means it can be expressed as the ratio of two integers, a and b , where b is non-zero:

We can further assume that a and b are not both even (i.e., they do not share a common factor of 2), because if they were, we could divide both by 2 and obtain a smaller representation of √ 2 . Squaring both sides of the equation, we get:

This means that a 2 is even, because it is equal to 2 times an integer ( b 2 ). Therefore, a must also be even, because the square of an odd number is always odd. Since a is even, we can write a = 2 c , where c is an integer. Substituting this into the equation above, we get:

This means that b 2 is even, and therefore b is also even. However, this contradicts our assumption that a and b are not both even. Therefore, our original assumption that √ 2 is rational must be false, and √ 2 is actually irrational. This proof is often attributed to the ancient Greek mathematician Euclid, and it is still considered one of the most elegant and simple proofs of the irrationality of √ 2 .

Multiplying both sides by b 2 , we get:

Expanding the right-hand side, we get:

Dividing both sides by 2, we get:

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The main text describes limitations of the work that the authors were able perceive at submission time.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: This paper's method is well documented and can be easily reproduced. The code will be made publicly available at a later date as well.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code will be made publicly available at a later date.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Yes, details are listed in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: It is standard practice in LLM quantization papers to not report error bars on metrics.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See the appendix for details.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We are not aware of any violations of the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper concerns foundational research on LLM quantization. There is not a direct path to any negative applications.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All such instances were properly cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Trellis-based compression for quantization. Essentially, the method applies incoherence processing on the weight matrices, turning the approximately into Gaussians, such that the best compression method available to compression roughly Gaussian data can be applied to it. In this case, the authors choose a Trellis-based compression scheme.\nThey also introduce a specific Trellis setup with little overhead, for the setting where Gaussian data is the name of the game, making the method feasible in practice. According to the authors, this approach is novel. \nThe authors show that there is a slight benefit of this compression method over other codebooking methods, such as GPTVQ, AQLM and QUIP#, at next to no extra overhead in terms of the implementation. \n\nAs a note for the other reviewers and authors - I have no idea about trelisses, or the relevant literature. I am only knowledgeable about neural network quantization. So the veracity of some of the claims, like a novel Gaussian specific Trellis, I cannot comment on.\n\nSTRENGTHS:\n- The paper is very well-written, and quite easy to follow. It also feels like it's high-quality, well-researched, and the authors clearly know this field and their Trellis field very well. \n- I have not seen this Trellis-based compression scheme in the deep learning literature before, so to my knowledge it is novel\n- I can believe this compression scheme is better than normal VQ-based methods. Once you've turned your data into Gaussians, anything that compresses Gaussian distributed data better is likely to work.\n- The paper actually implemented the idea on GPU kernels, and show that the overhead is negligible. Essentially saying the improvements are free.\n\nWEAKNESSES:\n- The results are not TOO significant. In terms of perplexity, we're talking 0.1 ppl compared to QUIP# here. And the zero-shot results are always noisy, so tough to say what's faster.\n- Because the compression is likely theoretically slightly more accurate, what remains is the inference-speed of this method. Now the authors, to their credit, do address this with their GPU implementation. But, because the improvements in perplexity are so little, for fair comparison we have to be absolutely sure that the latency for the kernel is the same. Otherwise, we might as well increase the codebook size for QUIP# a little and get a better trade-off, or have one layer in a slightly higher bit-width or whatever. The Speed analysis in Table 7 does not totally satisfy my need for a thorough analysis on this. Since this is so important for the paper, I would like to see an analysis for the kernels themselves, analyze the roofline points for the band-width/compute trade-offs (does your method do more compute than QUIP#?), and have this done for more bit-widths. I believe the authors have to be more thorough on this, else I would not recommend anyone implementing the paper, and the trellis compression is relegated to being a fun curiosity instead of a practical method.\n- We don't all run things on GPUs - there are devices, with hardware accelerators, CPUs etc. What does hypothetical overhead look for this method compared to simple codebook look-ups? Some hardware has actual Sillicon for codebook look-ups, or specific instructions for it like in ARM CPUs. Would your method in these cases also still be faster? Since quantization is not only a 'speed-things-up-on-gpu' topic, I would like to see a discussion on this in the paper.\n\nEditorial Notes\n5 - […] PTQ approaches have converged. I don’t think convergence has really happened, so papers do this, some don’t. \n24-32 What is k? it’s undefined in the intro\n160 psuedorandom -> pseudorandom\n\nQUESTIONS:\nIs section 24-32 entirely correct? Technically you have 2^kd vectors, but nobody ever fills out the whole space with the codebook. So it’s always a subset \\subset R^{2^kd} x d you are selecting for your codebook C. So it’s not really fair to say VQ requires exponential time and space… if you keep your codebook size similar, it does not require exponential time and space. Similarly, the O(2^{kd}d ) complexity is a bit misleading. You can also say the lookup is log(|C|) depending on the size of the codebook you chose. I am not being festicious here, vector quantization methods often take a fixed set of vectors \n\nIs the analysis is 116 sensible? Why would one ever have an unstructured k-bit codebook? There is always a way to add structure. \n\nTable 1 - Is this done keeping the overhead of each method the same? I can always increase the dimensionality of VQ to get lower values right? This comparison should only make sense if the compute/storage overhead is the same.  \n\nHow does this Trellis thing actually work? I'm trying to follow figure 2. I'm assuming the firs thing you need to store is the starting value 00. And then you follow the up-and-down errors each costing you a bit, so the red arrows should give you down, down, up, down down... So how do we end up with 0010110? Wouldn't it be 00 | 00 1 00 or 00 | 11 0 11? \nAlso - what happens in your code if the next value cannot be transitioned to? In your example, if 0.1 follows 0.3, you're out of luck... does it then give you the closest value 0.3?\n\nDo you have a simple explanation for why Trellis-based coding would work better than normal codebooks? Just 'dimensionality' doesn't sit well with me, because that doesn't really explain anything concrete. \n\n\nIf I get more confidence from the authors on the speed of the kernels and the overhead of this method, with a discussion about non-gpu devices, especially given the very small gains they achieve - I am happy to increase my rating.\n\nLIMITATIONS:\nNo pain points here", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes QTIP, a new method for efficient post training quantization of LLMs. QTIP is a vector quantization method inspired from Quip# [1], but with no limitation in the dimension of the codebook. The work’s fundamental contributions are: 1) Propose to adapt trellis quantization to the compression of LLM pre-trained matrices. 2) Design efficient trellis and codebook in dimension (or 'sequence length') $=256$. 3) Integrate the trellis quantization into the Quip# [1] optimization process to fine-tune the pre-trained weights. The numerical experimental results of the proposed method outperform AQLM [2] and Quip# [1] on the C4 and Wikitext2 datasets.\n\n---\n\n[1] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks, 2024.\n\n[2] Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization, 2024.\n\nSTRENGTHS:\n1) The authors proposed an interesting description of the limitations (in terms of codebook dimension) related to standard vector quantization, and a sound introduction to trellis quantization.\n2) This paper presents two innovative lookup-free code for (non-fine-tunable) trellis weight quantization: '1mad' and '3inst'. These methods obtain sota performances with respect to other non-fine-tuned approaches, and are efficiently implemented without the need to store the codebooks.\n3) The author also propose an hybrid (fine-tunable) trellis weight quantization: 'hyb'. Though 'hyb' requires to store the codebook, it allows one to fine-tune the codewords, and thus enable QTIP to compete with fine-tuned sota approaches [1,2].\n\n---\n\n[1] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks, 2024.\n\n[2] Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization, 2024.\n\nWEAKNESSES:\n1) QTIP is mainly based on Quip# [1]: 'we use QTIP as a quantizer in QuIP#’s BlockLDLQ' (line.224). The code is not provided, but the authors explain that the whole quantization and fine-tuning process is based on Quip# [1]. Hence the contribution is more to the field of signal quantization in general, and LLM compression is an application.\n2) QTIP natively reuses the incoherence processing from [1]. But this pre-processing step may erase the native correlation patterns that pre-trained LLM matrices bear [2,3]. No discussion about this process (and the blockLDLQ) is provided.\n3) QTIP (with fine-tuning) update the pre-trained weights 'in a blockwise fashion' (line.247). This step is computationally intensive regarding the size of the matrices (and the optimizer states related to it), and it may impact the efficiency of the quantization process.\n4) Very few details are given regarding the experiments design (which calibration dataset is used for fine-tuning 'hyb' codewords?), what may affect the results reproducibility (in addition the code is not publicly available).\n\n---\n\n[1] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks, 2024.\n\n[2] Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. (2023). Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629.\n\n[3] Guo, H., Greengard, P., Xing, E., and Kim, Y. (2023). Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. In The Twelfth International Conference on Learning Representations.\n\nQUESTIONS:\n1) Table.1 shows the superiority of QTIP over scalar quantization and Quip# [1]. How does it translate to the case of 'real' pre-trained LLM matrices? Can you show some results similar to [2], to identify which matrix presents the lowest quantization error ? This would enable us to decouple the gains provided by QTIP, and understand better which part (the quantization process itself, or the fine-tuning machinery) is key for sota performances.\n2) Is 'hyb' codebook unique? Or do you have a different (fine-tunable) codebook for each LLM block?\n3) Do you think QTIP fine-tuning step is (computationally) competitive with respect to (low-rank) adapters methods such as LoRA [3] and LQ-LoRA [2] ? Or QTIP is only tailored for LLM compression ?\n\n\n---\n\n[1] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks, 2024.\n\n[2] Guo, H., Greengard, P., Xing, E., and Kim, Y. (2023). Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. In The Twelfth International Conference on Learning Representations.\n\n[3] Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. (2021). Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.\n\nLIMITATIONS:\nA comparison of inference speedups is provided in section 4.3.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces trellis coded quantization (TCQ) into large language model (LLM) quantization, achieving ultra-high-dimensional quantization with less inference burden compared to traditional vector quantization (VQ) methods. The main innovations are a hardware-efficient \"bitshift\" trellis structure and fast compute-based Gaussian codes, enabling high-quality quantization and fast inference.\n\nSTRENGTHS:\n1. The motivation to address the drawbacks of VQ is clear and intuitive.\n2. The proposed \"bitshift trellis\" and \"compute-based random Gaussian codes\" are innovative and improve the inference efficiency of TCQ.\n3. The performance is impressive, significantly enhancing 2-bit performance even without fine-tuning and optimizing quantization bits to 3-bit.\n4. The method significantly improves inference speed.\n\nWEAKNESSES:\nThis is a solid paper introducing a novel and robust quantization format. Inference efficiency is crucial for new quantization formats, but this paper only presents the inference speed on the RTX 4090 with 2-bit quantization. Providing inference speeds for more bits (e.g., 3-bit, 4-bit) and more devices (e.g., RTX 3090, A100) could enhance the paper further.\n\nIt is possible that existing quantization computation kernels achieve speedup on the RTX 4090 but not on the A100. Such phenomena are normal, and adapting a new kernel to different devices is challenging. However, I encourage the authors to report inference speeds on various devices. Even if results are not favorable, an in-depth analysis with potential solutions would provide a more comprehensive understanding without affecting the paper's contribution.\n\nQUESTIONS:\nPlease see the weaknesses for details.\n\nLIMITATIONS:\nThe paper lacks inference speed testing on more bit levels and additional devices.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a new method of weight-only post-training quantization (PTQ) that uses trellis-coded quantization (TCQ) to achieve ultra-high-dimensional quantization. Although TCQ was introduced by Mao and Gray, QTIP transfers it into the LLM space and introduces new algorithms to make the method hardware efficient, thereby enabling fast inference.\n\nSTRENGTHS:\n1. The idea of using TCQ for LLM is pretty interesting and, to the best of my knowledge, new.\n2. The paper is well-written, with explanatory figures and detailed explanations of the algorithms.\n3. Paper includes a comprehensive amount of experiments.\n4. The authors  provide extensive details to reproduce paper and promised to make the code public.\n\nWEAKNESSES:\n1. The ablation study is almost non-existent. For example, there is no analysis of the inference speed dependency on each introduced trick.Another thing you can do is experiment - perplexity (ppl) without the tricks, using L, k, and V values that still fit in the L1 cache, alongside the introduced algorithms.\n2. The L, K, and V parameters were not explored. It would be beneficial to see different V values compared at the same bit width.\n3. The paper contains a lot of notation and symbols that are either described once briefly or assumed to be known (e.g., see kd in line 27 line 128). This made reading the paper hard for me, as I had to constantly refer back to previous pages to understand what each symbol represents. For example, see line 229. This issue persists throughout the paper. Maybe it would be good to soft reintroduce symbols for time to time.\n4. To be frank, the performance gains for me are not that impressive, except perhaps for LLama-3 at 2 bits.\n\nQUESTIONS:\n1. In the paper Line 236, you mentioned that QTIP can be fine-tuned. Did you try it? Did it provide any performance boost?\n2. Can you please add the formula to calculate the average bit? While it can be deduced from the details given, having the formula would make readability clearer.\n3. There are misspellings in line 46 and line 128: tk/kt , this should be corrected.\n4. Is the formula in line 125 for the trellis structure correct? If we look at line 101, one of these cases seems to be a misspelling.\n5. The numbers in Table 5 are not properly highlighted, especially where other methods are winning or matching.\n6. The inference speed reported for AQLM is very odd. The AQLM paper claims around a 1.3x speed-up over FP16, but in your case, it is much slower. I suppose this may happen due to other library overheads like transformers, Pytorch and Python. Did you try the code with compiled CUDA graphs? Please take a look at [this example](https://colab.research.google.com/github/Vahe1994/AQLM/blob/main/notebooks/aqlm_cuda_graph.ipynb) from there repository.\n\nOther:\n1. Line 70 begins with \"The effectiveness of these methods.\" It is a new section, and because of this, it is not clear which methods you are referring to.\n2. Cite the linear congruential generator (LCG) paper.\n3. In Line 240, should be mentioned that  AQLM doing block fine-tuning. Quip# doing full model fine-tuning. Full model fine-tuning also can be applied at AQLM, and the numbers will much Quip#. See Table 4 https://arxiv.org/pdf/2401.06118.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}