QTIP: Quantization with Trellises and Incoherence Processing

Authors: albert  tseng
qingyao  sun
david  hou
christopher  de sa
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 22:40:47

Abstract: Post-training quantization (PTQ) reduces the memory footprint of LLMs by quantizing weights to low-precision datatypes.
Since LLM inference is usually memory-bound, PTQ methods can improve inference throughput.
Recent state-of-the-art PTQ approaches use vector quantization (VQ) to quantize multiple weights at once, which improves information utilization through better shaping.
However, VQ requires a codebook with size exponential in the dimension.
This limits current VQ-based PTQ works to low VQ dimensions ($\le 8$) that in turn limit quantization quality.
Here, we introduce QTIP, which instead uses trellis coded quantization (TCQ) to achieve ultra-high-dimensional quantization. 
TCQ uses a stateful decoder that separates the codebook size from the bitrate and effective dimension. 
QTIP introduces a spectrum of lookup-only to computed lookup-free trellis codes designed for a hardware-efficient "bitshift" trellis structure; these codes achieve state-of-the-art results in both quantization quality and inference speed.

Main paper: 
 introduction: 
 Large language models (LLMs) have accelerated advancements in fields ranging from natural language processing [34] to scientific modeling [28] . However, the largest LLMs have hundreds of billions of parameters that can take over a terabyte of memory to load in half-precision; this size poses significant challenges for the practical deployment of LLMs [33, 17, 2] . For example, smallbatch autoregressive decoding, a common form of inference for LLMs, is memory bound [35] . Even on a modern datacenter GPU with ≈ 3TB/s memory bandwidth, a large LLM (> 200GB) can only be directly run at < 20 tokens per second and may require multiple devices [35] . One way to accelerate inference is by compressing LLMs. This directly reduces the memory footprint of the model and increases the theoretical maximum inference throughput on any given machine.One form of compression, weight-only post-training quantization (PTQ), quantizes trained model weights to lower precision datatypes [9, 35, 5] . The latest state-of-the-art weight-only PTQ methods, QuIP# and AQLM, use vector quantization (VQ) to achieve high-quality 2-bit models [35, 11] . In VQ, a vector x ∈ R d is quantized to one of 2 kd vectors in R d that form a codebook C ∈ R 2 kd ×d . A higher vector dimension d allows for better codebook shaping and packing density, improving information utilization [19] . However, unstructured VQ requires exponential time and space in both the bitrate and dimension, limiting its practicality. During quantization, VQ costs O(2 kd d) time to perform nearestneighbor rounding to C, and during inference, C must fit in hardware cache for fast lookups. This exponential scaling limits how high d can be and thus the advantages of VQ over scalar quantization. Figure 1 : QTIP performs ultra-high dimensional (> 100) quantization by using Trellis Coded Quantization, which has linear cost in dimension. This enables QTIP to outperform Vector Quantizationbased approaches (QuIP#, AQLM) that are limited to low dimensions. With QTIP, 2 bit models scale better than theoretically optimal 4 bit models.To address this limitation, we propose QTIP, which uses trellis-coded quantization (TCQ) to enable tractable ultra-high-dimensional (> 100) quantization and improve quantization quality over prior VQ-based approaches. In the simplest scalar form of TCQ, a length-T sequence S is statefully quantized using a trellis -a directed graph G with 2 L nodes, each with 2 k incoming and outgoing edges and a scalar value [24] . The reconstructed sequence Ŝ corresponds to the node values of a length-T walk on G, and quantization finds the walk that minimizes some distortion metric on S and Ŝ. Since neighboring entries in Ŝ are connected by one of 2 k edges, we only need to store which edge an entry came from, which takes k bits. For additive distortion metrics such as squared error, the optimal Ŝ can be found with the Viterbi algorithm, which runs in O(2 L T ) time [13, 24] . This means that the cost of quantization is independent of the bitrate k and linear in the sequence dimension T , enabling tractable high dimensional quantization. However, TCQ is not free. During inference, vanilla TCQ requires storing both G and the size 2 L × V node value codebook, which can be too large to fit in cache. TCQ-quantized sequences also cannot generally be decoded in parallel, as tth elment of Ŝ could depend on up to the first tk encoded bits. In QTIP, we solve these issues by introducing a series of fast compute-based Gaussian codes designed for the hardware-efficient "bitshift trellis." Specifically, the bitshift trellis supports parallel decoding, does not require storing G, and our compute-based codes eliminate needing to store a large node value codebook. This enables high-quality quantization of Gaussian sources while supporting fast inference, and we adopt incoherence processing with the random Hadamard transform to ensure that LLM weights are approximately i.i.d Gaussian distributed. Altogether, QTIP 1. Achieves a state-of-the-art combination of weight-only LLM PTQ quality and fast inference through hardware-efficient trellis and codebook design.2. Introduces multiple novel hardware-efficient (≤ 4 instructions per weight) compute-based random Gaussian codes for TCQ on i.i.d. Gaussian sources.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The main claims made in the abstract and introduction reflect the paper's contributions and scope.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
background and related works: 
 We focus on weight-only post-training quantization (PTQ) of LLMs in this work; other modelcompression approaches include quantization-aware training (QAT) and pruning. These methods are not strictly orthogonal to each other, as one could both prune and quantize a model. Since QTIP is a weight-only PTQ method, the rest of this section focuses on this area. Most current state-of-the-art PTQ methods round to minimize the per-layer proxy loss from Nagel et al. [27] .EQUATIONHere, Ŵ ∈ R m×n is the quantized weight matrix, x ∈ R n is an input activation, and H = E x xx T ∈ R n×n is interpreted as a proxy Hessian matrix. This objective is defined per-layer, making it tractable for very large models. However, minimizing it is difficult due to the nondifferentiable nature of quantization. Instead many works have proposed algorithms such as Hessianbased adaptive rounding, alternating optimization, and even coordinate descent to approximately minimize the proxy error [11, 5, 35, 14] .The effectiveness of these methods depends on properties of W . For example, many works have observed that weight and activation outliers cause poor quantization quality [10, 20, 29] . In QuIP, Chee et al. [5] proposed that incoherence was important for quantifying this effect.Definition 2.1 (Chee et al. [5] ). A HessianH ∈ R n×n is µ-incoherent if its eigendecomposition H = QΛQ T has max i,j |Q ij | = max i,j |e T i Qe j | ≤ µ/ √ n. A weight matrix W ∈ R m×n is µ- incoherent if max i,j |W ij | = max i,j |e T i W e j | ≤ µ∥W ∥ F / √ mn.Essentially, incoherence means the weights and important rounding directions (Hessian eigenvectors) are not too large in any direction, aiding quantization. To make W, H incoherent (small µ), one can perform incoherence processing (IP) by conjugating W, H with random orthogonal matrices U, V :W ← U W V T , H ← V HV T . QuIP# introduced IP with the random Hadamard transformation (RHT), which performs W ← V m S m W S n V T n , H ← V n S n HS n V T n where V k is a k × kHadamard matrix and S k is a length k random sign vector. The RHT achieves, with probability ≥ 1 -δ, µ W = 2 log(4mn/δ), meaning that W 's entries are approximately independently Gaussian distributed, which can aid quantization [35, 3] . We choose to build on incoherence processing here because the independent Gaussian-like weights it produces are suitable inputs for trellis coding [23] .k-bit VQ quantizes a d dimensional vector S to one of 2 kd d-dimensional vectors that form a codebook C ∈ R 2 kd ×d [1] . Since C is an unstructured collection of arbitrary vectors, VQ enables better shaping and packing density than scalar product quantization (SPQ), where each entry in S is quantized independently [19] . However, this also comes at the cost of exponential time quantization and exponential space inference: finding the nearest neighbor in C requires O(2 kd d) time, and storing C requires O(2 kd d) space. The current crop of state-of-the-art LLM PTQ methods, QuIP# and AQLM, both use VQ to achieve high-quality 2-bit models. Since the shaping advantage of VQ comes from high dimensionality, both QuIP# and AQLM attempt to maximize dimensionality. AQLM's uses a large 8D codebook (1MiB) that does not fit in L1 cache. QuIP# uses an 8D compressible codebook based on the E 8 lattice, which is highly symmetric. This codebook is compressible by 256× and barely fits in L1 cache. In either case, the VQ dimension is effectively hardware-limited to ≤ 8, motivating methods that enable even higher-dimensional quantization.TCQ was first proposed by Marcellin and Fischer [24] to apply the benefits of trellis coded modulation, a conceptually dual problem, to quantization. Define a (L, k, V ) trellis G as a directed graph with 2 L nodes, each of which has 2 kV incoming and outgoing edges and a value ∈ R V ; these values form a codebook C ∈ R 2 L ×V . To quantize a length-T sequence S ∈ R T , each contiguous length-V subsequence of S is assigned to a node ∈ G, with the restriction that the assigned nodes form a walk. The reconstruction Ŝ of S is then given by concatenating node values in the walk. When V = 1, this setup describes Marcellin and Fischer [24] 's original scalar TCQ. When V > 1, this describes TCVQ, which applies TCQ to vectors [12, 37] .Finding the optimal Ŝ under an additive distortion metric can be done with the Viterbi algorithm in O(2 L T ) time. This is linear in sequence length, enabling ultra-high dimensional quantization. For exposition, we briefly describe the Viterbi algorithm here. Concretely, if we want to quantize a Tlength scalar sequence reinterpreted as a sequence of vectors s 1 , s 2 , . . . , s T /V ∈ R V using a trellis code with graph G and codebook C, this corresponds to solving the optimization problem minimize T /V i=1 ∥C xi -s i ∥ 2 over x 1 , x 2 , . . . , x T /V the vertex sequence of a walk on graph G.This optimization problem can be solved exactly with dynamic programming via the value functionV t (x) = min t i=1 ∥C xi -s i ∥ 2 x 1 , x 2 , .. . , x t the vertex sequence of a walk on G and x t = x using the update rule V t (y) = min(x,y)∈G V t-1 (x) + ∥C y -s t ∥ 2 .This Viterbi approach clearly takes time linear in T and in the number of edges of G; with a few simple optimizations this can be brought to O(2 L T ). In comparison, brute-force-searching all possible 2 kT codes-which is what we would need to do for an unstructured k-bit T -dimensional codebookwould take time proportional to 2 LT /V . The ability to tractably find the closest representable vector in R T , even for large T , is in some sense the "main benefit" of trellis coding. For i.i.d sources, as L increases, TCQ efficiently approaches the infinite-length distortion-rate D R , which lower bounds the attainable distortion of a k-bit quantizer [19] . As shown in Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The main text describes limitations of the work that the authors were able perceive at submission time.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
qtip: 
 Quantizing with TCQ requires storing both the codebook (2 L × V ) and trellis structure (2 L × 2 kV ) during inference. These components are too large for fast inference when L ⪆ 12, which is necessary for high quality. Furthermore, for a generic trellis, recovering the state (and so the decoded value) at step tth requires a graph walk using the first kt bits: this prevents parallel decoding. QTIP solves these problems with a novel combination of incoherence processing, a hardware-efficient "bitshift trellis," and fast compute-based random Gaussian codes. Incoherence processing makes W approximatelly i.i.d Gaussian, which reduces quantization to Gaussian source coding. The bitshift trellis removes needing to store the trellis structure during decoding and also enables parallel decoding. Finally, the fast compute-based random Gaussian codes remove the need to store the full codebook, completing the equation for fast inference. On the quality side, the fast random Gaussian codes enable the simple bitshift trellis to match complicated trellises and achieve state-of-the-art quantization quality.The main focus of QTIP is on what to quantize with (i.e. TCQ) and not how to quantize (e.g. adaptive rounding or descent methods). The general construction of QTIP can be used as a drop-in replacement for VQ in any rounding framework. In the following sections, we first describe the "bitshift" trellis (Section 3.1). Then, we describe a series of fast compute-based codes for i.i.d Gaussian sources, aligning with different types of hardware (Sections 3.1.1 and 3.1.2). Finally, we give an approximation for the tail-biting trellis problem, which lets us more efficiently load weights in hardware (Section 3.2).The bitshift trellis was introduced by Mao and Gray [23] as part of the "random permutation trellis coder" (RPTC). In the bitshift trellis, node i has an edge to node j if ∃c ∈ Z, 0 ≤ c < 2 kV , s.t. j = (i2 kV mod 2 L ) + c. Essentially, the top L -kV bits of j equal the bottom L -kV bits of i. This means that the first group of V weights depends only on the bits at positions {1, 2, . . . , L}, the second only on bit positions {kV + 1, kV + 2, . . . , kV + L}, and in general the tth on bit positions {(t -1)kV + 1, . . . , (t -1)kV + L}. During inference, obtaining the next compressed group of V weights in a sequence only requires bitshifting by kV bits, which is supported on virtually all hardware. Furthermore, since each group of V weights only depends on a contiguous window of L bits in Ŝ, decoding can be parallelized. Figure 2 shows a simple (L = 2, k = 1, V = 1) bitshift trellis. Note that edges only exist between nodes that overlap by 1 bit, and storing the quantized length 6 Ŝ indeed only requires 6 bits (plus the initial state).Quantizing an i.i.d. source with the bitshift trellis is nontrivial because neighboring groups of weights sharing many bits can potentially lead to strong correlations (Figure 3 Ŝ is also tail-biting, so the last L -kV = 1 bits can be dropped to give Ŝ = 001011. the codebook to decorrelate neighboring weight groups (Figure 3 RR) [23] . However, this requires storing the codebook or storing and applying the permutation, both of which are prohibitively expensive during decoding. Instead, QTIP introduces a series of compute-based codes to produce a psuedorandom code, which has the same decorrelating effect and admits fast inference. To match approximately i.i.d. Gaussian RHT-transformed matrices, these codes produce psuedorandom approximate Gaussians in as few as 2 instructions per weight (see Table 1 and Figure 3 ). To the best of our knowledge, these code constructions alone are novel and we are the first to propose a lookupfree Gaussian trellis code.Here, we present two pure-computed lookup-free codes that produce a pseudorandom approximately Gaussian number from a L bit word, enabling fast decoding on cache-limited hardware. These codes avoid strong correlations and can be implemented in ≤ 4 hardware instructions per weight on NVIDIA GPUs. We present two codes here to illustrate that multiple such codes are possible: in practice a lookup-free code can be designed to use the instructions available on whatever hardware we want to run on.Algorithm 1 (1MAD) first runs a linear congruential generator (LCG) to produce a pseudorandom 32-bit word [32] . This requires 2 instructions (MAD and &). It then sums the 32-bit word as four 8bit unsigned integers; this sum is approximately Gaussian distributed. This requires 1 instruction (vabsdiff4). Finally, this sum must be scaled and shifted (another MAD). Although there are only 2 10 representable values even when L > 10, this does not empirically affect quantization quality. 1MAD requires choosing a and b to avoid strong correlations; we set a = 34038481 and b = 76625530(Figure 3 LC).Algorithm 2 (3INST) also first runs an LCG to produce a random 32-bit word X. Then, it XORs the bottom 16 bits of X with the mantissa bits, bottom two exponent bits, and sign bit of a magic FP16 number m to produce an FP16 number m 1 . It then repeats this with the top 16 bits of X to produce m 2 and returns m 1 + m 2 . This entire process can be implemented in 3 ALU instructions1 with a MAD for the LCG, a lop3 to mask and XOR with a packed duplicated m, and then summing m 1 and m 2 . Here, we describe a hybrid computed-lookup code that computes a pseudorandom (or hashed) index into a 2D vector codebook (V = 2). This code is tailored for modern GPUs, which have enough cache for a small in-memory LUT-one benefit of using such a LUT over a purely computed codebook is that a LUT can be fine-tuned after quantization. Algorithm 3 first performs the hash X ← X 2 + X to mix the lower order and upper order bits of X [18] . Then, it takes bits (14 -Q + 1) -14 (0 indexed) as an index into a 2 Q × 2 LUT to get two 16-bit floats. (The reason why we chose a 2D codebook here is that shared memory on NVIDIA GPUs is accessed in 32-bit-word elements, and each such word can contain two 16-bit floats.) Finally, it XORs bit 15 of X to flip the sign of the second entry of the codebook vector. Algorithm 3 can be implemented with MAD, bitshift, mask, and lop3, giving an amortized 2 instructions per weight. This effectively assigns a L bit word to one of 2 Q+1 2D vectors, each of which can be fine-tuned to improve quality. Algorithm 3 can also be implemented to XOR bit 31 alongside bit 15 (this is free in the lop3) to give an effectively 2 Q+2 -sized codebook, which can improve quantization quality. We only realized this after running all the experiments, so the numbers in this paper use the "one sign flip" version of Algorithm 3. In QTIP, we initialize the LUT using K-means on an empirical 2D i.i.d. Gaussian distribution.Directly quantizing a length-T sequence to a (L, k, V ) trellis results in a total of kT + L -kV bits since the starting state takes an additional L -kV bits to store. If we run inference on a machine with w-bit words where w|kT , we must read an extra ⌈ L-kV w ⌉w -(L -kV ) wasted bits per sequence. For common w (e.g. 32), setting L = kV + w makes the Viterbi algorithm intractable. One way to solve this is by enforcing that the start and end state share L -kV bits, i.e. the trellis is tail-biting [4] . Exactly solving the tail-biting trellis problem via dynamic programming takes time quadratic in the state space (2 L ), making this problem intractable for reasonable L ≥ 12 [30] . However, since RHT-processed weights are approximately i.i.d., simple algorithms can be effective for approximately solving the tail-biting problem. We propose Algorithm 4, which first rotates the sequence by T /2, quantizes it, and then extracts the overlap between the rotated start and end states. It then requantizes the original sequence with this overlap as the tail-biting overlap. This only requires two Viterbi calls Algorithm 2 Computed Gaussian Code "3INST" input L-bit 0 left-padded integer x, uint32 a, b, float16 m.x ← (ax + b) mod 2 32 {run LCG to get uniform random x} {modify sign, mantissa, and bottom 2 exponent bits of m and sum, this is approximately Gaussian}EQUATION)) {apply sign flip} output Pseudorandom approximate Gaussian vector v. in total. Table 2 shows that in practice, Algorithm 4 can find close-to-optimal tail-biting sequences while being significantly cheaper to run than other tail-biting approximation algorithms [30] .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: This paper concerns foundational research on LLM quantization. There is not a direct path to any negative applications. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiments: 
 Here, we present experiments quantizing the Llama family of models with QTIP [33, 34, 26] . These models offer strong performance across a wide range of sizes, allowing us to compare how different quantization methods perform and scale. We primarily compare QTIP against QuIP# and AQLM. For Llama 1, we include GPTVQ-2D instead of AQLM since AQLM does not publish Llama 1 numbers [36] . GPTVQ-2D performs 2D VQ inside GPTQ and offers strong performance. These methods outperform scalar quantization methods including GPTQ, AWQ, and OmniQuant; comparisons to those methods can be found in QuIP# and AQLM [20, 14, 29, 35, 11] . We mainly focus on the hybrid code (Section 4.2) since it is tailored for modern GPUs, and present a full suite of results for it. For the computed codes (Section 4.1), we present results for Llama 2.Since the proxy error is not an additive distortion metric, we cannot minimize it by quantizing W as one sequence. Instead, for all experiments, we use QTIP as a quantizer in QuIP#'s BlockLDLQ, which allows us to simultaneously achieve high dimensionality and low proxy error [35] . Specifically, we quantize a block of T x × T y weights as a sequence, where T x and T y span the output and input dimensions of W , respectively. Since BlockLDLQ only specifies feedback along the input dimension, this is equivalent to BlockLDLQ with g = T y but a vector dimension of T x T y ≫ T y . This has the benefit of limiting the effect of g in BlockLDLQ's error bound gmµ 2 σ 2 tr(H 1/2 ) 2 /n while achieving a high dimension for TCQ. Algorithm 5 in the Appendix describes this in more detail.Here, we use 1MAD and 3INST with L = 16, V = 1, T x = T y = 16. Setting T x = T y = 16 enables using a 16 × 16 MMA tile per trellis sequence to perform matrix multiplication during inference. 16 × 16 MMA tiles form the basis of many types of "AI hardware," making fast decoding relatively simple [6] . We do not perform fine-tuning since the codes themselves are not tunable, but these codes are fully compatible with QuIP#-style fine-tuning (recall that QuIP#'s codebook is also not tunable). Table 3 shows that both 1MAD and 3INST significantly outperform QuIP# without fine-tuning (AQLM does not have numbers without fine-tuning). Even at 4 bits, where all methods are close to lossless, QTIP results in significant improvements. Notably, the computed-code QTIP variants without fine-tuning outperforms both QuIP# and AQLM with fine-tuning on almost all models and sizes, showing that fine-tuning is not a silver bullet. Here, we use the hybrid lookup-computed code withL = 16, V = 2, T x = T y = 16, Q = 9. Setting Q = 9gives a 2KiB codebook, which fits in L1 cache even after duplication for bank conflicts (32×) on modern GPUs. This codebook is differentiable, so we can finetune it: to evaluate this, we fine-tune using QuIP#'s methodology, tuning both the codebook entries and the as-yet-unquantized weights in a blockwise fashion.Table 5 shows the perplexity of quantized Llama 1 and 2 models. In all cases, QTIP outperforms the other vector quantization-based methods. Even at 3 and 4 bits, where QuIP# and AQLM are close to lossless, QTIP roughly halves the perplexity gap. These results also show the importance of dimensionality. Note that the 3-and 4-bit Llama 2 70B numbers here match those in 3. Since Table 3 uses a purecomputed code without fine-tuning, fine-tuning has no effect in these regimes and the improvement over QuIP# is purely from dimensionality.Table 6 shows zeroshot results computed with LM Eval, which are slightly random; QTIP generally matches or exceeds QuIP# and AQLM on these tasks [15] . Table 7 contains results on Llama 3. Like other works, we have observed that Llama 3 (especially 70B base) is harder to quantize than Llama 2 [16] . Since the contribution and focus of this work is what to round with (TCQ) and not how to round (BlockLDLQ), we only compare against the proximal baseline QuIP#, which uses BlockLDLQ with VQ. QTIP significantly improves upon QuIP# at all model sizes and bitrates, once again showing the dimensionality advantage of TCQ over VQ. Table 8 shows results for Llama 3.1 instruct-tuned models, including Llama 3.1 405B. At all sizes, QTIP achieves strong results. Notably, QTIP is able to match or exceed PV-Tuning, a recent quantization method that focuses on better finetuning algorithms [22] . However, PV-Tuning is based off of AQLM and inherits its slow inference speed, making it significantly slower than QTIP. Finally, Table 9 shows results for quantizing Llama 3.2 instruct-tuned models to 4 bits. Since the embedding layers are very large relative to the decoder layers for small Llama 3 models (≈ 500 -750MB), quantizing the decoder layers to fewer than 4 bits does not make a significant difference on the final model size. Here, QTIP is still able to achieve a meaningful end-to-end compression rate (2.5-3X) without degrading the final model. Table 4 shows the batch size 1 inference speed of QTIP, QuIP#, and AQLM on Llama 2 7B and 70B with matrix fusion. Here, the design choices of QTIP and QuIP# become apparent. Whereas AQLM uses a codebook that is too large to fit in cache and thus prevents fast inference, both QTIP and QuIP# achieve significant speedups over FP16. Furthermore, while it is impressive that both QuIP# and QTIP are > 2× faster than AQLM, it is even more impressive that QTIP is able to match QuIP#'s throughput with an effective dimension size of 256, or 32× larger than QuIP#'s. This means that the improved quantization quality of QTIP comes with no additional inference-time cost.Although our empirical throughput numbers were timed on NVIDIA GPUs, QTIP can be fast on a broad class of accelerators due to its flexibility. QTIP only requires generating a pseudorandom Gaussian efficiently, and can work on devices with no cache as well as devices with lookup hardware. For example, if we were using a ARMv8 CPU, we could use the vqtbl4q_u8 NEON intrinsic to look up 16 indices in a 64-entry codebook. This would let us use a 6 bit 1D codebook with the HYB code (Q=6, V=1). Quantizing Llama 2 7B to 2 bits with this setup and w/out fine-tuning gives 6.89 Wikitext2 perplexity -essentially the same state-of-the-art quality as 3INST. 
conclusion: 
 We present QTIP, a weight-only post-training quantization algorithm that achieves state-of-theart results through the use of trellis-coded quantization (TCQ). TCQ enables tractable ultra-high dimensional quantization, significantly reducing quantization distortion over vector quantization (VQ). However, naive TCQ does not admit fast inference due to sequential bottlenecks during decoding and needing to store a large codebook. QTIP solves this problem through a novel combination of incoherence processing, the hardware-efficient bitshift trellis, and fast computed codes. Specifically, QTIP introduces a series of compute-based pseudorandom Gaussian codes that, when used in  
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All such instances were properly cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Table 10 shows an ablation on L for quantizing Llama 2 7B with K = 2, V = 1, the bitshift trellis, a pure-lookup codebook, and no fine-tuning. L = 8 is the largest L achievable if we had to store the trellis and codebook in the same amount of cache as the HYB code (2KiB). L = 10 is the largest L achievable if we only had to store the codebook. As expected, increasing L improves quality. Table 10 also shows very little difference between an equal-sized LUT codebook and QTIP's codes, meaning that QTIP isn't sacrificing quality for speed. However, an equal-sized LUT would need > 10× more cache than the latest GPUs have, making the bitshift trellis and compute-based codes necessary to achieve both quality and speed. Table 11 shows an ablation on V with L = 12 and 16, K = 2, and the same settings as Table 10 . Increasing V generally decreases quality, but this can be recovered with a larger L. It is hard to measure V 's impact on decoding speed since this is highly implementation and hardware dependent, so V is more of a user-chosen hyperparameter. Here, we use a pure-lookup code ∼ N (0, 1) with L = 14, V = 1, T x = 32, T y = 8, and QuIP#'s fine-tuning scheme. These parameters show what performance QTIP could achieve if we did not care about fast inference today. Specifically, a pure-lookup codebook is tunable, and setting T y = 8 reduces the BlockLDLQ group size while maintaining high dimensionality (256). This codebook uses 32KB; this only fits in GPU L1 cache with bank conflicts. Setting T x = 32, T y = 8 corresponds to using a larger MMA tile size than current GPUs allow for. The largest tile size is usually 16 in the T x dimension, meaning that a 32 × 8 trellis needs two tiles. Thankfully, hardware required to serve Here, we detail how we use TCQ within BlockLDLQ to produce our experimental setup. Essentially, QTIP is used as a high dimensional T x T y quantizer within BlockLDLQ and is a drop-in replacement for vector quantization in BlockLDLQ. The regular blockLDLQ step Q(W + (W -Ŵ )A) is exactly the same, and the only difference is in how Q rounds. Instead of rounding each row of x = W +(W -Ŵ )A independently, it groups T x rows into a block to round as m/T x high-dimensional sequences.Our code is available at https://github.com/Cornell-RelaxML/qtip.Hessian matrices were generated with 6144 sequences of length 2048 for Llama 1, 6144 sequences of length 2048 for Llama 2, 4096 sequences of 8192 for Llama 3, and 4096 sequences of 8192 for Llama 3.1 except for 405B, which only used 2048 sequences due to time constraints. All sequences were sampled from the RedPajama dataset [7] .Algorithm 5 QTIP with BlockLDLQ input W ∈ R m×n , H ∈ R n×n , T x , T y , L, k, V, code C. Ŵ ← 0 m,n LDL T ← T y -block LDL decomposition of H A ← L -I for j ∈ {n/T y -1, n/T y -2, ..., 0} dox ← W :,jTy:(j+1)Ty + (W :,jTy: -Ŵ:,jTy: )A jTy:,jTy:(j+1)Ty x ← x.reshape(m/T x , T x T y ) x ← Viterbi(x, (L, k, V ) bitshift trellis, C) (row-wise) Ŵ:,jTy:(j+1)Ty ← x.reshape(m, T y ) end for output Quantized Ŵ .We use Hadamard matrices from Neil Sloane's website [31] .We use the OPTQ "Wikitext2" and "C4" test splits to calculate perplexity [14] .We use the same fine-tuning hyperparemters as QuIP#, except that we do not perform matrix fusion [35] . This results in having to quantize 7 matrices (q, k, v, o, up, gate, down) instead of 4 (qkv, o, upgate, down). We quantize and fine tune in the following order: (v, q, k, o, up gate, down).The following are some sample generations from Llama 2 70B (not instruction tuned) quantized to 2 bits with QTIP, truncated to 256 tokens.Prompt: It is a truth universally acknowledged QTIP: It is a truth universally acknowledged that a single man in possession of a good fortune must be in want of a wife. However little known the feelings or views of such a man may be on his first entering a neighbourhood, this truth is so well fixed in the minds of the surrounding residents, that he is considered the rightful property of some one or other of their daughters. "My dear Mr. Bennet," said his lady to him one day, "have you heard that Netherfield Park is let at last?" "But it is," returned she; "for Mrs. Long has just been here, and she told me that Netherfield is taken by a young man of large fortune from the north of England; that he came down on Monday in a chaise and four to see the place, and was so much delighted with it that he agreed with Mr. Morris immediately; that he is to take possession before Michaelmas, and that he is now in town in search of a wife." "I see no occasion for that. You and the girls may go, or you may send them by themselves, which do you prefer? Only let me know what you are meaning to do, for I shall certainly not be at home. You Prompt: Call me Ishamel QTIP: Call me Ishmael. Some years ago-never mind how long precisely-having little or no money in my purse, and nothing particular to interest me on shore, I thought I would sail about a little and see the watery part of the world. It is a way I have of driving off the spleen, and regulating the circulation. Whenever I find myself growing grim about the mouth; whenever it is a damp, drizzly November in my soul; whenever I find myself involuntarily pausing before coffin warehouses, and bringing up the rear of every funeral I meet; and especially whenever my hypochondriacal palate is out of sorts, and I am desperately dull, and in the dumps, and have such an overpowering sense of fatality that I find myself involuntarily gravitating toward Cemeteries-and especially toward those open ones which are occasionally cheered by the gleam of a bonfire, or a cigar, or a pipe, or a lantern, or a candle, or a torch, or the moon, I then take to the ship. There is nothing surprising in this,The following is a sample generation from Llama 3.1 405B Instruct quantized to 2 bits with QTIP.Prompt: Give a proof for why the square root of 2 is irrational: QTIP: The proof that the square root of 2 is irrational is as follows: Assume, for the sake of contradiction, that √ 2 is rational. This means it can be expressed as the ratio of two integers, a and b, where b is non-zero:√ 2 = a/bWe can further assume that a and b are not both even (i.e., they do not share a common factor of 2), because if they were, we could divide both by 2 and obtain a smaller representation of √ 2. Squaring both sides of the equation, we get:2 = a 2 /b 2Multiplying both sides by b 2 , we get:2b 2 = a 2This means that a 2 is even, because it is equal to 2 times an integer (b 2 ). Therefore, a must also be even, because the square of an odd number is always odd. Since a is even, we can write a = 2c, where c is an integer. Substituting this into the equation above, we get:2b 2 = (2c) 2Expanding the right-hand side, we get:2b 2 = 4c 2Dividing both sides by 2, we get:b 2 = 2c 2This means that b 2 is even, and therefore b is also even. However, this contradicts our assumption that a and b are not both even. Therefore, our original assumption that √ 2 is rational must be false, and √ 2 is actually irrational. This proof is often attributed to the ancient Greek mathematician Euclid, and it is still considered one of the most elegant and simple proofs of the irrationality of √ 2.NeurIPS Paper ChecklistAs there is currently no instruction on NVIDIA GPUs that sums the top and bottom half of a 32-bit word as two FP16s, this requires an extra data movement instruction to "split" the 32-bit word into two 16-bit registers. 


Decision: Application of TCQ for quantizing trained weights results in strong performance and fast inference, as the reviewers mostly agreed. TCQ is a well-established practice in the communicaiton field and this paper is mainly about borrowing it to model qualitization in ML; neverthless, bitshift trellis and compute-based random Gaussian codes are important new features and the value is significant. Ablation requests were complied to the satsifaction of the reviewer. Studies on larger bit width and variety of devices were also added, which made the work more convincing.

Reviews: review: paper_summary: paper_summary: This work presents a new method of weight-only post-training quantization (PTQ) that uses trellis-coded quantization (TCQ) to achieve ultra-high-dimensional quantization. Although TCQ was introduced by Mao and Gray, QTIP transfers it into the LLM space and introduces new algorithms to make the method hardware efficient, thereby enabling fast inference.
strength_weakness: strength_weakness: 1. The ablation study is almost non-existent. For example, there is no analysis of the inference speed dependency on each introduced trick.Another thing you can do is experiment - perplexity (ppl) without the tricks, using L, k, and V values that still fit in the L1 cache, alongside the introduced algorithms.
2. The L, K, and V parameters were not explored. It would be beneficial to see different V values compared at the same bit width.
3. The paper contains a lot of notation and symbols that are either described once briefly or assumed to be known (e.g., see kd in line 27 line 128). This made reading the paper hard for me, as I had to constantly refer back to previous pages to understand what each symbol represents. For example, see line 229. This issue persists throughout the paper. Maybe it would be good to soft reintroduce symbols for time to time.
4. To be frank, the performance gains for me are not that impressive, except perhaps for LLama-3 at 2 bits.
questions: questions: 1. In the paper Line 236, you mentioned that QTIP can be fine-tuned. Did you try it? Did it provide any performance boost?
2. Can you please add the formula to calculate the average bit? While it can be deduced from the details given, having the formula would make readability clearer.
3. There are misspellings in line 46 and line 128: tk/kt , this should be corrected.
4. Is the formula in line 125 for the trellis structure correct? If we look at line 101, one of these cases seems to be a misspelling.
5. The numbers in Table 5 are not properly highlighted, especially where other methods are winning or matching.
6. The inference speed reported for AQLM is very odd. The AQLM paper claims around a 1.3x speed-up over FP16, but in your case, it is much slower. I suppose this may happen due to other library overheads like transformers, Pytorch and Python. Did you try the code with compiled CUDA graphs? Please take a look at [this example](https://colab.research.google.com/github/Vahe1994/AQLM/blob/main/notebooks/aqlm_cuda_graph.ipynb) from there repository.

Other:
1. Line 70 begins with "The effectiveness of these methods." It is a new section, and because of this, it is not clear which methods you are referring to.
2. Cite the linear congruential generator (LCG) paper.
3. In Line 240, should be mentioned that  AQLM doing block fine-tuning. Quip# doing full model fine-tuning. Full model fine-tuning also can be applied at AQLM, and the numbers will much Quip#. See Table 4 https://arxiv.org/pdf/2401.06118.
limitations: limitations: yes
paper_summary: paper_summary: This paper introduces trellis coded quantization (TCQ) into large language model (LLM) quantization, achieving ultra-high-dimensional quantization with less inference burden compared to traditional vector quantization (VQ) methods. The main innovations are a hardware-efficient "bitshift" trellis structure and fast compute-based Gaussian codes, enabling high-quality quantization and fast inference.
strength_weakness: strength_weakness: This is a solid paper introducing a novel and robust quantization format. Inference efficiency is crucial for new quantization formats, but this paper only presents the inference speed on the RTX 4090 with 2-bit quantization. Providing inference speeds for more bits (e.g., 3-bit, 4-bit) and more devices (e.g., RTX 3090, A100) could enhance the paper further.

It is possible that existing quantization computation kernels achieve speedup on the RTX 4090 but not on the A100. Such phenomena are normal, and adapting a new kernel to different devices is challenging. However, I encourage the authors to report inference speeds on various devices. Even if results are not favorable, an in-depth analysis with potential solutions would provide a more comprehensive understanding without affecting the paper's contribution.
questions: questions: Please see the weaknesses for details.
limitations: limitations: The paper lacks inference speed testing on more bit levels and additional devices.
paper_summary: paper_summary: This paper proposes QTIP, a new method for efficient post training quantization of LLMs. QTIP is a vector quantization method inspired from Quip# [1], but with no limitation in the dimension of the codebook. The work’s fundamental contributions are: 1) Propose to adapt trellis quantization to the compression of LLM pre-trained matrices. 2) Design efficient trellis and codebook in dimension (or 'sequence length') $=256$. 3) Integrate the trellis quantization into the Quip# [1] optimization process to fine-tune the pre-trained weights. The numerical experimental results of the proposed method outperform AQLM [2] and Quip# [1] on the C4 and Wikitext2 datasets.

---

[1] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks, 2024.

[2] Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization, 2024.
strength_weakness: strength_weakness: 1) QTIP is mainly based on Quip# [1]: 'we use QTIP as a quantizer in QuIP#’s BlockLDLQ' (line.224). The code is not provided, but the authors explain that the whole quantization and fine-tuning process is based on Quip# [1]. Hence the contribution is more to the field of signal quantization in general, and LLM compression is an application.
2) QTIP natively reuses the incoherence processing from [1]. But this pre-processing step may erase the native correlation patterns that pre-trained LLM matrices bear [2,3]. No discussion about this process (and the blockLDLQ) is provided.
3) QTIP (with fine-tuning) update the pre-trained weights 'in a blockwise fashion' (line.247). This step is computationally intensive regarding the size of the matrices (and the optimizer states related to it), and it may impact the efficiency of the quantization process.
4) Very few details are given regarding the experiments design (which calibration dataset is used for fine-tuning 'hyb' codewords?), what may affect the results reproducibility (in addition the code is not publicly available).

---

[1] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks, 2024.

[2] Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. (2023). Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629.

[3] Guo, H., Greengard, P., Xing, E., and Kim, Y. (2023). Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. In The Twelfth International Conference on Learning Representations.
questions: questions: 1) Table.1 shows the superiority of QTIP over scalar quantization and Quip# [1]. How does it translate to the case of 'real' pre-trained LLM matrices? Can you show some results similar to [2], to identify which matrix presents the lowest quantization error ? This would enable us to decouple the gains provided by QTIP, and understand better which part (the quantization process itself, or the fine-tuning machinery) is key for sota performances.
2) Is 'hyb' codebook unique? Or do you have a different (fine-tunable) codebook for each LLM block?
3) Do you think QTIP fine-tuning step is (computationally) competitive with respect to (low-rank) adapters methods such as LoRA [3] and LQ-LoRA [2] ? Or QTIP is only tailored for LLM compression ?


---

[1] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks, 2024.

[2] Guo, H., Greengard, P., Xing, E., and Kim, Y. (2023). Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. In The Twelfth International Conference on Learning Representations.

[3] Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. (2021). Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.
limitations: limitations: A comparison of inference speedups is provided in section 4.3.
paper_summary: paper_summary: The paper introduces Trellis-based compression for quantization. Essentially, the method applies incoherence processing on the weight matrices, turning the approximately into Gaussians, such that the best compression method available to compression roughly Gaussian data can be applied to it. In this case, the authors choose a Trellis-based compression scheme.
They also introduce a specific Trellis setup with little overhead, for the setting where Gaussian data is the name of the game, making the method feasible in practice. According to the authors, this approach is novel. 
The authors show that there is a slight benefit of this compression method over other codebooking methods, such as GPTVQ, AQLM and QUIP#, at next to no extra overhead in terms of the implementation. 

As a note for the other reviewers and authors - I have no idea about trelisses, or the relevant literature. I am only knowledgeable about neural network quantization. So the veracity of some of the claims, like a novel Gaussian specific Trellis, I cannot comment on.
strength_weakness: strength_weakness: - The results are not TOO significant. In terms of perplexity, we're talking 0.1 ppl compared to QUIP# here. And the zero-shot results are always noisy, so tough to say what's faster.
- Because the compression is likely theoretically slightly more accurate, what remains is the inference-speed of this method. Now the authors, to their credit, do address this with their GPU implementation. But, because the improvements in perplexity are so little, for fair comparison we have to be absolutely sure that the latency for the kernel is the same. Otherwise, we might as well increase the codebook size for QUIP# a little and get a better trade-off, or have one layer in a slightly higher bit-width or whatever. The Speed analysis in Table 7 does not totally satisfy my need for a thorough analysis on this. Since this is so important for the paper, I would like to see an analysis for the kernels themselves, analyze the roofline points for the band-width/compute trade-offs (does your method do more compute than QUIP#?), and have this done for more bit-widths. I believe the authors have to be more thorough on this, else I would not recommend anyone implementing the paper, and the trellis compression is relegated to being a fun curiosity instead of a practical method.
- We don't all run things on GPUs - there are devices, with hardware accelerators, CPUs etc. What does hypothetical overhead look for this method compared to simple codebook look-ups? Some hardware has actual Sillicon for codebook look-ups, or specific instructions for it like in ARM CPUs. Would your method in these cases also still be faster? Since quantization is not only a 'speed-things-up-on-gpu' topic, I would like to see a discussion on this in the paper.

Editorial Notes
5 - […] PTQ approaches have converged. I don’t think convergence has really happened, so papers do this, some don’t. 
24-32 What is k? it’s undefined in the intro
160 psuedorandom -> pseudorandom
questions: questions: Is section 24-32 entirely correct? Technically you have 2^kd vectors, but nobody ever fills out the whole space with the codebook. So it’s always a subset \subset R^{2^kd} x d you are selecting for your codebook C. So it’s not really fair to say VQ requires exponential time and space… if you keep your codebook size similar, it does not require exponential time and space. Similarly, the O(2^{kd}d ) complexity is a bit misleading. You can also say the lookup is log(|C|) depending on the size of the codebook you chose. I am not being festicious here, vector quantization methods often take a fixed set of vectors 

Is the analysis is 116 sensible? Why would one ever have an unstructured k-bit codebook? There is always a way to add structure. 

Table 1 - Is this done keeping the overhead of each method the same? I can always increase the dimensionality of VQ to get lower values right? This comparison should only make sense if the compute/storage overhead is the same.  

How does this Trellis thing actually work? I'm trying to follow figure 2. I'm assuming the firs thing you need to store is the starting value 00. And then you follow the up-and-down errors each costing you a bit, so the red arrows should give you down, down, up, down down... So how do we end up with 0010110? Wouldn't it be 00 | 00 1 00 or 00 | 11 0 11? 
Also - what happens in your code if the next value cannot be transitioned to? In your example, if 0.1 follows 0.3, you're out of luck... does it then give you the closest value 0.3?

Do you have a simple explanation for why Trellis-based coding would work better than normal codebooks? Just 'dimensionality' doesn't sit well with me, because that doesn't really explain anything concrete. 


If I get more confidence from the authors on the speed of the kernels and the overhead of this method, with a discussion about non-gpu devices, especially given the very small gains they achieve - I am happy to increase my rating.
limitations: limitations: No pain points here

Comments: title: Thanks
Thanks
Thanks
Response to comment part 2
Response to comment part 1
Update on score
Responses to Further Comments
cutlass
Extra Note
Further comments
Responses to Further Questions
Hardware Supported Datatypes and Other Quantizers (re ZTfk)
Further questions
FP8?
More discussion on QTIP inference speed analysis (comment for rebuttal to W2)
comment: Thank you for your review, we will be sure to clarify how trellis coding works in an updated manuscript. We uploaded a response to F569's latest questions, which you may find useful.
Thank you for your review!
Thank you for your review, we will be sure to add clarifying information into an updated manuscript.
> A fair comparison would detail the expected activation error ...

The rebuttal does have this information for the 10_down layer of Llama 2 70B (see the paragraph after the table). QTIP achieves a lower proxy error than QuIP (scalar quantization) and QuIP# (vector quantization). We are not familiar with the AQLM codebase so we did not attempt to measure its proxy error. Due to time, we won't be able to get proxy error measurements for all the layers before the end of the discussion period. However, below are proxy errors for layers 39 and 79 of Llama 2 70B for scalar quantization, vector quantization (QuIP#), and trellis quantization (QTIP) when quantized to 2 bits. As expected, QTIP achieves the lowest proxy error in all cases.

**Note: there was a bug in the scalar quantization proxy error code in the first version of this response. We have updated the table with the correct results. The SQ proxy error in the original rebuttal was not affected by this bug.**

|  Layer  | 2 Bit Scalar Quantization Proxy Error | 2 Bit Vector Quantization (QuIP#) Proxy Error | 2 Bit Trellis Quantization (QTIP) Proxy Error |
|:-------:|:-------------------------------------:|:---------------------------------------------:|:---------------------------------------------:|
|   39 q  |                 0.0105                |                     0.0088                    |                   **0.0065**                  |
|   39 k  |                 0.0091                |                     0.0074                    |                   **0.0056**                  |
|   39 v  |                 0.0567                |                     0.0458                    |                   **0.0341**                  |
|   39 o  |                 0.0372                |                     0.0318                    |                   **0.0246**                  |
|  39 up  |                 0.0626                |                     0.0531                    |                   **0.0407**                  |
| 39 gate |                 0.0435                |                     0.0367                    |                   **0.0282**                  |
| 39 down |                 0.0680                |                     0.0582                    |                   **0.0441**                  |
|   79 q  |                 0.0064                |                     0.0054                    |                   **0.0041**                  |
|   79 k  |                 0.0049                |                     0.0041                    |                   **0.0031**                  |
|   70 v  |                 0.0453                |                     0.0387                    |                   **0.0291**                  |
|   79 o  |                 0.0069                |                     0.0053                    |                   **0.0041**                  |
|  79 up  |                 0.0116                |                     0.0099                    |                   **0.0075**                  |
| 79 gate |                 0.0116                |                     0.0099                    |                   **0.0075**                  |
| 79 down |                 0.0012                |                     0.0010                    |                   **0.0007**                  |

Regarding LQ-LoRA, our understanding is that LQ-LoRA focuses on obtaining a better quantized + low rank initialization for the final goal of downstream fine-tuning to new tasks. Post training quantization methods like QuIP#, AQLM, and QTIP focus on finding a quantized model that is *as close to the original model as possible*, which is a fundamentally different problem. LQ-LoRA mainly presents results on downstream fine-tuning, where they can get lower perplexity on C4 than even the original unquantized model. This is impressive, but should also make it obvious that LQ-LoRA is solving a different problem than PTQ.

> I believe some layers would suffer more from the incoherence processing step; see for e.g. early layers in Fig.1 in [1]

We're not sure what you mean by "suffer more" from incoherence processing. Incoherence processing is a fully invertible transformation up to numerical error. This means that incoherence processing does not result in information loss unless there is catastrophic numerical imprecision. As mentioned in the rebuttal, we did not observe differences from running our experiments in FP32 vs. FP64, so numerical imprecision is not an issue here. If you are questioning the efficacy of incoherence processing in the quantization process, we recommend taking a look at QuIP and QuIP#. Both papers have theory showing that the proxy error can be bounded by the incoherence $\mu$ of the weight matrix when using LDLQ variants. The bound gets tighter as $\mu$ gets smaller, which is what incoherence processing does.
Splitting into multiple parts due to the character limit

> However, my Q.1 remains, 'how to decouple the gains provided by QTIP, and understand better which part (the quantization process itself, or the fine-tuning machinery) is key for sota performances?'.

The main difference between our QTIP experiments and QuIP# is the quantizer. The QTIP experiments use our fast trellis quantizer and QuIP# uses a vector quantizer. Therefore, the difference between QTIP and QuIP# is the difference from using a better quantizer. Our QTIP experiments and QuIP# both use the BlockLDLQ rounding algorithm and the same fine-tuning algorithm. 

We are not claiming that trellis coding alone is sufficient for state of the art quantization performance. Doing so would ignore the vast model quantization literature that shows the efficacy of algorithms such as adaptive rounding and fine-tuning. Rather, all we are saying is that trellis coding's gains over vector quantization translate to LLM quantization, *even after adaptive rounding and fine-tuning*. This means that although adaptive rounding and fine-tuning are important to achieve strong performance, there is still a significant benefit to using a better quantizer. QTIP lets us achieve fast decoding with trellis coding, making trellis coding practical for LLM quantization.

> Your rebuttal provides a comparison of a single LLaMA layer and iid Gaussian vectors, but I believe this comparison is not informative regarding Q.1.

This table was in response to "How does it translate to the case of 'real' pre-trained LLM matrices? Can you show some results similar to [2], to identify which matrix presents the lowest quantization error ?" Our understanding is that Figure 1 in the LQ-LoRA paper measures the RMSE (sqrt distortion) of the original weight matrix. The table in the rebuttal does this as well and shows that after incoherence processing, "real pre-trained LLM matrices" have the same quantization distortion as an i.i.d Gaussian with QTIP. This is significant because, to the best of our knowledge, trellis coding achieves the lowest empirical distortion on memoryless Gaussian sources (https://ee.stanford.edu/~gray/trellis.pdf). Having this result translate to LLM matrices means that QTIP is indeed a better quantizer than VQ, even on actual pre-trained matrices.
Dear Authors,

I have read the rebuttal. I will keep my positive rating.
After finally understanding the paper properly, I believe I can now say that I think the QTIP method is solid and worthwhile considering as a method to encode and decode weights for LLM inference. I think the approach is novel enough on top of existing codebook approaches. Although I think the result improvements are still minor, and thus do not warrant a best-paper award, I believe any smidgeon of improvement counts. If we were to reject a paper just because the improvement is small, we'd have a big problem in our field of efficient deep learning. Especially since the benefits would be essentially free, if a suitable implementation exists.

I do however still have two import notes:
- It took me quite a while to understand the method - and with me likely the average reader that's not inundated in the field of Trellis codes. I would strongly encourage the authors to more clearly explain their method in the paper, so new readers will not have to resort to Openreview or send you an e-mail to understand what is going on.
- I am also a bit partial to reviewer F569's question on showing the benefit of this code versus other codes for more examples. Incoherence processing only approximately and probabilistically makes your distributions Gaussian - I've seen that in practice outliers can still occur, and we're left guessing now what happens with this method in such cases. A more extensive expose of this method on more than a few distributions that occur in the wild would do it good.

That said, the positives outweigh the negatives and I've increased by score to a weak accept.
First, let me thank you for your response and for running the experiments. I have a few comments regarding your response:

(W4) Generally, I would not recommend trusting PPL results at this scale. There is a risk of overfitting on the data. You can see this yourself by looking at Table 5 at zero-shot results. While QTIP ppl constantly better at 4 bits (although by not much), the difference at zero shots are not significant.

(Q1) This was not clear to me from reading the paper. I would suggest making it clearer wherever you are using fine-tuning version.

(W1, W2) I would suggest adding this information in the appendix, if it fits.

(Q6, W3) Great, thank you!

I will keep my score.
Thank you for your answers. However, my Q.1 remains, 'how to decouple the gains provided by QTIP, and understand better which part (the quantization process itself, or the fine-tuning machinery) is key for sota performances?'. Your rebuttal provides a comparison of a single LLaMA layer and iid Gaussian vectors, but I believe this comparison is not informative regarding Q.1. A fair comparison would detail the expected activation error (proxy error, Eq 1) or the MSE for QTIP **and** other methods (e.g. LQLoRA, Quip#, AQLM, etc.) for all layers in a LLM (I believe some layers would suffer more from the incoherence processing step; see for e.g. early layers in Fig.1 in [1]). I will keep my score.



[1] Guo, H., Greengard, P., Xing, E., and Kim, Y. (2023). Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. In The Twelfth International Conference on Learning Representations.
> I guess a final question for me to understand ...

To clarify, these computed codes are not random, they just do a good job of decorrelating the state ID from the computed value. Like your example FP8 casting code, our codes are deterministic in their outputs. The reason why we want a code that appears random is because under the bitshift trellis, random codes are asymptotically optimal as L increases (see the RPTC paper for more details https://ee.stanford.edu/~gray/trellis.pdf). Table 1 in the RPTC paper shows that an optimal code with a small L has higher distortion than a random code with a large L. This means that we should always increase L as much as possible, and our compute-based codes let us do that without impacting decoding speed.

> One thing that worries me is that for a low number of bits, your 'approximately Gaussian' doesn't really hold very strongly, and there will be quite some bias in the number format you choose.

In trellis coding, the codebook assigns values to the *states* (2^L), so the number of representable codebook values is independent of the bitrate K. These codes are approximately Gaussian for large enough L ($\gtrapprox$ 10), and our experiments use L=16. Figure 3 shows the set of representable neighboring values in a bitshift trellis for various codebooks when L = 16. Indeed, 1MAD and 3INST give very similar coverage as a random Gaussian. Figure 3 also shows why we want a “randomized” codebook. Since neighboring states in the bitshift trellis share L-KV bits, a poorly designed computed code will have strong correlations that result in unrepresentable neighboring values. The leftmost plot in Figure 3 shows such a code, where large portions of the state space are unrepresentable with the bitshift trellis. A code that does direct casting to FP8 (what you suggested above) would have extreme correlations since the mantissa bits of a state become the exponent and sign bits of the next state. Openreview isn’t letting me send an image of the correlation plot, but you can generate a plot with the Python script below to verify for yourself. Section 3.1 in the paper also has more details on random codes.
```
import torch
import matplotlib.pyplot as plt

L = 8
# bitrate, adjust accordingly. V =1 for plotting purposes.
K = 4

states = torch.arange(1 << (L+K), dtype=torch.int32)

left = (states >> K).to(torch.uint8)
right = (states & ((1 << L) - 1)).to(torch.uint8)

lval = left.view(torch.float8_e5m2).float()
rval = right.view(torch.float8_e5m2).float()

plt.scatter(lval, rval)
plt.show()

```

> On the exponential scaling ...

The focus of the paper is indeed to show that QTIP achieves a better quality/speed tradeoff over existing quantization methods. Our empirical results show that QTIP outperforms VQ-based methods such as QuIP# and AQLM while offering the same fast inference as QuIP#. The section about VQ’s exponential scaling was to explain why we can’t increase the VQ dimension to improve quality while preserving fast inference. We know from information theory that the distortion of a vector quantizer is lower bounded by a strictly decreasing function of the dimension (http://vkostina.caltech.edu/pdfs/2012KostinaVerdu-lossycomp.pdf). This means that for optimal codebooks and a fixed bitrate, the only way to improve VQ’s quality is by increasing the dimension. TCQ lets us achieve lower distortion than the best “fast inference” VQ, which translates into higher quality quantized LLMs. QTIP’s contribution is enabling fast decoding with TCQ so we can quantize LLMs with it while supporting fast inference.

> It would be great if you could be a little bit more explicit ...

Most of our responses are based off of information already in the paper. We will make this information more clear in an updated manuscript. We will also add new information from the rebuttal/discussion period, such as those on hardware requirements for QTIP's kernels.
NVIDIA TensorCores do not take mixed-precision inputs natively, so kernels like Cutlass and BitBLAS usually upcast the lower precision operand and do the matmul in the higher precision operand’s datatype. For memory bound inference, this doesn’t affect throughput since the decompression step is dominated by reading the weights and activations from memory.
Thanks for the comparisons and extra explanations!

Small note: There is nothing stopping the Nvidia hardware from doing mixed-precision kernels, e.g. Cutlass generates mixed-integer-precision like w4a8 kernels just fine :)
Thanks for the explanation on how the codes are used. I understand it now. Essentially, you are trading off compute for memory this way. Instead of storing a codebook, there is a procedure for generating the codebook on the fly. When you decode, you run it through the function and you get e.g. 11 -> 0.332 which is the decoded value. And I’m assuming the codebooks would otherwise be much larger than the programming code to do this on-the-fly generation.

I guess a final question for me to understand why this method could be better theoretically: Why would these randomly generated codes be better than something that is not randomly generated? E.g., another value generating function could just be something that maps bits to a floating-point number, arithmetically: e.g. 8 bits could map to something with 1 sign, 5 mantissa and 2 exponent bits in FP. Something we also know approximates a N~[0, 1] Gaussian well. For e.g. 4 bits you can probably find the optimal mapping of 4 bits to whatever mantissa/exponent bit configuration is the most accurate for matching a Gaussian as well. This can be done arithmetically too without an explicit codebook. What is the benefit over something that’s seemingly randomly generated over something that has such structure? One thing that worries me is that for a low number of bits, your 'approximately Gaussian' doesn't really hold very strongly, and there will be quite some bias in the number format you choose. 
I guess, in the end all these codes are doing are defining a different (random) way to put representable points on the number line. Could you plot for your experiments where the representable values finally lie on your number-line? 

On the exponential scaling. I don’t believe the exponentially of the codebook sizes in the dimensions directly matters, since the problem is rather: Given a certain accuracy, what is the most efficiency I can get. Or in codebook terms, given a codebook size (e.g. something that fits in L1 cache), what is the best accuracy I can get. This is very different from combinatorial optimization problems, where you might have a problem with trying a TSP solver on a small graph, and then when you scale it to industrial sizes the algorithms don’t work and they break because it’s too expensive. This is not the case here in this setting, which has mostly fixed constraints. Dimensionality, or scalability with respect to dimensionality, is not the goal by itself here and it’s solely subject to the accuracy/performance trade-off. I believe your story would be easier to comprehend if that was the focus, not the fact that your algorithm somehow scales better in big-O notation.

It would be great if you could be a little bit more explicit with what unclarities/comments you agree with, and commit to adding to your paper, and what not. Some things that are unclear to me, like e.g. how are the look-up free codes used in the algorithm, I’m assuming would be unclear to others as well. It would be great to see if you could commit to making these things clearer in the paper for folks that have not read up on Trellis-based encoding and can infer what you are doing. 

Converged -> How about "Recent state-of-the-art 4 PTQ approaches have utilized using vector quantization (VQ)”
> Thanks for your explanation on the Trellis decoding ...

We will add it to the paper.

> I am still unclear on what the Lookup-free computed codes ...

These codes take a L bit word and return a FP16 number (or vector for HYB). For a uniform distribution of L bit inputs, the outputs of these codes are approximately i.i.d Gaussian. These codes are used to generate the codebook during trellis quantization. Recall in trellis quantization that the trellis has 2^L nodes/states, each with 2^{kV} directed edges to other states where k is the bitrate and V is the number of weights we quantize in a single step. Each state has a V-dimensional vector assigned to it, and a length-T input sequence is quantized to a length (T/V)-1 walk on this graph. The reconstructed sequence is the concatenation of the state values on this walk. 

For example, in Figure 2 in the paper, the walk consists of nodes (00, 01, 10, 01, 11, 10), so the reconstruction is (0.5, 0.1, 0.8, 0.1, 0.3, 0.8), since these are the values that correspond to those nodes. In QTIP, the codes are used to generate the value of node i by running the algorithm on i. Since there are 2^L nodes, i spans 0 to 2^L-1, so the codebook consists of all possible outputs of a code. This means that the codebook is approximately i.i.d Gaussian, which is what we want for quantizing a i.i.d. Gaussian source.

To actually quantize a sequence of T weights, we can run the Viterbi algorithm on this graph. The Viterbi algorithm performs dynamic programming by storing the minimum distortion achievable at step t, 0 <= t <= T/V - 1. Section 2.3 in the paper has more details, but the algorithm itself is well documented (https://ieeexplore.ieee.org/document/1450960). In our experiments, we split the weight matrix into 16x16 tiles and quantized each one as a length 256 sequence using BlockLDLQ from QuIP#. This is just one way of applying QTIP, and you could alternatively quantize the entire matrix as a single sequence if the way you quantized (e.g. LDLQ, direct optimization a la AQLM, etc.) was compatible with that.

> On the discussion of the runtime and it’s exponential time ...

Can you explain why exponential scaling doesn’t matter? As you noted above, VQ is practical when applied at lower dimensions like 4 and 8 because the codebook can be kept small. We know from Table 1 and the QuIP# paper that going from 4D VQ to 8D VQ significantly improves quality, so going past 8 dimensions would further improve quality. However, 8 is the highest dimension current VQ methods can achieve while still maintaining fast inference because higher dimension codebooks would not fit in cache. This means that we are indeed limited by the size of the codebook and VQ’s space complexity; we cannot arbitrarily improve VQ’s quality by increasing dimension while still maintaining inference speed. In contrast, TCQ doesn’t suffer from these problems. We can do TCQ without any stored codebooks at all, and decoding is linear in the bitrate.

As a side note, I don’t think we fundamentally disagree about VQ’s problems. Perhaps “practicality” is the wrong word here and something like “scalability” would be better. We’re open to using other words if you’d like to suggest one.

> For some of the comments I made ...

We plan on adding revisions from discussions with all reviewers after the review period. 

> Editorial Note: Even if AQLM and QUIP# are ...

We are also open to using a different word than “converged,” if you’d like to suggest one.
>Wouldn't a ~2x speed-up of e.g. 4-bit tok/s compared to FP16 tok/s get negated when the FP8 format is used? Also, wouldn't the fairer comparison be more of what happens when you use 4-bit native kernels versus 4-bit kernels with the trellis coding?

Not really. QTIP 4 bit is actually closer to 3X faster than FP16 on Ada/Hopper, which is what you’d need for hardware FP8 support. Of the 3 GPUs we tested on, only the RTX 6000 Ada has hardware support for FP8, so you wouldn’t get any compute speedup from FP8 on the 3090 and A6000 Ampere. QTIP could also be made faster by fusing some of the Hadamard transforms like QuaRot does “for free”. We can test this by fusing the same RHTs as QuaRot’s. Here, we get 151 tok/s on the A6000 Ada for 7B 4 Bit. 

Quantizing to 8 bits also won’t give a 2X inference speedup due to overhead from other parts of the model. We can emulate inference with an 8 bit model by simply dividing half the matrix dimensions by 2 in a FP16 model, which conveniently means doing half the compute and reading half the memory. The alternative would be using the TransformerEngine FP8 GEMM kernel, which is complicated to integrate. On an A6000 Ada, this gives 98.4 tok/s for Llama 2 7B, which is only 1.77x the speed of FP16 and 35% slower than 4 bit QTIP with fused RHTs.

Another caveat with using hardware supported datatypes like FP8 and INT4 is needing to quantize *both the activations and weights*, since we don’t have hardware support for mixed precision GEMMs. Quantizing the activations means sacrificing quality, and existing methods give worse performance/speed tradeoffs than QTIP. For example, to the best of our knowledge, QuaRot and SpinQuant are the two best performing W4A4 methods, and they both perform significantly worse than 4 bit QTIP (QuaRot 6.10 ppl, SpinQuant 5.9 ppl, QTIP 5.52 ppl, FP16 5.47 ppl, Wikitext 2, ctx. 2048). SmoothQuant’s W8A8 setup does have similar quality as 4 bit QTIP, but here again the speedup will be less than 4 bit QTIP in memory bound scenarios (see above).

However, we don’t actually need hardware support for smaller datatypes to do memory bound inference. As long as we can write a kernel to do decompression during the matrix multiplication, then we can use any quantizer. This is what QTIP and existing weight-only quantization methods like QuIP# and AQLM do. To test speed here, we used Microsoft’s BitBLAS library as a drop-in matmul in torch’s Linear layer. BitBLAS contains highly optimized kernels to do FP16 x dtype matmuls, where dtype is an IEEE dtype. Since these datatypes are much easier to dequantize than QTIP’s codes, these kernels should be about as fast as possible for a weight-only quantization method. BitBLAS FP16 x INT4 gets 150 tok/s on the A6000 Ada, which is essentially the same as QTIP’s 151 tok/s. However, QTIP is still doing some RHTs, so QTIP’s 4 bit kernel is actually faster than BitBLAS’s 4 bit kernel.

From the QuIP# paper, we know that doing INT4 quantization without the RHT (e.g. GPTQ 4 bit) results in significantly worse quality than QUIP#, which has worse quality than QTIP at all bitrates. This means we can’t actually use BitBLAS FP16 x INT4 to reach the same quality as 4 bit QTIP, *and* QTIP is faster. To reach the same quality as 4 bit QTIP, we would need INT8 quantization, which we already know is 1/3 slower than 4 bit QTIP. Finally, we note that compared to QuIP# and AQLM, our two main baselines, QTIP is both faster and higher quality at all bitrates. This means that QTIP strictly improves over those two VQ-based methods, showing that TCQ is indeed practical. We are also not professional kernel writers, so it is very possible that someone could write a faster kernel than us, giving further speedups.
Some more questions and comments:

Thanks for your explanation on the Trellis decoding - I believe it would also be wise to add this explanation to the paper, as as a layman in the information theoretical source coding field, I did not get it at first. 

I am still unclear on what the Lookup-free computed codes actually do. As I understand it, you can take any word, run it through your 1MAD or 3INST algorithm, and out comes the number you decode right? Do they still have anything to do with the trellis coding that you mentioned? Given a weight tensor, how do you actually find which words encode this value the best? 

On the discussion of the runtime and it’s exponential time - I believe I understand what you are saying, but I also think it’s a bit unfair. Your introduction essentially reads: There’s these VQ methods, and they are exponential in the dimensionality, our method achieves better scaling in dimensionality, so it’s better…our problem sizes are fixed, we’re compressing a fixed-size network. These compression methods are never used in a method where the exponential scaling in dimensions actually matters. The authors of the other papers find an optimal trade-off in dimensions/num_clusters->codebook size and performance. This is not like a travelling salesman problem where your O() notation actually matters if you want to scale up your problem to larger sizes.  The dimensionality is a choice, and the scaling being exponential is not the actual fact that matters. Rather your question is: At the same codebook sizes/decoding time, what is the best packing I can achieve. You claim ‘VQ requires exponential […]  which limits its practicality', which I think is too strong. These methods are very practical because they are applied on lower dimensions, and the codebook size is explicitly limited. Given that this argument is so important for your paper, I would take some care :)

For some of the comments I made, I was also secretely asking that they would be addressed in a new revision of the paper, any thoughts on that? ;)

Editorial Note: Even if AQLM and QUIP# are the highest quality LLM PTQ methods currently known to you, it does not mean the field has converged. I don’t believe any model efficiency committee with prominent researchers created a consensus of what is best, and concluded that there is nothing better out there. If anything, it's highly dependent on what's available on the hardware as well.
Wouldn't a ~2x speed-up of e.g. 4-bit tok/s compared to FP16 tok/s get negated when the FP8 format is used? 
Also, wouldn't the fairer comparison be more of what happens when you use 4-bit native kernels versus 4-bit kernels with the trellis coding?
In general, reducing the size of the uncompressed codebook reduces the cost of lookup. QuIP# is forced to use a battery of bit manipulation operations (amortized to 3 per weight) to compress its 65536-entry V=8 codebook down to a 256*32-bit LUT. Due to the trellis-based quantization algorithm, QTIP HYB requires a much smaller uncompressed codebook (1024 entries at V=2) to achieve better accuracy. We decompress this from a 512 * 32-bit LUT in only 0.5 integer ALU instructions per weight, versus QuIP# which requires 3. QTIP HYB has an additional cost of ~1.5 integer ALU ops to unpack the trellis and calculate the codebook index, which comes out to a total of 2 ALU ops per loaded weight. At 2 bits, this is 8 ALU ops per loaded byte. On the flip side, QTIP at V=2 requires 4x more codebook lookups than QuIP# at V=8. The effect of these differences on inference speed depends on the details of the specific target microarchitecture.

Essentially,
- QTIP reduces the necessary uncompressed codebook size, which reduces the necessary runtime resources (smaller LUT hardware or fewer ALU ops: 0.5 ALU ops for HYB vs 3 for QuIP# E8P).
- QTIP HYB requires 1.5 ALU ops per weight to unpack and map trellis entries to codebook indices.
- QTIP HYB requires more lookups per weight than QuIP#. This is fine on modern GPUs, which have fast banked shared memory. On other architectures, performance will depend on the SIMD shuffle or L1 cache throughput. However, QTIP can be done without any lookups at all (e.g. 3INST and 1MAD) while QuIP# cannot.
- The decompress overhead is essentially constant for all bit widths for QTIP. VQ costs more as bit width increases.
- QTIP 3INST and 1MAD require a fixed number of ALU instructions to decode, and do not use a LUT, reducing cache considerations.

where “ALU ops” here refers to INT32 ALU ops.

**Openreview is not letting me make a comment to the rebuttal, so you are going to see this before the rebuttal. Please read the rebuttal first before reading this.**