Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment
Authors: Deokjae Lee
Hyun Oh Song
Venue: NeurIPS 2025 poster ()
Keywords: LLM quantization
Post-training quantization
Mixed scheme quantization
Data-free quantization
Primary area: deep_learning
TL;DR: We develop Q-Palette, a quantizer suite with efficient inference CUDA kernels and wide fractional-bit support, enabling mixed-scheme quantization that achieves ~36% faster LLM decoding than NormalFloat while improving accuracy.

Abstract: We study weight-only post-training quantization (PTQ), which quantizes the weights of a large language model (LLM) without retraining, using little or no calibration data. Weight-only PTQ is crucial for reducing the memory footprint and latency of LLM inference, especially in memory-bound, small-batch inference scenarios, such as personalized inference on edge devices. Despite its importance, irregular weight distributions with heavy-tailed outliers in LLMs complicate quantization, recently motivating rotation-based methods that transform weights into near-Gaussian distributions, which are more regular with fewer outliers, thereby reducing quantization error. In this work, we first derive the information-theoretically optimal bit allocation for Gaussianized weights under given bit budgets, revealing that fine-grained fractional-bit quantizers approaching the Gaussian distortion-rate bound are essential to achieve near-optimal quantization performance. To bridge this theoretical insight and practical implementation, we introduce Q-Palette, a versatile collection of fractional-bit quantizers that range from trellis-coded quantizers offering near-optimal distortion to simpler vector and scalar quantizers optimized for faster inference, all efficiently implemented with optimized CUDA kernels across various bitwidths. Furthermore, leveraging Q-Palette as a foundational component, we propose a novel mixed-scheme quantization framework, jointly optimizing quantizer choices and layer fusion decisions given resource constraints. The code is available at https://github.com/snu-mllab/Q-Palette.

Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment
Deokjae Lee 1.2
Hyun Oh Songl.2+
Seoul National University, Neurrl Processing Research Center {bdbj, hyunoh) snu kc
Abstract
We study weight-only post-training quantization (PTQ), which quantizes the weights of large language model (LLM) without retraining, using little or no calibration data. Weight-only PTQ is crucial for reducing the memory footprint and latency of LLM inference, especially in memory- bound, small-batel inference scenarios, such personalized inference on edge devices. Despite its importance, irregular weight distributions with heavy-tailed outliers LLMs complicate quan- tization, recently motivating rotation-based methods that transform weights into eear-Gaussian distributions, which are more regular with fewer outliers, thereby re- ducing quantization error. In this work, we first derive the information-theorriice. optimal bit allocation for Gaussianized weights under given bit budgets, revealing that fine-grained fractional-bit quantizers approaching the Gaussian distortion-rate bound are essential to achieve near-optimal quantization performance. To bridge this theoretical insight and practical implementation, we introduce Q-Pulette, ver- satile collection of fractional-bis bit quantizers that range from trellis-coded quantizers offering optimal distortion to simpler vector and scalar quantizers optimized for faster inference, all efficiently implemented with optimized CUDA kernels across various bitwidths. Furthermore, leveraging Q-Palette component, we propose a novel mixed-scheme quantization framework, jointly optimizing quantizer choices and layer fusion decisions given resource constraints. The code is available at http:://tttibbccommmmmmmaa/////p
foundational
1 Introduction
Large language models (LLMs) have recently achieved significant success across diverse tasks and are increasingly being deployed on resource- edge devices, such as laptops or smartphones [39 (3)... However, these edge devices typically have limited memory resources and often process small-batch workloads, making inference severely memory-bound. Weight-only quantization has thus become essential, allowing models to achieve significantly greater compression at similar levels of performance compared to quantizing both weights and activations. Moreover, recent studies have demonstrated that weight-only quantization, beyond its well-known compression advantages, can also significantly accelerate inference speed in small-batch decoding scenarios by alleviating memory bottlenecks (28 40 20). Specifically, we address weight-only post-training quantization (PTQ), enabling model quantization without costly retraining or extensive calibration data, which are common constraints in real-world deployments 24
However, quantizing LLM weights remains challenging due inherently irregular, heavy-tailed distributions containing outliers that significantly broaden quantization ranges [23] 28 29 54]. To
address this, recent research introduced a theoretically grounded approach known as incoherence
*Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) NF3064 w/ FLUTE kemne 1
Bits
266
Q-Palette
Baseline
Wiki2 Perp.: 7.70, Speedup: 2.33x
Trellis-Coded Coded Quant.
TCQ-3.25
Rige
Rup
Ours
WikiZ Perp.: 6,48, Speedup: 2.72x
TD
Z+
25
Q-Palette (Wik12 Perp.: 6.46, Speedup: 2,99x))
frsed
Wo.
frsed
Vector Quant.
(c) Fusion-aware MSQ O-Palette, Perp.: Speedug: x.17x)
R...
s0
uniform Quami
IF
Ont
Attention
blocks wi fased quomtiedd logen Figure I: Qualitative comparison of quantization frameworks based on Q-Palette against the Nor- malFloat baseline with FLUTE kernels [[] (0),, evaluated on the LLaMA 3. 3.I-8B model using an RTX4090 GPU with a batch size of 1. Compared to (a) NormalFloat, (b) single-scheme quantiza- tion with TCQ-3.25 achieves a 17% inference speedup. (c) MSQ with Q-Palette provides a 28% speedup, and (d) fusion-aware MSQ further yields 36% speedup alongside reduced Wiki'eext2 perplexity, highlighting the practical effectiveness of Q-Palette and our MSQ framework. In the MSQ visualizations, columns represent transformer blocks, and rows represent linear layers, with colors indicating selected quantization bitwidths. The right visualization illustrates fused layers within the 30-th transformer block from configuration (d). Refer Appendix F for the experimental details.
processing, which applies rotation matrices (e.g., random Hadamard transforms) weight matrices, reducing outliers by modifying distributions into approximately Gaussian forms (1 49).
We begin with natural question: if ideal Gaussian quantizers were available at arbitrary fractional bitwidths, how should we allocate bits across layers to minimize performance degradation under fixed memory budget? Building upon the linearity theorem [35, which approximates performance degradation by quantization as a weighted sum of layer-wise wise mean squared errors, we derive an iiformation-hhoortiice
optimal bit allocation strategy for Gaussianized weights. Our analysis reveals that fine ten-iiri fractional bit quantizers that closely match their theoretical distortion bounds are essential to approaching the quantization performance predicted by theory. However, existing sophisticated Gaussian quantizers. such as trellis-coded quantization, have only been implemented with fused kernels for limited set of integer bitwidths (e.g., 2, 3, 4 bits), with little or no support for
batch sizes larger than one [19 15 49 50].
To bridge theoretical insights with practical quantization, we introduce Q-Palette, a versatile set of fractional-bit quantizers, ranging from trellis-coded quantizers (TCQ) for near-optimal distortion to simpler vector and scalar quantizers for low latency, covering diverse aecuracy-latent trade-offs. We provide optimized CUDA kernels supporting a wide range of fractional bitwidths with broader batch size support than prior sophisticated quantization methods (e.g., QTIP (50). To enable even finer bitwidth control, we further propose half-TCQ, novel TCQ variant that mixes two TCQ quantizers of different bitwidths within a single layer (e.g., 2.5 and 3.0 bits) to realize intermediate bitwidths such as 2.75 bits, and extend our CUDA kernels to support this variant.
We integrate Q-Palette into resource-constrained mixed-scheme quantization (MSQ) framework to demonstrate its practical utility. To further improve accuracy-latency trade-offs, we propose fusion-aware MSQ, the first MSQ approach that jointly optimizes quantizer selection and layer fusion, introducing a new optimization dimension (see Figure D. Here, linear layers sharing the same input (e.g., query, key, and value projections in Transformer block) can be fused into single linear layer, reducing memory accesses and kernel launches [26] 12 40). By incorporating layer fusion, fusion-aware MSQ achieves significant gains eecuracy-latencee trade-offs. Extensive experiments on LLaMA 2, L.LMAA 3, and Qwen models demonstrate that our MSQ framework with Q- Q-Palette consistently outperforms strong data-free and data-aware weight-only PTQ baselines under both memory- and cinetey-onsttriied settings.
2

---

2 Preliminaries
2.1 Linearized surrogate objective for post-training quantization
Previous studies have approximated the performance degradation in neural networks induced by PTQ as linear combination of layer-wise surrogate losses derived from second-order Hessian approximations [14 77. the context LLMs, where performance typically measured by perplexity, the linearity theorem shows that the perplexity increase induced by quantization be accurately approximated as a weighted sum per-layer quantization errors [35]. This surrogate is especially valuable for fata-free quantization scenarios where Hessian-based surrogate losses unavailable. Formally, the linearized surrogate is expressed as:
C(O(WW))=) - C(tW)) -CU
Σ ai 1Q(W)) - wwll/
crr(Q:we
where L(-) denotes the perplexity loss, Wi E xef" is the weight matrix of layer I, Q(-) is quantization function, dr is the empirically estimated sensitivity coefficient for layer and err Q; Wi) is the normalized quantization error of layer L.
Leveraging this surrogate, the Q can be formulated as a multiple-choice knapsack problem (MCKP);
MSQ problem with a set of candidate quantizers
minimize (o,E(0,1)
a1
Piq err(Qu; Wi)
(1)
10
subject to
Pig = 1,
AI VIIIS
12
Σ
E Pig bit( Qgi W, Widdfayn s M, 1=1
where Qy denotes candidate quamtizer, bit(Qq: Wi) is the average number of bits per weight component for the weight matrix Wi quantized by Qq. Ply E 1] is binary indicator selecting quantizer Qg for layer and M denotes the total memory budget (in bits) allocated for quantized model [46). This formulation explicitly casts MSQ as a combinatorial optimization problem grounded in linearized performance surrogate, providing a principled framework for optimal bit allocation under strict memory constraints 35 7
Q-Palette: fractional-bit quantizers
3.1 Motivation and design goals
Building on the theoretical foundation introduced in Section we now derive the information- theoretically optimal bit allocation strategy and discuss its implications for the design of practical quantizers. Under the assumption that weight matrices are Gaussianized via incoherence processing, the quantization problem be viewed
Gaussian source coding problem. In this setting, classical inte-distortion theory establishes a fundamental lower bound on expected quantization error as [()rr(()) 2 2 2-2ba(Q) [1O]. Assuming ideal Gaussian quantizers that achieve this bound at arbitrary fractional bitwidths by n, the nemoryyccostraingg MSQ problem simplifies to:
minimize b
0,2 2b
(2)
l=1
subject
bidit yout < M,
where by is the fractional bitwidth allocated to layer 1, and 1/ > 0 is minimum bitwidth threshold introduced avoid degenerate cases such as assigning 0-bit layer. This formulation admits closed-form solution stated in Theorem
3

---

Table 1: Quantizers, kernel implementations, and supported bitwidth intervals in Q-Palette.
Quantization scheme
implementations
Supported hitwidths (bits)
Non-uniform scalar quantizution (NUQ)
Tensor Core, CUDA Core
2.0, 3.0, 4.0, 5.0, 7.0,
Vector quantization (VQ)
Tensor Core, CUDA Core
1,5, 2.0, 2.5,
4.5,0,5,
Tensor Cote (TCQ)
1.5, 2,5, 4.0,
Trellis-coded quantization (TCQ)
Tensor Core (Half-TCQ)
1.75, 2.25, 2.75, 3.25, 3.75, 4.25, 4.75
Theorem 3.1 (Optimal bit allocation with ideal Gaussian quantizers). If the budget M is feasible, i.e., M Ei=1 dft ditt. then the optimal fractional bit allocation {b} for problem is given by
+ + VISISL,
bi max The (1n
In
diddpu
for the constant C. that satisfies the memory constraint
Σ b7 M.
Proof. See Appendix A
In practice, however, quantization must be performed using a finite of non-ideal quantizers Q = (Q.,..... Jv.. which introduces gap between theoretical optimality and actual achievable factors: (I) how closely each quantizer
performance. The extent of this gap depends primarily approaches the ideal distortion bound 2 2-bbit(Q), and (2) how finely the available bitwidths can approximate the optimal fractional bit allocations b7. Please refer to Appendix B for further analysis.
This motivates the need for practical quantizers that are both accurate and available at fine-grained fractional-bi intervals. Moreover, quantizers often exhibit a trade-off between distortion and computa- tional efficiency: more sophisticated quantizers may offer lower error but incur higher inference costs. These considerations motivate the design practical quantization suite that supports fine-grained fractional bitwidths while also accounting for trade-offs between quantization error and computational efficiency. Guided by these goals, we design Q- Q-Palette as a versatile collection of quantizers tailored to balance quantization error and computational efficiency across deployment scenarios.
3.2 Quantization schemes in Q-Palette
Q-Palette includes three quantizer families, non-uniform scalar quantization (NUQ), vector quantiza- tion (VQ), and trellis-coded quantization (TCQ), spanning range of quantization error and inference latency trade-offs. In this section, we briefly introduce these quantization schemes.
scalar quantization. NUQ quantizes each scalar weight by assigning it to an entry non- uniformly spaced lookup table (LUT), in contrast to uniform scalar quantizers, which use equally spaced intervals [27]. We construct NUQ codebooks via k- means clustering on random Gaussian samples (34), thus optimizing the codebook entries for Gaussianized weights. NUQ incurs low dequantization overhead and
Non-uniform
0.75
VQ
TCQ
0.5
Optimal
RMSE
0.25
enables efficient inference 220 40).
Vector quantization. VQ partitions weight vectors into groups of fixed dimension, assigning each group to the nearest vector entry in precomputed codebook (5]]. In Q-Palette, we specifically implement 2D VQ, gener- ating codebooks via k-means clustering on random 2D Gaussian samples. Efficient kernel implementations rely on LUTs whose sizes are powers of two, enabling compact biv-level representations. Consequently, frac- tional bitwidths at 0.5-bit intervals become natural can- didates for efficient implementations. For instance, (23, 2)-shaped LUT (eight 2D vectors) encodes two elements with bits, effectively achieving 1.5 bits per scalar weight. Such constructions enable VQ to support fractional bitwidths at intervals of 0.5 bits.
1.5
2.5
3.0
3.5
4.0
4.5
5.0
Bitwidth
Figure 2: Gaussian quantization error of Q- Palette quantizers (NUQ, VQ. TCQ) com- pared to the uniform baseline.

---

Table 2: Decoding-latencyy speedup of quantized LLaMA 3.1-8B models relative to the FPI6 baseline on an RTX 4090 GPU. 'TC' and 'CC" denote Tensor Core and CUDA Core kernels, respectively.
Decoding-latency speedup compared to FP16 (batch size
Quantizer
2.0
2.25
2.5
2.75
3.0
3.25
3.5
3.75
4.0
4.25
NF FLUTE 11
20
2.33x
2.20x
QTIP 50
2.91x
2.49
2.23x
Ours-NUO-TC
3.64x
2.97
2,57
Our--NUQ-CC
3.70x
3.07
2.61x
Ours VQ-TC
3.63x
3,24x
2.97
2.69x
2.57
Ours-VO VQ-CC
3.69x
3,36x
2.82x
2.60x
Ours-TCQ-TC
3.57x
3.23x
3.13x
2
2.96x
2.72x
2.70x
2.61x
2.59x
2.37
Decoding latency speedup compared to FP16 (batch size 8)
Quantizer
2.0
2.25
2.5
2.75
3.0
3.25
3.5
3.75
4.0
4,25
NF FLUTE 11 20
2.28x
2.13x
QTIP 550
0.74x
0.55
0.47
Ours-NUQ-TO
3.28
2,78
2,46x
Ours-NUQ-CC
2,80x
2.56x
2.11x
Ours-VQ-TE VC-TC
3,28
2.87
2.74x
2.34x
2.47x
Ours-VO -Q-C-
2.94x
2.72x
2.66x
2.6xx
2.37x
OCrS-TQQ-TC
3.16x
2.92
2.80x
2.75x
2.74%
2.56x
2,47%
2.49
2.46x
2.27x
Trellis-ooded quantization. TCQ is known as sophisticated Gaussian source quantizer, achieving near-optimal distortion performance [9]. Recently, QTIP introduced optimized CUDA kernels for the bitshift variant of TCQ [36] [50], which encodes each high-dimensional real-valued vector into a binary bitshift representation r:
vii- V (i+ vj LUT (rli s LJ),
where each V-dimensional subvector represented by sliding bit-window of length L, shifted by bits at each step. This encoding achieves effective fractional bitwidth of s/V. While previous TCQ kernel (QTIP) supported integer bitwidths (e.g., 2, 3. bits with shifts s = 4. 6. for V = 2) and were limited to single batch, significantly expand practical applicability by introducing fractional
bitwidth support (e.g., 1.5, 2.5, 3.5, 4.5, 5.0 bits corresponding to shifts = 3, 5, 7, 9, 10 for = 2) and optimized kernels supporting inference at batch sizes up to 8. For constructing the LUT,, follow the protocol of QTIP, which also based on k-means clustering of Gaussian samples.
Furthermore, we introduce half-TCQ, simple extension enabling quantization intermediate fractional bitwidth intervals. Specifically, given a weight matrix row-wise and applies different bitwidth quantization each partition. For example, to achieve 2.75 quantization, half-TCQ quantizes W[: d,,/2 /2] at 2.5 bits and the remaining half W [ddm./ /2 at 3 bits. To preserve computational efficiency. we implement a dedicated CUDA kernel that performs fused dequantization and matrix multiplication for half-TCQ in a single kernel call. As illustrated in Figure TCQ-based schemes, including half-TCQ, consistently achieve quantization error close to theoretical lower bounds, outperforming simpler quantizers. For a more detailed explanation of these quantizers, including quantization algorithms, please refer Appendix C
ddn., half-TCQ partitions the
3.3 Implementation details for efficiency
Reducing rotation overhead. Incoherence processing rotates weights along both input and output dimensions (W + RW R') with per-tensor scaling, yielding approximately standard Gaussian distributions. However, each rotation also requires rotating activations online during inference (e.g., X - XR), incurring significant computational overhead [6 49 50). We reduce this overhead by rotating weights only along the input dimension (W + RW) and applying per-output-channel scaling, normalizing each rotated column to approximately standard Gaussian distribution. Additionally, we share rotation matrices among linear layers with identical inputs (e.g., query/key/value or gate/up projections in Transformer blocks). Combining these techniques reduces the number of online rotations per Transformer block from 14 4.
Kernel implementation. We implement two types of CUDA kernels optimized for different inference scenarios: (i) Tensor Core-based kernels and (ii) CUDA Core-based kernels. Our Tensor Core- based kernels, supporting TCQ, NUQ, and VQ, extend the single batch implementation from QTIP,
5

---

which leverages warp-level mma (matrix (matrix-multiply dequantization logic into this framework involves non-trivial engineering efforts, including the precise mapping of quantized weights to mma instruction fragments. Overall, our implementation extends kernel support from integer bitwidths (2, 3, bits) to fractional bitwidths (1.5, 2.5, 3.5, 4.5, 5.0 bits). To further minimize overhead at larger batch sizes, we traverse each quantized weight exactly once, directly loading and performing register-level dequantization without intermediate storage. Input activations are cached in shared memory for efficient reuse across multiple weight multiplications, significantly improving inference efficiency.
instructions 38 Integrating efficient
Our CUDA Core-based kemels, supporting NUQ and VQ, extend the Any Any-Precision LLM ker- nels [40], originally designed for NUQ. Specifically, we replace bit-plane encoding with simpler bit- bit-packing encoding to simplify the dequantization process. Additionally, we incorporate Any- precision's s table lookup merging technique into both Tensor Core and CUDA Core-based NUQ kernels for bitwidths 2, 3. and 4, further reducing dequantization overhead. Table summarizes the supported kernel implementations and bitwidths for quantization schemes in Q-Palette. For transparency and reproducibility, the full implementations of both kernel types are included in our code release, and additional kernel analysis is provided in Appendix D
Table summarizes the end-to-end decoding-laten -ltenny speedup for LLaMA 3.1 8B models quantized various bitwidths (2.0-4.25 bits) using quantizers in Q-Palette, compared against two baselines: NormalFloat with FLUTE kernels and QTIP [T] 20 [50]. Our quantizers consistently deliver superior inference speed while supporting wide range of finer- finer-grained fractional bit quantization. For smaller batch sizes (batch size = 1), CUDA Core-based kernels typically outperform Tensor Cose-based kernels, whereas Tensor Core- based kernels often provide better latency at larger batch sizes (batch size = 8). Although TCQ incurs slightly higher latency overhead compared to NUQ and VQ. our TCQ quantizers still achieve significantly faster decoding speed compared to baseline quantizers, clearly demonstrating practical efficiency improvements for real-world inference workloads. Note that our TCQ quantizers extend QTIP and, at batch size I, primarily differs by supporting a wider range of bitwidths and reducing the number of online Hadamard transforms, thereby lowering rotation cost and improving decoding-latenn speedup, e.g., from 2.91 x to 3.57 x at 2-bit.
Mixed-scheme quantization with Q-Palette
4.1 Mixed-scheme quantization under resource constraints
The memory-constrained MSQ formulation intro-
duced in Section 2 naturally generalizes to broader resource constraints such as inference latency. Fol- lowing prior one-shot mixed-precision quantization
234
TOQ-2,34
Wiki2 Perplexity ())
TOQ-ALL
frameworks [14 7 35), we formulate resource-
constrained MSQ as MCKP:
I
10
minimize (0,1
Piy big
(3)
1=
5.61
subject to
Pig 1,
v1 VISIS
2.5
3.0
3.5
Bitwidth (L)
L 1 Plg Cly S C,
Figure 3: Performance comparison of memory-constrained MSQ for different quan- tizer sets in Q-Palette on LLaMA 3.1.
[=1 1=1
where the loss term Eip denotes the estimated loss in performance incurred by selecting quantizer Q for layer 1, the cost term Ciq cost(Q Wi) represents the profiled resource cost (e.g., memory or latency), and the total resource constraint is denoted by C. The loss term bg can be instantiated in various ways depending on available information. For example, in data-free settings, we approximate the loss term as by dr err(Q,; WT), as in Equation [1. where az layer-wise sensitivity coefficient computed following the protocol of HIGGS K5 We estimate err(Qu; Wi) using the precomputed distortion of Qg. obtained by quantizing random Gaussian matrix. This enables fast estimation of loss terms in data-free scenarios without fully realizing the quantization pipeline.

---

While a dynamic programming algorithm exists for solving MCKP 43 we simply use the SCIP solver provided by Google OR-Tools for flexibility and ease of implementation 42)
Please refer to
Appendix E for additional details on loss term computation and cost profiling.
We illustrate the effectiveness of Q-Palette in Figure 3 by comparing memory-constrained MSQ results across different quantizer subsets within Q-Palette. Specifically, TCQ-ALL includes all TCQ quantizers available in Q-Palette, while TCQ-2,3,4 and VQ-2,3,4 reflect integer-bitwidth quantizers supported in QTIP and HIGGS, respectively (50] B5J. Although QTIP is originally single-scheme baseline, our constructed integer- subset 7CQ-2,3,4 serves as reasonable reference to evaluate the benefit of broadened quantizer support. The results clearly highlight the advantage of TCQ-ALL, demonstrating that the broadened quantizer support provided by Q-Palette consistently yields superior performance.
4.2 Fusion-aware mixed-scheme quantization
Layer fusion is widely used optimization for accelerating inference speed of DNN models 26 12] Within each Transformer block, certain linear layers, such as (query, key, value) projections or uup, gate) projections, share the same input and can be fused into single linear layer. For example, instead of separately computing XW,, g. XWs, and X we can concatenate the weight matrices and compute X XW, We @ W.), followed by splitting the output (see the right side of Figure Layer fusion can reduce the number of kernel launches and memory accesses, thereby providing further opportunities for inference speedup.
We propose fusion-aware MSQ. a novel MSQ framework that jointly optimizes quantization with the additional design dimension of layer fusion. Fusion-aware MSQ simultaneously determines 1) how to group layers for fusion and 2) which quantizer to assign to each fused group. Whereas standard MSQ (Equation B)) introduces one binary decision variable per (layer, quantizer) pair, fusion-aware MSQ instead defines one binary variable per (fusible layer group. quantizer) pair. Here, a fusible layer group is set of layers sharing the same input. For generic Transformer models, we write the set of all fusible layer groups for each Transformer block b as
Go = {( [ko). {uo), {90. ko {90. vb {ka, 2/ {40, kbg 2b), (oo) (wo). (90), (ub. go {do}..
The overall set of fusible layer groups is G = - Go where B is the number of Transformer blocks. For each 9 E G and quantizer Qu Q, we introduce binary variable Pgg E {0, indicating that all layers in 9 are fused and quantized by Qq-
The fusion-aware MSQ problem is formulated as:
[
minimize (0,1)
Eig
(4)
99
9E9 1=1
iEg
Q
B
subject to
Σ
P =1 1,
V E
U
(go, ks ub, ob, 4b. 96. do).
(C1)
gEGNES q=1
d=1
Q
ΣΣ
Pou Cgg C.
(C2)
DEG q=1
where byg is the loss term, Cgg represents the profiled cost (e.g., latency) of the fused layer correspond- ing the group g quantized by Qq-
To ensure valid solutions, two constraints are imposed. Exclusive assignment (CI): every layer must belong to exactly one active (group, quantizer) pair; among all fusible groups g that contain given layer only one associated variable Pgg can be 1. Resource constraint (C2): the total profiled cost of all activated groups must not exceed the resource budget C.
This formulation explicitly captures latency improvements enabled by fusion, thus providing improved accuracy-latency trade-offs compared to the MSQ formulation that neglects layer fusion (see Figure Since both the objective and constraints are linear Pay- Equation (4) is also an ILP. We solve this IIP using the SCIP solver in OR-Tools [2]. Note that, compared to non non-fusion-awar MSQ (Equation (3), ausion-aware MSQ introduces 1.71 x more decision variables while maintaining the same number of constraints.
7

---

Table Data-free quantization results on LLaMA 3 models for various bitwidths.
aM
a.MMA 3.2-38
Method
Bits
Wiki2
Acc (1)
Bits (1)
Wiki2
Acc
Bits
Wiki2 (1)
Acc (1)
FP16
16.00
5.61
69.3
16.00
8.64
55.9
16.00
6.98
63.7
Data-free QTIP
3,00
6.81
66.9
3.00
13,35
49.0
3.00
8.89
58.2
Ours- Our--TCQ-3
3.00
6.78
66.0
3.00
12.39
50.7
3.00
8.67
60.3
Ours-MSQ-Maan MSQ-Mem
3.00
6.28
67.5
3.00
10.51
53.2
7.81
61.7
3.25
7.70
3.25
17.73
46.8
10.06
59.3
HQQ
3.25
8.29
63.2
3.25
26.42
42.9
3.25
11.68
54.2
HIGGS
3.25
6.64
66.4
3.25
12.19
51.1
3.25
8.67
60.1
O0Q--3QQ-3325
HIGGS-MSQ
3.25
6.39
66.7
3.25
11.08
52.5
3.25
8.01
61.1
Ours Muss-MSQ-Men
3.25
6.10
67.6
3.25
10,00
53.7
3.25
7.60
61.9
NF
4.02
6.22
67.8
4.02
10.70
$2.9
4.02
7.82
62.1
HQQ
4.02
6.52
67.5
4.02
13.47
51.4
4,02
8.67
60.2
HIGGS
4,02
5.98
68.7
4.02
9.64
53.6
4,02
7.46
62.3
Data free QTIP
4,00
5.94
68.4
4,00
9.53
54.9
4,00
7.41
62.8
Ours- TCQ-4
4.00
4.00
9.45
7.37
63.4
HIGGS-MSQ
4.00
5.91
4.00
9.52
55.0
7.40
62.2
Our-MSQQ-Mem
4,00
5.81
69.0
4.00
9.14
55.2
4,00
7.22
63.2
FP16 HQQ NarnePPhed
Nommafles CTTP DireTTD Lu
Narralloot
Datafrre
OTIP
Wik Paplecity
MSQ
NSS
Ous MSQ-Maa
5.61
1500
Dirwidth (1)
Decoding Thrughput (Toksfsed) (?)
Decoding Throughput (Tekshee)
(a) Memory
(b) Throughput (b.s.= 1)
(c) Throughput (b.s.= 8)
Figure 4: Performance trade-offs of quantized LLaMA 3. 3.1-8B models under different constraints the data-free setting on an RTX 4090 GPU: (a) memory constraint; (b) latency constraint (single
batch); (c) throughput evaluation (batch size 8) of the quantized models in (b).
Experiments
We evaluate the quantization performance of our methods against baselines on the LLaMA series (LLaMA 3.1-8B 70B, 3,2-IB, 3B), LLaMA 2 series (LLaMA 2-7B, 13B), and Qwen 2.5-7B
18 48 (5). For the data-free scenario, we consider single-scheme quantization baselines HQQ (uniform), NormalFloat (NUQ), HIGGS-SingL (VQ), and data-free QTIP (TCQ) well as the
MSQ baseline HIGGS-MSQ 2 I 50 35). For the data-aware scenario, we use QTIP as the baseline. For all experiments, both our methods and baselines are evaluated strictly in the PTQ setting, without any retraining. Performance is measured primarily via WikiText2 perplexity and average zcro-shot accuracy across ARC-easy, ARC-challenge, HellaS wagg PiQA, and WinoGrande [37 56 3 44).
For latency evaluations, we use Gemlite kernels for HQQ at higher bitwidths (4.02, 4.25 bits), and
FLUTE kernels for NormalFloat (3.25 4.25 bits) and HQQ at 3.25 bits 22 20 Because QTIP supports only single batch inference, we simulate larger batch sizes by repeated kernel invocation.
We denote Ours-TCQ-x as single-scheme quantization using TCQ-z from Q-Palette, Ours- as menory-connttriieed MSQ (Section 4.1) using Q-Palette's TCQ quantizers, and Murs-MSQ-Lat as latency-constrained fusion-aware MSQ (Section 4.2) using all Q-Palette quantizers with Tensor-Core kernels. Please refer to Appendix G for additional results on other models, ablation studies, and detailed experimental settings.

---

Table 4: Data-aware quantization results on LLaMA 2 models (throughput on an RTX4090 GPU).
LLaMA
LLaMA 13B
Throughput (Toks/s)
Throughput (Toks/s)
Method
Bits
Wiki2 ())
Acc (t)
B
B 8
Bits
Wiki2 (1)
Acc (?)
B B=
B
FP16
16.00
5.12
64.9
71
527
16.00
4.57
67.9
00M
00M
QTIP
2,00
6.84
58.9
209
386
2,00
5.62
63.6
131
154
Our-MMQ--Mem
2,00
6.47
60.3
272
1684
2,00
5.35
64.2
152
928
QTIP
3,00
5,39
63.3
304
3.00
4.76
67.0
110
153
Our--MSQ-Mem
3,00
5,34
63,9
224
1489
3,00
4.74
67.0
126
738
5.1 Data-free quantization results
S data-free quantization performance on LLaMA 3 models. Ours-TCQ-x consis- tently outperforms all single-scheme baselines in WikiText2 perplexity and achieves competitive zero-shot accuracy. Notably, Ours-MSQ surpasses all baseline methods, clearly demonstrating the effectiveness of Q-Palette. Figure 44 evaluates Ours-MSQ-Men across broader memory range (2.25-4.25 bits). Our method achieves -treto-dominant performance, significantly outperforming baseline methods. Remarkably, our 2.875-bit model achieves comparable WikiText2 perplexity to the 3.25-bit HIGGS-MSC model, resulting in 1.13x higher compression ratio and superior perplexity. Figures 4b and 4c compare the buroughput-pepplexity trade-offs of Ours-MSQ-Lai and Ours-TCQ-x against baseline methods. Both variants achieve significant throughput improvements over baseline methods, substantially expanding the Pareto frontier in both batch sizes 1 and 8
5.2 Data-aware quantization results
We further evaluate our methods in the data-aware setting by comparing our Owrs-MSQ-Mem approach against the state-of-the-as QTIP baseline (without retraining) on the LLaMA 2-7B and 13B models. For this setting, we utilize the same proxy Hessian used in QTIP during the quantization and compute the loss term Eig for our MSQ as the actual validation perplexity degradation induced by quantizing the weight Wi using the quantizer Qg. As summarized in Table 4 our method consistently achieves superior perplexity and zero-shot accuracy compared the baseline. Additionally, our optimized kernels achieve over 4x throughput improvements at batch size 8 for both LLaMA 2 models at both 2 and 3 bits, demonstrating the practical benefits of our optimized kernel for batch size 8.
Related works
Incoherence processing, Previous methods for handling outliers in LLM quantization have primarily relied on heuristic techniques 288 54 29]. Recently, a theoretically grounded approach, incoherence processing, has been introduced to systematically address weight irregularities [6]. Incoherence processing applies rotation matrices to weight matrices prior to quantization, significantly suppressing outliers and transforming distributions into approximately Gaussian forms 66 45). This Gaussian-
ization enables the use of sophisticated Gaussian quantizers such as lattice vector quantization [49] and trellis-coded quantization [50]. However, current implementations support efficient kernels only for limited integer bitwidths and small batch sizes, constraining their practicality, a limitation that our proposed Q-Palette directly addresses by introducing fractional-bit quantizers and optimized CUDA kernels with broader batch size support.
More recent rotation-based approaches further enhance quantization performance by applying learned matrix transforms such as scaling or affine transformations B2]
30 23 47. However, these methods mainly target weight-activation quantization and require calibration data to learn the transforms, whereas our work focuses on weight-only PTQ, which remains applicable even in data-free settings and is particularly suited for memory -bound, small-batch inference.
Other weight-only post-training quantization methods. Several simpler PTQ methods prioritize computational and implementation efficiency. HQQ employs data-free uniform quantization via half- quadratic optimization (2]. NormalFloat constructs lookup tables for non-uniform sealar quantization using Gaussian quantiles [T]. FLUTE offers state-of-the-ard CUDA kernels for LUT-based non-

---

uniform quantizers with per per-group scaling 200] Despite their efficiency, these approaches generally incur higher quantization errors compared sophisticated quantizers such as TCQ.
Mixed-precision and mixed-scheme quantization. Mixed-precision quantization (MPQ) optimizes layer-wise bit allocation under given constraints [53]_ For vision models, HAQ and HAWQ-V2
introduced surrogate objectives based on second-order information for MPQ [13] 14. Chen et al. generalized these approaches by explicitly incorporating diverse resource constraints, such as latency, and formulated the problem as an MCKP [7]. Recently, HIGGS introduced the linearity theorem, a data-free linear surrogate specifically tailored for LLM quantization (35), Building upon these works and drawing insights from compiler optimization research [26] 12). we propose novel fusion-awee aware mixed-selle sccemee quantization framework that jointly optimizes quantizer selection and layer fusion decisions, achieving superior accuracy-latenge trade-offs.
7 Conclusion
In this paper, we have investigated weight-only PTQ as a solution for compressing LLMs, particularly beneficial for memory-bound inference tasks with small batch sizes. Considering that irregular weight distributions in LLMs have complicated quantization, we leveraged recent rotation- based methods that Gaussianize weight distributions, enabling theoretical analysis of optimal bit allocation. Based optimal bit allocation strategy under fixed
on this perspective, we derived an information-theoree bit budgets, demonstrating that fine-grained fractional quantizers closely approaching the Gaussian distortion-rate bound are essential for achieving near-optimal quantization efficiency. To translate this theoretical finding into practical benefits, we introduced Q-Palette, a versatile suite of fractional-bit quantizers, from sophisticated trellis-coded quantization schemes offering near-optimal distortion to simpler vector and scalar quantizers optimized for fast inference, each efficiently implemented with optimized CUDA kernels across a wide range of bitwidths. We further integrated Q-Palette into an MSQ framework, proposing novel fusion-aware MSQ approach that jointly optimizes quantizer selection and layer fusion decisions under given resource constraints, effectively improving inference latency. Experimental evaluations validated that MSQ framework with Q-Palette and fusion-aware optimization consistently outperforms existing baseline methods, achieving superior accuracy-memory and accuracy-latency trade-offs on LLaMA 2 and LLaMA 3 models.
Impact statement
Q-Palette introduces a versatile set of quantizers with broad -rawtio support, which can serve foundational building block for evaluating and developing MSQ algorithms. Q-Palette's quantizers are usable in data-free scenarios, offering off-the-shelf applicability like NormalFloat and HQQ [I] 20 21 which lowers the barrier for practitioners lacking calibration data. Importantly, Q- Palette supports a wide spectrum of performance-efficiine trade-offs, enabling practitioners to select quantization configurations that best match their specific deployment workloads. This adaptability is valuable for real-world applications where resource constraints and performance requirements vary significantly. Moreover, our results challenge the misconception that sophisticated quantizers such as TCQ are computationally prohibitive for practical use beyond batch size I [35]. We demonstrate that TCQ achieves efficient decoding speeds for batch sizes up to 8. making practically suitable for edge-device workloads. By correcting this misunderstanding, our work may encourage further investigation into TCQ and other quantizers previously considered computationally expensive.
Acknowledgements and Disclosure of Funding
We would like to thank Jinuk Kim for insightful discussions and helpful feedback on this work. This work was supported by Samsung Electronics Co., Ltd. perience (MX) Business, Samsung Electronics Co., Ltd., Institute of Information & Communica- tions Technology Planning Evaluation (IITP) grant funded by the Korea government (MSIT) [No. RS- 21-2002-11200888, (SW STAR LAB) Development of deployable learning intelligence via self-sustainable and trustworthy machine learning, No. RS-2021-1122 1343, Artificial Intelligence Graduate School Program (Seoul National University)], and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2024-00354036) Hyun Oh Song is the corresponding author.
Mobile eX-
10

---

References
[1] Salch Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated 1lms. In NewrIPS, 2024.
[2] Hicham Badri and Appu Shaji. Half-quadratic quantization of large machine learning models, github http/hqg_blogg
2023. URL
[3] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In AAAI, 2020.
[4] Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
[5] Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. CVPR, 2020.
[6] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. Quip: 2-bit quantization of large language models with guarantees. In NeurlPS, 2023.
[7] Weihan Chen, Peisong Wang, and Jian Cheng. Towards mixed-precision quantization of neural networks via constrained optimization. In ICCV, 2021.
[8] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arxivv1800.00455v1,, 2018.
[9] LMDeploy Contributors. Lmdeploy: A toolkit for compressing, deploying, and serving 1lm. https://githut: com wwwm///neeeiii///m/ee/pollo 2023.
[10] Thomas M. Cover and Joy A. Thomas. Elements of Information Theory 2nd Edition. Wiley- Interscience, 2006.
[11] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized 1lms. In NeurIPS, 2023.
[12] Yaoyao Ding, Ligeng Zhu, Zhihao Jia, Gennady Pekhimenko, and Song Han. los: Inter-operator scheduler for cnn acceleration. In MLSys, 2021.
[13] Zhen Dong, Zhewei Yao, Amir Gholami, Michael Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precision. In ICCV, 2019.
[14] Zhen Dong. Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq-v2: Hessian aware trace-weighted quantization of neural networks. In NeurIPS, 2020.
[15] T.R. Fischer, M.W. Marcellin, and M. Wang. Trellis-coded vector quantization. IEEE Transac- tions on Information Theory, 1991.
[16] G.D. Fomeyy The viterbi algorithm. Proceedings of the IEEE, 1973.
[17] Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. In ICLR, 2023.
[18] Aaron Grattafiori, Abhimanyu Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, and et al. The llama herd of models. In arxXiv2407.21788, 2024.
[19] R.M. Gray and D.L. Neuhoff. Quantization. IEEE Transactions on Information Theory, 1998.
[20] Han Guo, William Brandon, Radostin Cholakov, Jonathan Ragan-Kelley, Eric P. Xing, and Yoon Kim. Fast matrix multiplications for lookup table-quantized 1lms. In EMNLP findings, 2024.
[21] A. Hedayat and W. Wallis. Hadamard Matrices and Their Applications. The Annals of Statistics, 1978.

---

[22] Appu Shaji Hicham Badri. Gemlite: Towards building custom low-bit fused cuda kernels, 2024.
URL httpp::/mobiuumll.giiii
io/ gemlite_blog
[23] Xing Hu, Yuan Cheng, Dawei Yang, Zukang Xu, Zhihang Yuan, Jiangyong Yu, Chen Xu, Zhe Jiang, and Sifan Zhou. Ostquant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting. In ICLR, 2025.
[24] Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Accurate post training quantization with small calibration sets. In ICML, 2021.
[25] Jake Hyun. Log-time k-means clustering for ld data: Novel approaches with proof and implementation. In arXiv:2412.15295, 2024.
[26] Zhihao Jia, Oded Padon, James Thomas, Todd Warszawski, Matei Zaharia, and Alex Aiken. Taso: optimizing deep learning computation with automatic generation graph substitutions. In SOSP, 2019.
[27] Jinuk Kim, Marwa Halabi, Wonpyo Park, Clemens JS Schaefer, Deokjae Lee, Yeonhong Park, Jae W. Lee, and Hyun Oh Song. Guidedquant: Large language model quantization via exploiting end loss guidance. In ICML, 2025.
[28] Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. In ICML, 2024.
[29] Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. Owq: Outlier- aware weight quantization for efficient fine-tur fine-tuning and inference of large language models. In AAAI, 2024.
[30] Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quantized 1lms. In NeurlPS, 2024.
[31] Yujun Lin Haotian Tang*, Shang Yang*, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. Qserve: W4a8ky4 quantization and system co-design for efficient 11m serving. In MLSys, 2025.
[32] Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: LLM quantization with learned rotations. In The Thirteenth International Conference on Learning Representations, 2025.
[33] llama.cpp Contributors. llama.cpp. https://githun
com www.hul..oooogggmmmge. aogglooorgllama, cpp 2023.
[34] Stuart Lloyd. Least squares quantization in pem. IEEE transactions information theory, 1982.
[35] Vladimir Malinovskii, Andrei Panferov, Ivan Ilin, Han Guo, Peter Richtárik, and Dan Alistarh. Higgs: Pushing the limits of large language model quantization via the linearity theorem. In ACL, 2025.
[36] Mark Mao and Robert Gray. Stationary and trellis encoding for iid sources and simulation. Data Compression Conference Proceedings, 2010.
[37] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 160.v 07843, 2016.
[38] NVIDIA Corporation. Nvidia tensor cores. programming teuorio----- 9/ 2018
hnttps://developer: http://eeopppe.iiiiiia..ooooooo combblog//
[39] OpenAl. Gpt-4 technical report. In arXiv:2303.00774 2023.
[40] Yeonhong Park, Jake Hyun, SangLyul Cho, Bonggeun Sim, and Jae Lee. Any-precision lmm: Low-cost deployment of multiple, different-sized 1lms. In ICML, 2024.
12

---

[41] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer, R. Weiss, V. Dubourg. J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and F. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 2011.
[42] Laurent Perron and Vincent Furnon. Or-tools. URL httpss://deveeloprss.ooot com/ optimization/
[43] Ulrich Pferschy and Rosario Scatamacchia. Improved dynamic programming and approximation results for the knapsack problem with setups: ru. pferschy and r. scatamacchia. International Trunsactions in Operational Research, 2017.
[44] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In arxiiv:907.10644 2019.
[45] Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. NeurlPS, 2024.
[46] Prabhakant Sinha and Andris A. Zoltners. The multiple-choice knapsack problem. Operations Research, 1979.
[47] Yuxuan Sun, Ruikang Liu, Haoli Bai, Han Bao, Kang Zhao, Yuening Li, Jiaxin Hu, Xianzhi Yu, Lu Hou, Chun Yuan, et al. Flatquant: Flatness matters for IIm quantization. In ICML, 2025.
[48] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, and et Llama Open foundation and fine-tuned chat models. In 2.X.v:22307.09288, 2023.
[49] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quipii: Even better IIm quantization with hadamard incoherence and lattice codebooks. ICML, 2024.
[50] Albert Tseng, Qingyao Sun, David Hou, and Christopher De Sa. Qtip: Quantization with trellises and incoherence processing. In NeurlPS, 2024.
[51] Mart Baalen, Andrey Kuzmin, Markus Nagel, Peter Couperus, Cedric Bastoul, Eric Mahurin, Tijmen Blankevoort, and Paul Whatmough. Gptvq: The blessing of dimensionality for Ulm quantization. In x.xxi:24022. 15319, 2024.
[52] Maurice Weber, Daniel Y. Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexan- drov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. Redpajama: an open dataset for training large language models. NeurIPS Datasets and Benchmarks Track, 2024.
[53] Bichen Wu, Yanghan Wang, Peizhao Zhang, Yuandong Tian, Peter Vajda, and Kurt Keutzer. Mixed precision quantization of convnets via differentiable neural architecture search. In 1rXiv:1812.00000, 2018.
[54] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In ICML, 2023.
[55] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, and et al. Qwen2.5 technical report. In arXiv:2412.15115, 2025.
[56] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In ACL, 2019.
13

---

A Optimal bitwidth proof
In this section, we formally derive the optimal bit allocation result stated in the main paper. Under the assumption that weight matrices have been Gaussianized through incoherence processing, the quantization problem can be viewed as a Gaussian source coding problem. We recall the memory- constrained mixed-scheme quantization (MSQ) formulation as:
minimize (0,1(
Σ
aj Σ
Pig ear (2g: Wi)
1Q1
subject to
Pig
1,
v1 VI
q=1
Pig bit(Qg; Wi)dF dith s M,
where aj the empirically estimated sensitivity coefficient for layer 1, err Q; Wi) is the normalized quantization error of layer Qg denotes candidate quantizer, bit(Q,; Wi) is the average number of bits per weight component for the weight matrix Wi quantized by Qg. Plg E {0, indicator selecting quantizer Q2 for layer I, and M denotes the total memory budget (in bits) allocated
binary
for quantized model 46 35
We recall that classical rate-distortion theory provides a fundamental lower bound on the expected quantization error for Gaussian sources: [[err(Q))] 2 2 [TO]. Further assuming we have access to ideal Gaussian quantizers capable of exactly achieving this theoretical distortion bound [(err(Q)] 2- (Q) at any fractional bitwidth by 2 7) the memory-constrained MSQ (problem
can again written as the continuous optimization problem:
minimize 2
0,2
subject to
Σ
bid ypur S M,
where by is the fractional bitwidth allocated to layer I, and 7) > 0 is minimum bitwidth threshold introduced avoid degenerate cases such as assigning 0-bit to layer. Here, we replace the actual quantization error err(Q) with its expectation [[err(())]. We empirically justify this replacement by demonstrating extremely low variance quantization errors for typical weight matrix dimensions encountered in LLMs (see Table Additionally, assume that the sensitivity coefficients a non non-negative (a: 2 0), a reasonable assumption given that pretrained weights typically represent local optima. Given this simplified optimization problem, now derive the closed-for fosed-form solution for the optimal fractional bit allocation.
Theorem 3.1 (Optimal bit allocation with ideal Gaussian quantizers). If the budget M is feasible, i.e., M 2 17 Ele dite then the optimal fractional bit allocation {bς for problem is givee by
at din
+c].
b7 = max 17. 2 (2))
In
VISISL,
for the constant C' that satisfies the memory constraint Σ, b7 df = M.
Proof. Let's start by formulating the Lagrangian function for problem 2 explicitly including the constraint by ny via Lagrange multipliers μr and the budget constraint via À 2
Clbi,,,, (m+)) = Σ a,2 2h + A
bidj du
M
Σ 1m1
pi(bi - 11)-
14

---

Table Empirical distortion statisties for quantizing random Gaussian matrices (4096 4096) with Q-Palette quantizers over 32 trials.
Quantizer
Mean distortion
Std. deviation
Ours-TCQ-2
0.07101 0.11747
8.36E-06
Ours-NUQ-2
4.24E-05
Ours-VQ2 VQ-2
0.10857
2.93E-05
By differentiating the Lagrangian with respect to by and setting it equal zero find the stationary points, we have:
OC bb,
-2 (2) a:2 Adj apnt 141 = O.
Since. for all L, p4 2 0 and complementary slackness requires pb - -n) we have two cases:
Case 1: If by 7), complementary slackness implies μ and thus:
Ad aput
2 2-26
l2(2))
Taking logarithms on both sides and rearranging terms explicitly, obtain:
1
(in
br 2 (2)
In din duu
+ 2ln(2) (In(2In(2)) In())) = 21n(2)
djt C(A).
where, the constant term C()) defined as C(A) 21n(2) (In(2In(2)) In())).
Case 2: If 17, we directly have:
7
Combining these cases yields the optimal fractional allocation:
b7 max n 2 2ln(2)
In
din
at
+
where the constant C(A) is chosen such that the memory constraint
L
Σ
bi di dout < M,
tight (i.e., equality holds). This equality condition emerges naturally, as the objective function non-increasing in by due to the non-negativity assumption (a; 2 0). Therefore, increasing C(A) until the constraint exactly met cannot worsen the objective, completing the proof.
To empirically validate our approximation of quantization errors by their expectation, quantized random Gaussian matrices multiple times and observed consistently low variance in the quanti- zataion emrors (distortion). Specifically, we performed quantization 32 random standard Gaussian matrices of shape (4096, 4096), consistent with the LLaMA 3. -8B self-attention query projec- tion layer. Table 5 reports the mean and standard deviation of the normalized quantization error <IIW>
W W
values. The results demonstrate low variance, supporting our assumption.
B Analysis of the quantization optimality gap
In practice, we typically have access only to finite set of quantizers Q = (Q1.... Qx }. which may not achieve the theoretical distortion-rate optimality. Under this constraint, the original memory- constrained MSQ problem (1) can still be solved, but the resulting solution may deviate from the optimal solution derived under the assumption of ideal Gaussian quantizers (Theorem 3.T this section, we formally analyze this quantization optimality gap.
15

---

Table Optimality gap analysis for different quantizer sets on LLaMA 3.1-8B. TCQ-ALL includes
all fractional TCQ bitwidths from 1.5 to 5.0 in Q-Palette.
Quantizer pool
Bitwidt
Distortion gap (4)
Bit allocation (4)
Total gap (J)
Surrogute objective (L)
VQ-2.3,4
3.25
0.0586
0.0130
0.0716
0.1219
TCQ-2,3,4
3.25
0.0198
0.0129
0.0327
0.0830
TCQ-ALL Idcal Gaussian quantizer
3.25
0.0145
0.0023
0.0168
0.0671
3.25
0
0.0503
VQ-2,3,4
2.50
0.1282
0.0178
0.1460
0,2883
TCQ-2,3,4
2.50
0.0306
0.0178
0.0484
0.1907
TCQ-ALL
2.50
0,0260
0.0034
0.0294
0.1717
Ideal Gaussian quantizer
2.50
0
0.1423
Let Of denote the optimal quantizer selected from the quantizer set Q for each I, obtained by solving problem (T). Then, the quantization optimality gap, defined as the difference in the objective values between the practical optimal solution and the theoretically optimal fractional bitwidth solution by
(derived in Theorem 3.1 can be expressed as:
Σ aj err(Qi: Wi) 22-20)
(5)
We decompose the total gap into intuitive terms, the distortion gap and the bit allocation gap as follows:
eer((Qu Q1) - 2--26
Σ at eer((1)) 2222b1((2)) 1=
(Q7) -2 2--2%
at
Total gap
Distartion gap
allocation gap
where abbreviate err(Q): Wi) by err(Q1) and bit(Q?: Wi) by bit(Q?) for the simplicity.
Due to classical rate-distortion theory and the optimality bj as the solution the continuous optimization problem (2), each term in this decomposition is non-negative. Specifically, the first term, (err(Q)) - 2bbi quantifies how closely each practical quantizer Qi approaches the theoretical Gaussian distortion bound. The second term, (2 (2))bi(())) 2 2-26;) measures how well the available bitwidths (bi Q E Q} approximate the optimal bit allocation {b}}-
To investigate how quantizer-set design affects each component of the gap, Table 6 reports the distortion and bit allocation gaps for L.LaMA 3.1-8B under 2.5- and 3.25-bit constraints. Two key observations emerge:
Effeet of quantizer quality. VQ-2,3,4 and TCQ-2,3,4 show comparable bit allocation gaps, but TCQ-2,3,4 yields much smaller distortion gaps. Thus, their performance difference mainly stems from quantizer quality rather than bit allocation.
Effect of broader bitwidth support. Comparing TCQ-2,3,4 to TCQ-ALL reveals sub- stantial reduction in bit allocation gap, demonstrating that richer fractional- sactional-bitwidthh bitwidth support enables more accurate bit allocation and closer match to the theoretical ideal.
This analysis motivates the design of Q-Palette, which provides high-quality TCQ quantizers and broad paetiooal-bitwidde coverage to reduce both distortion and bit allocation gaps.
Analyzing these factors provides insight into improving practical quantizer designs and selecting more effective quantizer sets to reduce the quantization optimality gap. Motivated by this analysis, we specifically designed O-Palette as versatile set of fractional-bir quantizers, including TCQ, which closely approaches the theoretical distortion bound, and providing broad fractional bitwidth support.
C Additional details on quantizers in Q-Palette
In this section, we provide implementation details for each quantizer family in Q-Palette. For each quantizer, we describe: (i) codebook construction, (ii) dequantization, and (iii quantization
16

---

procedures. The quantization step relies on quantizer-specifee round-to-neared (RTN) operators, with procedures differing based on data availability:
Data-free scenario: We partition each weight matrix into scalar elements (NUQ) or vec- tors (VQ. TCQ). Each partition is independently quantized using the RTN operator, and their resulting binary representations are concatenated to form the final quantized weight representation.
Data-aware scenario: We adopt block LDLQ framework as introduced in previous methods [17 49 50). Specifically, for each weight matrix, we perform quantization block-wise manner guided by Hessian approximations, with quantizer-specifie block sizes: 1 for NUQ, for VQ, and 16 for TCQ. This method sequentially processes weight rows from first to last, iteratively updating weights based on the Hessian and cumulative quantization errors, and quantizing each updated weight via the RTN operator. For detailed formulations and additional theoretical background, please refer to QUIP# and QTIP 49 [50]
C.1 Non-uniform scalar quantization (NUQ)
Codebook construction. For NUQ at bitwidth b, we construct the LUT using flash 1 dkmeans, a fast ID k-means algorithm [25]. applied to 10 randomly sampled standard Gaussian values. We set the number of clusters to = resulting in a LUT I
Dequantization. Given the LUT, binary representation {0, 1) is dequantized as:
Vq(r;LUT) LUT(int(r)),
where ir) converts the binary representation {00, 13 its corresponding integer index in the range [0, 2 1].
Quantization. NUQ's RTN operator RTN LUT entry:
IR (0. 1) maps scalar input IR the nearest
RTN(;: LUT): argmin |0 LUT(int(r)]]-
(
Quantization follows the general procedures described above for data-free and data-aware scenarios.
C.2 Vector quantization (VQ)
Codebook construction. For VQ at bitwidth 6, we construct the codebook using the scikit-learn implementation of the 2D k-means algorithm, which employs Lloyd's algorithm [4] (4]. We the hyperparameters to max maa_iite=300 and tol=1e-6, 6, and apply the algorithm to random standard Gaussian samples, using 10 samples for bitwidths < and 10" samples for bitwidths b > 5. The
number of clusters is set to = 22h resulting in a LUT e 2 consisting of 2 number of 2D vectors.
Dequantization.
Given the LUT, binary representation r E {0, 1]26 is dequantized similarly to NUQ, now mapping vector:
dq(r; LUT) LUT((nt(rr] R²,
where (r) converts the binary representation r E { 1325 into its corresponding integer index in the range [0, 22b
Quantization.
The RTN operator specific VQ. RIN R2 [0, 1).h maps 2D input vector E IR2 to the nearest LUT entry:
RTN(v; LUT) = argmin ||v LUT[int(r)]ll-
re (0.1)
Quantization then follows the general procedures described above for data-free and data-aware scenarios.
17

---

C.3 Trellis-coded quantization (TCQ)
C.3.1 Generic TCQ
Codebook construction. We follow the same protocol as QTIP 150, using scikit-learn's 's k-means
implementation based on Lloyd's algorithm [4] 34 Specifically, we cluster 2 randomly sampled 2D standard Gaussian vectors (with appropriate scaling) into 2 clusters, obtaining cluster centroids tlut & T codebook construction using the following quantlut_s) function from the QTIP codebase:
x2 We then construct the final codebook LUT E R22 x2 using the hybrid
def quantlut_eym L, tlut_bits):
with ttoch..raaaae arange L, device=tlut.devics.
lut (lut 1) * lut
sflp
((lut 15) 1)
lut (lut >> (16 tlut_bits 1)) & ((1 << tlut_bits) 1)
lut tlut [lut]
lut 0] = lutl:, 0] sflp
return lut
Following QTIP, we set L = 16 for all TCQ quantizers. We set tlut_biisi to 9 for bitwidths S 4. and to 10, for new fractional bitwidths 4.5, 5.0, respectively.
We adopt the bitshift variant TCQ with tail-biting from QTIP Given a binary representation E (0, 1)=/VV we define parameters explicitly as follows:
Dequantization.
8: shift size, set as s = 26 for bitwidth b.
V: vector size, fixed V
L: codebook length (or sliding window size), fixed to L = 16,
trellis size, set as T 256.
The dequantization then proceeds via sliding-window window LUT indexing with tail-biting:
dq(r; LUT) concat T/V-1 LUT [rli : + L.] E R.,,
where indices exceeding the length of wrap around due to tail-biting, resulting in s/V bitwidth.
Quantization. Given target vector R quantize, we use the same RTN operator as detailed in QTIP, which leverages the Viterbi algorithm to find the optimal binary representation r E {0, 10 that is dequantized into the vector = dq(r; LUT) closest to the vector v [66] [50] Quantization procedures follow the general data-free and data-aware frameworks described earlier.
C.3.2 Half-TCQ
Codebook construction. For half-TCQ, which quantizes half the weight using bitwidth and the other half using b + 0.5, we follow exactly the same codebook construction procedure described above for TCQ at bitwidth b ++00 0.5.
Dequantization. Dequantization separately processes two partitions of the weight matrix: the first half using binary representations of bitwidth b, and the second half using bitwidth b + 0.5. The resulting vectors from each half are then concatenated into complete dequantized weight vector.
Quantization. We apply the RTN operator corresponding TCQ-b the first half of the weights, and the RTN operator corresponding to TCQ-(b + 0.5) to the second half. This procedure is consistently used in both the data-free and data-aware (block LDLQ) scenarios.
D Additional details and performance analysis of CUDA kernels
We implemented two types of CUDA kernels: (i) Tensor Core-based kernels and (ii) CUDA Core- based kernels. Here, we first detail our Tensor Core-based kernel implementations. Then, describe
18

---

Table 7: Decoding -aateny speedup of quantized LLaMA 3.1-8B models relative to the FPI6 baseline on an RTX3090 GPU. 'TC' and 'CC' denote Tensor Core and CUDA Core kernlss, respectively.
Decoding-latenge speedup compared to FP16 (batch size 1)
Quantizer
2.0
2.25
2.5
2.75
3.0
3.25
3.5
3.75
4.0
4.25
NF FLUTE 11 20
1.63x
1.63x
QTIP 50
2.17x
2.02x
1.92
Ours-NUO-TC
2.82x
2.53x
2.28
Our--NUQ-CC
3.07x
2.75
2.38 x
Ours VQ-TC
2.83x
2.54x
2.54x
2.10x
2.28x
Ours-VO VQ-CC
3.11x
2.74x
2.52x
2.36x
Ours-TCQ-TC
2.56x
2.32x
2.29x
2.24x
2.36X
2.13x
2.14x
2.16x
2.25x
1.94x
Decoding latency speedup compared to FP16 (batch size 8)
Quantizer
2.0
2.25
2.5
2.75
3.0
3.25
3.5
3.75
4.0
4.25
NF FLUTE 11 20
1.55x
1.55x
QTIP 550
0.43x
0.39
0.36x
Ours-NUO-TC
2.20
2.00
1.85
Ours-NUQ-CC
1.75x
1.70x
1.49
Ours-VQL VQ-TC
2.20x
1.99
1.97x
1.65x
1.84x
Ours-VQ-CE
1.88x
1.81
1.78x
1.69x
1.69
OCrS-TQQ-TC
2.04x
x.89
1.84%
1.88
1.93%
1.79
1.72x
1.83x
1.89
1.67x
CUDA Core-based kernel implementations. Finally, provide additional performance analysis on our kernels.
D.1 Tensor Core-based kernel implementation
Our Tensor Core-based kernels support various quantization schemes (TCQ, NUQ, and VQ) and implemented by extending the QTIP kernels, which originally supported TCQ at integer bitwidths (2, 3, 4 bits) [50] Specifically, for TCQ, we introduce optimized support for fractional bitwidths at fine-grained intervals (e.g.. 1.5, 2.5, 3.5, 4.5, 5.0 bits) by carefully extending the warp-level mma instruction-based implementation provided by QTIP. Additionally, we adapted the QTIP's kernel design principles to implement efficient Tensor Core-based kernels for NUQ and VQ. These extensions involved non-trivial engincering efforts, particularly for precisely mapping quantized weights into Tensor Core mma instruction fragments. To further reduce overhead at larger batch sizes, we traverse each quantized weight exactly once, directly performing register-level dequantization upon loading without intermediate storage. Input activations are cached in shared memory to enable efficient reuse across multiple weight multiplications, substantially improving inference efficiency.
Simplified Kernel Structure. Below, we provide brief kernel structure highlighting key functions, their purposes, and file locations:
// kernels/tcq /eerme/lsttoeekeemel/////////rre wwt.elaa/arciinnerreeee..
device void load_reg_cst caogg_css><<cooo (opreee idx, laneld, &regs) { // Maps quantized TCQ weights (bitwidth R/2) fragments
// Supports fractional bitwidths
}
// /aemrpt//ggttaaooeeeeeeaaaa//rrr
device void load reg_ ca<R, LUT_TYPE TYPE L> L>Cooppressed, idx, laneld, &regs) (L LUT :
// Maps quantized NUQ weights (bitwidth R) to mma fragments else if (L JUT_TYPE:: VQ_LUT_2)
// Maps quantized vq weights (bitwidth R/2) to mma fragments
}
// /Gnerrll Tensor Core kernel structure
// Bernel/s/tag-eemel//rrrddr
kermei_decompress.
19

---

1/ for TCQ fused kernel
// sermel/ssta-kemmllsssssdddo www.la//rrr/iinorreceeeeeee cu:
// kernels/tcq
cu: kernel_@ecomprressgemmmcco
//
for TCQ-Half fused kernel
cu: kernel, sernel_decompress_eee
// for VQ and NUQ fused kernel
g global void vensor_core_tee (orre_errel(..)) {
// Manages LUT and imputs in shared memory
// Manages quantized weights in registers
// Maps quantized weights to Tensor Core fragments via 'load_reg_ca
// Uses Tensor Core mma instructions for matmul routine
// Writes final results after reduction
}
// Dequ-ottization-onlly kernels
//
deroep_decompeess
// for TCQ dequantization kernel
// Neemee///tcg-eeeeee///////eeeeee
kernel_
// for TCQ-Half dequantization kernel
//
cu: kernel_decompres // for VQ and NUQ dequantization kernel
global void kernel_ weweeedddccoppesss...
Dequantizes weights independently (no matmul)
}
Full Implementation. The complete Tensor Core-based kernel implementations are included in our public code release (directories kernels sernels/tcq-kerneel. and kernels/vq-tensee http://vvttetrreeeeeeeeeee -ernees)).
D.2
CUDA Core-based kernel implementation
Our CUDA Core-based kernels explicitly leverage CUDA Core instructions support NUQ and VQ quantization schemes, extending the Any-Precision LLM kernels originally developed NUQ [40]. Specifically, we replaced the original bit-plane encoding with simpler bit-packing encoding to streamline the dequantization procedure.
Kernel Structure and Implementation Below, we provide brief kernel structure highlighting key functions, their purposes, and file locations:
//
device void _o_pack__equant.cmm dequunteuuutouooeeeeee cogpuaattroutiinecbiittss vec_sz> (Bcode, B_row, shC) // Unpack quantized VQ weights 'Bcode' (bitwidth nbits/vec_sz) \
// to the half2 array 'B_row' using the lookup-table 'shC'
// (e.g., 1.5-bit quantization: nbits=3, vec_s2-2)
}
// kernelss/a--cuaa-kennell/// device void pack decckddettattntbit>> (Bcode_roodo B_row, shC) { // Unpack quantized NUQ weights 'Bcode" (bitwidth nbits) \
// to the half2 array 'B_row' using the lookup-table 'shC'
// General CUDA Core kernel structure
// https//s-cuuaa--eeeeeeee paarness/eemmrouttine cu: eq-gemm_fple
// for NUQ fused kernel
// htw.://///ooo.aoaaaaa/////////ee
vq_pack.genm_Epll
//
for VQ fused kernel
gl global void cuda, core kernel(...) {
// Manages LUT in shared memory
20

---

// Manages quantized weights in registers
Unpacks quantized weights half2 array via corresponding
// back_dequant, eeauuantrroutieeee
// Uses CUDA Core half-precision FMA (hfma2) instructions for matmul
// Writes final results after reduction
}
// Dequantiization-only kernels
// berneets/gggcccaaaeeooooo//o.aa
cu: pack__eeuant_
// for NUQ dequantizattin kernel
//
//
vq_pack, vq_pack,dequant, dequant sawtantt_bbit.o store
// for VQ dequantization kernel
global void permel_decompeess(..
// Dequantizes weights independently (no matmul)
Full Implementation. The complete CUDA Core-based kernel implementation is included in our public code release (directories kernels/sq-cue cudaas//ss-uaa--aaeaeaaaaaa--aa- kernel and kernels keraes..
D.3 Additional performance analysis
This section complements Table in the main paper by providing additional performance analysis on a different hardware configuration (with RTX3090 GPU). Table 7 summarizes the decoding-latente speedup quantized LLaMA 3. 3.B models relative to the FPI6 baseline on an RTX3090 GPU, complementing Table 2 in the main paper, which reports results consistently outperform baseline methods across both evaluated batch sizes (1 and 8).
RTX4090 GPU. Our quantizers
Notably, for batch size 1 on RTX3090, the latency speedup gap between our CUDA Core-based ('CC) and Tensor Core-based ('TC") kernels larger than observed on RTX4090 (Table This difference highlights significant hardware dependencies in kernel performance, justifying the importance of providing multiple kernel implementations. Such flexibility enables optimal kernel selection tailored to specific hardware platforms and workload requirements.
E Implementation details of mixed-scheme quantization
E.1 Loss term computation
Data-free loss term. In data-free scenarios, we estimate the loss term via the linearity theorem 35 i.e,, by aj crr(Qq; Wi). Specifically, approximate the quantization pre-computed distortion values obtained by quantizing random standard Gaussian matrices, thereby avoiding explicit quantization for each weight matrix Wt. To determine the sensitivity coefficient as we adopt the procedure introduced in HIGGS (35]. First, we randomly generate 128K tokens from the given LLM. Then, for each layer we inject random Gaussian noise scaled to specific norms vi for 1 < i < 16 and measure the resulting increase in the KL-divergence loss computed
err(Qq: Wi) using
nu 16
over these 128K tokens:
AC#: ([Wr + 8p1- nu ||Wi]| (allaee///lllllaaaaaa C ((Wr)))))) El (O,,
where 8p1 the Kronecker delta (1 if = I, else O). ensuring noise is injected exclusively into layer 1, and e standard Gaussian noise matrix matching the dimensions of layer 1. Due to the linearity theorem, the increase in loss approximately follows the linear relation ALn: = nf,ar, enabling us to estimate az by linearly fitting the data points (mī, ALu). This procedure requires 16 L computations of the KL-divergence loss, where L is the total number of layers, but it can be performed in an embarrassingly parallel manner. Furthermore, the computed sensitivity coefficients aj can be reused for all data-free MSQ scenarios, incurring only one-time computational cost.
Data-aware loss terms. For data-aware scenarios, we employ two different types of loss terms: linearity and actual.
21

---

Linearity-based loss term. Similar to the data-free scenario, we utilize the linearity theorem-baa based approximation. However, we replace the KL-divergence loss computed over randomly generated 128K tokens with the perplexity loss computed over 1M tokens from the RedPajama dataset [52].
Actual loss term. Here, we explicitly calculate the actual perplexity increase caused by quantization. Specifically, we use 256K tokens from the RedPajama dataset and compute the loss term follows:
Bgg L(1Q (Wr)) replace 1-th layer with Q₂(Wi) LoaQaafatt (Wr))),
where Qa denotes the default quantizer for layer (e.g., we use TCQ-2 for 2-bit quantization, TCQ-3 for 3-bit quantization). This explicitly measures the actual empirical perplexity increase caused by quantize layer with a quantizer Qq.
For the data-awa Lata-aware experiments in Table of the main paper, we exclusively employ the 'actual' loss term. Additionally, in the ablation study provided in Table (see Appendix G we explicitly compare the performance using the two types of loss terms, 'actual' and 'linearity"
E.2 Latency profiling
For latency profiling, we measure the execution time of the CUDA kernels corresponding to each quantizer. Since computations such normalization layers, rotations, and self-attention operations remain identical across quantizers, we specifically profile the latency of each fused dequantization and matrix multiplication kernel. To accurately estimate the overall inference latency, we first measure the end-to-end inference lateney of several randomly selected quantization configurations using torch. compile. We then subtract the sum of kernel latencies estimate the latency overhead caused by common computations (e.g., normalization layers), uniformly distributing this overhead across all quantizer profiles.
When considering fused kernels, we separately measure kernel latencies corresponding to each fusion pattern. To account for latency overhead variations due to different fusion patterns, we adjust this overhead bias accordingly.
E.3 Optimizer
To solve the mixed-scheme quantization optimization problem, employ Google's OR-Tools optimization suite [42] specifically utilizing the SCIP solver with a time limit of 60 seconds.
F Settings for Figure!
In Figure we present qualitative comparisons among quantization frameworks based on Q-Palette and the NormalFloat baseline with FLUTE kernels M 20], evaluated on the LLaMA 3.I-8B model using an RTX4090 GPU at a batch size of 1. Specifically, we validate WikiText2 perplexity at sequence length of 8192 and measure the inference speedup compared to the FP16 baseline under the following detailed settings:
NormalFloat (Baseline). For NormalFloat, we employ FLUTE with codebook size of 2 and group size of 64, resulting in an average bitwidth of 3.25. We utilize the FLUTE kernels released in the FLUTE codebase [11, (20), optimized for inference on an RTX4090 GPU, to measure inference latency and compute the speedup.
Single-scheme quantization with TCQ-3.25 (Ours). We apply data-free quantization uniformly to all layers of the LLaMA 3.1-8B model using our TCQ-3.25 quantizer from Q-Palette. This corresponds to the half-TCQ scheme which quantizes half of the weight matrix at bitwidth 3.0 and the other half at 3.5.
MSQ with Q-Palette (Ours). We leverage the full set of quantizers available in Q-Palette as our search space. For NUQ and VQ quantizers, both Tensor-Core and CUDA-Core kernel implemen- tations are considered during optimization. Sensitivity coefficients a are computed following the
22

---

HIGGS protocol [35] detailed in Appendix and we utilize pre-computed distortion values as explained in Section 4.1 of the main paper. Given these sensitivity coefficients, distortion values, and pre-profiled latency measurements, we solve the atency-constrained MSQ optimization (Fquation (3) the main paper) to identify Pareto-optimal quantizer selections under various latency constraints. From this resulting accuracy-latency trade-off curve, we select the configuration that clearly improves both latency and perplexity over the TCQ-3.25 baseline.
Fusion-aware MSQ with Q-Palette (Ours). We further incorporate layer fusion into our MSQ formulation by additionally profiling latency measurements fused linear-layer combinations and solving the fusion-aware optimization (Fquation (4) in the main paper). This approach explicitly captures the latency reductions achievable via layer fusion, enabling joint optimization of quantization schemes and layer fusion decisions. We select quantization configuration that clearly improves both inference speed and perplexity compared to the MSQ baseline without layer fusion.
The detailed quantization configurations correspond to these scenarios are visualized in Figure 1.
G Experimental settings and additional results
G.1 Experimental settings
G.1.1 Evaluation metric details
For evaluating language modeling performance, we measure perplexity on the WikiText2 dataset [3]] using sequence lengths of 4096 tokens for LLaMA models and 8192 tokens for LLaMA 3 models. Additionally, we report zero-shot accuracy on five downstream tasks: ARC-easy, ARC-challenge. HellaSwag, PiQA, and WinoGrande 37 56
Zero-shot evaluations are conducted using the
1m_ evvlll library (version 0.4.4).
G.1.2 Device details
RTX 4090 GPU experiments. We conduct our RTX 4090 GPU experiments using a cloud environ- ment provided by RunPod, with the following hardware and software specifications:
GPU: NVIDIA RTX 4090
CPU: AMD EPYC 7B13 64-Core Processor
OS: Ubuntu 22.04.5
CUDA Version: 12.4
RTX 3090 GPU experiments. machine, detailed as follows:
We conduct our RTX 3090 GPU experiments using our local
GPU: NVIDIA RTX 3090
CPU: AMD EPYC 7402 24-Core Processor
OS: Ubuntu 22.04.1
CUDA Version: 12.4
G.1.3
Baseline configurations
HQQ (2].
According the official documentation, inference acceleration kernels (e.g., Gemlite) are supported only for configurations with axis=1. Thus, we use the following configurations:
4.25-bit: nbits=4, group_size=64, axis=1 (Gemlite kernel),
4.02-bit: nbits=4, group_size-1024, axis=1 (Gemlite kernel).
b.t5-bit: mbits=3, group_size=64, axis=1 (FLUTE kemel).
For 4-bit instances, we utilize Gemlite kernels following best practices from the HQQ documentation 22 2); for the 3-bit instance, we report inference time using the FLUTE kernel [20].
23

---

6.60
3.70
HP16
HP16
Ours-MS(--Mem
3.50
MSS--MM
6.50
HIGOS-MSQ
HIGGS-MSQ
Wiki Perplexity (J)
Wiki2 Perplexity
3.30
6.40
3.10
6.30
2.90
6.20
2.70
6.10
2.50
3.25
3.50
3.75
4.00
4.25
3,25
3.5
3.75
4.00
4.25
Bitwidth (1)
Bitwidth (1)
(a) Qwen 2.5-7B
(b) LLaMA 3.1-70B
Figure 5: Mixed-scheme quantization results Qwen 2.5-7B and LLaMA 3. 3.1-70B models. To accommodate the broader sensitivity range in LLaMA 3.1-70B, we extended the quantizer set to include higher-bitwidth options (NUQ 7/8 bits and VQ 5.5/6 bits), in addition to the TCQ quantizers.
NormalFloat 11
We utilize FLUTE's NormalFloat implementation with configurations similar to
HQQ 20 2
4.25-bit: nbits=4, group_size=64,
4.02-bit: nbits=4, group_size=1024,
b.t5-bit: nbits=3, group_size=64.
Since the publicly available optimized FLUTE kernel does not support group size 1024, we only report inference latency results for the 4.25-bit and 3.25-bit configurations.
For data-free QTIP, we approximate the Hessian as the identity matrix and follow the same algorithmic implementation the original data-aware QTIP. For data-aware QTIP, the publicly available Hessian approximation from the relax-m1 HuggingFace repository, computed using 6144 4096 tokens.
QTIP [50].
As the implementation of HIGGS is not publicly available, we directly report results from their paper. Specifically, for the mixed-scheme baseline, HIGGS provides only a single result for each bitwidth, which we directly use in our comparison. For the single-scheme VQ baseline, HIGGS reports multiple configurations for each bitwidth; we select the configuration achieving the lowest WikiText2 perplexity. Although these single-schens VQ configurations may not be efficiently realizable in practice due non-powe non-power-of-tee codebook sizes, we include them for completeness and comparison purposes.
HIGGS 35
G.2 Additional results
G.2.1 Mentry-constrained mixed scheme quantization results on additional models
To demonstrate the generality of our method, we applied our MSQ method to Qwen 2.5-7B (non- LLaMA) and LLaMA 3.1-70B (large-scale), comparing against HIGGS-MSQ under various bitwidth constraints [55
18 35). Here, for LLaMA 3.BB7OB, due to its substantially larger model size, we used 64K tokens (i.e., half of the default setting explained in Appendix E) to estimate the sensitivity coefficients while keeping the rest the quantization pipeline unchanged.
As shown in Figure 5 our method consistently outperforms HIGGS-MSO under the same bitwidth constraints (3.25, 4.00, 4.25) on both models. Additionally, our method achieves comparable or better perplexity at lower bitwidths compared to HIGGS-MSQ. For Qwen 2.5-7B, our 3.5-bit model matches the performance HIGGS-MSQ at 4.00 bits, and our 3.75-bit result slightly improves upon the HIGGS-MSC result at 4.25 bits, yiclding up 12.5% memory savings. A similar trend is observed

---

Table Ablation study of layer fusion and CUDA-Core kernel usage on inference throughput and WikiText2 perplexity (LLaMA 3.1 3.B, batch size=l, RTX4090). 'TC' and 'CC' denote Tensor Core and CUDA Core kernels, respectively.
Method
Throughput (Toks/s) (1)
Wiki2 (1)
FP16
62
5.61
Ours-VQu VQ-2 (single scheme)
231
5905.08
Ours-MSQ-Lat (No Fusion, TC Only)
228
119.72
Ours-MSQ-Lat (Fusion-aware TC only)
232
8.93
Ours-MSQ-LLt (Fusion-aware, TC and CC)
231
8.71
Ours-TCQ-2 (single scheme)
223
37.95
Ours-MSQ-Lat (No Fusion, TC Only)
223
20.33
Ours-MSO-Lat (Fusion-awarr aware, TC only)
224
7.79
Ours-MSQ-Lat (Fusion-aware, TC and CC)
223
7.69
Ours-TCQ-3 (single scheme)
185
6.78
Ours-MSQ-Lat (No Fusion, TC Only)
187
6.47
Ours-MSQ-LLt (Fusion-aware, TC only)
186
6.06
Ours-MSQ-Lat (Fusion-aware, TC and CC)
185
6.03
Table 9: Ablation comparing 'linearity" 'actual' loss terms in data-aware MSQ (LLaMA 2-7B, RTX4090 GPU)
LLaMA 7B
Throughput (Toks/s)
Method
Bits
Wiki2 (4)
Acc (t)
B 8
FP16
16.00
5.12
64.9
71
527
QTIP
2.00
6.84
58.9
209
386
Our--MSQ-Mem (linearity)
2.00
6.69
60.3
270
1690
Ours-MSQ-Mem (actual)
2.00
6.47
60.3
272
1684
QTIР
3.00
5.39
63.3
184
304
Ours-MSQ-Mem (linearity)
3.00
5.38
64.3
225
1501
Ours-MSQ-Mem (actual)
3.00
5.34
63.9
224
1489
on LLaMA 3. 3.BB where our 3,40-bit and 3.63-bit results slightly outperform HIGGS-MSQ, at 4.00 and 4.25 bits, respectively. resulting in up 15% memory savings at better perplexity. These results demonstrate the broad applicability of our MSQ method.
G.2.2 Ablation on layer fusion and CUDA-Core kernel integration
Table presents an ablation study evaluating the impact of layer fusion and the integration of CUDA-Cord kernels on inference throughput and WikiText2 perplexity for the quantized LLaMA 3.BBBB model. We compare single-scheme baselines (VQ-2, TCQ-2, and TCQ-3) in Q-Palette against several MSQ variants: MSQ without layer fusion (Tensor Core kernels only), fusion-aware MSQ using only Tensor Core kernels, and fusion-aware MSQ combining both Tensor Core and CUDA Core kernels. Results demonstrate significant improvements WikiText2 perplexity when applying fusion- aasirn-aware MSQ compared to single-scheme quantization and MSQ without fusion, highlighting effectiveness jointly optimizing quantization schemes and layer fusion. For example, MSQ without fusion achieves 20.33 perplexity at 223 tokens/sec, while our fusion-aware MSQ achieves significantly reduced perplexity of 7.79 at 224 tokens/sec. Additionally, integrating CUDA Core kernels alongside Tensor Core kernels provides further performance improvement.
25

---

Table 10: Per-group quantization results on LLaMA 3.1-8B (group size = 64) without IP. For MSQ. we used a pool of per- group NUQ quantizers ranging from 2 to 8 bits with group size 64. All results are reported without IP.
Method
Bitwidth
Bit allocation strategy
Wiki2 (L)
G64-NUQ-3
3,25
Uniform bitwidth
8063.91
G64-MSQ-Mee
3,25
MSQ with Gaussian-assune error
7.34
G64-MSQ-Mem
3.25
MSQ with true quantization error
7.33
G64-NUQ-4
4.25
Uniform bitwidth
6.10
G64-MSQ-Mem
4.25
MSQ with Gaussian-assumed error
5.89
G64-MSQ-Mem
4.25
MSQ with true quantization error
5.90
G.2.3 Effect of loss-term choice in data-aware MSQ
Table provides an ablation study comparing two different loss-term definitions ('linearity' vs. "actual" used in data-aware MSQ quantization for LLaMA 2-7B. The 'linearity" loss term efficiently approximates the increase in perplexity loss via sensitivity coefficients measured using the linearity theorem, enabling reuse across multiple quantization configurations, similar to the data-free scenario (see Appendix E In contrast, the 'actual' loss term explicitly computes the empirical (validation) perplexity increase caused by quantization. Our results demonstrate that using the computationally efficient 'linearity' loss term achieves comparable zero-shot accuracy improvements to those obtained with the 'actual" loss term, indicating that the simpler and reusable linearity- -based approach is also effective in practice. Additionally, both loss-term approaches achieve similar inference throughput, reinforcing the practicality of the computationally efficient 'linearity' loss term.
G.2.4 Applicability of MSQ with iinearity-theoree surrogate without incoherence processing
A natural question is whether the MSQ framework based on the linearity-theore surrogate [35 remains applicable when incoherence processing (IP) is not available due to the hardware constraints. This surrogate objective requires sensitivity coefficients and per-layer quantization errors; with IP, weights are nearly Gaussian, allowing these errors to be precomputed from random Gaussian matrices as explained in Appendix B
To examine the no-IP case, where per-group quantization is typically adopted to handle weight outliers, we disable IP and quantize on per-group basis (group size = 644 using fixed Gaussian- trained codebooks. We compare three bit allocation strategies: 1) uniform bitwidth, 2) MSQ with Gaussian-as -asuueed error, which relies on cached Gaussian distortion estimates for err(Qq; Wi) and solves Equation (T), and 3) MSQ with true quantization error, which measures layerwise distortion err (Q,; Wi) directly and solves Equation
As shown in Table TO MSQ significantly outperforms uniform bitwidth even in the no-IP case. Moreover, the Gaussian-assunie quantization eиror (e.g.. 7.34 vs. 7.33 at 3.25 bits), providing preliminary evidence that cached Gaussian-based error estimates may remain reliable in the no-IP case.
error achieves perplexity almost identical to that from the true
H Limitations and future work
We introduce Q-Palette, a comprehensive suite of quantizers spanning a wide range of trade-off's across memory footprint, inference latency, and quantization error, offering versatile options suitable for diverse deployment scenarios. To demonstrate its effectiveness, we integrate Q-Palette into an MSQ framework and validate its ability achieve improved performance-effciiency trade-off' under PTQ settings. However, our framework is designed around one-shot MSQ objectives, which rely on layer-wise second-order approximations of end-to-end loss and are primarily applicable scenarios that do not involve retraining [35]. While Q-Palette can also serve as a building block for retraining- training, which may be preferable cases where larger computational budgets and data are available, we have not evaluated its effectiveness in that setting, as our primary focus is on data-free or calibration-light PTQ. Extending Q-Palette to retraining based quantization workflows thus remains a promising direction for future work.
based quantization workflows such as quantization-awan
26

---

Another limitation lies in the cost of computing the sensitivity coefficients. Currently, evaluating these coefficients requires (LL) computations of the KL-divergence loss in data-free setting as explained in Appendix E which can become a bottleneck as the model size grows. Although this computation can be performed in an embarrassingly parallel manner and the resulting coefficients can be reused across all MSQ runs, thus representing one-time cost, the overhead may still be non-negligibe when the set of target memory or latency is fixed and reuse is limited. Developing methods to further reduce this cost is therefore an interesting direction for future research.
One promising direction is the extension of Q-Palette to weight-activatiin quantization. In this work we focus on welght-only PTQ, which is particularly effective in memory- bound inference settings with small batch sizes, such as on laptops or mobile devices where memory bandwidth, rather than compute. is the primary bottleneck. However, on some hardware accelerators, such as the Qualcomm Hexagon NPU, which natively support only integer (e.g., INT8) GEMM, activation quantization is essential for exploiting their full performance. Thus, extending Q-Palette to support weight-activation quantization is natural direction for broader deployment. One potential approach is two-stage scheme: (I) first quantize weights to INT8 using uniform W8A8 quantizers for hardware compatibility; and (2) then apply secondary compression step that further quantizes the INT8 weights into æ-bit representations using variant of Q-Palette quantizers whose codebooks are constrained to the INT8 grid. During inference, the compressed weights are dequantized back to INT8 and then processed using integer GEMM with INT8 INT8- quantized activations, enabling compatibility with only hardware while reducing memory usage. A similar idea was introduced in Q-Serve, which quantizes weights in two stages, first to symmetric INT8 grid and then to asymmetric INT4 31
We consider exploring such extensions an interesting direction for future work.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction clearly reflect the paper's contributions and scope by explicitly highlighting our key contributions: (1) deriving the information-theoreetioo optimal bit allocation strategy, (2) introducing Q-Palette, a comprehensive suite of fractional- bit quantizers (including trellis-coded, vector, and scalar quantizers) with efficient CUDA kernel implementations, and (3) proposing novel fusion-aware mixed-scheme quantization framework. All these claims are thoroughly supported through theoretical analyses, detailed methodology, kernel implementation descriptions, and extensive empirical evaluations presented in the main paper.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We discuss the limitations of our work in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
28

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We state the theoretical result in Section 3.1 The assumptions and complete
proofs corresponding our theoretical results are provided Appendices A and B
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all the necessary details to reproduce the main experimental results in Appendix G (for experimental settings), Appendix D (for kernels), Appendix (for Figure D. and Appendix (for details on MSQ).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
29

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We include the implemented CUDA kernels and the codebase required to reproduce the experimental results as supplemental material in the submission, and we have released the code GitHub.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
public//giidee/CodeSuuiiti wow.eoobbssoioooocoooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips https///cccpllicgggeeeee/e/eeeee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All essential details regarding experimental settings are clearly documented in Appendices to
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
30

---

Justification: We comprehensively evaluate our methods across multiple models (LLaMA 2-7B, 13B, LLaMA 3.1-8B, 3.2-IB, 3B) and various bitwidths using standard metrics (WikiText2 perplexity and zero-shot accuracy measured by 1m_ an_eval). However, we do not explicitly report error bars or statistical significance tests, quantization literature to demonstrate robustness through comprehensive evaluations across multiple models and bitwidths rather than conducting isolated statistical tests.
common practice in
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Information regarding the computing resources used is provided in Appendix for CUDA kernel evaluations and Appendix for the main experimental settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Yes, our research complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
31

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Potential societal impacts discussed in the Impact statement section of the main paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work focuses on quantization methods and its compute kernels. does not involve releasing models, data, other artifacts with high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this. but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
32

---

Answer: [Yes]
Justification: We credit the libraries used in our experiments (Appendix G and acknowl- edge the original kernel implementation upon which our kernel implementation builds (Appendix D).
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwpsswipeaaaaeeeeeeeeeeeee thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide our CUDA kernel implementations and the codebase for reproduc- ing results with README. md file, zip file in the supplemental material submission, and we have released the code on GitHub.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/mmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work focuses quantization methods and its compute kernels. It does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
33

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were only used for editing purposes and did not affect the core method- ology, scientific rigor, or originality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hbtpap///eorrgsscc////offrrrerep for what should or should not be described.
34


Reviews:
[{"review_id": "getQfnCWEz", "reviewer": "Reviewer_skQe", "summary": "This paper proposes a multi-scheme quantization framework for post-training quantization of neural networks, including large language models (LLMs). Specifically, it formulates the problem of choosing the most suitable quantizer for each layer—selected from a suite of quantizers with varying precision levels—as an optimization problem that can be numerically solved to balance accuracy and model size. The authors further extend their approach to account for layer fusion, noting that the fusion-aware optimization problem is NP-hard but can still be addressed numerically in practice.", "questions": "- Can the computation of the layer sensitivities $\\alpha_l$ be explained in greater detail, particularly in the context of data-free settings?\n- Do the experimental results include the fusion-aware framework? Does including/not including it make a significant difference in the results?\n- Does the linearization of the model loss hold for non-standard architectures such as those with many skip layer connections?", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 1, "strengths_and_weaknesses": "The paper’s main strengths are its practical contributions: the authors provide efficient CUDA implementations of the quantizers used in their framework, making the method readily usable for practitioners interested in deploying their solution. Moreover, the experimental results show some improvement over both uniform bitwidth baselines and a previous work that allocated bit-widths to layers non-uniformly.\n\nThe main weakness of the paper lies in its limited novelty. The quantizer selection through optimization closely parallels the principles established in Malinovskii et al. ([30]) for bitwidth allocation, and the approach appears to build directly on that prior framework. Additionally, the quantizers included in the QPalette suite are classical, well-known algorithms, and the work does not introduce new quantization techniques. As a result, while the practical contributions are clear and valuable for deployment, the paper’s research novelty may not be sufficient to justify acceptance.", "was_revised": true, "final_justification": "I thank the authors for their detailed response. Despite the noteworthy practical value demonstrated, as stated in my final response to the authors, I will keep my original score of 3 because of my concerns with the novelty.", "rebuttal": "We thank the reviewer for recognizing the practical value of our work, including our efficient CUDA kernel implementations and improvements over uniform and non-uniform bitwidth baselines.\n\n---\n\n**W. Regarding novelty and contributions beyond prior work**:\n\nWhile Q-Palette builds on established quantizer families (NUQ, VQ, TCQ), we emphasize that supporting efficient kernel execution for fine-grained fractional-bit variants and larger batch sizes required substantial engineering effort, including the precise mapping of quantized weights to warp-level `mma` instruction fragments for each quantizer configuration. As the reviewer acknowledged, these kernels make Q-Palette “readily usable for practitioners” and contribute significantly to real-world deployability.\n\nBeyond these engineering aspects, our work introduces several novel contributions that distinguish it from prior work, including HIGGS \\[30].\n\n**First, we propose fusion-aware MSQ, addressing the reviewer’s concern that our optimization framework closely follows HIGGS \\[30].**\n\nAs stated in the paper, our standard MSQ formulation in Section 4.1 adopts a well-established mixed-precision quantization framework based on the linear surrogate loss introduced in prior works \\[13, 6, 30]. We use this formulation to evaluate the effectiveness of our quantizer set (Q-Palette) under memory constraints, but we do not claim it as our contribution.\n\nIn contrast, under latency-constrained scenarios, we introduce fusion-aware MSQ (Section 4.2), which extends the standard MSQ formulation by jointly optimizing quantizer assignment and operator fusion decisions. Unlike standard MSQ, which considers only the cost and loss associated with each individual layer, fusion-aware MSQ incorporates the potential cost reduction from operator fusion directly into the optimization process, enabling additional opportunities for performance improvement. This leads to substantial improvements in the accuracy-latency trade-off in practical deployment scenarios. The empirical results demonstrating the effectiveness of fusion-aware MSQ are provided in our response to Q2.\n\n\n**Second, we introduce half-TCQ, a simple yet novel variant that enables finer control over bitwidths.**\n\nWhile Q-Palette builds upon established quantizer families, it also includes new extensions that go beyond prior work. In particular, half-TCQ splits a weight matrix and applies different TCQ bitwidths to each half (e.g., 2.5 and 3.0 bits), achieving a 2.75-bit representation. This design enables finer-grained bitwidth control beyond the 0.5-bit intervals. To support efficient deployment, we implement CUDA kernels that perform dequantization and matrix multiplication for a half-TCQ layer in a single kernel call. To our knowledge, half-TCQ is a novel quantization technique that has not appeared in prior work and directly addresses the reviewer’s concern regarding the lack of new quantization techniques.\n\n\n**Third, our theoretical analysis in Section 3.1 provides a principled motivation for the design of Q-Palette.**\n\nTheorem 3.1 and the analysis in Appendix B show that to improve MSQ performance, it is important to use practical quantizers that achieve distortion close to the theoretical optimum for a given bitwidth, and are available at fine-grained fractional bitwidths. This insight motivates the design of Q-Palette and offers guidance for future quantizer development tailored to optimal bit allocation.\n\nWe plan to revise the introduction in the camera-ready version to more explicitly highlight these contributions.\n\n---\n\n**Q1. Can the computation of the layer sensitivities be explained in greater detail, particularly in the context of data-free settings?**\n\nThank you for the question. The procedure for computing the sensitivity coefficients $a_l$ in data-free settings is described in **Appendix E.1**. We briefly summarize it here for clarity.\n\nIn data-free scenarios, we estimate the surrogate loss as\n\n$$\n\\ell_{lq} = a_l \\cdot \\mathrm{err}(Q_q; W_l),\n$$\n\nwhere $\\mathrm{err}(Q_q; W_l)$ is approximated using the precomputed distortion of quantizing standard Gaussian matrices. To compute the sensitivity coefficient $a_l$, we adopt the procedure introduced in the HIGGS \\[30]. Specifically, we:\n\n* Generate 128K random tokens from the LLM. We used `top_k=50`, `top_p=0.98` with `temperature=1.0` for generation.\n* Inject scaled Gaussian noise into each weight matrix $W_l$ at 16 different norm levels\n  $n_{li} = \\frac{\\sqrt{i}}{16} \\\\|W_l\\\\|_2$ for $i = 1, \\dots, 16$.\n* Measure how much the model’s KL-divergence loss increases on randomly sampled 128K tokens when Gaussian noise of each level is injected into layer $l$.\n\nThis increase in loss is approximately linear in $n_{li}^2$, and we estimate $a_l$ by linear regression on ($n_{li}^2$, increase in loss) data. This process requires $16 \\times L$ forward evaluations of the loss function but can be done in embarassingly parallel manner. Once computed, the sensitivity coefficients can be reused across all data-free MSQ scenarios with no additional cost.\n\nIf the explanation in Appendix E.1 was unclear, we hope this summary helps clarify the practical steps and rationale.\n\n\n---\n\n**Q2. Do the experimental results include the fusion-aware framework? Does including/not including it make a significant difference in the results?**\n\nThank you for your question regarding the fusion-aware MSQ (Section 4.2). We demonstrate the impact of fusion-aware MSQ through experiments comparing it to standard MSQ.\n\nFirst of all, Figure 1 in our paper shows the qualitative results on effectiveness of fusion-aware MSQ. In Figure 1(c), we show the quantizer allocation using the standard MSQ without considering fusion, while Figure 1(d) presents the allocation achieved with fusion-aware MSQ. As seen in the figure, the fusion-aware MSQ outperforms the standard MSQ in both perplexity and speedup. Specifically, **fusion-aware MSQ** achieves a perplexity of **6.39** and a speedup of **3.17x** which outperforms  the standard MSQ, which achieves a perplexity of 6.46 and a speedup of 2.99x,  in both perplexity and speedup. This demonstrates that fusion-aware MSQ leads to better accuracy while maintaining a higher speedup.\n\nAdditional quantitative results are presented in Table 7 of Appendix G in supplementary material, where we compare the perplexity of fusion-aware MSQ against standard MSQ on LLaMA 3.1-8B across various throughput levels. For example, on an RTX 4090, with batch size 1, standard MSQ achieves 20.33 perplexity at 223 tokens/sec, while our **fusion-aware MSQ** achieves a significantly reduced perplexity of **7.79** at **224** tokens/sec.\n\nThese results confirm that the fusion-aware framework brings substantial improvements over standard MSQ. We will further emphasize these findings in our camera-ready version to ensure the impact of fusion-aware MSQ is clearly communicated.\n\n\n---\n\n**Q3. Does the linearization of the model loss hold for non-standard architectures such as those with many skip layer connections?**\n\nThank you for the thoughtful question. We would like to clarify that standard transformer architectures, including all models evaluated in our paper (LLaMA 2-7B, 13B, LLaMA 3.2-1B, 3B, and 3.1-8B), already include two skip connections per block: one around the self-attention module and another around the MLP. As such, our results provide empirical evidence that the linearization-based surrogate objective works effectively in architectures with multiple skip connections.\n\nWe do not analyze the surrogate’s behavior in architectures with significantly different connectivity patterns, such as DenseNet  for computer-vision tasks. Exploring such cases could be worthwhile in broader contexts, but this falls outside the scope of our study, which focuses on quantizing pretrained Transformer-based LLMs."}, {"review_id": "jfDMhPCBMa", "reviewer": "Reviewer_LLDE", "summary": "This paper presents a set of fractional-bit quantizers aimed at accelerating weight-only PTQ for LLMs. The proposed CUDA kernels, together with a mixed-scheme quantization framework, demonstrate competitive performance in decoding speed.", "questions": "1. What is the effectiveness of Q-Palette and MSQ on larger LLMs (e.g., LLaMA3-70B) and on other architectures such as the Qwen series?\n2. Could Q-Palette be extended to support or improve weight-activation quantization methods, particularly rotation-based techniques?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The custom CUDA kernel implementation is technically solid and shows real-world speedups on NVIDIA GPUs.\n2. The quantizers introduced in Q-Palette achieve significant decoding speed improvements compared to existing baselines such as NF and QTIP.\n\n**Weaknesses**\n\n1. The experimental evaluation is primarily limited to LLaMA series models and small-scale models. Broader coverage would strengthen the conclusions.\n2. Some relevant rotation-based quantization methods, such as DuQuant [1] and OstQuant [2], are not discussed in the Related Work section. Please consider citing and briefly discussing these in Section 6.\n3. The solution to fusion-based mixed-scheme quantization in Section 4.2 is not clearly explained. Additional details are needed to understand how the solution is derived and implemented.\n\n\n[1]. DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs. NeurIPS 2024.\n\n[2]. OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting. ICLR 2025.", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude for your encouraging comments regarding our “technically solid” CUDA kernel implementation and the “significant decoding speed improvements” achieved by Q-Palette.\n\n---\n**W1 & Q1. What is the effectiveness of Q-Palette and MSQ on larger LLMs (e.g., LLaMA3-70B) and on other architectures such as the Qwen series?**\n\nThank you for raising this important point. We agree that evaluating our framework on both larger-scale and non-LLaMA architectures is crucial to demonstrating generality.\n\nTo this end, we applied our MSQ method with Q-Palette to LLaMA 3.1-70B (large-scale) and Qwen 2.5-7B (non-LLaMA), comparing against HIGGS-MSQ [3] under various bitwidth constraints.\n\n\n*Table A. Mixed-scheme quantization results on Qwen 2.5-7B. The FP16 perplexity is 6.13.*\n\n| Bitwidth | Wiki2 perplexity of Ours-MSQ-Mem (↓) | Wiki2 perplexity of HIGGS-MSQ (↓) |\n| -------- | ------------------------------------ | --------------------------------- |\n| 3.25     | **6.41**                             | 6.60                              |\n| 3.50      | 6.33                                 | -                                 |\n| 3.75     | 6.27                                 | -                                 |\n| 4.00      | **6.24**                             | 6.33                              |\n| 4.25     | **6.21**                             | 6.28                              |\n\n*Table B. Mixed-scheme quantization results on LLaMA 3.1-70B. The FP16 perplexity is 2.54. To accommodate the broader sensitivity range in LLaMA 3.1-70B, we extended the quantizer set to include higher-bitwidth options (NUQ 7/8 bits and VQ 5.5/6 bits), in addition to the TCQ quantizers.*\n\n| Bitwidth | Wiki2 perplexity of Ours-MSQ-Mem (↓) | Wiki2 perplexity of HIGGS-MSQ (↓) |\n| -------- | ------------------------------------ | --------------------------------- |\n| 3.25     | **3.28**                             | 3.68                              |\n| 3.40      | 3.06                                 | -                                 |\n| 3.63    | 2.94                                 | -                                 |\n| 4.00      | **2.77**                             | 3.13                              |\n| 4.25     | **2.70**                             | 2.96                              |\n\nAs shown in Tables A and B, our method consistently outperforms HIGGS-MSQ under the same bitwidth constraints (3.25, 4.00, 4.25) on both models. Additionally, our method achieves comparable or better perplexity at lower bitwidths compared to HIGGS-MSQ. For Qwen 2.5-7B, our 3.50-bit model matches the performance of HIGGS-MSQ at 4.00 bits (6.33), and our 3.75-bit result slightly improves upon the HIGGS-MSQ result at 4.25 bits (6.27 vs. 6.28), yielding up to 12.5% memory savings. A similar trend is observed on LLaMA 3.1-70B, where our 3.40-bit and 3.63-bit results slightly outperform HIGGS-MSQ at 4.00 and 4.25 bits, respectively, resulting in up to 15% memory savings at better perplexity. \n\nThese results demonstrate the broad applicability of our framework. We will include these additional results in Appendix G of the camera-ready version.\n\n\n\n---\n\n**Q2. Could Q-Palette be extended to support or improve weight-activation quantization methods, particularly rotation-based techniques?**\n\nIn this work, we focus on weight-only PTQ, which is particularly effective in memory-bound inference settings with small batch sizes, such as on laptops or mobile devices where memory bandwidth, rather than compute, is the primary bottleneck.\n\nHowever, on hardware accelerators like the Qualcomm Hexagon NPU in Snapdragon 8 Gen 3, which natively support only integer (*e.g.*, INT8) GEMM, activation quantization is essential for leveraging their full performance. Thus, extending Q-Palette to support weight-activation quantization is a promising direction for broader deployment.\n\nOne potential approach is a two-stage scheme: (1) first quantize weights to INT8 using uniform W8A8 quantizers for hardware compatibility, and (2) then apply a secondary compression step that further quantizes the INT8 weights into x-bit representations using a variant of Q-Palette quantizers whose codebooks are constrained to the INT8 grid. During inference, the compressed weights are dequantized back to INT8 and then processed using integer GEMM with INT8 quantized activations, enabling compatibility with INT8-only hardware while reducing memory usage.\n\nWe consider this an important direction for future work and will discuss it in Appendix H of the camera-ready version.\n\n---\n\n**W2. Some relevant rotation-based quantization methods, such as DuQuant [1] and OstQuant [2], are not discussed in the Related Work section. Please consider citing and briefly discussing these in Section 6.**\n\nThank you for pointing this out. We will revise Section 6 to cite and briefly discuss recent rotation-based quantization methods including DuQuant [1] (which applies rotation and permutation), OstQuant [2] (which uses learned scaling and rotation matrices), and FlatQuant [4] (which adopts learned affine transformations). These methods primarily target weight-activation quantization, whereas our work focuses on weight-only PTQ which is especially suited for memory-bound, small-batch inference. We will clarify this distinction and update the Related Work section accordingly in the camera-ready version.\n\n---\n\n\n\n\n\n\n**W3. The solution to fusion-based mixed-scheme quantization in Section 4.2 is not clearly explained. Additional details are needed to understand how the solution is derived and implemented.**\n\nThank you for your question regarding the fusion-aware MSQ formulation in Section 4.2. We are happy to provide further clarification.\n\nWe begin by briefly reviewing operator fusion. Within each Transformer block, certain linear layers, such as {query, key, value} or {up, gate}, share the same input and can be fused into a single matrix multiplication. For example, instead of separately computing $ \\\\mathbf{x}W_q$, $ \\\\mathbf{x}W_k$, and $ \\\\mathbf{x}W_v$ for an input $\\\\mathbf{x}$, we can concatenate the weight matrices and compute $\\\\mathbf{x}(W_q \\oplus W_k \\oplus W_v) $, followed by splitting the output. This common optimization is often used to improve inference speed by reducing the number of kernel launches and memory accesses. A visualization of fused layers is provided on the right side of Figure 1 in our paper.\n\n\nIn quantized models, however, fusion is only valid if the involved layers share both the input and **the same quantization configuration**. Fusion-aware MSQ is designed to jointly determine:\n\n1. how to group layers for fusion, and\n2. which quantizer to assign to each fused group.\n\nWhereas standard MSQ (Section 4.1) introduces one binary variable per (layer, quantizer) pair, fusion-aware MSQ defines one binary variable per (fusible layer group, quantizer) pair. For each Transformer block $b$, we define the set of fusible layer groups:\n\n$$\n\\mathcal{G}_b=\\\\{\\\\{q_b\\\\}, \\\\{k_b\\\\}, \\\\{v_b\\\\}, \\\\{q_b,k_b\\\\}, \\\\{q_b,v_b\\\\}, \\\\{k_b,v_b\\\\}, \\\\{q_b,k_b,v_b\\\\}, \\\\{o_b\\\\}, \\\\{u_b\\\\}, \\\\{g_b\\\\}, \\\\{u_b,g_b\\\\}, \\\\{d_b\\\\}\\\\}.\n$$\n\nand the full set is $\\mathcal{G} = \\\\bigcup_{b=1}^B \\mathcal{G}_b$, where $B$ is the number of Transformer blocks.\n\nEach binary variable $P_{gq} \\in \\\\{0,1\\\\}$ indicates whether the layer group $g$ is quantized using quantizer $q$. These variables jointly encode both the fusion and quantization decisions. \n\nTo ensure valid solutions, the problem imposes two constraints:\n\n1. **Exclusive assignment**: each layer must appear in exactly one activated (group, quantizer) pair, *i.e.*, among all groups $g$ that contain layer $l$, only one corresponding $P_{gq}$ may be 1:\n\n$$\n\\sum_{g \\in \\mathcal{G}: l \\in g} \\sum_{q=1}^{|\\mathcal{Q}|} P_{gq} = 1, \\quad \\forall l \\in \\bigcup_{b=1}^B \\\\{q_b, k_b, v_b, o_b, u_b, g_b, d_b\\\\}.\n$$\n\n2. **Resource constraint**: the total profiled cost (*e.g.*, latency or memory) of all activated groups must not exceed the resource budget $C$:\n\n$$\n\\sum_{g \\in \\mathcal{G}} \\sum_{q=1}^{|\\mathcal{Q}|} P_{gq} \\cdot c_{gq} \\le C,\n$$\n\nwhere $c_{gq}$ denotes the profiled cost (*e.g.*, latency) of executing the fused layer group $g$ quantized by $q$.\n\nThe objective is to minimize total estimated performance loss across all groups:\n\n$$\n\\sum_{g \\in \\mathcal{G}} \\sum_{q \\in \\mathcal{Q}} P_{gq} \\cdot \\sum_{l \\in g} \\ell_{lq},\n$$\n\nwhere $\\ell_{lq}$ is the estimated loss from quantizing layer $l$ with quantizer $q$. The resulting optimization problem corresponds to Problem (4) in our paper.\n\nSince both the objective and constraints are linear in $P_{gq}$, the problem can be formulated as an integer linear program (ILP). We solve this ILP problem using the SCIP solver in OR-Tools and our implementation typically completes within seconds to minutes across all experiments. For implementation details, please refer to the supplementary material (`solve_optimal_quantizer` function in `codes/solve_lat_const.py`).\n\nWe believe that fusion-aware MSQ is one of the key contributions of our work. To the best of our knowledge, it is the first quantization framework to jointly optimize **layer fusion and quantizer assignment** under deployment constraints within a unified, practical ILP formulation. We will revise the camera-ready version to make this contribution more prominent and clearly presented.\n\n\n---\nReferences\n\n[1]. DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs. NeurIPS 2024.\n\n[2]. OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting. ICLR 2025.\n\n[3]. Pushing the Limits of Large Language Model Quantization via the Linearity Theorem. NAACL 2025.\n\n[4]. FlatQuant: Flatness Matters for LLM Quantization. ICML 2025."}, {"review_id": "QZO9WNBUA0", "reviewer": "Reviewer_hvcr", "summary": "This paper has two main parts: a bit allocation algorithm that chooses how many bits to compress each linear layer with, and a series for scalar, vector, and trellis coded quantization that enable the bit allocation algorithm to be useful in practice. The bit allocation algorithm builds upon the \"linearity theorem\" in HIGGS by formulating bit allocation as a multi choice knapsack problem. The kernels build upon existing quantizers in the literature, with the main contribution being a set of fast trellis coded kernels that support a wider range of bitrates and decoding batch sizes than the existing QTIP kernels while also being faster. The empirical results in this paper show that Q-Palette's bit allocation algorithm achieves better compression-distortion tradeoffs than uniform bitrate methods and that Q-Palette's kernels are more flexible and faster than existing kernels.", "questions": "See above", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- This paper has two key components: a bit allocation algorithm and a set of kernels. Based on the empirical results, the kernels are faster and more flexible than the existing QTIP kernels without sacrificing quality. This alone is an important contribution, since a major hurdle to adopting advanced quantizers is good kernel support. The bit allocation algorithm works well in practice and has some basis in theory, although I have some questions (see below).\n- The bit allocation algorithm works with data-free quantization algorithms.\n- The paper is well written and easy to follow.\n- Although this is a more \"minor\" part of the paper, the authors introduce a way to perform fractional trellis coded quantization within a single weight matrix. To the best of my knowledge, this is new and nice to see.\n\nWeaknesses/Questions:\n- The bit allocation algorithm assumes that the input weight distribution has been whitened to be ~ iid Gaussian with incoherence processing. It isn't always possible to apply IP due to hardware or latency constraints. Does the bit allocation algorithm still work when the weight distribution isn't iid Gaussian? That is, if you produce a set of bitrates with a Gaussian assumption and then quantize a model with these bitrates but without IP, do you still get an improvement over the uniform bitrate case?\n- Theorem 3.1 seems suspicious to me. max(0, bitrate) implies that if the computed bitrate is negative, this linear layer should be pruned from the model since it gets 0 bits. Surely this isn't correct? \n- The paper says that the bit allocation algorithm works better with a better gaussian quantizer (ie closer to the rate distortion limit). The main experiments mostly use the QP trellis quantizers. Do you have experiments that show that the bit allocation algorithm performs worse when used in the VQ or SQ case?", "was_revised": true, "final_justification": "(copy of my final response to the authors). Thanks for your follow up experiments. I think they confirm my initial thoughts about the paper, which is that the bit allocation algorithm and better tcq kernels are somewhat orthgonal to each other. The bit allocation theorem could also be improved due to the possibility of just breaking the model in half (which apparently happens empirically without a bit floor). However, I think both components of the paper are useful and practical contributions that are not just incremental. I think this paper should be accepted, and if we were on the old scale I would give it a weak accept, which seems to be a 4.5 on this scale. I will keep my score at 4 for now since IMO a 5 would be a 7-8 on the old scale, and I don't think this paper is at the level of 7-8.", "rebuttal": "Thank you for your thoughtful and encouraging review. We sincerely appreciate your recognition that the paper is \"well written and easy to follow,\" that our fractional TCQ variant is \"new and nice to see,\" and that the proposed bit allocation algorithm \"works well in practice and has some basis in theory.\" Your insightful questions helped us further validate and clarify key aspects of our work.\n\n--- \n\n\n**W1. Does the bit allocation algorithm still work without incoherence processing (IP)?**\n\nWe appreciate this important question regarding the robustness of our bit allocation strategy when the Gaussianity assumption is violated.\n\nTo assess this, we compare single-scheme quantization and mixed-scheme quantization (MSQ) on LLaMA 3.1-8B using non-uniform scalar quantizers (NUQ), in a setting where IP is disabled.\n\n*Table C. Performance comparison at 3-bit quantization with and without IP on LLaMA 3.1-8B.*\n\n| Method       | IP | Bit allocation strategy          | Perplexity ↓ |\n| ------------ | -- | ---------------------------------- | ------------ |\n| Ours-NUQ-3   | ✗  | Uniform bitwidth                             | 36.86        |\n| Ours-MSQ-Mem | ✗  | Bit allocation by Gaussian-assumed error             | 11.73        |\n| Ours-MSQ-Mem | ✗  | Bit allocation by true quantization error                 | 10.76        |\n| Ours-MSQ-Mem | ✓  | Bit allocation by Gaussian-assumed error             | 6.84     |\n\n*Table D. Performance comparison at 4-bit quantization with and without IP on LLaMA 3.1-8B.*\n\n| Method       | IP | Bit allocation strategy          | Perplexity ↓ |\n| ------------ | -- | ---------------------------------- | ------------ |\n| Ours-NUQ-4   | ✗  | Uniform bitwidth           | 6.21         |\n| Ours-MSQ-Mem | ✗  | Bit allocation by Gaussian-assumed error             | 6.17         |\n| Ours-MSQ-Mem | ✗  | Bit allocation by true quantization error | 6.13         |\n| Ours-MSQ-Mem | ✓  | Bit allocation by Gaussian-assumed error             | 5.92     |\n\nTables C and D show the quantization result for various bit allocation strategies without IP. Even without IP, our MSQ method significantly outperforms uniform bitwidth setting. For example, at 3 bits, perplexity improves from 36.86 (uniform) to 11.73 (MSQ with Gaussian-assumed error), and further to 10.76 when using true quantization error. A similar trend holds at \\~4 bits, where MSQ improves over the uniform baseline from 6.21 to 6.17 (Gaussian-assumed error), and further to 6.13 (true quantization error). Although applying IP consistently yields the best performance (6.84 at 3 bits, 5.92 at 4 bits), these results confirm that the bit allocation algorithm remains effective even without IP.\n\n**Experimental details**. In the no-IP setting, directly using Q-Palette’s per-tensor codebooks (trained on Gaussian-distributed weights) leads to severe performance degradation due to distribution mismatch. To mitigate this, we replaced Q-Palette’s per-tensor codebooks with per-column codebooks fitted to each column of the weight matrix via k-means clustering. We used a pool of NUQ quantizers ranging from 2 to 8 bits for MSQ. Bit allocation was computed by solving the ILP in Problem (3) of the paper, where the per-layer quantization error $\\\\text{err}(Q_q; W_l)$ was instantiated either as (a) precomputed distortion on Gaussian weights (“Gaussian-assumed error”), or (b) actual distortion measured from the retrained codebooks without IP (“true quantization error”).\n\nIn conclusion, while incoherence processing provides the most favorable quantization performance, our bit allocation algorithm maintains strong performance even without it. We will include these ablation results and analysis in Appendix G of the camera-ready version.\n\n---\n\n**W2. Theorem 3.1 seems suspicious to me. max(0, bitrate) implies that if the computed bitrate is negative, this linear layer should be pruned from the model since it gets 0 bits. Surely this isn't correct?**\n\nThe derivation of Theorem 3.1 is mathematically correct under its stated assumptions. The theorem is derived by minimizing a surrogate objective based on the linearity theorem, assuming that optimal distortion quantizers exist for any non-negative bitwidth. Under this formulation, it is not mathematically incorrect for the optimal bitwidth of extremely low-sensitivity layers to become zero.\n\nAlthough this behavior may appear counterintuitive, it is not due to a flaw in the theorem, but rather reflects the limitations of the surrogate approximation. The surrogate is based on a second-order Taylor expansion of the perplexity loss and is accurate when quantization errors are small. However, at very low bitwidths, the quantization error becomes large, and the surrogate may incur significant approximation errors, leading to seemingly unintuitive solutions in extreme cases.\n\nTo prevent such extreme solutions caused by surrogate inaccuracies, a practical refinement is to introduce a minimum bitwidth  threshold \\$\\eta\\$, restricting the optimization to bitwidths \\$\\ge \\eta\\$, where the surrogate approximation expected to remain valid. This leads to a refined solution:\n\n$$\nb_l^* = \\\\max\\\\left\\\\{\\\\eta,\\\\frac{1}{2\\\\ln(2)}\\\\left(\\\\ln\\\\frac{a_l}{d_l^{\\\\mathrm{in}} d_l^{\\\\mathrm{out}}}\\\\right) + C\\\\right\\\\},\n$$\n\nwith \\$C\\$ chosen to satisfy the overall memory constraint.\n\nIn practice, we already adopt this principle by including only quantizers of 1.5 bits or higher in Q-Palette. Our solver never assigns zero-bitwidths, and all layers are quantized with strictly positive values. We will clarify this discussion and incorporate the refined version of the theorem in Appendix A of the camera-ready version.\n\n---\n\n**W3. Do you have experiments that show that the bit allocation algorithm performs worse when used in the VQ or SQ case?**\n\nYes, we provide such comparisons in Figure 3 of the paper. Figure 3 visualizes the WikiText2 perplexity achieved by our memory-constrained MSQ algorithm using different sets of quantizers under varying bitwidth constraints.\n\n* The **blue curve** corresponds to MSQ using only **TCQ quantizers at 2, 3, and 4 bits**.\n* The **cyan curve** corresponds to MSQ using only **VQ quantizers at 2, 3, and 4 bits**.\n\nAs shown, the blue curve (`TCQ-2,3,4`) consistently outperforms the cyan curve (`VQ-2,3,4`) across all tested bitwidth constraints. This empirically supports our claim that the bit allocation algorithm is more effective when used with quantizers that better approximate the rate-distortion limit, such as TCQ. Compared to VQ and SQ, TCQ achieves lower distortion at the same bitwidth (see Figure 2), resulting in better perplexity-bitwidth tradeoffs.\n\nWe will clarify this interpretation of Figure 3 in the camera-ready version to explicitly support the reviewer’s question."}, {"review_id": "38U6AgoHPg", "reviewer": "Reviewer_4xNx", "summary": "This paper aims to develop a finegrained fractional bit quantizer to approach the optimal bit allocation rate for Gaussianized weights after rotation-based transformation. The paper proposes a collection of fractional bit quantizers and corresponding CUDA kernels to achieve optimal compression rate and speedup.", "questions": "Please explain the difference between the proposed quantizer and the TCQ quantizer cited in the paper.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strength\n1. This paper pays special attention to the quantizer design for the post-rotation weight distribution, which is an area largely ignored by previous rotation-based quantizer design.\n2. The paper makes solid emperical contribution by exploring the fractional-bit quantizers and the implementation of CUDA kernel, achieving both improved performance and real speedup\n3. The paper is well-motivated, clearly-written, and easy to follow.\n\n## Weakness\n1. Novelty-wise, the fractional precision quantization and the kernel implementation proposed in the paper appears to have been based on existing work, as cited in the paper. The contribution of the paper is mainly implementing these techniques on rotational-quantized LLMs\n2. The conclusion that fractional-bit quantization leads to better performance-efficiency tradeoff is not suprising. Though the paper brings practical advances on the model compression frontier, the research insight is less significant.", "was_revised": true, "final_justification": "After checking the rebuttal and the author's reply, I'm confident in my judgement that the paper is solid. Though the novelty on the algorithm side may be limited, the practical contribution is adequate and should be encouraged.", "rebuttal": "Thank you for your thoughtful and constructive review. We appreciate your positive assessment that the paper is “well-motivated, clearly-written, and easy to follow,” and that it makes a “solid empirical contribution” by exploring fractional-bit quantizers and implementing CUDA kernels that achieve both improved performance and real speedup.\n\n---\n\n**Regarding our contributions and novelty (W1 & W2).**\n\nAs the reviewer correctly noted, one of the key contributions of this work lies in the practical realization of fractional-bit quantizers through efficient CUDA kernel implementations. Supporting a wider range of bitwidths and larger batch sizes required substantial engineering effort, including the precise mapping of quantized weights to `mma` instruction fragments for each quantizer configuration. These implementations make Q-palette quantizers practically deployable and enable strong empirical results across various model sizes and compression levels.\n\nBeyond implementation, our work offers contributions and novel ideas in several other aspects, summarized below:\n\n* First, we introduce **fusion-aware MSQ**, the first framework to jointly optimize operator fusion decision and quantizer selection under resource constraints. Within each Transformer block, certain linear layers, such as {query, key, value}, share the same input and can be fused into a single matrix multiplication. For example, instead of separately computing $ \\\\mathbf{x}W_q$, $ \\\\mathbf{x}W_k$, and $ \\\\mathbf{x}W_v$ for an input $\\\\mathbf{x}$, we can concatenate the weight matrices and compute $\\\\mathbf{x}(W_q \\oplus W_k \\oplus W_v) $, followed by splitting the output. This fusion technique is often used to improve inference speed by reducing kernel launches and memory accesses. Our method integrates operator fusion directly into the MSQ framework, formulating a unified ILP. This leads to substantial improvements in the accuracy-latency trade-off. For example, on LLaMA 3.1-8B (batch size 1, RTX 4090), standard MSQ achieves 20.33 perplexity at 223 toks/sec, while our fusion-aware MSQ reduces perplexity to **7.79** at a similar throughput of **224** toks/sec (Appendix G, Table 7). Qualitative results are shown in Figure 1.\n\n* In addition, Section 3.1 provides a **theoretical motivation** for the design of Q-Palette. Theorem 3.1 and the analysis in Appendix B show that to improve MSQ performance, it is important to use practical quantizers that achieve distortion close to the theoretical optimum for a given bitwidth, and are available at fine-grained fractional bitwidths. This insight motivates the design of Q-Palette and offers guidance for future quantizer development tailored to optimal bit allocation.\n\n* We also introduce a novel quantization scheme, **half-TCQ**, which splits a weight matrix and applies different TCQ bitwidths to each half (*e.g.*, 2.5 and 3.0 bits), achieving a 2.75-bit representation. This enables finer-grained bitwidth control beyond 0.5-bit intervals. To support efficient deployment, we implement CUDA kernels that perform dequantization and matrix multiplication for a half-TCQ layer in a single kernel call.\n\n\nWe will revise the introduction in the camera-ready version to more clearly reflect these contributions.\n\n---\n\n**Q1. Please explain the difference between the proposed quantizer and the TCQ quantizer cited in the paper.**\n\nWhile our TCQ quantizers are based on the bitshift variant of TCQ introduced in QTIP, our TCQ quantizers introduce several key enhancements that significantly improve practicality.\n\nFirst, while QTIP supports only a limited set of integer bitwidths (2, 3, 4 bits), our implementation extends support to a broader range of fractional bitwidths (1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0 bits). In addition, we introduce half-TCQ, a novel scheme that enables intermediate bitwidths such as 1.75, 2.25, 2.75, 3.25, 3.75, 4.25, 4.75 by applying different bitwidths to each half of the weight matrix.\n\nMoreover, our kernels support batch sizes up to 8, whereas QTIP was restricted to batch size 1, substantially improving deployment flexibility.\n\nFinally, we reduce runtime overhead by rotating weights only along the input dimension and reusing rotation matrices across layers that share inputs, effectively reducing the number of online rotations (Section 3.3)."}]

Author rebuttal:

Meta-review: The paper considers the problem of post-training quantization of LLMs. It tries to identify the optimal bit allocation, and quantization schemes to use at each layer of an LLM for achieving optimal latency and quality tradeoffs. The paper poses this as a search problem over a discrete space and relies on integer linear program solvers to identify a good solution. The particular search space considered is as follows
- **quantization schemes**: non uniform scalar quantization implemented using k-means clustering, 2D vector quantization,  and Trellis-coded quantization (TCQ)
- **bits**: apart from the usual  integer bit quantization schemes, the paper also considers fractional bit quantizers.  For instance, the authors propose to different number of bits for quantization first half of the weights in a layer compared to the second half. This effectively mimics fractional bit quantization.

While this framework itself is not novel, the paper introduces a new dimension to the search space by introducing layer fusion, where linear layers with same input can be fused together for reducing memory access. The other main contribution of the is to provide efficient CUDA, Tensor core kernels for the above quantization schemes.

The empirical results in the paper look strong. Compared to many baselines, the proposed technique achieves better quality-latency tradeoffs. The being said, there were a number of concerns raised by reviewers
- the main contribution of the work seems to be in the design of kernels for supporting various quantization schemes at various bitwidths. Unfortunately the paper doesn't provide many details on the kernel implementation. Without these details, it is hard to understand where the gains are coming from. For instance, one of the reviewers wondered if the speedups are simply coming from  reducing the number of online Hadamard transforms from 14 to 4.
- the paper never discusses the computational cost of the technique. Given that it requires running various PTQ algorithms at every layer, it can easily be an order of magnitude slower than traditional PTQ techniques. It is not clear how well the technique scales to larger models. 

Despite these limitations, I believe the paper could be interesting to practitioners. So I recommend accepting it. But I encourage the authors to take this feedback into account to improve the camera-ready version of the paper.
Decision: Accept (poster)