Title: Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM quantization

Post-training quantization

Mixed scheme quantization

Data-free quantization
Abstract: We study weight-only post-training quantization (PTQ), which quantizes the weights of a large language model (LLM) without retraining, using little or no calibration data. Weight-only PTQ is crucial for reducing the memory footprint and latency of LLM inference, especially in memory-bound, small-batch inference scenarios, such as personalized inference on edge devices. Despite its importance, irregular weight distributions with heavy-tailed outliers in LLMs complicate quantization, recently motivating rotation-based methods that transform weights into near-Gaussian distributions, which are more regular with fewer outliers, thereby reducing quantization error. In this work, we first derive the information-theoretically optimal bit allocation for Gaussianized weights under given bit budgets, revealing that fine-grained fractional-bit quantizers approaching the Gaussian distortion-rate bound are essential to achieve near-optimal quantization performance. To bridge this theoretical insight and practical implementation, we introduce Q-Palette, a versatile collection of fractional-bit quantizers that range from trellis-coded quantizers offering near-optimal distortion to simpler vector and scalar quantizers optimized for faster inference, all efficiently implemented with optimized CUDA kernels across various bitwidths. Furthermore, leveraging Q-Palette as a foundational component, we propose a novel mixed-scheme quantization framework, jointly optimizing quantizer choices and layer fusion decisions given resource constraints. The code is available at https://github.com/snu-mllab/Q-Palette.


Peer reviews:
comments: SUMMARY:
This paper aims to develop a finegrained fractional bit quantizer to approach the optimal bit allocation rate for Gaussianized weights after rotation-based transformation. The paper proposes a collection of fractional bit quantizers and corresponding CUDA kernels to achieve optimal compression rate and speedup.

STRENGTHS AND WEAKNESSES:
## Strength
1. This paper pays special attention to the quantizer design for the post-rotation weight distribution, which is an area largely ignored by previous rotation-based quantizer design.
2. The paper makes solid emperical contribution by exploring the fractional-bit quantizers and the implementation of CUDA kernel, achieving both improved performance and real speedup
3. The paper is well-motivated, clearly-written, and easy to follow.

## Weakness
1. Novelty-wise, the fractional precision quantization and the kernel implementation proposed in the paper appears to have been based on existing work, as cited in the paper. The contribution of the paper is mainly implementing these techniques on rotational-quantized LLMs
2. The conclusion that fractional-bit quantization leads to better performance-efficiency tradeoff is not suprising. Though the paper brings practical advances on the model compression frontier, the research insight is less significant.

QUESTIONS:
Please explain the difference between the proposed quantizer and the TCQ quantizer cited in the paper.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper has two main parts: a bit allocation algorithm that chooses how many bits to compress each linear layer with, and a series for scalar, vector, and trellis coded quantization that enable the bit allocation algorithm to be useful in practice. The bit allocation algorithm builds upon the \"linearity theorem\" in HIGGS by formulating bit allocation as a multi choice knapsack problem. The kernels build upon existing quantizers in the literature, with the main contribution being a set of fast trellis coded kernels that support a wider range of bitrates and decoding batch sizes than the existing QTIP kernels while also being faster. The empirical results in this paper show that Q-Palette's bit allocation algorithm achieves better compression-distortion tradeoffs than uniform bitrate methods and that Q-Palette's kernels are more flexible and faster than existing kernels.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- This paper has two key components: a bit allocation algorithm and a set of kernels. Based on the empirical results, the kernels are faster and more flexible than the existing QTIP kernels without sacrificing quality. This alone is an important contribution, since a major hurdle to adopting advanced quantizers is good kernel support. The bit allocation algorithm works well in practice and has some basis in theory, although I have some questions (see below).\n- The bit allocation algorithm works with data-free quantization algorithms.\n- The paper is well written and easy to follow.\n- Although this is a more \"minor\" part of the paper, the authors introduce a way to perform fractional trellis coded quantization within a single weight matrix. To the best of my knowledge, this is new and nice to see.\n\nWeaknesses/Questions:\n- The bit allocation algorithm assumes that the input weight distribution has been whitened to be ~ iid Gaussian with incoherence processing. It isn't always possible to apply IP due to hardware or latency constraints. Does the bit allocation algorithm still work when the weight distribution isn't iid Gaussian? That is, if you produce a set of bitrates with a Gaussian assumption and then quantize a model with these bitrates but without IP, do you still get an improvement over the uniform bitrate case?\n- Theorem 3.1 seems suspicious to me. max(0, bitrate) implies that if the computed bitrate is negative, this linear layer should be pruned from the model since it gets 0 bits. Surely this isn't correct? \n- The paper says that the bit allocation algorithm works better with a better gaussian quantizer (ie closer to the rate distortion limit). The main experiments mostly use the QP trellis quantizers. Do you have experiments that show that the bit allocation algorithm performs worse when used in the VQ or SQ case?\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

comments: SUMMARY:
This paper presents a set of fractional-bit quantizers aimed at accelerating weight-only PTQ for LLMs. The proposed CUDA kernels, together with a mixed-scheme quantization framework, demonstrate competitive performance in decoding speed.

STRENGTHS AND WEAKNESSES:
**Strengths**

1. The custom CUDA kernel implementation is technically solid and shows real-world speedups on NVIDIA GPUs.
2. The quantizers introduced in Q-Palette achieve significant decoding speed improvements compared to existing baselines such as NF and QTIP.

**Weaknesses**

1. The experimental evaluation is primarily limited to LLaMA series models and small-scale models. Broader coverage would strengthen the conclusions.
2. Some relevant rotation-based quantization methods, such as DuQuant [1] and OstQuant [2], are not discussed in the Related Work section. Please consider citing and briefly discussing these in Section 6.
3. The solution to fusion-based mixed-scheme quantization in Section 4.2 is not clearly explained. Additional details are needed to understand how the solution is derived and implemented.


[1]. DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs. NeurIPS 2024.

[2]. OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting. ICLR 2025.

QUESTIONS:
1. What is the effectiveness of Q-Palette and MSQ on larger LLMs (e.g., LLaMA3-70B) and on other architectures such as the Qwen series?
2. Could Q-Palette be extended to support or improve weight-activation quantization methods, particularly rotation-based techniques?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a multi-scheme quantization framework for post-training quantization of neural networks, including large language models (LLMs). Specifically, it formulates the problem of choosing the most suitable quantizer for each layer—selected from a suite of quantizers with varying precision levels—as an optimization problem that can be numerically solved to balance accuracy and model size. The authors further extend their approach to account for layer fusion, noting that the fusion-aware optimization problem is NP-hard but can still be addressed numerically in practice.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper’s main strengths are its practical contributions: the authors provide efficient CUDA implementations of the quantizers used in their framework, making the method readily usable for practitioners interested in deploying their solution. Moreover, the experimental results show some improvement over both uniform bitwidth baselines and a previous work that allocated bit-widths to layers non-uniformly.\n\nThe main weakness of the paper lies in its limited novelty. The quantizer selection through optimization closely parallels the principles established in Malinovskii et al. ([30]) for bitwidth allocation, and the approach appears to build directly on that prior framework. Additionally, the quantizers included in the QPalette suite are classical, well-known algorithms, and the work does not introduce new quantization techniques. As a result, while the practical contributions are clear and valuable for deployment, the paper’s research novelty may not be sufficient to justify acceptance.\n\nQUESTIONS:\n- Can the computation of the layer sensitivities $\\alpha_l$ be explained in greater detail, particularly in the context of data-free settings?\n- Do the experimental results include the fusion-aware framework? Does including/not including it make a significant difference in the results?\n- Does the linearization of the model loss hold for non-standard architectures such as those with many skip layer connections?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-03"}