QSpec: Speculative Decoding with Complementary Quantization Schemes

Authors: Juntao Zhao, Wenhao Lu, Sheng Wang, Lingpeng Kong, Chuan Wu
Primary area: optimization
Keywords: quantization, compression, large language models, reasoning, speculative decoding

Abstract:
Quantization has been substantially adopted to accelerate inference and reduce memory consumption of large language models (LLMs).
While activation-weight joint quantization speeds up the inference process through low-precision kernels, we demonstrate that it suffers severe performance degradation on multi-step reasoning tasks, rendering it ineffective.
We propose a novel quantization paradigm called QSPEC, which seamlessly integrates two complementary quantization schemes for speculative decoding. 
Leveraging nearly cost-free execution switching, QSPEC drafts tokens with low-precision, fast activation-weight quantization, and verifies them with high-precision weight-only quantization, 
effectively combines the strengths of both quantization schemes.
Compared to high-precision quantization methods, QSPEC empirically boosts token generation throughput by up to $1.80\times$ without any quality compromise, distinguishing it from other low-precision quantization approaches. 
This enhancement is also consistent across various serving tasks, model sizes, quantization methods, and batch sizes.
Unlike existing speculative decoding techniques, our approach reuses weights and the KV cache, avoiding additional memory overhead. Furthermore, QSPEC offers a plug-and-play advantage without requiring any training.
We believe that QSPEC demonstrates unique strengths for future deployment of high-fidelity quantization schemes, particularly in memory-constrained scenarios (e.g., edge devices).

Decision: reject
Reviewer scores: 6, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730048678345

thread_type:
official_review

<thread_object>
creator: reviewer_fprc

<content>
summary:
The authors investigate the impact of LLM quantization on multi-step reasoning tasks and find that current methods are insufficient in some cases. They introduce a system design for speculative decoding where activation quantization is disabled during the verification stage. They find that this can improve HW performance without degrading model quality.

strengths:
Improving the HW performance of multi-step reasoning is a relevant and impactful problem. The authors provide a very insightful analysis into both the impact of quantization on multi-step reasoning and the impact of activation quantization on token prediction probabilities, both of which are novel insights to my knowledge.

weaknesses:
While the results are promising, the benchmarking may be incomplete. As an alternative to this approach, one could just use a smaller draft model without weight sharing assuming memory is not a constraint. Often the significantly smaller draft model is tuned to the verification model to improve acceptance rates. There is no comparison against this dual-model architecture to establish a concrete baseline. As such, it is unclear which approach one should take.

questions:
Have you considered the scenario where a draft model is significantly smaller than the verification model and fine-tuned specifically to maximize acceptance rates? Are the memory benefits from weight sharing still large enough to warrant using QSpec over such an approach?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730051961970

thread_type:
official_review

<thread_object>
creator: reviewer_9onb

<content>
summary:
The authors propose a weight-sharing speculative decoding framework for quantized models with varying activation precisions. The core insight of this work is that aggressive weight-activation quantization methods, such as W4A4, often result in performance degradation on complex multi-step reasoning tasks. To address this performance gap while retaining the efficiency of low bit-width models, the framework employs W4A4 to draft the output and W4A16 to verify it, sharing 4-bit weights but producing outputs at different precisions. The KV-cache is subsequently updated by the high-precision model, enhancing the framework with a high-quality KV-cache. Experiments demonstrate that the framework achieves comparable performance to full W4A16 models, while delivering higher throughput with W4A4-based drafting.

strengths:
The paper presents an interesting observation that claims of "lossless" performance in many quantization studies are often skewed by limited evaluation benchmarks, which overlook the negative impacts of activation quantization. To address this issue, the authors propose a weight-sharing mixed-precision speculative decoding framework. Unlike conventional heterogeneous speculative decoding, which uses a small model for drafting and a significantly larger model for verification, the authors argue that homogeneous speculative decoding with weight-sharing is more suitable for edge scenarios. Although the framework may appear similar to conventional speculative decoding, experiments demonstrate its advantages in both accuracy and throughput.

weaknesses:
While the observation and intuition behind this work are compelling, some experiments seem to be missing, which weakens the support for the authors' claims. For instance, Tables 1 and 3 lack comparisons with FP16 models. Demonstrating the performance differences between FP16 and W4A16 on complex multi-step reasoning tasks would help the community better understand the true performance gaps between FP16 models and their quantized counterparts. Additionally, Table 4 lacks throughput evaluations for W4A4. Including a latency and accuracy trade-off comparison between the framework and FP16, W4A16, and W4A4 models would provide valuable insights. The absence of these experiments may mislead readers and hinder a full assessment of the proposed framework. I am willing to raise my score if authors address aforementioned issues.

questions:
- Could the authors provide the accuracy of the FP16 models in Tables 1 and 3?
- Could the authors provide the throughput of the FP16 and W4A4 models in Table 4 as well as in Figures 4 and 5?
- Could the authors include a figure illustrating the throughput and accuracy trade-offs for the proposed framework compared to all baselines?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730578242994

thread_type:
official_review

<thread_object>
creator: reviewer_mqt3

<content>
summary:
The paper brings an elegant idea to use faster 4b quantized activations for drafting, and use full 16b activations for decoding.

There are significant weaknesses in completeness of the analysis, specifically lack of comparison with other quantization methods in terms of speedup.

strengths:
It is an elegant idea to use faster 4b quantized activations for drafting, and use full 16b activations for decoding - it saves the memory for weights and helps to align draft / target distributions.

weaknesses:
1) The paper does not reported acceptance rates

2) The paper no comparison to other lossless speculative decoding methods, namely Eagle2, Medusa2 with top speedups, BiTA with negligible VRAM overhead.

3) table 1: WikiText-2 PPL # 8.70 7.87 (+9.20%) 8.58 (+1.15%) - why ppl is lower for W4 methods? 

4) table 4: 7B W4A16 HumanEval seems too fast (300+ vs other methods)

5) the paper fails to notice that W4A4 is more memory efficient - need to compare using same memory budget

6) the abstract should mention average improvement, not peak

7) Since this is a lossless method, tests should mostly focus on speed, not on quality. So please add speedup comparisons vs other quantization methods, not just to W4A16 baseline.

8) Llama family should refer to L2 paper, not just L3

9) Table4 - add model names

questions:
1) How would your method compare in speedup to other lossless speculative decoding methods, namely Eagle2, Medusa2 (with top speedups), BiTA (with negligible VRAM overhead).

2) How would your method performs with tree-based speculation?

3) Is the method good for batched generation? What is its performance vs batch size?

4) please report your observed acceptance rates
</content>
</thread_object>

</thread>
<thread>
creation date:
1730646081237

thread_type:
official_review

<thread_object>
creator: reviewer_pctg

<content>
summary:
This paper introduces a new method called QSPEC, which seamlessly integrates two complementary quantization schemes for speculative decoding. QSPEC utilizes W4A4 quantized LLMs to draft tokens and verifies these drafted tokens with W4A16 quantized LLMs, achieving results comparable to those of W4A16 quantized LLMs in generation tasks.

strengths:
1. The authors highlight that quantized models perform worse on generation tasks (e.g., code and math) compared to multiple-choice tasks, particularly when using 4-bit quantized activations. This is a significant issue that the community should address.

2. To my knowledge, this paper is the first to combine quantization and speculative decoding. The authors analyze their compatibility, focusing on token prediction similarity, acceptance rate, shared weights, and key-value caches.

weaknesses:
1. The novelty of this paper is limited. While the combination of speculative decoding and quantization is an interesting topic, the authors do not introduce any new methods. I suggest that the authors explore ways to enhance quantization methods in speculative decoding. For instance, do different quantization techniques, such as Atom and QuaRot, result in varying acceptance rates?

2. It is recommended to make more comparisons with previous speculative decoding methods in order to better demonstrate the benefits of integrating quantization into speculative decoding in Table 4. Due to the shared weights and key - value cache, the memory consumption might be one of the advantages of this method, and this aspect can also be added to Table 4.

3. The speedup ratio compared with previous speculative decoding methods such as Medusa and EAGLE is weak.

questions:
Why does the speedup ratio relative to W4A16 decline as the batch size increases, as depicted in Table 4? I'm intrigued by the overall inference latency difference between the W4A16 and W4A4 models. Could you provide the latency of W4A4 and W4A16 at both the generation and prefilling stages?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces QSPEC, a method that combines low-precision (W4A4) and high-precision (W4A16) quantization schemes to improve token generation throughput in large language models (LLMs) through speculative decoding. The method drafts tokens with low-precision quantization and verifies them with high-precision quantization, offering up to 1.78x throughput improvement without sacrificing output quality. The approach is claimed to be memory-efficient, leveraging shared weights and key-value caches, and does not require retraining, making it suitable for memory-constrained environments.

The paper demonstrates clear performance gains in token generation throughput, making it a relevant contribution for improving the efficiency of LLMs. The empirical results are promising, showing improvements across various tasks, model sizes, and batch sizes. Additionally, the method avoids increasing memory consumption by reusing weights and key-value caches, which is beneficial for deployment in resource-limited scenarios. The fact that the method does not require retraining is another practical advantage, facilitating easy adoption.

Reviewers raised concerns about the limited novelty of the paper, as it primarily integrates existing quantization techniques (W4A4 and W4A16) with speculative decoding. They also requested more robust comparisons, particularly with other low-bitwidth quantization methods like EAGLE and W4A4, and more detailed performance metrics (e.g., speedup vs. batch size vs. quality). Although the authors provided a detailed rebuttal and made some revisions, these concerns were not fully addressed.

The paper lacks significant novelty, relying on existing quantization techniques and speculative decoding without introducing fundamentally new methods. While the empirical results are promising, the comparisons with other methods remain insufficient, and the revisions did not fully address the key concerns raised by the reviewers. Given the highly competitive nature of the review process this year and the unresolved issues, the paper does not meet the acceptance criteria and is therefore rejected.