Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding
Authors: Pei-Shuo Wang
Jian-Jia Chen
Chun-Che Yang
Chi-Chih Chang
Ning-Chi Huang
Mohamed S. Abdelfattah
Kai-Chiang Wu
Venue: NeurIPS 2025 poster ()
Keywords: speculative decoding
offloading
quantization
compression
large language models
inference
Primary area: deep_learning
TL;DR: We propose a lossless and training-free speculative decoding method to accelerate LLMs that requires offloading on a single memory-limited cosumer GPU.

Abstract: The immense model sizes of large language models (LLMs) challenge deployment on memory-limited consumer GPUs.
    Although model compression and parameter offloading are common strategies to address memory limitations, compression can degrade quality, and offloading maintains quality but suffers from slow inference.
    Speculative decoding presents a promising avenue to accelerate parameter offloading, utilizing a fast draft model to propose multiple draft tokens, which are then verified by the target LLM in parallel with a single forward pass. This method reduces the time-consuming data transfers in forward passes that involve offloaded weight transfers.
    Existing methods often rely on pretrained weights of the same family, but require additional training to align with custom-trained models. Moreover, approaches that involve draft model training usually yield only modest speedups. This limitation arises from insufficient alignment with the target model, preventing higher token acceptance lengths.
    To address these challenges and achieve greater speedups, we propose SubSpec, a plug-and-play method to accelerate parameter offloading that is lossless and training-free. SubSpec constructs a highly aligned draft model by generating low-bit quantized substitute layers from offloaded target LLM portions. Additionally, our method shares the remaining GPU-resident layers and the KV-Cache, further reducing memory overhead and enhance alignment.
    SubSpec achieves a high average acceptance length, delivering 9.1$\times$ speedup for Qwen2.5 7B on MT-Bench (8GB VRAM limit) and an average of 12.5$\times$ speedup for Qwen2.5 32B on popular generation benchmarks (24GB VRAM limit).

Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding
Pei-Shuo Wang', Jian-Jia Chen*1, Chun-Cht Yang*! Chi-Chih Chang? Ning-Chi Huang'. Mohamed S. Abdelfattah", and Kai-Chiang Wu"
National Yang Ming Chiao Tung University Cornell University
Abstract
The immense model sizes of large language models (LLMs) challenge deployment on memory-limited consumer GPUs. Although model compression and parameter offloading are common strategies to address memory limitations, compression can degrade quality, and offloading maintains quality but suffers from slow infer- ence. Speculative decoding presents a promising avenue to accelerate parameter offloading, utilizing a fast draft model to propose multiple draft tokens, which are then verified by the target LLM in parallel with single forward pass. This method reduces the consuming data transfers in forward passes that involve offloaded weight transfers. Existing methods often rely pretrained weights the same family, but require additional training to align with custom-trained models. Moreover, approaches that involve draft model training usually yield only modest speedups. This limitation arises from insufficient alignment with the target model, preventing higher token acceptance lengths. To address these challenges and achieve greater speedups, we propose SUBSPEC, a plug-and -nd--alydpay method to ac- celerate parameter offloading that is lossless and training-fiee freee SubSpec constructs a highly aligned draft model by generating low-bit quantized substitute layers from offloaded target LLM portions. Additionally, GPU-resident layers and the KV-Cache, further reducing memory overhead and enhance alignment. SubSpec achieves high average acceptance length, delivering 9.1 x speedup for Qwen2.5 7B on MT-BBech (8GB VRAM limit) and an average of 12.5 speedup for Qwen2.5 32B popular generation benchmarks (24GB VRAM limit). The code is available at http://hhbb.ooo/oyoo---/////////
method shares the remaining
Introduction
Large language models (LLMs) [ 37] have achieved widespread popularity in tasks ranging from chat models to code generation. Local deployment of these models on consumer hardware offers compelling advantages: data privacy, potential cost reductions compared API access, freedom for model customization, and direct control over the inference process
The substantial memory requirement is the primary barrier to such local deployment. Popular open source model families like Llama [15 39 Qwen 44 43), and DeepSeek [26] often exceed the memory constraint in common lonsumer-level GPUs (typically ranging from 8GB to 24GB), For
These authors contributed equally to this work
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Qwen2.5 7B with Offlaading (8GB VRAM Limit)
SubSpec
400
Memory Transfer
9.10x
Latency (ms token)
350
Target Compute
300
Draft Compute
Speedup
250
5.61x
200
150
2.83x
100
50
2
0
0
16
32
48
64
Vanilla
EAGLE-2 Qwen2 1.58 SubSpec
# Draft Depth
(no SD)
Figure 1: Impact of draft model characteristics on speculative decoding performance, tested under the MTT-Bench [46] benchmark. Left: Maximal speedup achieved by different draft models varies with draft depth in ree-based speculative decoding. Right: Average inference latency per token for Qwen2.5 7B (&GB GPU memory constraint) different methods with optimal draft depths. SubSpec utilizes higher draft model computation to minimize costly memory transfers of target model parameters.
instance, Llama3.1 8B requires 16GB VRAM, surpassing the capacity of consumer card such as an RTX 3060 (12GB).
common strategy to accommodate LLMs within limited memory is model compression, primarily Quantization reduces model memory demands by encoding weights in low-precision formats (e.g., INT8, INT4); however, this lossy compression inevitably alters model outputs and impairs model quality.
via techniques like quantization [13 25
Parameter offloading presents a lossless alternative, storing inactive layers host memory and streaming them the GPU when needed [20]. Unfortunately. frequent data transfers over the PCIe bus severely throttle throughput. This results in prohibitively long latencies for token generation, often only to two tokens per second consumer cards like the RTX 4090 supporting PCle 4.0. Such latency undermines practical usability and fails to satisfy demands for real-time interaction.
Speculative decoding (SD) has emerged excessive latency due to parameter offloading. SD utilizes smaller, faster draft model to rapidly propose multiple tokens, which are then verified by the larger target model in parallel using single forward pass. Potentially accepting multiple tokens, SD) can substantially reduce the number of expensive forward passes involving target model weight transfers. However, some existing SD approaches [10, 355 achieve notable speedups by relying on smaller pretrained models from the same family that align well with the target LLM. These strategies cannot be directly applied to custom- custom-traines trained models and require additional training for alignment. Other techniques involving draft model training [24] 45) often yield relatively modest speedups when applied on offloading scenarios. This limitation stems from insufficient alignment, leading to average acceptance lengths generally below seven tokens.
promising acceleration technique to mitigate the
Hence, we introduce Substitute Speculative Decoding (SUBSPFC), a plug-and-play and training approach designed to maximize inference throughput for offloaded LLMSs consumer-grade grade devices. SubSpec constructs a highly aligned draft model by sharing GPU-resident weights and KV-Cache with the target model, while maintaining low-bit substitute weights for the offloaded layers. This innovative design ensures exceptionally close alignment between the draft and target models and enables deployment of large models locally with competitive inference speeds. The effectiveness of this approach in reducing data transfer latency is demonstrated in Figure
In this work, our key contributions are as follows:
Our analysis identifies the predominance of the alignment and draft depth of the draft model under parameter offloading scenarios, compared to its speed.

---

We propose a plug-and- and-play and training- free method that constructs highly aligned draft models using shared components and low-bit substitutes, minimizing VRAM overhead.
We introduce refinements to draft tree construction, such as probability sharpening, to effectively leverage deeper trees and boost acceptance length.
We developed efficient system accelerate offloading. Using this system, popular models like Qwen2.5 7B achieve 25 tokens per second on a single consumer GPU across diverse benchmarks (within 8GB VRAM limit), demonstrating over 10x speedups compared to baseline offloading inference.
2 Background
2.1 Parameter Offloading
Parameter offloading is a technique commonly employed by inference frameworks 14 18 21 to manage models whose memory requirements exceed GPU capacity. This strategy primarily stores model parameters in CPU memory, transferring them to the GPU only when required for computation. After the GPU completes computations for specific layer, its parameters may either discarded or overwritten by those required for subsequent layers. Typically, frameworks aim to retain as many model layers possible directly within GPU memory, offloading only the remainder to minimize data transfer overhead.
However, these frequent parameter transfers between CPU and GPU memory introduce considerable latency, as GPU operations often stall while awaiting data. This latency significantly impacts the total inference time, making performance highly sensitive PCIe bandwidth limitations.
While techniques like Deepspeed-inferenee innferenee [3] and FlexGen (34 improve throughput via large batches, such approaches are not suitable for latency-critical online inference, where small batch processing standard. Each forward pass remains constrained by the PCIe bus data transfer bottleneck in such scenarios.
2.2 Speculative Decoding
Speculative decoding 7 [22 accelerates the target autoregressive LLM by generating multiple tokens per iteration, rather than just one. In each iteration, a smaller, faster draft model quickly produces a set of draft tokens. These draft tokens are then evaluated parallel by the larget model (the original LLM being accelerated) with a single forward pass. Tokens confirmed by the target model are then accepted, reducing the total forward passes required to generate the full context. The average acceptance length (r) is the mean number of accepted draft tokens per iteration.
Miao et al. further advanced this approach with tree decoding. This method improves the number of lokens accepted per iteration while conserving computational efficiency. Tree decoding maintains a hierarchical token structure instead of parallel beams. Multiple branching token paths are flattened and evaluated in one forward pass. Positional encodings and attention masks are modified to preserve have built on this foundation. These works developed advanced tree-based speculative decoding strategies, reporting 2 to 4x faster than standard autoregressive decoding when no offloading is required.
tree structure dependency. Subsequent works 6 10 23 24
Recent methods illustrate the benefits of using SD to accelerate offloading scenarios. SD significantly reduces data transfer overhead by reducing the total number of forward passes required by the target model, without losing quality. For example. SpecExec (35] demonstrates that speculating and verifying with a larger budget (from 128 2048 draft tokens) per iteration can achieve higher average acceptance length and additional speedup. In contrast, Dovetail (45) focuses on accelerating smaller LLMs on heterogeneous setups, running Llama-2 2 7B by offloading partial computation the CPU. The authors of Dovetail also trained a draft model larger than EAGLE alignment. These techniques have showcased notable speedups. Such speedups result from choosing mid-sized, accurate draft models and speculating deeper token trees. These approaches either assume access to a compatible smaller model in the same family or require fine-tuning to align draft and target distributions.
[3 to achieve better
3

---

3 Analysis of Speculative Decoding Speedup in Offloading Scenarios
3.1 Theoretical Speedup Analysis
This subsection derives the theoretical speedup of speculative decoding (SD) over autoregressive decoding (AR). The goal is to clarify the factors governing SD performance and highlight the distinct optimization challenges posed by standard (model fully GPU-resident) versus parameter-offloddle inference scenarios.
First, we establish the original time required for the token generation of the AR-based target LLM. The total time An required to generate A tokens using autoregressive decoding is directly proportional to the number of tokens, as each token necessitates one forward pass of the target model:
AR = N |Target:
where Target represents the latency of a single forward pass of the target model.
In contrast, the total time SD to generate N tokens using speculative decoding (SD) is given by:
TW N- D. Draaft + Y Taarget
1 ID+1
Dr is the latency of a single draft model forward pass, and D is the draft depth. For each iteration, the draft model runs DD forward passes to generate draft token sequence tree of depth DD (as to "speculate"). The target model then performs a single forward pass over all draft tokens, checking their correctness (as to "verify"). The factor 7 reflects the relative cost of this parallel verification compared to normal AR forward pass (Target (lypically ranging from < 7 S 2). The term т, known as the average acceptance length, denotes the mean number of tokens accepted per iteration (potentially including a bonus token derived from the final accepted token, thus 1 < t S D + 1).
The theoretical speedup is then the ratio T TVY Combining Equations and ( yields:
TW
T ..aarge
1 1
(3)
T:Y
D DDrrff + Y Target
D-EDeaft brarget
+
Equation 3 reveals key trade-off when selecting the draft model and its draft depth (D). In standard settings, both the target latency Target and the speculation overhead D DDrrftt impact the denominator. Increasing draft depth (D) to improve J potentially must therefore be balanced against this linear rise in speculation overhead. This balance typically favors smaller, faster draft models (f Draft Target)) with moderate D. though such models often offer lower alignment with the target, potentially capping the achievable T.
Conversely, this optimization landscape changes dramatically in parameter offloading scenarios where data transfers significantly increase tarrget- Here, the relative impact of the speculation latency (D- tDraft) diminishes against the target verification cost ( orgget) Maximizing the average acceptance length (T) thus becomes crucial to minimize the frequency of expensive target model forward passes. This priority favors draft models with superior target alignment and contextual quality, even if their Draft is larger, as reducing calls to the costly target model results in greater overall speedup.
3.2 Empirical Validation and Motivation for Efficient Draft Model Generation
To empirically demonstrate these contrasting dynamics of speculative decoding (SD) in standard versus crrameter-offloadiinn scenarios, we present an illustrative evaluation with Qwen2.5 7B as the target model. Figure 2 showcases the performance of two representative existing draft model types: EAGLE-Qwen2..0 moderately-siee sized model from the same family that might offer better intrinsic alignment. Figure also provides an early glimpse of SubSpec, our proposed method, which will be detailed in the next section.
[23), a smaller, generally faster draft model, and Qwen2.5 1.5B [43),
"The draft model weights for Qwen2.5 were obtained from http:://hnggiiggieeccoooooogooop 5. Instrucct.

---

30
EAGLE-2
Throughout (takens/s)
80
Throughput (tokens/s)
Qwen2255 1.58
Avg. Accept Length
25
SubSpec
60
20
15
Baselina: 43.07 tokenals
10
10
20
Baseline: takens/s
16
32
48
64
16
32
48
64
16
32
48
64
# Draft Depth
# Draft Depth
# Draft Depth
(a) Average acceptance length (1)
(b) Standard (w/o offloading)
(c) With offloading (8GB limit)
vs. draft depth (D)
Comparative performance of Qwen2.5 7B using SD with different draft approaches across varying draft depths, tested under the MTT-Bench benchmark.
Figure 2:
the faster EAGLE-Qwen2.5 draft model performed best in standard settings due its low overhead, while the better-aligned Qwen2.5 1.5B model achieved superior speedup in the offloading setting, where large (Target dominates due obtaining higher average acceptance length (7). These empirical findings highlight a critical challenge: maximizing SD benefits in offloading scenarios demands highly aligned draft models. This then raises practical question for us: How can we efficiently obtain such highly aligned draft models, particularly for custom or fine-tuned LLMs?
The results presented in Figure 2 confirm our theoretical speedup analysis (Equation 3h:
While open-source model families offer pretrained models of various sizes, allowing smaller versions to serve as potentially efficient and aligned draft models, this option is often unavailable for tittone- fine-tuned LLMs. Creating sufficiently aligned draft model for such custom models typically necessitates additional training or distillation. These processes introduce further costs (computation, memory, data, and time), preventing widespread SD adoption in many real- world deployments.
To bridge this gap, propose practical and efficient alternative: utilizing low-bis quantized version of the target LLM itself as the draft model, which remains fully GPU-resident. This approach capitalizes efficient data-free quantization techniques, eliminating the need for training datasets and resource-intensing training runs. Detailed later in Section 4 our method also incorporates weight and KV-Cache sharing, significantly reducing VRAM overhead while enhancing draft model alignment.
Substitute Speculative Decoding (SubSpec)
We introduce Substitute Speculative Decoding (SubSpec), novel method enabling efficient LLM inference on consumer-grand GPUs, particularly when model weights exceed available GPU memory. SubSpec achieves this by constructing highly aligned, fully GPU-resident draft models and performing tree-based speculative decoding, constructing deep draft trees with optimizations.
SubSpec constructs the draft model with quantized 'substitute" layers for the offloaded portions of target model, while GPU-resident layers and the KV-Cache are shared between the draft and target. This design creates a draft model that is highly aligned with the target model. By constructing deep draft token trees with such model, obtains extremely high average acceptance lengths (T), significantly reducing the expensive data transfers, boosting overall inference throughput. The subsequent sections detail the draft model construction (Section 4.1). adaptations for draft tree construction (Section 4.2) and complementary performance optimizations (Section
4.1 Draft Model Construction
SubSpec employs three synergistic strategies to ensure that the draft model remains entirely on the GPU. These strategies involve using substitute weights, sharing GPU-resident layers, and employing shared KV-Cache, illustrated in Figure 3
5

---

Target Model (w/ Offloading)
1. Prefill (Target
Layer
Layer
Layer
Layer
left
keys
Speculate (Draft xD)
aay
keys
the
bit
Shared
Substitute
left
mry
keys
si
the
office
Weight
3. Verify (Target
Layer
Layer
Layer
left
keng
hore
office
Draft Model (GPU-Resident)
4. Update
ma
Weight
Almaady
Weight
CPU-GPU
Generated Target Model
Genersted by Draft Modal
an GPU
on GPU
on CPU
Transfer
Figure 3: Left: Draft model architecture of SubSpec. SubSpec maintains additional low-bit substitute weights to keep the full draft model on the GPU. Right: Shared KV-Cache generation pipeline of SubSpec. The draft model reuses the KV-Cache of the target model to achieve better alignment and memory efficiency. This illustration serves as a simple sequential demonstration, while in practice, maintain flattened token tree for tree decoding.
Quantized Substitute Weights for Offloaded Layers. A core principle of SubSpec involves replacing the offloaded layers the target model with lightweight, low-bit 'substitute' layers within the draft model. These substitutes reside entirely the GPU and approximate the functionality of their corresponding target layers. For example, layers and 3 on the left side of FigureBlor of the target model are offloaded and the corresponding low-bit substitutes of the target layers utilized for the GPU-resident draft model. We generate these substitutes using fast, data-free quantization methods (e.g., HQQ [5]. HIGGS (28 which require minimal processing time (under minutes for 7B to 70B parameter models on a single consumer GPU). During inference, these quantized layers enable rapid execution through highly optimized low-bit GEMM kernels [6).
GPU-Resident Layer Sharing. The draft model reuses target model layers that remain in GPU memory. For example, the weight of layer one on the left side of Figure 3 GPU-resident and shared by both the target model and our draft model. This sharing strategy maximizes GPU resource utilization and inherently improves the alignment between the draft and target models, given that identical weights are employed for these shared layers.
Shared KV-Cache. The structural similarity between the draft model of SubSpec and the target model allows for unified KV-Cache. This sharing approach yields significant advantages: it halves the KV-Cache memory footprint compared to using separate caches and enhances alignment by ensuring that both models operate on identical contextual history. Furthermore, sharing the KV-Cache eliminates the need for a distinct prefilling phase for the draft model, directly contributing faster overall inference. The demonstration of this pipeline is illustrated on the right side of The draft model extends new KV-Cache values on speculation, which the target model then overwrites during verification ensure identical results.
Figure 3
4.2 Optimized Draft Tree Construction
Constructing Deep Context-Aware Dynamic Draft Tree. The high alignment of the SubSpec draft model (detailed in Section 4.1 enables the exploration of deeper draft trees to achieve higher average acceptance lengths (T). Our sampling approach thus extends established context-aware dynamic draft tree methods like EAGLE-2 24 and SpecExec (35 to support these greater depths.
A context-aware dynamic draft tree is built by iteratively generating draft tokens with the draft model over D future time steps. In each of these D forward passes, all leaf nodes are input to the draft model, each yielding probability distributions for the potential next tokens. The score for each potential next token is cumulative product of its conditional generation probability (from the draft model) and its parent path score. The top-k tokens with the highest scores are selected to form the new k leaf nodes for the subsequent time step. This iterative procedure produces draft tree of depth D, presenting & D draft tokens for target model validation (not including the root token).

---

Golden: "They are a subset text data
Draft with
Draft tree with draft temperature
draft temperature
(Original)
(Draft Sharpening)
(0.188)
(0:041)
9000
(0
0:39
0.4
6,8001
6,817
0.12
8,009]
Correct
're
franco
on [0.112)
vast (0.058)
Correct
trared
Ieey
Trom C0O7)
tokens
(0.15)
tokens
10.00011
Correct path stopped growing due to low cumulative probability.
massive (0.04)
Figure 4: Demonstration of the false positive path issue (with k = 2 for simplification). Correct draft tokens may dropped due to lower cumulative probability score. The number the parentheses in each token denotes the cumulative probability score.
Addressing Cumulative Probability Divergence. While exploring substantially deeper draft trees than typically used in prior work (often D < 7) showed promise for increasing т, construction for greedy decoding (target temperature initiated by less probable tokens could accumulate an erroneously high overall likelihood through high-probability subsequent selections. This phenomenon
0) revealed subtle issue. We observed that paths
lead falss-ppssttvve paths that achieve a higher cumulative probability than the genuinely optimal path (illustrated Figure 4 Such divergence is problematic in greedy decoding, as it can misguide path selection, potentially causing early termination of the correct sequence and thereby limiting the achievable T.
Draft Probability Sharpening. To counteract this divergence under greedy decoding, we employ draft probability sharpening. This technique involves applying low temperature (= 0.2) to the output distribution of the draft model before caleulating cumulative probabilities for tree sampling. Such sharpening makes the probability distribution more peaked, reducing the probability mass allocated tokens with lower initial probabilities. Further analysis is shown Appendix D
4.3 Complementary Performance Optimizations
Beyond the core SubSpec framework for draft construction and sampling, two complementary techniques are integrated further enhance performance and efficiency:
Asynchronous Data Transfer. We mitigate the computation cost of the target layers by overlapping
computation with data movement. While the GPU processes the current layer, the parameters for the next required offloaded layer are concurrently prefetched from CPU memory. Unlike some prefetching strategies limited adjacent layers within the same decoder block, our implementation operates across decoder layers, maximizing potential computation hiding. Furthermore, these prefetched layers are loaded into the same reused memory regions to avoid increasing peak memory usage. This asynchronous data transfer technique effectively conceals the computation time of the SD verification step during forward passes of large draft trees.
Chunked Prefill for Long Contexts. Prefilling long input prompts can lead to substantial peak GPU memory usage because intermediate activation sizes scale with context length, restricting the number of target model layers that can remain GPU-resident. To address this, we employ chunked prefill, where the input is segmented into Exed-length chunks (e.g., 256 tokens), and the KV-Cache is built incrementally. This method significantly reduces peak memory during the prefilling phase. While Sarathi-Serve serving, our adaptation specifically focuses on curtailing peak memory of the prefilling phase for long contexts to maximize the GPU residency of target model layers, further enhancing overall inference efficiency.
22 introduced this approach primarily to minimize pipeline bubbles in token
7

---

Table 1: Throughputs and average acceptance lengths by using different draft models. L31 represents Llama-3. -.amm-3.1-IIstttee 1-Instruct, L32 represents Llama-332-Instruee Q represents Qwen-2.5 None represents vanilla autoregressive decoding is used. The number in each parentheses below the target model name denotes the restricted VRAM limit.
HumanEval
GSMBK
Alpaca
CNN/DM
Mean
Target
Draft
tokens/s
tokens's
tokens/s
tokensis
tokens's
tokens's
Temperature D
None
2,40
2.39
2.40
2.40
1
2.34
2.39 (1.00 (1.00x)
L31
EAGLE-2
7,56
3,90
8.58
4.43
8.05
4.15
7.59
3,90
6.45
3.34
7.65 (3.20x)
3.95
(8GR)
L32
15:14
20.23
15.71
11.54
17.23 (7.22xx
13.61
SubSpec
24.29
28.35
28.09
31.63
27.78
31.55
22.13
25.38
22.60
31.39
24.98 (10.47x)
29.66
None
2.77
2.77
2.77
2.77
2.34
2.68 (1.00x)
Q 7B
EAGLE-2
8.00
4.08
3.85
3.47
x)
(8GB)
Q 1.5H
18.78
12.27
28.29
21
22,86
17.59
12:37
9.47
9. 34
7.88
19.73 (6.61
13.82
SuliSpee
25.35
27.08
33.48
34,77
33,04
34.18
22.30
23.19
21.28
26,33
27.09 (10.10%)
29.11
None
1.22
1.22
1.22
1
1.22
1
1.17
1.20 (1000)
14B
O 1.5B
8.17
10.81
16.20
21.45
12.58
16.54
6.46
8.43
4.40
6.40
9.56 (7.92x)
12.73
(12GB)
SubSpec
12.05
26.36
15.70
33.76
15.44
32.94
10.34
21.79
9.27
24.08
12.56 (10.4x)
27.79
None
0.52
0.52
0.52
1
0.52
1
0.50
0.52 (1.00x)
Q 7B
3,68
12:55
6.09
20.73
5.30
17.91
3.20
10.74
2.15
8.32
4.08 (7.86 x)
14.05
(24GB)
1.5B
4.49
10.87
8.22
19.90
6.74
16.17
3.61
8.62
2.
6.46
(9.
12.40
SulSSpee
6.33
27.53
7.58
32.70
7.96
33.66
24.50
4,80
26.48
6.50 (12.50
28.97
Temperature 0.6
None
2,40
2.39
2,40
2.40
1
2.39 (1 (0X0)
L31
EAGLE-2
7.38
3,81
8.30
4.29
7.60
3.92
7.49
3,86
6.19
3.21
7.39 (3. 10x)
9.96
(8GB)
L32 IB
12.30
9.69
21.37
16.82
15.88
12.42
12.80
9.83
8.16
6.93
14.10 (5.91x)
11.14
SabSSpee
14,62
17.58
22.59
26.04
16.51
19.54
13.17
15.46
11.61
15.37
(6.58x)
18.80
None
2.77
2.77
1
2.77
1
2.77
1
2.34
1
2.68 (1.00x)
1
Q (8GB)
EAGLE-
7.42
3.45
8.49
3.96
3.76
6.47
3.01
5.69
2.66
7.23 (2.69
3.37
Q 1.58
13:19
10.43
22.96
18.06
19.19
15.02
10.59
8.21
7,63
6.44
14:71 (5.48k)
11.63
SubSpee
15:92
17.09
29.13
30,57
23.89
25.32
14.43
14.98
10.15
12.07
18.70 (6.97%)
20.00
1.22
1.22
1.22
(1.00x)
Q 14B
O 1.5B
6.53
8.62
11.90
15.73
9.87
12.99
5,43
7.09
3.77
5.42
7.50 (6.21
9.97
(12GB)
SuhSpee
6,90
15.22
12.17
26.74
9.56
21.02
6.14
13.05
4,40
10.90
7.83 (6.49 x)
17.39
0.52
32B
Q 7B
2.64
9.08
4.88
16.67
3,90
13.24
2.49
8.37
1.76
6.61
3.13 (6.03xx
10.79
(24GB)
1.5B
3,71
9.03
5.72
13.93
4.80
11.61
2,94
7,06
1.94
5,26
3.82 (7.35x)
9.38
SubSpec
3.74
26.54
19.32
14.16
13.33
(7.82x)
5 Performance Evaluation
5.1 Evaluation Setup
Evaluation Benchmarks. We evaluated performance across five diverse generative tasks, consistent with the benchmarks from EAGGLE (23] and Spec-Bench (41). These tasks included multi-turn conver- sation (MT-Bench [46)), code generation (HumanEval (9). mathematical reasoning (GSM8K [I],
instruction following (Alpaca (36)), and summarization (CNN/Daily Mail (31).
Hardware and Simulated Environments. All experiments were run on a system with an RTX 4090 GPU, an Intel 17-13700K CPU, a PCIe-4.0 x16 bus, and 128GB of DDR5 RAM. GPU memory utilization during evaluations was programmatically restricted to 8GB, 12GB, and 24GB VRAM capacities to simulate diverse consumer device environments.
Comparative Methodology and Parameters. We compared the end-to-end speedup of SubSpec against EAGLE-2 and chat models from the Qwen2.5 (7B, 14B, 32B) and Llama3. (8B) families. Evaluations used a batch size of under both greedy (target temperature 0) and stochastic (target temperature = 0.6) generation. For fair comparison, all methods used an identical context-aware dynamic draft tree algorithm without additional tree pruning techniques. A portion of the target model decoder layers was kept resident on the GPU within the VRAM limits. All SD methods were evaluated 20 identical samples, randomly selected from each dataset. The baseline (offloading with no SD) used the initial five samples due to its significantly longer runtime.
The key parameters were configured as follows: The low-bit substitute layers in SubSpec were quantized to bits with a group size 64 using HQQ. EAGLE-2 used its default published parameters
'The draft model weights for Llama3.1 were obtained from their official repository.
8

---

= D - 6). For SubSpec and the smaller pretrained draft models, the top-k value of tree construction was set = Their optimal draft depths (D), identified through the grid search reported in Section 3.2 (results shown in Figure 2 were 'D = 48 for SubSpec and 'D = 32 for the pretrained models. While further parameter tuning might yield additional improvements, such exhaustive optimization was beyond the scope of this research. Chunked prefill was also applied to prevent out-of-memory (OOM) errors and maximize the number of decoder layers on the GPU.
To better reflect typical real-world usage, all draft models were executed using torch compi le with the vax-autotung configuration. static KV-Cache with context length of 2048 tokens was applied consistently across all methods. The comparative results are summarized in Table 1
5.2 End-to-end Performance
The results in Table [ demonstrate the effectiveness and robustness of SubSpec. SubSpec consis- tently achieved average acceptance lengths (T) near 30 across tasks, delivering the highest average throughput. This performance translates to a speedup of 10x 12.5 between different model sizes, underscoring the broad applicability and significant performance gains offered by SubSpec. We also evaluated the performance of the reasoning models on additional reasoning benchmarks listed in the Appendix G
Further highlighting its efficiency, SubSpec achieves additional 30% to 50% speedup compared to smaller draft models from the same family as the target model, without any additional training. This advantage underscores the critical role of the enhanced draft alignment of SubSpec in accelerating offload scenarios. All SD methods showed reduced performance in stochastic settings (target temperature 0.6). For SubSpec, this meant decrease in speedup of approximately 60%. Despite this, SubSpec maintained a considerable speedup of 5.8x to 7.8x, showcasing its resilience and sustained effectiveness even under less favorable generation conditions. We achieved low standard deviation of 0. 001 tokens/sec on five independent runs of SubSpec on MTT-ench benchmark.
5.3 Ablation Study
Table 2: Ablation study of SubSpec component contributions for accelerating Qwen2.5 7B target model on MT-Bench under greedy decoding (8GB VRAM limit). Performance is shown the components were added cumulatively. The final row represents the complete SubSpec system.
Finally, we performed an ablation study on MT- Bench under an 8GB VRAM constraint to assess the impact of individual SubSpee components. The results are detailed in Table 2 A base- line configuration of only implementing the core concept of SubSpec (a quantized, GPU-resident draft model using only 'substitute' layers for of- floaded portions and shared GPU-resident target model layers), achieved 7.05 x speedup (19.54 tokens/s).
Method
tokens/s
Substitule and layer sharing
19.54 (7.05x)
23.07
+ Shared KV-Cache
21.99 (7.94x)
25.14
Draft prob. sharpening
23.66 (8.54x)
27.08
+ Asyuc data transfer
25.35 (9.15x)
27.08
Sequentially integrating additional enhancements of shared KV-Cache, draft probability sharpening, and asynchronous data transfer yielded further performance gains. Each of these components contributed an approximate 7% to 13% increase in throughput. The complete SubSpec system, incorporating all optimizations, ultimately delivered a 9. 151 speedup and a throughput of 25.35 tokens/s. These results affirm the individual and collective efficacy of the components of SubSpec.
6 Conclusion
This paper addressed the challenge of efficiently performing the inference of large language models on memory-constraine consumer GPUs using parameter offloading. Our analysis confirmed that highly aligned draft model is crucial for speculative decoding to accelerate parameter offloading effectively. We introduced SubSpec, a novel lossless and training- technique based on this insight. SubSpec constructs aligned draft model by utilizing low-bit substitute layers for offloaded portions of the target LLM while sharing GPU-resident components. Evaluations demonstrate that SubSpec is robust across various model sizes and benchmarks under realistic memory limits, achieving substantial average speedups of 10x to 12.5x compared to baseline offloading inference. These results significantly advance the feasibility of deploying large, high-quality LLMs locally on widely available consumer hardware.

---

Acknowledgments
The authors would like thank ASUS for their generous support in providing the computing resources necessary for this work.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303. 230.07340.. 2023.
[2] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gula- vani, Alexey Tumanov, and Ramachandran Ramjee. Taming throughput-latency tradeoff in llm inference with sarathi-serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 117-134, 2024.
[3] Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, al. Deepspeed- inference: enabling efficient inference of transformer models at unprecedented seale. In SC22: International Conference for High Performance Computing, Networking. Storage and Analysis, pages 1-15. IEEE, 2022.
[4] Anthropic. Claude:
A conversational ai assistant, 2023. URL http:ttss://ww /ttt:::://ww.atthoppicc.ooo com/ claude Large Language Model. Version 1.0. Accessed: 2025-03-13.
[5] Hicham Badri and Appu Shaji. Half-quadratic quantization of large machine learning models, November 2023. URL hattps://mobiusml..hhh
[6] Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng. Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple Ilm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10771 2024.
[7] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint r3Xiv:2302..01313 2023.
[8] Longze Chen, Renke Shan, Huiming Wang, Lu Wang, Ziqiang Liu, Run Luo, Jiawei Wang, Hamid BlinejadRRokny, and Min Yang. Clasp: In-context layer skip for self-speculative decoding, arXiv preprint arXiv:2505.241100
2025.
[9] Mark Chen, Jerry Tworek, Heewoo Jun. Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Broekman, et al. Evaluating large language models trained on code. arXiv preprint wxXiv:2107.03321 2021.
[10] Zhuoming Chen, Avner May. Ruslan Svirschevski, Yu-Hsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. Sequoia: Scalable and robust speculative decoding. Advances in Neural Information Processing Systems, 57:112531-129560 2025.
[11) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
[12] DeepSeck-AI. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025. URL /ttpps////a/ www.org/bbboo abs/ 12948
[13] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17322. 2022.
[14] Georgi Gerganov. agerganov/llama.cp:: Port of facebook's Iaama model in c/c++. hetphubb.cca/gggggaao///laaa cpp. 2023.
https
[15] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
10

---

[16] Han Guo, William Brandon, Radostin Cholakov, Jonathan Ragan-Kelley, Eric Xing, and Yoon Kim. Fast matrix multiplications for lookup table-quantized llms. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 12419-12433, 2024.
[17] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https: /
[18] HuggingFace. Hugging face accelerate. https: httpp///mugnnnaaeecocooooosccacs index 2022.
[19] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024. URL htthhtpps// hhttps///axxi> //axxvv org /bs/ 2403.07974
[20] Xuanlin Jiang, Yang Zhou, Shiyi Cao, lon Stoica, and Minlan Yu. Neo: Saving gpu memory crisis with epu offloading for online IIm inference. arXiv preprint arXi::2411.01142 202,
[21] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[22] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274-19286. PMLR, 2023.
[23] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE: Speculative sampling requires rethinking feature uncertainty. In International Conference on Machine Learning. 2024.
[24] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-2: of language models with dynamic draft trees. In Empirical Methods in Natural Language Processing. 2024.
Faster inference
[25] Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-awe aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
[26] Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint ar arXiv:2405.0441
2024.
[27] Fangcheng Liu, Yehui Tang, Zhenhua Liu, Yunsheng Ni, Duyu Tang, Kai Han, and Yunhe Wang. Kangaroo: Lossless self-speculative decoding for accelerating Ilms via double early exiting. Advances in Neural Information Processing Systems, 37:1 1946-11965, 2024.
[28] Vladimir Malinovskii, Andrei Panferov, Ivan Ilin, Han Guo, Peter Richtárik, and Dan Alistarh. Pushing the limits of large language model quantization via the linearity theorem. arXiv preprint urXiv:2411.121 17525, 2024.
[29] Michael R Metel, Peng Lu, Boxing Chen, Mehdi Rezagholizadeh, and Ivan Kobyzev. Draft on the fly: Adaptive self-speculative decoding using cosine similarity. arXiv preprint arXiv:2410.01025 2024.
[30] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang. Xiaoxiang Shi, et al. Specinfer: Accelerating generative large language model serving with tree-based speculative inference and verification. arXiv preprint arXiv:2305.0978. 2023,

---

[31] Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. Abstractive text sum- marization using peaeeenceto-sqqueeee mns and beyond. arXiv preprint arXiv: 2016.
[32] Mathematical Association of America. Aime, February 2024.
URL https://
tttofproblemsolviing wwt.wwoioiinneeeeccceeeeeeeeeeee wwwp:eee.ppPAAAMMPPPPaaPPP Problemss and Sadd___oltioos////
[33] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petry, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman, Gpqa: A graduate-level google-proof q&a benchmark, 2023. URL httph: http:///axxvvv.o//aaaaee
12022
[34] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning, pages 31094-31116. PMLR, 2023.
[35] Ruslan Svirschevski, Avner May, Zhuoming Chen, Beidi Chen, Zhihao Jia, and Max Ryabinin. Specexec: Massively parallel speculative decoding for interactive Ilm inference on consumer devices. Advances in Neural Information Processing Systems, 37:16342- 2025.
(36] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Perey Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction- following model. Stanford Center for Research on Foundation Models. https://crfi. stanford. edw/2023033/1 3/alpaca. html, 3(6):7, 2023.
[37] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
[38] Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL httpps://wennmm. github. io/ii//bbll hotpl////g/g///////////0
[39] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0922: 2023.
[40] Heming Xia, Yongqi Li, Jun Zhang, Cunxiao Du, and Wenjie Li. Swift: On-the-fly fly self- speculative decoding for Ilm inference acceleration. arXiv preprint arXiv:2410.06916, 2024.
[41] Heming Xia, Zhe Yang. Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. arXiv preprint arXiv:2401.074 00851,, 2024.
[42] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint a4Xiv:2407.1067 2024.
[43] An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, Junyang Lin, Kai Dang, Kexin Yang, Le Yu, Mei Li, Minmin Sun, Qin Zhu, Rui Men, Tao He, Weijia Xu, Wenbiao Yin, Wenyuan Yu, Xiafei Qiu, Xingzhang Ren, Xinlong Yang, Yong Li, Zhiying Xu, and Zipeng Zhang. Qwen2.5-1m technical report, 2025. httpss::: httpit//s/://ii.....// org abs/ abs/2501.... 15383
[44] Hanling Yi, Feng Lin, Hongbin Li, Peiyang Ning, Xiaotian Yu, and Rong Xiao. Generation meets verification: Accelerating large language model inference with smart parallel auto-correct decoding. arXiv preprint 240. 2024.
[45] Libo Zhang, Zhaoning Zhang, Baizhou Xu, Songzhu Mei, and Dongsheng Li. Dovetail: A cpu/gpu heterogeneous speculative decoding for Ilm inference. arXiv preprint arXi::44/2.18334, 2024.
[46] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang. Joseph E. Gonzalez, and Ion Stoica. Judging LLMM-as-a-judge with MT-Bench and chathot Arena. arXiv preprint arXiv:2306.0566 2023.
12

---

[47] Yue Zheng, Yuhao Chen, Bin Qian, Xiufang Shi, Yuanchao Shu, and Jiming Chen. A review on edge large language models: Design, execution, and applications, 2025. URL https
7/arxiv. argg/bbs/4100 11845
13

---

A Limitations
Minimum GPU Memory Requirement. SubSpec has higher minimum GPU memory require- ment. A general prerequisite for speculative decoding to work effectively is that the entire draft model must be GPU-resident. SubSpec architecture necessitates maintaining low-bit substitutes for offloaded target model layers. An approximate 7.1GB minimum GPU memory is required for Qwen2.5 7B, thus fewer layers of the target model can remain GPU-resident compared to some alternative speculative decoding methods. Despite this, SubSpec demonstrates superior throughput in all benchmarks.
Quantization Granularity. Our research only experimented with 4-bit quantization for substitute layers. Although this might affect draft model alignment, more aggressive methods (e.g.. 2-bit or 3-bit quantization) could further reduce VRAM demands. Such VRAM savings could permit strategic memory reallocation, for instance, retaining critical draft or target model layers at full GPU precision. A thorough exploration of these trade-offs between draft quality, VRAM usage, and performance represents an important direction for future research.
Applicability to Model Architectures. SubSpec mainly suits dense LLM architectures. Applying SubSpec to alternative architectures, such as Mixture-of-Expers (MoE) models, requires further adaptation and research.
B Extended Discussion
B.1 Comparison with Standalone Quantized Models
Table 3: Throughput comparison of various quantization methods for the 1lama3.1 8B model on the MTT-Bench (8GB VRAM).
While standalone 4-bit quantized model may fully fit on GPU and eliminate the offloading bottleneck, this approach inevitably alters outputs and degrades model quality. In contrast, SubSpec avoids this accuracy trade-off entirely.
Model Configuration
tokens/s
SubSpec makes parameter offloading practical. As shown in Table 3, this method accelerates naive offloading from an unus- able 2.4 tokens/sec use on consumer-grant hardware. We therefore position Sub- Spec simple, training-free solution for users who refuse compromise on the model quality.
Original (fp16)
2.40
GPTQ 13
(int4)
58.87
acceptable 24 tokens/sec for interactive
AWQ 25
(int4)
52.32
HQQ S (int4)
135.84
SubSpec
24.29
B.2
Why Tree-Based Speculative Decoding
This research focuses tree-based speculative decoding (SD) because tree-based SD generally achieves higher average acceptance lengths (T) than sequential or self-speculation methods [23] 30 41]. Besides, high values are crucial to minimize expensive forward passes of the target model, especially in offloading scenarios that involve significant data transfer overhead.
B.3 Future Outlook and Technological Advancements.
Forthcoming interconnect advancements (e.g., PCIe 5.0 and 6.0), along with the continuous progress on model compression methods and kernel optimizations, are anticipated to further enhance inference performance. Each PCIe generation roughly doubles raw bandwidth, halving expensive target model data transfers. While faster PCIe reduces target model latency, potentially increasing the relative cost of speculation iterations, concurrent GPU computation and kernel efficiency improvements expected to accelerate speculation proportionally. Consequently, SubSpec is projected to retain its relative speedup advantage standard autoregressive decoding. These combined technological trends promise a progressive reduction in the performance gap between offloaded and non-offloaded LLM inference.
Forr the GPTQ and AWQ methods, we loaded their corresponding pre-quantized weights directly from the Hugging Face Hub.
14

---

C Related Works
SpecExec [35]. SpecExec speculative decoding method that introduces a pruning strategy during tree construction and an early-exit mechanism to reduce speculation time. However, these features increase computational complexity and hinder the application of torch. compi le due to dynamic shapes from pruning.
Table 4: Performance of SpecExee using Llama3. 8B as target model and Llama3.2 IB draft model on MT-Bench (greedy decoding). "Budget" denotes the number of tokens concurrently verified, "VRAM" denotes the peak GPU memory.
We performed parameter sweep for SpecExec the MT-Bench benchmark, with results in Table While SxecExec exceeds the 8GB VRAM constraint, SubSpec achieves 76% speedup without requiring an additional draft model.
Budget
VRAM (GB)
tokens/s
64
8.38
11.75
8.64
128
8.43
12.21
9,42
256
8.55
13.02
10.79
512
8.79
13.77
12.40
1024
9.27
12.95
12.93
Dovetail 45]. Dovetail is a speculative decoding
method to accelerate LLM inference on consumer-grade devices by offloading portions of the target computation to the CPU, with the draft model on the GPU. Unlike GPU processing, CPU computation for verification scales linearly with the number of draft tokens. This trait typically restricts the amount of verifiable draft tokens per iteration to maintain optimal speedup. For instance, Dovetail used only 16 draft tokens for verification, whereas our proposed SubSpec verified 288 draft tokens in parallel.
Self-speculative decoding builds draft model by reusing layers from target model to reduce parameter overhead and improve alignment. While many approaches add recent works propose training-free layer 29 40 This distinction highlights an opportunity for hybrid solutions. Future work could explore the combination of SubSpec with layer skipping to produce draft model with lower VRAM requirement and faster inference.
Seef-Speculative Decoding.
new parameters and require extra training [6] 24
skipping methods 8
D Analysis of Draft Probability Sharpening
Table discusses the results of varying draft temperatures among draft model types. The first result (denoted Self in the second row) performs speculative decoding using the tar- get model itself. i.c., Qwen2.5 7B, as the draft model. Although this setup does not make sense from both memory and speedup perspectives, the experiment clearly demon- strates the cumulative probability divergence problem mentioned in Section [.2] which
Table 5: Average acceptance length (r) for various draft models and temperatures with Qwen2.5 7B tar- get model on MT-Bench (greedy decoding), where a draft temperature of 1.0 represents the baseline with- out draft probability sharpening.
Temn.
0.4
Qwen2.5 7B (Self)
34.50
33.03
31.95
29.64
27.79
26.57
EAGLE22
3,42
3.62
3.70
3.71
3.76
3.77
Qwen2.5 1.5B
11.86
12.20
12.48
12.36
12,42
12.51
SuhSpec
27.08
27.19
27.16
26.09
25.29
23.92
shows that even the same model cannot correctly predict all the tokens when using the default draft tree method.
By lowering the draft temperature, the average acceptance length (r) increases from 27.79 to 34.50, showing that this aggressive sharpening effectively counters the "false positive" path issue. For SubSpec, 7 also increases from 23.92 to 27.08. This method does not yield a better due to the lower alignment of the pretrained models and EAGLE-2.
15

---

E Verification Pipeline with Asynchronous Data Transfer
Figure illustrates the execution timelines of the verification step in vanilla autoregressive decoding, speculative decoding, and speculative decoding with asynchronous data transfer. The bottom diagram (c) illustrates how asynchronous data transfer works. This method overlaps GPU computation (Q. K) with the data transfer of the subsequent layer (Loading K, V weights to the GPU) to hide the higher verification computation time for speculative decoding.
(a) Vanilla Autoregressive Decoding
GPU
CPU
Load Q
Load K
Load V
...
time
(b) Speculative Decoding
GPU
CPU
Load
Load
Load V
time
(c) Speculative Decoding Asynchronous Data Transfer
GPU
CPU
Load Q
Load K
Load V
time
Figure 5: Execution timelines of the verification step for different decoding strategies
Temporal Analysis: Speculation vs. Verification
For each iteration in speculative decoding, the draft model runs D forward passes generate a draft tree of depth 2 (speculate). The target model then performs single forward pass over all draft tokens, checking their correctness (verify). We compare the average execution time of these two phases, along with the actual obtained throughput, shown in Figure The optimal throughput for SubSpec occurs at D 48, where the speculation and verification times are nearly equal.
100
30.0
90
27.0
Execution Time (%)
(tokensh)
80
24.0
70
21.0
60
18.0
50
15.0
40
12.0
30
9.0
Speculation Time (%)
20
6.0
Verification Time (%)
10
Throughput (tokens/s)
3.0
0.0
4
8
16
32
48
64
Draft Tree Depth
Figure 6: Execution time breakdown (speculation verification) and average acceptance lengths (7)
of SubSpee accelerating 5wen2.5 7B under an 8GB VRAM limit.
16

---

G Performance Evaluation on Reasoning Models
To fürther demonstrate the robustness of the lossless and training- free method, SubSpec, we also
evaluated its performance on reasoning models (DeepSeek-R1 distilled variants [2]. QWQ 38
This evaluation utilized benchmarks widely used in the field: AIME 2024 [32] MATH 500
GPQA Diamond [33, and LiveCodeBench [9]. As shown in Table 6 SubSpec demonstrated
promising results of over 10x speedup on all benchmarks.
Table 6: Speedup ratios and average acceptance lengths (r) for SubSpec compared to baseline (None) on reasoning models under greedy decoding. DSL represents DeepSeek-R -1-eeekk------ 1--Distill-LLamm and DSQ
represents DeepSeek-RR Beieck-R---iiiie Distill-Qwer The number in each parentheses below the target model name
denotes the restricted VRAM limit.
AIME 2024
Math 500
GPQA Diamond
LiveCodeBench
Mean
Target
Draft
tokens/s
tokens/s
tokens's
tokeas/s
tokens/s
DSL 8B (8GB)
None SuhSpec
2.40
1
2.40
2.40
2.39
1
2.40 (1.00))
31.30
38.64
32.39
27.54
34.18
35.69
29.92 (12.49x)
DSQ (8GB)
None
2.77
2.77
2.77
2.76
2.77 (1.00x)
SubSpee
33.45
37.76
35:25
39.55
27.74
31.41
29.94
34.45
31.60 (11:22x
34.72
DSQ 14B (12GB)
None
1.22
1.22
1.21
1.21
1
1.21 (1,00x)
SubSpec
17.38
40.57
17.60
40.96
14.93
35.07
15.73
37.42
16.41 (13.51x)
38.51
DSQ 32B (24GB)
None
0.52
0.52
0.52
1
0.52
1
0.52 (1.00x)
SubSpee
9.56
43.59
9.70
44.00
8.57
39.45
8.47
39.70
9.07 (13.76x)
41.68
QWQ 32B (24GB)
None
0.52
0.52
0.52
0.52 (1.00x)
SubSpec
7.95
36.29
8.11
36.93
6.86
31.54
5.89
27.47
7.20 (13,76x)
33,06
H Quantized Target Model Scenarios
We analyzed the practical trade-off between model preci- sion and performance for an 8-bit quantized target model. As detailed in Table 7, this quantization nearly halves the memory footprint and offloading time relative the 16-bit version. While the process introduces an acceptable loss in accuracy, this configuration increases throughput by 36% with all other configurations held constant. This outcome demonstrates a compelling trade-off, where minor reduc- tion in precision yields a substantial performance gain.
Table 7: Performance of Llama3.1 8B across different SubSpec settings on MT-Bench (greedy decoding, 8GB VRAM).
Config (Target, Draft).
tokens/s
SubSpec (fp16, int4)
24.29
29.66
SubSpee (int8, int4)
33.03
29.76
I Supplementary Evaluation Configurations
We retain as many target model layers as possible directly within GPU memory to reduce the expensive data transfer overhead. The draft models and KV-Cache are default GPU-resident.
The following details the number of GPU-resident layers of the target model for various methods within VRAM limits. Embedding and head layers are default to be GPU-resident:
Vanilla (Baseline Offloading): 11 layers for 7B/8B target models, 15 layers for 14B target models, and 20 layers for 32B target models.
EAGLE-2: layers for 7B/8B target models.
SD with Mid-Size Pretrained Draft Model (1B/1.5B): layers for 7B/8B target models, layers for 14B target models, and 16 layers for 32B target models.
SD with Large Pretrained Draft Model (7B): 3 layers for the 32B target model.
SubSpec: All decoder layers of the target model were offloaded, with their 4-bit quantized substitutes retained on the GPU.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and Section accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

Justification:
We include our theoretical analysis in Section 3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We include the experimental hyper-parameters in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification:
All evaluation codes are the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pippe.cc/pullc//uuiii wwwiii/gguie///eeeeeeoeic CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We include the experiment setup Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too time-consuming
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
20

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We include the compute resources Section 5.1
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://wwrrr..coooooo///ccccaaaa
Answer: [Yes]
Justification:
The research is conducted with the NeurIPS Code of Ethics. conform.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The purpose of this paper is to accelerate the execution of full call-precision large language models running on consumer-level GPUs, without any negative societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The models, benchmarks, and codebase used in our experiment comply with open-source licenses and can be used for scientific research.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All pre-trained models were cited their authors. We reference them in the main text, supplement, and code
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
23

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy buttps://neure wwwpariipp...coooeeeeeeeeeeca for what should or should be described.
24


Reviews:
[{"review_id": "ihpQYrzJOD", "reviewer": "Reviewer_BRmM", "summary": "This paper addresses the significant inference latency of large language models (LLMs) when using parameter offloading to run them on consumer GPUs with limited VRAM. The authors propose Substitute Speculative Decoding (SubSpec), a novel, training-free method to accelerate these offloaded models.\n\nThe core contribution is a technique to construct a highly-aligned, GPU-resident draft model for speculative decoding. This is achieved by creating low-bit quantized 'substitute' layers for the offloaded parts of the target LLM, while directly sharing the layers that already fit in GPU memory as well as the KV-Cache. This design ensures high alignment between the draft and target models without requiring any training. The paper also introduces refinements to the draft tree construction, such as probability sharpening, to maximize the token acceptance rate.", "questions": "Please refer to the weakness part.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n* The paper is exceptionally well-written and clearly structured.\n* The central idea of using a low-bit quantized version of the target LLM as the draft model is both intuitive and highly effective.\n* The paper demonstrates outstanding empirical results, particularly under the greedy decoding setting (temperature = 0).\n\nWeakness:\n\nThe primary weakness of the paper lies in its experimental comparison framework, which may not fully address the most common and practical trade-offs for deploying LLMs on consumer hardware.\n* Missing Comparison with Standalone Quantized Models: The paper frames SubSpec as a solution to the high latency of offloading half-precision models. However, a very common and effective strategy for memory-constrained devices is to use a fully quantized model (e.g., 4-bit) that fits entirely within GPU VRAM, thus avoiding the offloading bottleneck from the start. The quality loss from modern quantization techniques is often considered an acceptable trade-off for the immense speed gain. The paper lacks a direct performance comparison against this strong and highly relevant baseline. It is unclear how SubSpec's performance (e.g., 25 tokens/s for Qwen2.5 7B) compares to simply running a 4-bit quantized version of the same model directly on the GPU.\n* Untapped Potential in \"Quantized-to-Quantized\" Scenarios: Building on the previous point, a more practical application of speculative decoding in resource-limited settings would be to accelerate an already-quantized target model. For example, one could use a highly compressed model (e.g., 2-bit or 3-bit) as a draft for a less compressed, higher-fidelity target model (e.g., 4-bit or 5-bit) that is also fully GPU-resident. This \"quantized-to-quantized\" speculative decoding represents a more realistic optimization scenario for end-users. The paper does not explore this direction.\n\nWithout this analysis, the experimental comparison is incomplete. Demonstrating a clear and favorable trade-off in this more practical context would significantly strengthen the paper's contribution. Addressing this limitation by providing a more comprehensive comparison against strong, quantized baselines would make the work's practical utility much clearer, and I would be inclined to raise my score accordingly.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the very thoughtful and crucial review. We will address the weaknesses in the following:\n\n### 1. Missing Comparison with Standalone Quantized Models\nWe agree that comparing SubSpec to a standalone quantized model is essential. As you suggested, we now include a direct performance comparison against fully GPU-resident 4-bit models. \n\nThe table below shows the throughput of Llama-3.1 8B on a single GPU with an 8GB VRAM limit:\n| Model Configuration | Throughput (tokens/s) | \n| :--- | :--- | \n| Full Precision (fp16) | 2.40 |\n| GPTQ (int4) | 58.87 | \n| AWQ (int4) | 52.32 | \n| HQQ (int4) | 135.84 |\n| **SubSpec** | 24.29 |\n\nWhile a standalone 4-bit model offers maximum throughput by accepting the quality trade-offs of quantization, SubSpec is designed for users who require lossless output that is bit-for-bit identical to the original FP16 model.\n\nFor this high-fidelity use case, SubSpec makes offloading practical. It accelerates naive offloading from an unusable 2.4 tokens/sec to an acceptable 24 tokens/sec for interactive use. This allows users to achieve full model fidelity at a speed that is comparable to some standalone quantized models, all on consumer-grade hardware. We therefore position SubSpec as a simple, training-free solution for users who cannot compromise on model quality.\n\n### 2. Untapped Potential in Quantized-to-Quantized Fully GPU-resident Scenarios\nWe agree it is an interesting direction. However, for a draft with 2-bit substitutes accelerating a 4-bit target, we believe the gains would be modest. A much more efficient draft architecture (e.g., EAGLE) is required to achieve decent speedup on fully GPU-resident cases, as shown in our analysis in Sections 3.1 and 3.2. The effectiveness of speculative decoding hinges on a draft model that is substantially faster than the target. Also, due to hardware and kernel overheads, current 2-bit quantization kernels are not yet fast enough compared to their 4-bit counterparts to create the large speed gap needed for significant improvement.\n\nInspired by this suggestion, we analyzed a different practical trade-off, where the accuracy degradation introduced by 8-bit quantization is acceptable. \n\nConsidering running Llama-3.1 8B on MT-Bench under an 8GB VRAM limit, while quantizing the target model to 8-bit still requires offloading, its memory footprint and offloading time are nearly halved compared to the 16-bit version. Our results show this configuration increases throughput by 36%, presenting a compelling trade-off between precision and performance:\n| Configuration (Target, Draft) | Throughput (tokens/s) | τ |\n| :--- | :--- | :--- | \n| SubSpec (fp16, int4) | 24.29 | 29.66 | \n| SubSpec (int8, int4) | 33.029 | 29.764 |\n\nThank you again for pushing us in these valuable directions. We will incorporate this analysis into the final paper."}, {"review_id": "Cy7QoWcVxH", "reviewer": "Reviewer_fdZL", "summary": "The authors propose SubSpec, an effective speculative decoding method designed for settings where parameter offloading is required. The core idea is to retain a low-bit quantized replica of the model on the GPU as a draft model, while periodically loading the full-precision parameters of the original model to perform lossless speculative decoding. Importantly, this approach does not require any additional training. In addition, the authors introduce several optimizations, such as modifications to the draft tree construction strategy and asynchronous execution, to further improve decoding speed.", "questions": "1. Could the authors provide experimental results comparing different quantization algorithms and target bit-widths?\n\n2. Although admittedly challenging, could the authors consider providing a direct comparison with modern quantized models?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The authors present a clear and practical use case, i.e. deploying larger-scale language models on mainstream consumer GPUs through hybrid optimization strategies.\n\n2. The experimental evaluation is thorough. The authors provide detailed deployment data, instilling confidence in reproducibility. They validate the method across models of varying sizes and settings, demonstrating its broad applicability. The reported improvements are substantial, with speedups of up to 30× in some scenarios.\n\n3. The authors keenly identify a key limitation in applying traditional draft tree strategies to SubSpec: SubSpec employs a significantly more capable draft model compared to traditional speculative decoding setups, where the draft model is relatively weak. As a result, conventional draft tree construction strategies may not be optimal. The authors propose a revised strategy accordingly, offering valuable insight for future draft tree designs.\n\n4. The paper includes ablation studies to quantify the contributions of different components to overall performance gains.\n\n**Weaknesses**\n\nThe discussion of quantization strategies and target bit-widths is somewhat limited. Comparative experiments in this area would strengthen the completeness of the study.\n\nThe primary weakness of the paper lies in the narrow interpretive framing of SubSpec. The method is discussed solely from the speculative decoding perspective, as an offload-aware approach to improving inference efficiency. However, SubSpec can also be viewed through the lens of quantization compensation: a low-bit quantized model is augmented by periodically verifying its predictions using a full-precision model, effectively correcting potential inference errors. From this viewpoint, the appropriate baselines include not only other speculative decoding methods, but also standard quantization techniques.\n\nTo fairly assess SubSpec's tradeoffs, it would be necessary to compare both the efficiency and capability of SubSpec against strong quantized baselines. Although SubSpec aims to be training-free, high-performance quantized models, especially those using state-of-the-art quantization algorithms, are often readily available. Furthermore, recent 4-bit quantization techniques (as used in this paper) tend to incur minimal accuracy degradation while delivering significant throughput gains by eliminating the need for offloading entirely. It remains unclear whether SubSpec offers a favorable efficiency–capability tradeoff, or if it sacrifices excessive efficiency for relatively minor capability gains.", "was_revised": true, "final_justification": "The authors have largely addressed the concerns I raised. Although the resulting trade-off, achieving a lossless 4-bit quantized model at the cost of roughly 50% efficiency, may be viewed differently depending on one’s perspective, I find the manuscript overall to be sufficiently coherent and complete.", "rebuttal": "We thank the reviewer for the insightful feedback. \n\nWe agree that viewing SubSpec as a form of quantization compensation is a valuable perspective that helps clarify its unique contribution.\nThe core design principle of SubSpec is to achieve acceleration while being lossless, guaranteeing outputs that are bit-for-bit identical to the full-precision (e.g., fp16) model. This fundamentally distinguishes our method from standard quantization techniques, which are inherently lossy and inevitably alter model outputs. This distinction is critical for applications where any deviation from the original model's behavior is unacceptable.\n\nWhile we acknowledge that modern 4-bit quantization techniques achieve only 1~2% accuracy degradation on multiple tasks [1,2], the accuracy degradation is not always negligible, especially on more challenging benchmarks [2]. To illustrate this, we evaluated several popular 4-bit quantization methods on MMLU-Pro [3], using a random sample of 50 questions per category. The results showed a non-trivial accuracy drop in accuracy compared to the fp16 baseline:\n| Model Configuration | Accuracy (%) |\n| :--- | :--- | \n| Full Precision (fp16) | 46.1 | \n| GPTQ (int4) | 41.8 | \n| AWQ (int4) | 44.1 | \n| HQQ (int4) | 42.2 |\n\nSubSpec avoids this accuracy-capability trade-off entirely. Nonetheless, we agree that a direct comparison against strong quantized baselines is essential. Hence, we now include a direct performance comparison against fully GPU-resident 4-bit models to provide a clearer picture of the efficiency-capability landscape. The table below shows the throughput of Llama-3.1 8B on a single GPU with an 8GB VRAM limit on MT-Bench:\n| Model Configuration | Throughput (tokens/s) | \n| :--- | :--- | \n| Full Precision (fp16) | 2.40 |\n| GPTQ (int4) | 58.87 | \n| AWQ (int4) | 52.32 | \n| HQQ (int4) | 135.84 |\n| **SubSpec** | 24.29 |\n\nWhile a standalone 4-bit model offers maximum throughput by accepting the quality trade-offs of quantization, SubSpec is designed for users who require lossless output that is bit-for-bit identical to the original FP16 model.\n\nFor this high-fidelity use case, SubSpec makes offloading practical. It accelerates naive offloading from an unusable 2.4 tokens/sec to an acceptable 24 tokens/sec for interactive use. This allows users to achieve full model fidelity at a speed that is comparable to some standalone quantized models, all on consumer-grade hardware. We therefore position SubSpec as a simple, training-free solution for users who cannot compromise on model quality.\n\n### Response to Questions\n> Could the authors provide experimental results comparing different quantization algorithms and target bit-widths?\n\n**Different quantization algorithms.**\nWe have shown speed results of different quantization methods above. \nThe reason we chose HQQ (int4) for our experiments is primarily due to its training-free nature and its high-performance triton kernels, which support efficient GEMM operations essential for tree-based speculative decoding.\n\n**Different target bit-widths.**\nWhile state-of-the-art low-bit methods like QUIP# [4] and QTIPS [5] (2-bit) show promising results, their current public kernels lack GEMM support, making them incompatible with tree-based speculative decoding methods. We thus consider exploring draft models with lower bitwidths (2/3 bits) as promising future work, contingent on kernel development. Please refer to our reply for Reviewer DA4c for more details about the system requirements.\n\nWe have also added experiments using SubSpec to accelerate an 8-bit target model to provide a more comprehensive analysis.\nThis scenario is considered as a different practical trade-off, where the accuracy degradation introduced by 8-bit quantization is acceptable. \n\nConsidering running Llama-3.1 8B on MT-Bench under an 8GB VRAM limit, while quantizing the target model to 8-bit still requires offloading, its memory footprint and offloading time are nearly halved compared to the 16-bit version. \nOur results show this configuration increases throughput by 36%, presenting a compelling trade-off between precision and performance:\n| Configuration (Target, Draft) | Throughput (tokens/s) | τ |\n| :--- | :--- | :--- | \n| SubSpec (fp16, int4) | 24.29 | 29.66 | \n| SubSpec (int8, int4) | 33.029 | 29.764 |\n\n> Although admittedly challenging, could the authors consider providing a direct comparison with modern quantized models?\n\nYes. We have compared the throughputs of different quantized models and discussed trade-offs between SubSpec and strong quantized baselines above, hoping to address your concerns.\n\n#### Reference\n1. An Empirical Study of LLaMA3 Quantization: From LLMs to MLLMs, Visual Intelligence, 2024\n2. Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models, COLM, 2025\n3. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark, NeurIPS, 2024\n4. QuIP#: even better LLM quantization with hadamard incoherence and lattice codebooks, ICML, 2024\n5. QTIP: Quantization with Trellises and Incoherence Processing, NeurIPS, 2024"}, {"review_id": "sdh3yXzeUO", "reviewer": "Reviewer_i2M6", "summary": "The paper proposed a technique for speculative decoding in offloading setting, where the inference is completely memory bound. The paper first identified that offloading workload requires different draft model from Eagle, where the draft model less needed to be fast. For offloading, the draft model needs to have strong performance in drafting deep and accurate trees for expensive offloading full model verification. The paper proposes a method for speculative decoding in offloading, where it used the quantized model as the draft model that can fit inside the consumer GPU VRAM. Model weights and KV cache are shared between draft and the full model.", "questions": "Under the same budget, quantized model might not have significant tree building capacity compared to using a small draft model and sample a huge tree. However, I think it really depends on the target hardware and the full model parameter count. I think there is some value in studying when it is better to use compressed model of the full as the draft model, when using a smaller draft model is more favorable.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths \n1. The paper is well-written and easy to follow \n2. The contrasts between offloading speculative decoding with normal SD is clear and insightful \n3. The paper show practical speedup, reinforcing the usefulness of the proposed method \n\nWeaknesses \n1. Important comparison missing, need to compare against SpecExec, as it is mentioned in the related work section, and it is clearly a much better baseline than Qwen Eagle \n2. The proposed method might encounter diminishing benefit when tackling 70B or beyond models offloading on constraint devices, as the quantized model might not even fit on the on-chip memory, and you have to resort to a non weight sharing draft model that is smaller (8B model). \n3. Minor, lack of discussion on prior self-speculative decoding works. \nElhoushi, M., Shrivastava, A., Liskovich, D., Hosmer, B., Wasti, B., Lai, L., ... & Wu, C. J. (2024). LayerSkip: Enabling early exit inference and self-speculative decoding. arXiv preprint arXiv:2404.16710.\nZhou, Y., Chen, Z., Xu, Z., Lin, V., & Chen, B. (2024). Sirius: Contextual sparsity with correction for efficient llms. arXiv preprint arXiv:2409.03856.", "was_revised": true, "final_justification": "My two concerns remain. \n1. SpecExec comparison is critical in establishing the conclusion of the paper. The current experiment results requires important clarification in the experiment setup, which is important for establishing the comparison. \n2. The method fundamental difficulty to extend to the large model regime also raises questions about the scope of the paper. \n\nBased on the above two reasons, my concern stands. Therefore, I will not raise the score.", "rebuttal": "We thank the reviewer for the detailed and constructive feedback. We have conducted new experiments based on your suggestions, which we believe have strengthened the paper. Please find our responses below:\n\n### 1. Missing Direct Comparison with SpecExec\nWe agree that a direct comparison with SpecExec is a crucial baseline. While we previously included a discussion in Appendix B.2, we have now performed a direct performance comparison. Under a similar decoding budget (~288 nodes), SubSpec significantly outperforms SpecExec:\n| Method | Throughput (tokens/s) | τ |\n| :--- | :--- | :--- |\n| **SubSpec (k6-d48)** | **24.29** | **29.66** |\n| SpecExec (k = 32, budget = 288) | 13.253 | 11.23 |\n\nSubSpec achieves ~83% higher throughput, driven by a 2.6x higher average token acceptance length (τ). This result validates SubSpec’s design principle: using a highly-aligned draft model combined with system co-design is more effective. We will prominently feature this direct comparison in the revised Appendix B.2.\n\n### 2. Diminishing Benefit When Scaling to Very Large Models\nThis is a valid point regarding the method's scope. Our approach is primarily designed for the common and practical scenario where a quantized version of the target model can fit into GPU VRAM. This covers a wide range of models (up to the 30B scale) on modern consumer GPUs.\nFor extreme cases, such as running a 70B+ model on a single constrained device where even the quantized draft will not fit, a different approach (e.g., using a separate, smaller draft model) would indeed be necessary.\nHowever, SubSpec remains applicable for these large models when multiple GPUs are available (e.g., via tensor parallelism), offering an efficient and training-free alternative where training a custom draft model might be infeasible.\n\n### 3. Missing Related Works\nThank you for pointing this out. We will add a new subsection in our related works to provide a comprehensive review of self-speculative decoding methods.\n\n### Response to Questions\n> Under the same budget, quantized model might not have significant tree building capacity compared to using a small draft model and sample a huge tree ... I think there is some value in studying when it is better to use compressed model of the full as the draft model, when using a smaller draft model is more favorable.\n\nThanks for this insightful question. We investigated this trade-off by testing whether a larger draft tree could compensate for a smaller draft model. We expanded the tree width (k) from 6 to 32 and observed the following:\n| Draft Model | Throughput (tokens/s) | τ |\n| :--- | :--- | :--- |\n| **SubSpec (k6-d48)** | **24.29** | **29.66** |\n| Llama-3.1 1B (k6-d32) | 15.14 | 11.91 |\n| Llama-3.1 1B (k32-d32) | 13.77 | 15.41 |\n\nOur results show that increasing the tree width yields minor gains in average token acceptance length (τ), while harming throughput due to the increased drafting overhead. We attribute this to the highly skewed probability distributions of LLMs, where a small number of top tokens account for the vast majority of the probability mass.\n\nThis finding suggests that SubSpec will always be more favorable against less-aligned small models of the family while sampling huge draft trees. We will add this valuable analysis to the final paper.\nThank you again for helping us improve our work."}, {"review_id": "ojG2yeW339", "reviewer": "Reviewer_DA4c", "summary": "This paper presents SubSpec, a speculative decoding method to accelerate inference for LLMs using parameter offloading. Specifically, SubSpec selects a number of layers to be quantized and the rest of layers to be GPU-resident, while sharing the KV cache. This leads to a training-free draft model with strong alignment with the target. The evaluation results show strong acceleration (up to 12.5$\\times$) on Qwen2.5 32B, with 24GB VRAM limit.", "questions": "- Could you please elaborate on how throughput translates to speedups? especially Tab 1 is very similar to that of Eagle-2, but Eagle-2 reported speedups. It would be helpful to include speedups in the results\n- Could you discuss the implementation complexity and system requirements? e.g. incorporating HQQ for quantization, and does the implementation of SubSpec pose any constraints on systems?\n- Thank you in advance for any response to the above.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n- The objective of improving SD acceleration on resource constrained hardware is well motivated. The paper is well-structured and clearly presented.\n- The proposed method of low-bit quantization of selected layers while keeping GPU resident layers and sharing KV cache is new, as far as I know, although there are existing works on self-speculative decoding.\n- The average accepted length and throughput achieved compare to Qwe 2.5 1.5B and Eagle-2 seems significant.\n\nWeakness\n\n- There doesn’t seem to be enough review of existing Self-speculative decoding methods, such as:\n1. Draft on the Fly: Adaptive Self-Speculative Decoding using Cosine Similarity, EMNLP, 2024.\n2. SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration, ICLR, 2025.\n3. Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding, ACL, 2024.\n4. Kangaroo: Lossless Self-Speculative Decoding for Accelerating LLMs via Double Early Exiting, NeurIPS, 2024.\n5. Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding, ACL, 2024.\n- Not necessarily a weakness, but if SubSpec can be considered as a plug-in module for other self-speculative method, it would make a much stronger impact.  Specifically, would it be possible to combine layer-skipping [1,2] above and layer-quantization ideas? How compatible is SubSpec with existing self-speculative decoding methods?\n- The statement of draft model being highly aligned doesn’t seem to be tested — although SubSpec achieves good end-to-end throughput improvement, would it be possible to showcase the output of draft and target are aligned at token level? perhaps a graph similar to Fig5 B in Eagle-2 paper would be helpful.\n- Results are shown without standard variation or error bars, making it difficult to assess statistical significance.", "was_revised": true, "final_justification": "My concerns have been sufficiently addressed, and I am raising my score to Accept.", "rebuttal": "We thank the reviewer for the insightful feedback and questions. Your suggestions and questions have helped us clarify the novelty and practical considerations of our work. Please find our detailed responses below.\n\n### 1. Missing Related Works\nThank you for pointing this out. We will add a new subsection in our related works to provide a comprehensive review of self-speculative decoding methods:\n1. Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding, ACL, 2024.\n2. Kangaroo: Lossless Self-Speculative Decoding for Accelerating LLMs via Double Early Exiting, NeurIPS, 2024.\n3. Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding, ACL, 2024.\n4. Draft on the Fly: Adaptive Self-Speculative Decoding using Cosine Similarity, EMNLP, 2024.\n5. SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration, ICLR, 2025.\n\nWe are willing to further expand this section in case we are missing important relevant works.\n\n### 2. Compatibility with Self-SD Methods\nThis is an excellent point. SubSpec is indeed designed as a modular framework and is highly compatible with other self-speculative methods like layer-skipping.\nThe integration could be straightforward. For example, to combine SubSpec with layer-skipping, one could simply skip a subset of the offloaded layers entirely rather than replacing them with their quantized substitutes. The feasibility of this is supported by prior work (e.g., \"Draft on the Fly,\" \"SWIFT\"), which shows high acceptance rates (α≈0.98) even when skipping 35-45% of the layers.\n\nApplying layer-skipping methods itself requires about 60% model VRAM usage, while a 4-bit substitute requires only 25%. A hybrid approach that combines layer-skipping and quantization could be even more powerful, potentially creating a draft model with a smaller VRAM footprint and faster generation speed than either method alone. We plan to investigate this direction further in our future work.\n\n### 3. Proves On Token-Level Alignment\nWhile we are unable to generate a new figure during the rebuttal period, we can provide two forms of evidence for the high alignment between our draft and target models.\n\n**Existing Evidence in the Paper.** The most direct measure is the average token acceptance length (τ). As shown in Figure 2(a), SubSpec's τ is nearly twice as high as that of a small draft model of the same family, providing strong quantitative proof of superior token-level alignment.\n\n**Additional Experimental Evidence.**  We performed a simple experiment by computing the average cosine similarity of different draft models' output probability distributions against that of the target model, on each of the first four output tokens generated by the target model (Llama-3.1 8B, 5 samples in MT-Bench):\n| Draft Model | 1st token | 2nd token | 3rd token | 4th token |\n| :--- | :--- | :--- | :--- | :--- |\n| Llama-3.1 1B | 0.7965 | 0.5761 | 0.4718 | 0.2969 |\n| SubSpec | 0.9728 | 0.6012 | 0.5950 | 0.6038 |\n\nAs shown, SubSpec maintains a much higher cosine similarity, confirming its superior token-level alignment compared to a smaller draft model of the same family.\n\n### 4. Missing Standard Variation or Error Bars\nThis is an important consideration. Due to the significant computational cost of our experiments, running multiple seeds for every configuration was infeasible.\n\nHowever, to validate the stability of our findings, we performed 5 independent runs on a key, representative result: SubSpec (8GB VRAM) on MT-Bench, obtaining a low standard deviation of 0.101 tokens/s. \nWe will add this information and a discussion of it to the final version.\n\n### Response to Questions\n> Could you please elaborate on how throughput translates to speedups? Especially Tab 1 is very similar to that of Eagle-2, but Eagle-2 reported speedups. It would be helpful to include speedups in the results.\n\nAll of our experiments are conducted with a batch size of 1, as noted in Section 5 (Comparative Methodology and Parameters). Therefore, the speedup is simply the method's throughput divided by the offloading baseline's throughput.\nWe chose to report absolute throughput primarily for transparency and fairness. We applied strong graph and kernel optimizations (torch.compile) to all methods, which significantly strengthened our baseline relative to those in other papers. Reporting only speedups against our optimized baseline could create an unfair comparison with previously published results.\n\n> Could you discuss the implementation complexity and system requirements? e.g., incorporating HQQ for quantization, and does the implementation of SubSpec pose any constraints on systems?\n\nThe implementation of our method is straightforward, but its performance is dependent on the capabilities of the underlying quantization libraries, which introduce specific system constraints.\n\n**Implementation complexity.** The main implementation complexity lies in tree-based SD with shared KV-cache, which includes handling KV cache position indices correctly among draft and target models, during both decoding and verification processes. The incorporation of HQQ for quantization and asynchronous offloading only adds minimal implementation complexity.\n\n**System requirements and constraints.**  The primary system constraint is the need for quantization kernels with efficient GEMM support. Tree-based speculative decoding processes multiple tokens in parallel, requiring matrix-matrix (GEMM) operations. Many low-bit (especially <4 bits) quantization libraries are only optimized for single-token autoregressive generation, which only relies on matrix-vector (GEMV) operations. This technical requirement is why our work focuses on HQQ with 4-bit quantization, as it is one of the few libraries with performant GEMM support for quantized models. We anticipate that broader low-bit GEMM support will become available as (tree-based) speculative decoding grows in popularity."}]

Author rebuttal:

Meta-review: (a) Summary

The paper proposes SubSpec which is a new speculative decoding method for offloaded LLMs on consumer GPUs. It builds a quantized substitute draft model from the target’s offloaded layers, shares GPU-resident layers and KV cache, and adapts draft-tree construction for high alignment.

(b) Strengths

- Addresses a practical deployment bottleneck with a simple, effective design.
- Strong empirical results:  speedup, alignment.
- Added key comparisons: 4-bit baselines, INT8+INT4 variant, SpecExec sweep under VRAM limits.

(c) Weaknesses

- No FLOPs-controlled fairness in SpecExec comparison (though VRAM-constrained sweep provided).
- Limited discussion of very large models (70B+)—acknowledged as out of scope.
- Related work on self-speculative decoding initially thin (to be added).

(d) Reasons

The method is well-motivated, practical, and now thoroughly evaluated. Authors addressed major concerns with new experiments and clarifications. Remaining issues are minor and suitable for camera-ready.

(e) Discussion & Rebuttal

Reviewers asked for quantized baselines, quantized-to-quantized scenarios, SpecExec comparison, and alignment evidence. Authors delivered all, plus clarified scope and added stability data. One reviewer (BRmM) didn't engage in the post-rebuttal discussion while mentioning he/she would adjust the score after the discussion. So, this review needed to be down-weighed. One reviewer maintained a borderline score citing FLOPs fairness, but this is a caveat, not a blocker. Overall, the concensus is that the rebuttal addressed major concerns.
Decision: Accept (poster)