Overfill: Two-Stage Models for Efficient Language Model Decoding
Authors: Woojeong Kim
Junxiong Wang
Jing Nathan Yan
Mohamed S. Abdelfattah
Alexander M Rush
Venue: COLM 2025 (2025)
Keywords: deep learning
large language models
inference efficiency
TL;DR: Leveraging more compute during prefill, OverFill improves generation quality with minimal latency overhead.

Abstract: Large language models (LLMs) excel across diverse tasks but face significant deployment challenges due to high inference costs. LLM inference comprises prefill (compute-bound) and decode (memory-bound) stages, with decode dominating latency particularly for long sequences. Current decoder-only models handle both stages uniformly, despite their distinct computational profiles. We propose Overfill, which decouples these stages to optimize accuracy-efficiency tradeoffs. Overfill begins with a full model for prefill, processing system and user inputs in parallel. It then switches to a dense pruned model, while generating tokens sequentially. Leveraging more compute during prefill, Overfill improves generation quality with minimal latency overhead. Our 3B-to-1B Overfill configuration outperforms 1B pruned models by 83.2%, while the 8B-to-3B configuration improves over 3B pruned models by 79.2% on average across standard benchmarks. Overfill matches the performance of same-sized models trained from scratch, while using significantly less training data. Our code is available at https://github.com/friendshipkim/overfill.

Published as a conference paper at COLM 2025
OverFill: Two-Stage Models for Efficient Language Model Decoding
Woojeong Kim, Junxiong Wang, Jing Nathan Yan, Mohamed Abdelfattah, Alexander M. Rush
Cornell University
{wk247, jw2544, jy858, Mohamed, aruus)@ccornell.edu
Abstract
Large language models (LLMs) excel across diverse tasks but face sig- nificant deployment challenges due to high inference costs. LLM infer- ence comprises prefill (compute-bound) and decode (memory-bound) stages, with decode dominating latency particularly for long sequences. Current decoder-only models handle both stages uniformly, despite their distinct computational profiles. We propose OverFill, which decouples these stages to optimize accuracy-efficiency tradeoffs. OverFill begins with a full model for prefill, processing system and user inputs in parallel. It then switches to a dense pruned model, while generating tokens sequentially. Lever- aging more compute during prefill, OverFill improves generation quality with minimal latency overhead. Our 3B-to-1B OverFill configuration out- performs 1B pruned models by 83.2%, while the 8B-to-3B configuration improves over 3B pruned models by 79.2% on average across standard benchmarks. OverFill matches the performance of same-sized models trained from scratch, while using significantly less training data. Our code is available at /https://gtt thub. ctt/:/riiennsiiiiim///eeeeee oo
1 Introduction
Large language models (LLMs) have achieved remarkable success on broad spectrum of tasks, from question answering code generation. Yet, their massive parameter counts pose significant challenges for practical deployment, with inference emerging as chief bottleneck. In modern LLMs, inference typically comprises two stages: prefill and decode. The prefill stage, where all input tokens are processed in parallel to build a Key-Value (KV) cache, is usually compute-boune the performance is primarily limited by the utilization of computational units. The subsequent decode stage is memory-bounde where it generates each output token autoregressively. The main bottleneck here is repeatedly loading the model's large feed-forward (FFN) layers into memory.
However, current LLM architectures do not exploit the distinct computational characteristics of these two stages. Our motivation stems from the distinct computational profiles of the prefill and decode stages, which have led to a growing trend of disaggregating them. The first line of work (Zhong et al., 2024 Patel et al., 2024) is system-oriented, focusing on stage-specific resource allocation and parallelism while still using a single model across both stages. The second line of work (algorithmii-orienteed) (Nair al. 2024 [ergner al., 2024) explores using models of different sizes for each stage, but often requires complex frameworks or delivers only small accuracy improvements. The question of interest is how to decouple the prefill and decode stages to achieve stronger balance of accuracy and efficiency.
We propose OverFill, a two-stage system that dedicates maximal capacity to the prefill stage, while pruning the costly decode stage to reduce parameter loading. Specifically, the larger model is used only once to process the user input into vector representation, and smaller, pruned model subsequently handles token-by-token generation. As a result, OverFill drastically cuts the memory footprint and latency of decoding, especially for longer
1

---

Published as a conference paper at COLM 2025
Prefill
Decode
Yes
it
is!
1
Transfor Block
0
Is apple fruit? <EOS>
Figure 1: Overview of OverFill. OverFill uses the full model for prefill and pruned model for sequential decoding. The yellow blocks represent the full model used for prefill. The decoder is initialized by selecting important channels (green blocks) from the full model. The blocks on the decoder side are in darker shades because they are updated during OverFill training. However, the full prefill model is kept frozen.
sequences. While it incurs a small prefill overhead compared to the standalone pruned model, this difference in cost is negligible compared to the dominant decoding latency. Moreover, OverFill starts with single model and prunes to build a smaller decoder, eliminating the extensive process of aligning two distinct models. Since only the small decoder updated during training, OverFill compatible with the original full model during serving. Importantly, OverFill is end-to-end trainable and does not require additional modules, allowing it to be optimized like any standard transformer architecture. Figure illustrates the overview of our framework.
We validate OverFill in diverse decoder scales and compare OverFill accuracy on tasks such as question answering, math, and thain-of-thougee reasoning. In 3B-to-1B configuration, OverFill outperforms standalone pruned models by 83.2% and instruction-turee by 52.6% in average. OverFill matches or even outperforms similarly sized models trained from scratch, while using significantly fewer training tokens. We also demonstrate OverFill is pareto-optimal in both scales. OverFill achieves significant accuracy gain over standalone small models while posing minimal latency overhead. These efficiency gains become even more pronounced when long outputs or multiple candidates are generated, as the pruned model remains active throughout the autoregressive process.
model
2 Related work
Model compression. Pruning enhances the efficiency of LLMs by removing model com- ponents that contribute the least to the output. Structured pruning eliminates entire groups
of parameters, such as channels, attention heads (Dery et al. 2024 Ma al. 2023 Xia
let al. 2023b Ashkboos et al. (024) or layers (Men et al. 2024 Yang et al. 2024] Kim et al.
2024). Structured pruning results
more compact model while preserving the underlying
architecture, keeping it hardware-friendly In this work, we adopt width pruning (Dery jtt al. 2024 Ma et al. 2023 Xia al. 2023b Ashkboos et al. 2024), which preserves accuracy better than depth pruning. We specifically use the approach from Sreenivas et al. (0024) but without the additional loss term, focusing on optimizing pruning to balance performance and computational efficiency.
Quantization (Frantar et al. 2022 Lin et al., 2024 Xiao et al. 2023a) is another effective model compression method. Standard quantization methods accelerate both prefill and decode. In our two-stage decoding process, we can also consider high-precision prefill and low-precision decode, which we leave for future work.
2

---

Published as a conference paper at COLM 2025
Decoding targeted speedups. Various methods tackle the serial bottleneck in LLM decod- ing. Speculative decoding [Leviathan et al. 2023) leverages available compute to propose tokens in parallel using small draft model. Researchers have explored specialized draft models (Sun et al. 2021 Xia et al. 2023a) and subnetworks of the target model (Schuster 2ia
et al. 2022] Elhoushi et al. | 2024 hhngg et al. 2023a Liu et al. 22024 Ankner et al. 2024). Among these, Du et al. (2024) Li et al. (2024) are particularly relevant to our work as they reuse target model representations
enhance drafting. Unlike speculative decoding, our approach eliminates rollbacks and calls the large model only once during prefill, avoiding parallel execution with the small model during decoding. This significantly reduces memory usage. We provide theoretical analysis in the Appendix.
KV cache compression. Many studies have explored KV cache compression to address memory bottlenecks with heavy batching and long contexts, using methods like token
eviction (Xiao et al., 2023b Zhang et al. 2023b Adnan et al. 2024), quantization (Sheng et al., 2023) Liu al. 2024b), and prompt compression (T'an al. 2024 Wingatte et al.) 2022).
Our approach targets scenarios where loading weights is the primary memory bottleneck.
While KV cache can dominate memory usage in certain scenarios, this typically occurs only at very long sequence lengths when using smaller models. For instance, in 7B parameter model with batch size 4, model weights is the primary bottleneck up 5K tokens (Adnan et al.] 2024). Several works target the opposite challenge as ours: reducing prefíll costs for very long contexts, where prefill becomes costly. These methods include architectural modifications (Sun et al., 2024),, chunking (Zeng et al., 20244, token dropping (Fu et al., 20244, and prompt packing (Zhao et al. 2024). Notably, such KV cache compression and prefill acceleration approaches can be applied top of our method for further optimization.
Method
We are interested in the setting of continual training of LLMs targeting instruction-tuninng In this setting we assume we have a large number of supervised examples of the form (x, where (x1, X2,, x2,.... XM) and y (y1, y2...., ......,y yN) are sequences of tokens, assumed for simplicity to be of a fixed length. We are particularly focused on N > M since the model may use methods like chain-of-thought
answer problems.
Formally LLMs model the probability of a sequence in conditional autoregressive manner: P( x) 1 M-t,, (c),,;(), where x<r denotes all tokens preceding Xt, where 0 the model. The core probability of P(y: y,t,,; is defined as a function of the Transformers cached hidden state,
P(y: x; & ((Cache((y<1, xl))
Where Cache is defined recurrently for any sequence of tokens a, b as,
Cache b]) = Transformer( a, Cache(b);0)
Due to this cache structure, when sampling from a language model, the computation happens in two-stages, prefill and decode. During prefill, the primary work is computing,
h hpre Transformer( (x 0;0),
which can be done in parallel for all (x1, ... XM). This stage is generally compute bound, since f can be loaded once and compute is parallelized across M.
During decode, autoregressively sample each y/ and recurrently update,
have, +- Transformer(y,, Cache Cache(h);0).
This stage has a serial dependency in that it requires previously generated tokens in order to update the cache. As such, it is cemorrybootleneccke in terms of speed as it needs to reload in 0 at each step and cannot fully use available parallel compute. Our primary goal will be to speed this stage up in practice by reducing the effective size of the during the decode stage.
3

---

Published as a conference paper at COLM 2025
Transformer block
Query
Proj
Key
Output Proj
Up Proj
Down
Attn
Proj
Proj
Value
Proj
Figure 2: Width pruning strategy: Diamonds indicate activation aggregation points for measuring channel importance. Yellow represents the hidden dimension, while green denotes the intermediate dimension.
Finally, to train instruction models, we simply maximize the likelihood of each (x,y) in- stance. During this stage, we do not maximize the probability of the conditioning term x but only of the generated y.
3.1 OverFill
We propose simple approach of using the model's capacity asymmetrically during the prefill and decode stage. The approach uses a better model prefill time to better utilize (overfill) the same cache of a smaller model. In our method, the same LLM parameters are used in two configurations:
Full Parameters (f) for processing prefill.
Pruned Parameters (#' C ()) a subset of the full parameters for processing decode.
Our contribution modifies both training and inference procedures so that the tokens x use the full network and the tokens y pruned sub-network. Formally, we define single set of model parameters but allow two "modes" (full vs. pruned), or more directly
hpre Transformer 2;0),
hdec, + Transformerry,, (ach(e))))
Upon deciding on the prune subset use, we freeze the full model and train only the pruned model with standard teacher forcing on the output tokens y. This approach ensures that the full model retains its well-initialized weights while accelerating training. Also, allows the pruned decoder to be seamlessly integrated into existing models.
3.2 Compatible pruning
In order for OverFill to be compatible method for LLM generation, it requires two aspects: (1) The pruned model must be significantly smaller than the original [0'] < [0] and (2) they must have compatible cache representation h. In Transformers, the KV Cache representation h is R2x1xD per sequence length, where D is the embedding dimension of the Transformer keys and values, and is the number of layers. While there are many different pruning methodologies that fit criteria (1), e-g. depth pruning and unstructured pruning, we target methods that can maintain criteria (2). Specifically we utilize a targeted form of width pruning, that avoid changing the cache dimension.
Our approach is based on the static channel pruning strategy from Sreenivas et al. 20024) Figure 2 gives schematic overview of the pruning template. Following the standard notation for transformers for the attention projections and FFNs we define our pruned parameters 0' as,
W w W WT E D'XD, D'DD, W DXD w R D'XAD' W| AD'
4

---

Published as a conference paper at COLM 2025
Full ((01)
Pruned (10"))
Pruning ratio (P)
Hidden dim. (D)
Layers (L)
3.21B
0.52B
0.7
921
28
3.21B
1.24B
0,45
1689
28
3.21B
2.01B
0.25
2304
28
8.03B
3.19B
0.43
2334
32
14.76B
7.62B
0.43*
2944
48
Table 1: Model sizes and width pruning configurations. *For the 14B model, we use harddare-friendly configuration by default, where different pruning ratios are applied to the hidden dimension and the intermediate dimension. See Table
This new network keeps subset D' < D for each of the weight matrices in the network. Layer norm and embedding parameters defined similarly.
To obtain the best starting compact sub-network, we first pick the pruning ratio P 1 D'D and then select the best rows and columns. To decide on what to prune, we pass a small calibration set through the model and aggregate activations as in Figure 2 three points per layer: before the attention projection,
before the FFN, and within the FFN. After aggregation we end up with tensor of shape batch by sequence length by dimension. We reduce this tensor using the L2 norm across the batch, and the mean across the sequences to derive an importance scores for each dimension. Finally, we retain the top (1 % of channels to calculate 6'.
As mentioned above, this pruning acts as starting point for determining the shape of 6'. Once determining this shape, additional finetuning is run on set of instruction examples to adjust the weights of the parameters to this new setting.
Experimental setup
4.1 Data
We use two instruction-tunning datasets for training: Oe. and Infinity-Instruct For main experiments, we adopt Infinity-Instruct (BAAT 2024) with 7M instances. We filter out non-English data using the provided language tags. For pruning ratio sweeps, we use OpenHermes-2.5 (Teknium/ 2023) which has 1M instances, of which 997k are used for training and 3k for validation. The total training tokens amount is 38M for -.e and 212M for Infinity- Instruct.
These datasets are well-suited for our task due to their natural separation between context and input. Both datasets are formatted with distinct tags: System, User, and Assistant. The System and User parts concatenated to the context, which is processed by the full model. The Assistant part serves as the target output to be predicted by the pruned model. To construct the model inputs, we inherit each dataset's original chat template.
4.2 Model
We evaluate our method on three base models in two model families: Llama 22-3B-Instruce Llama 3.1-8B-Instruct (Dubey et al., 2024), and Qwen 2.5-14B-Instruct (Team] 2024). For pruning, we apply the strategy outlined in Section 3.2 We do not prune attention heads or layers to preserve the dimensionality possible passed to the pruned decoder. Table [ presents the full and pruned model sizes along with their pruning configurations. We adopt the training hyperparameters from Tunstall et al.] presented in Table
the KV cache and retain as much information as

---

Published as a conference paper at COLM 2025
Model
Pruned*
OverFill
1B-Tuned*
1B-Inst
3B-Inst
Decoder Size
1.2B
1.2B
1.2B
1.2B
3.2B
GSM8K-CoT
45.4 (+1.4)
59.2 (+1.4)
47.6 (+1.4)
45.7 (+1.4)
78.4 (#111)
ARC
36.4 (#1.4)
77.8 (+1.2)
42.5 (+1.4)
56.5
78.2 (#1.2)
MMLU
33.7 (=0.4)
63.7 (+0.4)
38.7
(+004)
63.3 (#0.4)
MATH
6.1 (=:.3)
8.3 (#0.4)
5.2 (=
16.7 (=:0.)
35.0 (#0.6)
WMT16-DE-EN
14.7 (1:0.3)
31.4 (1:0.5)
28.0 (1.0.)
29.7 (10.4)
36.9 (10.4)
IFEval
26.4 (119))
44.2 (1.2.1)
26.1 (11.9)
48.1 (12.2)
69.5 (12.0)
NQ
(1.04)
12.1 (1.0.5)
7.8 (+005)
10.8 (10.5)
19.7 (+.07)
MMLU-Redux
26.06
40.93
27.29
18.21
56.95
CRUX 8.62
8.75
4.88
9.00
25.71
Table 2: Results in 1B scale. Bold indicates the best models under the same training data regime. means identically trained with the same data (less data compared to the Instruct models).
Model
Pruned
OverFill *
3B-Tuned*
3B-Inst
8B-Inst
Decoder Size
3.2B
3.2B
3.2B
3.2B
8.0B
GSMSK-CoT
55.4 (=1.4)
69.8 (#113)
61.8 (+1.3
78.4 (=111)
84.6 (=
ARC
42.5 (=144
83.4 (=1.2)
61.7 (=1.4)
78.2 (=1.2)
83.4 (=1.1)
MMLU
35.8 (+0.4)
69.4 (+0.44)
48.3 (+0.4)
(+-0.4)
69.4 69.4 (+0.4)
MATH
7.2 (=:.4)
15.1 (+0.5)
17.3 (+-0.5)
(4:06)
36.2 (+0.7)
WMT16-DE-EN
18.5 (+003)
35.6 (10.5)
25.8 (+0.5)
41.0 (+0.4)
IFEval
28.4 (119
44.0 (1221))
32.3 (12.0)
69.5 (12.0)
73.9 (1119)
NQ
8.0 (.055)
14.5 (10066
(1.0.)
19.7 (10.55
19.1 (10.7)
MMLU-Redux
28.80
43.12
43.95
56.95
61.66
CRUX 6.12
27.00
24.88
25.71
39.38
Table 3: Results in 3B scale. Bold indicates the best models under the same training data regime." means identically trained with the same data (less data compared to the Instruct models).
4.3 Downstream evaluation
We evaluate OverFill on downstream generation tasks, including math, code, question answering, and machine translation, using the LM Eval Harness (Gao et al. 2024) Details on evaluation metrics and the number of few-shot examples are provided in Table We use generation-based evaluation for all tasks, including multiple-choice question answering, whereas an alternative approach to compare the probability answer choices. assess longer-form generation, we MMLU-Redux and CRUXEval from the ZeroEval benchmark (Lin (0024). ZeroEval designed for evaluating instruction tunee models, with MMLU-Redux focusing general knowledge reasoning and CRUXEval assessing code reasoning, understanding, and execution capabilities. Models are prompted to provide both detailed reasoning steps and final answers in formttd output. We use greedy decoding for all generations.
Results
5.1 Accuracy
Table prresnnss the downstream evaluation results at the 1B scale. OverFill begins with 3B model and then pruned to a 1B-scale model for decoding. Our roofline model, Llama-3B- Instruct, handles both prefill and decoding.

---

Published as a conference paper at COLM 2025
Model
Pruned
OverFill
7B-Tuned*
7B-Inst
14B-Inst
Decoder Size
7.6B
7.6B
7.6B
7.6B
14.8B
GSM8K-CoT
75.3 (=1.2)
78,1 (+1.1)
81.0 (+111)
81.7 (=1.1)
75.9 (#1.1)
ARC
70.8 (+13
90.6 (10.9)
83.0 (+1.1)
(10.9)
90.6 (10.
MMLU
51.2 (+444
77.9 (+333)
65.3 (+0.8)
(+0.4)
77.9 (+0.3)
WMT16-DE-EN
32.0 (=0)3
38.2 (=0.4)
37.6 (+0.4)
37.5 (+0.4)
38.6 (#0.4)
IFEval
40.5 (12.1)
51.0 (12.11)
32.2 (12..0)
71.5 (12.0)
78.3 (1
Table 4: Results in 7B scale. Bold indicates the best models under the same training data
regime. means identically trained with the same data (less data compared to the Instruct models).
We compare accuracy across models trained under the same 1B decoder size and data regime. One baseline is the standalone pruned model, derived from the same original model, where a single model performs both prefill and decoding. Additional baselines include Llama 1B variants: (1) Llama 1B-base model finetuned on the same data as our pruned models and (2) Llama 1B-Instruct, a highly optimized model with more extensive instruction tuning. We observe that further tuning of Llama- consistently hurts performance across all tasks, as shown in Table 7 Therefore, we focus on comparisons with untuned Instruct models.
Our results highlight the following: OverFill consistently outperforms the standalone pruned model and finetuned 1B-Base model across all tasks by significant margin, demon- strating the effectiveness of the two-stage approach. Overfill shows accuracy improve- ments on both multiple-choice tasks, such as ARC-Challenge (Clark et al., 2018) and MMLU (Hendrycks et al., 2020), which typically involve short generations, and tasks requiring longer responses, such as GSM8K (Cobbe et a (021), MMLLU-Redux (Gema al., 2024), and CRUXEval (Gu et al. 2024.. This demonstrates that smart prefill benelits not only tokens close but also those generated later. When generation is short, OverFill can even match the full performance of the roofline model while maintaining a lower decoding cost. OverFill also outperforms Llama 1B-Instruct on out of 9 tasks. This is particularly notable given that Llama 1B- In-truttttt is highly optimized model at this scale, likely benefiting from more extensive data than our approach.
A similar trend is observed for 3B-scale models (Table 3). Here, OverFill with an 8B prefill and 3B decode outperforms the pruned model on all tasks and the trained 3B-Base model on 7 out of tasks. The 3B-Instruct model is a strong compact baseline, with performance close to that of 8B models, yet OverFill matches on several tasks. This pattern also extends to the Qwen2.5 family and a larger scale (14B to 7B). OverFill with a 7B decoder outperforms both the pruned model and the trained 7B-Base model on most tasks.
5.2 Efficiency
We show OverFill poses minimal overhead to the small standalone model by presenting end-to-end latency in 1B and 3B scales. We use vLLM (Srennivas et al., 2024) v0.8.5 for benchmarking and separately measure prefill and decode latency. All experiments are conducted on a single NVIDÍA A100 GPU. We slightly modify OverFill configuration to better leverage NVIDIA Tensor Cores, which is optimized for matrix multiplications in tiles (typically 16x8 or 16x16). For fair comparison, we maintain a larger parameter size than the one used for accuracy evaluation. The exact configurations are provided in Table We report the mean and standard deviation of 10 runs with 2 warm-ups.
Figure 3 presents latency across varying generation lengths, with fixed prompt length and batch size. The results show that Overfill asymptotically reaches the runtime of a small model and this trend becomes more pronounced in longer generations, where decoding cost dominates over prefill cost. At the 1B scale, both Pruned-1B and OverFill-1B exhibit higher latency compared to Llama-1B, as they have more transformer blocks, and transformers are more efficiently parallelized along width rather than depth. This suggests limitation of
7

---

Published as a conference paper at COLM 2025
14000
12300
20000
Latond (00)
10003
12000
Leaenyy
L0000
250
500
1000
1250
1500
1750
2000
230
1230
Generation Length
Length
Us
*** trasnd-15
Lams-d8
** d1
Liate
<#- Pruse0-15
. Owerth-18
Figure 3: Latency across different generation lengths. Prompt length is fixed to 128 with batch size
3500
1500
Lateee
1500
1000
500
Lamr--1
Lianx-18
Frunek38
Figure 4: Latency under different prompt lengths, with generation length fixed at 128 and batch size set 4. Prefill and decode latencies are shown stacked bars, with darker shades representing prefill and lighter shades representing decode.
2300
4000
2000
2300
1000
500
1.000
20
La
PruseSii
Linea-18
Linna St
Provdd30
Unass
OwePLi-1
Figure 5: Latency under different batch sizes. Prompt length and generation length are fixed to 128.
width pruning, highlighting the need for more balanced decoder architecture to improve both efficiency and accuracy. Still, both remain significantly faster than the full 3B model. At the 3B scale, where models have similar depth, Llama-3B, Pruned-3B, and OverFill-3B show nearly identical latency. Overall, OverFill achieves higher accuracy while introducing minimal latency overhead.
We measure latency while varying prompt lengths with fixed generation length, as shown in Figure Decoding latency remains the dominant factor over prefill latency in all cases, even when the prompt length is 16 times the generation length. Across all prompt lengths and model scales, OverFill consistently achieves speedups compared to the full model.
We also sweep batch size to account for diverse serving environments. The results are shown in Figure 5, where prompt and generation lengths are the same. In small batch scenarios, OverFill introduces minimal latency overhead to the pruned models. However, as batch size increases, prefill becomes relatively more expensive as decoding shifts from being memory-bound to compute-bound.
8

---

Published as a conference paper at COLM 2025
Mode Catogury Trand
Model Cutegocy
70
-Bo- Tured
Prusnt
Pruned
Anturicy
Acturacy
60
60
50
250
300
350
100
450
500
250
300
350
400
150
500
Latency
Latenty
Figure 6: Latency-accuracy tradeoff on GSM8K with temperature 0.01 (left) and temperature 1 (right), Models trained with the same approach connected by dashed lines, and dot size indicates the corresponding decoder size.
6
Analysis
6.1 Pareto optimality
We demonstrate that OverFill is Pareto optimal compared to models under the same training regime. 6 end-to-end latency versus accuracy on GSM8K with Chain-of-thoughe (CoT). CoT is standard practice that prompts models to reason step-by-step before pro- ducing a final answer, thereby boosting accuracy but resulting in longer generations. In our experiments, the average prompt length is 613 tokens with 4 demonstrations, and the average generation length is 120 tokens. Latency is measured using vLLM on a single NVIDIA H100 GPU. We fixed the batch size to across all experiments. OverFill achieves Pareto-optimality compared to finetuned and pruned models. Interestingly, Pruned models underperform compared to finetuned-Llama counterparts, likely because pruned architec- tures require extensive adaptation (Sreenivas et al. 2024 to recover their accuracy.
6.2 Comparison to speculative decoding
Speculative decoding and OverFill are similar in that both decode with the assistance of smaller model. We present speculative decoding latency results in Figure The original speculative decoding method (Xia et al. 2023a is theoretically lossless, meaning it exactly matches the target model's output. We also evaluate a lossy variant with lenient rejection sampling (Zhou et al.) 2023b), in which more tokens proposed by the draft model are accepted to increase speedup. In our setup, the target model is finetuned 3B model and the draft model is finetuned 1B model using the same data for both. We measure latency with a single batch size, which is the most common setting for speculative decoding benchmarks.
Under temperature-1 decoding, OverFill achieves better tradeoff. OverFill 3B-1B attains a 1.06x speedup over lossy speculative decoding at the same accuracy level. However, in near-greedy decoding (low temperature), speculative decoding performs particularly well because the outputs of the target and draft models are already very similar. As result, lenient rejection sampling provides limited additional benefit in this regime, since the acceptance rate is naturally high.
In more realistic serving scenarios with multiple concurrent requests, OverFill shows greater potential. Using the same vLLM setup with a maximum batch size of 256, our pruned decoder achieves throughput 7,913 tokens/s compared to 2,871 tokens/s for standard speculative decoding. This ad advantage comes from our approach accepting all tokens from the drafter without rejection sampling, allowing more tokens to be emitted per second. While recent speculative decoding variants have been proposed to improve throughput (Miao et al. 2024 Sadhukhan et al. 2024), we leave comparisons these methods for future work.
6.3 Impact of pruning ratio
We vary the pruning ratio while keeping the full model (3B) fixed to examine how the capacity gap between the full and pruned models affects the benefits of OverFill. As shown

---

tasks, both Published as a conference paper at COLM 2025
ARC Challenge
WM716
[rEval
80
30
60
Accuracy
20
30
10
0.3
0.4
0.5
0.6
0.7
0.3
0.4
0.5
0.6
0.7
0.3
0.4
0.5
0.6
0.7
Prusing Battio
Praning Natio
Praning Rettio
Figure Performance comparison between Pruned and OverFill across three tasks varying pruning ratios.
in Fig.1 7 OverFill consistently maintains significantly higher accuracy than the pruned model with the same decoder configuration. Interestingly, in ARC, where most information is processed during the prefill stage, a lightweight decoder with a pruning ratio of 0.7 shows no performance degradation. However, in translation and instruction-follow Pruned and OverFill experience performance drops with increased pruning. In some cases, OverFill degrades at slower rate than Pruned or follows similar trend more channels are pruned.
6.4 Accuracy by generation length
We observe that the benefits of OverFill per- sist for long generations. Figure 8 shows the probabilities assigned to correct tokens across their absolute positions in the out- put. The results show that OverFill consis- tently predicts tokens more accurately than the pruned model, demonstrating that the advantage of smart prefill extends to long generations.
0.70
Prumed-13
Overl8l-2
Mean Token Probaliity
0.50
0.35
0.50
However, the distance from the prefill grows, the gap gradually narrows, as the the
0.45
generation becomes more dependent smaller decoder. This suggests that while OverFill may not be optimal for extremely long generations, it remains highly effec- tive for many practical use cases, such bootstrapping multiple generations during testing. We plan further explore whether periodically refreshing the prefill can help maintain its benefits uniformly throughout the entire sequence.
0.40
Taken Position
Figure 8: Probability assigned to the correct token up to the 1024-th position, averaged across the sampled validation set.
Conclusion & Future work
This work presents a method for improving LLM generation with minimal latency increase by using a compatibly pruned subset of parameters for memory-bound decoding while retaining the full model for compute-bound prefill. We show that this approach outperforms fine-tuning base models of the same size and a standalone pruned model, with only minimal latency slowdowns. Our method is one of many possible strategies for training compatible pruned decoders and we believe there is a large design space of other architectures. For instance, pruning attention could further optimize KV cache size and decoding efficiency. Scaling this approach with larger training could also extend its benefits to even more memory-constrained models.
10

---

Published as a conference paper at COLM 2025
Acknowledgment
AMR was supported by NSF CAREER 2037519. This research is supported in part by the Office the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via the HIATUS Program contract #4202696884. We thank Yueying Li (Cornell Tech), Celine Lee (Cornell Tech), Ben Athiwaratkun (Together AI), and Muru Zhang (Together AI) for helpful discussions and feedback.
References
Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant Nair, Ilya Soloveychik, and Purushotham Kamath. Keyformer: Kv cache reduction through key tokens selection for efficient generative inference. Proceedings of Machine Learning and Systems, 6:114-127, 2024.
Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan-Kelley, and William Brandon. Hydra: Sequentially-dependee draft heads for medusa decoding. Xiv preprint arXim:2402.0511 02.00109, 2024.
Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. ar Xiv preprint ar Kit:24011110244 2024.
BAAL. Infinity instruct. ar Xiv preprint ar XXx::2406.XXXX 2024.
Benjamin Bergner, Andrii Skliar, Amelie Royer, Tijmen Blankevoort, Yuki Asano, and Babak Ehteshami Bejnordi. Think big, generate quick: Lim-to-sim for fast autoregressive decoding, Xiv preprint ar Xrv:22442 16444444..1844, 2024.
Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, et al. Findings of the 2016 conference on machine translation (wmt16). In First conference on machine translation, pp. 131-198. Association for Computational Linguistics, 2016.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. Xiv preprint
2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Relichiro Nakano, et al. Training verifiers to solve math word problems. Xiv preprint ar Xiv:221 14168, 2021.
Lucio Dery, Steven Kolawole, Jean-François Kagy, Virginia Smith, Graham Neubig, and Ameet Talwalkar. Everybody prune now: Structured pruning of iims with only forward passes. ar Xiv preprint ar arXin:2402.05406, 2024.
Cunxiao Du, Jing Jiang, Xu Yuanchen, Jiawei Wu, Sicheng Yu, Yongqi Li, Shenggui Li, Kai Xu, Liqiang Nie, Zhaopeng Tu, et al. Glide with a cape: A low-hassle method to accelerate speculative decoding. arviv preprint ar Xix:22402.00288, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arXiv preprint ar drx::22407.211 217883, 2024.
Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. Layerskip: Enabling early exit inference and self-speculative decoding. arxiv preprint ar arXi::2004.16710 2024.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh.
Gptq: Accurate post-training quantization for generative pre-trained transformers. ar arxio preprint arXio:2210.17323, 2022.
11

---

Published as a conference paper at COLM 2025
Qichen Fu, Minsik Cho, Thomas Merth, Sachin Mehta, Mohammad Rastegari, and Mahyar Najibi. Lazyllm: Dynamic token pruning for efficient long context Ilm inference. arXiv preprint ar 00t.24407.1405,, 2024.
Leo Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL https://zenodo. wttp://rrrrrrero///225/550000
Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, et al. Are we done with mmlu? arXiv preprint arXit:2406.04122 2024.
Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida Wang. Cruxeval: benchmark for code reasoning, understanding and execution. ar Xiv preprint ar 1ttt::240.030665 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, ar arXiv preprint #rXiv:209.003300, 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. Xrv preprint arXiv:2103.03874, 2021.
Bo-Kyeong Kim, Geonmin Kim, To Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung-Kyy Song. Shortened Ilama: A simple depth pruning for large language models. ar preprint arXiv:2402.02834 11, 2024.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Compulational Linguistics, 7:453-466 466, 2019.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint ar arXiv:2401.15077 2024.
Bill Yuchen Lin. ZeroEval: A Unified Framework for Evaluating Language Models, July 2024. URL
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangx- uan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
Fangcheng Liu, Yehui Tang, Zhenhua Liu, Yunsheng Ni, Kai Han, and Yunhe Wang Kangaroo: Lossless self-speculative decoding via double early exiting. ar preprint arXiv:2404.18911, 2024a.
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. ar Xiv preprint ar arXin:2402.02750 2024b.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. Lim-pruner: On the structural pruning of
large language models. Advances neural information processing systems, 36:21702-217200 2023.
12

---

Published as a conference paper at COLM 2025
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. ar Xiv preprint XrXi::2403.03853, 2024.
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceddings of the 29th ACM International Conference Architectural Support for Programmming Languages and Operating Systems, Volume 3, pp. 932-949, 2024.
Pranav Ajit Nair, Yashas Samaga, Toby Boyd, Sanjiv Kumar, Prateek Jain, Praneeth Netrapalli, et al. Tandem transformers for inference efficient 1lms, arviv preprint arXiv:2402.08644, 2024.
Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, et al. LImlingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. arXiv preprint ar xrXi::2403.12968, 2024.
Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, fñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative Ilm inference using phase splitting, In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), Pp. 118-132. IEEE, 2024.
Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisth Tiwari, Ruihang Lai, Jinyuan Shi, lan En-Hsu Yen, Avner May, Tianqi Chen, and Beidi Chen. Magicdec: Breaking the throuhytt tradeoff for long context generation with speculative decoding. Xiv preprint ar ArX:v:2408.11000 2024.
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Tay, and Donald Metzler. Confident adaptive language modeling. Advances in Neural Information Processing Systems, 35:17456-17472, 2022.
Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with single gpu. In International Conference Machine Learning, pp. 31094-31116. PMLR, 2023.
Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Lim pruning and distillation in practice: The minitron approach. arxiv preprint ar Xii::2408.11796, 2024.
Xin Sun, Tao Ge, Furu Wei, and Houfeng Wang. Instantaneous grammatical error correction with shallow aggressive decoding. Xrv preprint ar arXiv:2166.04970 2021.
Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. You only cache once: Decoder-decoder architectures for language models. Advances in Neural Information Processing Systems, 37:7339-736) 2024.
Qwen Team. Qwen2.5 technical report. arXiv preprint arXiv:2412.151155 2024.
Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist 11m assistants, wetttttttsseeeeen//eeeeeeeeeee
2023. Utt/httss://hugiing
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alvaro Bartolome, Alexander M. Rush, and Thomas Wolf. The Alignment Handbook. URL https:://st thub. www.oogggonnaaaaeeoooooooooooooe aigmmntn
David Wingate, Mohammad Shoeybi, and Taylor Sorensen. Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models. ar Xiv preprint arXtt:2200.03162, 2022.
13

---

Published as a conference paper at COLM 2025
Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. Speculative de- coding: Exploiting speculative execution for accelerating seq2seq generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 3909-3925, 2023a.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared Ilama: Accelerating language model pre-training via structured pruning. arXiv preprint ar Xrio:2331006694, 2023b.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099, PMLR, 2023a.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient stream- ing language models with attention sinks. arXiv preprint ar XrXio:2399.17553, 2023b.
Yifei Yang, Zouying Cao, and Hai Zhao. Laco: Large language model pruning via layer collapse. ar Xie preprint ar arXio:2402.11187 2024.
Zhiyuan Zeng, Qipeng Guo, Xiaoran Liu, Zhangyue Yin, Wentao Shu, Mianqiu Huang, Bo Wang, Yunhua Zhou, Linlin Li, Qun Liu, et al. Memorize step by step: Efficient long-context prefilling with incremental memory and decremental chunk. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 21021-21034, 2024.
Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft & verify: Lossless large language model acceleration via self-speculative decoding. Xiv preprint arXiv:2309.08168 2023a.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661-34710, 2023b.
Siyan Zhao, Daniel Israel, Guy Van den Broeck, and Aditya Grover. Prepacking: A simple method for fast prefilling and increased throughput in large language models. aiv preprint arXi::2404.0955 2024.
Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving. ar preprint arXiv:2401 000670,, 2024.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. instruction-followiin evaluation for large language models. arXiv preprint ar Xix::2311.07990
2023a.
Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Ros- tamizadeh, Sanjiv Kumar, Jean-François Kagy, and Rishabh Agarwal. Distillspec: Im- proving speculative decoding via knowledge distillation. arXiv preprint 4rXiv:2310.08461, 2023b.
14

---

Published as a conference paper at COLM 2025
A Appendix
A.1 Training hyperparmeters
LR
LR scheduler
Warmup
Max seq length
2e-05
cosine
0.01
2048
Table 5: Training hyperparameters
A.2
Downstream evaluation details
Task
Metric
Few-shot
GSM8K (Cobbe al. 2021
Accuracy
ARC-challenge (Clark et al.
2018
Accuracy
0
MMLU Hendrycks
2020
Accuracy
MATH Hendrycks et al.
202T)
Accuracy
4
WMT16 [Bojar et al. 2016
BLEU
IfEval (Prompt-level) (Zhou et al. 2023a
Accuracy
4
Natural Questions (wwitkowski et al. 2019
4
MMLU-Redux (Gema et al. 2024
Accuracy
0
CRUXEval (Gu et al. 2024)
Accuracy
0
Table 6: Evaluation details.
A.3
Finetuning Instruct models
Model
GSMSK
ARC
MMLU
MATH
WMT16
IfEval
NQ
1B-Instruct
45.7
56.5
47.9
16.7
29.7
48.1
108
E--Instruct-Tuned
40.8
49.4
42.4
6.9
28.9
34.4
3.1
3B-Instruct
78.4
78.2
63.3
36.9
78.5
69.5
19.7
3B-Instruct-Tume
64.4
70.6
55.7
12.8
34.2
53.2
8.9
Table 7: Finetuning Llama-Instruct models.
A.4 Speed benchmark model configurations
15

---

Published as a conference paper at COLM 2025
Model
Hidden dim.
Intermediate dim.
Layers
Params.
1B-Pruned
1689
4505
28
1.24B
standard
1792
4096
28
1.26B
3B-Pruned
2334
8171
32
3.19B
JD-Pruned-standan
2432
7680
32
3.21B
2944
11776
48
7.62B
Table 8: Speed benchmark model configurations.
16


Reviews:
[{"review_id": "j6zLNx1x8n", "reviewer": "Reviewer_paUo", "summary": "The paper introduces OverFill, a two-stage language model inference framework that leverages a full-size model for the compute-bound prefill stage and a pruned model for the memory-bound decode stage. This architecture achieves improved efficiency and generation quality by balancing computational costs and maintaining performance, showing significant gains over standalone pruned models with minimal latency overhead.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The main idea of splitting inference into compute-bound and memory-bound stages and optimizing each separately is well-motivated and practically significant.\n2. The empirical results are strong. OverFill consistently outperforms pruned baselines and tuned models across a diverse range of benchmarks (ARC, MMLU, GSM8K, etc.), often achieving performance comparable to larger models.\n3. The authors effectively showcase OverFill's balance of latency and accuracy, particularly under long generation and high-batch regimes.", "reasons_to_reject": "1. While OverFill performs well on standard benchmarks, the performance gap narrows as generation length increases. The authors acknowledge this but don't offer concrete mitigation strategies beyond future work.\n2. The approach is validated only on small scale models (<= 8b). It would be more informative to at least evaluate on middle scale models e.g. Llama 3.1 70b or Qwen 2.5 32b or 72b models.\n3. The approach relies on partial training of the pruned decoder. However, detailed analysis on training cost savings (vs. full finetuning) is limited.", "questions_to_authors": "1. How does OverFill interact with or complement existing model compression methods like quantization and distillation? Could the prefill stage be quantized more aggressively?\n2. You mention that OverFill may underperform for extremely long sequences and suggest refreshing the prefill. Can you elaborate on how this would be implemented practically without breaking autoregressive continuity?\n3. Have you tested or do you anticipate OverFill working with mixture-of-expert models e.g. deepseek, llama4, qwen3?", "extra_scores": {}}, {"review_id": "R87Lc8g1lD", "reviewer": "Reviewer_89kW", "summary": "This paper presents ​​OverFill​​, a two-stage framework to enhance inference efficiency in decoder-only LLMs by decoupling compute-heavy ​​prefill​​ (full model processes inputs in parallel) and memory-bound ​​decode​​ (switches to a pruned model for token generation). By strategically pruning model width (e.g., key channels in attention/FFN layers) without altering cache dimensions, OverFill reduces decoding bottlenecks while preserving architecture compatibility.\n\nEvaluations on GSM8K, MMLU, and CRUXEval show OverFill’s 3B→1B variant outperforms standalone 1B models by ​​83.2%​​, and 8B→3B exceeds 3B models by ​​79.2%​​, with minimal latency overhead. It matches same-sized models trained from scratch using ​​70% fewer tokens​​ and scales efficiently with longer outputs. On A100 GPUs, decoding latency nears that of smaller models, achieving ​​3× speedups​​ over full-model inference for long sequences.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed idea is simple yet effective, achieving both significant latency reduction and average quality improvement", "reasons_to_reject": "1. Limited novelty： The core width pruning method reuses existing techniques (Sreenivas et al., 2024), making the framework appear more like a special case of prior work rather than a novel contribution\n\n2.  Lack of direct comparisons with parameter-matched LoRA-based methods, failing to demonstrate the necessity of width pruning over alternative lightweight adaptation approaches", "questions_to_authors": "1. It’s unclear how the pruning ratio during the prefilling stage affects performance.\n\n2. This work could be more impactful by proposing guidelines (like a scaling law) that link teacher model sizes and pruning ratios.", "extra_scores": {}}, {"review_id": "5rKlpbUEt8", "reviewer": "Reviewer_Y7iV", "summary": "The paper proposes OverFill, a disaggregated prefill-decode approach for LLM inference that uses a smaller pruned model for decoding in combination with the original full model for prefill. This approach outperforms using models of the smaller size for both stages while only adding limited latency overhead, especially when operating at small batch sizes. This is a simple yet effective trick to increase the efficiency of serving LLMs and offers new ways to achieve different accuracy-latency-tradeoffs.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The method seems to be an effective way to increase the efficiency of serving LLMs and offers new ways to achieve different accuracy-latency-tradeoffs.\n- The experiments compare to a reasonable set of baselines, including finetuning on the same data, and the paper thoroughly analyzes the effect of prompt length, generation length and batch size on the latency.\n- The paper is well written and clearly explains the approach and experimental setup.", "reasons_to_reject": "- The paper focuses solely on latency but does not measure or discuss the impact on memory and thus throughput.\n- The paper compares to speculative decoding only theoretically and in the appendix, instead of having this as a baseline in the main experiment. There is also no discussion on why no direct comparison has been made. \n- The 1B/3B-Tuned baselines use fewer layers than the Pruned and Overfill models, making the comparison difficult. Different depth-to-width ratios show different accuracy across different types of benchmarks, some scale better with width (knowledge tass), other with depth (reasoning tasks). A clean comparison would be the architecture of Pruned trained from scratch.", "questions_to_authors": "Please respond to the points raised above", "extra_scores": {}}, {"review_id": "JEOp7r5Kx4", "reviewer": "Reviewer_wzXx", "summary": "The authors argue that the current prefill and decode stages during the LLM inference are not fully exploited, since each stage has its own computational characteristics. To better decouple the two stages, this paper proposes OverFill, which handles the two stages separately. For prefill stage, it uses a large model to dedicate maximal capacity, while in the decoding stage, it prunes smaller models to reduce parameter loading. Experiments verified the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The research question of exploiting and decoupling the stages of prefill and decoding is important for LLM inference.\n- The proposed method, OverFill, is generally sound.\n- The experiments and analyses are solid.", "reasons_to_reject": "- The readability of this paper needs improvement, particularly for readers who are not very familiar with this research area (like me). For example, in Figure 1, the authors should illustrate the relationships between the green blocks. I spent 3 times reading the figure till I fully understood this intention.\n- The motivation is not explained to me. In the introduction, the authors mentioned two lines of work, system-oriented and algorithmic-oriented. I don't quite understand the logic flow here, why these two work lead to the question of \"decoupling the prefill and decode stages\". Similarly, for the related work section. I think the related work section does not echo well with the motivation and the two lines.\n- It would be great to see experiment results on model architectures other than Llama.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviews are lukewarm about this paper, sitting just slightly above borderline. I believe there are genuine concerns about experiments such as scale of experiments, and fully validating claims through from-scratch trained of a pruned model. Going beyond LLaMa was done during rebuttal time. In general the authors have put a lot of work on demonstrating this efficiency technique, unfortunately efficiency techniques are expensive to validate for large language models without full scale experiments. Reviewers have largely overlooked the rebuttal so I think while some of the concerns are legitimate as AC I have looked at both points and I am also slightly over the fence for acceptance as some of the ideas on this work might survive scrutiny when tested at scale, particularly the idea of a prefill and an overfill model for decoder-only architectures.
Decision: Accept