Progressive Mixed-Precision Decoding for Efficient LLM Inference
Authors: Hao Mark Chen
Fuwen Tan
Alexandros Kouris
Royson Lee
Hongxiang Fan
Stylianos Venieris
Venue: ICLR 2025 Poster (2025)
Keywords: LLM Quantization
Efficient LLM Inference
Primary area: generative models
TL;DR: We propose a novel decoding scheme that adapts the precision throughout stages in LLM inference.

Abstract: In spite of the great potential of large language models (LLMs) across various tasks, their deployment on resource-constrained devices remains challenging due to their excessive computational and memory demands. Quantization has emerged as an effective solution by storing weights in reduced precision. However, utilizing low precisions (i.e.~2/3-bit) to substantially alleviate the memory-boundedness of LLM decoding, still suffers from prohibitive performance drop. In this work, 
we argue that existing approaches fail to explore the diversity in computational patterns, redundancy, and sensitivity to approximations of the different phases of LLM inference, resorting to a uniform quantization policy throughout.
Instead, we propose a novel phase-aware method that selectively allocates precision during different phases of LLM inference, achieving both strong context extraction during prefill and efficient memory bandwidth utilization during decoding. To further address the memory-boundedness of the decoding phase, we introduce Progressive Mixed-Precision Decoding (PMPD), a technique that enables the gradual lowering of precision deeper in the generated sequence, together with a spectrum of precision-switching schedulers that dynamically drive the precision-lowering decisions in either task-adaptive or prompt-adaptive manner. 
Extensive evaluation across diverse language tasks shows that when targeting Nvidia GPUs, PMPD achieves 1.4$-$12.2$\times$ speedup in matrix-vector multiplications over fp16 models, while when targeting an LLM-optimized NPU, our approach delivers a throughput gain of 3.8$-$8.0$\times$ over fp16 models and up to 1.54$\times$ over uniform quantization approaches while preserving the output quality.

Published as a conference paper at ICLR 2025
PROGRESSIVE MIXED-PRECISION DECODING FOR EFFICIENT LLM INFERENCE
Hao (Mark) Chen Fuwen Tan', Alexandros Kouris', Royson Lee', Hongxiang Fan 1.2 Stylianos I. Venieris "Samsung AI Center, Cambridge, UK Imperial College London, UK hc/620@ ic.ac.uk, [fuwen.tan, a.kouris, royson.lee) @ samsung.com hongxiangfan @ org,. s. s.venieris@ samsung.com
ABSTRACT
Despite the great potential of large language models (LLMs), deploying them on resource-constrained devices is challenging due to high computational and memory demands. Quantization has emerged
effective solution by storing weights in lower precision. However, Post- Trainiran Quantization (PTQ) at ex- tremely low precisions (i.e., 2/3-bit) still incurs severe performance degradation. In this work, we argue that existing approaches fail to explore the diversity in com- putational patterns, redundancy, and sensitivity approximations of the different phases of LLM inference, resorting to a uniform quantization policy throughout. Instead, we propose novel phase-aware method that selectively allocates pre- cision during different phases of LLM inference, achieving both strong context extraction during prefill and efficient memory bandwidth utilization during decod- ing. To further address the memory -bundeesbsu of the decoding phase, in- troduce Progressive Mixed-PPeeiise Decoding (PMPD), a technique that enables the gradual lowering of precision deeper in the generated sequence, together with a spectrum of lowering decisions in either task-ade or prompt- compttaddapive manner. Extensive evaluation across diverse language tasks shows that when targeting Nvidia GPUs, PMPD achieves 1.4- 12.2x speedup in LLM linear layers
schedulers that dynamically drive the precision-
fp16 models and up to 1.41x x over uniform quantization. When targeting an LLM- ILM-optimized NPU, our approach delivers throughput gain of 3.8 over fp16 models and up 1.54x over uniform quantization approaches while preserving the output quality. Our code is available at at/github.com adgithubcco//SmsuuglLlpPPPP/
INTRODUCTION
Modern large language models (LLMs) have demonstrated unprecedented capabilities across vari- ous natural language understanding and generation tasks. However, their computational and memory footprint, driven by billions of parameters and thousands-of-tooke icant challenges particularly in applications that require their deployment on embedded or mobile devices. LLM inference operates in two phases: i) the prefill phase, which processes all input prompt tokens in parallel, and ii) the decoding phase, which generates output tokens one by one in an autoregressive manner. Computationally, the prefill phase known be compute-bound, while the decoding stage is memory mouud, leading to underutilization of computational resources and often dominating inference latency (Kwon et al. 2023 Yuan et 2024). especially on resource- constrained devices where batching is not an option (Yu et al. 2022 Agrawal et al. 2024).
context windows, pose signif-
To enhance the hardware performance of LLM inference at the edge, techniques like model com- pression and adaptive inference have been explored. Quantization, in particular, reduces precision for weights (Yu et al. 2021), activations (Tan et al.) 2024),, and KV cache (Hooper et al. 2024). However, most methods apply uniform quantization across prefill and decoding, leading to two key issues: i) Ignoring distinctive fault tolerances in prefill and decoding stages, causing accuracy drops while reducing bit-wssths for mobile-scall LLMs; ii) Decoding is memory- memory-bound, so bit- bit-width reduction more effective here for speedup. Thus, uniform quantization yields limited gains, high- lighting the need for more targeted quantization strategies specifically tailored to LLM decoding.

---

Published as a conference paper at ICLR 2025
Conventional Approach
Progressive Precissoo Decoding (PMPD)
Profilling Phase
Decoding Phose
Motivation !
Prefiling Phase
Decoding Phase
Phase-Aware Error Resillence
Motivation ? Increasing Fault Tolerance (Decoding)
4-bit
4-bit Uniform High Precision
4-it
3-bit
2-bit
Progressie Lowering
Unitrrm Precision
Figure Illustration of the conventional paradigm of quantized LLM inference (left) and our method that comprises phase-aware precision allocation and progressive mixed-precision decod- ing (right), Our approach motivated by i) the distinct error resilience observed during the prefill and decoding phases, and ii) the increasing fault tolerance as decoding progresses to longer tokens.
By observing the key limitations in prior work, this paper identifies novel insight for LLM weight quantization: The prefill phase, as well as earlier parts of the decoding phase, are more sensi- five approximation errors from quantization, than later parts the autoregressive generation process. Building on this insight, we propose a novel LLM inference method that counteracts the limitations of existing approaches by means of phase-aware and progressive reduced-precision approach, shown in Figure 1 Specifically, our method departs from the existing deployment ap- proaches that employ uniform quantization precision throughout the LLM inference process and introduces phase-awer aware precision allocation. By considering the distinct redundancy, sensitivity to approximations, and arithmetic intensity of the prefill and decoding phases, our scheme tailors the arithmetic precision to each phase's 's characteristics. This leads to both maintained generation quality through high-precision prompt encoding during prefill, and improved sustained throughput through reduced precision during decoding.
To further alleviate the memory-boundednee of decoding, we introduce progressive mixed- precision decoding (PMPD). novel decoding scheme that gradually reduces precision throughout the generation the LLM response. Based on the observation that tokens generated later in the output sequence more resilient to approximations while carlier tokens more sensitive, method employs weight quantization as an approximation mechanism and progressively reduces the numerical precision during the decoding phase. To balance generation quality with decoding throughput, we formulate precision scheduling as a constrained optimization problem with the ob- jective of minimizing the average bitwidth while preserving output quality. As solution, de- sign two complementary precision-switching schedulers, one high-performance task-specific static scheduler and one flexible task agnostic learned scheduler, to strategically determine the highest- performing precision-reduce rrdduction time. By applying our method on Vicuna-7B, MobileLLaMA. 1.4B, Stable I.M Zephyr-3B, and Phi-1.5 across diverse language tasks, we achieve decoding throughput gains of 3.8-8.0x on NPU platform and 1.40- 14.220000x speedup on LLMM linear layer computations on GPU. The main contributions of this work can summarized follows:
A novel phase-aware precision allocation strategy that optimizes precisions differently for the prefill and decoding phases, leveraging the distinct error resilience of each stage achieve an extremely low average bitwidth for mobile-grade LLMs.
A progressive -ixeedprecision decoding scheme that progressively reduces the precision as decoding progresses longer token sequences, effectively improving the hardware per- formance in the memory- bound L.LM decoding stage.
A prompt-agnostic static scheduler and task-agnostic learned scheduler for precision switching, enabling the flexible deployment of our approach across diverse scenarios, ac- commodating varying data availability constraints and generation quality requirements.
BACKGROUND AND RELATED WORK
2.1 LOW-PRECISION LLM INFERENCE
Several quantization approaches have been employed to reduce the precision of the weights (Fran- tar et al., 2023), activations (Dettmers et al. 2022; [Tan et al., 2024) and KV cache entries of efffriiinnn.cc.c efficiency.
L.LMss (Hooper et al. 2024), aiming to boost their computational
2

---

Published as a conference paper at ICLR 2025
Weight-only Quantization. In particular, weight-only quantization approaches for LLMs result in remarkable speedups upon deployment on commodity platforms, by alleviating the volume of mem- ory transactions during decoding which, dominated by weight transfers, forms the main processing
bottleneck in LLM inference (Yuan et al.] (024). Along these lines, and inspired by relevant lit- erature on deep neural networks (DNNss (uu al. 202T), Any-Precision LLM (Park et al. 2024) enables the post-training quantization of single set of weights to multiple lower precisions. This creates a family of variably quantized model variants, without any storage overhead the original model. To further improve the performance of low-bit quantization, Dense-and-Sparre Quantization (DNS) (Kim al. 2024) increases the average bitwidth by storing small fraction of outlier weights in full precision while keeping the remaining weights in quantized format. Finally, BitNet (Ma et al.) 2024 introduces ternary weights, but requires costly quantization-awe aware training. contrast, this work focuses on low-overhead post-training quantization for efficient LLM inference.
Nonetheless, existing methods typically adopt uniform precision across phases, which also remains fixed across all input prompts. Although this approach can deliver near lossless approximate infer- ence with very low precisions on highly redundant large-scale LLMs, it often struggles to maintain performance under aggressive quantization (e.g. 2 or bits) for smaller-scal models. Our approach takes advantage of the low memory footprint solution provided by LLM inference, but adopts dynamic precision lowering methodology, which considers the LLM inference phase and input prompt at hand, to push the limits of the adopted precision.
LLM to accelerate
2.2 PRECISION-ADAPTIV APPROACHES
The proposed approach exploits weight quantization, through dynamic phase-aware and progres- sive precision lowering methodology. As such, more relevant to our method is a line of work that proposes ways to dynamically adapt bitwidth run time.
Precision-Adaptive Training, MUPPET (Rajagopal et al. 2020). CPT Fu et al. 2021) and AdaPT (Kummer et al. (202)) aim to improve the efficiency of the training stage by dynamically adjusting the arithmetic precision throughout the training process. This family of methods focuses primarily on classification tasks. In contrast, our method is optimized for the characteristics LLMs and focuses on improving the efficiency of the inference stage.
2020) allow Inference. HAQ (Wang et al. 2019) and HAWQ (Dong et al. 2019
for different precisions across the layers of a given DNN. Nonetheless. the selected precision re- mains fixed across all processed input samples. Bit-mixer (Bulat Tzimiropouloy (202T) allows a similar utilization of mixed precision across layers, but can adjust the selected bitwidths in per- sample manner. CascadeCNN (Kouris et al., 2018) adopts different-precision variants of the same DNN, organised in low-to-hin high precision classifier cascade. Samples that fail to meet hand-tuned confidence- baaddd criteria on the low-precision stage are propagated for re- re-computation with higher precision. As before, these approaches are developed for CNN-based classification tasks and are not off-the-shelve applicable to the autoregressive decoding process of LLM inference. Closer to our work, LLM-PQ (Zhao et al. 2024) adjusts model precision based on the hardware support of dif- ferent servers, along with a phase-aware model partitioning, targeting distributed LLM inference serving setup. In contrast, our approach adapts precision at different time steps rather than across different machines and is optimized for edge- based deployment.
MOTIVATION
3.1
OBSERVATION ONE: DIFFERENT ERROR RESILIENCE IN PREFILL AND DECODING
Recent work has shown that allocating larger model capacity to the prefill phase for natural language understanding allows for reduced capacity in the decoding phase for natural language generation, while producing responses with both high quality and improved efficiency (Aishwarya al.) 2024). Inspired by this, we aim to investigate the distinct redundancy and resilience exhibited by the prefill and decoding phases in LLM inference.
To this end, we target three diverse language tasks and compare the response from two variants of the same LLM (Vicuna-7B (Chiang al. 2023)): i) uniformly low-precision variant with 2-bit
3

---

Published as a conference paper at ICLR 2025
Translate following text fron French to Eng Lishh: Les architectes les Ingénieurs, teureusenert)
Humen: Krite poen about the ocean.
huran: What is the 16th Fiboracci number?
Assistant: sorry, nachine have ability poen.
Assistant:
Fibonacci number 16,
architectes les ingón
20f1
2bx
Hunan: Trans the fellowing
Numan: Mrite the
Human: Mhat is the 18th Fibonacci number?
French architectes ingér ingér,
is a vest and mysteripus Surrounding every iife, part the Piece Earth.
Assistants
Assistant: The 10th Ibonace a number is 5S.
Assistant: Architects and engineers,
stat prefi
fortunately!
364 profii
3BR profii
Figure 2: Phase-aware Precision Allocation. Here prefilling was performed by 2- (top) or 3-bit (bottom) Vicuna-7B, while decoding employed the 2-bit model for both cases. We observe three key improvements. i) enhanced reasoning abilities. The model demonstrated higher accuracy for numerical tasks, such calculating Fibonacci numbers (left). ii) improved instruction instruction-folloiingg The model understood and responded to user instructions (middle). iii) strengthened emerging abil- ities like creatívity. The model successfully handled open-ended tasks such as poem writing (right). More quantitative results are shown in Appendix A.2 and Figure
following.
quantized weights in both prefill and decoding, and ii) two-precision variant with 3-bit weights during prefill and lower-precision 2-bit weights during decoding.
Figure 2 shows the responses the two L.IM variants. We empirically observe that using the 3-bit model for prefilling (bottom row) significantly enhances the generative capabilities of the uniform 2- bit model (top row). We attribute this result to the significant steering role that the prefill phase has to the output generation process. Specifically, a high-quality KV cache extracted from the input prompt during prefilling provides good understanding of the task and information-rict context, enhancing in this way the generation capability of the -reeciiiioon model. This insight motivates us explore and study different precision policies for the prefill and decoding phases.
3.2 OBSERVATION TWO: PROGRESSIVELY LOWERING PRECISION DURING DECODING
To further increase the output quality of seduced-precision generation, we explore the optimiza- tion space of mixed-precision decoding. Figure 3 depicts our findings when applying different approaches of scheduling between low- and seshhpreciiion weights. Namely, we explore employing the nigh-precision weights in the first half, middle part, and last half of the generated tokens.
In line with our hypothesis, adopting high precision at the first half of the decoding stage yiclds the best performance, matching the performance of using high precision throughout the generation pro- cess. One intuitive explanation is that using high precision to generate the first few tokens minimizes the error accumulation that would affect the rest of inference. This observation also aligns with the "attention sink" phenomenon (Xiao et al. 2024), where it was observed that the attention scores tilt heavily towards the initial tokens, indicating their higher importance. As reference, we also tested the performance of alternating the two models at each token, which also proves to work well. How- ever, frequent precision switching introduces hardware overhead, both in terms of weights loading time and additional memory traffic, so this schedule is not considered practical for real deployment.
4 METHODOLOGY
Motivated by our findings in Section 3 suggesting that tokens deeper in the decoded sequence are more resilient to approximations, we adopt weight quantization as our approximation mechanism and propose new mixxed-reccisio decoding method, comprising two key techniques: i) phase- aware precision allocation and ti) progressive mixed-precision decoding (PMPD). With the objective of maintaining generation quality while maximizing efficiency during inference, the proposed tech- niques operate complementarily towards applying precision selection and scheduling strategies that tailored to the respective characteristics of the prefill and decoding phase of L.MMs. To enforce
4

---

Published as a conference paper at ICLR 2025
First
Middle
4.5
half
4b
4b
3b
3b
half
3b
4b
4b
3b
12.75
Ahemace
22.30
Last
half
3b
4b
Alternate
4b
4b
3b
Token from 3-bit model
Token from 4-bit model
4b
Figure 3: Different schedules mixing 3- and 4-bit models for decoding. We examined sum- marization on CNN/DM using Phi-1.5. To ensure fair comparison, the same number of tokens are generated by the 4-bit baseline. Moreover, the 4-bit model was used for prefilling for all schedules.
our approach with fine-gre aineggrinned control, we further introduce two types of runtime schedulers that drive precision-switching decisions in prompt- and task-agnostic manner, respectively.
Algorithm Saanthmmmppeeeee presents our end-to-end methodology, consisting of: the offline stage (lines 1-8) and ii) deployment (lines 9-22). The offline stage is responsible for producing variably quantized variants of a given LLM (line 1), conducting the phase-aware precision allocation (line 2), and configuring the selected type of precision-switching scheduler with optimized PMPD parameters for the target use case (lines 3-8). Upon deployment, the scheduler assigns the allocated precisions to the prefill and decoding stages, and triggers precision-switchiee events for each encountered prompt (line 20).
4.1 PHASE-AWARE PRECISION ALLOCATION
The proposed Phase-Aware Precision Allocation capitalizes on the unique characteristics of each LLM inference phase and tailors accordingly the per-phase weight precision. Specifically, ference scheme dictates that a high-precision model is used during the prefill phase which in turn enables the use of power-precision weights the memory-bound decoding phase, and thus yields more efficient utilization of the memory bandwidth. Since the prefill phase is compute-bound and the LLM inference latency is typically dominated by the decoding phase, using a high-preciste model during prefill introduces negligible latency overhead.
in-
Formally, our scheme parametrized with respect to: 1) a quantizer Q(-), which is typically post-training quantization (PTQ) method, 2) set of precisions P, indicating the supported weight bitwidths, 3) a reference value (eer in a tpsk-specific quality metric, and 4) a quality drop tolerance e. The objective of our method is find the pair of smallest weight precisions for the prefill and decoding phase, respectively, that achieves equal or greater algorithmic performance than Gref e.
We solve this objective by means of calibration step during the offline stage of our methodology. First, given a pretrained LLM m. we obtain set of variably quantized models (m,) containing one quantized model variant per arithmetic precision p in precision set P using quantizer Q(-) (line I). Next, we perform the phase-aware precision allocation step (line 2). Concretely, the pretrained L.MM is evaluated a calibration dataset, by assessing its algorithmic performance using different combinations of prefill and decoding bitwidths. Through this process, we determine the lowest pair of precisions that meet the quality target (line 2).
4.2 PROGRESSIVE MIXED-PRECISIG DECODING
Exploiting the insights of Section 3.2 we introduce Progressive Mixed-Precision Decoding (PMPD) to further improve decoding efficiency. As depicted Figure Right, PMPD exploits the variability between different tokens in terms of resilience to approximation, by gradually lower- ing the arithmetic precision weights as we go deeper the output sequence. this manner, we maintain higher precision for earlier tokens, which often play more decisive role in the response quality, while generating deeper tokens more rapidly through the use of narrower-bitwin weights.
We parametrize PMPD with i) the precision set P and ii) the precision- schedule = (st(p) | P). which consists of the possible switching points of each precision defined as st(p) E [1. OL), where OL is the maximum output length of given LLM. Here, the starting point st(p) is defined as the token index of the output sequence where we perform a switch from the cur- rent precision the lower precision p. Under this setup, obtain the highest- PMPD)
5

---

Published as a conference paper at ICLR 2025
Algorithm 1: Progressive
Decoding
Input:
LLM vi with max ourput length OL
Precision set P. Quantizer Q(-)
Domn. Task Task-specifis validation set 1/ (optional if using prompt agnostic satic scheduler) Reference quality Ree: Quality drop tolerance
Task agnosiic calibration
Output: Output token sequence (fa, (1....
-9---- Offline Stage ---
} Q(m.p) Vp E
variably quantized model
y prelll 1p dboood - PAPAlloc(P,, Dealhs Gest c)
Dr Phase-ase aware precision allocation
if ooperPMPSchssuulerr prompt-agnostic static then
PMPDScheduler.5 +- OfflineSolver( {mp) V. Radi
P. Offiine optimization of Eg.
end
it [yppePMMPScccculerr task-agnonicc learned then
PMPDSchechllermooel SchedulerTrainee {mp): Pan))
learned scheduler
end
19
Deployment Stage-..
while prompt queue empty
Process incoming prompts P. Prefill Phase
10
du, Kg Vo f greecii (prompt)
11
Sampler(da)
12
[opee(MMPSchcdulle task-agnostic learnd then PMPDSchedaleer.S PMPDSScheluleemmoalllll Ko Vo)
Generalte precision switching schedule before decoding
end 14
15
16
for 0. OL 1 do
D. Decoding Phase
17
d.t1. K Vi+1 +- myyew (ts, K.V.)
18
tge1 +- Sampler(de;+
if frai1 FOS then hreak plo PMPDSchstular(s +
End of sequence Precision-switching scheduler
20
21
end
22
end
configuration, we pose the following optimization problem:
min st(p), Vp E P {Pmin}
(1)
s.t. 4re - ((S)
0 05 st(p) < OL, pEP p>9 st(p) S st(g), P.pEP
(3)
where (8) is the achieved quality of schedule S. Our objective function aims minimize the number of tokens to be processed in each precision in the precision set -except for the lowest pre- cision Pmin which will be used last and until the end of the sequence for higher throughput- subject to meeting the specified quality target (first constraint in Eq. (2), not exceeding the LLM''s maxi- mum context length (second constraint in Eq. 2 and following a progressive precision lowering approach (precedence constraint in Eq.
4.3 PRECISION-SWITCHIEE SCHEDULER
Upon deployment, PMPD is enforced by means precision-swiie -witchinititttggg scheduler (Figure4.. The scheduler is configured with the PMPD parameters (P.S) and issues precision precision-switchingg switching actions to the system processor following schedule for each input prompt.
Given the set of variably quantized models {mp) that was generated in the phase-aware precision allocation step (Section 4.1 and line 1), the objective function of Eq. (1) can evaluated for all combinations of precision-switching schedules. Ideally, the highest-performing schedule considers all E P per-prompt per-step when deciding for precision switch without the precedence con- straint shown in Eq. (3). This schedule can be obtained through an exhaustive enumeration over [P] OL schedules. However, for realistic values of [P] and OT, this computation quickly becomes intractable. Besides, even if such schedule can be efficiently found. it is, in most cases, imprac- tical as it would require excessively frequent switching, e.g. per decoding step, between variably quantized weights, aggravating the memory bandwidth demand (Section 3.2
Instead, through our approach of progressive precision lowering during decoding, the total num- OL! (IP -1)! ber of schedules to be examined is reduced to E!P1-1 rH(L--]) -1()) where is the num- ber of times the precision switches. Nonetheless, the search time for the optimal schedule in- creases with OL and [P), which is still prohibitive at run time. Hence, to further reduce the search time for the optimal S to O(1), we parameterize the number of candidate precision-

---

Published as a conference paper at ICLR 2025
switching points, denoted by N, and add range constraint to the second constraint in Eq- (N-2)OL
as st(p) E {0} u (N 20L [N-1].-m (N-1) OL}. Under this setup, we propose two types of schedulers: i) a prompt-agnostic static scheduler, where determined prior to deployment, and ti) task-agnostic learned scheduler, where S is derived dynamically for each prompt.
Prompt-Agnostic Static Scheduler. To utilize an efficient precision scheduler, we propose static approach that selects schedule S through an offline optimization process (line 4) that utilizes calibra- tion set. While the precedence constraint in Eq. (3) and the optimization objective in Eq. (1) still hold, the achieved quality in Eq. (2) is approximated as (SS) = [V Evey 9e (S) where is the validation set and Ja (S) is the measured quality score of the schedule on sample x.
Output: Precision Switching Policy
Precision-Swiccingg Scheduler
Input: Per-prompt prefilled IV cache (For learned scheckiiess only)
As the validation set is defined per task, the static scheduler remains prompt-agnostic (i.e. uses the same precision-switching schedule for all prompts) and task-specific. As such, this type of scheduler is tailored to the resilience of the task at hand and has the advantage of inducing close to no runtime overhead. However, by adopting a uniform sched- ule within given task, it requires the availability of a rask-repreeentative validation set, which is often not a realistic assumption, and it does not exploit the variability difficulty and resilience to approximation across different prompts.
Figure 4: Precision-switching scheduler.
Task-Agnostic Learned Scheduler. To eliminate the need for a per-task validation set, we intro- duce a trainable scheduler that adjusts the schedule S to each input prompt. For a given LLM, our learned scheduler trained on generic, aank-agi dataset and then applied across various down- stream tasks, amortizing in this way its training cost. Specifically, we aim to devise a schedule given the features of each input prompt. Training the scheduler to make very fine-grained per-step deci- sions, however, would require both substantial model capacity and excessively frequent scheduler invocation, which induce significant runtime overhead and counteract PMPD's speedup benefits.
Hence, we utilize the features extracted in the prefill phase as inputs to our scheduler. With this approach, we are able to maintain a lightweight architecture for the scheduler and introduce minimal overhead during inference as the scheduler generates schedule once before decoding (line 13). In Section 5.3 we investigate different input features, i.e. KV caches and activations, from various layers of the model and find that utilizing the prefilled KV cache is efficacious in most cases. Given an input Key cache K E RTxxx and Value cache V E RTxD= the output of Learned scheduler is computed as MLP(Softmax(
VD. ge VV), where qo E R De is a learned query vector. V),
We also find that despite not requiring a task-speeific validation set and labeled ground truths, our leareed scheduler performs close to the static and, in some cases, even outperforms it in both algo- rithmic and hardware performance. Its architecture and training are detailed in Appendix A.I
5
EVALUATION
Models and Datasets. We conducted experiments on edge-deployable models, including Vicuna- 7B (Chiang al. 2023), MobileLLaMA-I.... (Chu et al. 2023). Stable LM Zephyr-3B and Phi- 1.5 (Li et al. 2023b), evaluating their zero-shot generative performance on news summarization, dialogue summarization, and translation tasks using the CNN/DM (Hermann al. 2015), Dialog- sum (Chen et al.) 2021). and IWSLT French-Engling datasets (Cettolo et al. (017). respectively. These tasks encompass context understanding and text generation, providing comprehensive eval- uation of PMPD We also tested open-ended question answering on MT-Bench (Chiang et al. 2023). special case where Statie scheduler is infeasible due to the absence of a validation set.
'eereafter, PMPD refers to approach including both phase-aware precision allocation and progressive mixed-precision decoding.

---

Published as a conference paper at ICLR 2025
CNN/DM
Dialogsum
IWSLT
Moszifj Vicun-7B
Modal1ii6 VAM
Moda VAMEm-
Method
Manie LMA, Mi-1
Mtatialaa P9-15
Miniel LMA, Jepken-3
Rouge-L/ BERTScore
Rouge-L/ BERTScore
BLEU
Bit
Bit
Bit
SacreBLEU
Baseline-1
2
8.30 78.4
7
1.2 1.2
Baseline-h
3
24.2 86.9
24.4 88.2
31.6 31.6
DNS
2.39
24.2/ 86,8
2.0
2.68
27.6/
PMPD- Static
2.39
24.3 /870
2,0
25.0
88.2
2.68
31.0/3111
PMPD-Leamen
2,43
24.0/86.7 86.7
2.74
24.5 88.2
2.37
29.9
Bascline-I
3
16.3783.3
15.87 84.1
9.879.83
Baseline-h
4
17.2/83.5
84.9
12.7 12.7
DNS
3.37
17.4/83.5
84.4
3.65
12.0/ 12.0
PMPD-Static
3.37
17.6 83.7
3.0
17.0 85.0
3.65
12.6 12.6
PMPD-Leamed
3,19
16.6/832 83.2
3.21
17.1 85.0
3.48
11.8 1L.8
Baseline-
3
13.4822.4
15.3785.1
3
21.17211
Baseline-h
4
16.2/84.0
18.0 86.1
30.4/
DNS
3.71
124/8188
3.30
16.1 85.7
3.34
28.2 28.2//282 28.2
PMPD-Static
3.71
16.2 84.0
3.30
86,2
29.7 29.7
PMPD-Lcamed
3.09
15.5 83.4
3.52
17.9 86.1
3.34
29.8 29.8
Table I: Performance comparison of Static and Learned schedulers against low-precision (baseline- 1) and high-precising (baseline-h) baselines, as well as DNS with the same average bitwidth as the scheduler with highest performance. For CNN/DM and Dialogsum, models used from top bottom are Vicuna-7B (2/3 bit), MobileLLaMA (3/4 bit), and Phi-1.5 (3/4 bit). For IWSLT, the models are Vicuna-7B, MobileLLaMA, and Zephyr-3B (3/4 bit). Pairwise winners between the scheduler and DNS are underlined, while the highest overall scores are in bold.
Baselines. We compare against single-precision quantized models, and quantized models enhanced with Dense-and-Sparse decomposition (DNS) (Kim et al. [2024), with DNS ratios adjusted match PMPD's average bitwidth. PMPD's average bitwidth calculated as the weighted average of bitwidths over decoding steps, weighted by the number of tokens generated each precision.
PMPD Implementation Details. For PTQ method, we adopted the nested quantization method of Any-Precision LLM (Park et al., 2024), ensuring that using multiple weight precisions incurs no memory footprint overhead. We further explore the effect of using other PTQ methods in Ap- pendix A.7 The Static scheduler finds a schedule that minimizes high-precision benchmark's validation set while maintaining lossless performance. The Learned scheduler was trained using the first 256 samples from the C4 test dataset as the seed dataset. The high-precision variant of each model is quantized to the lowest lossless precision determined by perplexity on the C4 dataset, while low precision is defined to be one bit lower than the high precision.
steps on each
GPU Latency. Following Any-Precision LLM (Park et al. 2024), we evaluated the latencies of linear layers in the LLMs across different Nvidia GPUs, including RTX 4090 and A40. As our implementation was based PyTorch (Paszke et al. 2019),, employed its built-in Profiler tool and reported the average self CUDA time metric over 100 forward passes to ensure reliable results.
NPU Simulation Measurements. To estimate the processing speed of PMPD when deployed on an NPU, we developed an analytical performance model of the hardware architecture of FlightLLM (Zeng al. 2024), LLMooptimized accelerator, adapted to support multi-precision weight loading. More details can be found in Appendix A.6
5.1 ALGORITHMIC PERFORMANCE
Table presents the algorithmic performance of PMPD in comparison with both single-precisioe baselines and DNS. We observe that both Static and Learned schedulers achieve significant bitwidth reduction while maintaining competitive performance.
PMPD vs. Baselines: Both schedulers deliver lossless performance on the CNN/DM and Di- alogsum datasets with up to 33% reduction in bitwidth, highlighting their ability to effectively capture context and generate high-quality outputs. In cases, PMPD high-preed sigh-precision baseline by up to 2.4%. On the IWSLT dataset. Static experiences a moderate perfor- mance drop of up 2.3%, while still reducing the average bitwidth by 9-33%. Notably, our results
outperforms the
8

---

Published as a conference paper at ICLR 2025
CNN/DM
Dialogsum 12
IVSLT
70
(samplesis
50
fp16
Throughhus (sangleec))
fp16
3bit Static
Sbit
50
3bit
30-
80
Learned
40
Static
Learned
30
40
Leamned
Throughpo
30
20
20
10
1o
10
Vicuna-76
MobileLiaMA
Phi-1.5
Vicuna-76
MobileLiaMA
Phi-1.5
Vicune-78
MobileLiaMA
Figure 5: End-to-end NPU throughput. Speedup ratios are obtained in comparison with with model.
Vicuna-7B
MobileLlama
Phi-1.5
Zephyr-3B
Attn Proj.
MLP Proj.
Attn Proj.
MLP Proj.
Attn Proj.
MLP Proj.
Attn Proj.
MLP Proj.
Baseline-h
6.25x
11.32x
1.25x
2.50x
1.32x
1,70x
3.33x
2.54x
RTX 4090
PMPD
6.25x
12.20x
1.0%X
2
1.51x
1.8x
2.73x
2,82x
Baseline-h
5.81x
3.00x
3.96x
2.57x
3,50x
3,02x
A40
PMPD
6.58x
4.60x
3.37x
4.74x
2.77x
3.51x
3.81x
4.27x
Table 3: GPU speedup ratios of PMPD for linear layers compared the fp/6 model. Two repre- sentative layers per model are shown: attention projection (Attn Proj.) and MLP projection (MLP Proj.). PMPD speedup was calculated weighted average of kernel latencies in two precisions, weighted by decoding steps. Additional details are included in Appendix A.5
show that using high-precision models in the decoding phase is not always necessary. Also, Static is able to accurately identify such scenarios, achieving the maximum bitwidth reduction possible.
PMPD DNS: DNS is robust baseline that maintains low-bit precision by selectively handling outlier values, Nonetheless, our Static scheduler consistently outperforms DNS at the same bitwidth across all tasks, achieving up 29% higher BLEU on the IWSLT dataset. Additionally, the per- formance gain increases with smaller models: DNS performs comparably to Static on CNN/DM for Vicuna-7B but lags behind by 23% Rouge-I. on MabileL.LaMA. This indicates that PMPD particularly well-suited for edge deployment scenarios.
Static vs. Learned Scheduler: On average, the Static scheduler achieves a slightly higher bitwidth reduction (0.66 vs. 0.63 bits) and frequently outperforms Learned, showing more stable performance with less variance. This is likely because Static leverages task-specific data, while Learned may face challenges from data distribution shifts between its training set and each test set.
Model
Method
Romge-L L/ BEKTScore
16.7
Vicuna-7B
PMPD- Learned
2.68
39.6 $8.9
DNS
2.68
36.7 88.2
3bit
25.5/72 87.2
Zephyn-3B
PMPD- Learned
4bst
3.48
40.8 /9.6
40.7 89.7
We used MT-Bench challenging benchmark covering a wide range of open-ended tasks. particular, we demonstrate the usefulness of Learned in this scenario where Static is not feasible due to the absence of task-specific validation set. We measured the faithfulness of the quantized models to the fp/6 model by evaluating the similarity of their outputs. As shown in Table 2 Learned achieves an average bitwidth reduction of 0.47 with no drop BERTScore and only up to 1.4 points drop in Rouge-L. Additionally, it consistently outperforms DNS with 13.9-18.4% higher Rouge-L, demonstrating that PMPD better preserves model fidelity in complex tasks. Moreover, Learned's s relative performance improves with smaller models, making particularly promising for reducing quantization error in mobile-friendle models.
Open-Ended Tasks:
MobileLLaMa
PMPD-Learas
3.39
29.1 83.4
85,8
DNS
3,39
25/4/54.
TEIT
27373357
Phi-1.5
PMPD-Lemmed
/77.9
Table 2: PMPD's 's MT-Bench perfor- mance with fp16 output reference.
5.2 HARDWARE PERFORMANCE
NPU Evaluation. We compare the attainable throughput of PMPD against fp16 and reduced- precision baselines on an LLM-optimized NPU (Zeng et al. 2024) by emulating their execution. As shown in Figure 55, PMPD achieves significant speedup of 3,0x over fpl6 across various models and datasets, with more pronounced speedup for the larger Vicuna-7B, due to its higher re- silience quantization emors,, enabling more aggressive b- precision lowering. Overall, PMPD

---

Published as a conference paper at ICLR 2025
CMNDD
IWSLT
Latency of Troft wtt 1:00 tokors
1,088 0,2%
30.0
71.
20.3
75.
Aaply
201
2611
4LC
336
00
Figure 6: Performance of the phase-aware precision allocation, 3/2 bits for Vicuna-7B, 4/3 bits for the other models.
achieves more than 50 tokens/s and 15 tokens/s for mobile-grade LLMs and for the larger Vicuna- 7B, respectively, in spite of the limited bandwidth (32 GB/s for NPU vs. 1008 GB/s for RTX4090 GPU), showcasing its effectiveness and suitability for LLM deployment mobile devices.
Focusing on PMPD's GPU deployment, Table compares the latency of fp16 and quantized linear layers, reporting CUDA kernel runtimes with batch size of 1. PMPD achieves speedups of 1.40-6.58x for smaller attention projection layers and 1.84-12.20s for larger feedfor- ward projection layers. We hypothesize that the lower speedup on smaller layers is primarily due to the memory-boume nature of matrix-vector multiplication. Despite the accelerated GPU execution, we observed that CPU-side processes for GPU kernel launching become a bottleneck, resulting in suboptimal GPU utilization. Potential solutions include leveraging CUDA Graph (Gray 2019) to launch multiple GPU operations simultaneously, which we leave for future research.
GPU Evaluation.
0.85
5.3 ABLATION STUDIES
0.80
Phase-Aware Precision Allocation. Figure analyzes the im- pact of phase-aware precision allocation on performance. Rouge- on CNN/DM improves by an average of 4.3 points, BERTScore
Normalized Score
0.75
0.70
4s act tre
0.65
on Dialogsum increases by 3.6, and BLEU improves by 9.45. The 2-bit variant of Vicuna- 7B benefits significantly from using a higher-precision model during the prefill phase, it struggles with instruction following and often outputs incomplete tokens or repeated prompts. Using 3-bit variant during prefilling improves language comprehension and generation. Latency overhead from using higher precision is minimal, ranging from 0.07% to 1.05 confirming that the prefill phase is compute-bound. bound. This supports the efficiency of phase-aware precision allocation for system opti- mization.
0.60
0.55
mit
mitder prior unncomanform
0.50
3.0
3.2 3.4
3.6
3.8
4.0
Precision
Figure 7: Normalized scores of different Learned schedulers variants across CNN/DM, IWSLT and MT-Bench on MobileLLaMA.
Learned Scheduler Design. We study different choices of inputs for Learned, including activa- ti and KV caches from the first, middle, and last Transformer attention blocks. We test two random schedulers: using a uniform distribution (random uniform) and ii) matching the label dis- tribution of the training dataset (random prior). As shown in Figure 7 using the KV cache from the last attention block achieves the highest normalized score with significant bitwidth reduction, highlighting its effectiveness in capturing contextual information for precise switch point predic- tion. This scheduler also outperforms the two random schedulers, indicating that is effectively learning patterns in the KV cache.
6 CONCLUSION
Taking advantage of the insight that later stages of the LLM decoding process demonstrate enhanced resilience to approximations, we introduced PMPD, novel technique that progressively reduces the precision of the model's weights during inference. The proposed approach considers the distinct LLM inference phases (prefill vs decoding) and the depth of the decoding sequence to schedule precision changes in either a prompt-agnostic static or task-agnostic learnable manner. We show that PMPD can generate high-quality responses to prompts, while significantly reducing the adopted average bitwidth, leading to remarkable gains in inference speed across GPU and NPU platforms.
"Specifically, the input activations the final projection layer of the attention block.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming Throughput-Lattnncy Tradeoff in LLM Inference with Sarathi-Serve. In USENIX OSDI, 2024.
AI@ Meta. Llama 3 model card. 2024. URL /tt.:tttss////tttbccc/mmmtttttte Ttob/main/MMDEL.CCM
ama 3/
PS Aishwarya, Pranav Ajit Nair, Yashas Samaga BL, Toby James Boyd, Sanjiv Kumar, Prateek Jain, and Praneeth Netrapalli. Tandem Transformers for Inference Efficient LLMs. In International Conference Machine Learning (ICML), 2024.
Adrian Bulat and Georgios Tzimiropoulos. Bit-mixer: Mixed-precision networks with runtime bit-
width selection. International Conference on Computer Vision (ICCV), 2021.
Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Niehues, Sebastian Stüker, Katsuhito Su- doh, Koichiro Yoshino, and Christian Federmann. Overview of the IWSLT 2017 evaluation cam- paign. In Proceedings of the 14th International Conference on Spoken Language Translation, pp. 2-14, Tokyo. Japan, December 14-15 2017. International Workshop on Spoken Language Translation. URL
Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. DialogSum: A real-life scenario dialogue summarization dataset. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, 2021.
Wei-Lin Chiang. Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality, March 2023. URL
Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang. Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, and Chunhua Shen. MobileVLM A Fast, Strong and Open Vision Language Assistant for Mobile Devices. arXiv preprint arXiv:2312. 16886, 2023.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. GPT3.int8O) 8-bit Matrix Multiplication for Transformers at Scale. Advances in Neural Information Processing Systems (NeurIPS), 2022.
Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. HAWQ: Hes- sian Aware Quantization of Neural Networks with fixed-Precision. In IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
Zhen Dong, Zhewei Yao, Yaohui Cai, Daiyaan Arfeen, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. HAWQ-v2: Hessian Aware Weighte Quantization of Neural Networks. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate Post-training
Compression for Generative Pretrained Transformers. International Conference on Learning Representations (ICLR), 2023.
Yonggan Fu, Han Guo, Meng Li, Xin Yang, Yining Ding, Vikas Chandra, and Yingyan Lin. CPT: Efficient Deep Neural Network Training via Cyclic Precision. In International Conference on Learning Representations (ICLR), 2021.
Alan Gray. Getting Started with CUDA Graphs, 2019.
URL https://
Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching Machines to Read and Comprehend. In Advances on Neural Information Processing Systems, 2015.
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. KVQuant: Towards Million Context Length LLM Inference with KV Cache Quantization. arXiv preprint arXiv:2401. 18079 2024.

---

Published as a conference paper at ICLR 2025
Schoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong. Xiuyu Li, Sheng Shen, Michael Ma- honey, and Kurt Keutzer. SqueezeLLM: Dense-and-Sye SSpars Quantization. In International Con- ference Machine Learning (ICML), 2024.
Alexandros Kouris, Stylianos I. Venieris, and Christos-Savvas Bouganis. CascadeCNN: Pushing the Performance Limits Quantisation in Convolutional Neural Networks. In International Confer- ence on Field Programmable Logic and Applications (FPL), 2018.
Lorenz Kummer, Kevin Sidak, Tabea Reichmann, and Wilfried Gansterer. Adaptive Precision Train- ing (AdaPT): A dynamic quantized training approach for DNNs. In SIAM International Confer- ence on Data Mining (SDM), 2023.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang. Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient Memory Management for Large Language Model Serving with PagedAttentiin.. In Symposium on Operating Systems Principles (SOSP), 2023.
Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. How long can context length of open-source Ilms truly promise? In NeurlPS 2023 Workshop on Instruction Tuning and Instruction Following. 2023a.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report, 2023b.
Shuming Ma, Hongyu Wang, Lingxiao Ma. Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits. arXiv preprint arXiv:2402. 17764, 2024.
Yeonhong Park, Jake Hyun, SangLyul Cho, Bonggeun Sim, and Jae W. Lee. Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sizer LLMs. In International Conference on Ma- chine Learning (ICML), 2024.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy. Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In NeurlPS, 2019.
Aditya Rajagopal, Diederik Vink, Stylianos Venieris, and Shristos-Savvar Bouganis. Multi- Precision Policy Enforced Training (MuPPET): A Precisinn-Swichhngg strategy for Quantised Fixed-Point Training of CNNs. In International Conference on Machine Learning (ICML), 2020.
Fuwen Tan, Royson Lee, Lukasz Dudziak, Shell Xu Hu, Sourav Bhattacharya, Timothy Hospedales, Georgios Tzimiropoulos, and Brais Martinezs. MobileQuant: Mobile-friendly Quantization for On-device Language Models. In Conference Empirical Methods in Natural Language Pro- cessing (EMNLP), 2024.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. HAQ: Hardware-Aware Automated Quantization with Mixed Precision. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2019.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In International Conference Learning Representations (ICLR), 2024.
Gycong-In Yu, Joo Scong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orea: A Distributed Serving System for Transformer-Bart Generative Models. In USENIX OSDI, 2022.
Haichao Yu, Haoxiang Li, Humphrey Shi, Thomas S Huang, and Gang Hua. Any-Precision Deep Neural Networks. In AAAI Conference on Artificial Intelligence (AAAI), 2021.
Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Zhe Zhou, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu. Yong Jae Lee, Yan Yan. Beidi Chen, Guangyu Sun, and Kurt Keutzer. LLM Inference Unveiled: Survey and Roofline Model Insights. arXiv preprint 2rX. 16363, 2024.
12

---

Published as a conference paper at ICLR 2025
Shulin Zeng, Jun Liu, Guohao Dai, Xinhao Yang. Tianyu Fu, Hongyi Wang, Wenheng Ma, Hanbo Sun, Shiyao Li, Zixiao Huang, et al. Fii Efficient Large Language Model Inference with Complete Mapping Flow on FPGAs. In ACM/SIGDA International Symposium on Field Programmable Gate Arrays (ISFPGA), 2024.
Juntao Zhao, Borui Wan, Chuan Wu, Yanghua Peng, and Haibin Lin. POSTER: LLM-PQ: Serv- ing LLM on Heterogeneous Clusters with Phase-Awar Partition and Adaptive Quantization. In Annual Symposium on Principles and Practice of Parallel Programming (PPoPP), 2024.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 LEARNED SCHEDULER DETAILS
A.1.1 MODEL ARCHITECTURE
To avoid runtime overhead, we employ lightweight attention module for Learned scheduler. Given an input Key cache K E RT'x>, and Value cache V E RR'xD the output of Learned scheduler is computed as:
O Softmarl qe KT
I vDk
(4)
Logils Softmar So.Immar(MLPPOO))
(5)
where MLP feedforward neural network with hidden layer and ReLU its activation, and & R Px is learned query vector. In other words, the trainable parameters of the network
include the query vector and the parameters of the MLP.
A.1.2 TRAINING APPROACH
To prepare the training dataset, sample to create a completion task. For each sample, we generate sequences, each corresponding to high-precision step from {0, N-I+ N-I..... N-1 OL}. The ground-truth label is set to the lowest high-pree gich-prccision step such that the Rouge-L score of the sequence matches or exceeds that of the fp/6 model output.
the C4 dataset the seed dataset and randomly truncate each
OL 201 [N-2]0L
During training, we use cross-entropy loss function as the objective.
A.2 ERROR RESILIENCE OF PREFILL AND DECODE BITWIDTH
Decoding/Prefill Strategy
Rouge-L
2bit Prefill, 2bit Decode 3bit Prefill, 2bit Decode
18.3
28.1
2bit Prefill, 3bit Decode
22.2
Table 4: Rouge-L scores for different decoding and prefill bitwidth allocations. Vicuna-7B and MT- Bench used. Outputs from fp16 model used reference answer.
observed that using higher bitwidth for prefilling significantly improves output quality. On the other hand, using higher bitwidth for decoding does not improve the perfor- mance much. This demonstrates that higher bitwidth should be used for prefilling.
As shown in Table 4
A.3 PERFORMANCE OF PMPD FOR LARGER MODELS
While PMPD mainly targets edge and mobile applications, it would be interesting to see larger models could also benefit from PMPD. We conducted additional experiments using Llama3- 8B @ Meta 2024) and longchat-16ks 13B (Li et al.) 2023a) models, with static scheduler for PMPD given the availability of validation data. The results shown in Table 5 demonstrate our method's generalizability across model scales.
A.4 GPU TND-TO-END LATENCY SPEEDUP
We provide the GPU end-to-end latency speedup as compared to FP16 models Table 6 PMPD consistently speedup improvement over the uniform quantization baseline upto 0.31 x.
14

---

Published as a conference paper at ICLR 2025
CNN/DM
Dialogsum
IWSLT
Modd (3x Liana3-88
Meezl Lama3-88
Medel Lamal-88
Method
Longhe--33 138
Eungitasi33
Rouge-L/ BERTScore
Rouge-L/ BERT
BLEU/
Bit
Bit
Bit
SacreBLEU
Baseline-I
75.8
5.58
2
0.0.1
Baseline-h
18.3 85.2
19.0/ 85,8
3
23.1/
PMPD-Static
2.66
17.8 84.8
2.31
18.7 85.8
2.71
22.1 /2.11
Baseline-1
9.29 72.8
10.6 80.8
2
2.94
Baseline-h
19.9 86.4
3
19.9 / 86.6
3
21.6/
PMPD-Static
2.37
19.9 86.4
2.0
20.3 / 86.5
2.71
21.4/214
Table 5: Performance of PMPD on models larger than 7B.
GPU 4090
Method
Vicuna-7B
MobileLlama
Phi-1.5
Zephy-3B
Baseline-h
3.05x
1.80x
1.75x
2.07x
PMPD
3.36x
1.90x
1.83x
2.22x
A40
Baseline-h
2.67%
1.71x
1.66x
1.91x
PMPD
2.91x
1.82x
1.74x
2.07x
Table 6: End-to-End GPU latency speedup as compared to FP16 model.
A.5 GPU MATRIX MULTIPLICATION LATENCY EVALUATION
Tab Table 8 report the GPU kernel latency using the self CUDA time metric on RTX 4090 and A40 GPUs. For PMPD, the latency calculated as a weighted average of the fixed hitddbbtwidthh latencies, where the weights correspond to the average number of decoding steps performed at each bitwidth across datasets. select between the Static and Learned schedulers, we choose the one with the highest performance. For MobileLLaMA and Vicuna-7B, bitwidths are averaged over the CNN/DM, Dialogsum, and IWSLT datasets. For Phi-1.5, bitwidths are averaged over the CNN/DM and Dialogsum datasets, while for Zephyr-3B, the bitwidth is from the IWSLT dataset.
A.6 NPU SIMULATION DETAILS
We developed an analytical performance model to emulate the deployment of PMPD on FlightLLM (Zeng et al., 2024). an NPU architecture that has been optimized for LLM inference. FlightLLM consists of: a unified Matrix Processing Unit that can perform multiple types of mul- tiplications between matrices and/or vectors through a hierarchical structure of multiply-accumulate (MAC) units, üi) a Special Function Unit where L.LM-seccfiic operations such as softmax, layer normalisation etc are mapped and iii) optimized memory hierarchy for the computational pattern of LLMs, while keeping all activation on-chip during decoding. The underlying design space of FlightLLLM's NPU architecture is traversed through a design space exploration method that takes into consideration the need to support different precisions of LLM weights. For our experiments, we instantiate two NPU configurations consisting of: 4K and 16K MAC units with GHz clock frequency (i.e. 8 and 16 teraops/sec (TOPS) peak throughput, respectively) for the deployment of smaller- (MobileLLaMa- 1.4B, Phi-1.5) and larger-scale LLMs (Vicuna-7BB respectively, and with 32 GB/s off-chip memory bandwidth.
A.7 ALTERNATIVE PTQ METHOD
Our approach offers a plug-and-play framework compatible with any PTQ method. To showcase its generalizability. evaluate the performance of both Static and Learned schedulers using GPTQ, uniform quantization technique (Frantar et al., (023). As shown in Table 9 both schedulers deliver performance that closely matches the high-precision baseline.
15

---

Published as a conference paper at ICLR 2025
Model
proj
k proj
proj
proj
up proj
down proj
gate proj
NobieeLLaW.A.L.BB-Chatt
fp16
5
5
13.3
15
w2
3
3
3
4
w3
3
3
3
4.3
4.57
w4
4
4
4
5.1
5.97
phi-1
fp16
5
5
5
10.6
11.8
w2
2.96
2.82
2.93
2.82
4.96
4.94
w3
3
2.81
2,89
2.91
5.36
5.92
w4
3.98
3.79
3,86
3.9
6.92
6.93
stablclm-eehyyr-Bb
fp16
10
10
10
10
11
18
11
W2
2
2
4
6
4
w3
4
4
w4
5
vicuma-7b-v1.5
fp16
25
25
25
25
97.1
101.93
97
w2
4
4
3.8
4
7
8
7
w3
4
4
4
8.1
w4
12
Table 7: GPU matrix kernel latency (uS) on RTX 4090.
Model MobieLLaMMA-1AB-Chhll
q-proj
k_proj
v-proj
o.proj
up- up-proj
down down-proj
gate.proy
fp16
18
18
18
18
42.4
47.5
42
w2
4.9
4.9
4.7
7.4
7.5
w3
5
5
5
8
w4
phi-1 5
fp16
18
18
18
18
62.2
60.3
w2
5.9
5.7
5.9
5.9
10
11
w3
6
12
13
w4
20.3
21.3
stablelm-zephyr--bb
fp16
28
28
28
28
66.1
70.1
66.4
w2
6
6
11
10
11
w3
13
12,9
w4
22.1
vicuna-7b-v1.5
fp16
64
64
64
64
170
162
170
w2
9
9
24.3
30.9
24.2
11.2
33.9
33.4
w4
21.2
21.5
21.3
21.2
46.6
58.9
45.9
Table 8: GPU matrix kernel latency (us) on A40.
A.8 EXAMPLE OUTPUTS BY PMPD
this section, we include some example outputs by Vicuna-7B with PMPD on MT-Bench, The precisions used are and 4 bits.
Success Case
Question: How do the stages of life shape understanding of time and mortality?
Answer: The stages of life shape understanding of time and mortality in several ways.
1. Childhood: Children moment and less concerned with the future. This can lead to a more relaxed and flexible understanding of time and mortality.
often more focused the present
16

---

Published as a conference paper at ICLR 2025
Method
Bit
Rouge-L. 9.71
BertScore
3bit
82.7
4bit
16.0
84.2
PMPD-Static
3.63
16.0
84.6
PMPD-LLearned
3.02
14.2
84.2
Table 9: Performance of GPTO-quantized MobileLLaMA on Dialogsum.
2. Adulthood: As we enter into adulthood, we become more aware of the passage of time and the possibility of death. This can lead to a more pessimistic view of time and mortality.
3. Midlife: Midlife is time of significant life changes, including career changes, family responsibilities, and the possibility death. This can lead to more realistic and optimistic view of time and mortality.
4. Later life: As we age, we may experience changes in our lives and the possibility of death. This can lead to a more realistic and pessimistic view of time and mortality.
Overall, the stages of life can shape our understanding of time and mortality by shaping our understanding of the present moment and the possibility of death.
The PMPD model able to produce sensible results for humanity subjects.
A Failure Case
When a number is divided by 10, the remainder is 4. What is the remainder when twice the number is divided by 4?
Question:
Answer: To find the remaindes when a number divided by 4, you can use the following steps:
1. Write down the number you want to find the remainder of.
2. Write down the number you want to find the remainder of again.
3. Multiply the number you wrote down times.
4. Write down the result.
Here, while coherent texts are generated, the answer not very relevant to the question.
17


Reviews:
[{"review_id": "Etmty98Lxf", "reviewer": "Reviewer_RDtY", "summary": "The authors point out that different stages of LLM inference require different quantization bit-widths. In oracle experiments, they discovered that increasing the model weight bit-width during the prefill phase can enhance model performance. They also find that using a higher bit-width in the early stages of decoding helps maintain model performance. Based on these findings, the authors search for different quantization bit-widths for the prefill and decode stages. Additionally, they use an optimized scheduler to gradually reduce the model bit-width during decode stage. Using these two schemes, the quantized model can ensure output quality on the challenging MT-Bench task. Meanwhile, the quantized model achieves a throughput improvement of 3.8-8.0× on NPU and a linear layer acceleration of 1.4-12.2× on GPU.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Compared to previous methods of mixed-precision quantization within models, this paper innovatively considers introducing mixed-precision quantization at different stages of LLM inference.\n\n2. This paper not only validates the algorithmic performance of the proposed method but also measures the hardware performance like speedup ratio and throughput on both GPUs and NPUs.", "weaknesses": "1. In Section 3.1 of this paper, the experiment results are insufficient to demonstrate the validity of Motivation One. \n\n    - First, the authors only increase the bit-width during the prefill stage and found that model performance was better than that when using uniform bit-width, but do not increase the bit-width during the decode stage to evaluate model performance. Therefore, the results do not demonstrate that the prefill and decode stages have different error resilience. \n\n    - Secondly, this section only presents the model's outputs for three specific inputs, lacking quantitative results.\n\n    - It is recommended that the authors supplement the results of high-precision decoding with low-precision prefill and provide performance of the model on MT-Bench to support Motivation One.\n\n2. In Section 4.1 of this paper, the authors search for the optimal bit-widths for the prefill and decode stages assuming that the decode stage is conducted with a uniform bit-width. However, during actual inference, the model bit-width in the decode stage gradually decreases. Therefore, it cannot be guaranteed that the previously found results remain optimal. Then what is the significance of this search step? Is it possible to directly use the bit-width of the prefill stage as the starting point for the bit-width decrease in the decode stage?\n\n3. In Section 4.3 of this paper, the authors design the Task-Agnostic Learned Scheduler to predict the steps at which the bit-width should decrease during the decode stage. Theoretically, the optimal precision switch points may depend on both the model's inputs and the outputs. However, this scheduler only uses the token features generated during the prefill stage as input. Is it reasonable to overlook the information from the model's generated content?\n\n4. The experimental results in this paper are not comprehensive enough. First, the maximum parameter size of the model used in the experiments is 7B, and larger models are not evaluated. Secondly, the paper only provides GPU speedup ratios for the linear layers and does not present end-to-end speedup data.\n\n5. This paper has misunderstandings regarding some concepts. For example, the maximum context length of LLMs (CL) refers to the sum of the model's maximum input and output lengths, but in Section 4.3, the paper incorrectly asserts that CL is the model's maximum output length.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wSR3Z8JMiE", "reviewer": "Reviewer_A5Dp", "summary": "This work tackles the difficulty of deploying large language models (LLMs) on resource-constrained devices by introducing a “phase-aware” quantization method that adapts precision to different inference stages, unlike traditional uniform quantization approaches. The authors develop Progressive Mixed-Precision Decoding (PMPD), a technique that lowers precision progressively as the generated sequence deepens, using schedulers to adapt precision based on tasks or prompts. Experimental results demonstrate that PMPD provides substantial speedups on Nvidia GPUs and optimized NPUs, while preserving output quality and surpassing the performance of both fp16 and uniform quantization models.", "questions": "1. How do you implement the precision switch operation during the decoding stage? And what is the actual cost of such switching?\n\n2. How's the result on Llama3? Or other GQA models?\n\n3. How's your method compared with static quantization methods like GPTQ and AWQ both for accuracy and efficiency?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The author discusses many design choices, with their pros and cons.\n- The discussion on the tolerance of quantization in different depths of the decoding stage is informative", "weaknesses": "- The paper does not discuss much about how they do the precision switching in their implementation, which could be a potential risk of memory usage (loading the same model with different precision) or additional latency (loading the model weight on the fly).\n- The accuracy results are all based on MHA models, lacking discussion on other types of models like GQA models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hO5x3jOTxD", "reviewer": "Reviewer_TCS8", "summary": "Authors propose a novel phase-aware method that selectively allocates precision during different phases of LLM inference: use higher precision during prefill stage and then reduce precision for efficient memory bandwidth utilization during decoding.\nThey introduce Progressive Mixed-Precision Decoding (PMPD) which controls gradual lowering of weights precision in the generated sequence with task-adaptive or prompt-adaptive manner. It is weights only post training quantization(PTQ) which adaptively controls weights precision depending on decode stage (prefill, decode) and decoded token. They use  Any-Precision LLM (Park et al., 2024), as PTQ which keeps multiple weight precisions with no memory footprint overhead.\n\nAuthors show that PMPD achieves 1.4−12.2x speedup in LLM linear layers over fp16 models (on Nvidia GPUs).\nAlso throughput gain of 3.8−8.0x over fp16 models and up to 1.54× over uniform quantization (on NPU) approaches while preserving the output quality.", "questions": "Q : What quantization method is used for Baseline-l and Baseline-h?\n\nQ: Why GPTQ are not presented as baselines in Table 1?\n\nQ: How would you explain that PMPD-Static has better performance in comparison to PMPD-Learned on Table 1?\n\nQ: Please explain what precision range was used for training PMPD shown on Table 1, e.g. (2bits, 3bits, 4bits) for all experiments or you use (2bits 3bits) for one one set of experiments and (3bits, 4bits) for another set of experiments?\n\nQ: For PMPD method you will need to keep pre-computed e.g. 2bits 3bits weights and then switch between different precision depending on input token, please explain how do you avoid overhead of keeping weights with different set of precision in memory in comparison to a baseline method which keeps weights with one set of precision e.g. 3bits only?\n\nQ: On Table 1, experiment on data Dialogsum, how do you explain that Baseline-l with 2bits has accuracy 10.2 / 75.5 but PMPD-Static with the same amount of bits = 2.0bits has better accuracy 25.0 / 88.2? Please explain what set of bits PMPD-Static uses, e.g. if it was trained with (2bits 3bits) then it means that PMPD-Static used only 2bits during inference, so that final number of bits is 2.0 (reported in Table 1).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "PMPD vs. Baselines: Both schedulers deliver lossless performance on the CNN/DM and Dialogsum datasets with up to 33% reduction in bitwidth, highlighting their ability to effectively capture context and generate high-quality outputs.\n\nNovelty in applying different precision for model weights depending on input token during decode mode.\n\nConducted thorough experiments on LLM edge-deployable models.\n\nProduced End-to-end NPU throughput and Speedup comparison vs fp16 model. Shows that PMPD achieves a significant speedup of 3.8-8.0× over fp16 across various models and datasets.\n\nGood ablation studies showing that  indicating that PMPD-Learned is effectively learning patterns in the KV cache", "weaknesses": "Q: in Table 1, PMPD-Static has better accuracy score in 11 cases; PMPD-Learned has better accuracy score in 3 cases; PMPD-Static uses less bits 5 cases; PMPD-Learned uses less bits 4 cases. It looks like PMPD-Learned under-performing on both accuracy and bits usage in comparison to PMPD-Static. Please analyze why PMPD-Learned underperforms compared to PMPD-Static. I see that PMPD-Learned is preferable in case when validation data are missing, because PMPD-Static can not be trained on data which are missing validation data sets.\n\nQ: Authors report 1.4−12.2x speedup in LLM linear layers over fp16 models for PMPD methods. Please also add speed up in comparison to 3bit and 4bit baseline models (you show it in Figure 5 and Table 3, but do not report in the abstract of the paper).\n\nQ: One of the goal of this paper is to show that proposed quantization approach has no accuracy drop in comparison to the float baseline model. Float baseline is missing in Table 1, so it is hard to make a conclusion that proposed methods do not drop accuracy in comparison to float baseline. Please include the float baseline results in Table 1 or explain why you chose not to include them (for example in Figure 5 you show fp16 latency of evaluated models). Same comment for Table 2.\n\nQ: It would be great to add in Table 1, other state of the art post training quantization methods, e.g. :\n* Fast and Efficient 2-bit LLM Inference on GPU: 2/4/16-bit in a Weight Matrix with Asynchronous Dequantization\n* QuIP: 2-Bit Quantization of Large Language Models With Guarantees or (QuIP#: Even Better LLM Quantization with\nHadamard Incoherence and Lattice Codebooks)\n* GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers\n\nPlease explain why above methods were not included as baselines, and how your method compares conceptually to these approaches if direct comparison is not feasible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zxO3wH6m4J", "reviewer": "Reviewer_FUCY", "summary": "The paper proposes phase-aware precision/quantization recipe which allocates higher bit depth for the prefill phase and lower/adaptive bit depth for the decoding phase in LLMs. They motivation is based off of previous work & representative examples shown in the paper that the prefill phase (language understanding phase) is likely to benefit from lower quantization errors than the decoding phase.\nAdditionally, authors propose a well executed \"adaptive\" precision strategy in the decoding phase, where the precision is reduced after every few steps of the generation. \nThe experiments on three small-scale mobile LLMs are show the effectiveness of the approach in terms of quality and latency/throughput of the kernels as compared to the other methods which use fixed bit-depth through out the generation.", "questions": "1) IMO the most interesting section of the paper is the scheduler design in Section 4.3. As such a lot of core ideas/implementation details of the scheduler are relegated to the Appendix. It maybe better to add scheduler model/training design to the main paper, while reducing redundant text in the introduction, methodology and the motivation section. \n2) When adding the constraint in line 325-327, it's not entirely clear how is N chosen, is this a hyper-parameter, what is recommended value? Does it depend on task/model that were studied? A detail ablation can be performed in the experiment section, but a brief summary for practioners can be provided here.\n3) Table 1 caption says the phi-1.5 models are 2/4 bit, but the table itself shows they are 2/3 bit?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "It is very critical to address the memory-boundedness of small LLMs operating on mobile phones. Quantization is the most prevalent method for model compression but suffers from quality-latency tradeoff. The paper proposes \"adaptive\" precision in the decoding stage, by training a \"precision scheduler\", which is very interesting. It exploits the fact that later half of the tokens in the generation process can afford higher quantization error in the weights. Evaluation shows that the proposed methods achieves better (or at-least maintains) better quality than high bit-depth quantization methods, while increasing the throughput which is critical for mobile performance of small LLMs. The motivation is clear and backed with relevant prior work.", "weaknesses": "1) There is a lot of repetition of concepts in the first few pages of the paper. The same idea is explained several times in the first three sections. It maybe better to show more representative examples similar to Figure 2 -- perhaps some high level stats/evals on how different tasks/models perform when changing the precision in the decode/prefill phase.\n2) The work misses several critical LLM benchmarks around Math, reasoning and especially long context. It would be interesting to see the effect of the proposed approach on a wider range of LLM evals. \n3) This experimentation section fails to provide more statistics on evaluation, for example, what is the average decode length for the three tasks for the thee models, what is the impact of changing N (the number of \"switches\" in precision during decoding phase). It is not entirely clear what precision the scheduler choses and for how long at each decoding step, -- for three different tasks and the three models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)