Progressive Mixed-Precision Decoding for Efficient LLM Inference

## Abstract

Abstract In spite of the great potential of large language models (LLMs) across various tasks, their deployment on resource-constrained devices remains challenging due to their excessive computational and memory demands. Quantization has emerged as an effective solution by storing weights in reduced precision. However, utilizing low precisions ( i.e. 2/3-bit) to substantially alleviate the memory-boundedness of LLM decoding, still suffers from prohibitive performance drop. In this work,
we argue that existing approaches fail to explore the diversity in computational patterns, redundancy, and sensitivity to approximations of the different phases of LLM inference, resorting to a uniform quantization policy throughout.
Instead, we propose a novel phase-aware method that selectively allocates precision during different phases of LLM inference, achieving both strong context extraction during prefill and efficient memory bandwidth utilization during decoding. To further address the memory-boundedness of the decoding phase, we introduce P rogressive M ixed- P recision D ecoding (PMPD), a technique that enables the gradual lowering of precision deeper in the generated sequence, together with a spectrum of precision-switching schedulers that dynamically drive the precision-lowering decisions in either task-adaptive or prompt-adaptive manner.
Extensive evaluation across diverse language tasks shows that when targeting Nvidia GPUs, PMPD achieves 1.4 − - - 12.2 × \times × speedup
in LLM linear layers
over fp16 models, while when targeting an LLM-optimized NPU, our approach delivers a throughput gain of 3.8 − - - 8.0 × \times × over fp16 models and up to 1.54 × \times × over uniform quantization approaches while preserving the output quality.

## 1 Introduction

Modern large language models (LLMs) have demonstrated unprecedented capabilities across various natural language understanding and generation tasks.
However,
their computational and memory footprint, driven by billions of parameters and thousands-of-tokens context windows, pose significant challenges particularly in applications that require their deployment on embedded or mobile devices.
LLM inference operates in two phases: i) the prefill phase, which processes all input prompt tokens in parallel, and ii) the decoding phase, which generates output tokens one by one in an autoregressive manner.
Computationally, the prefill phase is known to be compute-bound, while the decoding stage is memory-bound, leading to underutilization of computational resources and often dominating inference latency (Kwon et al., ; Yuan et al., ), especially on resource-constrained devices where batching is not an option (Yu et al., ; Agrawal et al., ).

To enhance the hardware performance of LLM inference at the edge, several approaches have been proposed, including model compression and adaptive inference techniques. Particularly, quantization has been widely studied, to adopt low-precision representations for the weights (Yu et al., ) and/or activations (Tan et al., ), as well as its Key-Value (KV) cache (Hooper et al., ).
Nevertheless, most existing approaches apply a uniform quantization scheme across both prefill and decoding stages, which presents two key limitations:
i) They overlook the distinct fault tolerances against quantization errors in prefill and decoding stages, leading to a significant accuracy drop while reducing the bit-width for mobile-scale LLMs; ii) Since the decoding phase is memory-bound, reducing the bitwidth in this stage is more effective in improving hardware performance.
As such, previous uniform quantization approaches can only achieve limited speedups, underscoring the need for more targeted quantization strategies specifically tailored to LLM decoding.

Figure: Figure 1: Illustration of the conventional paradigm of quantized LLM inference (left) and our method that comprises phase-aware precision allocation and progressive mixed-precision decoding (right). Our approach is motivated by i) the distinct error resilience observed during the prefill and decoding phases, and ii) the increasing fault tolerance as decoding progresses to longer tokens.
Refer to caption: x1.png

By observing the key limitations in prior work,
this paper identifies a novel insight for LLM weight quantization: The prefill phase, as well as earlier parts of the decoding phase, are more sensitive to approximation errors from quantization, than later parts of the autoregressive generation process.
Building on this insight, we propose a novel LLM inference method that counteracts the limitations of existing approaches by means of a phase-aware and progressive reduced-precision approach, shown in Figure 1.
Specifically, our method departs from the existing deployment approaches that employ uniform quantization precision throughout the LLM inference process and introduces phase-aware precision allocation. By considering the distinct redundancy, sensitivity to approximations, and arithmetic intensity of the prefill and decoding phases, our scheme tailors the arithmetic precision to each phase’s characteristics. This leads to both maintained generation quality through high-precision prompt encoding during prefill, and improved sustained throughput through reduced precision during decoding.

To further alleviate the memory-boundedness of decoding, we introduce progressive mixed-precision decoding (PMPD), a novel decoding scheme that gradually reduces precision throughout the generation of the LLM response. Based on the observation that tokens generated later in the output sequence are more resilient to approximations while earlier tokens are more sensitive, our method employs weight quantization as an approximation mechanism and progressively reduces the numerical precision during the decoding phase.
To balance generation quality with decoding throughput, we formulate precision scheduling as a constrained optimization problem with the objective of minimizing the average bitwidth while preserving output quality. As a solution, we design two complementary precision-switching schedulers, one high-performance task-specific static scheduler and one flexible task-agnostic learned scheduler, to strategically determine the highest-performing precision-reduction time.
By applying our method on Vicuna-7B, MobileLLaMA-1.4B, Stable LM Zephyr-3B, and Phi-1.5 across diverse language tasks, we achieve decoding throughput gains of 3.8-8.0$\times$ on NPU platform and 1.40-12.20$\times$ speedup on LLM linear layer
computations on GPU.
The main contributions of this work can be summarized as follows:

- •
A novel phase-aware precision allocation strategy that optimizes precisions differently for the prefill and decoding phases, leveraging the distinct error resilience of each stage to achieve an extremely low average bitwidth for mobile-grade LLMs.
- •
A progressive mixed-precision decoding scheme that progressively reduces the precision as decoding progresses to longer token sequences, effectively improving the hardware performance in the memory-bound LLM decoding stage.
- •
A prompt-agnostic static scheduler and a task-agnostic learned scheduler for precision switching, enabling the flexible deployment of our approach across diverse scenarios, accommodating varying data availability constraints and generation quality requirements.

## 2 Background and Related Work

### 2.1 Low-Precision LLM Inference

Several quantization approaches have been employed to reduce the precision of the weights (Frantar et al., ), activations (Dettmers et al., ; Tan et al., ) and KV cache entries of LLMs (Hooper et al., ), aiming to boost their computational efficiency.

Weight-only Quantization. In particular, weight-only quantization approaches for LLMs result in remarkable speedups upon deployment on commodity platforms, by alleviating the volume of memory transactions during decoding which, dominated by weight transfers, forms the main processing bottleneck in LLM inference (Yuan et al., ). Along these lines, and inspired by relevant literature on deep neural networks (DNNs) (Yu et al., ), Any-Precision LLM (Park et al., ) enables the post-training quantization of a single set of weights to multiple lower precisions. This creates a family of variably quantized model variants, without any storage overhead to the original model. To further improve the performance of low-bit quantization, Dense-and-Sparse Quantization (DNS) (Kim et al., ) increases the average bitwidth by storing a small fraction of outlier weights in full precision while keeping the remaining weights in quantized format. Finally, BitNet (Ma et al., ) introduces ternary weights, but requires costly quantization-aware training. In contrast, this work focuses on low-overhead post-training quantization for efficient LLM inference.

Nonetheless, existing methods typically adopt a uniform precision across phases, which also remains fixed across all input prompts. Although this approach can deliver near lossless approximate inference with very low precisions on highly redundant large-scale LLMs, it often struggles to maintain performance under aggressive quantization (e.g. 2 or 3 bits) for smaller-scale models. Our approach takes advantage of the low memory footprint solution provided by Any-Precision LLM to accelerate LLM inference, but adopts a dynamic precision lowering methodology, which considers the LLM inference phase and input prompt at hand, to push the limits of the adopted precision.

### 2.2 Precision-Adaptive Approaches

The proposed approach exploits weight quantization, through a dynamic phase-aware and progressive precision lowering methodology. As such, more relevant to our method is a line of work that proposes ways to dynamically adapt bitwidth at run time.

Precision-Adaptive Training. MuPPET (Rajagopal et al., ), CPT (Fu et al., ) and AdaPT (Kummer et al., ) aim to improve the efficiency of the training stage by dynamically adjusting the arithmetic precision throughout the training process. This family of methods focuses primarily on classification tasks. In contrast, our method is optimized for the characteristics of LLMs and focuses on improving the efficiency of the inference stage.

Mixed-Precision Inference. HAQ (Wang et al., ) and HAWQ (Dong et al., ; ) allow for different precisions across the layers of a given DNN. Nonetheless, the selected precision remains fixed across all processed input samples. Bit-mixer (Bulat & Tzimiropoulos, ) allows a similar utilization of mixed precision across layers, but can adjust the selected bitwidths in a per-sample manner. CascadeCNN (Kouris et al., ) adopts different-precision variants of the same DNN, organised in a low-to-high precision classifier cascade. Samples that fail to meet hand-tuned confidence-based criteria on the low-precision stage are propagated for re-computation with higher precision. As before, these approaches are developed for CNN-based classification tasks and are not off-the-shelve applicable to the autoregressive decoding process of LLM inference. Closer to our work, LLM-PQ (Zhao et al., ) adjusts model precision based on the hardware support of different servers, along with a phase-aware model partitioning, targeting a distributed LLM inference serving setup. In contrast, our approach adapts precision at different time steps rather than across different machines and is optimized for edge-based deployment.

## 3 Motivation

### 3.1 Observation One: Different Error Resilience in Prefill and Decoding

Recent work has shown that allocating larger model capacity to the prefill phase for natural language understanding allows for reduced capacity in the decoding phase for natural language generation, while producing responses with both high quality and improved efficiency (Aishwarya et al., ). Inspired by this, we aim to investigate the distinct redundancy and resilience exhibited by the prefill and decoding phases in LLM inference.

To this end, we target three diverse language tasks and compare the response from two variants of the same LLM (Vicuna-7B (Chiang et al., )): i) a uniformly low-precision variant with 2-bit quantized weights in both prefill and decoding, and ii) a two-precision variant with higher-precision 3-bit weights during prefill and lower-precision 2-bit weights during decoding.

Figure 2 shows the responses of the two LLM variants. We empirically observe that using the 3-bit model for prefilling (bottom row) significantly enhances the generative capabilities of the uniform 2-bit model (top row). We attribute this result to the significant steering role that the prefill phase has to the output generation process. Specifically, a high-quality KV cache extracted from the input prompt during prefilling provides good understanding of the task and information-rich context, enhancing in this way the generation capability of the low-precision model. This insight motivates us to explore and study different precision policies for the prefill and decoding phases.

Figure: Figure 2: Phase-aware Precision Allocation. Here prefilling was performed by 2- (top) or 3-bit (bottom) Vicuna-7B, while decoding employed the 2-bit model for both cases. We observe three key improvements. i) enhanced reasoning abilities. The model demonstrated higher accuracy for numerical tasks, such as calculating Fibonacci numbers (left). ii) improved instruction-following. The model understood and responded to user instructions (middle). iii) strengthened emerging abilities like creativity. The model successfully handled open-ended tasks such as poem writing (right).
Refer to caption: x2.png

### 3.2 Observation Two: Progressively Lowering Precision during Decoding

To further increase the output quality of reduced-precision generation, we explore the optimization space of mixed-precision decoding. Figure 3 depicts our findings when applying different approaches of scheduling between low- and high-precision weights. Namely, we explore employing the high-precision weights in the first half, middle part, and last half of the generated tokens.

In line with our hypothesis, adopting high precision at the first half of the decoding stage yields the best performance, matching the performance of using high precision throughout the generation process. One intuitive explanation is that using high precision to generate the first few tokens minimizes the error accumulation that would affect the rest of inference. This observation also aligns with the “attention sink” phenomenon (Xiao et al., ), where it was observed that the attention scores tilt heavily towards the initial tokens, indicating their higher importance. As a reference, we also tested the performance of alternating the two models at each token, which also proves to work well. However, frequent precision switching introduces hardware overhead, both in terms of weights loading time and additional memory traffic, so this schedule is not considered practical for real deployment.

Figure: Figure 3: Different schedules mixing 3- and 4-bit models for decoding. We examined text summarization on CNN/DM using Phi-1.5. To ensure fair comparison, the same number of tokens are generated by the 4-bit baseline. Moreover, the 4-bit model was used for prefilling for all schedules.
Refer to caption: x3.png

## 4 Methodology

Motivated by our findings in Section 3, suggesting that tokens deeper in the decoded sequence are more resilient to approximations, we adopt weight quantization as our approximation mechanism and propose a new mixed-precision decoding method, comprising two key techniques: i) phase-aware precision allocation and ii) progressive mixed-precision decoding (PMPD). With the objective of maintaining generation quality while maximizing efficiency during inference, the proposed techniques operate complementarily towards applying precision selection and scheduling strategies that are tailored to the respective characteristics of the prefill and decoding phase of LLMs. To enforce our approach with fine-grained control, we further introduce two types of runtime schedulers that drive precision-switching decisions in a prompt- and task-agnostic manner, respectively.

Algorithm 1 presents our end-to-end methodology, consisting of: i) the offline stage (lines 1-8) and ii) deployment (lines 9-22).
The offline stage is responsible for producing variably quantized variants of a given LLM (line 1), conducting the phase-aware precision allocation (line 2), and configuring the selected type of precision-switching scheduler with optimized PMPD parameters for the target use case (lines 3-8). Upon deployment, the scheduler assigns the allocated precisions to the prefill and decoding stages, and triggers precision-switching events for each encountered prompt (line 20).

### 4.1 Phase-Aware Precision Allocation

The proposed Phase-Aware Precision Allocation capitalizes on the unique characteristics of each LLM inference phase and tailors accordingly the per-phase weight precision. Specifically, our inference scheme dictates that a high-precision model is used during the prefill phase – allowing the extraction of rich context from the input prompt – which in turn enables the use of lower-precision weights in the memory-bound decoding phase, and thus yields more efficient utilization of the memory bandwidth. As discussed in Section 3, since the prefill phase is compute-bound and the LLM inference latency is typically dominated by the decoding phase, using a high-precision model during prefill introduces negligible latency overhead.

Formally, our scheme is parametrized with respect to: 1) a quantizer $Q(\cdot)$, which is typically a post-training quantization (PTQ) method, 2) a set of precisions $\mathcal{P}$, indicating the supported weight bitwidths, 3) a reference value $q_{\text{ref}}$ in a task-specific quality metric, and 4) a quality drop tolerance $\epsilon$. The objective of our method is to find the pair of smallest weight precisions for the prefill and decoding phase, respectively, that achieves equal or greater algorithmic performance than $q_{\text{ref}}-\epsilon$.

We solve this objective by means of a calibration step during the offline stage of our methodology. First, given a pretrained LLM $m$, we obtain a set of variably quantized models $\{m_{p}\}$ containing one quantized model variant per arithmetic precision $p$ in precision set $\mathcal{P}$ using quantizer $Q(\cdot)$ (line 1). Next, we perform the phase-aware precision allocation step (line 2). Concretely, the pretrained LLM is evaluated on a calibration dataset, by assessing its algorithmic performance using different combinations of prefill and decoding bitwidths. Through this process, we determine the lowest pair of precisions that meet the quality target (line 2).

### 4.2 Progressive Mixed-Precision Decoding

Exploiting the insights of Section 3.2, we introduce Progressive Mixed-Precision Decoding (PMPD) to further improve decoding efficiency.
As depicted in Figure 1-Right, PMPD exploits the variability between different tokens in terms of resilience to approximation, by gradually lowering the arithmetic precision of weights as we go deeper in the output sequence. In this manner, we maintain higher precision for earlier tokens, which often play a more decisive role in the response quality, while generating deeper tokens more rapidly through the use of narrower-bitwidth weights.

Figure: Algorithm 1 Progressive Mixed-Precision Decoding

We parametrize PMPD with i) the precision set $\mathcal{P}$ and ii) the precision-switching schedule $S=\{st(p)\leavevmode\nobreak\ |\leavevmode\nobreak\ p\in\mathcal{P}\}$, which consists of the possible switching points of each precision defined as $st(p)\in[1,CL)$, where $CL$ is the maximum context length of a given LLM. Here, the starting point $st(p)$ is defined as the token index of the output sequence where we perform a switch from the current precision to the lower precision $p$. Under this setup, to obtain the highest-performing PMPD
configuration, we pose the following optimization problem:

$$ $\displaystyle\min\leavevmode\nobreak\ st(p),\quad\forall p\in\mathcal{P} \setminus\{p_{\text{min}}\}$ (1) $\displaystyle\text{ s.t. }\quad q_{\text{ref}}-\epsilon\leq q(S)\quad\&\quad 0 \leq st(p)<CL,\quad p\in\mathcal{P}$ (2) $\displaystyle\phantom{\text{ s.t. }}\quad p>q\implies st(p)\leq st(q),\quad p, q\in\mathcal{P}$ (3) $$

where $q(S)$ is the achieved quality of schedule $S$. Our objective function aims to minimize the number of tokens to be processed in each precision in the precision set –except for the lowest precision $p_{\text{min}}$ which will be used last and until the end of the sequence for higher throughput– subject to meeting the specified quality target (first constraint in Eq. (2)), not exceeding the LLM’s maximum context length (second constraint in Eq. (2)), and following a progressive precision lowering approach (precedence constraint in Eq. (3)).

### 4.3 Precision-Switching Scheduler

Upon deployment, PMPD is enforced by means of a precision-switching scheduler (Figure 4). The scheduler is configured with the PMPD parameters $\left<\mathcal{P},S\right>$ and issues precision-switching actions to the system processor following schedule $S$ for each input prompt.

Given the set of variably quantized models $\{m_{p}\}$ that was generated in the phase-aware precision allocation step (Section 4.1 and line 1), the objective function of Eq. (1) can be evaluated for all combinations of precision-switching schedules. Ideally, the highest-performing schedule considers all $p\in\mathcal{P}$ per-prompt per-step when deciding for a precision switch without the precedence constraint shown in Eq. (3). This schedule can be obtained through an exhaustive enumeration over ${|\mathcal{P}|}^{CL}$ schedules. However, for realistic values of $|\mathcal{P}|$ and $CL$, this computation quickly becomes intractable. Besides, even if such a schedule can be efficiently found, it is, in most cases, impractical as it would require excessively frequent switching, e.g. per decoding step, between variably quantized weights, aggravating the memory bandwidth demand (Section 3.2).

Figure: Figure 4: Precision-switching scheduler.
Refer to caption: x4.png

Instead, through our approach of progressive precision lowering during decoding, the total number of schedules to be examined is reduced to $\sum^{|\mathcal{P}|-1}_{r=0}\frac{CL!}{r!(CL-r)!}\frac{(|\mathcal{P}|-1)!}{r!(
|\mathcal{P}|-1-r)!}$ where $r$ is the number of times the precision switches. Nonetheless, the search time for the optimal schedule increases with $CL$ and $|\mathcal{P}|$, which is still prohibitive at run time. Hence, to further reduce the search time for the optimal $S$ to $O(1)$, we parameterize the number of candidate precision-switching points, denoted by $N$, and add a range constraint to the second constraint in Eq. (2) as $st(p)\in\{0\}\cup\{\frac{CL}{(N-1)},\frac{2CL}{(N-1)},...,\frac{(N-2)CL}{(N-1)
},CL\}$.
Under this setup, we propose two types of schedulers: i) a prompt-agnostic static scheduler, where $S$ is determined prior to deployment, and ii) a task-agnostic learned scheduler, where $S$ is derived dynamically for each prompt.

Prompt-Agnostic Static Scheduler. To utilize an efficient precision scheduler, we propose a static approach that selects schedule $S$ through an offline optimization process (line 4) that utilizes a calibration set. While the precedence constraint in Eq. (3) and the optimization objective in Eq. (1) still hold, the achieved quality in Eq. (2) is approximated as $q(S)\approx\frac{1}{|\mathcal{V}|}\sum_{x\in\mathcal{V}}q_{x}(S)$
where $\mathcal{V}$ is the validation set and $q_{x}(S)$ is the measured quality score of the schedule on a sample $x$.

As the validation set is defined per task, the static scheduler remains prompt-agnostic (i.e. uses the same precision-switching schedule for all prompts) and task-specific. As such, this type of scheduler is tailored to the resilience of the task at hand and has the advantage of inducing close to no runtime overhead. However, by adopting a uniform schedule within a given task, it requires the availability of a task-representative validation set, which is often not a realistic assumption, and it does not exploit the variability in difficulty and resilience to approximation across different prompts.

Task-Agnostic Learned Scheduler. To eliminate the need for a per-task validation set, we introduce a trainable scheduler that adjusts the schedule $S$ to each input prompt. For a given LLM, our learned scheduler is trained on a generic, task-agnostic dataset and then applied across various downstream tasks, amortizing in this way its training cost. Specifically, we aim to devise a schedule given the features of each input prompt. Training the scheduler to make very fine-grained per-step decisions, however, would require both substantial model capacity and excessively frequent scheduler invocation, which induce significant runtime overhead and counteract PMPD’s speedup benefits.

Hence, we utilize the features extracted in the prefill phase as inputs to our scheduler. With this approach, we are able to maintain a lightweight architecture for the scheduler and introduce minimal overhead during inference as the scheduler generates schedule $S$ once before decoding (line 13). In Section 5.3, we investigate different input features, i.e. KV caches and activations, from various layers of the model and find that utilizing the prefilled KV cache is efficacious in most cases, supporting our claim in Section 3.1 that the prefilled KV cache provides good task-specific context.
We also find that despite not requiring a task-specific validation set and labeled ground truths, our learned scheduler performs close to the static and, in some cases, even outperforms it in both algorithmic and hardware performance. Its architecture and training are detailed in Appendix A.1.

## 5 Evaluation

Models and Datasets. We conducted experiments on edge-deployable models, including Vicuna-7B (Chiang et al., ), MobileLLaMA-1.4B (Chu et al., ), Stable LM Zephyr-3B(^1^11https://stability.ai/news/stablelm-zephyr-3b-stability-llm), and Phi-1.5 (Li et al., ), evaluating their zero-shot generative performance on news summarization, dialogue summarization, and translation tasks using the CNN/DM (Hermann et al., ), Dialogsum (Chen et al., ), and IWSLT French-English datasets (Cettolo et al., ), respectively. These tasks encompass context understanding and text generation, providing a comprehensive evaluation of PMPD(^2^22Hereafter, PMPD refers to our approach including both phase-aware precision allocation and progressive mixed-precision decoding.). We also tested open-ended question answering on MT-Bench (Chiang et al., ), a special case where Static scheduler is infeasible due to the absence of a validation set.

Baselines. We compare against single-precision quantized models, and quantized models enhanced with Dense-and-Sparse decomposition (DNS) (Kim et al., ), with DNS ratios adjusted to match PMPD’s average bitwidth. PMPD’s average bitwidth is calculated as the weighted average of bitwidths over decoding steps, weighted by the number of tokens generated at each precision.

PMPD Implementation Details. For PTQ method, we adopted the nested quantization method of Any-Precision LLM (Park et al., ), ensuring that using multiple weight precisions incurs no memory footprint overhead.
We further explore the effect of using other PTQ methods in Appendix A.4.
The Static scheduler finds a schedule that minimizes high-precision steps on each benchmark’s validation set while maintaining lossless performance.
The Learned scheduler was trained using the first 256 samples from the C4 test dataset as the seed dataset. The high-precision variant of each model is quantized to the lowest lossless precision determined by perplexity on the C4 dataset, while low precision is defined to be one bit lower than the high precision.

GPU Latency.
Following Any-Precision LLM (Park et al., ), we evaluated the latencies of linear layers in the LLMs across different Nvidia GPUs, including RTX 4090 and A40. As our implementation was based on PyTorch (Paszke et al., ), we employed its built-in Profiler tool and reported the average self CUDA time metric over 100 forward passes to ensure reliable results.

NPU Simulation Measurements. To estimate the processing speed of PMPD when deployed on an NPU, we developed an analytical performance model of the hardware architecture of FlightLLM (Zeng et al., ), an LLM-optimized accelerator, adapted to support multi-precision weight loading. More details can be found in Appendix A.3.

### 5.1 Algorithmic Performance

**Table 1: Performance comparison of Static and Learned schedulers against low-precision (baseline-l) and high-precision (baseline-h) baselines, as well as DNS with the same average bitwidth as the scheduler with highest performance. For CNN/DM and Dialogsum, models used from top to bottom are Vicuna-7B (2/3 bit), MobileLLaMA (3/4 bit), and Phi-1.5 (2/4 bit). For IWSLT, the models are Vicuna-7B, MobileLLaMA, and Zephyr-3B (3/4 bit). Pairwise winners between the scheduler and DNS are underlined, while the highest overall scores are in bold.**
| Method | CNN/DM | Dialogsum | IWSLT |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model ($\downarrow$): Vicuna-7B | Model ($\downarrow$)): Vicuna-7B | Model ($\downarrow$)): Vicuna-7B |  |  |  |  |
| MobileLlaMA , Phi-1.5 | MobileLlaMA, Phi-1.5 | MobileLlaMA, Zephyr-3B |  |  |  |  |
| Bit | Rouge-L/ | Bit | Rouge-L/ | Bit | BLEU/ |  |
| BERTScore | BERTScore | SacreBLEU |  |  |  |  |
| Baseline-l | 2 | 8.30 / 78.4 | 2 | 10.2 / 75.5 | 2 | 1.2 / 1.2 |
| Baseline-h | 3 | 24.2 / 86.9 | 3 | 24.4 / 88.2 | 3 | 31.6 / 31.6 |
| DNS | 2.39 | 24.2 / 86.8 | 2.0 | - | 2.68 | 27.6 / 27.6 |
| PMPD-Static | 2.39 | 24.3 / 87.0 | 2.0 | 25.0 / 88.2 | 2.68 | 31.0 / 31.1 |
| PMPD-Learned | 2.43 | 24.0 / 86.7 | 2.74 | 24.5 / 88.2 | 2.37 | 29.9 / 29.9 |
| Baseline-l | 3 | 16.3 / 83.3 | 3 | 15.8 / 84.1 | 3 | 9.8 / 9.83 |
| Baseline-h | 4 | 17.2 / 83.5 | 4 | 16.8 / 84.9 | 4 | 12.7 / 12.7 |
| DNS | 3.37 | 17.4 / 83.5 | 3.21 | 14.7 / 84.4 | 3.65 | 12.0 / 12.0 |
| PMPD-Static | 3.37 | 17.6 / 83.7 | 3.0 | 17.0 / 85.0 | 3.65 | 12.6 / 12.6 |
| PMPD-Learned | 3.19 | 16.6 / 83.2 | 3.21 | 17.1 / 85.0 | 3.48 | 11.8 / 11.8 |
| Baseline-l | 3 | 13.4 / 82.4 | 3 | 15.3 / 85.1 | 3 | 21.1 / 21.1 |
| Baseline-h | 4 | 16.2 / 84.0 | 4 | 18.0 / 86.1 | 4 | 30.4 / 30.4 |
| DNS | 3.71 | 12.4 / 81.8 | 3.30 | 16.1 / 85.7 | 3.34 | 28.2 / 28.2 |
| PMPD-Static | 3.71 | 16.2 / 84.0 | 3.30 | 18.1 / 86.2 | 3.0 | 29.7 / 29.7 |
| PMPD-Learned | 3.09 | 15.5 / 83.4 | 3.52 | 17.9 / 86.1 | 3.34 | 29.8 / 29.8 |

Table 1 presents the algorithmic performance of PMPD in comparison with both single-precision baselines and DNS. We observe that both Static and Learned schedulers achieve significant bitwidth reduction while maintaining competitive performance.

PMPD vs. Baselines: Both schedulers deliver lossless performance on the CNN/DM and Dialogsum datasets with up to 33% reduction in bitwidth, highlighting their ability to effectively capture context and generate high-quality outputs.
In some cases, PMPD even outperforms the high-precision baseline by up to 2.4%.
On the IWSLT dataset,
Static experiences a moderate performance drop of up to 2.3%, while still reducing the average bitwidth by 9-33%.
Notably, our results show that using high-precision models in the decoding phase is not always necessary.
Also, Static is able to accurately identify such scenarios, achieving the maximum bitwidth reduction possible.

PMPD vs. DNS: DNS is a robust baseline that maintains low-bit precision by selectively handling outlier values. Nonetheless, our Static scheduler consistently outperforms DNS at the same bitwidth across all tasks, achieving up to 29% higher BLEU on the IWSLT dataset. Additionally, the performance gain increases with smaller models: DNS performs comparably to Static on CNN/DM for Vicuna-7B but lags behind by 23% in Rouge-L on MobileLLaMA. This indicates that PMPD is particularly well-suited for edge deployment scenarios.

**Table 2: PMPD’s MT-Bench performance with fp16 output as reference.**
| Model | Method | Bit | Rouge-L/ |
| --- | --- | --- | --- |
| BERTScore |  |  |  |
| Vicuna-7B | 2bit | 2 | 16.7 / 80.4 |
| 3bit | 3 | 41.0 / 89.9 |  |
| PMPD-Learned | 2.68 | 39.6 / 88.9 |  |
| DNS | 2.68 | 36.7 / 88.2 |  |
| Zephyr-3B | 3bit | 3 | 28.6 / 87.2 |
| 4bit | 4 | 40.8 / 89.6 |  |
| PMPD-Learned | 3.48 | 40.7 / 89.7 |  |
| DNS | 3.48 | 35.3 / 88.6 |  |
| MobileLLaMa | 3bit | 3 | 22.3 / 82.4 |
| 4bit | 4 | 28.9 / 85.8 |  |
| PMPD-Learned | 3.39 | 29.1 / 85.4 |  |
| DNS | 3.39 | 25.4 / 84.2 |  |
| Phi-1.5 | 3bit | 3 | 27.9 / 85.7 |
| 4bit | 4 | 35.4 / 87.8 |  |
| PMPD-Learned | 3.56 | 35.7 / 87.9 |  |
| DNS | 3.56 | 29.9 / 86.7 |  |

Static vs. Learned Scheduler: On average, the Static scheduler achieves a slightly higher bitwidth reduction (0.66 vs. 0.63 bits) and frequently outperforms Learned, showing more stable performance with less variance. This is likely because Static leverages task-specific data, while Learned may face challenges from data distribution shifts between its training set and each test set.

Open-Ended Tasks: We used MT-Bench as a challenging benchmark covering a wide range of open-ended tasks. In particular, we demonstrate the usefulness of Learned in this scenario where Static is not feasible due to the absence of task-specific validation set. We measured the faithfulness of the quantized models to the fp16 model by evaluating the similarity of their outputs. As shown in Table 2, Learned achieves an average bitwidth reduction of 0.47 with no drop in BERTScore and only up to 1.4 points drop in Rouge-L. Additionally, it consistently outperforms DNS with 13.9-18.4% higher Rouge-L, demonstrating that PMPD better preserves model fidelity in complex tasks. Moreover, Learned’s relative performance improves with smaller models, making it particularly promising for reducing quantization error in mobile-friendly models.

### 5.2 Hardware Performance

Figure: Figure 5: End-to-end NPU throughput. Speedup ratios are obtained in comparison with fp16 model.
Refer to caption: x5.png

**Table 3: GPU speedup ratios of PMPD for linear layers compared to the fp16 model. Two representative layers per model are shown: attention projection (Attn Proj.) and MLP projection (MLP Proj.). PMPD speedup was calculated as a weighted average of kernel latencies in two precisions, weighted by decoding steps. Additional details are included in Appendix A.2.**
|  |  | Vicuna-7B | MobileLlama | Phi-1.5 | Zephyr-3B |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Attn Proj. | MLP Proj. | Attn Proj. | MLP Proj. | Attn Proj. | MLP Proj. | Attn Proj. | MLP Proj. |
| RTX 4090 | Baseline-h | 6.25$\times$ | 11.32$\times$ | 1.25$\times$ | 2.50$\times$ | 1.32$\times$ | 1.70$\times$ | 3.33$\times$ | 2.54$\times$ |
| PMPD | 6.25$\times$ | 12.20$\times$ | 1.40$\times$ | 2.81$\times$ | 1.51$\times$ | 1.84$\times$ | 2.73$\times$ | 2.82$\times$ |  |
| A40 | Baseline-h | 5.81$\times$ | 3.77$\times$ | 3.00$\times$ | 3.96$\times$ | 2.57$\times$ | 2.83$\times$ | 3.50$\times$ | 3.02$\times$ |
| PMPD | 6.58$\times$ | 4.60$\times$ | 3.37$\times$ | 4.74$\times$ | 2.77$\times$ | 3.51$\times$ | 3.81$\times$ | 4.27$\times$ |  |

NPU Evaluation.
We compare the attainable throughput of PMPD against fp16 and reduced-precision baselines on an LLM-optimized NPU (Zeng et al., ) by emulating their execution. As shown in Figure 5, PMPD achieves a significant speedup of 3.8-8.0$\times$ over fp16 across various models and datasets, with more pronounced speedup for the larger Vicuna-7B, due to its higher resilience to quantization errors, enabling more aggressive 2-bit precision lowering.
Overall, PMPD achieves more than 50 tokens/s and 15 tokens/s for mobile-grade LLMs and for the larger Vicuna-7B, respectively, in spite of the limited bandwidth (32 GB/s for NPU vs. 1008 GB/s for RTX4090 GPU), showcasing its effectiveness and suitability for LLM deployment on mobile devices.

GPU Evaluation. Focusing on PMPD’s GPU deployment, Table 3 compares the latency of fp16 and quantized linear layers, reporting CUDA kernel runtimes with batch size of 1. PMPD achieves speedups of 1.40-6.58$\times$ for smaller attention projection layers and 1.84-12.20$\times$ for larger feedforward projection layers. We hypothesize that the lower speedup on smaller layers is primarily due to the memory-bound nature of matrix-vector multiplication. Despite the accelerated GPU execution, we observed that CPU-side processes for GPU kernel launching become a bottleneck, resulting in suboptimal GPU utilization. Potential solutions include leveraging CUDA Graph (Gray, ) to launch multiple GPU operations simultaneously, which we leave for future research.

### 5.3 Ablation Studies

Figure: Figure 6: Performance of the phase-aware precision allocation, 3/2 bits for Vicuna-7B, 4/3 bits for the other models.
Refer to caption: x8.png

Figure: Figure 7: Normalized scores of different Learned schedulers variants across CNN/DM, IWSLT and MT-Bench on MobileLLaMA.
Refer to caption: x12.png

Phase-Aware Precision Allocation.
Figure 6 analyzes the impact of phase-aware precision allocation on performance. Rouge-L on CNN/DM improves by an average of 4.3 points, BERTScore on Dialogsum increases by 3.6, and BLEU improves by 9.45. The 2-bit variant of Vicuna-7B benefits significantly from using a higher-precision model during the prefill phase, as it struggles with instruction following and often outputs incomplete tokens or repeated prompts. Using a 3-bit variant during prefilling improves language comprehension and generation. Latency overhead from using higher precision is minimal, ranging from 0.07% to 1.05%, confirming that the prefill phase is compute-bound. This supports the efficiency of phase-aware precision allocation for system optimization.

Learned Scheduler Design.
We study different choices of inputs for Learned, including activations(^3^33Specifically, we use the input activations to the final projection layer of the attention block.) and KV caches from the first, middle, and last Transformer attention blocks. We test two random schedulers: i) using a uniform distribution (random uniform) and ii) matching the label distribution of the training dataset (random prior). As shown in Figure 7, using the KV cache from the last attention block achieves the highest normalized score with significant bitwidth reduction, highlighting its effectiveness in capturing contextual information for precise switch point prediction. This scheduler also outperforms the two random schedulers, indicating that it is effectively learning patterns in the KV cache.

## 6 Conclusion

Taking advantage of the insight that later stages of the LLM decoding process demonstrate enhanced resilience to approximations, we introduced PMPD, a novel technique that progressively reduces the precision of the model’s weights during inference. The proposed approach considers the distinct LLM inference phases (prefill vs decoding) and the depth of the decoding sequence to schedule precision changes in either a prompt-agnostic static or a task-agnostic learnable manner. We show that PMPD can generate high-quality responses to prompts, while significantly reducing the adopted average bitwidth, leading to remarkable gains in inference speed across GPU and NPU platforms.

## Appendix A Appendix

### A.1 Learned Scheduler Details

#### A.1.1 Model Architecture

To avoid runtime overhead, we employ a lightweight attention module for Learned scheduler. Given an input Keys cache $K\in\mathbb{R}^{T\times D_{k}}$ and Values cache $V\in\mathbb{R}^{T\times D_{v}}$, the output of Learned scheduler is computed as:

$$ $\displaystyle O=Softmax(\frac{q_{\theta}K^{T}}{\sqrt{D_{k}}})V$ (4) $\displaystyle Logits=Softmax(MLP(O))$ (5) $$

where $MLP$ is a feedforward neural network with one hidden layer and a ReLU as its activation, and $q_{\theta}\in\mathbb{R}^{D_{v}}$ is a learned query vector. In other words, the trainable parameters of the network include the query vector and the parameters of the MLP.

#### A.1.2 Training Approach

To prepare the training dataset, we use the C4 dataset as the seed dataset and randomly truncate each sample to create a completion task. For each sample, we generate $N$ sequences, each corresponding to a high-precision step from $\{0,\frac{CL}{N-1},\frac{2CL}{N-1},\ldots,\frac{(N-2)CL}{N-1},CL\}$. The ground-truth label is set to the lowest high-precision step such that the Rouge-L score of the sequence matches or exceeds that of the fp16 model output.

During training, we use a cross-entropy loss function as the objective.

### A.2 GPU Matrix Multiplication Latency Evaluation

**Table 4: GPU matrix kernel latency ($\mu$s) on RTX 4090.**
| Model | q_proj | k_proj | v_proj | o_proj | up_proj | down_proj | gate_proj |
| --- | --- | --- | --- | --- | --- | --- | --- |
| MobileLLaMA-1.4B-Chat |  |  |  |  |  |  |  |
| fp16 | 5 | 5 | 5 | 5 | 13.3 | 15 | 13 |
| w2 | 3 | 3 | 3 | 3 | 4 | 5 | 4 |
| w3 | 3 | 3 | 3 | 3 | 4.3 | 5 | 4.57 |
| w4 | 4 | 4 | 4 | 4 | 5.1 | 6 | 5.97 |
| phi-1_5 |  |  |  |  |  |  |  |
| fp16 | 5 | 5 | 5 | 5 | 10.6 | 11.8 | - |
| w2 | 2.96 | 2.82 | 2.93 | 2.82 | 4.96 | 4.94 | - |
| w3 | 3 | 2.81 | 2.89 | 2.91 | 5.36 | 5.92 | - |
| w4 | 3.98 | 3.79 | 3.86 | 3.9 | 6.92 | 6.93 | - |
| stablelm-zephyr-3b |  |  |  |  |  |  |  |
| fp16 | 10 | 10 | 10 | 10 | 11 | 18 | 11 |
| w2 | 2 | 2 | 2 | 4 | 6 | 4 | 6 |
| w3 | 4 | 4 | 4 | 4 | 6 | 6 | 6 |
| w4 | 5 | 3 | 5 | 3 | 8 | 7.1 | 6 |
| vicuna-7b-v1.5 |  |  |  |  |  |  |  |
| fp16 | 25 | 25 | 25 | 25 | 97.1 | 101.93 | 97 |
| w2 | 4 | 4 | 3.8 | 4 | 7 | 8 | 7 |
| w3 | 4 | 4 | 4 | 4 | 8.1 | 9 | 8 |
| w4 | 5 | 5 | 5 | 5 | 12 | 12 | 12 |

**Table 5: GPU matrix kernel latency ($\mu$s) on A40.**
| Model | q_proj | k_proj | v_proj | o_proj | up_proj | down_proj | gate_proj |
| --- | --- | --- | --- | --- | --- | --- | --- |
| MobileLLaMA-1.4B-Chat |  |  |  |  |  |  |  |
| fp16 | 18 | 18 | 18 | 18 | 42.4 | 47.5 | 42 |
| w2 | 4.9 | 4.9 | 5 | 4.7 | 7.4 | 8 | 7.5 |
| w3 | 5 | 5 | 5 | 5 | 8 | 9 | 8 |
| w4 | 6 | 6 | 6 | 6 | 11 | 12 | 10.2 |
| phi-1_5 |  |  |  |  |  |  |  |
| fp16 | 18 | 18 | 18 | 18 | 62.2 | 60.3 | - |
| w2 | 5.9 | 5.7 | 5.9 | 5.9 | 10 | 11 | - |
| w3 | 6 | 6 | 6 | 6 | 12 | 13 | - |
| w4 | 7 | 7 | 7 | 7 | 20.3 | 21.3 | - |
| stablelm-zephyr-3b |  |  |  |  |  |  |  |
| fp16 | 28 | 28 | 28 | 28 | 66.1 | 70.1 | 66.4 |
| w2 | 6 | 6 | 6 | 6 | 11 | 10 | 11 |
| w3 | 7 | 7 | 7 | 7 | 13 | 12.9 | 13 |
| w4 | 8 | 8 | 8 | 8 | 22.1 | 23.2 | 22 |
| vicuna-7b-v1.5 |  |  |  |  |  |  |  |
| fp16 | 64 | 64 | 64 | 64 | 170 | 162 | 170 |
| w2 | 9 | 9 | 9 | 9 | 24.3 | 30.9 | 24.2 |
| w3 | 11.2 | 11 | 11 | 11 | 33.9 | 43 | 33.4 |
| w4 | 21.2 | 21.5 | 21.3 | 21.2 | 46.6 | 58.9 | 45.9 |

Table 4 and Table 5 report the GPU kernel latency using the $self\ CUDA\ time$ metric on RTX 4090 and A40 GPUs. For PMPD, the latency is calculated as a weighted average of the fixed-bitwidth latencies, where the weights correspond to the average number of decoding steps performed at each bitwidth across datasets. To select between the Static and Learned schedulers, we choose the one with the highest performance. For MobileLLaMA and Vicuna-7B, bitwidths are averaged over the CNN/DM, Dialogsum, and IWSLT datasets. For Phi-1.5, bitwidths are averaged over the CNN/DM and Dialogsum datasets, while for Zephyr-3B, the bitwidth is from the IWSLT dataset.

### A.3 NPU Simulation Details

We developed an analytical performance model to emulate the deployment of PMPD on FlightLLM (Zeng et al., ), an NPU architecture that has been optimized for LLM inference. FlightLLM consists of: i) a unified Matrix Processing Unit that can perform multiple types of multiplications between matrices and/or vectors through a hierarchical structure of multiply-accumulate (MAC) units, ii) a Special Function Unit where LLM-specific operations such as softmax, layer normalisation etc are mapped and iii) an optimized memory hierarchy for the computational pattern of LLMs, while keeping all activation on-chip during decoding. The underlying design space of FlightLLM’s NPU architecture is traversed through a design space exploration method that takes into consideration the need to support different precisions of LLM weights.
For our experiments, we instantiate two NPU configurations consisting of: 4K and 16K MAC units with 1 GHz clock frequency (i.e. 8 and 16 teraops/sec (TOPS) peak throughput, respectively) for the deployment of smaller- (MobileLLaMa-1.4B, Phi-1.5) and larger-scale LLMs (Vicuna-7B), respectively, and with 32 GB/s off-chip memory bandwidth.

### A.4 Alternative PTQ Method

Our approach offers a plug-and-play framework compatible with any PTQ method. To showcase its generalizability, we evaluate the performance of both Static and Learned schedulers using GPTQ, a uniform quantization technique (Frantar et al., ). As shown in Table 6, both schedulers deliver performance that closely matches the high-precision baseline.

**Table 6: Performance of GPTQ-quantized MobileLLaMa on Dialogsum.**
| Method | Bit | Rouge-L | BertScore |
| --- | --- | --- | --- |
| 3bit | 3 | 9.71 | 82.7 |
| 4bit | 4 | 16.0 | 84.2 |
| PMPD-Static | 3.63 | 16.0 | 84.6 |
| PMPD-Learned | 3.02 | 14.2 | 84.2 |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: See weaknesses
soundness: 3
strengths: 1. Compared to previous methods of mixed-precision quantization within models, this paper innovatively considers introducing mixed-precision quantization at different stages of LLM inference.

2. This paper not only validates the algorithmic performance of the proposed method but also measures the hardware performance like speedup ratio and throughput on both GPUs and NPUs.
summary: The authors point out that different stages of LLM inference require different quantization bit-widths. In oracle experiments, they discovered that increasing the model weight bit-width during the prefill phase can enhance model performance. They also find that using a higher bit-width in the early stages of decoding helps maintain model performance. Based on these findings, the authors search for different quantization bit-widths for the prefill and decode stages. Additionally, they use an optimized scheduler to gradually reduce the model bit-width during decode stage. Using these two schemes, the quantized model can ensure output quality on the challenging MT-Bench task. Meanwhile, the quantized model achieves a throughput improvement of 3.8-8.0× on NPU and a linear layer acceleration of 1.4-12.2× on GPU.
weaknesses: 1. In Section 3.1 of this paper, the experiment results are insufficient to demonstrate the validity of Motivation One. 

    - First, the authors only increase the bit-width during the prefill stage and found that model performance was better than that when using uniform bit-width, but do not increase the bit-width during the decode stage to evaluate model performance. Therefore, the results do not demonstrate that the prefill and decode stages have different error resilience. 

    - Secondly, this section only presents the model's outputs for three specific inputs, lacking quantitative results.

    - It is recommended that the authors supplement the results of high-precision decoding with low-precision prefill and provide performance of the model on MT-Bench to support Motivation One.

2. In Section 4.1 of this paper, the authors search for the optimal bit-widths for the prefill and decode stages assuming that the decode stage is conducted with a uniform bit-width. However, during actual inference, the model bit-width in the decode stage gradually decreases. Therefore, it cannot be guaranteed that the previously found results remain optimal. Then what is the significance of this search step? Is it possible to directly use the bit-width of the prefill stage as the starting point for the bit-width decrease in the decode stage?

3. In Section 4.3 of this paper, the authors design the Task-Agnostic Learned Scheduler to predict the steps at which the bit-width should decrease during the decode stage. Theoretically, the optimal precision switch points may depend on both the model's inputs and the outputs. However, this scheduler only uses the token features generated during the prefill stage as input. Is it reasonable to overlook the information from the model's generated content?

4. The experimental results in this paper are not comprehensive enough. First, the maximum parameter size of the model used in the experiments is 7B, and larger models are not evaluated. Secondly, the paper only provides GPU speedup ratios for the linear layers and does not present end-to-end speedup data.

5. This paper has misunderstandings regarding some concepts. For example, the maximum context length of LLMs (CL) refers to the sum of the model's maximum input and output lengths, but in Section 4.3, the paper incorrectly asserts that CL is the model's maximum output length.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. How do you implement the precision switch operation during the decoding stage? And what is the actual cost of such switching?

2. How's the result on Llama3? Or other GQA models?

3. How's your method compared with static quantization methods like GPTQ and AWQ both for accuracy and efficiency?
soundness: 2
strengths: - The author discusses many design choices, with their pros and cons.
- The discussion on the tolerance of quantization in different depths of the decoding stage is informative
summary: This work tackles the difficulty of deploying large language models (LLMs) on resource-constrained devices by introducing a “phase-aware” quantization method that adapts precision to different inference stages, unlike traditional uniform quantization approaches. The authors develop Progressive Mixed-Precision Decoding (PMPD), a technique that lowers precision progressively as the generated sequence deepens, using schedulers to adapt precision based on tasks or prompts. Experimental results demonstrate that PMPD provides substantial speedups on Nvidia GPUs and optimized NPUs, while preserving output quality and surpassing the performance of both fp16 and uniform quantization models.
weaknesses: - The paper does not discuss much about how they do the precision switching in their implementation, which could be a potential risk of memory usage (loading the same model with different precision) or additional latency (loading the model weight on the fly).
- The accuracy results are all based on MHA models, lacking discussion on other types of models like GQA models.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Q : What quantization method is used for Baseline-l and Baseline-h?

Q: Why GPTQ are not presented as baselines in Table 1?

Q: How would you explain that PMPD-Static has better performance in comparison to PMPD-Learned on Table 1?

Q: Please explain what precision range was used for training PMPD shown on Table 1, e.g. (2bits, 3bits, 4bits) for all experiments or you use (2bits 3bits) for one one set of experiments and (3bits, 4bits) for another set of experiments?

Q: For PMPD method you will need to keep pre-computed e.g. 2bits 3bits weights and then switch between different precision depending on input token, please explain how do you avoid overhead of keeping weights with different set of precision in memory in comparison to a baseline method which keeps weights with one set of precision e.g. 3bits only?

Q: On Table 1, experiment on data Dialogsum, how do you explain that Baseline-l with 2bits has accuracy 10.2 / 75.5 but PMPD-Static with the same amount of bits = 2.0bits has better accuracy 25.0 / 88.2? Please explain what set of bits PMPD-Static uses, e.g. if it was trained with (2bits 3bits) then it means that PMPD-Static used only 2bits during inference, so that final number of bits is 2.0 (reported in Table 1).
soundness: 3
strengths: PMPD vs. Baselines: Both schedulers deliver lossless performance on the CNN/DM and Dialogsum datasets with up to 33% reduction in bitwidth, highlighting their ability to effectively capture context and generate high-quality outputs.

Novelty in applying different precision for model weights depending on input token during decode mode.

Conducted thorough experiments on LLM edge-deployable models.

Produced End-to-end NPU throughput and Speedup comparison vs fp16 model. Shows that PMPD achieves a significant speedup of 3.8-8.0× over fp16 across various models and datasets.

Good ablation studies showing that  indicating that PMPD-Learned is effectively learning patterns in the KV cache
summary: Authors propose a novel phase-aware method that selectively allocates precision during different phases of LLM inference: use higher precision during prefill stage and then reduce precision for efficient memory bandwidth utilization during decoding.
They introduce Progressive Mixed-Precision Decoding (PMPD) which controls gradual lowering of weights precision in the generated sequence with task-adaptive or prompt-adaptive manner. It is weights only post training quantization(PTQ) which adaptively controls weights precision depending on decode stage (prefill, decode) and decoded token. They use  Any-Precision LLM (Park et al., 2024), as PTQ which keeps multiple weight precisions with no memory footprint overhead.

Authors show that PMPD achieves 1.4−12.2x speedup in LLM linear layers over fp16 models (on Nvidia GPUs).
Also throughput gain of 3.8−8.0x over fp16 models and up to 1.54× over uniform quantization (on NPU) approaches while preserving the output quality.
weaknesses: Q: in Table 1, PMPD-Static has better accuracy score in 11 cases; PMPD-Learned has better accuracy score in 3 cases; PMPD-Static uses less bits 5 cases; PMPD-Learned uses less bits 4 cases. It looks like PMPD-Learned under-performing on both accuracy and bits usage in comparison to PMPD-Static. Please analyze why PMPD-Learned underperforms compared to PMPD-Static. I see that PMPD-Learned is preferable in case when validation data are missing, because PMPD-Static can not be trained on data which are missing validation data sets.

Q: Authors report 1.4−12.2x speedup in LLM linear layers over fp16 models for PMPD methods. Please also add speed up in comparison to 3bit and 4bit baseline models (you show it in Figure 5 and Table 3, but do not report in the abstract of the paper).

Q: One of the goal of this paper is to show that proposed quantization approach has no accuracy drop in comparison to the float baseline model. Float baseline is missing in Table 1, so it is hard to make a conclusion that proposed methods do not drop accuracy in comparison to float baseline. Please include the float baseline results in Table 1 or explain why you chose not to include them (for example in Figure 5 you show fp16 latency of evaluated models). Same comment for Table 2.

Q: It would be great to add in Table 1, other state of the art post training quantization methods, e.g. :
* Fast and Efficient 2-bit LLM Inference on GPU: 2/4/16-bit in a Weight Matrix with Asynchronous Dequantization
* QuIP: 2-Bit Quantization of Large Language Models With Guarantees or (QuIP#: Even Better LLM Quantization with
Hadamard Incoherence and Lattice Codebooks)
* GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

Please explain why above methods were not included as baselines, and how your method compares conceptually to these approaches if direct comparison is not feasible.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: 1) IMO the most interesting section of the paper is the scheduler design in Section 4.3. As such a lot of core ideas/implementation details of the scheduler are relegated to the Appendix. It maybe better to add scheduler model/training design to the main paper, while reducing redundant text in the introduction, methodology and the motivation section. 
2) When adding the constraint in line 325-327, it's not entirely clear how is N chosen, is this a hyper-parameter, what is recommended value? Does it depend on task/model that were studied? A detail ablation can be performed in the experiment section, but a brief summary for practioners can be provided here.
3) Table 1 caption says the phi-1.5 models are 2/4 bit, but the table itself shows they are 2/3 bit?
soundness: 3
strengths: It is very critical to address the memory-boundedness of small LLMs operating on mobile phones. Quantization is the most prevalent method for model compression but suffers from quality-latency tradeoff. The paper proposes "adaptive" precision in the decoding stage, by training a "precision scheduler", which is very interesting. It exploits the fact that later half of the tokens in the generation process can afford higher quantization error in the weights. Evaluation shows that the proposed methods achieves better (or at-least maintains) better quality than high bit-depth quantization methods, while increasing the throughput which is critical for mobile performance of small LLMs. The motivation is clear and backed with relevant prior work.
summary: The paper proposes phase-aware precision/quantization recipe which allocates higher bit depth for the prefill phase and lower/adaptive bit depth for the decoding phase in LLMs. They motivation is based off of previous work & representative examples shown in the paper that the prefill phase (language understanding phase) is likely to benefit from lower quantization errors than the decoding phase.
Additionally, authors propose a well executed "adaptive" precision strategy in the decoding phase, where the precision is reduced after every few steps of the generation. 
The experiments on three small-scale mobile LLMs are show the effectiveness of the approach in terms of quality and latency/throughput of the kernels as compared to the other methods which use fixed bit-depth through out the generation.
weaknesses: 1) There is a lot of repetition of concepts in the first few pages of the paper. The same idea is explained several times in the first three sections. It maybe better to show more representative examples similar to Figure 2 -- perhaps some high level stats/evals on how different tasks/models perform when changing the precision in the decode/prefill phase.
2) The work misses several critical LLM benchmarks around Math, reasoning and especially long context. It would be interesting to see the effect of the proposed approach on a wider range of LLM evals. 
3) This experimentation section fails to provide more statistics on evaluation, for example, what is the average decode length for the three tasks for the thee models, what is the impact of changing N (the number of "switches" in precision during decoding phase). It is not entirely clear what precision the scheduler choses and for how long at each decoding step, -- for three different tasks and the three models.

Meta-review: This paper introduces a phase-aware quantization strategy for LLM inference, allocating higher precision to the prefill stage and adaptively reducing precision during decoding. Its core strengths lie in demonstrating consistent throughput gains. Initially, concerns were raised regarding writing, limited benchmarks, insufficient details, uncertainty about choosing the hyperparameter N, etc. However, the authors’ rebuttal effectively resolved these points, offering additional experiments, clarifications, and corrections—thereby strengthening the overall contribution.

Decision: Accept (Poster)