Title: DP-LLM: Runtime Model Adaptation with Dynamic Layer-wise Precision Assignment
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM Quantization

LLM Inference

Efficiency

ML System
Abstract: How can we effectively handle queries for on-device large language models (LLMs) with varying runtime constraints, such as latency and accuracy? Multi-scale quantization addresses this challenge by enabling memory-efficient runtime model adaptation of LLMs through the overlaying of multiple model variants quantized to different bitwidths. Meanwhile, an important question still remains open-ended: how can models be properly configured to match a target precision or latency? While mixed-precision offers a promising solution, we take this further by leveraging the key observation that the sensitivity of each layer dynamically changes across decoding steps. Building on this insight, we introduce DP-LLM, a novel mechanism that dynamically assigns precision to each layer based on input values. Experimental results across multiple models and benchmarks demonstrate that DP-LLM achieves a superior performance-latency trade-off, outperforming prior approaches.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes DP-LLM, a dynamic precision assignment framework for on-device LLM inference. Traditional multi-scale quantization techniques statically assign bitwidths across all or per-layer configurations. DP-LLM improves upon this by dynamically adjusting layer-wise precision at each decoding iteration based on runtime input sensitivity.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n\n1. The paper identifies a previously overlooked behavior: the dynamic change in sensitivity of layers during token decoding.\n\n2. DP-LLM Integrates dynamic bitwidth switching in a practical and low-overhead manner.\n\n3. Experiment results shows significant improvements in perplexity and task accuracy over baselines, and ablation studies on latency and approximation show that design choices are well-justified.\n\nCons:\n\n1. Lack of experiment on larger LLMs, the scalability of LLM quantization methods. Previous works like  AWQ, SqueezeLLM scale to 70B models. Given the complex architecture of DP-LLM, the complexity and cost in finetuning and inference in larger PLMs should be demonstrated.\n\n2. Lack of low bit evaluation. The performance degradation becomes more obvious and a greater challenge in low-precision (as demonstrate in QuiP for 2-bit). There are also works like [2] show that quantize the whole layer into 2 bit will have a huge performance degradation. How DP-LLM performs in such scenario? Will the performance of DP-LLM highly depend on the Any-precision LLM method?\n\n3. Lack of comparison on integer-bit. As a mixed-precision solution, it's also important to show the performance compared to existing works on integer-bit quantization task. [3] shows that SliM-LLM (which is also a mixed-precision method) achieve 6.4 ppl on wiki2 and 9.5 ppl on c4 on 4bit task for LLaMa3-8B. The comparison of such methods should be discussed.\n\n\n\n\n[1] Chee, Jerry, et al. \"Quip: 2-bit quantization of large language models with guarantees.\" Advances in Neural Information Processing Systems 36 (2023): 4396-4429.\n\n[2] Chen, Zihan, et al. \"Channel-wise mixed-precision quantization for large language models.\" arXiv preprint arXiv:2410.13056 (2024).\n\n[3] Huang, Wei, et al. \"An empirical study of llama3 quantization: From llms to mllms.\" Visual Intelligence 2.1 (2024): 36.\n\nQUESTIONS:\n1. Layer-wise maximum precision is determined in Appendix A. Does the B[i] equal to b where $c_{i,b}=1$? The constraint in (6) guarantees that when we use b where $c_{i,b}=1$, the memory usage for quantized model still below the memory budget. Why we need to further determine layer precision, but can't direct quantize each layer to its maximum precision? Is there some misunderstanding? \n\n2. How to guarantee the fair comparison with baselines? Though G requires mild GPU memory overhead, we can also achieve performance improvement by protecting a very small fraction of weights in FP16. How does DP-LLM perform when have exactly storage cost with baselines.\n\n3. The latency evaluation is not that straightforward. Methods like GPTQ, SqueezeLLM usually measure the seconds for generation of tokens and compared with FP16 models to show the efficiency. How does DP-LLM perform?\n\nLIMITATIONS:\nSee Cons and Questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a lightweight mechanism that extends AnyPrecisionLLM to enable dynamic layerwise precision selection based on their saliency metric. AnyPrecisionLLM (prior work) solves the problem of storing multi-precision weights while DP-LLM (introduced here) proposes a solution for dynamic, fine-grained bit width allocation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n- While mixed-precision quantization is increasingly being studied, the problem of dynamic, fine-grained bit width allocation does not seem well-explored. The insight that layer importance may change over decoding steps is interesting.\n- I thought the idea of approximating $||\\Delta W x||_2$ by using the Johnson-Lindenstrauss lemma was quite nice. \n\n\nWeaknesses:\n\n- While the insight that layer importance may change over decoding steps is interesting, it is only briefly discussed and loosely analyzed. It would be great to have a deeper analysis and discussion.\n- As a more general concern, the paper is not particularly well positioned in the wider context and history of quantization, and model optimization in general. For example, fine-grained saliency estimation has been studied for quite a while, but this is not addressed in the paper. \n- The experiment design does not take into account the current state-of-the-art in LLM quantization. \n \nTo expand on some of these issues, saliency metrics are well studied, and have been since the 90's including in \"Optimal Brain Surgeon\" in 1993, \"Deep Compression\" in 2016, and LLM “super weights” in 2024. Additionally, the paper only benchmarks against 2 static mixed-precision techniques and does not consider cheaper techniques that are known to be sufficient saliency estimators (e.g., weight and/or activation magnitude). As such, it is unclear if the paper’s proposal (which introduces compute and memory overhead) can be undercut by a simpler metric.\n \nWhile it is clear that query budgets can change, there is not enough discussion or analysis on how much the budget can vary. This is foundational of the utility of the work, so this should be part of the motivation. \n\nFurthermore, techniques such as Once-for-All in 2019 and EagleEye in 2020 propose methods for encoding several mixed-precision models and architectures, similar to AnyPrecisionLLM, but more fine-grained. If the goal is to adjust precision to query budgets, this seems like a natural benchmark as well, since these techniques can instead adjust neural architecture based on inference cost constraints.\n \nFinally, when benchmarking against static precision allocation, one should benchmark against the state-of-the-art techniques for PTQ (e.g., SpinQuant, OPTQ, GPFQ, OSTQuant, etc.) and QAT (ParetoQ). Although these papers study uniform precision, a primary contribution of this paper is saliency estimation, which should be compatible with static bit width allocation as well. Given that, for example, SpinQuant + OPTQ can yield high quality sub-4-bit quantized models in a PTQ setting and that ParetoQ models are trained to use 2-bit or even 1.58-bit weights, it seems reasonable that one can use these techniques to statically optimize a model to the tightest runtime constraints and circumvent the need for dynamic bit width allocation.\n\nQUESTIONS:\n- It is unclear what precision activations are assumed to be at. Are those dynamically selected as well, or fixed at high precision?\n- In lines 231-237, it is stated that the choice of $k$ in the JL projection can limit the \"relative error estimation difference within 8% with 95% confidence\". Is this an empirical or theoretical statement? \n- In section 6.3, I didn't quite understand why the geometric mean was being reported. Could you please explain? My concern is that the geometric mean can be quite small relative to the arithmetic mean and can hide outliers.\n\nLIMITATIONS:\nPlease see questions and weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identifies a new perspective in LLM quantization: while previous works consider layer-wise sensitivity as static, the authors observe that the sensitivity of each layer varies across decoding iterations. Motivated by this, they propose DP-LLM, a runtime quantization strategy that dynamically selects bitwidths per layer at each decoding step, taking an instance-specific view of quantization. The method includes a principled mechanism to allocate bit budgets per layer under an average precision constraint, along with lightweight error estimators to ensure quality guarantees. Extensive experiments demonstrate that DP-LLM achieves improved performance compared to static mixed-precision methods under the same average bitwidth, validating the benefit of dynamic, layer-wise quantization.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper is generally well-organized and well-written.\n\n2. The core insight—that layer-wise quantization sensitivity varies across decoding iterations—is original and not addressed in prior work.\n\n3. The newly proposed dynamic per-layer precision allocation effectively improves model performance while satisfying the same average bitwidth constraint.\n\n4. The authors conduct extensive experiments on two recent large models (LLaMA-3-8B and Phi-3-Medium) across multiple datasets and downstream tasks.\n\nWeakness:\n\n1. While the idea of per-iteration, per-layer precision assignment is theoretically appealing, the current design of DP-LLM makes it difficult to support standard batched inference across multiple queries. Although autoregressive generation is inherently token-by-token within each query, it is common in practice to process multiple queries in parallel by batching them at each decoding step. However, DP-LLM selects bitwidths dynamically for each layer and each decoding iteration based on the input vector of that specific query. This leads to query-dependent computation paths and weight selection, which prevents efficient batching due to divergent bitwidth configurations across queries. In contrast, static mixed-precision approaches assign fixed bitwidths per layer and are thus fully compatible with batch-level parallelism, making them more practical for high-throughput deployment.\n\n2. Although the proposed dynamic precision selection aims to improve the accuracy-latency trade-off, the paper does not provide a thorough comparison of latency overheads against strong static baselines such as LLM-MQ and HAWQ-V2. Given that DP-LLM involves runtime computations for precision selection—either via relative error estimation or linear predictors—it is important to quantify how much latency these mechanisms add relative to static mixed-precision methods, which do not incur such runtime costs. A more detailed latency analysis would strengthen the empirical justification of DP-LLM’s design and better inform practical deployment decisions.\n\n3. The formulation of the integer programming problem in Appendix A—particularly Equation (6)—lacks important clarification regarding Q. Especially, the relationship between Q and b is unclear. If Q is assumed to be fixed in advance, it remains unclear how the decision variables c_{i,b} can be optimized. \n\n4. In Algorithm 1, the computation of the output y is expressed as a weighted sum over bitwidth candidates, but the current form effectively reduces toy = 1 * W_{i,b}, since only one c_{i,b}  is active (equal to 1). This appears inconsistent with the linear interpolation form described in the main text (e.g., Line 194), where y is computed as a convex combination of W_{i,b}x and W_{i,b+1} x,. It is unclear whether this discrepancy is an intentional simplification, a design decision only for the integer programming phase, or simply a typo. Clarifying this would help improve the rigor and consistency of the method presentation.\n\n5. The paper includes a per-query QoS analysis showing that DP-LLM introduces small deviations between actual and target bitwidths across queries (as in Table 6). However, it remains unclear whether such deviations are unique to DP-LLM or if similar fluctuations would occur in static methods like LLM-MQ or HAWQ-V2.\n\nQUESTIONS:\n1.\tDP-LLM performs dynamic, per-layer, per-token bitwidth selection during autoregressive decoding. This implies that, for any given forward pass, a layer may require either W_l or W_h.\nCould the authors clarify how weights are stored and accessed at runtime? Specifically:Is DP-LLM implemented using a Matryoshka quantization approach, where only the highest-precision weights (e.g., 6-bit) are loaded, and lower-precision weights are derived on-the-fly via truncation?\n2.\tOther concerns：\nFor additional questions and clarifications regarding runtime assumptions, implementation details, and ablation completeness, please refer to the identified weaknesses above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DP-LLM, a dynamic layer-wise precision assignment method for quantized LLMs. The key idea is to dynamically adjust each layer’s quantization bitwidth during decoding iterations based on estimated relative error, instead of relying on static precision assignment. The results show improvements in perplexity and downstream tasks over different baselines, with minimal latency overhead.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\n1.The idea of adjusting bitwidths dynamically for each token is quite innovative and makes sense for balancing performance and efficiency.\n2.The paper is well structured, with a clear explanation of the method and easy to understand.\n3.The writing is generally clear, and the explanations are easy to follow。\n4.The authors provide code and helps others reproduce and build on this work.\n\n**Weaknesses**\n\n1.The analysis in section 2.4 lacks detailed quantification. Figure 3 shows dynamic sensitivity, but it doesn’t explain which types of tokens are sensitive to which layers, or which layers are most affected. It’s also unclear which dataset was used for this figure and whether the results generalize to other tasks and models.\n2.The experiments are mainly done in a PyTorch setup, but real-world systems often use frameworks like vLLM. It’s unclear how the method would perform there, especially for latency . Also, the latency ablation study does not provide detailed performance changes for different setups.\n3.The method seems quite complex, but the performance improvements shown in the tables are modest. Also, there’s no information about variability or statistical significance, so it’s hard to judge how reliable the gains are. This raises questions about whether the method is practical to deploy.\n\nQUESTIONS:\n1.Does this method work with batch sizes larger than one? If so, how do you handle situations where tokens in the same batch might need different bitwidths?\n2.While the trade-off between performance and memory is intuitive, how is the trade-off between performance and latency manifested in practice?\n3.The framework has multiple components—thresholds, error estimators, asynchronous steps, etc. How much does each part contribute to the final results? Are all of them really needed, or could the system be simplified?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}