ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization
Authors: Zechun Liu
Changsheng Zhao
Hanxian Huang
Sijia Chen
Jing Zhang
Jiawei Zhao
Scott Roy
Lisa Jin
Yunyang Xiong
Yangyang Shi
Lin Xiao
Yuandong Tian
Bilge Soran
Raghuraman Krishnamoorthi
Tijmen Blankevoort
Vikas Chandra
Venue: NeurIPS 2025 poster ()
Keywords: compression
Primary area: deep_learning

Abstract: The optimal bit-width for achieving the best trade-off between quantized model size and accuracy has been a subject of ongoing debate. While some advocate for 4-bit quantization, others propose that 1.58-bit offers superior results. However, the lack of a cohesive framework for different bits has left such conclusions relatively tenuous. We present ParetoQ, the first unified framework that facilitates rigorous comparisons across 1-bit, 1.58-bit, 2-bit, 3-bit, and 4-bit quantization settings. Our findings reveal a notable learning transition between 2 and 3 bits: For 3-bits and above, the fine-tuned models stay close to their original pre-trained distributions, whereas for learning 2-bit networks or below, the representations change drastically. By optimizing training schemes and refining quantization functions, ParetoQ surpasses all previous methods tailored to specific bit widths. Remarkably, our ParetoQ ternary 600M-parameter model even outperforms the previous SoTA ternary 3B-parameter model in accuracy, using only one-fifth of the parameters. Extensive experimentation shows that ternary, 2-bit, and 3-bit quantization maintains comparable performance in the size-accuracy trade-off and generally exceeds 4-bit and binary quantization. Considering hardware constraints, 2-bit quantization offers promising potential for memory reduction and speedup.

ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization
Zechun Liu'
Changsheng Zhao'
Hanxian Huang',
Sijia Chen' Yangyang Shi!
Jing Zhang Lin Xiao Tijmen Blankevoort
Jiawei Zhao'
Scott Roy!
Lisa Jin
Yunyang Xiong'
Yuandong Tian
Bilge Soran'
Raghuraman Krishnamoorthi
Vikas Chandra
Meta Al
Abstract
The optimal bit-width for achieving the best trade-off between quantized model size and accuracy has been a subject of ongoing debate. While some advocate for 4-bit quantization, others propose that 1.58-bit offers superior results. However, the lack a cohesive framework for different bits has left such conclusions relatively tenuous. We present ParetoQ, the first unified framework that facilitates rigorous comparisons across 1-bit, 1.58-bit, 2-bit, 3-bit, and 4-bit quantization settings. Our findings reveal notable learning transition between and 3 bits: For 3-bits and above, the fine-tuned models stay close to their original pre-trained distributions, whereas for learning 2-bit networks or below, the representations change drastically. By optimizing training schemes and refining quantization functions, ParetoO toQ surpasses all previous methods tailored specific bit widths. Remarkably, our ParetoQ ternary 600M-parameter model even outperforms the previous SoTA ternary 3B-parameter model in accuracy, using only one-fifth of the parameters. Extensive experimentation shows that ternary, 2-bit, and 3-bit quantization maintains comparable performance in the size-accuracy trade-off and generally exceeds 4-bit and binary quantization. Considering hardware constraints, 2-bit quantization offers promising potential for memory reduction and speedup.
Introduction
As deep learning continues to scale toward larger models and datasets, significant attention has been devoted studying the scaling laws that trade-off between model and dataset size to optimize performance and computational efficiency (Hoffmann et al. 2022 Kumar et al.) 2024 Dettmers and Zettlemoyer 2023). In the meantime, the field is shifting toward lower-precision computation, particularly in large language models, driven by the substantial benefits of memory savings and computational efficiency [Liu et al.) 2023a Ma et al., 2024). This shift necessitates a rethinking of scaling laws to account for the effects of quantization resulting quantized model performance.
When allowing for lower-bit quantization, can freely trade off the bit-width and the number of parameters. Keeping the amount of memory used the same, we could have an 8-bit model, or a 4-bit model twice the size. This begs the question: What the optimal trade-off between bit-width and model size? Recent papers [Dettmers and Zettlemoyer 2023 Kumar et al. 2024) on scaling laws for low-precision conclude that 4 or 6-bit quantization often resides on the Pareto frontier balance accuracy and efficiency. Other studies Ma et al.) 2024 Kaushal et al. 2024 suggest that bit-widths as low 1.58-bit per parameter hold significant promise for the optimal scaling law trade-off. These opposing conclusions highlight the challenges of studying scaling laws in the low-precision domain.
'Correspondence to: Zechun Liu aachunliu@meta.com
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

61
Accuracy
57
2-bit
4-bit
49
8-bit
16-bit
500
1000
1500
2000
2500
3000
Effective Quantized Model Size
Figure 1: Pareto curves of accuracy-siz size trade-offs.
In this paper, we demonstrate that previous conclusions on the low-bit scaling laws can be significantly sharpened by better quantization scheme design and training improvements. While previous works define the search space of the QAT scaling laws solely as a function of model parameters (A), token count (D), and quantization precision (P), we emphasize the critical role that the training scheme (Strain) and the bit-specific quantization function (F) play in the equation. We formalize the search space C(N, D, Suain F), comprising five dimensions.
To disentangle these complexities, we finst identify the optimal training strategy for plausible quanti- zation functions in each bit width, L(N, D, Suain P,F). Subsequently, with the optimal training strategy (Strai)) and the token count (D*) required for saturation, we determine the best quantiza- tion function for each bit, L(NFF P, D" Strinn) Results highlight that quantization grids and ranges are pivotal in the sub-4-bit regime, with a sharp learning behavior transition between bhit/1.58-bit/2-bit and 3-bit/4-bit.
Based on the findings, we derive ParetoQ, the first framework that unifies the training and quanti- zation scheme in sub 4-bit regime. Rather than fitting hypothetical scaling laws for quantization, ParetoQ demonstrate its robustness by yielding state-of-the-art (SOTA) models all bit widths, surpassing prior works tailored for individual bit levels.
These SOTA points in the Pareto chart ensure that our scaling law comparisons are both reliable and consistent, as they derive from homogeneous settings. Leveraging ParetoQ, we identify the optimal bit-width for minimizing loss within the effective quantized model size, C(N P F* D Stran) Our scaling laws reveal that binary quantization significantly compromises accuracy, while ternary, 2-bit and 3-bit quantization are tied in performance, often surpassing 4-bit. The tiebreaker lies in the kernel implementation, which drives real memory savings and speedups. 1it8-bit and 3-bit quantization are in general less hardware- friendly than 2-bit. We implemented an optimized 2-bit CPU kernel and our results indicate that 2- quantization achieves higher speed at the same accuracy compared to 4-bit.
The key contributions of this study are as follows:
We present comprehensive study on the intertwined effects of QAT budget allocation and specific choices of quantization functions across 8 models (125M to 3B) and 5 quantization strategies. Our study highlights the unique characteristics and challenges of binary, ternary, and 2/3/4-bit quantization, offering actionable insights and best practices for achieving optimal eecuracy-efficiemn
trade-offs. poples-to-apples comparison of quantization functions
We introduce ParetoQ, the first systematic, at extreme low-bit settings. Each point in the Pareto chart outperforms prior methods optimized for specific bit widths. Specifically, the 1.58-bit ParetoQ LLaMA-3 8B model reduces the performance gap to full precision by relatively 37.8% compared to the 1-bit Era's 's LLaMA-3 8B model [Ma et al.) 2024], while using only 30% of the training tokens.
Our research highlights the potential of 2-bit quantization as prospective alternative to the traditional 4-bit approach, offering improved accuracy-size trade-off, as underlined in Figure [ Preliminary speed benchmarks also demonstrate promising efficiency gains with 2-bit quantization. Nevertheless, widespread adoption will require community- efforts, such as INT2 support in NVIDIA tensor cores, to unlock the full benefits of 2-bit quantization.
Better QAT Scheduling Strategy for Extreme Low-Bit LLMs
In this work, we systematically investigate trade-offs involving bit precision (P), quantization func- tions (F), model size (N), training strategies (Strain) and training token (D): L(P, F D).

---

45
Accuracy
40
1.58-bit
2-bit
3-bit
35
4-bit
0.0
0.2
0.4
0.6
0.8
1.0
B5p
Berain
Figure 2: With fixed total training budget of 100B tokens (Buinn) where BFF + Beaa Buain. explore optimal allocation between full-precision pretraining (BFp) and QAT fine-tuning (Borr). "0,0" represents QAT from seratch, while "1.0" indicates free-precision pretraining followed by PTQ. Results on MobileLLLM M0LLMM1255X 125M show peak accuracy with 900% of the budget for full aull-precision pretraining and ~00% for QAT fine-tuning.
1-bit
1.58-bit
2 2-bit
65
60
Accuray
55
55
50
50-
50-
45
455
40
125M 350M 600M
1.38
18
38
3-bit
4-bit
MobileLLM
LLaMA-3
60
60
FP
Accursty
555
55
finetune 108 token
50
50
finetune 1008 token
45
scratch
scratch 10B token
125M 350M ECOM
18
1.58
18
125M 350M 600M
16
1.58
16
3B
scratch
MobileLLM
LLaMA-3
MobiletiM
LLaMA-3
scratch 1008 token
Figure 3: Analysis of training token requirements for quantization- aware fine- fine-tuning and training from scratch across 1-bit, 1.58-bit, 2-bit, 3-b and bib settings. Fine-tuning typically saturates 10B tokens for 3-bit and b-b and 30B tokens for 1-bit, 1.58-bit, and 2-bit. Fine-tuning consistently outperforms training from scratch both accuracy and token efficiency
all configurations.
Given the vast search space defined by these variables, first fix the quantization method (F) and explore the dimensions of bit precision (P), training strategies (Strain) and training tokens (D) in this section.
2.1 Training Budget Allocation
Post- cest-Triinng Quantization (PTQ) and Quantization-Awarr Training (QAT) two primary quanti- zation approaches. PTQ applics guantization after full-precision training, simplifying deployment but often leads significant performance loss at bit widths below 4 bits. In contrast, QAT incorporates quantization during training to optimize model performance for low-bit-width representations.
Here we start by answering a key question:
Given fixed training budget (in #tokens) Brain BEFT + BQAT, how should the budget be optimally allocated between full-precision training (ByPT) and quantization-aware train- ing/finetuning (BQAT) to maximize the accuracy of the quantized model?
This question is both technically intriguing and practically significant. Our approach begins with analyzing the pretraining phase to determine the optimal switching point from FPT to QAT, aiming to minimize the loss:
arg min L(BEFT, No, Bipt BOAT
Berr Baga -B Buin

---

1.0
Norma lized Deviation
0.8
0.6
0.4
0.2
0.0
1-bit
1.58-bit
2-bit
3-bit
4-bit
Figure 4: LI norm difference between QAT-finetuned weights and full- full-precision initialization (l]Minemn:: - Waallln lnn
Win lt). Models quantized 1, 1.58, and bits show larger weight changes, indicating distinct "compensation" behavior higher-bit quantization and "reconstruction" in lower-bit settings.
where Eer and BAT describe the optimal allocation computational budget Brain- We utilize Buain to incorporate training tokens utilization (D) into the training strategy (S). Specifically, evaluate various allocation ratios of BEPT and BOAT on MobileLLM- 125M across four bit-widths ( 1.58-bit, 2-bit, 3-bit, and 4-bit). The FP models undergo complete learning rate scheduling cycle for Berr tokens, followed by another cycle for QAT for BOAT tokens. Detailed experimental settings are provided in the appendix.
Figure revealls distinct upward trend in the full- cell-precision pre-training proportion versus accuracy curve. Notably, accuracy peaks at 90% FPT allocation for almost every bit-width choice, then decline sharply when FPT exceeds 90%, likely because this leaves insufficient tokens and training capacity for QAT. This leads to our first key finding:
Finding-1 QAT finetuning consistently surpasses both PTQ with Birr Bunin and QAT from scratch with BeAT = Briin Optimal performance is nearly achieved by dedicating the majority of the training budget to full precision (FP) training and approximately 10% to QAT.
2.2 Finetuning Characteristics
Then we investigate the impact of finetuning tokens across various bit choices, spanning 7 architec- tures and 5 bit levels. Results in Figure offer ooffe several key insights:
finetuning benefits across all bit-widths: This observation challenges recent methodologies that trains ternary LLMs from scratch [Kaushal et al.) [2024] Ma et al. 2024). Instead, we suggest leveraging pre-trained full-precision models for initialization more effective approach for training quantized networks, including binary and ternary.
2. Optimal finetuning budget and bit width: Lower bit quantization (binary, termary, 2-bit) requires more finetuning than higher bit quantization (3-bit, b-ti.). 3-bit and 4-bit reach near full precision accuracy after 1OB tokens, while lower-bit quantization saturates around 30B tokens.
QAT behavior transition between bit-widths: Networks quantized to 3-bit/4-bit recover near full- null-precision accuracy after finetuning, while binary, ternary, and 2-bit saturate before achieving full accuracy. We hypothesize that QAT acts as "compensation" for bit-widths above 2-bit, adjusting weights within adjacent quantization levels, and as "reconstruction" below 2-bit, where weights adapt beyond nearby grids to form new representations. This is supported by weight change analysis in Figure 4 showing smaller adjustments in 3-bit/4-bit (10-20%) and larger shifts in lower-bit quantization (~40%), indicating substantial value reconstruction.
Finding-2 While finetuning enhances performance across all bit widths, binary and ternary, optimal finetuning effort inversely correlates bit-width. For and 4-bit weights, finetuning adjusts within nearby grid to mitigate accuracy loss, and requires less finetuning tokens. In contrast, binary and ternary weights hreak the grid, creating new semantic representations to maintain performance, requiring longer finetuning.

---

Elastic Temarizution
2-bit 2-hic
Binary
-1.5
15
4.0
-30
2.0
-.00
1.0
2.0
3.0
(d)
Elustic Quant (SPQ)
SRQ 2-bir
SPQ 3-hit
10
05
035
1.0 03
13
2.0
4.0
3.0
2.0
100
1.0
2.0
3.0
4.0
Figure 5: Impact of quantization grid choice across bit widths. Binary quantization uses sign function; Temary and 2-bit prefer more balanced output levels and range coverage; For 3-bit and higher, including "0"" in quantization levels is more favorable.
3 A Hitchhiker's Guide to Quantization Method Choices
We have examined the impact of training strategy and budget allocations (Buain BQAT) scaling laws. Building on the optimal training practices outlined in Section 2 we focus on a critical yet often overlooked factor: the choice of quantization functions (F).
F arg min L(FIP, ((AT)
The significance of this choice has been largely underestimated in prior scaling law studies (Kumar al. 2024). Our results show that, especially at sub-4-bit quantization, the choice of function highly sensitive and can drastically alter scaling law outcomes. An improper selection can distort performance and lead to entirely different conclusions, underscoring the need for careful design of F.
3.1 Preliminary
In general, uniform quantization function expressed as W - aL Wir ++ where Wa W represents quantized weights, WR R denotes their real-valued counterparts (Nagel et al. 2021 Krish namoorthi 2018). Key design choices focus on scale a and bias 3. For symmetric min-max max quantiza- [all and B 0. In asymmetric min-max quantization, a max(Wa) 2 2N-1- 3 min W R). Symmetric min-max quantization is prevalent for weights > 4 bits, while sub-4-bit quantization requires distinct functions.
tion, a
and
2d
For binary quantization, assigning the sign of ail--prccisio weights (Wp R) to binary weights (WB) is a commonly used approach [Rastegari et al. 2016 Liu al. 2018): W a-Sign W R. where
|alll In ternary quantization, ternary weights are often given by W T a- a-Sign W nWR
W
Σ. W -wi Zhang et al. 2020 Liu et al. 2023a Besides
with A 0.7-1| Walli and Or Σ, 1w "Wa
binary and ternary quantization, there is less work targeting 2-bit 3-bit integer quantization function design. Directly using min-max quantization for them will lead to performance collapse.
3.2 Introducing ParetoQ
In sub-4-bit quantization, design requirements vary significantly across bit levels. Equal attention to each bit choice is crucial for accurate, reliable comparisons.
3.2.1 Trade-offs
We identify two key trade-offs in low-bit quantization for LLMs: (1) Outlier precision vs. intermediate value precision and (2) Symmetry inclusion of "0" at the output level.
(1) Range clipping Outliers challenge LLM quantization [Ln et al., 2023 Liu et al. 2024a), especially when using min-max ranges for weight quantization for extremely low-bit quantization. As seen in Figure (b)-(e), min-max quantization works at bits but loses accuracy at lower bit-widths. On the other hand, range clipping improves bit quantization but harms 4-bit accuracy. We refer
5

---

[b) 1.58-biit Leamable Statistics
[e] 2-bit Learnable
(d) 3-bit Learnable Statshiiss
(e) 4-bit Learnable Statsifics
VS.
quantization
method
60
50
50-
Vahue
40
300
125
30
Nodel
[LaN6-]
Leamasle
52-
15
50
6000%
1,30
1.5.4
(201
12.0
350
550
35/00
620010
1.30
MOBIILLUM
LLWMA-33
LL.394-3
[La866-3
LLMMA3
Figure 6: Comparison of quantization methods across different bit widths. Extreme low-bit quantization highly sensitive quantization function selection. (b)-(e) show that learnable policy with range clipping updated final loss consistently outperforms stats-based methods with fixed range clipping. From (f)(1), the SEQ works better for ternary and 2-bit quantization, while and 4-bits favor LSQ.
to range-setting methods based on weight statistics as "stats-based" approaches. The effectiveness of these quantization functions varies with different bit choices.
Learnable scales, however, optimize quantization ranges as network parameters, balancing outlier suppression and precision. Solutions like LSQ [Esser al. 2019 and its binary (Liu et al., 2022] and ternary [Liu et al., 2023a] extensions exist. While prior work favored learnable policies for activations but used statistics-based quantization for weights (Liu et al. [202bb), find that, with appropriate gradient scaling, learnable scales yield stable, superior performance for weights. As shown in Figure(o(b)-(e)) learnable policies consistently outperform stats-based methods across all bit widths.
(2) Quantization grids Level symmetry in quantization grids is crucial for lower- bit quantization, yet it is rarely discussed. The "0" in quantization output levels is essential for nullifying irrelevant in- formation, but in even-level quantization (e.g., 2-bit, 3-bit, 4-b)t), including "0" results in imbalanced levels. For example, in 2-bit quantization, options like (-2. 1, O, 1) and (-1.5, 0..5, 0.5, 1.5) exist. The former limits representation with only one positive level, while the latter offers balanced distribution. Inspired by this, we propose Stretched Elastic Quant (SEQ), an amendment to LSQ for
lower-bit scenarios: W = a (LClip Wi: 1,1) x 0.5] + 0.5) /k: x Here, denotes number of quantization levels. Figure 5 visualizes quantization grids, showing that SEQ not only balances output quantized levels but also evenly divides the full-precision weight span to quantization levels, which turns be crucial for extremely low-bit quantization. Figure (f) demonstrate SEQ's superiority in ternary and 2-bit quantization, while LSQ with "0" in output level slightly outperforms in 3 and 4-bit cases.
(f)-(i)
Finding-3 Extreme low-bit quantization is highly sensitive to quantization function selection, with no single optimal function for all bit widths. Learnable range settings outperform statistics-based methods due to their flexibility optimizing range parameters with respect to the final loss. Temary and 2-bit quantization favor symmetric levels balanced range coverage quantization grid configuration, while imbalance levels with "0" in output levels are more effective for and b-bit guantization.
3.2.2 Quantization Function
Based on our analysis, we integrate the optimal quantization functions identified for each bit- wit-width into one formula, denoted as ParetoQ. This includes Elastic Binarization (Liu et al. 2022 for 1-bit quantization, LSQ [Esser et al. 2019 for 3 and 4-bit quantization, and the proposed SEQ for 1.58 and 2-bit quantization:
a- Li-Sinn(ae (Wig a([Clip( w -1,1) k/2 0.5) 2, Noa 1.58, (3)
if Noa =
W = a
al Clippp W
n,
if Nois = 3.4

---

[a] Pareto optimal quantized MobileLLM
Pareto optimal quantized LLaMA
Pareto optimial CPU speed (token/s)
62
(ecura
57
1-bit
Accuracy
1-bit
559
1.-biiit
60
1.58-bil
2-bit
2-bit
2-bit
53
3-bit
55
3-bit
4-bit
45
4-bit
4-bit
50
16-bit
0
200 400
6000
1000
2000
3000
4000
20
40
60
80
100
Effective Quantized Model Size
Effective Quantized Model Size
CPU
Speed (token/s)
Figure 7: (b) bub regime, 1.58-bit, 2-bit, and 3-bit quantization outperform b- in the accuracy- model size trade-off. (c) Under hardware constraints, 2-bit quantization demonstrates superior accuracy-specd trade-off's compured to higher-bit schemes.
Here & equals 3 in the ternary case and Nott otherwise; = 2 and 2 - 1. In the backward pass, the gradients to the weights and scaling factor can be easily calculated using straight-through estimator:
Sign Wi). Wi
if Nou =1 No.: =1.58,2
if Now 1,,58,
a STE
Wi Ke'
aw q
W
oW; i
Wk
if No4=3344
W
Wie
Ni. = =3,4
(4)
(5)
For the initialization of a, we use |wall for the binary case, since the scaling factor has
"Wr
the closed-form solution minimizing quantization error: = lla WR l12. For the other cases, we simply initialize a as the maximum absolute value of the weights. For ternary and 2- bit quantization, a max ((r.)). RR), associated with SEQ quantizer, and for 3-bit and 4-bit cases,
a max([Wn] associated with LSQ quantizer.
With ParetoQ, we present a robust comparison framework across five bit-widths (1-bit, 1.58-bit, 2-bit, 3-bit, 4-bit), each achieving state-of-the-ane accuracy. This facilitates direct, apple-to-apple comparisons to identify the most effective bit-width selection.
Pareto-Optimality of Extremely Low-Bit LLM
To ensure a consistent apples-to-appie performance comparison across different bit-width config- urations, we first determined the optimal training setup (Bt (Birain) in Section and the quantization function (F**) for each bit Section 3 Using this unified framework for all widths, we examine the trade-off between model size and quantization bit: C(P,NIJ F Birain).
4.1 Accuracy-comppession Trade-off
In on-device deployment scenarios, such as wearables and portables, storage constraints often limit the capacity of large language models (LLMs). To optimize performance within these constraints, quantization is essential. A common dilemma is whether to train a larger model and quantize it to a lower bit- width to train smaller model and quantize it to higher bit-width.
4-bit quantization-awee
training (QAT) achieves near-lossless compression in many scenarios, making it widely adopted. However, the landscape below 4-bit remains unclear, with limited comparative analysis. Previous claims about ternary models matching 16-bit performance [Ma et al.) 2024] were based on lower FP16 basclines than current standards. Spectra's comparisons between ternary QAT and 4-bit PTQ fall short a fair evaluation due to inconsistencies in the training schemes used KKushhal et al. 2024
With ParetoQ, we are able to improve the analysis. Figure (a) demonstrates that sub-4-bit quanti- zation, including binary, ternary, 2-bit, and 3-bit, often surpasses 4-bit. Notably, 2-bit and ternary models reside on the Pareto frontier. For instance, 2-bit MobileLLM-111 model achieves 1.8 points higher accuracy than 4-bit MobileLLLM-O0OM model, with even smaller model sizes. This trend persists across larger I.LaMA models, as shown Figure 7 (b), demonstrating the potential of lower-bit quantization for achieving both higher accuracy and compression. We calculate the effective
7

---

quantized model size as (#weights x weight-bits + #embedding x (abbeddiingbits/// More comprehensive analysis provided in the Appendix.
4.2 Hardware Implementation Constraints
In practical deployment, both memory limitations and hardware constraints must be considered. While 2-bit and ternary quantization sit on the accuracy-size Pareto frontier, 2-bit quantization is generally more feasible due to practical challenges. Ternary quantization, using 1.58-bit format with values {-1.0, 0, }. appears more storage-efficiet but inefficient implementation. Storing ternary values with sparsity exploitation is effective only when sparsity exceeds 90%, due to high indexing costs. Packing temary values into an Int32 offers limited compression but complicates GEMM. Some approaches [Yang et al., 2024] even store ternary values 2-bit signed integers, negating the expected storage benefits. In contrast, 2-bit quantization directly maps bit pairs to values, reducing unpacking and conversion overhead, which can be more efficient for custom GEMM kernels. As result, 2-bit quantization is often a more practical choice for deployment.
4.3 Accuracy-speed Trade-off
To evaluate potential speedup benefits beyond memory reduction, we implemented 2-bit quantization kernels on the CPU and compared them with 4-bit quantization. The curves in Figure demonstrate that, within experimental range, 2-bit quantized models consistently outperform 4-bit models in terms of securacy--peedd performance, positioning 2-bit quantization as a superior choice for on-device applications where both latency and storage are critical. See appendix for detailed settings.
5 Experiments
In this section, we compare each point on our Pareto chart with prior methods in the literature. As the first approach to unify training and quantization schemes the sub-4-bit regime, we evaluate our method against specialized techniques for each bit setting. This includes binary quantization
methods: BiLLM (Huang et al.) 2024], ARB-LLM [Li et al. 2024]. PB-LLM (Shang et al. 2023).
and DB LL- [Chen et al. [204a| | ternary quantization methods: TernaryLLM [Chen et al. 2024c
1-bit Era [a et al. 2024] and lower-bit QAT methods: LLM-QAT (Liu et al.
[023c] and Efficien
1QAT [Chen al. 2024b well as PTQ methods like GPTQ [Frantar al.
2022 OmniQ Shao
et al. 2023), SpinQuant [Luu et al. 2024a QulP [Chee et al. 2024] and AWQ
Lin et al.
2023
We
also compare with post-training vector quantization method AQLM (Egiazarian al.
2024
demonstrate that ParetoQ, with unified scheme spanning five distinct bit settings (1, 1.58, 3, and 4 bits), consistently outperforms previous methods specialized for each bit level, including both PTQ and QAT approaches. The performance gains are particularly pronounced in the 1, 1.58, and 2-bit settings, underscoring the robustness and reliability of our conclusions regarding scaling laws.
5.1 Experimental Settings
We conduct experiments on eight models including MobileLLM Liu et al. 2024b 125M/350//00M//// BTA////0000/1//1/55/5 and LLaMA-3 [eIMMeta] 2024) IB/3B/SB. Our evaluation was carried out on eight zero-shot commonsense reasoning tasks and Wiki2 Meeity et al. 2016) test set.
During the quantized network training process, we initialized the models with pre-trained weights. Following the common practice [Frantar et al. 2022 Liu al. 2023c)) all weights except for the embedding and output layers are quantized. We employed the Adam [Loshchiloy and Hutter 2017] optimizer with zero weight decay for optimization. The training was distributed across 16 GPUs, with each GPU handling a batch size of 8. For binary, ternary, and 2-bit quantization settings, the optimization process spanned 120,000 iterations with initial learning rate of 2 x 10-5, For 3-bit and bit settings, the process involved 40,000 iterations with initial learning rate of 1 x 10 5 The learning rate decayed to zero following cosine learning rate decay.
5.2 Main Results
1 / 1.58 / 2-bit Comparison on 8B Model Let's first examine the comparison on 8B parameter models. As depicted in T in the 2-bit quantization setting, previous methods, including both PTQ and QAT, experience significant drop in accuracy, Among PTQ methods, the vector quantization method
8

---

Table 1:
Comparison of 1-bit, 1.58-bit and 2-bit quantization methods LLaMA-3 8B model. Results for LLM-QAT, EfficientQAT, GPTQ, AWQ, SpinQuant, OmniQ, AQLM, 1-bit era and BiLLM were obtained using their publicly released models and codebase. The results of DB-LLM, PB-LLM, QulP and TernaryL.MM are quoted from the TeM paper. The results of ARB-LLM is soureed from their paper. All methods employ integer quantization, except AQLM, which uses vector quantization with a vector dimension of 16.
Group Size
A ARC PIQA HellaS WinoGi Avg. Wikiz
Method
#Bits
(1)
(1)
(t)
(t)
(1)
(t) (L)
FP
16
81.0
57.7
81.0
79.5
73.9
74.6 6.15
RTN
ichannel
27.2
25.1
49.7
26.1
50.5
35.7 1.2c6
GPTQ
channel
27.4
24.6
51.0
25.9
50.6
35,9 1.6c2
OmniQ
ichannel
27.3
22.8
49.5
25.3
49.4
34.8
SpinQuant
cchannel
32.4
21.8
53.4
31.9
50,9
38. 31.2
AWQ
cchannel
26.0
27.1
51,4
26.1
49.8
36,1
QuIP
cchannel
85.1
AQLM
12.27:
Ix16
75.5
$1.8
78.8
75.3
69.8
70.2
DB-LLM
22.12
128
13.6
PB-LIMM
12.12)
24.7
LLM-QAT
|channel
54.8
35.9
68.0
58.0
54.7
54.3
29.5
EfficiemQATT2.12
128
69.3
46.8
76.4
69.0
66.3
65.5
9.6
ParetoQ
[channel
78.5
54.5
79.2
73.8
70.0
71.2
8.0
PB-LLM 1.7 128
41.8
1-bit era 1 11.58 channel
72.8
45.4
81.0
70.6
58
65.6
11.7
TemaryI.. 1 1.58, channel
11.2
ParetoQ 11.58 channel
76.3
51.4
77.7
71.9
67.7
69.0
8.6
BiLLM 11.06 128
33.2
25.6
54.6
32.7
50.5
39.3
38.5
ARB-I.I.
1.06
channel
27.4
ParetoQ channel
75.5
$1.9
76.6
69.4
65.6
67.8
9.5
2-bit Quanization
3-bit Quantization
4-bit Quantization
70
70
70
65-
Average 0 shot
60
60-
GPTQ
over
50
AWO
55
OmniQ
40
50
SpinQuant
FaretoQ
125M
350M
600M
18
1.58
16
38
125M
350M
600M
18
1.58
18
38
88
125M
350M
600M
18
1.58
18
38
88
MobileLLM
LLaMA-3
MobileLLM
LLaMA-3
MobileLLM
LLaMA-3
Figure 8: Accuracy comparison on eight models. ParetoQ outperforms all sfate-of-the-at art PTQ and QAT methods.
AQLM Egiazarian et al. 2024] effectively mitigates some of the quantization loss, achieving 64.1 points, falls 10.5 points short of full precision. The best quantization-awae
training method, EfficientQAT [Chen et al. 2024b), still suffers a 9. 9.1-point decline in average accuracy. ParetoQ dramatically narrows the 2-b quantization gap to full precision to just 3.4 points, outperforming the best QAT method by 5.7 points and the vector quantization method by 7.1 points.
In ternary cases, the accuracy drop is more pronounced, highlighting the effectiveness of different quantization methods. A follow-up work of the 1-bit Era [Mekkouri et al.) (024)), which trains b-bit LLaMA-3 8B models using 100B tokens and complex techniques like binary relax with sigmoid schedulers, still experiences a 9.0-point accuracy drop. In contrast, ParetoQ requiring only 30B tokens and utilizing standard Adam optimization with cosine learning rate decay, narrows the gap to just 5.6 points. This underscores the robustness of our quantization function design.
Furthermore, ParetoQ significantly outperforms previous binary quantization techniques, such as BiLLM and ARB-LLM, reducing WikiText perplexity from 27.4 to 9.5.
1.58-bit Comparison on Sub-8B Models Figure 9 illustrates that ParetoQ also excels in sub- 8B models, consistently outperforming previous methods targeting at ternary quantization aware training such as 1-bit Era [Ma et al., 2024]. Given that full- full-precision LLaMA-3 3B model achieves 65.2 accuracy, it's remarkable that ParetoQ ternary 3B-parameter model narrows the gap to just 3.3 points, while previous methods experience drops exceeding 7.4 points. Additionally, our 600M -arammeree ternary model achieves 58.7 accuracy, even surpassing 1-bit era ternary 3B model with only 1/5 parameters.
9

---

Average 0-shot Accuracy
60
55
50
45
1-bit era
RTN
40
LLM-QAT ParetoQ
35
500M
1.08
1.5B
2.08 2.5B
3.0B 3.5B
Number
Figure 9: Temary quantization accuracy averaged across eight zero-shot commonsense reasoning tasks. ParetoQ consistently outperforms all prior methods in ternary quantizationawwre training.
2-bit / 3-bit /--iit Comparisons As evidenced by Figure 8 compared to previous state-of-the-an PTQ and QAT methods on 2, 4-bit quantization settings, approach consistently resides the Pareto front, with particularly pronounced advantage in lower-bit quantization settings. These results confirm that our bit-accuracy trade-off conclusions are benchmarked against SoTA results across all bit settings, ensuring its reliability.
Related Work
The quantization of Large Language Models (LLMs) has emerged as pivotal research area, driven by the imperative to reduce computational and memory demands while preserving model performance
Liu et al. 2023c Dettmers et al.] 2022 Xiao et al. 2022 A notable trend is the quantization of
LLMs lower bit-widths [M et a.. 2024 Kaushal et 2024).
Initial efforts, such as LLM..nn8() [Dettmers et al.
2022 and SmoothQuant [Xiao et al. 2022 con-
centrated quantizing I.LMs to 8-bit weights and 8-bit activations. Subsequently, numerous studies have demonstrated the feasibility of quantizing LLMs to 4-bit with minimal accuracy degradation, employing both post-training quantization (PTQ) methods [Kim et al.) [023 Frantar al. 2022 Liu et al. 2 2023b and quantization-awee aware training (QAT) Liu et al. 2003c Chen et al. 2024b
Bondarenko 2021).
Recently, research has shifted towards sub-4-bit quantization, Some PTQ methods target 3-bit or 2-bit integer quantization (Shao al.) 2023 Zhao al., 2023 Chee al.
2024 Ashkboos al.
2023 Lin et al., 2023 Frantar et al. 2022]] or employ vector quantization Egiazarian et al. 2024
Tseng al.
2024 van Baalen et al. [0223 Other PTQ approaches even achieve binary weight
quantization Huang et al. 2024 Shang et al. 2023 Chen et al. 2024a ei al.
2022). Most recently,
two QAT studies have claimed that ternary quantized models, trained from scratch, match the
accuracy of full-precision models with equivalent training [Ma et al.) 20024] Kaushal et af.) 2024).
generated heated debate within the field, with many practitioners expressing reservations about this conclusion. To our knowledge, no existing work unifies sub-4-bit quantization schemes to derive solid conclusion on which bit-width achieves the Pareto optimal the efficiency-accuracy trade-off. This work presents ParetoQ to fill that gap.
7 Conclusions
In this study, we have performed an in-depth analysis of the intricate relationships among model parameters (NV,, training data volume (D), quantization training schemes (Biain), quantization precision (P), and the selection of quantization functions (F) in relation to the model's final loss, expressed as
fNN P. Brain F). To address these multifaceted challenges, we propose ParetoQ, an advanced quantization framework that achieves ff-te-of-the-- performance across all bit-width levels. This framework uniquely enables a direct, consistent comparison across different bit-widths, ensuring an equitable evaluation of performance metrics. Our empirical analysis indicates that quantization 1.t8-bit, 2-bit, and 3-bit offers a superior trade-off between accuracy and effective quantized model size compared to 4-bit, highlighting their potential for optimized model deployment.
10

---

References
Al@Meta. Llama 3 model card. 2024. URL 00t.1//00//////////////////////0
CARD. md]
Salch Ashkboos, Amirkeivan Mohtashami, Maximilian L.. Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. 2023.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the challenges of efficient transformer quantization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7947-7969, 2021.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024.
Hong Chen, Chengtao Lv, Liang Ding, Haotong Qin, Xiabin Zhou, Yifu Ding, Xucho Liu, Min Zhang, Jinyang Guo, Xianglong Liu, et al. Db-lIm: Accurate dual-binarizatio for efficient llms. arXiv preprint arXiv:2402. 19600, 2024a.
Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang. Peng Gao, Kaipeng Zhang. and Ping Luo. Efficientqat: Efficient quantization-wwar training for large language models. arXiv preprint arXiv:2407. 11662, 2024b.
Tianqi Chen, Zhe Li, Weixiang Xu, Zeyu Zhu, Dong Li, Lu Tian, Emad Barsoum, Peisong Wang, and Jian Cheng. Temaryllm: Ternriizdd large language model. arXiv preprint arXiv:2406.07177, 2024c.
Christopher Clark, Kenton Lee. Ming-Wes Wei Chang. Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty natural yes/no questions. arXiv preprint arXiv:1905.10444 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1.X3::1803.054555 2018.
Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. In International Conference on Machine Learning, pages 7750-7774. PMLR, 2023.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.in8()): 8-bit matrix multiplication for transformers at scale. 2022.
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401.06/1 2024.
Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S Modha. Learned step size quantization. arXiv preprint arXiv:1902.088 2019.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre- preitrained transformers. arXiv preprint arXiv:2210.1732. 2022.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 2022.
Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and Xiaojuan Qi. Billm: Pushing the limit post-training quantization for lims, arXiv preprint Xii:22402.0429, 2024.
Mandar Joshi, Eunsol Choi, Danicl Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXivv 2017.
Ayush Kaushal, Tejas Vaidhya, Arnab Kumar Mondal, Tejas Pandey, Aaryan Bhagat, and Irina Rish. Spectra: Surprising effectiveness of pretraining ternary language models at scale. arXiv preprint 1232 2024.
Sehoon Kim. Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li. Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and and--adddaasra sparse quantization. arXiv preprint arXiv:2306. 076299 2023.
11

---

Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint 106.v:1806.08342, 2018.
Tanishq Kumar, Zachary Ankner, Benjamin F Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan. Scaling laws for precision. arXiv preprint arXiv:4411.04330 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gon- zalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy, Race: Large-scale reading comprehension dataset from examinations. arXiv preprint 1704 04683, 2017.
Zhiteng Li, Xianglong Yan, Tianao Zhang. Haotong Qin, Dong Xie, Jiang Tian, Linghe Kong, Yulun Zhang, Xiaokang Yang, et al. Arb-llm: Alternating refined binarizations for large language models. arXiv preprint arXiv:2410.03129, 2024.
Ji Lin, Jiaming Tang. Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306. 00978, 2023.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang. Wei Liu, and Swang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. In Proceedings of the European conference on computer vision (ECCV), pages 722-737 2018.
Zechun Liu, Barlas Oguz, Aasish Pappu, Lin Xiao, Scott Yih, Meng Li, Raghuraman Krishnamoorthi, and Yashar Mehdad. Bit: Robustly binarized multi-distilled transformer. arXiv preprint arXiv:2205. 13016, 2022.
Zechun Liu, Barlas Oguz, Aasish Pappu, Yangyang Shi, and Raghuraman Krishnamoorthi. Binary and ternary natural language generation. arXiv preprint arXiv:2306.0188 2023a.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Lim-qat: Data-free quantization aware training for large language models. arXiv preprint Xr0::2300. 17888, 2023b.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Lim-qat: Data-free quantization aware training for large language models. arXiv preprint 230. 2023c.
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi,
q Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant-IIm quantization with learned rotations. arXiv preprint 240.::22005..60006 2024a.
Zechun Liu, Changsheng Zhao, Forrest landola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Emic Chang, Yangyang Shi, Raghuraman Krishnamoorthi, et al. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases. arXiv preprint arXiv:2402. 14905, 2024b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 10.0001 2017.
Shuming Ma. Hongyu Wang. Lingxiao Ma, Lei Wang, Wenhui Wang. Shaohan Huang. Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The era of 1-bit Ilms: All large language models are in 1.58 bits. arXiv preprint a40iv:2402. 17764, 2024.
Mohamed Mekkouri, Mare Sun, Leandro Werra, Pedro Cuenca, Omar Sanseviero, and Thomas Wolf. Fine-tuning 11ms 1.58bit: extreme quantization made easy, September 2024. URL/https:/huuuiiggfee co/blog/ Baalllgg1___6_11 1_58_11m.extreme 1es_1mmmexterme.uuuntiiaaaaaaa
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint 1cXiv:1609.07853 2016.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint arXiv: 10001100880880 2018
Markus Nagel, Marios Fournarkkis, Rana Ali Amjad, Yelysei Bondarenko, Mart van Baalen, and Blankevoort Tijmen. A white paper on neural network quantization. arXiv preprint arXiv:2
2021.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint 160iv 160.0660.. 2016.
12

---

Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference computer vision, pages 525-542. Springer, 2016.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communications of the ACM, 64(9):99-166 2021.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:I904,000 9999009722, 2019.
Yuzhang Shang. Zhihang Yuan, Qiang Wu, and Zhen Dong. Pb-llm: Partially binarized large language models. arXiv preprint arXiv:2310.00034, 2023.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308. 13137, 2023.
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher Sa. Quipit: Even better quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.04390 2024.
Mart van Baalen, Markus Nagel Andrey Kuzmin, Peter Couperus. Cedric Bastoul, Eric Mahurin, Tijmen Blankevoort, and Paul Whatmough. Gptvq: The blessing of dimensionality for Ilm quantization. 2023.
Johannes Welbl, Nelson Liu, and Matt Gardner. Crowdsoureing multiple choice science questions. arXiv preprint arXiv:/707.062000 2017.
Lucas Wilkinson. Introducing machete, mixed-input gemm kernll optimized nvidia hopper gpus, October 2024. URL https:h//neurallagggggg/// wttp://wwwa.......aaaaaaaeeeeeae cww Lntroducingggmchheeee
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In CVPR, 2022.
Chenglin Yang, Celong Liu, Xueqing Deng, Dongwon Kim, Xing Mei, Xiaohui Shen, and Liang- Chhic Chen. 1.58-bit flux. arXiv preprint arXiv:2412. 18653, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Farhadi, and Yejin Choi, Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: 1905.0780, 2019.
Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. Ternarybert: Distillation- aware ultra-low bit BERT. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, EMNLP, 2020.
Yilong Zhao, Chien Yu Lin, Kan Zhu, Zihao Yc, Lequn Chen, Size Zheng, Luis Coze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate Ilm serving. arXiv preprint arXiv:2310.1 19102, 2023.
13

---

A Appendix supplemental material
A.1 Complete Results of Figure
2 the numerical results of Figure 9 We evaluate accuracy across eight zero-shot com-
monsense reasoning tasks: ARC-easy, ARC-challenge [Clark al. 2018 BoolQ [Clark et al. 2019) PIQA [Bisk et al. 2020]. SIQA [Sap et al. [219 HellaSwag [Zellers et al. 2019 OBQA MMihaylov al. 2018 and WinoGrande (Sakaguchi al. 2021], along with perplexity the WikiTex12 test set Merity et al. 2016]. Our results are compared against prior state-of-the- ternary quantization- aware training works, such 1-bit era [Ma al. 2024). We also include the comparison to LLM-QAT [Luu et al., [023c] Consistent with previous methodologies [Ma et al. 2024). we quantize all weights low-bit, excluding the embedding and output layers. The ParetoQ 3B ternary model
is quantized from LLaMA3 [Al Meta 2024 3B model, while other models are quantized from
MobileLL.M [Lu et 2024b
Table 2: Comparison of ParetoQ ternary quantization with QAT methods, including general LLM-QAT Liu
2023c] and sernary-specifie QAT methods as 1-bit Era [Ma al. 2024].
Method
# Params
ARC-e
(t)
ARC-c
(t)
BoolQ
(1)
PIQA
(f)
SIQA HellaSwag DBQA WinoGrande Avg.
(f)
(f)
(1)
Wiki2 (1)
RTN
125M
25.5
26.5
37.8
49.6
36.3
26.3
49.3
34.9
4.0c5
LLM-QAT
125M
54.6
39.0
87.3
ParetoQ
125M
49.3
30.9
61.0
62.1
41.0
34.3
52.9
46.5
19.8
RTN
350M
26.6
25.1
37.8
48.7
16.7
26.5
27.5
50.2
34.9
3.7e5
LLM-QAT
350M
39.1
24.1
61.6
55.5
39.9
30.4
32.1
50.6
41.7
68.6
ParetoQ
350M
56.8
36.3
67.1
43,5
44,0
46.3
55.2
51.4
14.4
RTN
26.2
49.5
36.3
26.1
6.6e5
LLM-QAT
600M
34.0
23.0
59.4
53.6
38.9
28.7
32.3
51.4
40.2
71.7
I-bit cra
700M
49.5
29.0
59.2
67.5
43.6
43.2
38.9
$3.5
48.0
17.3
ParetoQ
600M
65.5
43.8
62.3
70.6
44.7
$1.3
47.1
58.8
$5.5
11.4
RTN
1B
25.7
24.8
37.8
49.3
37.1
26.2
25.2
50.2
34.5
1.4c5
LLM-QAT
1B
36.0
39.7
31.3
33.5
39.9
$6.9
I-bit era
L.3B
52.4
69_1
44.7
47
50.8
23.6
ParetoQ
18
68.5
47.6
62.8
72.1
45.3
$7.4
$2.9
61.3
58.5
10.0
RTN
1.5B
25.5
26.8
37.8
49.0
37.6
26.0
30.5
50.2
35.4
9.7c4
LLM-QAT
1.5B
41.1
26.1
60.5
$7.6
39.5
35.0
31.9
49.8
42.7
39.7
L.5B
70.2
73.4
47.3
55.3
9.0
RTN
26.9
$1.3
37.6
27.0
49.3
4.4e5
LLM-QAT
3B
44.5
30.7
62.1
62.7
41.0
43.4
35.0
50.6
46.3
6.5e2
bit era
3B
58.7
37.2
61.3
71.3
45.2
56.0
45.8
60.3
$4.5
265.6
ParetoQ
3B
71.5
48.6
68.2
75.5
46.4
67.9
54.3
63.1
61.9
9,9
A.2 Complete Results of Figure 8
In Tables 3 and
5 we provide detailed results corresponding to Figure
We compare ParetoQ
against LLM-QAT QAT
Liu et al. 2023c), GPTQ Frantar al. 2022]. AWQ Lin al.
2023 Omni-
Quant [Shao et al. 2023), and SpinQuant [Liu et al. 2024a]. Following the common practice Frantar
al. 2022 Liu al. 2023c we apply low-bit quantization to all weights, except for the embedding
and output layers.
A.3 CPU Latency Experimental Setup
We measure the CPU latency of five MobileLLM models on an Apple M1 MacBook Pro (32GB RAM) using 6 threads. Each evaluation uses prompt tokens and generates 122 tokens. For the quantized models, embedding and output layers quantized to 8-bit precision using channel- wise guantization, while weights in fully connected layers are quantized to 2-bit or 4-bit precision. Accuracy and decoding speed (in tokens/s) were measured under identical settings.
A.4 GPU Latency Experimental Setup and Results
We measured the latency of LLaMA 3.2 models (IB, 3B, 8B) on an H100 NVL GPU (94GB memory).
The W4A16 kernel used the Machete kernel from vLLM (K et al. 2023 Wilkinson 2024]. while W2A16 kernel was implemented based on the CUTLASS mixed precision backbone kernel. All tests were performed
single GPU with a context length of 2048 tokens. For kernel-level latency, we compared the 2-bit kernel the 4-bit Machete kernel across three weight shapes: (4096 4096), (8192 x 8192), and (16384 x 16384).
14

---

Table Complete results of 2-bit quantization WikiText2 and Zero-shot Common Sense Reasoning tasks.
Model Name
Method
ARC-e
ARC-c (t)
BoolQ (f)
PIQA
SIQA
HellaSwag OBQA
WinoGrande
Wikiz
(1)
(1)
(f)
(1)
56.0
34.5
56,3
42.0
42.2
51.3
48.5
RTN
26.1
24.1
62.2
50.3
37.1
26.6
26.6
49.4
38.1
6.465
MobilLLM-1125M
AWO
25.8
24.2
44.2
50.7
38.8
51.6
36.3
6.5c3
OmniQ
32.4
22.7
38.1
39.4
28.2
49.9
36,9
LLM-QAT
34.9
23.3
61.8
$1.8
39.3
29.1
274
51.3
40.1
66.8
SpinQuant
31.6
23.3
40.3
$2.2
40.5
28.6
28.9
50.1
36.9
68.7
Paretoq
50.7
12.7
59.8
63.3
41.0
36.3
40.6
52.7
47.1
25.1
FP
65.5
42.3
$7.4
71.0
43.5
$3.3
47.3
S8.3
54.8
10.4
62.2
49.8
37.7
26.3
$1.2
38.2
GPTO
28.6
21.5
40.5
38.8
26.6
50.4
35.5
1.6c2
MobileLLM-330M
AWQ
27.0
23.5
47.6
49.4
38.2
26.4
26.2
49.5
36.0
7.2e4
DmniQ
33.9
23.4
39.6
$4.9
38.4
28.0
29.4
49.7
37.2
80.8
LLM-QAT
40.6
25.9
62.0
55.6
40.0
318
31.1
$2.6
42.5
8.2el
SpinQuant
32.4
25.0
37.8
54.6
40,
29.2
27.5
48.9
36,9
67.5
ParetoQ
59.0
39.4
63.5
68.8
43.1
47.3
44.1
57.5
52.8
17.7
FP
68.5
47.6
60.5
72.5
44,4
59.5
51.
61.
58.2
9.0
RTN
25.8
26.2
37.8
49.8
37.6
25.9
26.8
509
35.1
2.7c2
GPTQ
27.9
26.6
48.2
49.5
39.0
25.9
26.8
4944
36.6
3.422
MobieLLM-600M
AWQ
26.4
25.2
40.6
50.7
38.7
26.5
21.6
49.1
35.1
K9c1
OmniQ
39.0
24.5
55.8
55.9
4112
30.1
32.1
51.3
41.1
68.1
LLM-QAT
42.7
25.6
62.1
56.0
38.8
33.7
29.6
51.5
42.5
4.7e2
SpinQuant
28.2
22.4
39.8
$2.0
38.0
27.9
22.1
49.1
34,9
2.1e2
ParetoQ
67.7
45.3
63.0
72.1
44.8
49.8
58.4
56.6
15.4
FP
73.4
50.8
67.6
74.1
46.7
64.7
56.6
62.7
62.1
8.0
RTN
26.3
16.3
62.
49.
36,9
26,0
25.8
48.
31.7
6.0c
GPTQ
29.7
25.4
38.7
50.3
38.9
26.1
26
49.6
35.6
4.7e2
MobileLLM-IIB
AWO
26.6
26.8
59.1
50.2
37.1
26.0
24.0
50.4
37.5
1.6e5
OmniQ
38.0
26.1
41.7
54.6
40.1
31.3
51.4
39.5
46.3
LAT
42.6
26.7
49.7
ST.7
40.4
34.9
314
49.2
41.6
1.96
SpinQuant
35.3
23.9
42.8
40.5
29.7
49,8
38.2
35.7
ParetoQ
65.7
61.0
FP
73.9
51.4
70.0
74.8
46.6
66.4
35.1
632
62.7
RTN
25.2
25.3
37.8
49.3
36.0
26.4
25.0
48.5
34.2
1.7e2
GPTO
29.8
12.3
45.3
$3.4
39.3
27.0
25.8
$1.4
36.8
1.
MobileLLMM11..
AWQ
28.9
26.1
43.7
$1.1
37.7
26.6
24.4
49
36.0
7.
OmniQ
44,9
LAT
SpinQuant
52.3
29.5
50.5
389
ParetoQ
75.3
47.5
70.1
74.1
46.8
64.6
$5.5
62.5
6188
11.7
64.8
42.5
74.8
44.8
61.5
58.5
RTN
26.5
16.3
62.2
$1.0
36.8
15.9
28.5
502
38.5
1.5eb
GPTO
29.3
27.6
37.8
50.8
36.8
3.322
LLaMA-IB
AWO
2.05
35.6
LLM-QAT
49.2
33.3
62.0
41.1
41.5
54.
47.9
1.365
SpinQuant
62.4
$1.6
36.1
25.8
50.
38.3
46.7
ParetoQ
64.8
41.7
62.8
73
44.0
56.6
$2.0
58.5
56.7
12.5
72.6
74.6
78.2
28.5
74.3
53.7
69.1
65.2
7.7
Re5
28.6
464
30
50.1
36.7
2.7e2
LLaMA-3B
AWO
27.3
38.2
50.7
35.6
OmniQ
28.3
24.6
378
50.5
38.0
25.3
26.6
50.2
35.2
65c3
M.M-QAT
49.3
35.3
63.5
65.2
41.7
48.9
52.2
48.5
2965
SpinQuant
28.3
23.7
53.2
$1.1
38.8
26.1
25.8
49.0
37.0
57.4
Paretog
73.9
49.0
68.8
76.4
47.0
69.2
$6.6
64.
63.2
9.1
FP
81.0
51.7
83.6
61.0
49.3
79.5
35.7
73.9
70,2
6.2
27.2
25.1
37,4
26.1
505
35.0
GPTQ
27.0
61.6
50.5
37.4
260
27.5
49.7
38.2
1.fe2
LLMMA-BB
AWQ
26.0
27.1
58.3
$144
38.0
26.1
218
49.8
37.6
1
OmniQ
27.3
22.8
37.9
49.5
38.7
25.3
234
494
34.3
7.64
LLM-QAT
54.8
35.9
64.8
68.0
41.8
$8.0
35.7
S4.7
51.7
29.5
SpinQuannt
32.4
22.0
59.0
51.2
38.4
31.9
28.0
49.9
39.3
31.2
ParetoQ
78.5
54.5
7644
79.2
48.9
73.8
54.5
70.0
67.0
8.0
For smaller models (IB, 3B, 8B), the performance speed-up from reducing weight precision from 4-bit to 2-bit is minimal. This is due to the impact of conversion overhead, which becomes more pronounced when the weight size is small. Since the in in-kernel conversion latency ratio is higher for smaller models. the benefits of 2-bit quantization 4-bit quantization achieves more favorable speed-au aeceurcc trade-off in these settings, offering better overall performance. In comparison, for larger weight shapes (16384 x 16384), the 2-bit kernel provides a substantial speedup, achieving 4. 14x faster performance than FP16 and 1.24x faster than the Machete 4-bit kernel.
outweighed by the overhead. Consequently,
A.5 QAT Scheduling Experimental Setup
The total training budget (Buain) is set to 100B tokens. We vary the proportion tokens allocated for full-precision training versus quantization-wwar training (QAT) finetuning, sweeping the ratio across
15

---

Table 4: Complete results of 3-bit quantization WikiText2 and Zero-shot Common Sense Reasoning tasks.
Model Name
Method
ARC-e
ARC-c (1)
BoolQ (f)
PIQA
SIQA
HellaSwag OBQA
WinoCrandee
Wikiz
(1)
(1)
(f)
(1)
56.0
34,5
56,3
42.0
42.2
51.3
48.5
RTN
43.7
30.0
59.0
60.3
20.4
34.9
36.5
50.5
44.9
38.2
MobilLLM-112MM
AWQ
48.5
27.8
52.7
40.1
35.6
44.1
27.1
OmniQ
50.2
41.6
36.4
50.2
45.8
LLM-QAT
44.7
78.7
53.7
60.6
41.1
34.6
34.9
50.2
43.5
37.5
SpinQuant
50.9
30.8
46.7
62.1
41.5
37.3
39.1
48.9
44.7
17.6
Paretoq
53.5
13.7
56.1
65.6
41.7
20.0
41.2
51.3
47.9
21.6
FP
65.5
42.3
$7.4
71.0
43.5
53.3
47.3
S8.3
54.8
10.4
41.8
49.7
GPTO
59.8
60.6
67.5
42.1
$3.9
50.4
MobileLLMM330M
AWQ
39.5
35.7
57.5
66.9
42.1
47.0
53.8
50.6
DmniQ
58.0
36.2
612
67.2
42.4
46.1
42.
$2.0
50.7
13.5
LLM-QAT
54.6
35.4
60.5
63.9
42.2
42.6
41.9
S3.4
49.5
22.6
SpinQuant
57.9
35.3
59.3
67.0
414
47.5
43.2
$4.3
50.7
12.1
ParetoQ
63.9
40.5
6144
70.6
43.2
51.4
50.0
56.6
54.7
14.9
FP
68.5
47.6
60.5
72.5
44,4
59,5
31.
61.
58.2
9.0
RTN
55.3
32.
$7.0
63.1
42.1
40.6
37.7
54.1
478
12.0
GPTO
61.4
18.0
SS.7
68.5
42.5
51.
43.2
56.2
52.2
11.7
MobieLLM-600M
AWQ
63.6
19.5
55.6
70.0
43.1
53.
45.0
5K.0
53.5
12.9
OmniQ
64.9
41.6
63.4
69.8
42.1
530
45.4
58.2
54.8
11.3
LLM-QAT
61.8
38.0
62.1
68.5
43.6
48.9
44.2
$4.6
52.7
19.0
SpinQuant
63.4
42.9
60.9
68.7
42.4
S20
44.5
$7.4
$4.0
10.5
ParetoQ
68.2
47.4
44.2
50.2
62.4
58.5
13.2
FP
73.4
50.8
67.6
74.1
46.7
64.7
56.6
62.7
62.1
8.0
RTN
59.7
16.6
5K99
67.2
25.0
44.5
5174
50.7
19.
GPTQ
66.7
430
63.5
72.3
42.9
57.8
49.2
59.4
56.8
10.2
MobileLLM-IIB
AWO
68.8
43.5
62.9
71.1
43.7
51.9
49.2
$7.0
56.8
10.8
OmniQ
44.7
64.8
72.1
43.5
47.0
57.7
57.1
QAT
65.3
42.6
61.2
70.4
44.0
54.3
48.
56.8
35.5
17.4
SpinQuant
63.5
43,9
$9.0
56.6
ParetoQ
45.7
61.3
FP
73.9
51.4
70.0
74.8
46.6
66.4
35.1
632
62.7
7.8
RTN
63.2
38.0
58.5
67
43.6
47.9
45.9
56.0
52.5
10.2
GPTO
70.6
43.7
64.5
71.9
45.0
59.2
50.8
58.9
58.1
9.9
MobileLLMM-1..
AWQ
72.6
46.8
66.0
71.7
44,6
$2.0
62.4
59.7
Omni
59.7
LLM-QAT
SpinQuant
71.5
6788
61.3
61.
59.3
ParetoQ
72.6
49.9
70.6
75.7
47.7
66.0
56.2
64.5
62.9
114
64.8
42.5
74,8
44.8
61.5
58.5
RTN
28.5
25.0
55.9
$3.5
37.8
30.1
28
50.6
38.8
3099
GPTO
37.4
27.3
43.1
53,8
LLaMA-IB
AWO
42.0
28.8
LLM-QAT
52.7
32.4
60.5
44.0
43.2
53.8
49.2
20.7
SpinQuant
56.9
61.0
69.3
42.0
53.4
41.2
56.2
51.9
ParetoQ
65.3
41.9
64.2
73
43.9
613
47.7
59.5
57.2
1099
72.6
74.6
78.2
48.5
74.3
53.7
69.2
65.2
1.7
$2.9
45.2
LLaMA-3B
AWQ
65.4
65.1
66.6
70%
48.1
44.1
53.8
51.1
OmniQ
59.7
38.6
47.6
13.5
62.4
61.8
54.9
L.M-QAT
64.4
40.1
62.0
71.7
45.0
58.2
44.7
59.9
558
SpinQuant
66.4
43.8
70.8
73.9
47.7
67.6
$1.0
67.1
61.0
9,2
Paretog
72.3
49.8
73.3
76.7
48.8
71.9
$6.2
67.3
64.5
8.4
FP
81.0
51.7
83.6
81.0
49.3
70.5
35.7
73.9
70,2
6.2
RTN
42.4
41.0
31.3
$3.9
42.9
12.6
GPTQ
60.8
35.5
69.0
70.3
44.9
61.3
649
55.7
9.1
LLMMA-BB
AWQ
72.3
46.
74.9
75.9
48.2
70.8
$2.0
06.8
63.4
166
OmniQ
68.0
454
68.3
73.9
46.0
68.7
304
62.3
60.4
12
LLM-QAT
68.8
48.8
71.1
75.9
46.8
67.8
48.2
65.1
616
10.5
SpinQuantt
75.5
52.0
81.0
78.7
49.2
74.3
$3.6
70.5
66.9
7.4
ParetoQ
78.2
55.7
80.6
80.2
50.1
76.5
55.1
70.9
68.4
7.0
[0, 0.01, 0.05, 0.2, 0.0,0.6..... 0.8, 0.9, 0.95, 0.99, 1). Here, ratio of 0 corresponds QAT from scratch, while a ratio of represents full-precision training followed by post-training quantization (PTQ).
For full- precision training, we use 8x8 GPUs, batch size 16, a weight decay of 0.1, an initial learning rate of 2.5 10 and linear learning rate decay to zero. For quantized network training. we also use 8x8 GPUs but with a batch size of no weight decay, an initial learning rate of 1 x 10 and linear learning rate decay to zero.
A.6 Embedding Bit Precision vs. Accuracy Trade-off
Despite the prevalent practice of not quantizing embedding and output layers, as noted in prior works such Frantar et al. [Frantar et al.) 2022 and Ma et al. [Ma al. 2024] our study extends the

---

Table S: Complete results of 4-bit quantization WikiText2 and Zero-shot Common Sense Reasoning tasks.
Model Name
Method
ARC-e
(1)
ARC-c
(1)
BoolQ (f)
PIQA
SIQA
(1)
HellaSwag OBQA
WinoGrande
(f)
(1)
Wikiz
56.0
34,5
56,3
42.0
42.2
51.3
48.5
RTN
53.4
13.3
$3.9
64.7
a1.5
39.7
46.2
5188
47.3
MobilLLM-112MM
AWO
33.5
56.6
41.9
41.
47.9
OmniQ
52.8
33.5
56.1
41.4
39.2
50.8
47.1
LLM-QAT
54.2
33.4
52.2
64.7
42.4
79.0
42.7
51.7
47,5
52.1
SpinQuant
55.2
33.7
58.1
65.0
42.5
39.7
40.6
49.8
48.1
13.4
Paretoq
55.4
15.2
54.1
66.2
41.7
20.8
44.0
52.1
48.7
20.4
FP
65.5
42.3
$7.4
71.0
43.5
$3.3
47.3
S8.3
54.8
10.4
63.6
70.3
42.8
53.9
GPTO
63.8
$3.7
69.7
42.7
57.
53.3
MobileLLM-330M
AWQ
63.0
38.5
$7.1
70.7
43.6
51.6
45.8
55.2
53.2
112
DmniQ
63.9
37.4
56.2
69.8
42.4
50.9
46.6
34.2
52.7
111
LLM-QAT
63.4
42.0
39.8
70.1
43.6
51.5
47.0
57.5
54.4
17.1
SpinQuant
62.5
37.8
56.1
69.6
43.1
51.5
43.8
55.7
52.5
10.6
ParetoQ
64.9
41.6
57.8
71.3
44.4
53.5
48.2
57.9
55.0
14.2
FP
68.5
47.6
60.5
72.5
44,4
59.5
514
61.4
58.2
9.0
RTN
67.8
45.1
48.5
71.6
44.0
57.8
49.8
59.6
55.5
154
GPTO
68.5
47.0
$0.2
12.3
43.8
S7.7
49.6
58.9
56.0
9.4
MobieLLM-600M
AWQ
68.8
45.0
ML.5
72.1
44.0
58.3
48.2
59.8
57.1
977
OmniQ
68.4
45.0
59.5
71.5
43.7
58.1
49.0
59.0
36.8
9.5
LLM-QAT
67.2
47.4
65.2
718
43.8
57.8
50.6
59.8
57.9
11.0
SpinQuant
69.1
44.7
64.3
71.5
43.0
S7.4
49.0
57.1
$7.0
922
ParetoQ
69.3
48.9
64.8
62.1
39.2
13.2
FP
73.4
50.8
67.6
74.1
46.7
64.7
56.6
62.7
62.1
8.0
RTN
73.1
47.7
63.
73.0
73.7
62.
36.
612
60.6
11.1
GPTQ
72.6
50.7
65.5
74.8
45.9
63.7
56.6
62.3
61.5
8.4
MobilcLLM--IB
AWO
75.7
48.6
65.3
73.5
45.6
62.5
49.4
60.6
59.9
8,5
OmniQ
72.5
49.3
66D
74.3
45.0
62.5
$2.
62.1
60.5
8.4
L.M
72.1
49.5
66.1
73.9
46.2
63.0
$5.4
63.7
61.2
10.0
SpinQuant
70.5
44.2
62.4
59.8
ParetoQ
62.1
62.5
FP
73.9
51.4
70.0
74.8
46.6
66.4
35.1
632
62.7
7.8
RTN
73.7
49.5
66.0
74,5
46.4
65.5
$2.7
62.0
61.3
9.4
GPTO
73.9
49.9
68.9
73.7
46.6
64.9
34.5
62.0
61.8
822
MobileLLMM-1..
AWQ
74.9
68.1
46.3
$2.2
63.8
616
82
Omni
822
LLM-QAT
SpinQuant
73.8
61.5
ParetoQ
74.4
$1.7
71.8
75.3
47.3
67.2
$7.6
63.0
63.6
11.0
64.8
42.5
64.8
74,8
44.8
64,4
61.5
58.5
RTN
53.7
16.3
61.9
70.4
25.0
36.9
39.1
55.5
52.4
13.9
GPTO
$7.9
41.
52.8
13.4
LLaMA-IB
AWO
LLM-QAT
61.3
62.3
73.0
44.2
59.0
58.7
54.8
SpinQuant
62.2
40.3
64.1
72.3
44.0
61.6
47.9
59.
56.5
10.3
ParetoQ
67.4
43.4
64.4
74.8
44.4
63.5
304
614
58.7
10.3
72.6
74.6
78.2
48.5
74.3
$3.7
69.1
65.2
7.7
LLaMA-3B
AWO
64.7
46.7
66.5
72.9
75.3
47.0
49.9
72.8
66.7
60.2
63.6
OmniQ
70.6
47.5
73.9
71.0
46.9
72.0
67.1
63.5
L.M-QAT
71.8
48.1
74.6
76.6
48.1
71.4
$2.
674
638
8.2
SpinQuant
70.2
47.9
73.8
76.4
47.8
71.9
$4.3
08.0
63.8
8.0
Paretog
73.8
50.3
75.4
71.2
48.5
73.3
$7.0
67.7
65.4
8.0
FP
31.0
31.7
81.6
81.0
49.3
79.5
35.7
739
70.2
6.2
75.8
71.8
78.5
48.1
14.7
71.6
66.4
GPTQ
77.7
51.9
80.6
794
50.8
76.7
518
716
67.6
7.0
LLMMA-BB
AWQ
78.5
51.8
81.8
80.7
49.2
78.3
52.8
72.6
68.2
20
OmniQ
77.3
51.3
79.2
79.6
48.0
77.2
348
704
67.2
7.1
LLM-QAT
71.4
$4.0
82.9
79.1
49.2
71.6
S4.3
72.0
68.3
134
SpinQuant
78.8
56.0
82.5
79.7
49.5
78.5
$4.6
71.5
68.9
6.5
ParetoQ
78.6
55.6
80.2
80.4
31.5
77.8
$5.7
71.8
69.0
6.8
scaling law analysis by examining the impact of quantizing these layers. As illustrated Figure utilizing 4-bit embeddings matching the bit precision of embeddings that of weights positions these configurations on the Pareto front, in contrast to employing 8-bit or 16-bit embeddings.
A.7
Weight Bit Precision vs. Accuracy Trade-off
For the trade-off between weight-bit precision and model accuracy, we consider two configurations: 4-bit embeddings and embeddings with the same bit precision as weights. In both scenarios, lower-bit quantization, such as 1.58-bit, 2-bit, and 3-bit, consistently outperforms 4-bit quantization, as depicted in Figure I2
17

---

(a) Pareto optimal GPU Latency
GPU kemel decoding speed comparison
70-
bit
4-bit
4.13x
4-bit
BF16
2-bit
68
3.6-
3.32x
3.04x
Accuracy
66
2.7
2.74x
64
62
1.8
60
1. 09x
0.9
58
0.0
18. 4096, 4096)
(0, 8192. 8192) Configuration
18. 16384, 16394)
GPU Latency (msitaken)
Figure 10: (a) Accuracy versus end-to-end GPU latency trade-off analysis. (b) Speedup in GPU kernel lateney
relative to BF16.
MobiieLLM models
59
61
61
61
Accuracy
54
W2E2
56-
W3E3
56
W1.58F4
W2E4
W3E4
W4E4
49
W2EB
W3E8
W4E8
W1.58F16
46
100
200
300
200
400
200
400
600
250
500
750
67
69
LLa models
Accuracy
62
63
64
W2E2
W3E3
W1.88F1.58
57
W1.58E4
58
W2E4
59
W3E4
60
W4E4
W1.58F8
W1.58E16
W2E16
W3E16
W4E16
52
54
55
1000
2000
3000
1000
2000
3000
2000
4000
2000
4000
Effective Quantized Model Size
Effective Quantized Model Size
Effective Quantized Model Size
Effective Quartized Model Size
Figure Trade-off between model and accuracy the optimal embedding width. "WzEy" indicates quantized weights into z-bits and embeddings into y-bits
MobileLLM models
60
55-
W1. 5.88
W1.501
50
W2F4
W2E2
45
WAE4
W4E4
200
400
600
200
400
600
LLate 3 modass
65
65
Acouncy
60-
W1F4
60-
WIFI
58E4
W1.501 50
55
W3E4
W3E3
50
50
1000
2000
3000
4000
1000
3000
Effective Quantized Model
Effactive
Quantized
Size
Figure 12: Trade-off between model size and accuracy for the optimal weight bit width. "WæEy" indicates quantized weights into z-bits and embeddings into y-bits
A.8 Pareto Curve in More Tasks
Furthermore, we present results from a question-answering task, TriviaQA (TQA) Joshi et al. 2017), and reading comprehension benchmark, RACE [Lai et al. 2017). in Figures T3 The findings are consistent across these tasks: 1-bit quantization yields the lowest performance, whereas 1.58-bit, 2-bit, and 3-bit guantization comparable and generally surpass the performance of 4-bit quantization.
18

---

TOA
Raceridt
[c] Raceny
37.5
50
30
WIF4
W1E4
32.5
WIE4
20
30.0
W2F4
35
W7F4
W2E4
27.5
30
25.0
200
400
600
400
600
400
600
Effective Quantized Model
Effective Quantized Model
Effective Quantized Model
Figure 13: Trade-off between model size and accuracy on (a) TQA (b) Racensddl and (c) Racetigh- "WrEy" denotes quantized weights into x-bits and embeddings into y-bits.
(a) LAMBADA
(b) SciQ
0.55
0.50
Accurac
Accuracy
0.40
0.80
0.35
0.75
W2E4
W2E4
0.25
W4E4
0.70
W4E4
100
200
300
400
500
100
200
300
400
500
Effective Quantized Model
Effective Quantized Model
Figure 14: Trade-off between model size and accuracy on (a) LAMBADA (b) SciQ. "WaEy" indicates quantized weights into x- x-bits and embeddings into y-bits.
Additionally, for context-basee word prediction (LAMBADA Paperno et al.) 2016) and multiple- choice science questions (SeiQ [Welbl et al. 2017) in Figrue 14 the results also shows clear trend of 2-bit residing on the Pareto optimal frontier, outperforming 4-bit.
A.9 Limitations
This paper explores the Pareto frontier that characterizes the trade-off between model accuracy and size under quantization. Our findings indicate that lower-bit quantization performance is more sensitive to the fine-tuning data compared to higher-bit quantization. While our model's performance was evaluated in an academic setting, real-world applications may involve diverse training data and weight distributions. Consequently, the optimal bit-width may vary across use cases. ParetoQ provides tool to assess model's performance across different quantization levels. Although we identify 2-bit quantization as promising alternative to 4-bit, the generalizability of bit-width selection should evaluated on case-by-case basis in practical scenarios.
A.10 Broader Impacts
This work presents low-bit quantization for Large Language Models (LLMs). By lowering the precision required for inference, this technique can help reduce energy usage, making a promising strategy for cutting down the computational costs of deploying LLMSs.
19

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable "[No] ", it is perfectly acceptable answer "[No] provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" "we were unable find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased binary way. we acknowledge that the answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendíx. If you answer [Yes] a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer:(Yes
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
20

---

Justification:
See Section A.9
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
See the Sections 2 and 3.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See the experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
21

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will open-source our code and models.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.co son.esiiimsiiioooiccy.c. for more details. public/ Saaiii/gguiee///oeeSuumiimsss
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/ c/nips.cc/public// httpccccpbbblbccc http://////iieee/e//eiiiiii
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
22

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: See the experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
See the experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See the experiments section.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
23

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurIPS Code of
Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Section A.10
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: We have safeguards for responsible release of models that have a high risk for misuse
24

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Data and models used in experiments are porperly cited in the experiment section.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information. and terms of use in the package should be provided. For popular datasets, paperswis thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
We will release the code and models.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
25

---

Answer:
[NA]
Justification:
We didn't use any human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standaadd components.
Please refer to our LLM policy what should or should not be described.
for
26


Reviews:
[{"review_id": "Zyi2tSkBnz", "reviewer": "Reviewer_RkaM", "summary": "ParetoQ presents a unified framework for quantizing at sub-8-bit widths. They analyze the performance of quantized models as a function of dataset, number of parameters, quantization precision, quantization function, and training scheme. They present the right amount of token split for FP training, followed by QAT for the best results across different bitwidths. They also analyze different range clipping methods and show that learning based scale selection outperforms statistics-based methods across bitwidths. They propose Stretched Elastic Quant that balances output quantization levels and delivers superior performance for ternary and two-bit quantization. But by combining different quantization methods for different bitwidths, their framework is able to achieve the best-performing models at all the bitwidths under consideration.", "questions": "1. Are these observations specific to LLMs, or do they generalize to multimodal models?\n2. What is the performance with generative tasks?\n3. Why is there a drop in performance when switching from MobileLLM to Llama in Figure 2", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Their analysis of the performance of quantize models covers a wide range of bitwidths, model sizes, and quantization functions\n2. The results from the FP training and QAT split are extensive and are useful to the broader LLM community.\n3. Their framework is able to achieve the best performance across all the bitwidths and model sizes under consideration.\n4. They show practical benefits of quantization with efficient kernels to achieve speed-up gains.\n\nWeaknesses:\n1. They are able to achieve the best performance by bootstrapping the best methods at different bitwidths; there is no single method that works across different bitwidths\n2. The experiments are focused on discriminative or general language understanding tasks; generative tasks are not included\n3. All the observations are based on empirical results, questioning the generalizability of their framework. For example, based on these methods, what can we say about larger models beyond 8B?\n4. Do the same observations hold for Vision models or multimodal models?", "was_revised": true, "final_justification": "The rebuttal from the authors clears most of my questions. I am keeping my favorable score.", "rebuttal": "We thank the reviewer for the overall positive feedback on our paper, and especially for recognizing our main contributions and novelty. We address the reviewer’s comments and questions as follows:\n\n**[W1: No single method that works across different bitwidths.]**  \nThis observation actually reflects an inherent property of ultra-low-bit quantization, not a weakness of our method. Other literature [1] also mentioned that sub-4-bit quantization regimes are qualitatively different due to their drastically constrained representational capacity. For instance, binary quantization fundamentally collapses the value space to two levels, which makes its optimization dynamics fundamentally distinct from 2 - 4 bits. Likewise, from 2-bit to 3-bit quantization, it also exhibits different trade-offs between dynamic range and quantization error due to the significant increase in the number of quantization levels from 4 levels to 8 levels.  \nParetoQ adopts a unified core framework, utilizing a learnable scaling factor, while explicitly tailoring the quantization grid for each bitwidth because it is provably necessary.   \n[1] QuEST: Stable Training of LLMs with 1-Bit Weights and Activations.\n\n**[W2 & Q2: Generative tasks.]**   \nThanks for the question. In fact, we have already included generative benchmarks in the Appendix. Specifically, TriviaQA (Figure 13) and LAMBADA (Figure 14) are representative generative benchmarks. \n\n**[W3: Generalizability to larger models beyond 8B.]**  \nAs the model size increases, the accuracy gap between the full-precision network and the quantized network becomes smaller, as reflected in the results table of some PTQ works like OmniQuant [2]. This is because larger models have more redundancy and are generally easier to quantize. \nParetoQ quantization framework can be easily extended to different model families and scales. To address your question, we apply ParetoQ quantization on Qwen series, including Qwen-2.5 7B and 14B, to provide a more comprehensive analysis. ParetoQ quantization consistently outperforms previous baselines, such as RTN and GPTQ. In addition, a 2-bit Qwen 14B model achieves slightly higher accuracy than a 4-bit Qwen 7B model. We will include these results in the paper to provide further insight into the effectiveness of ParetoQ quantization.  \n|  | # Bits  | ARC-e | ARC-c | BoolQ | PIQA | SIQA | HellaSwag | OBQA | WinoGrande | Avg. |\n|-|-|-|-|-|-|-|-|-|-|-|\n| Qwen-14B |  |  |  |  |  |  |  |  |  |  |\n| RTN | 2 | 25.8 | 24.1 | 38.8 | 51.6 | 36.5 | 26.0 | 25.4 | 49.0 | 34.6 |\n| GPTQ | 2 | 48.4 | 30.5 | 60.4 | 58.5 | 41.7 | 37.6 | 36.5 | 55.2 | 46.1 |\n| ParetoQ | 2 | 75.5 | 47.8 | 81.5 | 77.4 | 52.3 | 78.1 | 45.8 | 66.4 | **65.6** |\n| RTN | 4 | 75.7 | 45.5 | 78.6 | 81.1 | 48.6 | 78.6 | 38.5 | 64.8 | 63.9 |\n| GPTQ | 4 | 74.0 | 45.4 | 80.2 | 81.6 | 51.6 | 81.0 | 42.9 | 70.2 | 65.8 |\n| ParetoQ | 4 | 73.3 | 49.4 | 77.8 | 81.3 | 54.2 | 79.3 | 53.0 | 73.2 | **67.7** |\n|  |  |  |  |  |  |  |  |  |  |  |\n| Qwen-7B |  |  |  |  |  |  |  |  |  |  |\n| ParetoQ | 4 | 77.2 | 46.4 | 79.7 | 79.5 | 53.6 | 74.1 | 43.3 | 67.5 | 65.1 |\n\n[2] OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models\n\n**[W4 & Q1: Do the same observations hold for Vision models or multimodal models?]**   \nThanks for the question. Our current study focuses on language models, and we have not yet extended QAT experiments to vision or multimodal models. We refrain from making conclusions or even strong hypotheses without empirical evidence, because VLMs differ significantly from LLMs in both architecture and data distributions.   \nExtending QAT to VLMs is not a straightforward adaptation of the LLM pipeline. It would require reconstructing a full SoTA VLM or multimodal training pipeline, including procedures like large-scale image-text pretraining and multimodal finetuning.   \nTherefore, analyzing Pareto-style trade-offs for VLMs would warrant a dedicated separate study. While this is out of scope for the current paper, we view it as an important and promising direction for future work.\n\n**[Q3: Why is there a drop in performance when switching from MobileLLM to Llama in Figure 2?]**  \nThat is because the baseline full-precision MobileLLM‑1B achieves higher accuracy than LLaMA‑1B on the evaluated benchmarks. MobileLLM is specifically optimized for constrained parameter size scenarios, leading to better accuracy at the same parameter scale. In contrast, LLaMA‑1B inherits a scaled-down version of the original LLaMA design, which is not fully optimized for the 1B parameter regime. \n\n*If our response has addressed your concerns, we would greatly appreciate it if you would consider further raising your scores. Thanks!*"}, {"review_id": "7OKElDWKLO", "reviewer": "Reviewer_m9vG", "summary": "The paper provides a comprehensive emprical analysis of relationship of final loss or model performance with combination of model parameters, training data, quantization training regime, quantization preicison and selection of quantization function. The authors propose a quantization framework namely ParetoQ, and achieves improved quantization performance across all bit-widths. ParetoQ is presented as first unified framework for quantization by optimizing training schemes and refining quantization functions for each specific bitwidth.", "questions": "- Can the authors provide a deeper intuition or empirical evidence as to why \"Stretched Elastic Quant\" specifically works better for 1.58-bit and 2-bit compared to other adaptive quantization methods? \n- Are there any new challenges or phenomena that emerge at the scales of >8B params for extremely low-bit quantization?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The unified framework presented in this paper is unique and novel. Thorugh this, the paper addresses a signficant gap in the literature by providing comparisons across a wide range of low bit quantization settings.\n- The paper shows the ParetoQ yields state of the art models at all low bitwidths, surprassing the previous literature across different precision.\n- The detailed analysis of training budget and quantization functions across different bitwidths provides valuable insights that holds significant practial value.\n- The emprical study provides extensive experiments spanning across 125M to 8B model parameters and various quantization strategies. The experimental evaluation is comprehensive and provides a thorough understanding on quantization trade-off in LLMs especially at low bitwidth.\n\nWeaknesses:\n- The study focuses on model scale from 125M to 8B parameter regime so its outcomes may or may not hold for larger scale LLMs. The targeted architectures are also focusing specifically on LLAMA and MobileLLM. It is unclear whether the empirical analysis holds in practise to other architectures.\n- The authors introduced SEQ for low bit scenarios to keep 0s in quantization output levels. It would be good to provide a discussion on this specific and why SEQ is better than LSQ.", "was_revised": true, "final_justification": "My comments have been addressed by the authors and therefore I stick to my current rating of \"Accept\".", "rebuttal": "We thank the reviewer for recognizing our paper’s contributions in addressing a significant gap in the literature by providing comparisons across a wide range of low bit quantization settings. We provide our point-by-point responses to the reviewers’ feedback below.\n\n**[Regarding the generalization to larger models and other model series]**  \nAs the model size increases, the accuracy gap between the full-precision network and the quantized network becomes smaller, as reflected in the results table of some PTQ works like OmniQuant [1]. This is because larger models have more redundancy and are generally easier to quantize.\nParetoQ quantization framework can be easily extended to different model families and scales. To address your question, we apply ParetoQ quantization on Qwen series, including Qwen 2.5 7B and 14B models, to provide a more comprehensive analysis. ParetoQ quantization consistently outperforms previous baselines, such as RTN and GPTQ. In addition, a 2-bit Qwen 14B model achieves slightly higher accuracy than a 4-bit Qwen 7B model. We will include these results in the paper to provide further insight into the effectiveness of ParetoQ quantization.\n|  | # Bits  | ARC-e | ARC-c | BoolQ | PIQA | SIQA | HellaSwag | OBQA | WinoGrande | Avg. |\n|-|-|-|-|-|-|-|-|-|-|-|\n| Qwen-14B |  |  |  |  |  |  |  |  |  |  |\n| RTN | 2 | 25.8 | 24.1 | 38.8 | 51.6 | 36.5 | 26.0 | 25.4 | 49.0 | 34.6 |\n| GPTQ | 2 | 48.4 | 30.5 | 60.4 | 58.5 | 41.7 | 37.6 | 36.5 | 55.2 | 46.1 |\n| ParetoQ | 2 | 75.5 | 47.8 | 81.5 | 77.4 | 52.3 | 78.1 | 45.8 | 66.4 | **65.6** |\n| RTN | 4 | 75.7 | 45.5 | 78.6 | 81.1 | 48.6 | 78.6 | 38.5 | 64.8 | 63.9 |\n| GPTQ | 4 | 74.0 | 45.4 | 80.2 | 81.6 | 51.6 | 81.0 | 42.9 | 70.2 | 65.8 |\n| ParetoQ | 4 | 73.3 | 49.4 | 77.8 | 81.3 | 54.2 | 79.3 | 53.0 | 73.2 | **67.7** |\n|  |  |  |  |  |  |  |  |  |  |  |\n| Qwen-7B |  |  |  |  |  |  |  |  |  |  |\n| ParetoQ | 4 | 77.2 | 46.4 | 79.7 | 79.5 | 53.6 | 74.1 | 43.3 | 67.5 | 65.1 |\n\n[1] OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models\n\n**[Regarding more discussion on why SEQ is better than LSQ]**  \nThanks for the question. In extremely low-bit quantization, the key challenge is to maximize the representational capacity of a very limited number of quantization levels. From an information entropy perspective, a more balanced distribution of quantized values increases the entropy of the quantized output, thereby retaining more information from the original full-precision weights, statistically speaking.\n\n- In the 2-bit case:  \n\nLSQ levels are {−2,−1,0,1}, and SEQ levels are {−1.5,−0.5,0.5,1.5}.  \n\nIn LSQ, explicitly including a zero level causes an imbalance in the distribution of quantization levels, leaving only one level on the positive side. This restricts the representation capability for positive values. It is an especially critical drawback when only 4 levels are available in 2-bit quantization.\n\nIn contrast, SEQ enhances overall information capacity by using symmetric positive and negative levels with equal spacing, which encourages a more uniform level occupancy. A more uniform level distribution yields higher entropy: $H = - \\sum p_i \\log p_i $.  If $p_i$​ is close to uniform, H is maximized. Therefore, SEQ tends to preserve more of the signal diversity under the same bitwidth.\n\n- In the 1.58-bit case:  \n\nLSQ scales weight to [-1,1], and maps [−1.0,−0.5] $\\rightarrow$ -1, [−0.5,0.5] $\\rightarrow$ 0, [0.5,1.0] $\\rightarrow$ 1. Here, the middle bin (−0.5,0.5) has double the width of the side bins, causing more concentration on the 0 level and lowering entropy.  \n\nSEQ scales weight to [-1.5, 1.5], and maps [−1.5,−0.5]→ -1, [−0.5,0.5] → 0, [0.5,1.5] → 1. All bins are equal-width, encouraging a more even probability distribution across levels, which maximizes entropy and better utilizes the limited quantization budget.\n\nWhile both LSQ and SEQ learn the scaling factor and the range changes dynamically during training, the initial quantization grid matters because it defines the early-stage optimization landscape. SEQ starts with a more entropy-preserving quantization grid, making it easier to retain information and stabilize training in extremely low-bit scenarios.  \n\n- In short:\n\nLSQ favors noise suppression by keeping 0 but sacrifices entropy.  \n\nSEQ favors balanced level usage, higher entropy, and better information retention, which is especially beneficial for ultra-low-bit quantization.  \n\n*If our response addressed your question, we would greatly appreciate it if you could consider raising your score even further!*"}, {"review_id": "noSjyg0pSJ", "reviewer": "Reviewer_hvF4", "summary": "The paper introduces, ParetoQ is a framework that compares different AI model quantization methods (1-4 bit) to find the best balance between size and accuracy. It shows that 2-bit and 3-bit quantization offer the optimal trade-off, outperforming previous methods.", "questions": "- Does the 90% observation from Figure 3 holds for bigger models? \n\n- Regarding the observed difference in distribution for 1bit and 2 bit models. Why does finetuning help if the final distribution is different from the initial one?", "limitations": "Limitation section is present. The authors are encouraged to move it to the main paper.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strenths:**\n\n- The paper is well written and easy to follow\n\n- The results obtained using the proposed approach are good, comparing favorable against state-of-the-art\n\n- Presence of large scale experiments\n\n**Weaknesses:**\n\n- Limited technological novelty, i.e. the quantization function and scheme, the training strategy etc., are all adopted from prior works. I do understand however that this is a study mainly and has a slightly different focus.\n\n- Many of the findings are known or straight-forward (e.g. of known: finetuning from FP16 model, higher quantization requires longer training scheduler, sensitivity to the quantization function). Some of this have large bodies of work behind, e.g: what is the best quantization function for low-bit quantization. \n\n- Some important, relevant works that study similar aspects were omitted, e.g:\n\n[1] Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for Quantized LLMs with 100T Training Tokens, Xu Ouyang, Tao Ge, Thomas Hartvigsen, Zhisong Zhang, Haitao Mi, Dong Yu, 2024\n[2] Continual Quantization-Aware Pre-Training: When to transition from 16-bit to 1.58-bit pre-training for BitNet language models?, Jacob Nielsen and Peter Schneider-Kamp and Lukas Galke, 2025\n\n- The problem of optimal quantization was studies under the umbrella of mixed quantization, where the goal is to identify the best configuration with best tradeoffs. How does the approach differ in scope from this works? Some of this work also provide or aim for pareto optimal curves. The authors should seek to analyze and compare with such works. Example:\n\n[3] One-Shot Model for Mixed-Precision Quantization, Ivan Koryakovskiy; Alexandra Yakovleva; Valentin Buchnev; Temur Isaev; Gleb Odinokikh, 2023\n[4] Bit-Mixer: Mixed-precision networks with runtime bit-width selection, Adrian Bulat, Georgios Tzimiropoulos, 2024\n[5] Edge Inference with Fully Differentiable Quantized Mixed Precision Neural Networks, Clemens JS Schaefer, Siddharth Joshi, Shan Li, Raul Blazquez, 2022\n[6] Rethinking differentiable search for mixed-precision neural networks, Zhaowei Cai, Nuno Vasconcelos, 2020\n\n- How does the conclusions change when considering the inference time as opposed to theoretical measures for Fig. 1?\n\n- Unclear how general \"finding 1\" is. How would a different learning scheduler and hyperparameters impact this ratio? As an example, would a higher learning rate for the finetuning part change the conclusions? Does the choice of the quantizer function have any impact on this?\n\n- Regarding \"finetuning benefits across all bit-widths\" where the paper suggests \"pre-trained full-precision models for initialization is a more effective\". This is widely known aspect for quantization in general. It's unclear why the paper presents this as a new observation, instead relevant works in the area should be cited.", "was_revised": true, "final_justification": "The authors have addressed my concerns, hence I have increased my score accordingly.", "rebuttal": "We thank the reviewer for recognizing the strength of our paper (solid experiments and outperforming previous methods) and giving us constructive suggestions on discussing more relevant works.\n\n**[Regarding the novelty]**\n\nThe main novelty of our work lies in establishing a unified framework that systematically covers binary, ternary, 2-bit, 3-bit, and 4-bit quantization—each achieving state-of-the-art performance compared to prior single-bit methods. To the best of our knowledge, no previous work has provided a rigorous, apple-to-apple comparison across all these bitwidths under strictly aligned conditions, employing consistent training schemes and achieved state-of-the-art quantization functions for each bitwidth, while deriving reliable accuracy-efficiency trade-off curves. This unified framework enables a fair and comprehensive evaluation, offering insights that isolated studies focused on individual bitwidths cannot reveal. We appreciate your recognition of the distinct focus of our work and believe this framework provides unique value to the community.\n\n**[Regarding finetuning from FP16 model]**\n\nActually, for extreme low-bit settings such as binary and ternary quantization, whether finetuning is necessary is not settled. The literature shows diverse practices and even contradictory trends. For example, in convolutional networks [7, 8], binary or ternary models are often trained from scratch, while for BERT, approaches like BinaryBERT [9] and TernaryBERT [10] usually fine-tune from a full-precision model. In contrast, recent works on LLMs, such as BitNet [11] and Spectra [12], again trained ternary LLMs directly from scratch, not for training efficiency, but for inference accuracy.\n\nThis lack of consensus can be confusing, as many papers simply adopt one scheme without explicitly justifying the choice, and previous scaling law work [13] do not mention this important choice at all. To provide clearer guidance for practitioners working on extreme low-bit quantization, we believe it is worthwhile to explicitly compare this design choice under a controlled setting, as they directly affect the scaling laws and conclusions, and highlight that all bit-widths, including binary and ternary, can still benefit from finetuning.\n\nAgain we are not claiming finetuning as a major discovery, instead, it builds a solid foundation for our scaling law because it achieves better accuracy than training from scratch.\n\n[7] XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks. Mohammad Rastegari, et al.  \n[8] Bi-Real Net: Enhancing the Performance of 1-bit CNNs. Zechun Liu, et al.  \n[9] BinaryBERT: Pushing the Limit of BERT Quantization. Haoli Bai, et al.  \n[10] TernaryBERT: Distillation-aware Ultra-low Bit BERT. Wei Zhang, et al.  \n[11] BitNet: Scaling 1-bit Transformers for Large Language Models. Hongyu Wang, et al.  \n[12] Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scale. Ayush Kaushal, et al.  \n[13] Scaling Laws for Precision. Tanishq Kumar, et al.  \n\n**[Regarding higher quantization requires longer training scheduler]**\n\nWe assume you meant *lower-bit* quantization needs a longer training schedule. We mentioned this because it is a building block in our scaling-law derivation. To keep the scaling law interpretable and practically useful, we clarify why specific training choices are necessary instead of treating them as ad hoc tricks.\n\n**[Regarding sensitivity to the quantization function]**\n\nThough important, prior scaling law studies (e.g., [1], [9]) didn’t consider quantization function sensitivity in their analysis. Other previous studies focused on single bit quantization lack cross-bit comparisons. Nevertheless, our ParetoQ achieves higher accuracy than methods specifically designed for a single bit setting.\n\n**[Some important, relevant works]**\n\nThanks for mentioning these relevant works. \n\n[1] primarily investigates scaling laws for 2-, 3-, and 4-bit quantization, focusing on the interactions among model parameters (N), training tokens (D), and precision (P). It omits the impact of quantizer designs, which we and the reviewer both agree to be a critical aspect in low-bit quantization. And it didn’t include binary or ternary cases. \n\n [2] investigates a training strategy for ternary pre-training specifically transitioning from 16-bit to ternary representations for BitNet-style language models. It is a study focused exclusively on ternary quantization.\n\nWe will include a discussion of these works in the Related Work section and highlight the conceptual differences.\n\n\n**[How does the approach differ in scope from mixed precision quantization?]**\n\nWe need to clarify that our work is not in the regime of mixed precision, and they are fundamentally different in both goal and method.\n\nMixed-precision optimize bit allocation across layers within a fixed model using techniques like sensitivity analysis or hypernetworks. In contrast, we investigate a global scaling tradeoff: given a fixed compute or memory budget, is it better to use a smaller model with higher precision or a larger model with lower precision? Thus, we are not searching for the best bit allocation within a single architecture but analyzing how model size and uniform precision jointly affect performance.\n\nMoreover, these works [3-6] focus on ConvNets, so experimental comparison is not applicable. We will add a related work section to compare conceptual differences.\n\n**[How does the conclusions change when considering the inference time as opposed to theoretical measures for Fig. 1]**\n\nThanks for the question. Besides accuracy-size trade-off, in Figure 7(c), we show the accuracy–latency trade-off. With properly optimized kernel implementations, 2-bit could be a promising alternative to 4-bit quantization when considering the accuracy-latency trade-off as well.\n\n**[How would different learning rates/schedulers or quantizers impact \"finding 1\"?]**\n\nWe adopt the widely used cosine learning rate scheduler. We also compared cosine and linear schedulers, but found no significant difference in performance. We also experimented with different learning rates during the setup phase and found that the chosen configuration yields the best overall accuracy. To obtain a meaningful performance curve, we therefore used this optimal learning rate—otherwise, curves with suboptimal learning rates (and lower accuracy) would not be informative. We also observed that varying the learning rate within a reasonable range (0.5×–2×) does not significantly affect final accuracy, indicating robustness to learning rate choices. \n\nThis conclusion about an effective fine-tuning ratio also holds when we switch to a different but reasonable quantization function. For example, in Figure 3, the 2-bit quantization results are obtained with SEQ; when we replace SEQ with LSQ, the same trend persists. The model achieves better performance when the FP training ratio is 90%, outperforming the 50% ratio by 0.7 points in zero-shot commonsense accuracy and exceeding both 0% and 100% by 4.1 and 10.7 points, respectively.\n\nIt is important to clarify that the key takeaway is to allocate the majority of the training budget to full-precision (FP) training and approximately 10% to QAT, as we mentioned in finding 1. The 90% ratio serves as a guideline rather than a strict optimum—values like 85% or 95% could still yield comparable or even slightly better results.\n\n**[Does the 90% observation from Figure 3 holds for bigger models?]**\n\nAs indicated in Figure 2, where the quantized model converges with a relatively small amount of fine-tuning tokens (<100B). For reference, the total training budget for MobileLLM is 1T tokens, and larger models like LLaMA use even more for pretraining.\n\nTherefore, the conclusion remains valid: optimal performance is achieved by dedicating the majority of the training budget to full-precision (FP) training and allocating approximately 10% to QAT.\n\nFigure 3 provides a rough range and qualitative overview, while Figure 2 presents a more detailed, model-specific, and bit-width-specific ablation. Moreover, Figure 2 includes results across different model sizes (125M, 350M, 600M, 1B, 1.5B, and 3B), supporting the generality of this trend and gives more quantitative guidelines.\n\n**[Why does finetuning help if the final distribution of 1-bit or 2-bit models differs from the initialization?]**\n\nQuantized networks, especially at very low bit-widths, operate on discretized values, which makes their loss landscapes much more rugged [14]. This increases the likelihood of getting trapped in suboptimal local minima. In contrast, full-precision networks have a smoother and more continuous loss landscape, allowing for better optimization. Fine-tuning in full precision helps position the model parameters in a more favorable region of the loss landscape before applying quantization, effectively placing the quantized model in a “better neighborhood” for further exploration.\n\nFrom Figure 5, we see that the distance between a full-precision model and its fine-tuned quantized counterpart ( $ ||W_{finetune} -W_{fp}||/||W_{fp}|| $ ) is around 0.4 on average for binary, ternary and 2-bit quantization, which is relatively small compared to the distance between the full-precision and its random initialization ( $ ||W_{fp} -W_{random}||/||W_{fp}|| $ ) which is $\\sim 1.0$. This suggests that starting from a well-trained full-precision model provides a strong initialization that helps the quantized model reach better local minima, even if its final distribution differs from the initial one.\n\n[14] How Do Adam and Training Strategies Help BNNs Optimization? Zechun Liu, et al.\n\n*We're glad to see this work sparking so many interesting discussions, which suggests it's a promising and worthwhile direction. If our response addressed your question, we’d greatly appreciate it if you could consider further raising your scores! Thanks!*"}, {"review_id": "afKlN2HSfx", "reviewer": "Reviewer_JKLc", "summary": "This paper presents ParetoQ, a novel framework that systematically compares various quantization functions specifically tailored for extremely low-bit quantization in large language models (LLMs).\n\nSpecifically, the article provides insights and practical quantization methods for binary, ternary, and 2/3/4-bit quantization to achieve the best accuracy-efficiency tradeoff, and also provides a series of LLaMA-3 efficient quantization schemes based on ParetoQ.\n\nExtensive experiments verify the effectiveness of the proposed quantization strategies.", "questions": "Please refer to weakness.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\\\n1.\tThe paper proposes the ParetoQ framework, which enables a systematic comparison of sub-4-bit quantization methods and provides new ideas for the design and selection of quantization functions. The paper proposes the ParetoQ framework, which enables a systematic comparison of sub-4-bit quantization methods and provides new ideas for the design and selection of quantization functions.\\\n2.\tThe paper is well-written and easy to follow.\\\n3.\tExperiments show that ParetoQ outperforms existing quantization methods and is more efficient in hardware adaptability.\\\n\n\nWeakness:\\\n1.\tAs mentioned in A.9 Limitations, although the paper explores the advantages of low-bit quantization, in practical applications, the diversity of different tasks and datasets may affect the generalizability of its results.\\\n2.\tThe design of the method relies on the Pareto curve, which is the trade-off between accuracy and speed. However, if the metric for measuring the model's ability is not accuracy, but some other task metric like FID in diffusion model, is the method still applicable? If possible, it is recommended to add such a discussion.", "was_revised": true, "final_justification": "Thanks for the reply. I will keep my score.", "rebuttal": "We thank the reviewer for  acknowledging ParetoQ is a novel framework that systematically compares various quantization functions specifically tailored for extremely low-bit quantization in LLMs. Below, we respond to the reviewers’ comments and questions in detail.\n\n**[Regarding generalizability of the results]**\n\nThanks for the question. What we mean in A.9 Limitations is that the optimal bit may vary in real scenarios, but our method is generalizable and provides a unified framework that allows practitioners to fairly compare different bit-width choices in their own setting and identify what works best for their specific use case.\n\n\n**[Regarding extending to other task metrics beyond accuracy, such as FID in diffusion models]**\n\nThank you for this thoughtful suggestion. We agree that extending the discussion to metrics beyond accuracy would be valuable. Indeed, we have already included generative benchmarks in the Appendix. Specifically, for the *TriviaQA* benchmark (Figure 13), we report the F1 score instead of accuracy.\n\nHowever, expanding the scope from LLMs to diffusion models would require solid supporting experiments, which entails non-trivial efforts (e.g., rebuilding the entire SoTA training pipeline from data preparation to fine-tuning). Without such experiments, we feel it would be irresponsible to make speculative claims. Nevertheless we will mention it as a promising direction for future work, highlighting the potential applicability of ParetoQ to other domains and metrics.\n\n*If our response has clarified your question, we’d be truly grateful if you could consider further raising your score. Thanks!*"}]

Author rebuttal:

Meta-review: Tis paper presents ParetoQ, a unified scaling law framework that considers a flexible schedule jointly leveraging full-precision pre-training and quantization-aware training (QAT) to train and quantize large language models (LLMs) with different bit-width settings (such as binary, ternary, and 2/3/4-bit quantization). By the proposed method with refining quantization functions, the authors claim that the best accuracy and training-efficiency tradeoff could be achieved. 

The paper was initially scored (4,5,4,4) by four reviewers, who are in agreement that the core contribution of this work is technical sound. Their initial concerns are about: 1) all the observations are based on empirical results, limited to tested LLMs (small Llama and MobileLLM models only); 2) the study focuses on a relatively narrow model size ranging from 125M to 8B parameters; 3) many of the observations are known or straight-forward; 4) technical novelty is not strong.

The authors provided detailed responses to these concerns, which were well recognized by all of four reviewers. Finally, all reviewers maintained their positive scores, and the paper got the scores of (4,5,5,4). The AC read the paper, the reviews, the rebuttal, the author-reviewer discussion and the reviewers' feedback, and generally agree with the reviewers' assessment. As the proposed method cannot be easily adopted in practice due to its requirement of huge computational resources (compared to low-cost post-training quantization (PTQ) methods for LLMs with different model architectures, this work needs full-precision pre-training and QAT fine-tuning together to attain quantization goal and only limited LLM structures with small model size are studied in experiments), I recommend to accept this paper as a poster. The authors are encouraged to carefully consider the reviewers' comments/suggestions and their rebuttal in the final paper revision.
Decision: Accept (poster)