Towards Quantization-Aware Training for Ultra-Low-Bit Reasoning LLMs
Authors: Yasuyuki Okoshi
Hikari Otsuka
Daichi Fujiki
Masato Motomura
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
Quantization
Primary area: foundation or frontier models, including LLMs
TL;DR: Two-stage QAT for reasoning LLMs: mixed-domain calibration preserves abilities and teacher-guided loss restores reasoning, boosting performance of ultra-low-bit  reasoning LLMs

Abstract: Large language models (LLMs) have achieved remarkable performance across diverse reasoning tasks, yet their deployment is hindered by prohibitive computational and memory costs. Quantization-aware training (QAT) enables ultra-low-bit compression (<4 bits per weight), but existing QAT methods often degrade reasoning capability, partly because complex knowledge structures are introduced during the post-training process in LLMs. In this paper, through a systematic investigation of how quantization affects different data domains, we find that its impact on pre-training and reasoning capabilities differs. Building on this insight, we propose a novel two-stage QAT pipeline specifically designed for reasoning LLMs. In the first stage, we quantize the model using mixed-domain calibration data to preserve essential capabilities across domains; in the second stage, we fine-tune the quantized model with a teacher-guided reward-rectification loss to restore reasoning capability. We first demonstrate that mixed-domain calibration outperforms single-domain calibration by up to 2.74% improvement on average over six tasks, including reasoning and pre-trained tasks. Following experiments on five reasoning benchmarks show that our 2-bit-quantized Qwen3-8B outperforms post-training quantization (PTQ) baselines by 50.45% on average. Moreover, compared to ultra-low-bit-specialized models such as BitNet-2B4T, our pipeline achieves approximately 2\% higher mathematical-reasoning accuracy with fewer than 1B tokens. Code is available: https://github.com/yasu0001/ReasoningQAT

Published as a conference paper at ICLR 2026
TOWARDS DUANTIZATION-AWAAT TRAINING FOR ULTRA-LOW-BIL REASONING LLMs
Yasuyuki Okoshi; Hikari Otsuka, Daichi Fujiki, Masato Motomura
AT Computing Research Unit Institute of Science Tokyo
ABSTRACT
Large language models (LLMs) have achieved remarkable performance across diverse reasoning tasks, yet their deployment is hindered by prohibitive com- putational and memory costs. Quantization-aware training (QAT) enables ultra- low-bit compression (< 4 bits per weight), but existing QAT methods often de- grade reasoning capability, partly because complex knowledge structures are in- troduced during the post-training process in LLMs. In this paper, through a systematic investigation of how quantization affects different data domains, we find that its impact on pre-training and reasoning capabilities differs. Build- ing on this insight, we propose a novel two-stage QAT pipeline specifically de- signed for reasoning LLMs. In the first stage, we quantize the model using mixed-domain calibration data to preserve essential capabilities across domains; in the second stage, we fine-tune the quantized model with teacher- guided reward-rectffication loss to restore reasoning capability. We first demonstrate that mixed-domain calibration outperforms single-domain calibration at maxi- mum 2.74% improvement on average over six tasks including reasoning and pre- trained tasks. Following experiments on five reasoning benchmarks show that our 2-bit-quantized Qwen3-8B outperforms post-training quantization (PTQ) base- lines by 50.45% on average. Moreover, compared to ultra-low-bit-ppp -itsb---ppciillzedddd models such BitNet-2B4T, our pipeline achieves approximately 2% higher mathemattical-rrasoning accuracy with fewer than 1B tokens. Code is available: htttasss//gtttnac.ccomm/sssss///
INTRODUCTION
Large language models (LLMs) have demonstrated remarkable performance across various tasks, including mathematics (Shao et al. 2024, Wang et al., Yang al. 2024), coding (Hui et al. 2024] Roziere et al., (2023), and snowledge-intens/i
question answering (Lu et al.. (022). However, their prohibitive computational and memory requirements pose significant challenges for deployment in inference. One promising direction for reducing these inference costs is weight quantization (Zhou) ct al. 2024; Lang et al. 2024). which employs low-bit widths for model weights. Among var- ious quantization methods, quantizaation-aware training (QAT), which fine-tunes the model with quantized weights, is especially effective for ultra-low-br widths (< 4 bits) (Wang et al., [2023] 2024]. enabling to deploy lightweight and fast LLMs. For example, 2-bit quantized LLMs via QAT can achieve performance comparable to their pre-quantized fp16
Ma et al. 2024 Xu et al.
counterparts (Ma al. 2024 Kaushal et al. 2024 Liu et (205cc).
Despite the promising performance of QAT, existing approaches suffer from severe performance degradation reasoning benchmarks (Du et al.) 2024), such as mathematics, and instruction- following tasks (Lee et al. (0025). We hypothesize that this degradation arises from the complex knowledge structures introduced during post-training. The post-training process is an extensive pro- cess that includes supervised fine-tuning (Wei et al. 2021) and preference optimization (Ouyang al. 2022 Rafailov et al., 2023), introducing new reasoning capabilities with existing common- sense knowledge acquired during pre- pre-training. While it creates heterogeneous knowledge structures, remains unclear how quantization affects the model's performance on reasoning capabilities and pre-trained commonsense knowledge.
"Equal contribution

---

Published as a conference paper at ICLR 2026
Existing QAT Pipeline for Instruction- Tuned LLMs
Calibration
stumped...
Pro Training Data (Sobaet)
INTZ
-1
(Ultra Reasoning
Supervised Fine. Tuning (2nd stage)
stage)
0.1
Quantized
Fine Turred (Quanitze))
Reasoning trosels trote. Dvoara iiac La the mirk stea step-
1.3
-.2
Our Proposed Pipeline
FP16
Calibration Data
Calibration Data
Original Weights
Instraction- Tuned LLM (Fulli Precision)
Inference
Reasoning
(Subued)
(Coda, Math etc.)
-1
Quantization tion stage)
RL Inspiede Fine- Tuning (2nd
INT2
Quantizedd LLM (Ultra Low
Quantined
Tuned (Ovamtioo))
String Reasoning Abillty
Figure 1: Comparison of the existing QAT pipeline with the proposed pipeline.
To address this gap, we conduct a systematic investigation of how quantization impacts different knowledge domains in post-training LLMs. Our analysis reveals that quantization creates inherent trade-offs between commonsense knowledge preservation and reasoning capability retention, where different domains exhibit varying sensitivity to quantization. Specifically, while performance on commonsense knowledge benchmarks remains relatively stable even with quantization using out- of-domain data, reasoning capabilities exhibit significant sensitivity to quantization data, suggesting that different domains have distinct requirements for effective quantization.
Based on this analysis, we introduce quantization framework specifically designed for post- trained LLMs that address diverse knowledge domains through novel two-stage pipeline. Following our observation, our quantization framework is designed to dedicate computational resources to maintain reasoning capability, with minimal efforts preserve general knowledge. Specifically, the first stage carries out block-wise guantization with mixed-domain calibration. This -omain calibration preserves essential reasoning capabilities that difficult to restore, while also maintaining com- monsense knowledge. Subsequently, we perform end-to-end fine-tuning with reinforcement learn- ing inspired objectives enhance reasoning capability. This unified framework enables extremely low-bit quantization post-trained LLMs with minimal reasoning performance degradation.
Extensive experiments five reasoning benchmarks demonstrate the effectiveness of our approach. Our method achieves significant improvements over existing post-training quantization methods for reasoning I.LMs. Specifically,
2-bit quantized Qwen3-8B outperforms other quantization methods by 50. 50.45% on average. Notably, even when compared to specialized ternary LLMs like BitNet-2B4T, our 2-bit model with 1.7B parameters demonstrates superior mathematical reasoning performance with substantially reduced training costs achieving 2.5% improvement using fewer than IB training tokens.
Our contributions can be summarized as follows:
We empirically demonstrate how quantization differently affects commonsense knowledge ac- quired during pre-training and reasoning capabilities developed in post-training. Our results highlight the importance of designing mixed calibration data to effectively preserve both of them.
We propose two-stage quantization pipeline for post-trained LLMs that combines mixed-domain calibration and RL-inspired fine-tuning preserve reasoning capabilities while achieving ex- tremely low-bit quantization.
We demonstrate that approach achieves state-of-the-art (SOTA) performance multiple reasoning benchmarks with both 2-bit and 3-bit quantization.
2 PRELIMINARIES
This section outlines the weight quantization and recent quantization-aware training (QAT) pipeline.
2

---

Published as a conference paper at ICLR 2026
Dataset Domain
Lefi Swil,, NARI-520 radee
Right Hiah
Pre-training
Code
Math
Science
- Invv
Use Code Berch Gogs Dlamend
40
Accuracy
Reasoning Data Ratio (%)
-20
(a) Performance for general (green and purple e) and mathematical (orange e, blue m. and red tasks on the reasoning data ratio.
(h) t-SNE visualization of the activations in the 14th Transformer block, where colors denote dataset mains.
Figure 2: Impact of data-domain composition on Qwen3-1.7B. In the ultra-low-bii quantized model, replacing a part of the calibration data drawn from the pre-training data (FineWeb-Eddh) with reason- ing data OOpenThoughee accuracy (left). In the full-precision model, t-SNE of the model's activations shows tight clusters for pre-training inputs but wide dispersion for reasoning inputs (right).
1.2M) leaves general-task accuracy unchanged while improving reasoning
Weight Quantization:
Weight quantization maps the model weights to low-bit width counter- parts. Given a full-precision weight w E IR, we obtain its dequantized approximation w/ via
g clamp (Lw/s] /s]
0, 0
w == z),
where s 0 the scale factor, 2 E F is the zero point, and N the target bit width. 1-] represents the nearest integer function (i.e., the round function), and clamp(ær, a, b) clamps input to the interval (a, bj. Since the scale s and zero point are shared across groups of weights (e.g., an entire matrix, channel, stored once per group, achieving a low-bit width per-weight representation.
block), each weight is represented only by an N-bit code with and
Weight quantization approaches are mainly categorized into two strategies: 1) post-training quanti- zation (PTQ), which converts pre-trained model weights to low-bit widths without retraining: and 2) quantization aware training (QAT), which quantizes and fine-tunes the weights simultancously. PTQ can quickly quantize weights with small or even without calibration data, while it struggles with ultra-low-bit quantization. In contrast, QAT can flexibly fine-tune the full-precision weights w, scaling factor s, and zero point 2, achieving performance comparable to the full-precision model in ultra-low-bit scenarios.
Quantization-Awarr Training Pipeline: As illustrated in the top panel of Figure 1 existing QAT pipelines generally comprise two stages: 1) an initial quantization stage; and 2) fine-tuning stage. The first stage initializes the quantized weights that serve the starting point for the sub- sequent stage. Some methods omit this step, whereas the latest state-of-the-art (SOTA) QAT ap- proaches (Chen et al. 2024 Du al. 2024) have demonstrated that quantizing weights using subset of the pre- traimignn dataset as calibration data enables stable fine-tuning during the subsequent stage. In the second stage, weights quantized in the first stage are fine-tuned by minimizing training objective. All parameters (i.e., weights, scale, zero point) or some of them are fine-tuned, and this paper fine-tunes only the scale, following one of the SOTA QAT approaches. EfficientQAT.
Also, the training objective is typically either a self-supervised pre-training loss (Liu et al. 2025c
Chen et al. 2024 or knowledge-distillan distillation loss (Du et al. 2024
Lee et al. 2025).
3 REASONING-ORIETTE TWO-STAGE QUANTIZATION AWARE TRAINING
This section proposes novel QAT pipeline that enables the preservation of reasoning capabilities after ultra-low-bit quantization. We first analyze the impact of quantization on various knowledge domains, and based on these findings, we introduce the reaeening-oriented QAT pipeline.
3

---

Published as a conference paper at ICLR 2026
3.1 QUANTIZATION IMPACTS ACROSS KNOWLEDGE DOMAINS
This section analyzes how domain selection for calibration data affects the overall model per- formance. Existing quantization approaches mainly perform quantization with either pre-training data (Liu et al.] [025c] Chen et al.] 2024), or domain-speciff data, such as mathematics (Liu et al., 2025a). While previous work has selected calibration data tailored specific target tasks, the cross- task implications of such task-specific calibration choices remain largely unexplored.
To investigate the effect of domain selection for calibration data, we analyze the impact of select- ing different calibration datasets performance across multiple tasks and knowledge domains. As shown in the results of Qwen3-1.7B quantized 3-bits by EfficientQAT (Chen et al. 2024) (Fig- ure 2ap. the tasks can be broadly grouped into two trends: 1) tasks for which performance improves as the amount of reasoning data increases; and 2) tasks for which performance remains almost constant regardless of the amount of reasoning data. Notably, all tasks in the first category rep- resented in the reasoning dataset, i.e., code (blue), mathematical tasks (orange), scientific questions and answers (red). These results indicate that reasoning data tends to suffer from domain shift, while tasks related to commonsense knowledg are less sensitive to calibration datasets. On the other hand, common tasks also demonstrate performance degradation when calibrated with pure reasoning data, suggesting the importance of dataset diversity even for tasks that appear less sensitive to calibration choices.
These distinct trends happen as the intermediate distributions between pre-trained data and reasoning data differ, as shown in Figure [b The distributional mismatch leads suboptimal quantization performance when calibrating on single-domain data, resulting in higher quantization errors for tasks that require domain- specifc representations.
3.2 PROPOSED METHOD
We now introduce the novel QAT pipeline for ultra-low-bit reasoning LLMs.
Knowledge Domain Selection in Calibration data: We first focus on the mixing ratio of the knowledge domain in calibration data. The results in Section 3.J illustrate the importance of select- ing appropriate calibration data when a QAT pipeline applied to post-trained LLMs. In particular, it is important to mix pre-training data and reasoning data in an appropriate ratio. Building on these findings, we propose using novel calibration data in the first stage of the QAT pipeline. This data is composed of 80% masoninggfocussee data and 20% pre-training data, designed to bias the calibration process toward reasoning while retaining coverage of pre-traie training distributions.
We secondly aim at the fine-tuning stage. Quantization of the first stage using proposed calibration data mixed with pre-training data preserves the fundamental capabilities of LLM, enabling us to focus on enhancing reasoning capabilities during the fine-tuning stage. A straightforward approach to enhance reasoning ability to perform supervised Ane-ttunig using reasoning data, but such training does not effectively generalize into reasoning data (Chu al. 2025). Employing reinforcement learning could improve generalization on reasoning data, but online text generation incurs auto-regressive text generations, resulting in huge training overhead. To balance training efficiency and generalization unseen data, we employ reweighted rectification (Wu et al. (2225) for supervised fine-tuning to make the objective function reinforcement-like.
Supervised Fine-Tuning With Reward Rectification Loss:
Reward rectification is scaling factors for the loss function in supervised fine-tuning. Given the datasets D {z, y²} and the supervised fine-tuning loss (SFr(O), reward rectification loss C(0) dynamically reweights the supervised loss as follows:
((Ø)= = (SFr(0) sg(1/w))
where w is the dynamic reweighting factor and sg(-) denotes the stop-gradient operator.
This formulation can be viewed as bridging supervised fine-tuning and reinforcement learning. In particular, choosing w = 1/To(y æ) yields gradient equivalent to an on-policy policy-gradient
update with the reward function:
r(z,v) = 1ly 10
4

---

Published as a conference paper at ICLR 2026
where (y 2) is the model's conditional probability of generating output y given input under parameters 0. This dynamic re- re-weighting can avoid cee-connentration on low-probalee low-probability reference tokens, improving generalization despite not using additional sampling reward func- tions.
While the original reward rectification uses the student model's own probability To | x)) for reweighting, in the QAT, the quantized model's distribution becomes less reliable due to precision loss. Using the quantized model's own probabilities for reweighting could amplify these errors.
Therefore, we leverage the teacher model's probability ( (ær) as a more reliable reference for the reweighting factor. This teacher-guided approach ensures that the reweighting process is based on the target distribution we aim to recover, rather than the potentially corrupted distribution of the quantized model.
Thus, we introduce leacher-guided reward rectification loss (()), where the teacher model TT con- trols the scale of supervised loss function. Given the teacher probability with labeled data Iy (y" Ix), teacher guided reward rectification loss can be represented as:
(0) (0) sg ( [z)).
Intuitively, this formulation represents that the supervised loss values are amplified when the prob- ability of the quantized model for the label smaller than that of the teacher probability. When the distribution of the quantized model becomes close to the original distribution, this scaling factor acts as the original reward rectification.
To align the overall probabilistic distribution of the quantized model with original LLMs, we further introduce an additional KI. divergence loss. Finally, our training loss function can be represented as:
[()) = aL, DDe.. (
where DKL (TT((I=)] (+1x)) E, π log is the KL divergence between the fp16 model and the quantized model, and a, 3 is hyperparameters that control the effects of teacher- guided reward rectification loss and kl divergence loss.
(r(w/ *s(ula)
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Training: We conduct experiments Qwen3 instruction-tunee models (Yang et al. 2025). For
block-wise calibration, we use a total 4, 096 samples with context length of 2, 048. Calibration datasets consist of 80% of sequences sampled from OpenThoughts-11 1.2M (Guha et al. 2025) and the remaining 20% sampled from FineWeb-BE Web-Edu (Lozhkov et al. 2024). We use different learning rates for quantization parameters (1c-4) and weight parameters (lc-5). For 2-bit quantization, we use larger learning rate of 2e-5 for weights.
During supervised fine-tuning, models are with 32, 768 samples from OpenThoughts 1.2M. We optimize all trainable parameters with the same learning rate. The learning rate for 3-bit quantization 1e-6, while we use larger learning rate for 2-bit quantization, 5e-6 for the 1.7B parameter, and le-4 for other parameters. We use the AdamW optimizers (Loshchilov & Hutter 2019) with the cosine annealing learning rate decay (Loshchilov
Hutter] (017). Models are fine-tuned with batch size of 64 and one epoch for 3-bit models, except for Qwen 1.7B at 2-bit which uses 3 epochs. We filter out the top-20 probabilities for the KL loss. We set a 0.2 and 3 = 1.0 in Equation (T) unless explicitly stated otherwise.
Evaluation: We evaluate the zero-shot accuracy on five benchmarks including We evaluate the zero-shot accuracy five benchmarks, including MATH-500 (Lightman et al.) 2023), Live Code Bench (White et al. 2024). MMLU-Redux (Gema et al.] 2024). GPQA- Diamond (Rein et al. 2024]. and IFEval (Zhou al. 2023), using the evalscope TTam 2024). These tasks evaluated in open-ended text generation. We use token-level sampling, whose tokens are sampled from the top 20 highest tokens with temperature of 0.6. We basically maximum sequence length of
5

---

Published as a conference paper at ICLR 2026
Table 1: Accuracy comparison for different calibration data on 6 benchmarks. Higher values better. We define the group size as 128. Mixed data contains 80% of reasoning data and 20% of pre-training data.
Reasoning Tasks
Pre-trained Tasks
Live
Model (Qwen3)
Bit Width
Dataset Type
MATH-
GPQA- Diamond
MMLU- Redux
Code
CSR
IFEVAL
Avg.
500
Bench
Pre-training
59.53
16.36
25.42
64.16
57.96
58.60
47.01
W3
Reasoning
81.33
29.35
26.60
63.58
55.82
59.52
52.70
Mixed
80.20
29.32
25.42
65.76
56.71
59.70
52.85
1.7B
Pre-training
0.13
0.00
0.00
0.00
50.75
13.86
10.79
w2
Reasoning
20.60
0.09
7.58
19.93
44.77
25.51
19.75
Mixed
18.68
0.47
7.58
28.92
49.11
24.58
21.56
Pre-training
81.80
35.42
41,92
77.98
63.94
71.16
62.04
w3
Reasoning
90.90
46.89
42.76
78.55
63.71
71.72
65.76
Mixed
90.70
46.85
45.45
78.91
63.97
74.86
66.79
4B
Pre-training
2,73
0.00
6.23
26.89
59.34
17.74
18.82
W2
Reasoning
33.80
5.78
11.62
46,26
53.41
32.90
Mixed
22.60
5.12
14.14
51.20
55.89
31.05
Pre-training
87.00
37.25
41.66
82.60
69.19
76.34
65.67
w3
Reasoning
91.80
53.84
51.52
81.75
68.14
75.79
70.47
Mixed
92.40
51.75
48.99
83.33
69.16
81.15
71.13
8B
Pre-training
5.33
0.28
4.55
41.72
63.35
19.41
22.44
w2
Reasoning
42.27
8.15
10.44
51.40
55.05
35.86
33,86
Mixed
40.00
7.30
11.11
57.11
60.81
43.25
36.60
32K for all benchmarks. However, we reduce the maximum sequence length to SK on the lower- performance models to avoid excessive text generation due to the absence of stop token. All evaluations are conducted three times, and we report the average accuracy.
We compare method with two PTQ quantiza- tion baselines, GPTQ (Frantar et al., 2022) and AWQ (Lin et al.] 2024), both of which are evaluated on reasoning benchmarks (Liu strategy as conducted by Liu et al. (2025a). Specifically, we perform quantization using 128 sam- ples from NuminaMath (LI et al. 2024). We reproduce these quantized models locally, except for 3 and 4 bits AWQ quantization for Qwen-8B, as reproduced performance is much inferior to the
Quantization baselines post-trained LLMs:
[025a To quantize these baselines, we follow similar
performance claimed in the paper (Liu et al. 2025a).
4.2 DATASET EFFECTS ON OVERALL PERFORMANCE
This section evaluates our proposed calibration datasets against single-domain calibrations using either pre-training data or reasoning data. We evaluate five benchmarks described in Section 4.1 with the additional commonsense reasoning tasks (CSR) evaluate general knowledge of quan- tized models. The CSR includes five subset tasks: ARC-e, ARC-c (Clark et al. 2018), PIQA (Bisk al. 2020), HellaSwag (Zellers al. 2019), and WinoGrande (aaaauuci al. (202T) Table demonstrates that mixed domain calibration outperforms single-domain calibration across various parameters in both 2-bit and 3-bit quantization settings. Compared with pre-training datasets, its performance improvements on reasoning benchmarks, including mathematical and coding tasks, are particularly notable. In addition. mixed data achieves comparable performance to pre-trained data and other benchmarks in reasoning tasks. When compared with reasoning-onl datasets, the per- formance of mixed datasets on reasoning benchmarks is close, while performance in pre-trained domain benchmarks tends to be superior: These results suggest that including small portion of pre-training data can effective in maintaining commonsense knowledge. resulting in better gen- eralization results across wide range of tasks.
4.3 COMPARISON WITH PRIOR QUANTIZATION APPROACHES FOR REASONING
This section compares our proposal with other quantization methods, including GPTQ and AWQ. Table shows that proposal significantly outperforms existing quantization approaches in both

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of quantization methods and bit-widths on Qwen3 models. Values % (higher is better). We define the group size 128.
Settings
Benchmarks (%)
Avg-
Model
Method
Bit Width(W/A
MATH500
LiveCodeBencch
MML AMLUU-Redux
GPQA-Diamond Diamond
[FEval
Avg.
Qwen3-1.7B
(Baseline)
bíoou116
53.6
74.7
GPTQ
4/16
86.3
36,9
70.2
34.3
66.2
58.8
AWQ
4/16
87.4
44.4
71.6
35.9
65.3
60.9
GPTQ
3/16
58.2
0.0
42.4
9.3
31.8
28.3
AWQ
3/16
58.0
5.8
53.5
17.5
47.7
Proposal
3/16
82.7
33.0
67.7
31.7
61.0
55.2
GPTQ
2/16
2.1
0.0
5,8
4.7
8.5
4.2
AWQ
2/16
27.3
8.1
9.5
Proposal
2/16
48.6
6.5
40.1
14.5
32.2
28.4
Qwen3-4B
(Baseline)
bflout1
93.6
71.2
84.3
83.6
76.8
GPTQ
4/16
93.4
66.2
82.1
50.7
81.2
74.7
AWQ
4/16
93.0
65.7
83.0
50.2
81.0
75.6
GPTQ
3/16
85.0
21.2
65.6
24.9
54.7
50.3
AWQ
3/16
88.3
37.2
74.2
71.9
61.1
Proposal
3/16
89.5
S0.2
79.8
46.8
75.3
68.3
GPTQ
2/16
3.7
0.0
7.5
8.9
8.5
5.7
AWQ
2.4
Proposal
2/16
77.1
19.5
61.7
26.9
50.1
47.1
Qwen3-8B
(Bascline)
bíloat16
94.0
73.0
87.3
86.5
80.5
GPTQ
4/16
94.6
69.6
86.8
58.1
86.9
79.2
AWQ
4/16
97.0
54.7
N/A
59.6
N/A
N/A
GPTQ
3/16
92.1
39.4
79.0
46.8
76.9
66.8
AWQ
3/16
92.9
35.3
N/A
46.8
N/A
N/A
Proposul
3/16
91.5
60.0
84.5
47.5
78.8
72.5
GPTQ
2/16
2.8
0.0
6.4
5.2
8.7
4.6
AWQ
2/16
3.9
10.2
4.0
Proposal
2/16
80.4
28.5
72.6
34.5
59.3
55.1
2-bit and 3-bit quantization settings. The performance improvements are particularly notable 2- bit quantization settings. While the performance of GPTQ AWQ quantized models is extremely low, our quantized models not only achieve solid performance but also exhibit steadily increasing accuracy the number of parameters grows. This trend suggests that quantization method scales effectively with model size even for extremely low-bit quantization. For 3-bit quantization, approach dramatically improves performance, particularly smaller models. For example, 3-bit quantization of Qwen3-1.77 achieves an average accuracy of 55.20% across five tasks, which 18.71% higher than existing PTQ methods. These results highlight that approach is especially effective when model capacity is constrained, such as in cases of ultra-low bit widths or limited parameter counts.
4.4 COMPARISON WITH QAT APPROACHES
This section compares our approach with two SOTA quantization-awarr training (QAT) approaches: BitDistiller (Du et al. 2024) and EfficientQAT (Chen et al. 2024). To ensure fair comparison in reasoning benchmarks, we reproduce these QAT approaches in our proposed framework. Specifi- cally, for both methods, we perform calibration using mixed dataset, and fine-tuning with the same number of training tokens sampled from OpenThoughts- 1,2M Datasets. As shown in Table 3 our approach achieves better performance in both 2-bit and 3-bit than two QAT baselines. These results demonstrate that our approach offers more effective strategy for low-bit quantization compared to conventional QAT methods.
4.5 ABLATION STUDY
There very few quantization-awan (QAT) training approaches that be directly compared to ours, as most existing methods target different evaluation settings. Instead, this section presents ab- lation studies that compare approach with key components derived from existing QAT methods.

---

Published as a conference paper at ICLR 2026
Table Comparison of SOTA quantization approaches with various bit-width on Qwen3-1.7B mod- els. Values are % (higher is better). We define the group size 128.
Settings
Benchmarks (%)
Avg.
Model
Method
Bit Width (W/A)
MATH500
LiveCodeBench
MMLU-Redux
GPQA- Diiqmodd
[FEval
Avg.
Qwen3-1.7B
(Baseline)
b6
89.0
74.7
38.4
65.2
EfficienQQT
3/16
80.8
29.8
66.2
30.3
60.4
53.5
BitDistiller
3/16
59.4
10.2
56.6
17.7
52.3
39.2
Proposal
3/16
82.7
33.0
67.7
31.7
61.0
55.2
EfficientQAT
24.8
0.6
29.1
12.1
25.1
18.3
BitDistiller
2/16
12.2
1.0
22.4
21.7
19.0
15.2
Proposal
2/16
48.6
6,5
40.1
14.5
32.2
28.4
Table 4: Ablation on block-wise calibration and loss choice: "S" denotes conventional supervised fine-tuning; "R" denotes our proposed loss; and "C" indicates the of block-wise calibration.
Table 5: Benchmark comparison our pro- posal with INT2 Qwen3 family quantized with our proposal and BitNet b1.58 2B. We represent Instruct Strict IS.
Training
$
C+S
C+R
Qwen3
Qwen3
Qwen3
MATH-500
1.4
1,60
28.57
22.70
38.13
Benchmark (Metric)
(Ours)
4B-ant2 (Ours)
1.78-int2 (Ours)
2B
Live Code Bench
0.0
0.0
0.47
0.00
5.75
Training Tokens
328M
328M
968M
MMLU Redux
3.54
3,53
35.78
36.64
Activation
bf16
br16
b16
int8
GPQA Diamond
6.06
6.06
7.58
14.31
14.14
MATH-500
80.
77.13
48.60
43.40
TFEval
10.72
12.20
24.58
23.66
31.61
GSMSK
88.93
81.71
57.47
58.38
(IS)
Average
76.13
69.91
50.75
51.75
Effectiveness of Block-wise Calibration: The main differences between our approach and ex- isting methods are the of block-wise calibration before fine-tuning and the loss function. To analyze the effects of these two components, we either replace the loss function with conventional cross-entropy loss, which is basically used in QAT (Liu al. 2025c). In these experiments, we fine-tune the 2-bit quantized Qwen-3 1. ..7B model for one epoch using 32K sequences with learn- ing rate of 5e-6. Table summarizes the results. Here, "S" denotes the conventional supervised fine-tuning with cross entropy loss function, "R" denotes the teacher-guided reward rectification loss Section 3.2 and "C" means the existence of a calibration stage. Therefore, "C+S" denotes supervised fine-tuning after calibration. As shown in Table 4 both calibration data and proposed loss function significantly enhance model performance. We also find that modifying the supervised loss led to substantial improvements reasoning benchmarks. In particular, MATH-500, the curacy increased by 15.43% when moving from cross entropy loss to our proposed loss, and on Live Code Bench, it increased 5.75%. More importantly, the performance degrades after conducting supervised fine-tuning with cross-entropy loss. These results indicate that conventional QAT approaches, which rely primarily on cross-entropy loss for supervised fine-tuning, are insufficient for post-trained LLMs.
reasoning benchmarks
Effectiveness of Loss Weighting: teacher- teacher-guided reward-rectification loss and the KL-divergence loss, to overall performance. We fine-tune 3-bit quantized 1wen3-11.7B for single epoch with different and B. We evaluate three different weighting schemes: (a,, B) = (1, O), which applies only the reward rectification loss; (o, 3) (0, 1), which applies only the KL divergence loss; and (a, 9) (0.2, 1) (i.e., our proposed configuration), which combines both losses with the specified weights. Table demonstrates that combining the losses improves overall model performance. These results demonstrate that our approach exhibits stronger gains in more aggressive quantization settings, particularly at 2-bit.
This section studies the contribution of two loss terms, the
4.6 COMPARISON WITH BITNETI 1.58 2B4T
This section compares our quantized models with BitNet1.58 2B4T, a native ternary LLM trained from seratch. To align the bit-width, our QAT pipeline quantizes Qwen3 models into 2-bit.
8

---

Published as a conference paper at ICLR 2026
Table 6: Effect of the loss function on reasoning performance. Comparison of 3-bit and 2-bit quan- tization on Qwen3 1.7B.
Bit Width
(a, (3
MATH-500
Live Code Bench
MMLU Redux
GPQA Diamond
IFEval
(1.0,0.0)
78.2
28.5
66.1
21.7
60.6
(0.0, 1.0)
82.8
32.4
67.0
30.3
63.0
(0.2, 1.0)
82.7
33.0
67.7
31.7
62.1
(1.0,0.0 0.0)
22.8
1.5
32.7
15.7
25.0
(0.0, 1.0)
0.0
6.6
2.0
8.0
(0.2, 1.0)
48.6
6,5
40.1
14.5
32.2
Table 5 describes the accuracies on two mathematical benchmarks including MATH-500 and GSM8K, and IFEval. We referred to the results of BitNet1. 1.58 2B4T from (Ma et al. 2025). As shown in Table 5 our INT2 quantized model achieves superior mathematical performance with lower parameter requirements and significantly fewer tokens required for the quantization process. These results demonstrate that by designing an appropriate QAT pipeline, it is possible leverage pre-trained features, leading to promising reasoning performance train high-accuracy 2-bit models at a fraction of the training costs.
In addition, our approach demonstrates superior scalability compared to BitNet 1.58 2B4T. Because we fine-tune pre-trained LLMs using only limited number of sequences, we can easily produce models with different parameter counts. This enables a flexible trade-off between performance and resource usage, as illustrated Table which demonstrates results of several parameter variations of our quantized models.
5 RELATED WORKS
In this section, we briefly summarize the quantization approaches. Quantization approaches be categorized into post-training quantization (PTQ) and quantization-awarr training (QAT) depending on whether fine-tuning performed not. This section deals with weight-only quantization of large language models (LLMs) addressed in this work.
Post-training quantization (PTQ) converts full-precision weights into lower-bit counterparts with- out relying on fine-tuning. To obtain better quantization parameters, recent methods optimize the
reconstruction problem either the linear projection level (Frantar et al. 2022 Lin al.) 2024)
or at the transformer block level (Lee et al. [023]] Shao et al.) (023). W While PTQ has achieved
strong initial success in LLMs, initial approaches still face limitations achieving extremely low-bit quantization without losing their performance. To overcome these challenges, research has shifted toward more aggressive quantization, such as 3-bit or 2-bit. Some approaches target such low-bit quantization with integer representation (Shao et al., 2023 Zhao et al. 2024 Chee et al. 2023) demonstrating noticeable performance at these bit-widths. To further improve the trade-offs be- tween accuracy and model size, recent approaches introduce vector quantization (Fgiazarian al.) 2024 Tseng et al. 2024 Malinovskii et al., (002)). Despite their promising performance, vector quantization introduces substantial overhead inference (Gong et al. 2024).
Quantization-aware training (QAT), in contrast, can enhance quantized model performance by incorporating fine-tuning. With the additional computational cost for fine-tuning, QAT enables the use of bardware-friendly numerical representations, such as integers, for low-bit quantization, re- sulting minimal overhead inference time. There are several choices for optimization targets for fine-tuning. LLM-QAT (Liu et al. 2023) and BitDistiller (Du et al.] 2024) explore knowledge distillation within QAT literature. BitNet bl. .58 (Ma et al. (024). Spectra (Kuushal et al.) 2024). and ParetoQ (Liu et al. 2025c) employ fine-tuning in self-supervised manner using pre-training data. By spending billions of tokens for fine-tuning, these approaches realize promising performance with ternary or 2-bit. Given the substantial training costs of these approaches, recent work has focused on improving the training efficiency of QAT approaches. EfficientQAT (Chen et al.) 2024) intro- duces two two-stage pipeline that perform end-to-end backpropagation following block-wise cali- bration. UPQ (Lee et al. 2025) modifies the two-stage QAT pipeline to use knowledge distillation and progressive quantization, demonstrating the promising performance instruction-tuned L.MMs.

---

Published as a conference paper at ICLR 2026
However, most existing quantization approaches have primarily focused on pre-training LLMs, with limited exploration of their effectiveness on complex reasoning capabilities that are crucial for mod- LLM upplications. In this paper, investigate how quantization affects reasoning performance and propose methods to preserve reasoning capabilities in quantized L.Mss.
Quantization and Reasoning. Several comprehensive analyses have explored the effects of quan- tization on reasoning capability. eL et al. (2025) and Liu et al.] (2025b) demonstrate that ultra-low- bit quantization leads severe performance drops on reasoning benchmarks such mathematical tasks. Liu et al. (022b) demonstrates that less-than-4-bit quantization leads to severe performance drops on reasoning benchmarks such as mathematical tasks. Mekala et al. (2025) systematically analyzes the effects of quantization on long-context reasoning tasks, demonstrating that even 4- bit models incur substantial losses. Although these analyses reveal critical challenges for existing quantization approaches, few studies have explored effective strategies to maintain reasoning per- formance under such aggressive settings. However, one notable example is BitNet 2B4T (Ma al.) 2025), which demonstrates strong performance mathematical benchmarks with ternary LLMs by performing quantization-avee more efficient approach for ceasoning-orienter LLMs. By combining block-wise quantization with RL-inspired fine-tuning using limited tokens, we obtain highly accurate 2- and 3-bit LLMs with significantly fewer fine-tuning sequences.
training over four trillion tokens. In this paper, we explore a
CONCLUSION
This paper addresses the critical challenge of maintaining reasoning capabilities ultra-low-bit quantized large language models (LLMs). Through systematic analysis, demonstrate that quan- tization affects different knowledge domains bust, reasoning capabilities show severe degradation. Building this insight, we develop novel two-stage quantivation-aware training pipeline specifically designed for post-trained reasoning LLMs. Our approach combines mixed-domain calibration with teacher-guided reward rectification loss to preserve and restore reasoning abilities under aggressive quantization. Experiments across five reasoning benchmarks validate our method, with 2-bit quantized Qwen3-8B achieving 50.45% average improvement over existing approaches. Notably, our method outperforms BitNet 2B4T on mathematical reasoning while requiring dramatically fewer training resources. We establish the first quantization framework specifically targeting reasoning-oriented LLMs, providing practical solu- tions for efficient model compression without sacrificing cognitive capabilities. This work provides foundation for future developments in efficient, perforefformncc quantized reasoning models, enabling broader deployment of sophisticated AI systems.
pre-training knowledge remains ro-
ACKNOWLEDGMENT
This work was carried out using the TSUBAME4.0 supercomputer at Institute of Science Tokyo. This work was supported in part by JSPS KAKENHI Grant Numbers JP23H05489, JP25K03092, and JP23KJ0955, and by JST-ALCA-Nex Japan Grant # JPMJAN24F3.
REFERENCES
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7433 2020.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36:4396 4429, 2023.
Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, and Ping Luo. Efficientqat: Efficient quantization-aware training for large language models. arXiv preprint arXiv:2407.11062. 2024.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501. 17161, 2025.
10

---

Published as a conference paper at ICLR 2026
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.088
2018.
Dayou Du, Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao, Xiaowen Chu, and Ningyi Xu. Bitdistiller: Unleashing the potential of sub-4-bit Ilms via self-distillation. arXiv preprint arXiv:2402. 10631, 2024.
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Al- istarh. Extreme compression of large language models via additive quantization. arXiv preprint MwXiv:2401.06118, 2024.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint 22X.i::210..732.. 2022.
Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, et al. Are done with mmlu? arXiv preprint arXiv:240.004222 2024.
Ruihao Gong, Yang Yong. Shiqiao Gu, Yushi Huang, Yunchen Zhang, Xianglong Liu, and Dacheng Tao. L.m-qbench: benchmark towards the best practice for post-training quantization of large language models. CoRR, 2024.
Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smymis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, Ashima Suvarna, Benjamin Feuer, Liangyu Chen, Zaid Khan, Eric Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, Wanjia Zhao, John Yang, Shreyas Pimpalgaonkar, Kartik Sharma, Charlie Cheng-Jie Ji, Yichuan Deng, Sarah Pratt, Vivek Ramanujan, Jon Saad-Falcon, Jeffrey Li, Achal Dave, Alon Albalak, Kushal Arora, Blake Wulfe, Chinmay Hegde, Greg Durrett, Sewoong Oh, Mohit Bansal, Saadia Gabriel, Aditya Grover, Kai-Wei Chang, Vaishaal Shankar, Aaron Gokaslan, Mike A. Merrill, Tatsunori Hashimoto, Yejin Choi, Jenia Jitsev, Reinhard Heckel, Maheswaran Sathiamoorthy, Alexandros G. Dimakis, and Ludwig Schmidt. Openthoughts: Data recipes for reasoning models, 2025. URL https: / org/ abs 2506. 04178
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint MxXi::5501.1244: 2025.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. /2 2166, 2024.
Ayush Kaushal, Tejas Vaidhya, Arnab Kumar Mondal, Tejas Pandey, Aaryan Bhagat, and Trina Rish. Spectra: Surprising effectiveness of pretraining ternary language models at scale. arXiv preprint arXiv:2407. 12327, 2024.
Jiedong Lang, Zhehao Guo, and Shuyu Huang. A comprehensive study on quantization tech- niques for large language models. In 2024 4th International Conference on Artificial Intelligence, Robotics, and Communication (ICAIRC), pp. 224-231. IEFE, 2024.
Jung Hyun Lee, Jeonghoon Kim, Se Jung Kwon, and Dongsoo Lee. Flexround: Learnable rounding based on element-wise division for post- post-training quantization. In International Conference on Machine Learning. pp. 18913-18939, PMLR, 2023.
Jung Hyun Lee, Seungjae Shin, Vinnam Kim, Jaeseong You, and An Chen. Unifying block-wise ptq and distillation-based qat for progressive quantization toward 2-bit instruction-tuned llms. arXiv preprint arXiv:2506.099 09) 004, 2025.
Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. co/AI-MO/ ho/AAI-M0///mmmaaaaahhhhh (atps: http://ssss..aaaao.ca/ccoo/ttaaa nimo- itte:pggrressspppr///////ttttttr oftp://ooooooooooooeaaaasataee.i 2024.

---

Published as a conference paper at ICLR 2026
Zhen Li, Yupeng Su, Runming Yang. Congkai Xie, Zheng Wang, Zhongwei Xie, Ngai Wong, and Hongxia Yang. Quantization meets reasoning: Exploring Ilm low-bit quantization degradation for mathematical reasoning. arXiv preprint arXhv:2501.03035 2025.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation- weight quantization for on-device IIm compression and acceleration. Proceedings of machine learning and systems, 6:87-100, 2024.
Ruikang Liu, Yuxuan Sun, Manyi Zhang, Haoli Bai, Xianzhi Yu, Tiezheng YU, Chun Yuan, and Lu Hou. Quantization hurts reasoning? empirical study quantized reasoning models. In Second Conference on Language Modeling. 2025a. URL http:://ppeerevieer..
Forum?id=Bil 95
Ruikang Liu, Yuxuan Sun, Manyi Zhang, Haoli Bai, Xianzhi Yu, Tiezheng Yu, Chun Yuan, and Lu Hou. Quantization hurts reasoning? an empirical study on quantized reasoning models. arXiv preprint arXivv2504.0483 0482244400,,, 2025b.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305.17888 2023.
Zechun Liu, Changsheng Zhao, Hanxian Huang, Sijia Chen, Jing Zhang, Jiawei Zhao, Scott Roy, Lisa Jin, Yunyang Xiong, Yangyang Shi, et al. Paretoq: Scaling laws in extremely low-bit Ilm quantization. arXiv preprint arXiv:2502.02633 026302...2.,..... 2025c.
Iyya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In In- ternational Conference on Learning Representations, 2017. URL https:://ppenreviev..
net/forum?id=3keee
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL
Bky6
Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest
collection of educational content, 2024. URL httpss///hucgnggffa....o/ttteee
HuggingFaceTW/ www.ce/wwwwwiwwbeeeeeeeeeeee
Pan Lu,, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, Advances in Neural Information Processing Systems. 35:2507-2521, 2022.
Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Lifeng Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The of 1-bit 1lms: All large language models are in 1.58 bits. arXiv preprint ar 2402 177,, 1(4), 2024.
Shuming Ma, Hongyu Wang, Shaohan Huang, Xingxing Zhang, Ying Hu, Ting Song, Yan Xia, and Furu Wei. Bitnet bl. 58 2b4t technical report. arXiv preprint arXiv:2504. 2tti:25504.22888 12285, 2025.
Vladimir Malinovskii, Denis Mazur, Ivan Ilin, Denis Kuznedelev, Konstantin Burlachenko, Kai Yi, Dan Alistarh, and Peter Richtarik. Pv-tuning: Beyond straight-through estimation for extreme Ilm compression. Advances in Neural Information Processing Systems, 37:5074-5121, 2024.
Anmol Mekala, Anirudh Atmakuru, Yixiao Song, Marzena Karpinska, and Mohit Iyyer. Does quan- tization affect models' performance on long-context tasks? arXiv preprint arXiv:2505.20276 2025.
12

---

Published as a conference paper at ICLR 2026
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728-5533 53344,, 2023.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a bench- mark. In First Conference on Language Modeling, 2024.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code Ilama: Open foundation models for code. arXiv preprint 2rXiv:2308./290 2023.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106, 2021.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308. 13137, 2023.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits mathematical reasoning in open language models, 2024. URL URL https:/arxiv org/abss2402..03300, 2(3):5, 2024.
ModelScope Team. EvalScope: Evaluation framework for large models, 2024. URL https:
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better Ilm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.04396, 2024.
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijic Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Wu, and Furu Wei. Bitnet: Scaling 1-bit transformers for large language models. arXiv preprint arXiv:2310.I 1453, 2023.
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations, 2024. URL https:/arxiv.. org/abs/2312.08933
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc Le. Finetuned language models are zero-shot learners. arXiv preprint 109.0220 2021.
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz- Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, et al. Livebench: A challenging, contamination- limited I1m benchmark. arXiv preprint arXiv:2406.199 19314, 2024.
Yongliang WU, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuan Yang, and Xu Yang. On the generalization of sft: A reinforcement learning perspective with reward rectification. arXiv preprint arXiv:2508. 05629, 2025.
Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. Onebit: Towards extremely low-bit large language models. Advances in Neural Information Processing Systems, 37:66357-66382. 2024.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409. /2 122, 2024.
13

---

Published as a conference paper at ICLR 2026
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint S2kY::25005003888, 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint 1rXiv:/905.07890 2019.
Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate Ilm serving. Proceedings of Machine Learning and Systems, 6: 196-209, 2024.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-follooiing evaluation for large language models. arXiv preprint arXiv:2311.079111 2023.
Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou. Luning Wang, Zhihang Yuan, Xiuhong Li, et al. A survey on efficient inference for large language models. arXiv preprint arXiv:2404. 14294, 2024.
14

---

Published as a conference paper at ICLR 2026
Table 7: Comparison of training cost and reasoning performance of Qwen3-1.7B under 3-bit and
2-bit quantization. GPU hours are measured using the H100 GPU.
2nd Step
Bit Width
GPU Hours
MATH-500
LiveCadellench
GPQA-Diamond
MMLU-Redux
IFEVAL
Avg.
GRPO (RL)
=220
31.5
25.8
66.5
55.5
Ours
=5
80.2
318
29.8
66.4
58.8
53.4
GRPO (RL)
=110
24.4
11.6
29.5
24.0
Ours
45.0
2.7
10.6
34,9
29.2
24.5
Table 8: Comparison of first-step and fine-tuning (FT) cost along with downstream accuracy under
3-bit and 2-bit quantization on Qwen3-1.7B. We report training time normalized to a single H100 GPU.
Method
Bit Width
First step
FT Epochs
FT Sequences
Avg. Acc.
EfficientQAT
3
4
22
1
32k
53.5
BitDistiller
3
0.25
26
32k
39.2
Block+GRPO
3
Rs 1180
1
32k
N/A
Ours
3
R2 4
Rs 23
1
32k
55.2
EfficientQAT
2
66
3
32k
18.3
BitDistiller
0.25
78
3
32k
15.2
GRoO
2
8 1
& 1016
3
32k
N/A
Ours
2
22 4
22 80
3
32k
28.4
A COMPARISON WITH ON-POLICY REINFORCEMENT LEARNING
While on on-policy reinforcement learning incurs substantial training overhead due autoregres- sive text generation, it is commonly observed that such methods generalize well reasoning tasks by learning from the model's own output distribution. Our approach, instead, doesn't It require autoregressive text generation because it only uses sequences sampled from datasets. Given these contrasting properties, it important to assess whether our proposed off-policy objective can match the generalization typically associated with on-policy methods.
To clarify this trade-off, we conduct a direct comparison between our proposal and on on-policy rein- forcement learning. Specifically, compare GRPO (Shao et al., 2024), SOTA on-policy RI. approach, with our off-policy loss. To ensure a fair comparison, we conduct the experiment us- ing 6K sequences. Following DeepSeek-RI (Guo al. 2025), we adopt accuracy and formatting rewards for GRPO. We filtered the OpenThoughts- 1ooohhhttt-.2M 1.2M dataset to retain only samples with ex- tractable answer labels to ensure that the accuracy reward over the training dataset is well-defined before randomly sampling 6K sequences.
As shown Table 7 our method achieves comparable performance 3-bit precision and outper- forms GRPO at 2-bit precision with significantly reduced training time. These results demonstrate that our off off-colicy approach both training-effice efficieet and particularly effective low-bit settings.
B TRAINING TIME ANALYSIS
observed Section our approach performs well on reasoning benchmarks. Despite this promising performance, the overall training cost is also critical for practical quantization. This section compares the training time both the first and fine-tuning stages to clarify this considera- tion.
Table compares the required training time for Qwen3-1.77 with other QAT approaches. The results demonstrate that method achieves favorable trade-off between accuracy and training time, par- ticularly under aggressive low-bit quantization. Notably, the 2-bit setting, our approach achieves over 10% higher accuracy than EfficientQAT, while maintaining comparable training efficiency.
15

---

Published as a conference paper at ICLR 2026
Table 9: Generation length comparison of Qwen3 models under different quantization bit-widths.
MATH-500
GPQA-diamond
LiveCodeBench
Model
Bit Width
Overall
Correct
Incorrect
Overall
Correct
Incorrect
Overall
Qwen3-1.7B
5905
4706
15610
6744
5561
7480
12329
Qwen3-1.7B
7325
5059
17247
8564
7168
9201
15309
Qwen3-1.7B
13803
6483
21065
14497
11539
14944
22466
Qwen3-8B 3BB
16
5513
4923
14753
7397
6252
9235
11100
OWB
3
6296
5359
16776
10473
8730
11896
14824
Qwen3-8B
2
9966
6136
25679
15738
13833
16779
20373
C ANALYSIS OF DECODING LENGTH
Analyzing additional reasoning metrics, such as token count, is important for understanding the qualitative behavior of quantized models. Differences in token usage induced by quantization may reveal shifts in reasoning style or depth that are not captured by standard benchmark accuracies. Evaluating these aspects provides more comprehensive view of how quantization affects the rea- soning process.
This section addresses this aspect by analyzing the decoding length across several reasoning bench- marks. As shown in Table | we find that the number of thinking tokens increases as the number of bits decreases. This increase in token length primarily arises from more frequent self-correction be- havior. In other words, quantized models tend to revisit and validate their answers more frequently, as illustrated in Table I0 These results indicate that quantization can introduce redundant correc- tive reasoning. Thus, mitigating such unnecessary thought processes during QAT is important for controlling the additional thinking tokens induced by quantization.
D FUTURE WORK
This section discusses several promising directions for further enhancing the quantized reasoning models.
Adaptive Mixing Ratio: Adapting the dataset mixing ratio in both the first and second stages is promising research direction. This flexible strategy can enable more effective dataset selection tailored to each training phase, potentially improving overall model performance. In particular, dynamically adjusting the proportion of supervised, preference, or reasoning- heavy data may help stabilize training and reduce overfitting to any single data source. Developing principled methods automatically determining these ratios could further enhance the efficiency and robustness of quantization-awen aware training. Additionally, adjusting the domain proportions within the reasoning dataset, such as increasing the ratio of code-related samples, may further enhance performance on more complex tasks like code generation. Exploring domain-specific mixing strategies remains an interesting avenue for future work.
Increasing Model Size: Applying approach over 10B models is also promising research direction for future research. While our current experiments focus on smaller-scale models, where the performance degradation of quantization is more severe, our promising results suggest that the benefits of our method may scale favorably with increased parameters. Due to the limited availability of computational resources, is difficult to conduct such larger-scale experiments. Nevertheless, we believe that our approach will scale effectively to larger LLMs.
As observed in Appendix C quantized models introduce addi- tional reasoning tokens. From practical perspective, this increase tokens reduces throughput and requires larger KV-cache memory. Moreover, a quantitative analysis of generated samples reveals that quantization tends not to make models think more carefully, but instead leads them to repeatedly verify answers that were already produced. Suppressing such redundant verification behavior may enable more efficient quantized models.
Reducing Redundant Thinking:
16

---

Published as a conference paper at ICLR 2026
Table 10: Reasoning text sample from 2bit Qwen3-8B quantized by our framework. The model can be seen revisiting its reasoning multiple times.
<think>
Okay, so need to find the length of DE in this ciagran. The problem says that sin D is 0.7. let me try to visual ize the diagram ritst. There's right angle mark at D, E, and F, maybe triangle D-K-H check the Asymptote code.
is right triangle?
The Asymptote code defines points D, E, F. is at (sqrt (51), 7), is at (0, 7), and is at (0, 0). Then they draw D-E-F and right angle mark at D, at single vertex. Maybe it's a right triangle with right angle at D? Let me think. If D is connected to and F, and there's a right angle mark at D, E, F, that might mean that the triangle is right-angled at D? But then E and F would be connected. Let me sketch mentally.
Wait, but right angles are usually marked
Point D is at (sqrt 7), E is at (0, 7), so E is directly to the Left of D the horizental line y=l. Then F is at (0, 0), which is directly below on the vertical line x-0. Sc connecting to check coordinates:
to to forms a triangle.
D: (sqrt (51), 7)
E: (0, 7)
F: (0, 0)
So DE is from D to E. Let me compute DE first. The coordinates of D are (51), 7) and is (0, 7). So the distance between D and z is [
02 2 (7 (7 2) sqrt (sqrt (51)) 0] cert(51),) Wait, sqrt (51) approximately 7.07. But the problem says sin D. is 0.7. Maybe need to find using that? Wait, but maybe the diagram is not to scale? Or maybe there's some triangle where angle D is involved?
wait, the Asymptote code draws D-E-F' as triangle and marks right angle at D, E, and F? But (... 192 words omitted)
Mait, in triangle DEF, angle at is between DE and DF. Let me confirm. IE the right angle is (... 17 words emitted)
First, 1et me compute DE, EF, DE.
DE is from D sqrt (sgrt (51) 02 (7 ") sqrt (51)
7.07.
RF is from B to F: sqrt ( (0 0) 12 + (7 22 - 7.
DF is from D
sqrt (sqrt (51) 0) 22 49) (... words omitted)
"2 sqrt
So triangle DEF has sides DE sqrt (51), , EF 7, DF 10. Wait, but if the right angle is equal 2 Let's check:
then by Pythagoras, DE EF should
DE*2 EF*2 51 49 = 100, which is equal to DF"2 = 10 100.
Perfect! So (... 33 words emitted)
Mait, the problem says verify is this is consistent (... words omitted)
0.7. So sin (angle D) 0.7. Let me
17

---

Published as a conference paper at ICLR 2026
In triangle DEE, angle D is between DE and DF. So in that triangle, sin (angle D) = opposite side / hypotenuse. Wait, opposite side to angle would be EF. Wait, in triangle DEF, angle at Di the side oppesite angle sin D) EF DF that matches the problem statement. Therefore, DE is sqrt( but the problem says sin D 0.7, but DE is already given? Wait, but the problem is asking for DE? Wait, hold on, maybe I misread the problem?
EF, which is 7. The hypoteause is DF 10. So 0.7. Oh! That's exactly 0.7. So
wait, the problem says "Suppose sin D 0.7 in the diagram below. what is DE?" But in the Asymptote code, DE is sqrt (51). But maybe the diagram is not to scale? Wait, but the coordinates are given. Let me check again. Wait, maybe the diagram is different? Wait, the Asymptote code defines D as (s1)t 7), E as (0, 7), as sqrt (51). So DE is sqrt (51). But then why is sin D given as 0.77 Because in the triangle DEF, sin is 7/10 0.7, so that consistent.
(0, 0) So DE is sqrt (sqrt (51) 01 (7 7)
Therefore, is the answer DE = sqrt((51) But maybe the problem is not referring to triangle DEF? Maybe there is another triangle in the diagram? Wait, the Asymptate code draws D-R-F as triangle, but there's also right angle mark diagram is different?
E wait, maybe the
But regardless, if the triangle DEF is right angled at E, with DE sqrt (51), EF 7, DF 10, then sin D is 7/10 0.7, which matches the problem statement. Therefore, DE sqrt (51). But maybe the problem is not referring to triangle DEF? Wait, the problem says "in the diagram belox", but the Asymptote code is given. Wait, maybe there's another triangle in the diagram? Let me check again.
But regardless,
the triangle DEF right-angled at E, with sert (51), EF = 7, DF = 10, (... 52 words omitted)
wait, the Asymptate code draws D-E-F, which is triangle DEF, and then draws right angle mark b, (... 120 words critted)
But regardless, the problem says sin D 0.7, and in triangle DEF, sin D is 0.7. Therefore, DE is (... 219 words emitted)
Alternatively, maybe the diagram is not triangle DEE, but another triangle involving D and E? For example, maybe triangle D-E-scmething (... 83 words emitted)
Wait, but the problem says "Suppose sin 0.7 in the diagram below. What is DE?" the diagram (... words omitted)
pair D,E,F; (0,0); (sqrt
g (0,7);
draw (D--8------)
draw (riightnaggglemmrkkkk (D, E,
label ("SDS", "VSDS",D,NE) NE);
label E"SES",E,N))
label ("$F$",F, ; label ("$75", (E+F)/2,W).
So, in the diagran, there is a right angle D, and F? Wait, no. The rightanglemark is drawn (... 159 words omi tted)
18

---

Published as a conference paper at ICLR 2026
Wait, but maybe the problem is not referring to triangle DEF, but another triangle? Let me think. The Asymptote code (... 101 words omitted)
But naybe the problem is not about triangle DEF? Wait, angle D given as sin D 0.7, (... 96 words omitted)
Wait, Let check the problem again: "Suppose sin 0.7 in the aiagram below. What is DE?" So (... words emitted)
wait, let me think again. If angle D is in triangle DEE, with sin D 0.7, and we can (... 171 words omitted)
Wait, maybe the diagram not triangle DEF? Maybe there's different triangle where angle D is involved. Wait, let (... 116 words omitted)
But maybe the problem is not referring triangle DEF? Maybe there's another triangle where DE is the hypotenuse or (... 38 words omitted)
Wait, but the Asymptote code draws only D-E-F. unless there is another point, mayse the diagran different? Wait, (... 137
words omitted)
But in that case, need to know EF, DE, and angle D. wait, but if we have sin D (... 69 words cmitted)
Wait, but if we don't have other sides, how can we find DE? Wait, unless there is a way to (... 181 words omitted)
But the problem says "in the diagram below", which given by the Asymptote code. Therefore, the answer is sqrt (51). (... 153 words emitted)
tterntivelly maybe there a misunderstanding in the problem. Maybe the diagram is not triangle DEF, but another figure? For (... 139 words omitted)
But maybe the problem is expecting different approach? Let think 0.7, and we need to find DE. If we don have
again. If sin coordinates, how would we de it? Let me consider triangle D-E-F, with right angle at E. Then, angle D is between DE and DF. So sin D opposite hypotenuse
EF / DF = = / 10 0.7. Therefore, if found via Pythagoras: DE72 sqrt((1).. Therefore,
we know EF 7, DF 10, then
7-2 10 DE"2 100 regardless of sin D, DE is sqrt (51). Therefore, the answer is sert (51). So maybe the problem is just giving sin D to contirm that the triangle is consistent, but the actual answer is (51), Therefore, the answer is \ooxed( (s11
51 =>
But let me check once more. If the problem had not given coordinates, but only sin 0.7 and other information, how would we find DE? and know
For example, if had right triangle with angle sin D, and maybe another side, then we could find DE. But in this case, we have right triangle with right angle at E, so we can use Pythagoras. So unless there is more information, the answer is sert (51), Therefore, think that is the answer.
Finill Answer**
\boxed[\ (esrrt (51)) </think>
19


Reviews:
[{"review_id": "ZtTJJnXWwP", "reviewer": "Reviewer_tqL3", "summary": "This paper addresses the significant challenge of performance degradation in large language models (LLMs) on reasoning tasks when subjected to ultra-low-bit (<4 bits) quantization. The authors hypothesize that this degradation stems from the fact that quantization disproportionately affects the complex knowledge structures introduced during post-training (e.g., SFT, preference optimization) compared to the commonsense knowledge acquired during pre-training.\nTo counter this, the paper proposes a novel two-stage Quantization-Aware Training (QAT) pipeline:\n\n- Stage 1 (Calibration): Performs an initial block-wise quantization using a mixed-domain calibration dataset (80% reasoning, 20% pre-training) to preserve both reasoning capabilities and general knowledge.\n\n- Stage 2 (Fine-Tuning): Fine-tunes the quantized model using a novel objective function called the \"teacher-guided reward-rectification loss\". This loss reweights the standard supervised fine-tuning (SFT) loss using the probabilities from the full-precision teacher model, combined with a KL divergence term. This is proposed as an efficient, offline alternative to RL-based methods.\n\nExperiments on Qwen3 models show that this pipeline dramatically outperforms standard post-training quantization (PTQ) baselines like GPTQ and AWQ, especially at 2-bit precision, and also shows competitive results against the specialized BitNet-2B4T model.", "questions": "- Regarding the baseline comparisons (Weakness 1), the authors are strongly encouraged to include a direct comparison against at least one SOTA QAT method (e.g., EfficientQAT, UPQ). This would substantially bolster the paper's claims.\n\n- To validate the claims about generalization (Weakness 2), it would be highly beneficial to add an ablation study that compares the proposed off-policy loss to a standard online RL baseline, such as PPO or DPO, on the same reasoning benchmarks.\n\n- In the ablation for the loss function (Table 4), the KL divergence term seems to be heavily weighted $(\\alpha=0.2, \\beta=1.0)$. This suggests much of the performance gain might come from simple distribution alignment. We would be interested to see more sensitivity analysis on the alpha parameter.\n\n- Given that \"deployment\" is a key motivator, the paper would be more complete if it included practical metrics like inference latency (tokens/sec) or memory footprint measurements, which are currently missing.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Clear Problem Statement and Novel Insight: The paper targets a critical problem (quantization failure on reasoning) and provides a well-articulated insight: the differential impact of quantization on pre-training vs. post-training knowledge (validated in Fig 2a).\n\n- Well-Motivated Methodological Components:\n  - The mixed-domain calibration (Stage 1) is a simple, effective, and well-supported solution (Table 1) to the domain-sensitivity problem.\n  - The teacher-guided reward-rectification loss (Stage 2) is a clever modification of SFT that correctly identifies the unreliability of the quantized student's probabilities and leverages the stable, full-precision teacher.\n\n- Impressive and Significant Empirical Results: The performance gains, especially for 2-bit quantization, are dramatic (Table 2). The method successfully retains performance (e.g., 55.07% avg for 8B) where SOTA PTQ methods (GPTQ, AWQ) fail completely (~3-5% avg). This is a significant practical achievement.\n\n- Strong Ablation Studies (Section 4.4): The ablations provide excellent support for the method's design. Table 3 compellingly demonstrates that both the calibration stage (C) and the proposed reward-rectification loss (R) are essential for success.", "weaknesses": "- Regarding Baseline Comparisons (QAT vs. PTQ): A primary concern is the comparison between the proposed QAT method and the PTQ (GPTQ, AWQ) baselines. While the results are impressive, it is generally expected that QAT methods (which involve fine-tuning) will outperform PTQ methods, especially at ultra-low bit-widths. The paper would be more convincing if it included a direct comparison against other state-of-the-art QAT methods (like EfficientQAT or UPQ, which are cited). The authors mention that other methods target different settings, but the paper would be significantly strengthened if the authors could reproduce at least one of these QAT baselines in their own evaluation framework. This would provide a more direct, like-for-like comparison.\n\n- Regarding the Justification for the Proposed Loss: The motivation for the \"teacher-guided reward-rectification loss\" is its efficiency compared to online RL methods. This is a valid point. However, we note that online, on-policy RL methods (like PPO) derive their generalization benefit from learning on the model's own distribution. The proposed method is an offline, off-policy approach. It would be very helpful if the authors could provide more discussion or empirical evidence to support the claim that this offline approach achieves comparable generalization to the online methods it seeks to replace. This is a central and very interesting claim, and validating it (perhaps with an ablation against a DPO or PPO baseline) would make the contribution even stronger.\n\n- Minor Clarification on Training Data: There appears to be a slight inconsistency in the reporting of training data, which could be easily clarified. The abstract mentions \"40K training sequences\", Section 4.1 lists \"32,768 samples\", and Table 5 notes \"0.2M\" and \"0.8M\" tokens. We would appreciate it if the authors could clarify the relationship between these numbers in the final version to avoid potential confusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PaxsZDQzK6", "reviewer": "Reviewer_b4un", "summary": "This paper analyzes the challenge of preserving reasoning capability under ultra low bit quantization and proposes a mixed calibration data strategy together with a two stage QAT pipeline. The method mixes reasoning and pre training data for calibration and then uses a teacher guided reward rectification loss in fine tuning to restore reasoning ability.", "questions": "Even with strong distillation, ultra low bit models still lag FP16 on tasks like LiveCodeBench. Are there other opportunities to further narrow this gap?\n\nCan the authors compare against efficient RL post training methods, for example DPO variants, to validate the claim that RL cost is prohibitive and the proposed loss is a more efficient substitute?\n\nHow does the training cost of the proposed method compare with EfficientQAT and BitDistiller in terms of GPU hours?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper studies an important issue of maintaining reasoning capabilities under aggressive quantization. \n\nThe analysis reveals that reasoning data has higher sensitivity to quantization than pre training data, and the mixed calibration design is logically motivated.", "weaknesses": "The novelty appears limited. The performance gain mainly comes from applying distillation on reasoning data, as shown in Table 4, where the KL or distillation term dominates the improvement and the proposed reward rectification formulation contributes marginal benefit. Distillation is a existing technique in low bit QAT.\n\nThe experimental comparison is not convincing. The main table (table 2) only compares with GPTQ and AWQ which are not QAT methods. Fairness requires comparison with recent QAT approaches, particularly distillation based ones such as EfficientQAT and BitDistiller. \n\nSome claims lack evidence. The paper states that RL methods incur large autoregressive training overhead, but there is no quantitative analysis or comparison against RL based fine tuning for QAT. Similarly the equivalence to on policy learning is mentioned, but no evidence is provided to show practical benefit over standard supervised tuning.\n\nEven with distillation, the model still shows a significant drop from BF16 performance on challenging tasks such as LiveCodeBench, suggesting that reasoning capability is not fully preserved and further improvements are required.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MBgkPo2I47", "reviewer": "Reviewer_Ncvd", "summary": "This paper proposes a reasoning-oriented quantization-aware training (QAT) pipeline for ultra-low-bit (2–3 bit) LLMs.\nThe key idea is that reasoning abilities (acquired in post-training) and commonsense knowledge (from pre-training) respond differently to quantization.\nTo address this, the authors design a two-stage QAT framework:\n\nMixed-domain calibration (80% reasoning data + 20% pre-training data) for balanced preservation of knowledge and reasoning.\n\nTeacher-guided reward-rectification fine-tuning, a reinforcement-inspired objective that restores reasoning accuracy efficiently.", "questions": "When evaluating reasoning LLMs, task accuracy alone may not capture the full effect of quantization. Could you also report metrics related to thinking tokens—for example, the number of intermediate reasoning or \"thought\" tokens generated during inference?\nIn particular, does quantization (especially at 2 bits) affect how long the model needs to “think” or the length of its reasoning chain before reaching an answer?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses an important and underexplored problem—maintaining reasoning capabilities during ultra-low-bit quantization.\n\nThe domain-sensitivity analysis (reasoning vs. pre-training) is interesting and helps motivate the mixed calibration design.\n\nThe pipeline is easy to follow and grounded in recent literature; figures and tables are clear and convincing.", "weaknesses": "The paper emphasizes accuracy but lacks analysis of training cost, throughput, and latency during inference—key aspects for practical quantization.\n\nThe 80/20 calibration ratio is fixed; it would be useful to see sensitivity analysis across ratios or datasets to validate robustness.\n\nOther reasoning metrics, such as thinking token count, is not analyzed, which might reveal qualitative differences induced by quantization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "25PF44qgPE", "reviewer": "Reviewer_d4eQ", "summary": "This paper proposes a reasoning-oriented quantization-aware training (QAT) framework for ultra-low-bit (<4-bit) large language models (LLMs). The method introduces a two-stage pipeline combining mixed-domain calibration (pre-training + reasoning data) and a teacher-guided reward-rectification loss to preserve reasoning capability. Experiments on Qwen3 models (1.7B–8B) show substantial gains over PTQ baselines (GPTQ, AWQ) and competitive performance against BitNet-2B4T with far fewer training tokens.", "questions": "How does the proposed teacher-guided loss scale computationally when the teacher is a large fp16 model?\n\nCould mixed-domain ratios be dynamically adapted during training?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Addresses an important and timely problem: maintaining reasoning ability under aggressive quantization.\n\nClear motivation and well-structured method.\n\nExtensive empirical validation across reasoning benchmarks.\n\nAblation studies convincingly support the proposed components.", "weaknesses": "The approach is only evaluated up to 8B models; scaling to 70B+ remains unverified, though critical for real deployment.\n\nThe paper focuses on accuracy but lacks quantitative analysis of training cost and hardware efficiency (e.g., fine-tuning time, energy, or throughput gains).\n\nSome comparisons (e.g., to full-training QAT methods like BitDistiller or Spectra) could be expanded to clarify relative trade-offs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)