QeRL: Beyond Efficiency - Quantization-enhanced Reinforcement Learning for LLMs
Authors: Wei Huang
Yi Ge
Shuai Yang
Yicheng Xiao
Huizi Mao
Yujun Lin
Hanrong Ye
Sifei Liu
Ka Chun Cheung
Hongxu Yin
Yao Lu
Xiaojuan Qi
Song Han
Yukang Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Quantization
RL
LLMs
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose QeRL, a Quantization-enhanced RL framework for LLMs that improves both efficiency and performance.

Abstract: We propose QeRL, a Quantization-enhanced Reinforcement Learning framework for large language models (LLMs). While RL is essential for LLMs' reasoning capabilities, it is resource-intensive, requiring substantial GPU memory and long rollout duration. QeRL addresses these issues by combining NVFP4 quantization with Low-Rank Adaptation (LoRA), accelerating rollout phase of RL while reducing memory overhead. Beyond efficiency, our findings show that quantization noise increases policy entropy, enhancing exploration in LoRA-based RL, and enabling the discovery of better strategies during RL. To further optimize exploration, QeRL introduces an Adaptive Quantization Noise (AQN) mechanism, which dynamically adjusts noise throughout training. Experiments demonstrate that QeRL delivers over 1.5× speedup in the rollout phase compared to QLoRA, and around 1.3× speedup compared to BF16 LoRA in 7B model. Moreover, this is the first framework to enable RL training of a 32B LLM on a single H100 80GB GPU, while delivering overall speedups for RL training. It also achieves faster reward growth and higher final accuracy than 16-bit LoRA and QLoRA, while matching the performance of full-parameter fine-tuning on mathematical benchmarks such as GSM8K (90.8%) and MATH 500 (77.4%) in the 7B model. These results establish QeRL as an efficient and effective framework for RL training in LLMs.

QERL: QUANTIZATION-ENHANCE LOW-RANK RE- INFORCEMENT LEARNING FOR LLMs
Wei Huang!. 13
Yi Ge
Shuai Yang'
Yicheng Xiao
Huizi Mao
Yujun Lin'
Hanrong Ye
Sifei Liu
Ka Chun Cheung
Hongxu Yin
Yao Lu
Xiaojuan Qi
Song Han
Yukang Chen
NVIDIA
MIT
HKU *THU
https///gthaub.com//aaaiSSELL
ABSTRACT
We propose QeRI., Quantization-enhaneed Reinforcement Learning framework for large language models (LLMs). While RL is essential for LLMs' reasoning capabilities, it is resource-intensive, requiring substantial GPU memory and long rollout durations. QeRL addresses these issues by combining NVFP4 quantiza- tion with Low-Rank Adaptation (LoRA), accelerating rollout phase of RL while reducing memory overhead. Beyond efficiency, our findings show that quantiza- tion noise increases policy entropy, enhancing exploration A-Raed RL, and enabling the discovery of better strategies during RL. QeRL further introduces an Adaptive Quantization Noise (AQN) mechanism. which dynamically adjusts noise during training. Experiments demonstrate that QeRL delivers over 1.5x speedup in the rollout phase compared to QLoRA, and around 1.3x speedup compared to BF16 LoRA A in 7B model. Moreover, this is the first framework to enable RI. training a 32B LLM on a single H100 80GB GPU, while delivering overall speedups for RL training, It also achieves faster reward growth and higher final accuracy than 16-bit LoRA and QLoRA, while matching the performance of full- parameter fine-tuning
mathematical benchmarks such as GSM8K (90.8%) and MATH 500 (77.4%) in the 7B model. These results establish QeRL as an efficient and effective framework for RI. training in LLMss
QeRL achieves a 1.7x speedup in RL for LLMs
QeRL has comparable accuracy to full training
QcRL OLORA LoRA
91.2
90.8
1641
1.5x
X1
1.7x
Accuracy (%)
90
88.1
77.4
77.0
77.4
1362
X0.7
85.0
76.4
2092
X1.2
LaRA
3LeRs
QLaRA
QcRL
Full
Fall
Rollout (tokens/s)
End-to-end (relative)
GSMBK
MATH 500
Figure 1: Rollout speedup and accuracy of QeRL on Swven2.--7BIIntreen QeRL achieves faster RL rollout and end-to-end training speeds (batch=8), while delivering performance superior to
vanilla LoRA and QLoRA, also comparable to full-parameter RL on mathematical benchmarks.
INTRODUCTION
The ability to perform multi-step reasoning critical for large language models (LLMs) to handle complex tasks, from theoretical problem solving to practical decision making (Sui et al., 2025; Xu et al., 2025; Chu et al., 2025; Yang et al., 2021). Supervised fine-tuning (SFT) a common method to improve reasoning by training models replicate explicit reasoning steps (Huang et al., 2024d; Min et al., 2024). However, this approach risks promoting imitation rather than encouraging genuine reasoning. In contrast, reinforcement learning (RL) uses verifiable reward signals to support adaptive learning, allowing models to explore diverse reasoning traces and identify more robust solutions (Lambert et al., 2024; DeepSeek-AI, 2025; Chen et al., 2025a).

---

LLM
Gradients 16-bit
LLM 4-bit
Gradients 16-bir
LLM NVIP
Gradients
16-bit
16-bir
qobb
Rewurds
Rewards
Rewards
Rollouts
Rollouts +
Rollouts
LLM
LoRA
Logils
LLM
LoRA
Logils
LLM
LoRA
Logits
16 bit
+
16-bit
bit
16-bit
NVTEA
16-bit
(a) RI. LoRA
(b) RI. QLoRA
(c) QeRL.
6
uuise
Adaptive Quantization Noise
Noiie Scheduler
Rewan
Stal CoRn
NVFP4 Weight
Training Steps
Quantization Noise
Steps
Figure 2: The illustration of QeRL. (a) RL via LoRA: reducing trainable parameters, but does not alleviate the rollout bottleneck. (b) RL via QLoRA: NF4 quantization with LoRA, but NF4 slower than LoRA. (c) QeRL: NVFP4 quantization with LoRA, reducing memory and enabling faster RL while matching full-parameter finetuning performance with adaptive quantization noise. AQN dynamically adjusts quantization noise, enhancing exploration in LoRA-based RL.
RL is effective for LLMs' reasoning but highly resource-intensive. RL requires substantial GPU memory, as multiple models, such as policy and reference models in GRPO (Shao et al., 2024), must concurrently. The large size of reasoning- focusee LLMs (DeepSeek-Al, 2025) further acerbates memory demands. Training is also slowed by multistage processes. including rollouts, reward computation, logit evaluation, and gradient updates. Rollouts are particularly costly, in- volving repeated sampling and processing of long sequences for complex tasks (Yu et al., 2025). Additionally, RL's inherent sample inefficiency (Hassani et al., 2024) further increases costs.
Improving RL efficiency in LLMs presents significant challenges. One approach, exemplified by Tina (Wang et al., 2025), leverages parameter-effiieet fine-tuning methods like Low-Rank Adap- tation (LoRA) (Hu et al., 2022) to reduce trainable parameters. However, similar to LoRA in SFT (Chen et al., 2024c), these methods fail address the core issue of slow rollout speeds. An- other strategy, demonstrated by FlashRL (Liu et al., 2025a), uses quantized rollout models to reduce computational costs. However, precision mismatches between the rollout model and logits model (e.g., 8-bit vs. 16-bit) require importance sampling to correct discrepancies, necessitating both 8- bit and 16-bit models run simultaneously, which increases memory usage. This usage pattern conceptually similar to QuaRL (Krishnan et al., 2019) (proposed for traditional RL task of small model), where a quantized model used for rollouts while full-precision weights are still needed for policy optimization; as result, the trai--inferenc mismatch (especially pronounced for LLMs) persists and the training-timm time memory footprint remains close to that of full- nallmparameter fine-tuning. To overcome these limitations, focus lower-bit quantization while avoiding duplicate models in memory. Additionally, using QLoRA (Dettmers et al., 2023a) in RL slows rollouts by 1.5-2x, further reducing efficiency. This slowdown occurs because QLoRA relies on NormalFloat 4-bit (NF4) precision, which requires unpacking and mapping to floating-point values via a lookup table before matrix multiplication.
To address the limitations of NF4 in QLoRA, a natural solution is to adopt higher-performance quan- tization. However, standard quantization methods introduce static and deterministic noise, which is non-beneficial to the later-stage RL training. To avoid this drawback, our analysis surprisingly reveals that quantization noise, with precise control, can benefit RL by increasing policy entropy (Fig.3). This added entropy enhances exploration by introducing uncertainty, similar to the effect of parameter noise in RL (Plappert et al., 2017; Pang & Jiang, 2021), and helps models discover better strategies (Cui et al., 2025). Our experiments show that well-designed noise strategy al- lows quantized LLMs to exploit this effect, reducing memory overhead while gaining better reward curves. This finding contrasts with results from SFT of LLMs (Dettmers et al., 2023a; Guo al., 2023), demonstrating that controllable quantization noise in LoRA-based RL enhances exploration and enables quantized frameworks to surpass 16-bit LoRA in both efficiency and performance.
2

---

Low Entropy
High Entropy
Supervised Finetuning
Reinforcement Learning
Probulallty Distributton
web Cuantoation
Probubiitty Distribation
Casntization
Accracy Roward
Quanization LoRs
More Explerationn
Worse
Better
102t - LoRA
10011 LoRA
Vocabulary Size
Vocabulary Size
Training Steps
Training Steps
Figure 3: Advancement of Quantization in RL. Quantization noise brings higher initialized entropy, which encourages exploration in LoRA-based RL training, accelerating the increase of reward.
We propose QeRL, quantization-bass RL framework designed train LLMs on reasoning tasks. As shown in Fig.2, QeRI. uses NVFP4 quantization for L.M weights and integrates Marlin- based (Frantar et al., 2024) approach in both rollout and prefilling stages. This design accelerates rollout and prefilling without sacrificing accuracy, with gradient backpropagation enabled through LoRA layers. To address static quantization noise, we introduce adaptive quantization noise (AQN), which injects channel-wise random noise during training and adjusts exploration noise dynamically using an exponential schedule. Additionally, we implement a noise-sharing strategy that merges the noise vector into the layer normalization layer, enabling zero-parameter overhead for noise injection. Compared to vanilla LoRA, QeRL. achieves faster rollout and better reward growth. For example, as shown in 1 QeRL. outperforms QL and vanilla LoRA in rollout and prefilling speeds on the Dwen2.5-7B-Instruu model, achieving GSM8K score of 108--surassinn both 16-bit LoRA and QLoRA while matching full fine-tuning accuracy on MATH 500. QeRL outperforms vanilla LoRA and QLoRA in both training speed and reward performance. Notably, it achieves approximately 1.8x speedup in end-to-end training, compared to QLoRA. Additionally, QeRL demonstrates the capability to train 32B model with GRPO on single H100 80GB GPU.
2 PRELIMINARY
Model Quantization Integer quantization requires mapping float-poin weights distributed within
the interval [ mins W max] to an integer range of 2 where N is the target bit-width. Given tensor W E Rdxk, this process is defined as:
Round
W ), Sw
W
W
min
(1)
9moz
where W represents the quantized weight matrix, sw is the scaling factor, and Qmaz defines the compressed range. For integer quantization, Qmar 2 1. contrast, for the floating-point quantization, such as FP4 format, Gman 6, achieved using a 1-bit mantissa and a 2-bit exponent (E2M1). 4-bit NormalFloat (NF4) is new data type (Dettmers et al., 2023a), designed for normally distributed weights. Recently, the latest Blackwell GPU architecture (NVIDIA, 2024) introduces hardware support for the advanced FP4 format, MXFP4 (Project, 2023) and NVFP4 (NVIDIA, 2024). MXFP4 adopts shared FP8 (E8MO) scaling factor across parameter blocks of 32 elements, while NVFP4 employs an FP8 (E4M3) scaling factor with smaller parameter blocks of 16 elements, enabling finer-grained scaling adjustments compared MXFP4, Both formats are seamlessly inte- grated into NVIDIA's Hopper (NVIDIA, 2023) and Blackwell (NVIDIA, 2024) GPUs.
Low-rank Adaptation LoRA (Hu et al., 2022) motivated by the observation that weight updates in large pre-trained models often lie in a low-dimensions subspace. Instead of directly fine-tuning all parameters, LoRA introduces a low-rank decomposition to model these updates efficiently:
W A W + BA
where B E RRdx and A E IRr with the rank r & min(d, k). In this setup, the original weight matrix kept frozen, and only the low-rank matrices and B are optimized during training. This formulation drastically reduces the number of trainable parameters and lowers both memory and computational cost, while retaining the expressivity required for domain adaptation. Within self- attention modules. LoRA is generally applied to the attention and feed-forwart projection matrices (W,, q. Wt, W,, W.. W gate: W up: W dowm). as these layers are the most critical in LLMs. Other related works discussed in Appendix D.
3

---

3 METHOD
Our experiments reveal that quantized LLMs can significantly enhance exploration. Applying parameter-effiiiet fine-tuning (PEFT) quantized models not only reduces training resource sumption but also outperforms vanilla LoRA in reward growth and evaluation scores (Fig.2). This challenges the conventional view in SFT that quantization degrades training effectiveness(Dettmees et al., 2023a; Guo et al., 2023). Notably, we observe that quantization error functions similarly to random noise in networks (Plappert et al., 2017; Eberhard et al., 2023; Osband al., 2016), promot- ing broader exploration of potential actions or tokens in RL by increasing entropy (Fig.3).
3.1 TRAINING FRAMEWORK OF QERL
QeRL based on the mainstream policy optimization algorithms of LLMs, such as GRPO (Shao et al., 2024) and DAPO (Yu et al., 2025).
Group Relative Policy Optimization (Shao et al., 2024) designed based on the Generalized Advantage Estimation (GAE) (Schulman et al., 2015), eliminating the need for a separately trained reward model, as required in Proximal Policy Optimization (PPO) (Engstrom et al., 2019; Schulman et al., 2017). Instead, for given input query q. multiple samples generated, resulting in a set of candidate outputs { [01, O2...., )G). These candidates are evaluated using a rule-based reward, and the average reward is used for updates. The optimization objective is defined follows:
lo:l
To (o(0).199) Toald (0,,e Ase, clip(
To(oi,e9) 1 aa, -a)++a))A.))
[(0)- g,t G
(min(
0
Te (oi
1=1
(3) where and Tref denote the policy model and reference model, respectively, and the clipping range (1 - a, 1 + a) stabilized the gradient steps of the policy model. KL penalty is used in GRPO to avoid the unexpected large change in updating (Schulman et al., 2017). Ar, is the antagonist of 2th completion, shared across all tokens in 0,, defined as:
-BDKL(TTll)
A T mean (2....
(4)
std((ri, ra))
Dynamic Sampling Policy Optimization (Yu et al., 2025) suggests higher clipping upper-bond can help avoid entropy collapse. Another improvement in DAPO is utilize the loss of token-level policy gradients. In DAPO, the KI. penalty from Eq.3 is removed to eliminate the upper limit on exploration in RI., thereby encouraging more optional tokens in the rollout process.
3.2 QUANTIZATION ENCOURAGE: EXPLORATION
To understand how quantization enhances RL, analyze its effect on the model's 's sampling behav- ior. Our central finding is that the noise introduced by quantization serves as an implicit exploration mechanism, similar explicit noise injection techniques in the parameter and action space (Plappert et al., 2017; Eberhard et al., 2023; Fortunato et al., 2018; Liu et al., 2025b).
Quantization Improves Sampling Entropy We study mats of FP4 (NVFP4, MXFP4, and NF4) on GSM8K (Cobbe et al., 2021). Our empirical study on Qwenn2.5-7B-Instruett (Team, 2024) reveals an intriguing finding: when applying PEFT- based RL, models quantized tently outperform their 16-bit counterparts. This advan- tage evident across two key metrics: significantly faster reward convergence during training and higher adjusted evaluation scores. As shown in Fig.4, the reward curves of the models exhibit a steeper upward trend compared to bit models, with convergence patterns closely resem- bling those of full- full-parameter fine-tuning in both DAPO and GRPO. Also, NVFP4 and MXFP4 both show better reward growth than NF4.
different quantization for-
NYFF+LORA(133)
0.3
4-bit precision consis-
NF4-LORA(132)
0.25
MXFP4-LORA(52)
Entropy
Flont16-LRA(22))
0.2
0.15
Relatively High
Entropy
0
50
100
150
200
Training Steps
Figure 5: Comparison of RL entropy.
4

---

DAPO
0.95
Accuracy Reward
XVFP4-LORA1E22
MXFP4LLORA(rS2)
NVTTS XN
0.55
Plaatl6--oRM(33)
Flottl6-oRA(E2))
NVEP+LORA(132)
MXFPLLLORA1r32)
6.
- Float16-Full
NF4-LORA(13)
0.15
100
L50
150
200
Training Steps
Training Steps
Training Steps
GRPO
Accuracy Reward
0.75
NVEPL-LOA(GA2)
MXFR+-LORAIS)
NASPA NIR
0.55
Flon16LoRMM322)
hyma
NVEPI-LORAGE2
0.35
WL
MXTP4LORAIFA)
Foort66FFlll
F6out16FFull
NF4-LRRMe33
0.15
100
150
200
100
200
Training Steps
Training Steps
Training Steps
Figure 4: Training reward performance. The upper figures illustrate the training rewards under DAPO, while the lower one GRPO. Although MXFP4 achieves higher scores in the early stages of training, NVFP4 ultimately converges to better final rewards. LoRA rank is set to 32.
This unexpected performance improvement prompted us to investigate the underlying mecha- nism. We discover that quantization inherently increases the sampling entropy, H(x(1g)) - -Eogy [o (q) log (o:]g)) where V is the vocabulary) of the policy during deployment (shown in Fig.5). During the forward pass, quantized model introduces small but systematic errors, which can be modeled as static network noise (Fan et al., 2020). This noise propagates across the net- work layers, perturbing the final logits before the softmax function is applied. Consequently, the output probability distribution over the vocabulary, denoted as Te())). becomes "flatter," with less pronounced peaks. This increase in sampling entropy plays crucial role RL by encouraging ex- ploration (Cheng et al., 2025; Eysenbach & Levine, 2021). It mitigates the model's overconfidence single "optimal" token and instead assigns more meaningful probabilities plausible next actions (Fig.3). The entropy of other model is provided in Appendix H.
wider range of
Quantization Noise Functionally, this effect resembles exploration in parameters (Eberhard al., 2023; Plappert et al., 2017), which deliberately injects noise into parameters to drive exploration:
@0 0)ora) (9 + Otora) Q(+) 0 Ae
(5)
where Q(0) denotes the de-quantized weight, and Ae is the quantization noise. Such exploratory noise emerges naturally as a computationally "free" byproduct of compressing model representa- tions. This contrasts starkly with SFT, where noise is often detrimental because the objective is to faithfully imitate the true data distribution rather than to discover novel high-reward outputs.
key limitation of quantization errors their deterministic nature, which fails to align with the dynamic exploration-exploiitioon trade-off required in RI. Unlike stochastic noise in traditional RL (Plappert et al., 2017; Osband et al., 2016), which is randomly sampled and independently applied at different training stages, quantization noise remains static throughout the process, lacking the adaptability needed to enhance exploration at critical phases.
3.3 ADAPTIVE QUANTIZATION NOISE IN PARAMETER SPACE
To transform static quantization noise into a dynamic exploration mechanism, we introduce an Adap- rive Quantization Noise (AQN) technique. The core idea is to introduce small set of structured modulation vectors that slightly perturb the otherwise static quantization noise. In our approach, we utilize advanced quantization format, NVFP4
NVFP4 Quantization NVFP4 represents weights using a dual-scaling mechanism: coarse, per- tensor global scaling factor in FP32, Syp32, and fine-grained tensor of block-wise FP8 (F4M3)
5

---

scalers, SEAM. The dequantization of a 4-bit W to the high-precision W follows:
W Dequant W) Spr32 (SEAMS O W)
(6)
where @ denotes block-wise scalar multiplication, broadcasting each scaler in SEAMS to its corre- sponding block of 4-bit weights in W. The quantization noise of each weight matrix, Ac = W-W, the difference between this reconstructed tensor and the original full- full-precision tensor W.
Adaptive Quantization Noise We introduce a noise vector to the static quantized weight. Specifi- cally, for each quantized linear layer, sample stochastic noise vector, Zueisy E RR where d the input dimension the layer. This vector is not fixed but resampled for each forward pass. We define it as: Znnisy E, E (0, I²1), where is a hyperparameter in different training stage governing the noise scale, and e is random vector whose elements are drawn independently from standard Gaussian distribution (Plappert al., 2017). Then the additive noise is defined as:
A = moissy + A maissy (w W
where Ac' is equivalent to the dynamic noise of each weight matrix. In our setting, we freeze the main branch weight and update the low-rank matrix during RI.. The W and W are consistent values. In the early stages, we leverage the inherent quantization noise to enhance the model's exploration capabilities. As training progresses, a gradually reduces following an exponential decay scheduler:
Gend
(k) = a7 ssarrt
(8)
start
where Gaan and Fend represent the initial and final noise levels, k is the current stage, and K is the total interval, which are evenly divided in the training steps (more scheduler comparison in Sec.4.2). For instance, our experiments in GSM8K with total of around 600 training steps, noise injected at 10 evenly spaced intervals, initialized with quantization noise, then from start to o Fend- This approach aims to balance exploration and exploitation (Fox et al., 2015).
Noise Merging While introducing noise vec- tor enables dynamic control over quantization noise, explicitly creating a separate vector for each quantized layer not feasible. First, imposes a burden parameter efficiency, in- creasing memory overhead. Moreover, high- precision noise cannot be directly added to quantized weights, as this would break the com- patibility of our inference kernel designed for NVFP4 x BF16 operations. We propose a sim- ple solution that integrates this noise vector di- rectly into the layer normalization parameters of LLM architectures.
Multi-Head Self-Attention
Feedforward Layer
Zenoion
4
Zerion
Zeries
Zanies
Zyates
Q
V NVEPA
Gate
Up
NNFFA
NVFP4
Down SVP24
Activation
Z Zooiee
Activation
noise
W
RMSNorm
RMSNorm
Figure 6: Deployment scheme of adaptive quanti- zation noise LLMs. Znoise is integrated in Lay- erNorm (e.g., RMSNorm) of each block in LLMs.
X
Z mosss + w) = Zunisy + +X W
By exploiting this equivalency in Eq.9, we subsume the role of Zuoicy into the learnable weight parameter of the LayerNorm operation (e.g. RMSNorm (Zhang & Sennrich, 2019)) that typically follows the scaling after normalization.
RMSNormppisee (x) W Wnoise
N 2i=1 +7 W Wnoise Znoise + w
x
(10)
where w represents the scaling factor of RMSNorm. In this configuration, channel- wise additive noise Zuaisy transfers to row-wise multiplicative noise Zuuaze + 7 of weight (proof provided in Ap- pendix G). Multiplicative noise has been shown to be effective in RL (Pang & Jiang, 2021; Zhang et al., 2025a). Due to the higher sensitivity of RL to multiplicative noise, we initialize the noise level with o stn = 1e-2 to ensure stability.
This approach extends adaptive quantization noise to the layer parameters W.. g. We. W.. W Wastes and W up within each block, these layers directly interact with normalized activations. To align with LLM architectures (Team, 2024; Grattafiori et al., 2024), WG, Wi. and W, share the same RMSNorm, while W gate and W share another (as shown in Fig.6).

---

(a) Performance of Qwwen.5-3BBInstreeee
(b) Performance of Qwen2.5-7BB-nntruuul
Model
W#
Training
GSM8K
Model
W#
Training
GSM8K
BF16
61.2
BF16
76.3
NF4
37.5 3.7
NF4
70.5
MXFP4
59.8 1.44
MXFP4
71.3 -..
NVFP4
59.4
NVFP4
73.4
Qwen2.5-3B
BF16
Full
84.4
Qwen2.5-7B
BFI6
Full
+1.2+
-Instruct
BF16
LoRA
76. 16.1
-Instruct
BF16
LoRA
88. 88.1+111.8
NF4
LoRA
76.1+14.9
NF4
LoRA
85.0
MXFP4
LoRA
73.4+12.2
MXFP4
LoRA
86.4+10.1
NVFP4
LoRA
83.3
NVFP4
LoRA
88.5+12.2
+AQN
83.7+22.8
+AQN
90.8 113.5
Table 1: Qwen2.5 Performance on GSM8K. GRPO algorithm is used to train 3B and 7B models on GSM8K dataset, while "Full" denotes the full-parameter training and "W#" represents the bit-width and data format of weight. + and are compared with original bfloat-16 16 (BF16) models.
Qwen2.5 7B
QwenZ.5 14B
Qwen2.5
Qwen2.5 7B
0.6
6 Full
0.6
6
Aucuray Neward
0,85
AQN
AQN
0.4
LoRA
QcRL
LoRA
AQN
Zacltn
0.35
w/b AQN
Accurry
0.2
0.2
0.2
Zrnse
Training Steps
Training Steps
Training Stops
Truining Steps
50
100
150
200
2.50
50
100 150 200
250
180
Figure 7: Training reward of 7/14B models.
Figure 8: Ablation of AQN on 3/7B model.
4 EXPERIMENT
4.1 EXPERIMENT SETTINGS
RL Training We conducted training experiments using DAPO (Yu et al., 2025) and GRPO (Shao al., 2024) on two prominent mathematical reasoning datasets: GSM8K (Cobbe al., 2021) and BigMath (Albalak et al., 2025). GSM8K comprises 7,500 samples with generation number of 8, while BigMath includes 122,000 samples with a generation number of 16. Both datasets feature problems of medium high difficulty, spanning levels 3 to 5. For GSM8K, we trained 3B and 7B models, whereas for BigMath, we trained 7B, 14B, and 32B models. Specifically, the 7B and 14B models were trained on problems ranging from levels to 5, while the 32B model exclusively trained on the more challenging level 4-5 problems. Training checkpoints were evaluated between 500 and 1000 steps. To account for the sensitivity of Znoise perturbation, we its range from 5e-2 to 5e-4 for dynamic noise estimation. In the main experiments, the LoRA rank is fixed at 32. The speedup tests are performed on a single H100 GPU, while the final evaluated model is trained using 8 H100 GPUs to ensure experimental efficiency on such large-scale data. Detailed hyperparameters and deployment of QeRL are provided in Appendix E and Appendix F.
Backbone Models We conduct experiments Qwen2.5 (Team, 2024) series, using basic without any mathematic data fine-tuning. For weight-only quantization, we applied AWQ (Lin et al., 2024) to MXFP4 and NVFP4 formats. The calibration dataset included 256 sequences, each 2048 tokens 114k (Guha et al., 2025). Weight-only formats also support long, sampled from NVIDIA-HI00 GPUs with the Marlin kernel (Frantar et al., 2024). For inference acceleration
NF4 quantization, we used the default configuration (Dettmers et al., 2023a).
Evaluation Benchmarks and Metrics We focus on several widely used mathematical reasoning benchmarks, including GSM8K (Cobbe et al., 2021), MATH500 (Lightman et al., 2023), AIME 2024/2025 (Li et al., 2024), and AMC 23 (Li et al., 2024), for evaluation. During inference, a temperature 0.6, completion length of 4096, and top-p sampling with p = 0.95. Each data set is evaluated multiple times, and report primarily the average accuracy of sample (Pass @ 1).

---

Model
W#
Training
MATH 500
AIME 24
AIME 25
AMC 23
Averagef
BF16
74.8
9.2
6.6
25.0
28.9
NVFP4
73.7
8.9
3.3
17.5
25.7
BF16
Full
77.4
16.747.5
10.0
45.0420..0
37.3 +8.4
7B
BF16
LoRA
77.042.22
13.3
10.0
42.5
35.7 +6.8
NVFP4
LoRA
76.8 76.842.0
13.74-15
10.0
47.5422.5
37.0
+AQN
77.4 77.4+2.6
15.5
10.0.3.4
42.5
36.4
BF16
78.6
11.3
9.2
45.0
36.0
NVFP4
76.4 2.2
11.2 0.1
8.3 0.9
40.0 -5.0
34.0 2.0
BF16
Full
83.2
20.0
15.1
55.0
43.3
14B
BF16
LoRA
81.0
14.0
13.3 +4.1
52.5
40.2
NVFP4
LoRA
79.4+0.8
16.7+5.4
13.3 13.3
52.5
40.5
+AQN
80.2 +1.6
17.5
12.6
57.5
42.0
BF16
81.4
14.0
10.8
52.5
39.7
NVFP4
80.6
11.3 2.7
10.0
45.0 7.5
36.7-3,0
BF16
Full
84.0
20.0
23.3 +1..5
57 57.545.0
46.2
32B
BF16
LoRA
83.6
16.7+3.7
13.3
55.0
42.2
NVFP4
LoRA
81.6
16.7
15.0
52.5
41.4
+AQN
83.3
16.7+3.7
19.2
63.3
45.645.9
Table 2: Performance across four benchmarks. DAPO algorithm is used to train Qwen2.5--/11/322.. Instruction models on BigMath dataset, while "Full" denotes the full-parameter training.
0.85
0.85
Accuracy Reward
Accuracy Reward
0.7
I Linear Decay
0.65
rank=16
0.55
Exponential Decay
rank=32
Cosine Decay
0.45
rank=64
0.4
Logarithmic Decay
rank=128
0.25
Training Steps
Training Steps
100
100
Figure 9: Comparison of noise schedulers.
Figure 10: Ablation of LoRA rank.
4.2 EXPERIMENT RESULTS
Reasoning Performance As shown in Tab. we report the GSM8k training results of the 3B and 7B models using GRPO. While quantized models exhibit performance degradation compared to BF16, applying PEFT with RL to the 3B model demonstrates that NVFP4 combined with AQN achieves performance of 83.7 from 59.4, surpassing the 76.1 achieved by 16-bit PEFT training and falling only 0.7 points below full-pane hall-parmmeter training. Similarly, for the 7B model, our method outperforms 16-bit LoRA by 1.7 points. Furthermore, compared to QLoRA, our approach improves average accuracy by 7.6 and 5.8 points for the 3B and 7B models, respectively. Tab.2 presents the results on the BigMath dataset for the 7B, 14B, and 32B models trained with DAPO. Across all datasets, QeRL consistently matches or exceeds the performance of 16-bit models trained with LoRA. No- tably, QeRL trains only about %% of the parameters required for full-parameter training while using just 40%-50% of the GPU memory of vanilla LoRA. For the 7B model, QeRL improves the aver- age score from 25.7 (quantized) 36.4, compared 35.7 with vanilla LoRA. Similar trends are observed in the 14B and 32B models, where QeRL consistently outperforms vanilla LoRA across benchmarks, further supporting the conclusion that quantization enhances RL. Remarkably, on the AMC 23 dataset, the 14B model with QcRL achieves 57.5, exceeding 55.0 of full- pala-ereeetter train- ing. In Tab. 13, we report both performance and resource consumption for QeRL and FlashRL on GSM8K.
Reward Visualization In Sec.3.2, we compare the accuracy rewards of quantized LoRA, vanilla LoRA. and full- tallmperameter training under GRPO and DAPO. Fig.7 presents the accuracy reward curves for the 7B and 14B models on the challenging BigMath dataset. Notably, QeRL achieves a rapid reward increase within 200 steps, while vanilla LoRA requires over 500 steps (Appendix H) to
8

---

Training Speedup (Batch Size)
Model
Method
W#
Model Size
2
4
8
LoRA
BF16
15.2 GB
Qwen2.5-7B-Insee
QLoRA
NF4
5.7 GB
x0.8
x0.8
x0.7 1
QeRI.
NVFP4
5.9 GB
x1.5 1
x1.4
x1.2 f
LoRA
BF16
29.6 GB
Qwen2.5-14B Instruct
QLoRA
NF4
10.2 GB
x0.9 x1.4 f
x0.7 J
x0.7 +
QeRL
NVFP4
10.6 GB
x1.2
x1.2 f
Table 3: Memory Saving and Speedup of 7B and 14B models. We report the end-to-end speedup the GRPO process each training step. Each input has a length of 256 tokens, and each max completion length is 2048. More results of other models shown in Appendix J.
Model
Method
Rollout
Reward
Update
Move Model
Total
BF16 LoRA
6.28
0.05
0.30
0.57
7.20
1rnn2.5-7B-Instruee
QLoRA
9.48
0.06
0.74
0.15
10.43
QeRL
4.00
0.05
0.53
0.14
4.75
BF16 LoRA
10.24
0.01
0.51
0.68
11.45
Qwen2.5-14B-Instren
QLoRA
12.30
0.01
1.05
0.21
13.56
QeRL
6.62
0.01
0.96
0.27
7.85
Table 4: Training Time Breakdown Comparison (s second). We report the breakdown time in the GRPO process each training step. Each input has a length of 256 tokens, and each max completion length is 2048.
Throughout (tokens/s)
LoRA
QQORA
QeRL
Throughput (tokens/s)
LoRA OOL
QeRL
80
92.9
2.0x
56.
2.3x
2,2x0
2,2x
513
40
23
R=16
R=32
R=64
20
R=16
R=64
Qwen2.5-14BB 1-nnnt--4B-
Qwenn.5-32B-instrucc
Figure 11: Rollout throughput of 14/32B model. The setting is aligned with Tab. 11 (batch is 1).
show improvement. This finding highlights that the inherent noise introduced by quantized LLMs enhances exploration in LoRA-hased RI., enabling faster reward growth and higher reward targets.
Noise Decay Schedule Fig.9 compares the performance of different noise decay functions for the 3B model: linear, exponential, cosine, and logarithmic decay. While their performance differences are negligible the early training stages, exponential decay achieves more stable improvements later by reducing noise to lower levels. The corresponding decay curves are provided in Appendix H.
Ablation of AQN Using default quantized noise throughout the training limits the exploration in RL. To address this, we propose the AQN. As shown in Fig.8, when we start with the default quan- tized noise and periodically inject additional noise in later stages, the reward curve grows more steadily. Notably, when the reward approaches convergence, AQN effectively expands the model's exploration space, enabling further improvements reward.
Ablation of LoRA Rank Fig. compares the reward curves of the 3B model during QeRL with different LoRA ranks. Specifically, ranks of 16. 32, 64, and 128 exhibit similar trends and reward growth rates, with rank 16 converging slightly faster, making more economical choice.

---

(a) Performance on SafeRLHF-QA.
(b) Performance on Commonsense@A.
Model
W#
Training
Acc.
Model
Wif
Training
Acc.
BF16
78.4
BF16
66.9
Qwen2.5-3B
BF16
Full
87.2+8.8
@wen2.5-3R -Instruct
BF16
Full
79.2--1.33
-Instruct
BFI6
LoRA
83.545..1
BF16
LoRA
73.5.-..6
NVFP4
QeRL
86.3
NVFP4
QeRL
73.2
Table 5: Swen2.5--B-IInstree Performance on SafeRLHF-QA and CommonsenseQA. GRPO algo- rithm is used to train 3B model, while "Acc." denotes the accuracy scores.
4.3 MEMORY SAVING AND SPEEDUP
Tab.3 compares the quantized model sizes and end-to-end RL training speedup of these PEFT meth- ods, with all experiments conducted on a single NVIDIA H100-80GB GPU (NVIDIA, 2023). For 7B and 14B models, both QLoRA (NF4) and QeRL (NVFP4, supported by the Marlin kernel (Fran- tar et al., 2024)) significantly reduce memory usage, shrinking the model sizes to 35%%30% of their 16-bit counterparts. Due to the limitations of NF4 generation speed (Egashira al., 2024), QLoRA slows to 0.7x-0.8% x across different batch sizes. In contrast, QeRL achieves around a 1.2x-1.5x speedup compared BF16 LoRA in end-to-end RL training while drastically reducing memory us- age, and a 1.5x speedup compared to QLoRA, benefiting from the generation speed of long reasoning sequences. This efficiency is particularly evident in RI., where the computational de- mands of long-horizon rollouts emphasize QeRL's advantage. Moreover, detailed breakdown time of QeRI., BF16 LoRA and QLoRA are shown in Tab.
Notably, our speedup measurements are based the average speed during the first 30 steps, where the output token length relatively short. In later stages of training, as the model generates longer outputs, the speed advantage of QeRL becomes even more pronounced. Its dual benefits in memory efficiency and training speed make QeRL highly effective for end-to-end RL workflows, especially in scenarios requiring extensive rollouts. Fig.11 shows rollout performance across various LoRA ranks, with QeRI. achieving over 2x speedups on 14B and 32B models. On other GPU architectures such Ampere,
can still apply QeRL. using slower dequantization schemes but not NVFP4 BF16 kernel. Under this setting, the throughput gains are smaller, but we still observe advantages from the reduced memory footprint and improved exploration behavior brought by QeRL. More efficiency comparisons for other models and settings are in Appendix J.
4.4 MORE RESULTS ON OTHER RL TASKS
In this section, we extend our experiments beyond math reasoning. The training results of Com- monsenseQA (Talmor al., 2019) (dialogue) and SafeRLHF (Ji et al., 2025) (safety-critical) are reported in Tab.5. We observe that safety-critical tasks often require careful step-by hep-bb--tepp analysis to produce safe and accurate responses, so they benefit from maasoning--nhaneee RL. In this regime, QeRL consistently outperforms BF16 LoRA, similar to the gains seen math benchmarks. In con- trast, CommonsenseeA relies more on knowledge recall than on complex reasoning, so the benefit from exploration and feasoee RL is less pronounced.
CONCLUSION
This paper presents QeRL, an efficient training framework for RL on LLMs, which integrates NVFP4 precision quantization with LoRA fine-tuning. The framework is based on the novel ob- servation that quantization can enhance exploration in LoRA-based RL, contrary to findings in SFT. Quantized LLMs not only surpass vanilla 16-bit LoRA training but also approach full full-parameter fine-tuning performance. To address the static nature of quantization noise, we introduce an AQN mechanism, which dynamically adjusts noise during training to enhance RL stability. Extensive ex- periments show that QeRL significantly improves accuracy across models of various sizes compared to both 16-bit LoRA and QLoRA. Additionally, with NVFP4 kernel support, QeRL achieves round 1.5x speedup in end-to-end RI. training while drastically reducing memory usage.
10

---

REFERENCES
Alon Albalak, Duy Phung, Nathan Lile, Rafacl Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, et al. Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models. arXiv preprint arXiv:2502. 17387, 2025.
Shengnan An, Yifei Li, Zeqi Lin, Qian Liu, Bei Chen, Qiang Fu, Weizhu Chen, Nanning Zheng, and Jian-Guang Lou. Input-tuning: Adapting unfamiliar inputs frozen pretrained models. CoRR, 01122003.0311,, 2022.
Roberto L Castro, Andrei Panferov, Soroush Tabesh, Oliver Sieberling, Jiale Chen, Mahdi Nikdan, Saleh Ashkboos, and Dan Alistarh. Quartet: Native fp4 training can be optimal for large language models. arXiv preprint arXiv:2505. 200::2555.16699 14669, 2025.
Mengzhao Chen, Yi Liu, Jiahao Wang, Yi Bin, Wenqi Shao, and Ping Luo. Prefixquant: Static quantization beats dynamic through prefixed outliers in llms. 2024a.
Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, Yu Qiao, and Ping Luo. Efficientqat: Efficient quantization-aware training for large language models. arXiv preprint arXiv:2407.11062, 2024b.
Tianqi Chen, Bing Xu, Chiyuan Zhang. and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv: XXKiv:6044.00 (604.06/74, 2016.
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. In ICLR, 2024c.
Yukang Chen, Wei Huang, Baifeng Shi, Qinghao Hu, Hanrong Ye, Ligeng Zhu, Zhijian Liu, Pavlo Molchanov, Jan Kautz, Xiaojuan Qi, et al. Scaling rl to long videos. arXiv preprint arXiv:2507.07966 2025a.
Zaiwei Chen, Siva Theja Maguluri, and Martin Zubeldia. Concentration of contractive stochastic approximation: Additive and multiplicative noise. The Annals of Applied Probability, 35(2): 1298-1352, 2025b.
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506. 14758, 2025.
Brian Chmiel, Maxim Fishman, Ron Banner, and Daniel Soudry. Fp4 all the way: Fully quantized training of llms. arXiv preprint arXiv:2505.19/13 2025.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shenghang Tong, Saining Xie, Dale Schuurmans, Quoe V. Le, Sergey Levine, and Yi Ma. SFT memorizes, RI generalizes: A comparative study of founda- tion model post-training. CoRR, abs/2501.17161, 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14668, 2021.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan. Huayu Chen, Weize Chen, al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617, 2025.
DeepSeek-AL. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning.
CoRR, abs/2501. 1222225511..188.... 2025.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 O: 8-bit matrix multiplication for transformers scale. NeurIPS, 35:30318-30333 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. In NewrIPS, 2023a.
11

---

Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashk- boos, Alexander Borzumov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantizel repre- sentation for near-lossless Ilm weight compression. arXiv preprint arXiv:2306.03300 03078, 2023b.
Onno Eberhard, Jakob Hollenstein, Cristina Pinneri, and Georg Martius. Pink noise is all you need: Colored noise exploration in deep reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023.
Kazuki Egashira, Mark Vero, Robin Staab, Jingxuan He, and Martin Vechev, Exploiting IIm quan- tization. Advances in Neural Information Processing Systems, 37:41709-41732, 2024.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: A case study on ppo and trpo. In International conference on learning representations, 2019.
Benjamin Eysenbach and Sergey Levine. Maximum entropy rl (provably) solves some robust rl problems. arXiv preprint arXiv:2103.06257, 2021.
Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Rémi Gribonval, Herve Jegou, and Armand Joulin. Training with quantization noise for extreme model compression. arXiv preprint arXiv:2004.07:22 2020.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Os- band, Alex Graves, Volodymyr Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In International Conference on Learn- ing Representations, 2018 URL https:ttp//.. //pp///vpppeeeeeee/// net/ forum htte:/oormmiiddryyeo 2id=rywicPkam
Roy Fox, Ari Pakman, and Naftali Tishby, Taming the noise in reinforcement learning via soft updates. arXiv preprint arXiv: 151.008522.8 2015.
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint 10Xiv:2210.17322 2022.
Elias Frantar, Roberto L Castro, Jiale Chen, Torsten Hoefler, and Dan Alistarh. Marlin: Mixed-precision auto-regressive parallel inference on large language models. arXiv preprint arXiv:2408. 1743, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smymis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reason- ing models. arXiv preprint arXiv:2506 251.::2200001178.8 2025.
Han Guo, Philip Greengard, Eric P Xing, and Yoon Kim. Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. arXiv preprint arXiv:2311. 1.202,,,, 2023.
Hossein Hassani, Roozbeh Razavi-Far, Mehrdad Saif, and Liang Lin. Towards sample-efficiency and generalization of transfer and inverse reinforcement learning: A comprehensive literature review. CoRR, abs/2411. 111122281,, 2024.
Juan Camilo Gamboa Higuera, David Meger, and Gregory Dudek. Synthesizing neural network con- trollers with probabilistic model-based reinforcement learning. In 2018 IEFF/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2538-2544. IEEE, 2018
Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning. Yanning Chen, and Zhipeng Wang. Liger-kernel: Efficient tri- ton kernels for LLM training. In Championing Open-source DEvelopment in ML Workshop @ ICM1.25, 2025. URL https: /etps://ppeeeee //opppeevviieeeeeeeee ee wwwp//eet//otmee wow.fooddddd3s
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
12

---

Wei Huang, Yue Liao, Jianhui Liu, Ruifei He, Haoru Tan, Shiming Zhang, Hongsheng Li, Si Liu, and Xiaojuan Qi. Mixture compressor for mixture-of-experts ilms gains more. arXiv preprint arXiv:2410.06270, 2024a.
Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and Xiaojuan Qi. Billm: Pushing the limit of post-training quantization for llms. arXiv preprint arXiv:2402.04291, 2024b.
Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Xianglong Liu, Luca Benini, Michele Magno, and Xiaojuan Qi. Slim-llm: Salience-driven mixed sixee-precision quantization for large language models. arXiv preprint arXhv:2405.14917, 2024c.
Wei Huang, Yue Liao, Yukang Chen, Jianhui Liu, Haoru Tan, Si Liu, Shiming Zhang, Shuicheng Yan, and Xiaojuan Qi. Mc#: Mixture compressor for mixture-of-experts large models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026.
Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu. O1 replication journey part Surpassing o1-preview through simple distillation, big progress or bitter lesson? CoRR, abs/241 1.16, 2024d.
Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Alex Qiu, Jiayi Zhou, Kaile Wang, Boxun Li, et al. Pku-saferlhf: Towards multi-level safety alignment for Iims with human preference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 31983-32016, 2025.
Srivatsan Krishnan, Maximilian Lam, Sharad Chitlangia, Zishen Wan, Gabriel sandra Faust, and Vijay Janapa Reddi. Quarl: Quantization for fast and environmentally sustain- able reinforcement learning. arXiv preprint 1910.0000 2019.
Alek-
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang. Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tülu 3: Pushing frontiers in open language model post-training. CoRR, abs/2411. 1.1422 2024.
Janghwan Lee, Jiwoong Park, Jinseok Kim, Yongjik Kim, Jungju Oh, Jinwook Oh, and Jungwook Choi. Amxfp4: Taming activation outliers with asymmetric microscaling Seating-point for 4-bit I1m inference. arXiv preprint Kii::2411009900 2024.
Brian Lester, Rami A1-Rfou, and Noah Constant, The power of scale for parameter-efficient prompt tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), EMNLP, pp.
2021.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9):9, 2024.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), ACL, pp. 4582-4597, 2021.
Baohao Liao and Christof Monz. Apiq: Finetuning of 2-bit quantized large language model. arXiv preprint arXiv:2402.051477 2024.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2023.
Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6: 87-100, 2024.
13

---

Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-effiiient fine-tuning is better and cheaper than in-context learn- ing. In NeurIPS, 2022.
Liyuan Liu, Feng Yao, Dinghuai Zhang, Chengyu Dong, Jingbo Shang, and Jianfeng Gao. Flashrl: 8bit rollouts, full power rI, 2025a. URL https: fwngyaongggaoo//oo not.on.iiiii....ooo.m site/ flash-ril
Shih- Yang Liu, Chien- Yi Wang. Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang. Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In ICMI., 2024a.
Xiangyan Liu, Jinjie Ni, Zijian Wu, Chao Du, Longxu Dou, Haonan Wang, Tianyu Pang, and Michael Qizhe Shieh. Noisyrollout: Reinforcing visual reasoning with data augmentation. arXiv preprint arXiv:2504. 1000:250444.055.. 13055, 2025b.
Yifei Liu, Jicheng Wen, Yang Wang, Shengyu Ye, Li Lyna Zhang, Ting Cao, Cheng Li, and Mao Yang. Vptq: Extreme low-hit vector post-training quantization for large language models. arXiv preprint arXiv:2409. 17066, 2024b.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. LIm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305. 17888, 2023.
Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng. Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang, and Ji- Rong Wen. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems. CoRR, abs/2412.09413. 2024.
NVIDIA. Nvidia h100 tensor core GPU architecture overview. http://resoorress..dddaa.cooorss tensor-core, 2023.
NVIDIA. Nvidia blackwell archittture technical brief. auruua-blaackeel---ccchhtttttrr 2024. Accessed: 2025-05-13.
www.:eessssiiiiiiii......oo. com/
OpenAl. Introducing GPT-5. http:////////eaaaaaaaa.a/aaaaaaa 2025. Accessed: 2025-09-21.
aug
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29, 2016.
Bo Pang and Zhong-Ping Jiang. Robust reinforcement learning for stochastic linear quadratic control with multiplicative noise. Trends in Nonlinear and Adaptive Control: A Tribute to Laurent Praly for his 65th Birthday, pp. 249-277, 2021.
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. arXiv preprint arXiv: 10..1106600000 706.01905, 2017.
Open
Project. tion version 1.0.
Compute
Ocp microscaling formats
(mx) specifica-
https: /www www.oopppc.oocoocoooo wttpee/or///doooodde///// cep-microscallee waano--forats--neee 1 -pec-----pe----niie fpaal-------aa..-....-. 2023. Accessed: 2023-09-13.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv: 1:Xi::/506.024330 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06377, 2017.
Yuzhang Shang, Zhihang Yuan, Qiang Wu, and Zhen Dong. Pb-llm: Partially binarized large lan- guage models. arXiv preprint arXiv:2310.00033. 2023.
14

---

Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2338. 13137, 2023.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.0330X 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient RLHF framework. In EuroSys, pp. 1279-1297, 2025.
Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, and Xia Ben Hu. Stop overthinking: survey on efficient reasoning for large language models. CoRR, abs/2503. 16419, 2025.
Yi-Lin Sung, Varun Nair, and Colin Raffel. Training neural networks with fixed sparse masks. In NewrlPS, pp. 14193-24208 2021.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lan- guage Technologies, Volume (Long and Short Papers), pp. 1pp.4149 4158- 2019.
Qwen Team. Qwen2 technical report. arXiv preprint arXiv:2407.1066) 2024.
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#:: Even better IIm quantization with hadamard incoherence and lattice codebooks. arXiv preprint vrXiv:2402.04399 2024.
Albert Tseng. Tao Yu, and Youngsuk Park. arXiv:2502.20588. 2025.
Training Ilms with mxfp4. arXiv preprint
Shangshang Wang, Julian Asilis, Ömer Faruk Akgül, Enes Burak Bilgin, Ollie Liu, and Willie Neiswanger. Tina: Tiny reasoning models via lora. CORR, abs/2504. 15777, 2025.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099, PMLR, 2023.
Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong. Tianjian Ouyang, Fanjin Meng. Chenyang Shao, Yuwei Yan, Qinglong Yang. Yiwen Song, Sijian Ren, Xinyuan Hu, Yu Li, Jie Feng, Chen Gao, and Yong Li. Towards large reasoning models: survey of reinforced reasoning with large language models. CoRR, abs/2501 ,09686, 2025.
Yi Yang, Yueting Zhuang, and Yunhe Pan. Multiple knowledge representation for big data artificial intelligence: framework, applications, and case studies. Frontiers of Information Technology & Electronic Engineering, 22(12):1551-1558, 2021.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: an open-source LLM reinforcement learning system at scale. CoRR, 114/7503..4476.... 2025.
Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. Rptq: Reorder-based post-training quantization for large language models. arXiv preprint arXiv:2304.0100 2023.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-efffcient fine-tuning for transformer-based masked language-modell In ACI., pp. 1-9, 2022.
15

---

Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in neural infor- mation processing systems, 32, 2019.
Hanfang Zhang, Bing-Chang Wang, and Ying Cao. Reinforcement learning solutions to stochas- tic multi-agent graphical games with multiplicative noise. IEEE Transactions Circuits and Systems I: Regular Papers, 2025a.
Jintao Zhang, Jia Wei, Pengle Zhang, Xiaoming Xu, Haofeng Huang, Haoxu Wang, Kai Jiang, Jun Zhu, and Jianfei Chen. Sageattention3: Microscaling fp4 attention for inference and an exploration of 8-bit training. arXiv preprint arXiv:2505. 11594, 2025b.
Jiaqi Zhao, Miao Zhang, Chao Zeng, Ming Wang, Xuebo Liu, and Liqiang Nie. Lrquant: Learnable and robust post-training quantization for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp- 2240-2255, 2024.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang. Jingren Zhou, and Junyang Lin. Group sequence policy op- timization. CoRR, abs/2507. 0os//557718071, 2025.
16

---

APPENDIX
A ETHICS STATEMENT
This work exclusively leverages publicly available open-source datasets that have been previously established and validated in academic research. No new text, video, or audio materials are generated or incorporated as part this study. The datasets utilized are strictly intended for research purposes and are not employed for any commercial applications.
B REPRODUCIBILITY STATEMENT
To ensure the research community replicate findings, this project will be released open- source software. The methodology described in detail in Sec.3, while Sec.4. and Appendix outline the complete training protocols and implementation details, including all hyperparameter settings.
C USE OF LARGE LANGUAGE MODELS
During the preparation of this manuscript, we utilized large language models-GPT-5 (OpenAI, exclssexyy to refine the language, focusing on improving grammar, flow, and tone the sentence and paragraph levels. These tools were not employed to generate ideas, design experiments, or draw conclusions. All technical content, methodologies, and interpretations were independently written, thoroughly verified, and approved by the authors. To minimize the risk of factual inaccu- racies or citation errors, every model-edited sentence underwent human review, and all references were carefully cross-checked with their primary sources. The authors accept full responsibility for ensuring the accuracy and integrity of this manuscript.
D RELATED WORK
Reinforcement Learning for LLMs Recent efforts have focused on enhancing reasoning in LLMs using RL (Min al., 2024; Chu et al., 2025). DeepSeekMath (Shao et al., 2024) im- proves mathematical reasoning by continuing pre-training on matth-nneensive data and introducing Group Relative Policy Optimization (GRPO) (Shao al., 2024). Building this, DeepSeek- R1 (eeepSeek-AA, 2025) demonstrates that RL alone can drive strong reasoning, achieving per- formance comparable proprietary models with large-scale training. Complementary system-level contributions, such as DAPO (Yu et al., 2025), offer an open-source RL framework with a de- coupled optimization strategy, achieving competitive results through simplified training pipeline. GSPO (Zheng et al., 2025) stabilizes RL training and reduces variance through sequence-leve op- timization, proving effective in large-scale mixture-of-experr models. HybridFlow (Sheng et al., 2025) introduces flexible RLHF framework with hybrid control flow and 3D-HybridEnginne Together, these works demonstrate significant progress in advancing LLM reasoning with RL.
Quantization for LLMs Quantization is key technique for compressing LLMs, improving effi- ciency by reducing parameter precision. The most common approach, Post-Training Quantization (PTQ) (Dettmers et al., 2022; Frantar et al., 2022; Xiao et al., 2023; Shao et al., 2023; Lin al., 2024; Chen et al., 2024a; Yuan et al., 2023), transforms pre-trained models cost-effectively without retraining. Vector quantization (Liu et al., 2024b; Tseng et al., 2024: Zhao al., 2024) is also ef- fective type to store fow-precision data, but still facing the speed problem in inference. Recent work has pushed quantization to ultra-low bit-widths while maintaining performance (Huang et al., 2024c; Dettmers et al., 2023b; Shang et al., 2023; Huang et al., 2024b; Liao & Monz, 2024; Tseng et al., 2024; Huang et al., 2024a; 2026), including advancements in Quantization Aware Training (QAT) to improve robustness (Liu et al., 2023; Chen et al., 2024b). Additionally, novel precision formats like NF4 (Dettmers et al., 2023a), FP4 (Tseng et al., 2025: Chmiel et al., 2025). and MXFP4 (Chmiel al., 2025) enable accurate weight representation, achieving high compression with minimal or im- proved accuracy loss. NVFP4 (NVIDIA, 2024) is a groundbreaking 4-bit floating seating-point format in- troduced with NVIDIA's Blackwell GPU architecture. This format expands the idea of compact,
17

---

low-bit "micro" Roating-poiin representations, offering developers enhanced versatility by adding another flexible option for their projects (Zhang et al., 2025b; Castro et al., 2025; Lee et al., 2024).
Efficient Fine-tuning Efficient fine-tuning is pivotal for adapting LLMs with minimal computa- tional cost. LoRA (Hu et al., 2022) pioneered this approach by adding low-rank adapters to frozen weight matrices. DoRA (Liu al.. 2024a) improved upon this by decomposing weight updates into directional and magnitude components, addressing low-rank constraints and enhancing stabil- ity. QLoRA (Dettmers al., 2023a) integrated LoRA with 4-bit quantization further reduce resource usage, while LongLoRA (Chen et al., 2024c) introduced fine-tuning methods for long- context processing. Tina (Wang al., 2025) demonstrated that compact models could gain rea- soning ability through RL with LoRA. Beyond the LoRA family (Hu et al., 2022), other efficient fine-tuning techniques include prompt tuning, prefix tuning, IA3, BitFit, Fisher-masked tuning, and input-tuning (Lester et al., 2021; Li & Liang, 2021; I; Liu et al., 2022; Zaken et al., 2022; Sung al., 2021; An et al., 2022; Guo al., 2023). These advancements underscore the importance of efficient fine-tuning for practical LLM adaptation.
E EXPERIMENT HYPERPARAMETERS
Training Data and Reward Function We trained the Qwen2.5-3B-Instruec, Qwen2.5-7B- Instruct, Qwen2.5-14B-Instuee and Qwen2.5-32B-Inse evaluating reasoning capabilities. Unlike other studies that rely on math-specialized models, we aim to evaluate training performance starting from general-purpose base models. Additionally, QeRI. can be smoothly transferred to other model families, such as the Qwen3 series. For the GSM8K dataset, we primarily trained the Qwen2.5-3B-Instre and Qwen2.5-7B-Instruee models using GRPO, while for the BigMath dataset, focused training the Qwen2.5-7B-Insieee models using DAPO. Specifically, for the 7B and 14B models, selected data with medium high difficulty levels (grades 3-5), and for the 32B model, we used high-difficulty data (grades 4-5). For problem prompts, we append the suffix Solve the follow following math problem step by step. The reasoning process and direct answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., <think> zeasoning process here </think> <answer> answer here </answer>; <think> </answer>,
models, which are widely used for
Owen2.5-144-Instruee
and Qwen2.5-32BBInsstnne
</think> <answer>
RL Training Configuration For both GRPO and DAPO, we use the hyperparameters in Tab.6, without using entropy or KI. losses. For 4-bit training, the learning rate is set to le 1 However, due to the fragile of the BF16 model with LoRA, the learning rate can not be larger than 5e-6, it will collapse in the late training stage.
Hyperparameter
Value
Optimizer
AdamW-Sbit
Policy learning rate
1e (QeRL., QLoRA) 5e (LoRA)
Training batch size
128
Samples per prompt
8 (GSM8K) / 16 (BigMath)
Policy updates per rollout
4 (GSM8K, off-policy) / (BigMath, on-policy)
Max response length
4096 (GSM8K) / 8192 (BigMath)
Rollout temperature
1.0
Clip range élow thigh
0.2, 0.28
Noise range Zstant Zend
le-2, 5e-4
Table 6: Hyperparameters of GRPO and DAPO training
F DEPLOYMENT OF QERL
In Algorithm 1, we provide detailed explanation how QeRL is deployed within the GRPO framework. During the steps in stage 0, the added noise is O, where only quantization noise
18

---

Algorithm Deploy GRPO with QeRL and Adaptive Quantization Noise
Input NVFP4 policy model Tgg reward function rs: task prompts D; hyperparameters: LoRA rank, LoRA alpha: number of stages dsun, dendd
1: policy model
2: for iteration = 1, ...., I do
3: reference model Fref f-
4: for step I...., M do
Divide total steps into K equal stages: steps per stage [M/K]
6:
Determine current stage k: &
step
stops per augo
k
7:
Set noise level a +
otherwise (exponential decay)
Jean
Sample a batch Dr from
9:
Update the old policy model with AQN: Thatd f ne + M00 o²) Sample G outputs {or Compute rewards {r:) for each sampled output by running Te
Neas g) for each question q D.
11:
12:
Compute A.,, for the f-th token of o, through group relative advantage estimation. for GRPO iteration = I.... do
13:
14:
Update the policy model me by maximizing the GRPO objective (Equation 3)
for end for
16:
17:
end for
Output
effects. At stage is initialized to suats and by the final stage (K-1) a gradually transitions to start- This progressive adjustment of noise ensures structured and controlled exploration process throughout the training stages, balancing stability and exploration effectively.
G PROOF OF NOISE SHARING
In this section, we further demonstrate the effectiveness of the noise-sharing operation proposed in 10. detailing the process by which additive noise is transformed into multiplicative noise. With AQN, input of each block follows:
Znoise
RMSNorm.oceed (X)
w I) @ RMSNorm(X),
(11)
where RMSNorm(-) (-) denotes the vanilla RMSNorm operation and is the original scaling factor in RMSNorm(-). The clement-wise multiplication (@) will be auto-broadcas during computing.
Then, the operation of the following linear computation is defined as:
Znoise +) @ RMMNNrr(XX) w RMSNorm(XX Znoise
+ 1)T W),
(12)
W
Thus, the additive Gaussian noise, when incorporated into the noise-sharing mechanism of Layer- Norm, can be equivalently regarded as multiplicative Gaussian noise (denoted as (Zuaice + I)) and applied row-wise to the weight matrix W. Since RMSNorm is only applied to the inputs of each attention block and feed-forward network (FFN) block, this mechanism ensures that the Q. K. and V matrices in the attention block share the same noise, while the down and up layers in the FFN block also share a single, identical noise set. This noise-injection strategy avoids disrupting the multiplication kernels of NVFP4 and BF16 in QcRL or introducing additional matrix multiplication operations.
w
Both additive and multiplicative noise have been shown positively contribute to exploration in RL (Plappert et al., 2017; Higuera et al., 2018; Chen et al., 2025b). However, multiplicative noise tends be sensitive, especially in deep networks like LLMs. To address this, initialize the noise standard deviation (σ) to 1e-2, which is smaller than the typical 1e-1 used in traditional noise-based networks.
19

---

Problem Level: 3-5
Problem Lvvel: 4-5
0.6
marlit Mmnl Wyya"
Accuracy Reward
0.5
QCRL
Accuracy Reward
0.5
0.4
LoRA
0.4
0.3
0.3
0.2
QcRL
02
LoRA
Training Steps
Training Steps
300
400
500
0
50
100
1.50
200
250
Figure 12: Training reward of 7B model.
Figure 13: Training reward of 32B model.
H ADDITIONAL EXPERIMENTS OF TRAINING
Training Rewards of Different Model Fig. 12 and Fig.13 13 further compare the performance of QeRL and 16-bit LoRA training on complex reasoning datasets. Fig. 12, we present the training rewards of the Owen2.5-7B-Instrwe
model on the Big.athh dataset with difficulty levels ranging from to 5, as an extension of Fig.7. Leveraging the exploration benefits of QeRL quantized models, a rapid increase in reward is observed after approximately 200 steps, whereas 16-bit LoRA requires over 500 steps to achieve a similar rise. Meanwhile, as shown in Fig. we trained the Qwen2.5-32B -nntttr in reward growth between QeRL. and LoRA is less pronounced in the 32B model compared to the smaller 3B, 7B, and 14B models, QeRL still consistently performs better than LoRA.
Instruct model the highest difficulty data (levels 4-5). Although the difference
More Experiments of Entropy As an extension of Fig.5, illustrates the entropy curve of the Qwen2.5- 14B-Instruct model at various training steps. Notably, the entropy of QeRI. remains consistently higher than that of LoRA throughout the RL process, particularly during the initial steps. This observation highlights the advantage of QeRL in promoting exploration, higher entropy in- dicates a broader search of the solution space. The in- creased exploratory capacity facilitated by quantization appears to enable the model to navigate complex environ- ments more effectively, ultimately supporting improved optimization. These results further validate the role of quantization in enhancing the exploration- exploitation balance in RL tasks.
0.5
Accuracy Reward
0.4
0.3
QeRI.
LoRA
0.2
Training Steps
300
400
500
Figure 14: Entropy in RL steps.
Noise Scheduler Fig. 15 illustrates the noise scheduler employed our experiments, show- ing four distinct decay strategies: linear, expo- nential, cosine, and logarithmic. The scheduler adjusts the noise level in 10 stages to guide the training process. The linear decay method re- duces noise uniformly across stages, ensuring consistent rate of change. The exponential de- cay rapidly decreases the noise at the beginning and uses smaller noise scales in later stages, which we found effective for achieving stable and higher rewards in later stages of training. The cosine decay follows smooth oscillatory pattern, gradually reducing noise with cosine curve, whereas the logarithmic decay decreases noise sharply in early stages and stabilizes later ones. Among these, we chose the exponential decay strategy due to its ability to maintain smaller noise scales during the later stages, resulting in more stable and higher reward curve. This flexibility in controlling noise levels plays a critical role in balancing exploration and convergence during training.
0.06
Linear Decay
Noise Scale (a)
Exponential Decay
0.04
Cosine Decay
Logarithmic Decay
0.02
0
1 2
3
5
10
Training Steps
Figure 15: Noise curve of different schedulers.
We use start noise levels in the range [le-1, 1e-2] and end noise levels in the range [le-3, 5c-4]. These are typical scales in traditional RI,, where they provide stable exploration without strongly
20

---

0.95
Asturacy Neward
0.8
2.0 Speed Up
Accerncy Reward
QcRL(1mSe-
LoRA(I-3o4)
0.5
0.4
www.
Training Steps
0.2
Training Steps
0
50
100
150
200
Figure 16: Ablation of learning rate in QeRL
Figure 17: Ablation of learning rate in LoRA (ewen2.5-7B-Instmee
Sween..--BBInstruue
Model
Setting
GSM8K
MATH500
76.3
74.8
Full
91.2
77.4
+ AQN
91.4
77.2
BFI6 LoRA
88.1
77.0
+ AQN
89.7
77.8
Qwen2.5-7B-Instruct
MXFP4 LoRA
86.4
+ AQN
88.1
NF4 LoRA
85.0
+ AQN
85.7
NVFP4 LoRA
88.5
76.8
+QQ
90.8
77.4
Table More Ablation of ANQ.
Model
start
end
GSM8K
le-1
1e-3
82.5
Bren2.5-3B-Instruee
le-2
1e-3
83.7
le-1
5e-4
83.1
le-2
5e-4
83.7
Table Comparison of Different Noise Scale.
disrupting the policy. We observe similar behavior in LLM RL: within this range, training remains stable and performance does not change sharply. To make this more quantitative, we added small sensitivity study on GSMSK with Qwen2.5-3BB-nsttree reasonably robust within this band, with all settings giving stable training and similar performance. We also notice that too large noise (above le-1) can start to hurt performance, so we keep the start noise at below le-1 in all our main experiments.
Tab. These results show that AQN is
Across full-parameter tuning and different quantization formats (NF4, MXFP4, BF16) with LoRA, adding AQN consistently gives modest gains, which is consistent with parameter-noise based explo- ration in (Fortunato et al., 2018; Plappert et al., 2017). We also observe that the relative improvement larger in the LoRA setting. Our interpretation is that full-parameter RL already fit the data well, leaving less headroom for additional exploration, whereas in low-rank (LoRA) updates, AQN- induced exploration provides stronger benefit.
ADDITIONAL ABLATION STUDY
Ablaioon of Learning Rate We examine the impact learning rate variations on the performance of quantized models compared 16-bit models. As illustrated in Fig. 16 and Fig. 17, with relatively small learning rate of 5e-6, QeRL marginally outperforms LoRA, achieving a reward close to 0.95. When the learning rate is increased to 3e-5, the larger update magnitude in the adapter results in faster reward growth and quicker model convergence. However, 16-bit models, the excessive update magnitude leads to instability, often causing the training process to collapse. In contrast,
21

---

Throughput (Tokens/s)
E2E RL Speedup
Method
W#
Model Size
BS#
Rollout Phase
Speedup
w/o GC
w/ GC
LoRA
BF16
6.2 GB
2
151.2
QeRL
NVFP4
2.8 GB
2
157.0
x1.0
x1.1
x1.0
LoRA
BF16
6.2 GB
8
2226.3
QeRL
NVFP4
2.8 GB
8
2271.4
x1.0
x1.1
x1.1
Table 9: Memory Saving and Speedup of Qwen2.5-3B-Instru. Model. The table reports the throughput (tokens/s) for the rollout phase under two batch size settings (2 and 8). Each input has length of 256 tokens, and each max completion length is 2048. "W#" denotes the data for- mat, "BS#" is the number of batch size, and "E2E" denotes the end-to-end speed of GRPO training. "GC" denotes gradient checkpointing.
Throughput (Tokens/s)
E2E RL Speedup
Method
W#
Model Size
BS#
Rollout Phase
Speedup
w/o GC
w/ GC
LoRA QeRL
BF16
15.2 GB
2
115.4
NVFP4
5.9 GB
2
151.6
x1.31 f
x1.2
x1.21
LoRA
BF16
15.2 GB
8
1641.1
QeRL
NVFP4
5.9 GB
8
2091.8
x1.3
x1.1
x1.1
Table 10: Memory Saving and Speedup of Owen2.5-7B-Inntten Model. The table reports the throughput (tokens/s) for the rollout phase under two batch size settings (2 and 8). Each input has length of 256 tokens, and each completion length is 2048. "W#" denotes the data for- mat, "BS#" is the number of batch size, and "E2E" denotes the end-to-end speed of GRPO training. "GC" denotes gradient checkpointing.
Throughput (Tokens/s)
E2E RL Speedup
Method
W#
Model Size
BS#
Rollout Phase
Speedup
w/o GC
w/ GC
LoRA QeRL
BF16
29.6 GB
2
65.4
NVFP4
10.6 GB
2
95.3
x1.3 1
x1.41
x1.41
LoRA
BF16
29.6 GB
8
737.2
OOM
QeRL
NVFP4
10.6 GB
8
1091.1
x1.5 f
OOM
x1.3 f
Table 11: Memory Saving and Speedup of Swwe2.5-14BB-nnstruc Model. The table reports the throughput (tokens/s) for the rollout phase under two batch size settings (2 and 8). Each input has length of 256 tokens, and each max completion length is 2048. "W#" denotes the data format, "BS#" is the number of batch size, and "E2E" denotes the end-to-end speed of GRPO training. "GC" denotes gradient checkpointing.
QeRL demonstrates remarkable robustness to larger learning rates due to the presence of NVFP4 quantization noise, which helps stabilize updates. This robustness enables QeRL to maintain stable training under high learning rates, achieving reward growth rate nearly twice as fast as the 16-bit model. These results underscore QeRL's superior adaptability and efficiency, particularly in challenging training scenarios with high learning rates.
J MORE EFFICIENCY EXPERIMENTS
Tab.9, Tab. 10, Tab, 11, and 12 provide additional speed benchmarks for the Qwen2.5-3B- Instruct, 1een2.5-7B-Instruee Qwen2.5-14B 14B--I4tBBuctt... and Swee2.5-32B-Instruu models, evalu- ated under batch sizes and 8. For the 3B and 7B models, we did not enable memory-efficient techniques such gradient checkpointing (Chen al., 2016) Liger loss (Hsu et al., 2025) in
22

---

Throughput (Tokens/s)
E2E RL Speedup
Method
W#
Model Size
BS#
Rollout Phase
Speedup
w/o GC
w/ GC
LoRA
BF16
62.3 GB
2
34.0
OOM OOM
OOM
QeRL
NVFP4
20.7 GB
2
60.0
x1.8
10.6 s/step
LoRA
BF16
62.3 GB
8
344.3
OOM
OOM
QeRL
NVFP4
20.7 GB
8
688.2
x2.0
OOM
12.2 s/step
Table 12: Memory Saving and Speedup of Qwen2.5-32B-Instruu Model. The table reports the throughput (tokens/s) for the rollout phase under two batch size settings (2 and 8). Each input has a length of 256 tokens, and each max completion length is 2048. "W#" denotes the data format, "BS#" is the number of batch size, and "E2E" denotes the end-to-end speed of GRPO training. "GC" denotes gradient checkpointing.
Model
Method
Rollout W#
Memory
GSM8K
Baseline
76.3
Qwen2.5-7B-Instruet
FlashRI. (w TIS)
FP8 NVFP4
70 GB
89.2
QeRL
15 GB
90.8
Table 13: Comparison of QeRL and FlashRL.
Model
BF16 (Tokens/s)
Model
NVFP4 (Tokens/s)
Rank 16
Rank 32
Rank 64
Rank 16
Rank 32
Rank 64
3B
151.2
148.8
138.6
3B
157.0
153.1
140.0
7B
115.4
113.2
108.3
7B
151.6
149.9
137.7
14B
65.4
63.1
61.2
14B
95.3
92.9
86.0
32B
34.0
33.3
31.9
32B
58.0
56.0
51.3
Throughput under different LoRA ranks in the rollout stage. We test the tokens/s for each model in the vLLMM engine, and the setting is aligned with Tab.11. We set the batch size 1.
Table 14:
order to maximize training speed. However, due to the substantial size of the 14B and 32B models and the computational overhead introduced by importance sampling with gradients during RL train- ing, we employ gradient checkpoint to accelerate computation. For training on GPUs with smaller memory capacity, enabling gradient checkpointing is recommended to reduce memory usage, al- though this may come at the cost of slower overall training speed. During the rollout phase, the precision of NVFP4, optimized by the Marlin kernel (Frantar et al., 2024), demonstrates signifi- cant acceleration, achieving speeds of 1.0 to 2.0x. In particular, performance gains become more pronounced as model size increases, with the 32B model achieving up 2.0x speedup. This indicates that NVFP4's advantages are particularly impactful for large-scale models, where compu- tational demands are higher.
In end-to-end RL efficiency evaluation, we report the per-step latency of GRPO training, defined as the wall clock time to complete an optimization step including rollout generation, log-probability computation, and parameter updates. We benchmark with rollout batch sizes of 2 and 8 while fixing the maximum input length to 256 tokens and the maximum completion length 2,048 tokens. For fairness, we match the vL.LM memory budget between BF16 and NVFP4 variants by setting the same gpu memory utilization in the engine: 0.20 for -wen2.5-3BBIInttru
0.30 for 7B, 0.45 for 14B, and 0.40 for 32B (the latter to enable single-GPU training). Under these controlled settings, the E2E latency reductions mirror the rollout phase acceleration and become more pronounced as the model size grows, with the largest gains observed Qwen2.5-14B Instrucct.
Additionally, 14 provides a comparison of inference speeds between 16-bit and NVEP4 main models across various LoRA ranks. NVFP4 consistently outperforms 16-bit models in terms of speed at all adapter ranks, showcasing its ability to maintain efficiency across diverse configura- tions. However, the rank increases, both NVFP4 and 6F16 experience gradual decline in rollout
23

---

speed within the vLLM engine, likely due to the increased computational overhead associated with higher ranks. Despite this, NVFP4 continues to demonstrate superior performance, highlighting its robustness and adaptability for both scall-scale and large-scale setups. These findings underscore NVFP4's potential to optimize inference efficiency, particularly when combined with advanced ker- nels and varying adapter configurations.
K LIMITATION ANALYSIS
We have demonstrated that method, QeRL, achieves superior performance in RL training for LLMs compared to 16-bit vanilla LoRA training. Additionally, QeRL matches the accuracy of 16- bit full-parameter reinforcement fine-tuning while delivering over 2x training speedup relative to both vanilla LoRA and QLoRA. However, since RL for LLMs inherently demands significantly greater computational resources than SFT, our experiments, conducted on model sizes ranging from 3B to 32B, do not yet establish whether QeRI. can maintain the same level of performance for mod- els exceeding 70B parameters, leaving that investigation for future work. Another limitation that RL training often requires tens or even hundreds of hours, and while we have provided comprehen- sive evaluations on reasoning benchmarks such as GSM8K, MATH 500, AIME 24, AIME 25, and AMC 23, we did not extend our evaluations other benchmarks or data types, such as code, to general-purpose purpose language tasks unrelated to reasoning. Nevertheless, our technique can be seam- lessly adapted richer and more diverse training datasets. We encourage the community to explore and apply this method to a broader range of tasks in future research.
24


Reviews:
[{"review_id": "lTw4ptLKkN", "reviewer": "Reviewer_g8xh", "summary": "This paper introduces QeRL, a framework that combines NVFP4 quantization with Low-Rank Adaptation (LoRA) for efficient training of large language models in reinforcement learning. The key insight is that quantization noise can enhance exploration during RL (unlike in supervised fine-tuning, where it's detrimental), leading to both improved efficiency and better performance. The authors propose an Adaptive Quantization Noise (AQN) mechanism that dynamically adjusts noise throughout training. Experiments on mathematical reasoning benchmarks (GSM8K, MATH500, AIME, AMC) demonstrate a 1.5× speedup in the rollout phase, the ability to train 32B models on a single H100 GPU, and performance matching or exceeding that of full-parameter fine-tuning on 7B models.", "questions": "Could you provide variance/error bars for multiple runs of GSM8K/MATH500?\n\nCan you include a direct comparison with FlashRL to quantify gains beyond precision-mismatch correction?\n\nHow sensitive is AQN to the start/end noise levels and decay schedule?\n\nDid you try stabilizing BF16 LoRA with smaller ranks or gradient clipping so it can use 1e-5 LR?\n\nPlease release a concise 32B single-GPU recipe (context length, microbatching, KV cache type, LoRA rank, and optimizer).\n\nAny early results on code or safety-oriented tasks to test AQN’s generalization? Can the authors provide more quantitative discussion about the failure modes of AQN or when exploration becomes detrimental, e.g., in very large models or with more complex tasks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Practical impact: Enables RL training of large models (up to 32 B) on a single GPU with significant speedups.\n Exploration insight: Demonstrates that adaptive quantization noise can raise policy entropy and aid reward learning. \nStrong empirical results: Matches or exceeds 16-bit LoRA / QLoRA on math reasoning tasks. \nGood ablation coverage: Studies noise schedules and LoRA ranks; efficient integration using Marlin kernels.", "weaknesses": "Positioning and comparison to recent quantized RL and post-training quantization work: While the paper surveys relevant literature, it does not cite or compare with several recent approaches critical to advances in quantized RL or LLM quantization. For instance, Quantized Reinforcement Learning (QuaRL) [Krishnan et al., 2020, 2022] and newer post-training quantization frameworks (RPTQ, VPTQ, LRQuant, PREFIXQUANT, QuIP$). This makes it challenging to position the core empirical claims and efficiency gains in relation to the current state-of-the-art.\n\nLimited statistical rigor: No multi-seed averages or error bars; robustness to randomness is unknown.\n\nScope restricted to math reasoning: No results for code, dialogue, or safety-critical RL tasks.\n\nMissing FlashRL baseline: Discussed in related work but not compared directly.\n\nHyperparameter fairness: Learning-rate differences may partly explain improved stability; further tuning of BF16 LoRA could close the gap.\n\nHardware specificity: To the best of my knowledge, NVFP4 / Marlin optimizations are NVIDIA-centric; portability to other accelerators is unclear.\n\nPotential overstatement of claims: In the introduction and conclusion, there are hints of broad generalization (e.g., “our technique can be seamlessly adapted to richer…datasets”, p. 21) without empirical demonstration. While acknowledging the focus on math reasoning, a slightly more tempered framing would match the current evidence base.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bBsGGnyvAD", "reviewer": "Reviewer_a31R", "summary": "This paper introduces QeRL, a novel framework that integrates NVFP4 quantization with LoRA to enhance the efficiency and effectiveness of RL for LLMs. The work is motivated by the significant computational bottlenecks—high memory usage and slow rollout speeds—inherent in RL training for LLMs. While existing approaches like QLoRA address memory concerns, they often introduce slowdowns and are not designed to leverage the unique dynamics of RL. QeRL's primary innovation lies in its dual approach: it achieves computational efficiency while also demonstrating, for the first time, that the quantization error (or \"noise\") can be a powerful, implicit mechanism for enhancing exploration in RL.\n\nTo overcome the limitation of static quantization noise, the paper further proposes an Adaptive Quantization Noise (AQN) mechanism. AQN dynamically injects and schedules noise, integrated in a zero-overhead manner into LayerNorm layers, to balance exploration and exploitation throughout training.\n\nThe experimental validation is extensive and convincing. The authors demonstrate that QeRL achieves over 1.5x speedup in the rollout phase and reduces memory usage by 50-60% compared to 16-bit LoRA, enabling the training of a 32B parameter model on a single H100 GPU. Crucially, QeRL does not sacrifice performance for efficiency; it consistently matches or surpasses the performance of 16-bit LoRA and often reaches the performance ceiling of full-parameter fine-tuning on challenging mathematical reasoning benchmarks like GSM8K and MATH.", "questions": "1. The exponential scheduler for AQN was chosen after comparing several options. Could you elaborate on the sensitivity of the performance to the key hyperparameters of AQN, such as $\\sigma_{start}$ and $\\sigma_{end}$? Was the choice of 10 evenly spaced intervals for noise adjustment based on empirical tuning, or is there a heuristic for setting this based on the total number of training steps?\n\n2. While the paper provides a strong empirical and intuitive explanation for how quantization noise increases entropy and aids exploration, could you elaborate on the theoretical connection between the specific properties of NVFP4 quantization noise (e.g., its distribution and magnitude) and established RL exploration theory? A deeper formal analysis could further solidify the foundational contribution of this work.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Novel and Counter-Intuitive Core Insight: The most significant strength of this paper is its conceptual leap: reframing quantization error from a detrimental artifact (as it is often viewed in SFT) into a beneficial exploratory force in RL. The careful analysis in Figures 3 and 5, showing the entropy increase and its correlation with improved reward growth, provides solid evidence for this claim.\n\n2. Holistic and Practical Framework (QeRL + AQN): The paper builds a complete, end-to-end framework. The combination of high-performance NVFP4 quantization with LoRA is a pragmatic solution to the memory and speed problem. The introduction of the Adaptive Quantization Noise (AQN) mechanism is a clever innovation that addresses the inherent limitation of static noise. Significantly, the noise vector can be elegantly merged into LayerNorm to avoid extra computational cost. This demonstrates a deep understanding of both algorithmic needs and system-level constraints.\n\n3. Extensive and Compelling Empirical Evaluation: The authors conduct a thorough ablation study across multiple model sizes, two  RL algorithms (GRPO and DAPO), and several challenging mathematical benchmarks (GSM8K, BigMath). On the efficiency front, QeRL achieves substantial memory reduction, shrinking model sizes to 25-30% of their 16-bit counterparts, while also providing a 1.2x to 1.5x end-to-end training speedup and a greater than 2x speedup in the rollout phase for larger models. Crucially, these efficiency gains do not come at the cost of performance; QeRL consistently achieves superior or comparable final accuracy and faster reward convergence compared to 16-bit LoRA and QLoRA, often matching the high-performance benchmark set by full-parameter fine-tuning.", "weaknesses": "1. Limited Task and Domain Generalization: The empirical validation, while extensive, is heavily focused on mathematical reasoning tasks (GSM8K, MATH, AIME, AMC). While these are excellent proxies for complex, multi-step reasoning, it remains an open question whether the benefits of QeRL generalize equally well to other domains such as coding.\n\n2. Dependency on Hardware and Kernels: The significant speedups of QeRL are contingent on the efficient NVFP4 format and the supporting Marlin kernel, which are tied to recent NVIDIA architectures (Hopper, Blackwell). This might limit immediate adoption for researchers without access to the latest hardware. A brief discussion on the performance or feasibility on more widely available hardware (e.g., Ampere) would be helpful for context.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mcgbcfuTyC", "reviewer": "Reviewer_9ptM", "summary": "This paper proposes QeRL, a quantization-aware RL framework for LLM reasoning. The method aims to improve RL efficiency by combining NVFP4 quantization with LoRA and an auxiliary “Adaptive Quantization Noise (AQN)” mechanism to encourage exploration. Experiments report faster rollouts, reduced memory usage, and higher exploration entropy compared to QLoRA-based baselines. The topic is relevant, and the goal of improving RLVR efficiency is valuable for the community.\n\nHowever, several aspects of the work require clarification, especially around the efficiency claims, the validity and generality of AQN, and the justification for the hypothesis that quantization inherently encourages exploration. These issues seem addressable, but resolving them is important for the credibility of the contribution.", "questions": "1. **Baseline consistency in efficiency claims**\n    The paper reports 1.5–2× speedups and significant memory savings, but these are measured against different baselines (speed vs QLoRA/NF4, memory vs BF16 Full). For example, in the **Conclusion (Line 485)**, the paper states: *“QeRL achieves around a 1.5× speedup in end-to-end RL training while drastically reducing memory usage.”* This phrasing may overstate the benefit of QeRL because it mixes different baselines into a single claim.\n   To ensure clarity and fairness, efficiency results should be reported against **BF16 Full** and **BF16 LoRA**, and each speedup claim in the abstract, Section 4.3, and the conclusion should explicitly name the corresponding baseline rather than using an unqualified number.\n\n2. **AQN ablation to validate its general effectiveness**\n   AQN is only evaluated under the NVFP4 + LoRA setting. To demonstrate that AQN is a generally useful exploration mechanism rather than a component that compensates for limitations specific to quantized LoRA, its evaluation should also include **Full-parameter RL** and cover **multiple quantization formats**. This would more convincingly establish the generality and independent contribution of AQN.\n\n3. **Justification for “Quantization Encourages Exploration”**\n    The evidence currently shows quantized+LoRA > BF16+LoRA (Section 3.2), which may reflect LoRA limitations rather than quantization benefits. Could you isolate the effect of quantization by comparing **Quantized Full RL vs BF16 Full RL**, and potentially provide analysis on *why* quantization increases sampling entropy?\n\n4. **Reporting time cost in main results and clarifying baselines for efficiency claims**\n    Table 1 and Table 2 currently report only mathematical performance metrics, without the corresponding time cost. Since efficiency is a core claim of the paper, these tables should also include **Rollout Time** or the **End-to-End Training Time** for each configuration, so that performance and efficiency can be evaluated together rather than in isolation.\n\n   In addition, the paper includes efficiency statements that do not explicitly state the baseline, which may lead to misinterpretation. For example:\n\n   - **Section Abstract (Line 20–21):** “over 1.5× speedup in the rollout phase”\n   - **Section Conclusion (Line 484–485):** “around a 1.5× speedup in end-to-end RL training while drastically reducing memory usage”\n\n   Both statements implicitly compare against **QLoRA**, but this is not mentioned. These speedup claims should explicitly state the baseline to avoid mixing different reference points and to ensure transparent and fair interpretation.\n\n5. **End-to-end speed vs rollout-only speed**\n   Since RLVR training involves rollout, reward, logprob, update actor, and additional overheads, rollout speed alone does not capture end-to-end efficiency. Reporting full training wall-clock time would provide a more complete picture of the practical gains.\n\n6. **Clarifying the ~2× speedup for 14B–32B models**\n   For the larger models, it would be helpful to clarify whether the baseline was affected by KV cache exhaustion. If the 14B/32B BF16 baseline was already near the memory boundary or encountering KV cache overflow, the slowdown may be caused by resource bottlenecks rather than reflecting the intrinsic advantage of QeRL. In such cases, restricting the comparison to a single GPU may not represent a realistic or fair setting, since practitioners would not normally train 32B-scale RL models on a single GPU. A comparison under a configuration where the baseline does not suffer from KV cache exhaustion would provide a more accurate and meaningful assessment of the true speedup.\n\n7. **Typos / Minor Issues**\n\n   L235–236: quotation formatting should be corrected (''flatter'' → ``flatter''). The same issue also appears on **L238–239** for ''optimal''.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper addresses an important challenge in RL for reasoning models, where rollout latency and memory are major bottlenecks. The attempt to unify quantization with RL training and exploration is timely. The empirical results show promising improvements under quantized LoRA settings, and the paper is generally easy to follow, with well-organized experiments and clear motivation for reducing RL compute.", "weaknesses": "While the paper targets an important problem and presents promising results under quantized LoRA settings, several aspects of the work require further clarification to validate the core claims. In particular, the empirical support for the efficiency gains, the generality of AQN, and the justification for “quantization encourages exploration” are not yet fully convincing. The experimental design could be more comprehensive to isolate the effect of quantization from LoRA, and additional baselines would strengthen the contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FgcPRsFhTi", "reviewer": "Reviewer_dYgm", "summary": "This paper introduces QeRL, a quantization-enhanced reinforcement learning framework for large language models (LLMs). QeRL integrates NVFP4 quantization with Low-Rank Adaptation (LoRA) to address the resource-intensive nature of RL training for LLMs, specifically targeting substantial GPU memory and long rollout durations.\n\nOne of the key findings is that quantization noise, when controlled precisely, can actually benefit RL by increasing policy entropy and enhancing exploration, leading to the discovery of better strategies. To further optimize this, QeRL incorporates an Adaptive Quantization Noise (AQN) mechanism that dynamically adjusts noise during training.", "questions": "please refer to the weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Improved Efficiency and Resource Utilization: QeRL directly tackles the substantial resource demands of RL for LLMs by combining NVFP4 quantization with LoRA. \n\n- Enhanced Exploration through Quantization Noise: Contrary to conventional understanding that quantization noise degrades training, this paper demonstrates that controlled quantization noise can actively benefit RL. By increasing policy entropy, it fosters better exploration and the discovery of more robust strategies, especially with the adaptive quantization noise (AQN) mechanism.\n\n- Superior Performance and Robustness: QeRL not only accelerates training but also achieves faster reward growth and higher final accuracy compared to existing methods like 16-bit LoRA and QLoRA.", "weaknesses": "- Though effective in specific tasks, whether LoRA based RL post training can achieve comparable performance in a wide range of tasks, is still under discussion in both academia and industry.  I am wondering whether QeRL can also be workable in full-parameter fine tuning?\n\n- The ablation of AQN, i.e., Figure 8 indicates that the curves with and without AQN are close to each other, will the authors give more experiments to demonstrate the effectiveness of AQN?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)