QuRL: Low-Precision Reinforcement Learning for Efficient Reasoning
Authors: Yuhang Li
Reena Elangovan
Xin Dong
Priyadarshini Panda
Brucek Khailany
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Quantization
Primary area: foundation or frontier models, including LLMs
TL;DR: We develop a lossless quantized reinforcement learning framework for LLM reasoning

Abstract: Reinforcement learning with verifiable rewards (RLVR) has become a trending paradigm for training reasoning large language models (LLMs). 
However, due to the autoregressive decoding nature of LLMs, the rollout process becomes the efficiency bottleneck of RL training, consisting of up to 70\% of the total training time. 
In this work, we propose Quantized Reinforcement Learning (QuRL) that uses a quantized actor for accelerating the rollout. 
We address two challenges in QuRL. First, we propose Adaptive Clipping Range (ACR) that dynamically adjusts the clipping ratio based on the policy ratio between the full-precision actor and the quantized actor, which is essential for mitigating long-term training collapse.
Second, we identify the weight update problem, where weight changes between RL steps are extremely small, making it difficult for the quantization operation to capture them effectively. 
We mitigate this problem through the invariant scaling technique that reduces quantization noise and increases weight update.
We evaluate our method with INT8 and FP8 quantization experiments on DeepScaleR and DAPO, and achieve 20% to 80% faster rollout during training.

Published as a conference paper at ICLR 2026
QURL: EFFICIENT REINEORCEMENT LEARNING WITH QUANTIZED ROLLOUT
Yuhang Reena Elangovan", Xin Dong', Priyadarshini Panda". BBucek Khailany
1 NVIDIA Research
Yale University
3 University of Southern California
yuhang. lilyllll.eedee.. edu, (relangovan, xdong, byhailaayyeeviddia.com
ABSTRACT
Reinforcement learning with verifiable rewards (RLVR) has become trending paradigm for training reasoning large language models (LLMs). However, due the autoregressive decoding nature of LLMs, the rollout process becomes the efficiency bottleneck of RL training, consisting of up to 70% of the total training time. In this work, we propose Quantized Reinforcement Learning (QuRL) that uses a quantized actor for accelcrating the rollout. We address two challenges in QuRL. First, we propose Adaptive Clipping Range (ACR) that dynamically ad- justs the clipping ratio based on the policy ratio between the full-precision actor and the quantized actor, which is essential for mitigating long-term training col- lapse. Second, we identify the weight update problem, where weight changes between RL steps are extremely small, making it difficult for the quantization op- eration to capture them effectively. We mitigate this problem through the invariant scaling technique that reduces quantization noise and increases weight update. We evaluate method with INT8 and FP8 quantization experiments DeepScaleR and DAPO, and achieve 20% to 80% faster rollout during training.
INTRODUCTION
The emergence of reasoning Large Language Models (LLMs) such OpenAI-O1 (Jaech et al., 2024) and DeepSeek-PR R1 (Guo et al., 2025) represents a fundamental transformation Al capabil- ities through the strategic scaling of inference-time computation. By enabling extensive Chain-of- Thought (CoT)) deliberation, these systems exhibit advanced problem-solving behaviors that deliver significant gains challenging domains, notably mathematical reasoning (Luo al., 2025b; Yu et al., 2025; Yue et al., 2025) and code synthesis (Luo et al., 2025a; Liu & Zhang, 2025). Such mod- els sacrifice computational economy favor of superior accuracy, producing elaborate reasoning chains that encompass systematic exploration, iterative verification, and strategic backtracking, Re- inforcement learning (RL) forms the cornerstone of these breakthroughs. Through direct optimiza- tion on verifiable objective functions instead of proxy reward models, RL--drven approaches cir- cumvent reward hacking pitfalls (Amodei et al., 2016; Wen al., 2024) while maintaining stronger fidelity to genuine reasoning patterns.
typical LLM RL training step comprises of three phases: actor rollout for response generation, forward pass to compute output probabilities, and backward pass for policy gradient updates. The autoregressive nature of LLMs imposes a fundamental bottleneck-eade token must be decoded sequentially during rollout, requiring extensive memory bandwidth for weight and KV cache access. This sequential dependency severely limits parallelization opportunities. Consequently, the rollout phase dominates training time (He et al., 2025; Zheng et al., 2025), consuming approximately 70% of the total latency. This bottleneck is further exacerbated in reasoning tasks, where competitive performance requires extended CoT traces.
Work done during internship NVIDIA Research.

---

Published as a conference paper at ICLR 2026
In this work, propose Quantized Reinforcement Learning (QuRL), efficient RL training algo- rithm through efficient inference. Specifically, we quantize the actor model for rollout while main- taining full- full-precision parameters for gradient updates. This approach transforms the on-policy RL into an off-policy setting: sequences are generated by quantized actor while policy updates occur in the full-precision parameter space. This necessitates careful importance sampling and trust region constraints, as formalized in Decoupled PPO (Fu et al., 2025; Liu et al., 2025). However, we iden- tify critical failure mode where the decoupled PPO objective leads training collapse at extended horizons, characterized by exponential growth of the divergence between the quantized actor and full precision actor. To address this instability, we propose Adaptive Clipping Range (ACR), which dynamically adjusts trust region bounds based on the policy divergence between the full-precision and the quantized actors.
Beyond importance sampling, we identify a critical scale mismatch between quantized and full- precision actors. RL. updates usually satisfy trust region constraints (Schulman et al., 2015), re- sulting in weight changes that orders of magnitude smaller than quantization errors. Conse- quently, the quantization operation fails to capture most weight updates, effectively decoupling the quantized model from the training dynamics. To address this fundamental mismatch, propose Update-Awer Aware Quantization (UAQ) that uses invariant scaling (Xiao et al., 2023a) to simultaneously reduce quantization error and amplify weight updates, ensuring that parameter changes exceed the quantization granularity threshold.
We validate our approach across multiple RL algorithms including PPO (Schulman et al., 2017), GRPO (Shao et al., 2024), and DAPO (Yu et al., 2025) on diverse reasoning benchmarks. Through 8-bit quantization (INTS or FP8), we achieve substantial inference acceleration for 7B, 14B and 32B models, demonstrating 20%-80% throughput improvements. Our experimental evaluation demon- strates that QuRL consistently outperforms naive combinations of RL with quantized rollout as well as concurrent approaches (Liu et al., 2025). Notably, INT8 QuRL achieves 55.5% average accu- racy on the DeepScaleR benchmark (Luo et al., 2025b) across five reasoning tasks, exceeding the baseline performance by 1.7%.
2 RELATED WORK
Reinforcement Learning for Reasoning. Al systems capable of extended reasoning constitute a distinct class of models that perform elaborate CoT deliberation prior to producing outputs, pi- oncered by OpenAl's ol series (Jaech et al., 2024). Following this breakthrough. DeepSeek (Guo et al., 2025) and Kimi (Team et al., 2025) have documented comprehensive frameworks for develop- ing reasoning models through RLVR. These contributions have established various RL algorithms as standard practice, including GRPO (Shao et al., 2024), Mirror Descent (Tomar al., 2020), RLOO (Ahmadian et al., 2024), among others. However, this scaling comes at the cost of per- forming significant amount of decoding, which severely under-utilizes modern hardware. To this end, the RL community has explored many ways for efficient training, including selective rollout generation (Zheng et al., 2025), rollout down- wwwn-samling (Xu et al., 2025), asynchronous multi-role distributed architectures (Fu et al., 2025). Despite these advances, achieving efficient RI. training while maintaining model performance remains a key challenge.
Quantization Quantization has emerged fundamental technique for compressing and acceler- ating large-scale models. Comprehensive surveys by Gholami et al. (2022) and Nagel et al. (2021) provide systematic analyses of quantization advancements. This section reviews key quantization methods with emphasis on LLM applications. Quantization techniques fall into two main cate- gories: Post-training Quantization (PTQ) and Quantization-Awee Training (QAT). PTQ methods operate directly on pre-trained models without additional training. Prominent approaches includ- ing Frantar et al. (2022b); Lin al. (2023); Wei et al. (2022; 2023); Shao et al. (2023); Chee et al. (2023); Liu et al. (2023a) enhance uniform quantization through strategic optimization of weight pa- rameters, scaling factors, and clipping boundaries. Alternative PTQ strategies explore non-uniform quantization schemes (Egiazarian et al., 2024; van Baalen et al., 2024; Elangovan et al., 2025) and mixed -recissre architectures such as LLM.int8 (Dettmers et al., 2022). QAT methods integrate quantization into the training process itself. LLM-QAT (Liu et al., 2023b) addresses data require- ments through synthetic generation, while Q-LoRA (Dettmers et al., 2023) combines quantization with low-rank adaptation to reduce memory overhead during fine-tuning.
2

---

Published as a conference paper at ICLR 2026
3 PRELIMINARIES
We begin with brief overview of the GRPO (Shao et al., 2024) algorithm. And then we introduce the basics of quantization operation and the subsequent challenges.
Group Relative Policy Optimization. GRPO adapts the PPO (Schulman et al., 2017) framework for training LLMs, notably by eliminating the need for a learned value function (critic). Instead of using generalized advantage cstimation (GAE), GRPO estimates the advantage A., at token t of output o, based on the relative rewards within a group of G outputs (01, .2,.,,, GG sampled from the old policy Te for the same prompt 9. The objective function is:
G
JCRPO(() = E.g. P
min (Ra,
+,1+))
(I)
(0,a is importance sampling ratio between the old actor and the current actor on the 1-th token of i-th generated response. Additionally, GRPO augments the PPO objective with an explicit KI. regularization term Dx (Te|| The reference model * (typically the initial supervised fine-tuned model) provides regularization, with the KL divergence computed using the k3 estimator from Schulman (2020).
where Rs, = =o(o..elg))
Quantization. Quantization maps the full precision parameters into low-bit parameters 0 = Q(0)- In general, b-bit quantized parameter can be expressed as
b-1 1 --
() = a x (-1)simm
siin x 2d x (1+
mi
(2)
2
where the representation consists of three components: sign, exponent, and mantissa, scaled by factor Here, sign [-1,+1) encodes the sign, d E [1, [% represents the exponent using bits. And the mantissa uses the remaining (b - 1 - e) bits with m; E 10. When e = 0, this formulation reduces to integer quantization. The scaling factor is determined by the maximum absolute value within a group of weights or activations. The granularity of quantization depends on the group size, ranging from channel-wise block-wise operations. To further reduce memory overhead, the scaling factor itself can be quantized, as in NVFP4 (NVIDIA, 2025).
QURL: QUANTIZED REINFORCEMENT LEARNING
To accelerate the rollout phase, we quantize the weights and activations of the old actor model Oold to lower-bit representations, enabling efficient matrix multiplication during inference lat = Qfooid, b). Fig. illustrates the pipeline for incorporating quantization into the RL.
Accelerated Decoding
Quantized Model
Actor
Rollout
((Oom))
Quant. Error! (0 0old
Our QuRL approach occupies a unique position between post-training quantization (PTQ) and quantization-aware training (QAT). Unlike QAT, we do not explicitly op- timize quantization performance through gradient de- scent-the actor undergoes one-shot quantization before deployment for rollout. However, unlike pure PTQ, the actor parameters are implicitly influenced by the gradi- ents computed from the quantized model's outputs dur- ing policy updates. This dual nature imposes specific re- quirements on our quantization strategy: it must be sufficiently simple to avoid complex calibration procedures while remaining expressive enough to preserve the learning dynamics. In the following sections, we present our methodology addressing both the reinforcement learning adaptations and the quantization operations necessary for efficient training.
Old Actor
Updated
Actor
Gola
Weight Update
0 Hotd
Figure 1: Overview of QuRL training. The sampling model Oatd is quantized to dota for rollout.
3

---

Published as a conference paper at ICLR 2026
(a) Mean Training Rewards
(b) Token Clipped Fraction
8.68
0.55
8.015
BF16 (Eq. 1)
INT8 (Eq. 1)
8.58
INTa (Eq. 3)
Value
0.010
INT8 (Eq. 4)
8.45
(Eq.
0.49
INT8
8.085
(Eq.
8.35
(Eq.
0.000
500
288
Step
Step
Figure 2: Comparison of (a) training rewards and (b) token clipped fraction under different training
objective or quantization.
4.1 ISSUES WITH IMPORTANCE SAMPLING AND CLIPPING
Given that rollout data has been sampled from the quantized actor, we can rewrite the RL objective
as
o. min (Rr.A,,,clip(RR.. 1,1+ e)Ai.) ())+)))
[(0) Eg Pog) Ford
(3)
where Ri,- (l(.)) aa (o.)) denotes the importance sampling ratio between the current full-precision
actor and the quantized old actor Tiai
We implement the objective of Eq. (3) and compare with full- preciisei GRPO experiments. The model and dataset follow the DeepScaleR setup (Luo al., 2025b). Unfortunately, the quantized model's importance sampling leads to training instability, with rewards collapsing after several RI. steps as shown in Fig. 2. Analysis of the token clipped fraction in Fig. 2(b) reveals that Aú,, exhibits significantly higher clipping rates than the -ree-iiiiision baseline. The fraction rapidly increases 1.5% before abruptly dropping to zero, indicating severe instability in the R.,, when applying clipping on top of it. Additionally, we test the objective of Eq. (1) with quantized rollout, which instead uses the full-precision old actor as the denominator for clipping and importance sampling. Fig. 2 shows that this objective can have stable training curve, but may produce a large gap between BF16 after 800 steps of RL training.
To address the instability in Eq. (3), we adopt the decoupled PPO objective (Hilton et al., 2022; Fu et al., 2025), which separates the behavior policy Teashav (for token sampling) from the proximal policy grom (for clipping):
T (0s,e) T (or,e)
Jaccoupled (0) Engn
min AR,, clip( 1 -c,, 1,,1 +c))
(4)
xo(0),)) denotes the ratio between the current policy and the proximal policy. where Ra,- Fepros (o1.)
For simpler notation, we integrate the averaging across responses and groups into expectation É, as they do not change the clipping/importance sampling outcome. In QuRL, we set the behavior policy as the quantized old actor (To. = T and the proximal policy as the full-precision
old actor (To ). Compared to Rs, that R., enables more tokens to trained via correct importance sampling. As shown in Fig. 2, this approach significantly improves training stability.
quantized actor to determine clipping,
FlashRL (Liu et al., 2025) observes that Tā is usually obtained from the inference engine such as vLLM (Kwon et al., 2023) and SGLang (Zheng et al., 2024). However, due to the implementa- tion difference between training (i.e., HuggingFace and Megatron) and inference (i.e., vLLM and SGLang) engines, an extra engineering discrepency between Tp and T is introduced and will hinder RL training. FlashRI. proposes Truncated Importance Sampling (TIS) reduce this
4

---

Published as a conference paper at ICLR 2026
(a) KL Ma [ MB...
(b) Max Ratio
0.025
QURL ACR
15008
QuRL w/ ACR
Value
18008
8.015
8.010
5000
268
408
600
890
1098
1290
288
499
690
800
1880
1298
Seep
Step
Figure 3: Training dynamics of QuRL. (a) Training collapses after 1000 steps due to increased KI. divergence between behavior and proximal policy, and (b) the maximum value of the proximal-to- behavior policy ratio.
difference, given by
T Alpr (ois)) Boaaii (ot.e) c) min (l, -,,1,,,))1,,
(O) = Eo bohav
min
(5)
where C' bounds the poxximal-to-behavior ratio. This formulation reduces computational overhead by directly accessing probabilities from the inference engine and naturally extends to other off- policy RL methods. However, even with these modifications, the decoupled objective alone cannot fully bridge the quantization gap QuRL, particularly during later training stages.
4.2 ADAPTIVE CLIPPING RANGE
The TIS essentially modifies the behavior policy by truncating the behavior policy if it is extremely small. To this, define
Ti bebav
max bechax'
T
Oprrxx
(6)
Now, we can rewrite TIS objective as the decoupled PPO objective form, given by
groxx (0i,e)
Idecoupled (0) Eo bobaar Tgrrune (or,))
min (Ri,AA,, clip(Ra, -, +)
(7)
Essentially, the gradient of the original decoupled PPO objective is scaled by Ti.
Wechaa (oLs)/s o/1 bebaay (o.,.). b. As shown in Fig. 3(b), the maximum proximal-to-behavio ratio can reach up to 105 causing an extremely large gradient norm using the decoupled PPO objective. The above equation effectively avoids the excessive gradient norm.
In practice we find that TIS works well under 500 steps of RL training. However, long training steps (e.g., > 1000 steps), observe the KL divergence between the behavior policy and the proximal policy (i.e., DKL(TTe | Eloog
"Bbaaax continues to increase. As shown in Wéprex
Fig. 3(a), the KL divergence increases from 0.002 to 0.025, which is 12x higher. This indicates that the truncated behavior policy also leads to biased gradient estimation, especially for large
To mitigate this problem, we examine the clipping mechanism in the decoupled PPO objective and propose the Adaptive Clipping Range (ACR). Our intuition is that, when the behavior policy is truncated, the factor ra,t implicitly affects the clipping. More concretely, given that 0 < ri, 1, absorb this factor into the clipping term well as its range, given by
cllpcip( e),
e), (1 e)) clip(r,, r.. e), (1 [
(8)
This operation shrinks both the upper and lower clipping range by factor of ri.. For negative advantage sequences, it does affect the clipping since the large ratios do not get clipped regardless. However, for positive advantage sequences, it reduces the upper bound. For large r,, where the difference likely due to training/inference engine execution, its biased estimation unexpectedly clips more tokens. To address this issue, we propose
fixed upper threshold + e), to allow
5

---

Published as a conference paper at ICLR 2026
AP axG
QURL w/ UAQ (INTB) QURL
8.818
w/o
UAQ
(INTS)
0.016
8.814
Quantize
WMN
0.812
Few weights changed quantized values
0.010
25
50
75
108
125
150
175
208
Step
(a) Weight change mismtach
(b) Visualization of average (Taas Té
Figure 4: The weight update problem. (a) An intuitive example showing weight quantization cannot sense the weight update suitably and (b) visualization of training dynamics showing the difference between Tana and R0.aa
more tokens to pass if the Thanhas (0t,e) is truncated. As result, we can rewrite our ACR into:
prrox (Oi,)) Buuutai (or,)) c) min [Ra,Ace, Ra,, (1 - t), (1 + Ac,
(AC B Bo pabaav
min
Vi.t
(9)
The ACR can dynamically adjust the clipping range based on the proximal-to-behaviee ratio: For tokens where Killpzox (os.t)
Kannhau (0s,a] C, Ti, enlarges the upper clipping bound, allowing more positive tokens to be updated. Otherwise, the threshold is the same as TIS.
4.3 UPDATE-AWARD QUANTIZATION
Another crucial challenge in QuRL is the mismatch of magnitude between the weight quantization change and the weight update change. During each RL step, Pold is quantized to dotd for rollout, then updated to to become the old actor for the next step. The weight change magnitudes follow:
10tal 26
Oold Gotd x
Gota x aG,
(10)
where a denotes the learning rate and G' the gradient. In typical RL experiments, we observe G E [0.1, with a 10-6 6 yielding weight updates of order 10-7 to 10-6, This is substantially smaller than the quantization error, the weight norm itself ranges from (0.001, 0.1). We also provide an example in Fig. 4(a) to illustrate this problem.
Empirical analysis confirms this mismatch. Comparing daia across RL steps with INT8 quantization in DeepScaleR experiments, the update is much smaller than quantization error. See Appendix A for more details. This indicates that quantization masks nearly all weight updates, effectively freezing the quantized model despite ongoing training. In Fig. 4(b), we measure the average difference between 7. O.n and To.ia of INT8 quantization in DAPO task (Yu et al., 2025).
Since QuRL operates between PTQ and QAT paradigms, neither approach offers an optimal solu- tion. Complex calibration algorithms like GPTQ (Frantar et al., 2022a) could theoretically capture finer weight changes if applied at each step, but would impose prohibitive training time overhead on rollout. For QAT, it will introduce additional discrepancies between training and inference engines, exacerbating importance sampling bias (Liu et al., 2025).
To mitigate this problem, propose Awarte-Aware Quantization (UAQ), one-time weight adjustment performed before RL training begins. Our ap- proach leverages invariant scaling of linear layers in trans- former blocks (Xiao et al., 2023b). Given weights W and input activations in a layer, invariant scaling preserves the output by
sWi
W
Q_proj
Wg
Layer
sX
K_proj
W =
(sX).
(11)
V_proj 5
Figure 5:
Invariant scaling of
Q/K/V layers.

---

Published as a conference paper at ICLR 2026
The scale is applied column-wise to W and row-wise X. The activation scaling can be absorbed into the preceding layer (e.g., LayerNonm), as illustrated in Fig. 5.
Unlike existing PTQ methods (Xiao et al., 2023b) that minimize quantization error [({ WX|I, strategically choose > 1 to balance weight quantization error against update magnitude such that
[0oil
Oold - Oold x
8 25 0 & 8 a.G..
(12)
The scaling factor reduces quantization error by a factor of while amplifying weight updates by the same factor. The weight update amplification occurs because gradients with respect to W are computed as Vw C vVrC) where the pre-scaled activations X have been multiplied by
This dual effect creates an s² improvement in the ratio between weight updates and quantization noise, enabling the quantized model to capture training dynamics more effectively. Empirically, we find 8 = 1.5 provides consistent improvements on INT8 and FP8 quantization, striking an effective balance between reducing quantization artifacts and maintaining numerical stability.
EXPERIMENTS
All our experiments were conducted with the hybrid engine-based RL framework, VeRL. (Sheng et al., 2024). We evaluate QuRI. across three distinct reinforcement learning configurations: (1) PPO training on GSM8K (Cobbe al., 2021), (2) DAPO (Yu al., 2025) optimization on AIME mathe- matical reasoning tasks, and (3) GRPO training on the DeepScaleR benchmark (Luo et al., 2025b).
Our quantization experiments employ both INTS and FP8 precision formats. Weight quantization utilizes channel-wis -wine scaling factors, while activation quantization applies token-wiss scaling. We leverage vLLM's optimized INT8 and FPS matrix multiplication kemnlls (Kwon al., 2023) to achieve computational acceleration during inference. Note that FP8 KV cache quantization remains suboptimally implemented in the current vLL.M version and does not yield measurable throughput improvements; consequently, exclude KV cache quantization from experimental evaluation.
5.1 REASONING RESULTS
PPO on GSMSK. We evaluate the PPO algorithm (Schulman et al., 2017) on the GSM8K dataset, which comprises 7.4k training examples and 1.3k validation examples. We use Qwen2.5-0.5B Instruct (Qwen et al., 2024) as the base model and conduct training over 15 epochs, corresponding to 435 RL optimization steps. Training employs batch size of 256 with a maximum response length of 512 tokens per rollout. Evaluation metrics are computed using greedy decoding on the test set (deterministic next-token selection based on maximum probability).
We compare against four experimental configurations: (1) Full-precision RL baseline, (2) Quan- tized rollout with standard importance sampling-applyy
Equation 1 to responses sampled from the quantized policy Tuu (3) FlashRL with Truncated Importance Sampling (TIS) (Liu et al., 2025), and (4) QuRI. with Adaptive Clipping Range. Note that Update-Aware Quantization dis- abled for this experiment due to the relatively high learning rate (10-5), which already provides suf- ficient weight update magnitude. Table 1 presents final checkpoint accuracies. The results demon- strate that naive INT8 quantization without decoupled behavior/proximal policies yields substantial performance degradation. FlashRL (Liu et al., 2025) achieves training stabilization through TIS and decoupled PPO, yet maintains notable accuracy gap relative BF16 baseline- parelinne-pattullarrly severe under INT8 quantization (4% degradation). In contrast, QuRI. with ACR reduces this gap to 2% for INT8 and approximately 1% for FP8, demonstrating superior quantization robustness across precision formats.
DAPO on AIME 2024. Next, we test the decoupled clip and dynamic sampling policy optimization (DAPO) (Yu et al., 2025) with Dwen2.5-7BBMatth We use the 17k dataset from the original paper and apply the decoupled clip where Enigh = 0.28 and foow = 0.2 for the default clipping range. We optimize the base model for 200 steps. and the learning rate is set to 1e 6. In each step, sample 512 queries and 16 rollout responses per query. Additionally, DAPO does not apply any KL diver- gence loss between the actor and reference models. For evaluation, we use metrics (Avg@)
7

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of GSM8K accuracy.
Figure 6: Convergence of INT8 experiment.
Method
Bitwidth
Accuracy
RL
BF16
55.35
GSMBK Accuracy
0.5
RL
INT8
48.78
0.4
FlashRL (Liu et al., 2025)
INT8
51.40
0.3
QuRL (Ours)
INT8
53.55
(BF16)
0.2
RL
FP8
0.0
RL (INT8)
FlashRL (Liu et al., 2025)
FP8
53.60
0.1
FlashRL (INT8)
QuRL (Ours)
FP8
54.28
QURL (INT8)
160
208
380
468
S Step
Table 2: Comparison of AIME 2024 accuracy.
Figure 7: Convergence of FP8 experiment.
Method
Bitwidth
Avg@1
Avg@
0.30
RL
BF16
33.33
31.67
AIME 2024 Avg832
0.25
RL
INT8
0.00
0.001
FlashRL
INT8
26.66
30.29
0.20
QuRL w/o UAQ
INT8
33.33
30.63
0.15
QuRL w/ UAQ
INT8
33.33
31.25
(BF16)
8.10
RL (FP81
RL
FP8
0.00
0.003
FlashRL (FP8)
FlashRL
FP8
30.00
32.60
0.85
QUAL w/o UAQ (FP8)
QURL UAQ (FP8)
QuRL w/o UAQ
36.66
33.12
0.00
QuRL w/ UAQ
33.33
33.27
58
75
100
125
159
175
208
Step
and (Avg @2) on the AIME 2024 dataset, where, Avg@) @1 1 represents the accuracy achieved using greedy decoding (deterministic next token prediction) and Avg represents the average accuracy of 32 sampled responses per problem, using temperature of 1.0 and top p of 0.7.
As shown Table 2, vanilla INTS/FP8 RL has near accuracy on the AIME 2024 dataset, indi- cating that biased estimation of the importance sampling will result in crashed performance. The convergence figure on the right shows that RL can converge well in the first 100 steps, but results in decreased performance for the latter 100 steps. This is due the increased gap between proximal and behavioral policy through training. FlashRI (Liu et al., 2025) converges better than RI. and has much better final accuracy than RI. For example, with INT8 quantization, FlashRI. achieves 30.3% Avg@3 32 accuracy, with 1.4% gap from the full precision baseline. Our QuRL, equipped with ACR, can successfully close the gap during training. The final accuracy also shows improved results, with 33.1 % Avg@ accuracy under FP8 quantization.
GRPO DeepScaleR. Finally, project, DeepScaleR (Luo et al., 2025b), which improves the reasoning boundaries of DeepSeek- Distill-Qwenl.5B models (Guo et al., 2025). The training dataset contains 40k math problems from AIME problems from 1983 to 2023, as well as Omni-Math Gao et al. (2024) and Still (Min et al., 2024). We train the actor for 3 stages, under 8k, 16k, and 24k context length, respectively. The training batch size is 256, with the learning rate of le 6. Following the official implementation,
test the performance of algorithm
open-source
Table 3: Comparison of Avg@32 32 accuracy across various math reasoning tasks of DeepScaleR.
Method
Bitwidth
AIME24
AMC
MATH
Minerva
Olympiad
Avg
Base
BF16
28.54
62.58
82.90
26.38
43.58
48.80
RL
BF16
40.73
73.45
87.71
30.56
49.59
56.40
RL
INT8
33,95
68.75
84.90
28.12
45.85
52.31
FlashRL
INT8
36.77
70.55
85.88
28.44
47.33
53.80
QuRL w/o UAQ
INT8
39.06
70.48
86.48
29.20
48.75
54.79
QuRL w/ UAQ
INT8
40.52
71.34
87.20
29.22
49.13
55.48
8

---

Published as a conference paper at ICLR 2026
DeepSeek Qwen-
DeepSnck Owen
DeepSeck -Qeen
BF16
Querles per second
70
INT8
INT8
50
INT8
60
50-
38
1.56x
1.33x
1.66x
28
20
1.40x
10
A180 GPU Model
A6088x2
H108x2
GPU Model Figure 8: Inference acceleration of INT8 quantization.
GPU Madel
for the first stage, we generate 8 rollouts per query with 8k context length and train the model for 800 steps. For the latter two stages, 16 rollouts per query are generated, and the actor is trained for 400 steps. The coefficient for KL divergence in GRPO is set to le 3. The temperature is set to 0.6
For evaluation, we follow the official implementation to compare the Avg results on AIME 2024 (Li et al., 2024), MATH 500 (Hendrycks et al., 2021), AMC 2023 (Ouyang et al., 2022), Minerva Math (Lewkowycz et al., 2022), and Olympiad Bench (He et al., 2024) as well as the average accuracy of all above. The results are shown in Table 3. The full-precision RL improves the base model by 7.6% average accuracy across tasks and notably 12% accuracy improvement on the AIME 2024 dataset. Instead, INT8 RL barely improve the base model accuracy, for example, 5% improvement on the AIME 2024 dataset. Averaging all tasks, INT8 RL. has a large gap of 4. compared BF16 RL. FlashRL achieves slight improvement over INTS RL, with a 1.5% higher average accuracy among tasks. On the other hand, QuRL. w/ UAQ significantly boosts the average accuracy of INT8 RL by 3%.
5.2 THROUGHIPT TEST
In this section, we demonstrate the throughput benefits of applying quantization during rollout (de- coding). We use the script from GuideLLm (Neural Magic, 2024) to evaluate the DeepSeek-Distie Bwen-7B/14B/322 models on the vLLM platform (Kwon al., 2023). We test INTS quantization across multiple GPU types, including A6000, A 1000, and H100. For 7B and 14B model, we evaluate throughput (queries per second) on one GPU, while for 32B model, we evaluate it with tensor parallelism across 2 GPUs.
The results are shown in Fig. For the 7B model, INT8 quantization can bring 20%~~30% accel- eration effect, while for the 32B model, we find INT8 quantization can bring 70% faster throughput on A100 and 90% faster throughput on H100. Generally, we observe that larger models benefit more from quantization. This is due the large model being bottlenecked by matrix multiplication, yet smaller models are usually bottlenecked by the I/O Nevertheless, we emphasize that QuRI. is compatible with other types of compression (Frantar & Alistarh, 2023; Liu et al., 2024).
5.3 ABLATION STUDY
In this section, compare the results of choosing different scales for UAQ. On hand, large scale contributes to a smaller gap between weight update and weight quantization noise. On the other hand, will also make the weight update more than usual, causing more clipped tokens and decreasing the RL performance. To demonstrate this effect, we compare scales & 1, 1.5. 2 and also test another alternative by directly increasing the learning rate on the DAPO tasks with INT8 quantization. The results are shown in Table 4. can be seen that s = 1.5 with the original learning rate provides the best results. The larger scale learning rate results in less stable training RL and decreases the accuracy AIME 2024.
Scale Learning Rate
Avg@32
8
a 10
30.63
$ = 1.5
a = 10 10
31.25
2
a 10-6
29.15
8
1.5 x 10-6
29.06
S a = 2 x 10
26.66
Table 4: Ablation scale and

---

Published as a conference paper at ICLR 2026
CONCLUSION
We present QuRL, an efficient RL training method that accelerates rollout generation through quan- tized inference. QuRL addresses two fundamental challenges: clipping instability and weight update-quantization mismatch. Our ACR prevents training collapse by dynamically adjusting the clipping, while UAQ bridges the scale gap between weight updates and quantization errors through invariant scaling. Experiments across multiple RL algorithms demonstrate consistent improvements over baselines and 20~80% throughput improvement over BF16 rollout.
ACKNOWLEDGMENT
The authors would like to thank Charbel Sakr for his helpful feedback. This work was supported in part by the National Science Foundation (CAREER Award, Grant #23 12366, Grant #23181522 the DARPA Young Faculty Award, the DoE MMICC center SEA-CROGS (Award #DE-SC0023198) and the Global Industrial Technology Cooperation Center (GITCC) program.
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learn- ing from human feedback in llms. arXiv preprint arXiv:2402.14740, 2024.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in ai safety. arXiv preprint 1rXiv:1606.06555, 2016.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. Quip: 2-bit quantization of large language models with guarantees, arXiv preprint arXiv:2307.13304 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint #rXiv:2110.14166 2021.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer: Llm. int8 0: 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.0733: 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305. 14314, 2023.
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Al- istarh. Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401.061 18, 2024.
Reena Elangovan, Charbel Sakr, Anand Raghunathan, and Brucek Khailany. Beq: Block clustered quantization for 4-bit (w4a4) 11m inference. arXiv preprint arXiv:2502.05376, 2025.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210. 20..:333,, 2022a.
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint 2rXiv:2210.1732.. 2022h.
Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang. al. Areal: A large-scale asynchronous reinforcement learning system for language reasoning. arXiv preprint arXiv:2505.24200 2025.
Bofei Gao. Feifan Song, Zhe Yang, Zefan Cai. Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07988. 2024.
10

---

Published as a conference paper at ICLR 2026
Amir Gholami, Sehoon Kim, Zhen Dong. Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. Low-Powel Computer Vision, pp. 291-229122 Chapman and Hall/CRC, 2022.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint 1xXiv:2402.14000 2024.
Jingkai He, Tianjian Li, Erhu Feng, Dong Du, Qian Liu, Tao Liu, Yubin Xia, and Haibo Chen. History rhymes: Accelerating IIm reinforcement learning with rhymerl. arXiv preprint arXiv:2508. 18588, 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint 2rrXiv:2105. 03874, 2021.
Jacob Hilton, Karl Cobbe, and John Schulman. Batch size-invariance for policy optimization. Ad- vances in Neural Information Processing Systems, 35:1 75086-177788 2022.
Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412. /6720, 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium Operating Systems Principles, 2023.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo et al. Solving quantitative reasoning problems with language models. Advances neural information processing systems, 35:3843-3857, 2022.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9):9, 2024.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for 11m compression and acceleration. arXiv preprint M.Xiv:2306.00978, 2023.
Jiawei Liu and Lingming Zhang. Code-rl: Reproducing rl for code with reliable rewards. arXiv preprint 1xXii:2503.18777 3, 2025.
Jing Liu, Ruihao Gong, Xiuying Wei, Zhiwei Dong, Jianfei Cai, and Bohan Zhuang. Qlim: Accurate and efficient low-bitwidth quantization for large language models. arXiv preprint arXiv:2310.08041 2023a.
Liyuan Liu, Feng Yao, Dinghuai Zhang, Chengyu Dong, Jingbo Shang, and Jianfeng Gao. Flashrl: 8bit rollouts, full power rl, August 2025. URL peattps:/feegyaart notion. site/ flash-rl.
notion.site./
Zechun Liu, Barlas Oguz, Changsheng Zhao. Emie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. LIm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305. /7888, 2023b.

---

Published as a conference paper at ICLR 2026
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A suning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750, 2024.
Michael Luo, Sijun Tan, Roy Huang, Xiaoxiang Shi, Rachel Xin, Colin Cai, Ameen Patel, Alpay Ariyak, Qingyang Wu, Ce Zhang, Li Erran Li, Raluca Ada Popa, Ion Stoica, and Tianjun Zhang. Deepcoder: A fully open-source 14b coder at 03-mini level. --wodderr--uull--peeee Source-- 4B Sarecce------odee---eeee- 10:003m-nn-- Level- 1cf81 190111 (e2ec14688bbbb 400.214440bbbeeeeeee... 2025a. Notion Blog.
https:/ https////eetttrrai..oo b75 wtwooo-....iiiiittoeooeeo notion.siter
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol-preview with
1.5b
model by scaling rl. DeepScaleR peepicalee-SSrrasssnng-e 01- seeeieewwithh-a 1-BB 11-5B-MMdee----- Model-by. Bondel-y-Sccalnn--t 12-1------11111111118881881 2025b. Notion Blog.
pooo-b5..nttiiii not notion.siee site/
Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, et al. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems. arXiv preprint arXiv:2412.09413, 2024.
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tijmen Blankevoort. A white paper on neural network quantization. arXiv preprint arXiv:2/06.0822 08295. 2021.
Inc. Neural Magic. Guidellm: Scalable inference and optimization for large language models. https: / hithbb....u...gggggg
p1ln-ppojjcctgguiieeiio. 2024.
NVIDIA. Introducing NVFP4 for Efficient and Accurate Low-Precision Inference - NVIDIA Technical Blog- - developer.widia.com. http::://Gevelopperr.v.... www.oodduinngnvvieeeee -tt.aaicoooooooo--------------a-
http::/wwwwwaaaaaacccooo/ooooo cwm/blog/
2025. [Accessed 1 11-09-2025].
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Yang Qwen, Baosong Yang, Zhang, B Hui, Zheng, B Yu, Chengpeng Li, D Liu, F Huang. H Wei, et al. Qwen2. 5 technical report. urXiv preprini, 2024.
John Schulman. Approximating KL Divergence - joschu.net.
k1 k1-approx.htmm html, 2020. [Accessed 1 1100-20025..
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308.1311 13137, 2023.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Rumxin Xu, Junxiao Song, Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 19256, 2024.
12

---

Published as a conference paper at ICLR 2026
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501.1259) 2025.
Manan Tomar, Lior Shani, Yonathan Efroni, and Mohammad Ghavamzadeh. Mirror descent policy optimization. arXiv preprini arXiv:2005.0988 2020.
Mart van Baalen, Andrey Kuzmin, Markus Nagel, Peter Couperus, Cedric Bastoul, Eric Mahurin, Tijmen Blankevoort, and Paul Whatmough. Gptvq: The blessing of dimensionality for Ilm quan- tization. arXiv preprint arXiv:2402. 15319, 2024.
Xiuying Wei, Yunchen Zhang. Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Feng- wei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer lan- guage models. Advances in Neural Information Processing Systems, 35:1 35:17402-1 17414, 2022.
Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo. and Xian- glong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. arXiv preprint MXXi:23304.09455 2023.
Jiaxin Wen, Ruiqi Zhong, Akbir Khan, Ethan Perez, Jacob Steinhardt, Minlie Huang, Samuel Bowman, He He, and Shi Feng. Language models learn to mislead humans via rlhf. arXiv preprint arXiv:2409. 22t.::220..1222,,, 2024.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099. PMLR, 2023a.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International conference on machine learning, pp. 38087-38099. PMLR, 2023b.
Yixuan Even Xu, Yash Savani, Fei Fang, and Zico Kolter. Not all rollouts usefull Down- Down-sampling rollouts IIm reinforcement learning. arXiv preprint 1tti::2504..88/65 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05118. 2025.
Haizhong Zheng, Yang Zhou, Brian R Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, and Beidi Chen. Act only when it pays: Efficient reinforcement learning for Ilm reasoning via selective rollouts. arXiv preprint arXiv:2506. 02177, 2025.
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs. Advances neural information processing systems, 37: 62557-62583, 2024.
13

---

Published as a conference paper at ICLR 2026
A
WEIGHT CHANGE VISUALIZATION
0.010
Relative L2 Norm
0.008
0.005
Normalized Weight Update
Norma lize Weight Quantization Error
0.004
0.002
0.000
8
200
400
600
888
Training Step
Figure 9: Visualization of normalized weight changes.
In this section, we visualize the weight change during RL learning. We conduct our analysis on the DeepScaleR task (Luo al., 2025b). Our metric include (1) Normalized Weight Update, defined as
NommaizzaWWighh)yyla((((() ||0 0' 1'1
(13)
|
where denotes the actor weights after RL steps. This metric captures the ratio between the total weight update and the initial weight values. The second metric adopt is the Normalized Weight Quantization Error, defined as
NormalizedWWighhuunanttt = ||Q(0") -01
(14)
||0:11
We use INT8 quantization, and compare their values in Fig. 9. can be found that the weight quantization error is much larger than the weight update, especially at the early training stages. Note that the update are measured across every 16 steps, which means the actual weight update per step could be far smaller than the plotted values.
B DEEPSCALER VISUALIZATION
0.40
AIME 2024 Avge32
8.35
0.30
0.25
RL (BF26)
RL. (INT8)
FLashRL (INTB)
0.20
QURL (INTB)
288
809
1288
1489
Step
Figure 10: Visualization of test accuracy (AIME 2024 @3))
Here, we provide the test accuracy of our DeepScaleR experiments. It can be observed that in the long-horizon training scenarios, INT8 RL incurs large gap with the BF16 RL. Although FlashRL ensures consistent improvement before 1200 steps, its test accuracy starts to drop after 1200 steps. While QuRL. can have consistent improvement across the whole training cycle.
14


Reviews:
[{"review_id": "mZ3KCsbbPx", "reviewer": "Reviewer_FcNP", "summary": "This paper presents QuRL, a practical recipe to accelerate reinforcement learning for reasoning LLMs by performing rollouts with a low-precision actor (INT8/FP8) while keeping full-precision weights for policy updates, which effectively decouples the behavior (quantized) and proximal (full-precision) policies and necessitates careful importance sampling and trust-region control. They propose Adaptive Clipping Range (ACR), which absorbs the truncation factor into PPO’s clipping term, shrinking the effective range (with a fixed upper cap) to stabilize gradients and prevent late-stage collapse. The paper also introduces Update-Aware Quantization (UAQ), a one-time invariant scaling of transformer linear layers that simultaneously reduces quantization error and amplifies effective updates, yielding an s² improvement in update-to-noise ratio. On GSM8K with PPO, QuRL closes much of the gap between naive INT8 and BF16 (53.55% vs 55.35%) and also improves FP8 robustness relative to baselines. Overall, QuRL stabilizes decoupled PPO (via ACR) and makes low-precision rollouts learnable (via UAQ), offering a simple, engine-compatible path to faster RL for reasoning LLMs.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper studies an important problem of model quantization in reinforcement learning, which can help speed up the RL training process. The work decouples rollout and optimization by quantizing the actor for sampling while keeping full-precision weights for updates; the paper cleanly frames this as decoupled/off-policy PPO and motivates the need for trust-region/IS control.\n\n2. Adaptive Clipping Range stabilizes late-stage collapse by adapting the clipping bounds to proximal-to-behavior ratios; Update-Aware Quantization (invariant scaling) reduces quantization error and amplifies effective updates with a simple, engine-compatible change.\n \n3. The paper conducts a fair comparison to demonstrate the effectiveness of QuRL for RL training.", "weaknesses": "1. Marginal accuracy improvements in places: on GSM8K the final gap vs BF16 remains (~53.55 vs 55.35), and DeepScaleR’s reported average gain is modest (+1.7%), which may feel incremental relative to the engineering complexity.\n\n2. Scale of core effectiveness studies: the detailed PPO accuracy study uses a 0.5B base model (Qwen2.5-0.5B); while throughput is shown for 7B–32B, end-to-end accuracy results beyond small models are limited. Therefore, the authors are suggested to conduct experiments on larger models and even MoE models to demonstrate the effectiveness of their method.\n\n3. Nowadays more models focus on reasoning or agentic capabilities, which requires stronger long-context capabilities of model. The paper does not provide direct evaluations on thinking models or agentic models to study whether the proposed methods works under these settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wIseDFyg0c", "reviewer": "Reviewer_PhnW", "summary": "This paper introduces Quantized Reinforcement Learning (QuRL), a method to accelerate the rollout phase in reinforcement learning (RL) for large language models (LLMs) by quantizing the actor model during inference. The authors identify two key challenges: (1) training instability due to policy divergence between the full-precision and quantized actors, and (2) the mismatch between the magnitude of weight updates and quantization errors, which hinders the quantized model from capturing training dynamics. To address these, QuRL proposes Adaptive Clipping Range (ACR) to dynamically adjust trust region bounds based on policy divergence, and Update-Aware Quantization (UAQ) to amplify weight updates relative to quantization noise using invariant scaling. Experiments on GSM8K, AIME 2024, and DeepScaleR benchmarks demonstrate that QuRL achieves 20–80% faster rollout with INT8/FP8 quantization while maintaining competitive accuracy compared to full-precision baselines and outperforming naive quantization approaches.", "questions": "See weakness", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles a critical bottleneck in RL training for LLMs -- rollout latency -- by integrating quantization into the RL loop. The idea of decoupling quantized rollout from full-precision updates is innovative and addresses a timely problem in scalable RL.\n\n2. The proposed solutions (ACR and UAQ) are well-motivated by empirical analyses of training instability and weight update mismatches. The ablation studies validate the contributions of each component.\n\n3. Experiments span multiple RL algorithms (PPO, DAPO, GRPO), model sizes (7B–32B), and reasoning benchmarks, demonstrating broad applicability. Throughput gains (20–80%) are substantial and empirically validated.\n\n4. The paper is well-structured, with clear explanations of challenges and methodologies. Figures effectively illustrate key issues like training collapse and weight update mismatches.", "weaknesses": "1. The work primarily contrasts QuRL with full-precision RL and naive quantization. Including comparisons to other efficiency methods (e.g., pruning, distillation) could better contextualize QuRL’s advantages.\n\n2. Although ACR mitigates collapse up to 1,200 steps, the scalability to even longer training horizons (e.g., thousands of steps) is not tested. Stability under extended training remains an open question.\n\n3. Robustness: How sensitive are ACR and UAQ to hyperparameters like the clipping bound or scale factor ? Are there guidelines for tuning them on new tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8GJTm0YTmr", "reviewer": "Reviewer_xNJr", "summary": "This paper introduces QuRL (Quantized Reinforcement Learning), a novel method to accelerate the rollout phase in reinforcement learning (RL) for large language models (LLMs). The authors address the critical bottleneck in RL training—where the autoregressive decoding process consumes most of thel training time—by quantizing the actor model during rollout while maintaining full-precision parameters for policy updates.", "questions": "(1) What are the implications of quantization on RL exploration-exploitation trade-offs, especially in tasks requiring high creativity or diversity?\n(2) How would QuRL perform under more aggressive quantization (e.g., 4-bit), and what additional adaptations would be necessary?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) it identifies and systematically addresses under-explored challenges at the intersection of quantization and RL, such as the divergence between behavior and proximal policies. The proposed ACR mechanism is both intuitive and effective, dynamically adjusting clipping bounds to sustain training stability over extended horizons. \n(2)UAQ cleverly leverages invariant scaling to amplify weight updates relative to quantization noise, enabling the quantized model to reflect training dynamics more faithfully.\n(3) the experimental validation is extensive, covering diverse model sizes (7B to 32B), tasks, and quantization formats (INT8/FP8), with consistent improvements over strong baselines like FlashRL.", "weaknesses": "(1) he UAQ method, though effective, introduces a hyperparameter (scale factor ss) that requires tuning, adding complexity. Also the adaption may add more computational overhead\n(2)while QuRL is positioned between PTQ and QAT, its dependence on one-time scaling may limit adaptability to highly dynamic training regimes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)