The Art of Scaling Reinforcement Learning Compute for LLMs
Authors: Fnu Devvrit
Lovish Madaan
Rishabh Tiwari
Rachit Bansal
Sai Surya Duvvuri
Manzil Zaheer
Inderjit S Dhillon
David Brandfonbrener
Rishabh Agarwal
Venue: ICLR 2026 Oral (2026)
Keywords: Scaling
LLMs
Reasoning
Primary area: foundation or frontier models, including LLMs
TL;DR: We study compute scaling properties of RL methods on LLMs

Abstract: Reinforcement learning (RL) has become central to training large language models (LLMs), yet the field lacks predictive scaling methodologies comparable to those established for pre-training.
    Despite rapidly rising compute budgets, there is no principled understanding of
    how to evaluate algorithmic improvements for scaling RL compute.
    We present the first large-scale systematic study, amounting to more than 400,000 GPU-hours, that defines a principled framework for analyzing and predicting RL scaling in LLMs.
    We fit sigmoidal compute-performance curves for RL training and ablate a wide range of common design choices to analyze their effects on asymptotic performance and compute efficiency. We observe:
    (1)
    Not all recipes yield similar asymptotic  performance,
    Details such as loss aggregation, normalization, curriculum, and off-policy algorithm primarily modulate compute efficiency without materially shifting the asymptote, and
    (3) Stable, scalable recipes follow predictable scaling trajectories, enabling extrapolation from smaller-scale runs.
    Combining these insights, we propose a _best-practice_ recipe, ScaleRL, and demonstrate its effectiveness by successfully scaling and predicting validation performance on a single RL run scaled up to 100,000 GPU-hours.
    Our work provides both a _scientific framework_ for analyzing scaling in RL and a practical recipe that brings RL training closer to the predictability long achieved in pre-training.

Published as a conference paper at ICLR 2026
THE ART OF SCALING REINFORCEMENT LEARNING COMPUTE FOR LLMs
Devvrit Khatri Ao
Lovish Madaan to Rishabh Tiwari Rachit Bansal " Sai Surya Duvvuri
Manzil Z Inderjit S. Dhillon David Brandfonbrener" Rishabh Agarwal
UT Austin Meta UCL UCB Harvard University TGoogle
ABSTRACT
Reinforcement learning (RL) has become central training large language mod- els (LLMs), yet the field lacks predictive scaling methodologies comparable to those established for pre-training. Despite rapidly rising compute budgets, there is no principled understanding of how to evaluate algorithmic improvements for scaling RL compute. We present the first large-scale systematic study, amounting to more than 400,000 GPU-hours, that defines a principled framework for analyz- ing and predicting RL scaling in LLMs. We fit sigmoidal compute-perfor performance curves for RI. training and ablate wide range of common design choices to ana- lyze their effects on asymptotic performance and compute efficiency. We observe: (I) Not all recipes yield similar asymptotic performance, (2) Details such loss aggregation, normalization, curriculum, and off-policy algorithm primarily mod- ulate compute efficiency without materially shifting the asymptote, and (3) Sta- ble, scalable recipes follow predictable scaling trajectories, enabling extrapolation from smaller-scale runs. Combining these insights, propose best-practics recipe, SCALERL, and demonstrate its effectiveness by successfully scaling and predicting validation performance hours. Our work provides both a scientific framework for analyzing scaling in RL and practical recipe that brings RL training closer to the predictability long achieved in pre-training.
single RI. run scaled up to 100,000 GPU-
Validation perf., Scaling RL Compute
AIME-24, Scaling RL Compute
0.76
0.72
*/*
training points
x/x
extended training points 8 .x02
0.65
Pass rate (log scale
0.68
filled curve
Pass rate (log scale
0.64
extrapolated curve
0.60
0.57
0.56
0.52
0.48
0.44
ScaleRL-88 Dense
0.41
ScaleRL-8 8B Dense
MaaleRL-178xll MoE
ScaleRt leRL 17Bx16 MoE
0.40
8
16 32
64
128
16 32
64 128
GPU Hours 10 (log scale)
GPU Hours 10a (log scale)
Figure 1: Predicatably Scaling RL compute to 100,000 GPU Hours (a) We run SCALERL for 100k GPU hours on an 8B dense model, and 50k GPU hours on a 17Bx16 MoE (Scout). We sigmoid curve (Equation (I) on pass rate (mean@16) on idd validation dataset up to 50k (and 16k) GPU hours and extrapolate to 100k (and 45k) on the 8B (Scout MoE) models respectively. We trained for 7400 steps for 8B and 7100 steps for Scout, which is 3.5x larger than ProRL (Liu et al. 2025a). The extrapolated curve (x markers) closely follows extended training, demonstrating both stability at large compute and predictive fits establishing SCALERL as a reliable candidate for RL scaling.
(b) Downstream evaluation on AIME- 24 shows a consistent scaling trend for SCALERL, thus generalizing beyond the training data distribution. Moreover, scaling model size substantially improves the downstream and asymptotic RL performance.
'Equal technical contribution

---

Published as a conference paper at ICLR 2026
Comparing ScaleRL with prevalent recipes
0.60
Pass rate (log scale)
ScaleRL
B: 1.97, A: 0.619
0.56
MiniMax
B: 1.77, A: 0.619
Magistral
B: 1.44, A: 0.535
0.52
Qwen2. 5 (DAPO)
B: 1.85,
A: 0.515
DeepSeek (GRPO)
B: 1.17,
A: 0.490
0.48
*
training points
X extended training points
0.44
X
fitted curve extrapolated curve
0.40
2
16 GPU Hours x 10 (log scale)
32
Figure 2: SCALERL is more scalable than prevalent RL methods. We fit sigmoid curves (Equation on iid validation dataset to comeonly-used training recipes like DeepSeek (GRPO) (Guo et al. 2025 ). Qwen- 2.5 (DAPO) (Yu al. 2025), Magistral (Rastogi et al. 2025), and Minimax-MI MI (MiniMax al. 2025). and compare them with SCALERL. SCALERL surpasses
other methods, achieving an asymptotic reward of = 0.61. Stars denote ovaluation points; solid curves show the fitted curve over the range used for fitting, dashed curves extrapolate beyond it. We validate the predictability by running each method for longer ("xx" markers), which align closely with the extrapolated curves for stable recipes like SCALERL and MiniMax. Further description of the individual recipes compared are given in Appendix A.11
INTRODUCTION
Scaling reinforcement learning (RL) compute is emerging as a critical paradigm for advancing large language models (LLMss.. While pre-training establishes the foundations of model; the subsequent phase of RL training unlocks many of today's 's most important LLM capabilities, from test-time thinking (OpenAl 2024. Guo al., 2025) agentic capabilities (Kimi Team al. 2025a). For instance, Deepseek-RI-Zen
used 100,000 H800 GPU hours for RL training - 3.75% of its pre- training compute (Guo et al. 2025). This dramatic increase RL compute amplified across frontier LLM generations, with more than 10x increase from ol to o3 (OpenAl 2025) and a similar 2025)
leap from Grok-3 to Grok-4 (xAl Team 2025).
While RL compute for LLMs has scaled massively, our understanding of how to scale RL has not kept pace; the methodology remains more art than science. Recent breakthroughs in RL are largely driven by isolated studies novel algorithms (e.g.. Yu et al. (DAPO, 2025 and model-specific training reports, such as, |MnnMMxx et al. (2025) and Magistral (Rastogi al. 2025). Critically, these studies provide ad-hoc solutions tailored to specific contexts, but not how to develop RL methods that scale with compute. This lack of scaling methodology stifles research progress: with no reliable way to identify promising RL candidates priori, progress is tied to large-scale experimentation that sidelines most of the academic community.
This work lays the groundwork for science of RL scaling by borrowing from the well-established concept of scaling laws from pre-training. While pre-training has converged to algorithmic recipes that scale predictably with compute (Kaplan et al., [2020, Hoffmann al., 2022 Owen 2024), the RL landscape lacks clear standard. As a result, RL practitioners face an overwhelming array of design choices, leaving the fundamental questions of how scale and what scale unanswered. To address these, we establish predictive framework for RL performance using sigmoid-lin like saturating curve between the expected reward (Rc) on an iid validation set and training compute (C):
Reward Gain
Asymptotic Reward Gain
Rc :o=
(A Ro
X 1
(fixed model and training data)
(I)
B
Compute Elficiency
2

---

Published as a conference paper at ICLR 2026
where 0 < A 1 represents the asymptotic pass rate, Ry represents initial reward 0 training compute, B > 0 is a scaling exponent that determines the compute efficiency, and Cmid sets the midpoint of the RL performance curve. A schematic interpretation of these parameters is provided in Figure
This framework in Equation I allows researchers to extrapolate performance from lower-compute runs to higher compute budgets, enabling them to evaluate scalability of RL methods without incur- ring the compute cost of running every experiment its computational limit.
Guided by this framework, we develop SCALERL, an RL recipe that scales predictably with com- pute. In massive 100,000 GPU-hours training run, we show that SCALERL''s performance closely matches the scaling curve predicted by our framework (Figure TT. Critically, scaling curves extrapolated from only the initial stages of training closely match the final observed performance, confirming the predictive ability of our framework to extreme compute scales.
The design of SCALERL grounded in a comprehensive empirical study of RL scaling that spanned over 400,000 GPU-hours (on Nvidia GB200 GPUs). This study explored numerous design choices at an 8B model parameters scale, where individual runs use up to 16,000 GPU-hours, making them 6x cheaper than experimenting at our largest training run scale. This investigation yielded three key principles:
RL Performance Ceilings are Not Universal: As we scale training compute for different methods, they encounter different ceilings on their achievable performance (A). This limit can be shifted by choices such as the loss type and batch size.
Embracing the Bitter Lesson: Methods that appear superior at small compute budgets can be worse when extrapolated to large-compute regimes (Figure Z We can still identify scalable meth- ods by estimating the scaling parameters (A, B) from the early training dynamics using our frame- work (Equation (I).
Re-evaluating Common Wisdom:
Common interventions thought to improve peak perfor- mance (e.g., loss aggregation, data curriculum, length penalty, advantage normalization) mainly adjust compute efficiency (B), while not changing the performance ceiling considerably.
Based on these insights, SCALERL achieves predictable scaling by integrating existing methods, rather than inventing novel methods. Specifically, SCALERL combines asynchronous Pipeline-RL RL setup (I3.I), forced length interruptions, truncated importance sampling RL loss (CISPO), prompt- level loss averaging, batch-level advantage normalization, FP32 precision logits, aero-variance filtering, and No-Positive-RRsamplin one-out ablation, consuming 16,000 GPU-hours per run.
with each component's contribution validated in leave-
SCALERL not only scales predictably but also establishes new state-of-the-art (Figure 2 it achieves higher asymptotic performance and compute efficiency compared to established RL recipes. Moreover, SCALERL maintains predictable scaling when increasing compute across mul- tiple training axes 5$ including 2.5x larger batch sizes, longer generation lengths up to 32,768 tokens, multi-task RL using math and code, and larger MoE (Llama-4 17B x 16); with benefits that consistently transfer to downstream tasks. Overall, this work establishes a rigorous methodology for cost-effectively predicting the scalability of new RL algorithms.
2 PRELIMINARIES & SETUP
We consider reinforeement learning with LLMs, where prompts x are sampled from data distri- bution D. Our setup follows a generator- trainer split across GPUs: subset of GPUs (generators) use optimized inference kernels for high-throughput rollout generation, while the remaining GPUs (trainers) run the training backend (FSDP) and update parameters. We denote by T and ? the model with parameters on the generator and training backends, respectively. For each prompt, the old policy Tooid on the generator GPUs produces candidate completions, which are then assigned scalar rewards. Policy optimization proceeds by maximizing clipped surrogate objective, taking expectations over x D and rollouts from x Pood
"gen
Training Regimen We mainly conduct our RL experiments using 8B dense model verifiable math tasks with a batch size of 768 and a maximum output sequence length of 14.336 tokens. More details about training, including SFT and hyper-parametent are in Appendix A.3
3

---

Published as a conference paper at ICLR 2026
RL Scaling 11
Expected Reward Rc
A-
Asymptotic reward
Scaling exponent & controls curve steepness
(1+ Ro]/22
compute at 50% of total gain
Grsd
observed training points extended training points
estrepulated fitted curve
Ro
Compute C (log scale)
Figure 3: Interpreting eq. I We provide an example illustrating the roles of parameters A, B, and Cmia- Cmis determines the compute point at which half of the total gain is achieved smaller values correspond to faster ascent toward the asymptote. B controls the curve's steepness, with larger values indicating greater efficiency. A represents the asymptotic performance reached large compute scales. Further discussion provided Appendix A.8
Base RL Algorithm As our starting point in $ 3 we start with "base" algorithm that resembles GRPO (Shao et al., 2024) without any KL regularization term, line with large-scale training reports (Rastogi et al. [2025] MiniMax al. 2025). Additionally, include the asymmetric DAPO clipping Yu et al.) 2025). because of its widespread adoption as a default approach to avoid entropy collapse and maintain output diversity.
For a given prompt x, the old policy (gen(fola) generates G candidate completions { each assigned a scalar reward ri. We compute advantages Å, and group-normalized advantages using:
A mean((r,)fe1)) A A
Each completion yi of length lys| contributes at the token-level importance sampling (IS) ratios (f), with asymmetric upper and lower clipping thresholds, akin to DAPO (Yu al. 2025
(0) = Train (Hit T, Vi,
Trrain (Vi.t) Bus Agen (yv,:)
clipasym (p. ( == clip(p, 1 1- -,,++t)).
Cat Agen Tgen(Bi, (ii,t x. Mi.<c))
(2)
We aggregate losses the sample level, i.e., averaging per-sample token losses before averaging across samples. The surrogate objective is given by:
min (p..()0))) clipasy((p::(0),. ,t)))))))))))))))
[()) = EIND. (1 ~ (-1ar
(3)
Controlling Generation Lengths prevent reasoning output lengths from exploding during training, which harms training stability and efficiency, use interruptions (GLM-V Team et al. 2025 Yang et al., 2025 Xu et al., (0255) that forcibly stop overly long generations by appending of phrase (e.g.. </think>"), signaling the LLM to terminate its reasoning and produce final answer. We revisit this choice later in Section and compare with length-penalty
that penalizes long generations (Yu et al. 2025 Kimi Team
Temi et 2025b
2.1 PREDICTIVE COMPUTE-SCALIN AND FITTING CURVES
Unlike pre-training, which typically uses pawer-law to fit predictive curves, model pass rate versus log(compute) with a sigmoidal function (Equation (I). We do so because we found the sig- moidal fit to be much more robust and stable compared power law empirically, which discuss further in Appendix A.4 Moreover, our choice is consistent with prior work that use sigmoid -iie power laws to capture bounded metrics such accuracy (Ruan al. 2024 Srivastava al. 2022).
Similar to pre-training studies (Li et al. 20256 Porian et al. 2025). we find that excluding the very early low-compute regime yields more stable fits, after which training follows predictable
4

---

Published as a conference paper at ICLR 2026
Comparing oft-policy algorithis
Companing loss functions
Precision Fix Head
0.54
0.58
Poss rate (log scale)
Ress rate (log tcSi))
0.50
0.54
0.54
0.46
pr0-df-pelsyy1:.
CISPO: B: 2.18, 0.595
prD-aff-olicy-lt- 8: 2:00, 8,500
0.40
GSPO: 1.80, 0.590
0.46
1p32
PpclincLL& oftollay. B: 4.55, 0.520
B: 4.53, 0.520
defadt: 4,55,
CPU Hours (log scale)
GPU Hours (log scale)
GPU (log scale)
(a)
(b)
(c)
Figure (a) Comparing "compute-scaling" of asynchronous off-policy RL setups. We report only
the B (scaling exponent) and A (asymptotic pass rate) parameters the fitted sigmoid curve (Equation [ PipelineRI-Ah --iiss is much more efficient and slightly better in the large compute limit. (b) Comparing loss func-
tions: DAPO Yuu et al. 2025), GSPO (Zheng et al., 2025a), and CISPO MMiniMax et al. (2025). We find CISPO/GSPO achieve higher asymptotic reward compared DAPO. (b) Using FP32 precision in the final layer (LM head) gives considerable boost the asymptotic reward.
trajectory. Unless noted otherwise, all our scaling fits begin after ~1.5k GPU hours. Further details of the fitting procedure are provided Appendix A.5 and the robustness of our curve fitting is discussed in Appendix A.7
Interpreting scaling curves Intuitively, sigmoidal curve captures saturating returns grows slowly the low-compute regime, accelerates sharply through a mid-range of efficient scaling, and then saturates at high compute. We also provide schematic interpretation of the parameters A, B, and Cmidd of the sigmoidal curve in Figure B We see that B, Cmis primarily affects the ef- ficiency of the run, and A denotes the asymptotic performance at large compute scale. Further discussion of these parameters provided in Appendix A.8
Scaling curve on held-out validation Consistent with pre-training practice (Hoffmaan et al. 2022 Porian et al.) 2025), we measure predictive performance on in-distribution validation data. Since our training runs span multiple epochs, we hold out randomly selected 1, 000 prompts from the Polaris-53k dataset for validation and use the remainder for training. The scaling curves fitted on the validation points, which measure the average pass rate every 100 training steps, with 16 generations per prompt the 1, 000 held-out prompts.
AN EMPIRICAL STUDY OF RL SCALING
In this section, conduct RL experiments using an 8B dense model verifiable math prob- we study several design axes in terms of their pre- dictable compute-scaling behavior, namely asymptotic performance (A) and compute efficiency (B), as shown in Figure 3
lems. Using the setup described in Section 2
We structure our experiments three stages we first ablate design choices on top of the base- line at 3.5k to 4% GPU-hours since some experimental choices destabilize beyond this scale (Ap- pendix (andix(A.16).. Whenever a design change proved stable, we trained for longer. Then, we combine the best choices into SCALERL and run leave-one-out (LOO) experiments for 166 GPU-hours in Section 4 Here, we assess predictability by fitting on the first 8k GPU-hours and extrapolating the remainder of the run. Finally, demonstrate predictable scaling with SCALERL, also consider training setups with larger batch sizes, mixture-of-experts model, multiple tasks (math and code), and longer sequence lengths in Section 5
3.1 ASYNCHRONOUS RL SETUP
We first investigate the choice of asynchronous off-policy RL setup (Noukhovitch al. 2024), governs training stability and efficiency, generally independent of other design choices. Specifically, consider two approaches for off- off-policy learning: PPO-off-policy-k and PipelineRI.-k.
5

---

Published as a conference paper at ICLR 2026
PPO-off-poliey-k is the default approach for asynchronous RL and has been used previously by Qwen3 (Yang et al. 2025) and ProRL (Liu et al. 2025a). In this setup, the old policy Thu erates reasoning traces for batch of B prompts. Each gradient update processes a mini-batch of B prompts, resulting in & = B/ gradient updates per batch. In our experiments, we fix B 48 prompts (with 16 generations each), and vary A E {1, 8) by setting B = k x 48.
agen gen-
PipelineRL-k is recent approach from Piche al. (2025) and used by Magistral (Rastogi al., 2025). In this regimen, generators continuously produce reasoning traces in a streaming fashion. Whenever trainers finish a policy update, the new parameters are immediately pushed to the genera- tors, which continue generating with the updated weights but a stale KV cache from the old policy. Once a full batch of traces is generated, it passed to the trainers for the next update. In our setup we introduce a parameter k: the trainers wait if they get le steps ahead of the generators.
We compare these approaches in Figure 4a PipelineRL and PPO-off oofl-oliccy achieve similar asymp- totic performance A, but PipelineRL. substantially improves the compute efficiency B; thus reaching the ceiling A faster. This because PipelineRL. reduces the amount of idle time in the training pro- cess. This choice yields reliable gains with fewer tokens, making larger sweeps at lower compute budget possible. We also vary the maximum off off-policyness for PipelineRL and find k = 8 to be optimal, discussed in Appendix A.
3.2 ALGORITHMIC CHOICES
Building the results above, we adopt PipelineRL-8 our updated baseline. We then study six additional algorithmic axes: (a) loss aggregation, (b) advantage normalization, (c) precision fixes, (d) data curriculum, (e) batch definition, and (f) loss type. In Section[4 we combine the best options into a unified recipe, termed BEALEELLSSaale-ee RL), and conduct leave-one-out experiments on a larger scale of 16, 000 GPU-Hours.
Loss type We compare the asymmetric DAPO loss (Eq. 8) with two recently proposed alterna-
tives: GSPO (Zheng et al.] 2025a) and CISPO (MiniMax et al., 2025 Yao al., (025). GSPO
applics importance sampling at the sequence level as opposed to GRPO's token-level formula- tion. Specifically, GSPO alters the token-level IS ratio (Eq. 2 sequence-level ratios: p,(0)
Strain (1.(2,9)/ Tyen (wel (r.a). CISPO simply combines truncated IS with vanilla policy gradient (Ion-
ides 2008). where sg is the stop-gratd gradient function:
(o)
sg(min(pt.. m) log (Teraen (v.,, vicce 01) T
(4)
D. 1 Togg (-) Maa
Figure (heuure4jsshnee shows that both GSPO and CISPO substantially outperform DAPO, improving the asymp- totic pass-rate A by a large margin. CISPO exhibits a prolonged near-linear reward increase, and is marginally better than GSPO later in training, so we opt for CISPO as our best loss type. Further discussion on off-policy loss types, and their hyperparameter robustness is detailed in Section 4 and Appendix A8
FP32 Precision for LLM logits The generators and trainers rely on different kernels for inference and training, leading to small numerical mismatches in their token probabilities (He & Lab 2025). RL training is highly sensitive to such discrepancies, since they directly affect the IS ratio in the sur- rogate objective. MMniMax et al. (2025) identified that these mismatches are especially pronounced at the language model head, and mitigate this by FP32 computations at the head for both the gen- erator and trainer. As shown Figure [4c the precision fix dramatically improves the asymptotic performance A from 0.52 to 0.61, and inclyde it in our SCALERL recipe.
Loss Aggregation We evaluate three strategies for aggregating the RI. loss: (a) Sample average where each rollout contributes equally (as in GRPO, Appendix A.2 (b) Prompt average where each prompt contributes equally (as in DAPO, Appendix A.2 (c) Token average where all token losses in the batch are averaged directly, without intermediate grouping. The comparison results are shown in Appendix A.9 (Figure T0a). We find prompt-average achieves the highest asymptotic performance and therefore use this choice for SCALERL.
Advantage Normalization We compare three variants of advantage normalization: (a) Prompt level where advantages are normalized by the standard deviation of rewards from the rollouts of the same

---

Published as a conference paper at ICLR 2026
10"
7 (pass--rate) (log scale)
training points
extended training points
Nethod
fitted fixed 0.685
original fitted paranaters
fitted
Can
/
10
extrapolated curve
ScaLeRL
2.81
2542
8.618
L00 proept: Ivl adv nore
1.82
2463
9.590
Length penelty
1.75
2660
10"
L00-batch
2503
9.595
L00-PPD-off- 8o
3489
8.685
L00-no- wpw--precisionn
2857
10
100-sanple-awy
2621
0.600
2621 0.605
L00-dapo
3093
10
16
GPU Hours 10 (log scale)
Figure 5: Leave-One-Out (LOO) Experiments: Starting from SCALERL, wC revert one design choice at a time to its baseline counterpart and re-train. Most LO0 variants reach a similar asymptotic reward, with SCALERL outperforming slightly overall, The main difference in these methods lies in efficiency. To highlight this, we re-arrange Equation d1) into F(R,) = C#, where F(R,) = Cma/ A-Ru Roo vs. log C. This makes slope B directly visible, showing that SCALERL has the highest compute efficiency.
Ra- 1), and plot log F(Re)
prompt (as in GRPO, Appendix A.2 (b) Batch level where advantages are normalized by the stan- dard deviation across all generations in the batch, as used by Hu et al.] (2025a); Rastogi et al. (2025) (c) No normalization where advantages are computed as raw rewards centered by the mean reward of the prompt's s generations, without variance scaling (as proposed in Dr. GRPO (Liu et al. 2025b)) A comparison plot is shown in Appendix A.. (Figure I0b), and all three methods are oberseved to yield similar performance. We therefore adopt batch-level normalization as it is theoretically sound and marginally better.
Zero- Varo-Varinnce Filtering Within each batch, some prompts yield identical rewards across all their generations. These "zero-variance" prompts have zero advantage and therefore contribute zero pol- icy gradient. The default baseline includes such prompts in loss computation, but it is unclear whether they should be included in the effective batch. To test this, we compare the default setting against an effective batch approach, where only prompts with non-zero variance are included in the loss calculation. as done by Seed et al. (2025). Note that zero- variance filtering differs from dynamic sampling DAPO (Yu et al.] 2025). The former merely drop the prompts, while latter resamples more prompts until the batch is full. We show in Appendix A.9 (Figure 11a that using the effective batch performs better asymptotically; and we adopt in our SCALERII recipe.
Adaptive Prompt Filtering A number of data curriculum strategies have been proposed for RL. training to improve sample efficiency (An et al., [2025] Zhang et al., [2025b] Zheng et al.] (0256). Here we evaluate a simple variant, introduced by An et al. (2025), with the key observation that once a prompt becomes too easy for a policy, typically remains easy. Since such prompts consume some compute but longer contribute useful gradient signal, it is better to exclude them from future training. We implement this by maintaining a history of pass rates and permanently removing any prompt with pass rate 2 0.9 from subsequent epochs-we we call this No-Positive Resampling. In Appendix MaaaiixXA9) (Figure TTb) we compare this curriculum against the default setting where all prompts are resampled uniformly throughout training. We see that the curriculum improves scalability and the asymptotic reward A.
no-Positive-Reeampling
4 SCALERL: SCALING RL COMPUTE EFFECTIVELY & PREDICTABLY
From the design axes studied above, we consolidate the best-performing settings into a single recipe, which we term SCALERL (Scale-able RL). SCALERL is an asynchronous RL recipe that uses PipelineRL with 8 steps off- off-policyness, interruption-basee length control for truncation, FP32 computation for logits, and optimizes the JSCALERI. (0) loss. This loss combines prompt-level loss aggregation, batch-level advantage normalization, truncated importance-sampling REINFORCE

---

Published as a conference paper at ICLR 2026
Scaling Generation Length
Scaling Model
Scaling Size
maining poinns
pointe
0.66
marting poims
5.550
3.62
ectropeitted
Frodd runce
8 0.54
acrrazo curve
tmae mire
0.50
fitedd care
8
9.90
46
Sextatt-ar7t 1 1.82 0.013
1.80,
5.830
LTB0e N6E-t5738 8: 1.00, 0.722
2:00
0/11
0.38
129
GPU Hours 102 (in scale)
CPU Hours x (log scale)
GPU
(a)
(b)
(c)
Figure 6: Scaling RL across several axes. We scale up the ScaleRL recipe across three axes: (a) sequence
length: Larger equence is slower but reaches higher asymptotic reward. (b) model size: using the larger 17Bx16 Llama-4 model reaches much higher terminal reward and shows better generalization, and (c) butch size: larger batch is slower training but settles higher asymptote.
Loss (CISPO), zero-variance filtering, and no- no-positive resampling:
JSCALERI
E anD {u
sg(min(pp...))44 log n Tlvain (Vr.e)
Bold (112)
Traain (Hi.e)
A Anorm
Aad, O < mean (1r)((())))))))) < pass rate(s)) < 0.9,
Pi,t
Tgen (3/,,)
where sg is the stop-gradient function, Ass is the standard deviation of all advantages Å, in a batch and pass rate(x)) is the historical pass rate of prompt For forced interruptions, use phrase: "Okay, time is up. Let me stop thinking and formulate an answer now. </think>".
Leave-One-Out (LOO) Ablations To validate that these choices remain optimal when combined, we conduct leave-one-our (LOO) experiments: starting from SCALERL, we revert one axis at a time to its baseline counterpart from Section 2 This ensures that each decision contributes positively even in the presence of all others. Figure 5 reports these experiments, each scaled to 16k GPU hours.
Across all axes, SCALERL consistently remains the most effective configuration, slightly outper- forming LOO variants either in asymptotic reward or in compute efficiency (refer to the last column in the Figure 5 table). Since most LOO variants reach similar asymptotic pass rates, we transform the sigmoidal fit to power-law fit, to highlight efficiency differences via the slope B (details in 5). Concretely, we average the asymptotic reward A across all runs, re-fit the curves with this fixed A, and then compare slopes (measuring efficiency) in Figure 5
In all our LOO experiments as well as independent SCALERL runs, we fit the sigmoidal curve up to 8k GPU-hours and extrapolate to 16k GPU-hours, observing that the predicted curves align closely with both training and extended points. This demonstrates the stability and predictability of SCALERL and other stable, scalable recipes under large-scale RL training.
5 PREDICTABLE SCALING RETURNS ACROSS RL COMPUTE AXES
Given a fixed or growing compute budget, which scaling knob -context length, batch size, genera- tions per prompt, and model size buys the most reliable performance gain, and how early can we predict that return? We answer this by (i) fitting the saturating power-law in eq. ( early in training for each setting (precisely, half the target budget), (ii) extrapolating to the target budget, and (iii) extending training to verify the forecast. Across all axes below observe clean, predictive fits whose extrapolated curves align with the extended trajectories, mirroring the behavior seen in our 100,000 GPU-hour run (Figure D). and the cross-recipe comparison in Figure 2
Model scale (MoE) Does ScaleRL remain predictive and stable on larger models? Training the 17Bx16 Llama-4 Scout MoE with ScaleRL exhibits the same predictable scaling behavior as the 8B model, with low truncation rates and no instability pathologies (Appendix(A.16]) A. Figure shows the training curve. The extended points align with the fitted curve, supporting the model-scale
8

---

Published as a conference paper at ICLR 2026
invariance of recipe. Moreover, the larger 17Bx16 MoE exhibits much higher asymptotic RL performance than the 8B dense model, outperforming the 8B's performance using only 1/6 of its RL training compute.
Generation length (context budget) Increasing the generation length from 14k to 32k tokens slows early progress (lower B and higher Ciii) but consistently lifts the fitted asymptote (A), yielding higher final performance once sufficient compute is provided (Figure This validates long-context RL as a ceiling-raising knob rather than a mere efficiency trade-off. Extrapolations made from the fit correctly forecast the higher 32k-token trajectory when training is extended.
Global batch size (prompts) Smaller-batch runs show early stagnation on downstream benchmarks even as in-distribution validation performance continues to improve. Larger batches reliably im- prove the asymptote and avoid the downstream stagnation we observe in smaller-batch runs, Fig- ure (c shows the same qualitative pattern at mid-scale: small batches may appear better early but are overtaken as compute grows. In our largest run in Figure moving to batch size of 2k both stabilized training and yielded a fit that extrapolated from 50k GPU hours to the final 100k point.
Generations per prompt (fixed total batch) For a fixed total batch, better to allocate more prompts more generations per prompt? Sweeping generations per prompt 8, 8,16,24,32 and ad- justing prompts to keep total batch fixed leaves fitted scaling curves essentially unchanged (Ap- pendix A.) suggesting that, at moderate batch, this allocation is second-order choice for both A and B. Clearer differences may emerge at much larger batches, which we leave as future work.
RELATED WORK
We detail two most relevant works to study in this section. ProRL (Liu al.) 2025a demon- strates that prolonged RL fine-tuning on LLMS (~ 2000 optimization steps, 64 batch size) for 16K GPU-hours using mix of reasoning tasks uncovers novel solution strategies beyond model's base capabilities. This longer training regimen delivered significant gains on a 1.5B model, rivaling the performance of larger models on some benchmarks. ProRL's contributions lie in specific heuristics for stability KLLrreulaaizztion, policy resetting, entropy controls, etc.) to achieve high performance out of 1 1.5B model.
Liu et al. (202cc) offer complementary perspective and ablates various design choices under con- sistent conditions on Qwen-3 4B/8B (Yang al., 2025), and presents a minimalist combination, LitePPO. that outperforms more complex methods like GRPO (Shao et al., (024) and DAPO (Yu jtt al. 2025) on smaller scale models and compute. This yields valuable algorithmic insights, but the focus is comparative empirical findings, rather than scaling behaviour. While focus defining universal scaling trajectories for reasoning tasks, Vattikonda et al. (2026) has employed ex- tensive hyperparameter sampling and bootstrapping provide statissical diagnosis' of RL training recipes in domain -specffc contexts like web-agent post-training.
None of these work study "scaling" properties of these methods. In fact, the main comparisons done on downstream evaluations, which may not be not the right metric to study predictable scaling. Rather, as done in pre-training and in our work here, we study performance on in-distribution held out eval set. In contrast to the mentioned related works, our work develops and validates a compute- performance framework with predictive fits, while operating 6z larger than ProRL) and model scale compared to the above studies. Additionally, our findings yield near state-of-the-an RL recipe that can scale predictably to over 100,000 GPU-hours without any stability issues. The rest of the related work is deferred to Appendix A.1
much larger compute budget (c.g,
7 DISCUSSION & CONCLUSION
In this work, we study the scaling properties different techniques used in RL for LLMs in pursuit of predictable scalable recipe. With this mission, we derive method for fitting predictive scaling fits for accuracy on the validation set that allows us to quantify the asymptotic performance and compute efficiency of an RL method. Using this methodology, primary contribution is to conduct a careful series of ablations of several algorithmic options that go into the RL recipe. For each ablation, choose the option with higher asymptotic performance when possible and improved

---

Published as a conference paper at ICLR 2026
efficiency otherwise. Combining these choices yields the SCALERL recipe which scales better than all existing recipes in our experiments. A few observations are in order:
Compute scaling extrapolation. An important insight of our scaling methodology is that we can use smaller-scale ablations in a systematic way to predict performance at larger scales. This allows to create our final scalable recipe.
Most important decisions. The off-policy algorithm, loss function, and model precision are the most important decisions from ablations. Each of the other decisions does not have a large individual effect, but as we see from the leave-one-out experiments, they still do have some cumulative impact (in terms of efficiency) when all combined.
Asymptotic performance vs. efliciency. For many of our ablations, we found the better option to improve both efficiency and asymptotic performance, but this is not always the case (e.g. for FP32, Figure Ac). When doing the "forward" ablations starting from the baseline method, opt for asymptotic performance first and foremost. Interestingly, when doing the "backward" leave-one-out one-out ablations from the SCALERL recipe, we find very little impact asymptotic performance from each decision, but each component of the algorithm seems to help efficiency. This shows that the cumulative effect of the changes is quite robust.
While we report transfer to downstream evaluations, our primary focus is studying predictive scaling, which is characterized through in-distribution performance curves on a held-out dataset from training prompts (Li et al., 2025b, Muennighoff et al.) 2025). This still leaves the question of how well the LLM would generalize from the training distribution to held out test sets. While a full characterization of generalization is beyond the scope of our work, we do observe correlation between in-distribution vali- dation and downstream generalization performance. However, there are some algorithmic choices that seem to help generalization more, that we want to note here including: larger batch size (Section 1.) reducing truncations (Section A.16). longer generation lengths (Figure (7b), and larger model scale (Section [
Generalization.
Figure
Multi-task RL. While our experiments focus mainly on the math domain, we also evaluate SCALERL under multi-task RL training. As shown in Figure joint training on math and code yields clean, parallel power-law trends for each domain, with extended runs remaining aligned with the extrapolated curves. While our preliminary results are promising, it would be interesting to thoroughly study predictability of compute scaling for multi-task RI. with different training data mixtures.
Future work A natural next step is to derive predictive "scaling laws" for RL across pre-training compute, model size, and RL training data. Future studies can also include other axes of RL compute scaling, such as incorporating structured or dense rewards (Setlur et al. 2024) and more compute- intensive generative verifiers (Zhang al.) (025a)) to find optimal compute allocation for RL train- ing. Finally, the framework introduced here can be applied to study the scaling behavior of other post-training regimes, including multi-turn RI,, agentic interaction, and long-form reasoning.
There are of course many design choices in RL, so we don't think that our SCALERL. recipe is the end of the story. We hope that our focus on scalable RL and methodology for predicting scal- ability can inspire future work to push the frontier of RL for LLMs even further. To enable future studies to fit compute-perfe performance RL scaling curves, we release minimal code repository at www.dewwrit.com/sctn www./r///aall...ooooooomomoo wwweeeee.ceeee.eeeeeeeeeee fiitti.
REFERENCES
Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong. Polaris: post-training recipe for scal- ing reinforcement learning on advanced reasoning models, 2025. URL https: http:/kkknpppp..p
github..o//blog//005/Pooss
AoPS. AIME problem set 1983-2025, U.L.attsss//arrffppoooooooooo//.
wi wiki/inden php/ TME. B0a0______b____ ems_and Solutions
10

---

Published as a conference paper at ICLR 2026
Quentin Carbonneaux, Gal Cohen, Jonas Gehring, Jacob Kahn, Jannik Kossen, Felix Kreuk, Emily McMilin, Michel Meyer, Yuxiang Wei, David Zhang, et al. Cwm: An research code generation with world models. arXiv preprint arXiv:2510.02387, 2025.
11m for
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. The entropy mechanism of reinforcement learning for reasoning language models, 2025. httL::tt:://// http:://aaaiiv... org.ii.oorrgaoooo 25w.:/555..o. 22617
GLM-V Team, Wenyi Hong. Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang. Yan Wang, Yean Cheng, Zehai He, Zhe Su. Zhen Yang. Ziyang Pan, Aohan Zeng, Baoxu Wang. Bin Chen, Boyan Shi, Changyu Pang, Chenhui Zhang. Da Yin, Fan Yang, Guoqing Chen, Jiazheng Xu, Jiale Zhu, Jiali Chen, Jing Chen, Jinhao Chen, Jinghao Lin, Jinjiang Wang. Junjie Chen, Leqi Lei, Letian Gong, Leyi Pan, Mingdao Liu, Mingde Xu, Mingzhi Zhang, Qinkai Zheng, Sheng Yang, Shi Zhong, Shiyu Huang, Shuyuan Zhao, Siyan Xue, Shangqin Tu, Shengbiao Meng. Tianshu Zhang, Tianwei Luo, Tianxiang Hao, Tianyu Tong, Wenkai Li, Wei Jia, Xiao Liu, Xiaohan Zhang, Xin Lyu, Xinyue Fan, Xuancheng Huang, Yanling Wang. Yadong Xue, Yanfeng Wang. Yanzi Wang, Yifan An, Yifan Du Yiming Shi, Yiheng Huang, Yilin Niu, Yuan Wang, Yuanchang Yue, Yuchen Li, Yutao Zhang, Yuting Wang. Yu Wang, Yuxuan Zhang, Zhao Xue, Zhenyu Hou, Zhengxiao Du, Zihan Wang, Peng Zhang, Dehing Liu, Bin Xu, Juanzi Li, Minlie Huang, Yuxiao Dong, and Jie Tang. Glm-4.5v and glm-4.1 1--thinkin: Towards versatile multimodal reasoning with scalable reinforcement learning, 2025. http:: /ttp///aaavvvvvvv org/ abs /2507 00006
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma. Xiao Bi, et al. Deepscek-rl incentivizes reasoning in ilms through reinforce- ment learning. Nature, 645(808) (55(8081)6633--11 1:633-638, 2025.
Horace He and Thinking Machines Lab.
Defeating nondeterminism in LLM infer- ence. Thinking Machines Lab: Connectionism, 2025. doi: (0.644344ml..22500910.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03877 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness both prompt and reward models, 2025a. URL. htttps://arxiv.crr///a
2501. 03262
Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zero An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290, 2025b.
Edward L lonides. Truncated importance sampling. Journal of Computational and Graphical Statis- fies, 2722)195--11, 2008
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836) 2020.
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534 2025a.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llss. arXiv preprint arXiv:2501. /2599, 2025b.

---

Published as a conference paper at ICLR 2026
Aonian Li. Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, et al. Minimax-01: Scaling foundation models with lightning attention. arXiv preprint arXhv:2501.08313, 2025a.
Margaret Li, Sneha Kudugunta, and Luke Zeuteeoyer. (mis)fitting: A survey of scaling laws, 2025b. URL htttp:://axxii wttpxiii..rggbb/////ee/2/ 18969
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoming boundaries in large language models, 2025a. URLhttps::///// /ttp/aaiiiv/v..///vv//e//e/aaaiv org/ abs
24864
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and
Min Lin. Understanding rl-zero-like training: A critical perspective, 2025b. URL https:
/aaxii.org/ab///5333 20783
Zihe Liu, Jiashun Liu, Yancheng Hc, Weixun Wang, Jiaheng Liu, Ling Pan, Xinyu Hu, Shaopan Xiong. Ju Huang, Jian Hu, Shengyi Huang, Siran Yang, Jiamang Wang, Wenbo Su, and Bo Zheng. Part i: Tricks traps? deep dive into rl for Ilm reasoning, 2025c. URL httth:///axxxiii
hts/abb/25005 08221
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https:
//arxiv. abs 01771105100
Michael Luo. Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Erran Li Li, Raluca Ada Popa, and lon Stoica. Deepcoder: A fully open-source 14b coder at 03-mini level, 2025. URL https:
/ //www. www..togetter wttpl///.aaaa////aaaaaaa/aoo//// deepcodee Notion Blog.
Aaron Meurer, Christopher P Smith, Mateusz Paprocki, Ondfej Čertík, Sergey B Kirpichev, Matthew Rocklin, AMIT Kumar, Sergiu Ivanov, Jason K Moore, Sartaj Singh, al. Sympy: symbolic computing in python. PeerJ Computer Science, 3:3103, 2017.
MiniMax, :, Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, Chengjun Xiao, Chengyu Du, Chi Zhang, Chu Qiao, Chunhao Zhang, Chunhui Du, Congchao Guo, Da Chen, Deming Ding, Dianjun Sun, Dong Li, Enwei Jiao, Haigang Zhou, Haimo Zhang, Han Ding, Haohai Sun, Haoyu Feng, Huaiguang Cai, Haichao Zhu Jian Sun, Jiaqi Zhuang, Jiaren Cai, Jiayuan Song, Jin Zhu, Jingyang Li, Jinhao Tian, Jinli Liu, Junhao Xu, Junjie Yan, Junteng Liu, Junxian He, Kaiyi Feng. Ke Yang, Kecheng Xiao, Le Han, Leyang Wang, Lianfei Yu, Liheng Feng. Lin Li, Lin Zheng, Linge Du, Lingyu Yang. Lunbin Zeng, Minghui Yu, Mingliang Tao, Mingyuan Chi, Mozhi Zhang, Mujie Lin, Nan Hu, Nongyu Di, Peng Gao, Pengfei Li, Pengyu Zhao, Qibing Ren, Qidi Xu, Qile Li, Qin Wang, Rong Tian, Ruitao Leng, Shaoxiang Chen, Shaoyu Chen, Shengmin Shi, Shitong Weng. Shuchang Guan, Shuqi Yu, Sichen Li, Songquan Zhu, Tengfei Li, Tianchi Cai, Tianrun Liang, Weiyu Cheng, Weize Kong, Wenkai Li, Xiancai Chen, Xiangjun Song, Xiao Luo, Xiao Su, Xiaobo Li, Xiaodong Han, Xinzhu Hou, Xuan Lu, Xun Zou, Xuyang Shen, Yan Gong, Yan Ma, Yang Wang. Yiqi Shi, Yiran Zhong, Yonghong Duan, Yongxiang Fu, Yongyi Hu, Yu Gao, Yuanxiang Fan, Yufeng Yang. Yuhao Li, Yulin Hu, Yunan Huang, Yunji Li, Yunzhi Xu, Yuxin Mao, Yuxuan Shi, Yuze Wenren, Zehan Li, Zelin Li, Zhanxu Tian, Zhengmao Zhu, Zhenhua Fan, Zhenzhen Wu, Zhichao Xu, Zhihang Yu, Zhiheng Lyu. Zhuo Jiang, Zibo Gao, Zijia Wu, Zijian Song, and Zijun Sun. Minimax-ml: Scaling test-time compute efficiently with lightning attention, 2025. URL https: //arxiv.orgaabs////
13585
Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Noua- mane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models, 2025. URL http:: // /tt////axvvv//////////eee/.aaava wrw.:oorg/aaaaee abs/ 2205. 16264
Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux. Arian Hosseini, Rishabh Agarwal, and Aaron Courville. Asynchronous rlhf: Faster and more efficient off-policy for language models. arXiv preprint 20.::2400./8522 2024.
12

---

Published as a conference paper at ICLR 2026
OpenAl. Openai ol system card. arXiv preprint drXtn:2412.16722,222
OpenAl. Introducing OpenAl o3 and 04-mini.
2025. Accessed: 22 September 2025.
David Owen. How predictable is language model benchmark performance? arXiv preprini
arXiv:2401.04757, 2024.
Alex Piche, Rafael Pardinas, Ehsan Kamalloo, and Dzmitry Bahdanau. Pipelinerl. 2025. URL co/b
inerl
Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. Resolving discrepancies in compute-optimal scaling of language models, 2025. URL https://arxiv
oxg/abs/2406.191
Abhinav Rastogi, Albert Q Jiang, Andy Lo, Gabrielle Berrada, Guillaume Lample, Jason Rute, Joep Barmentlo, Karmesh Yadav, Kartik Khandelwal, Khyathi Raghavi Chandu, et al. Magistral. arXiv preprint arXiv:2506. 10910, 2025.
Yangjun Ruan, Chris J. Maddison, and Tatsunori Hashimoto. Observational scaling laws and the pre- dictability of language model performance, 2024. UKLLhtttss///axxxi.orr///b//bb/
10938
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL h
ByteDance Seed, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, al. Seed1. 5-thinking: Advancing superb rea- soning models with reinforcement learning. arXiv preprint arXiv:2504.13914 2025.
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for I1m reasoning, 2024. URL
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang. Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.033000, 2024.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alona et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint M.XYv:2006.04615. 2022.
Dheeraj Vattikonda, Santhoshi Ravichandran, Emiliano Penaloza, Hadi Nekoci, Megh Thakkar, Thibault Le Sellier de Chezelles, Nicolas Gontier, Miguel Muñoz-Mármol, Sahar Omidi Shayegan, Stefania Raimondo, Xue Liu, Alexandre Drouin, Laurent Charlin, Alexandre Piché, Alexandre Lacoste, and Massimo Caccia. How train your Ilm web agent: statistical diagno- sis, 2026.
04103
Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari Morcos, Ali Farhadi, and Ludwig Schmidt. Stable and low-precision training for large-scale vision-language models. Advances in Neural Information Processing Systems, 36.10 2023.
XAI Team. Grok 4. 2025. URL httphhtts///..ii/iaatte
Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning- arXiv preprint 4rXiv:2405.0045). 2024.
Yuhui Xu, Hanze Dong. Lei Wang, Doyen Sahoo, Junnan Li, and Caiming Xiong, Scalable chain of thoughts via elastic reasoning, 2025. URL UNN.htpp::///rrswoorrr///aa 2250://..//250..000 25511
13

---

Published as a conference paper at ICLR 2026
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang. Wenbiao Yin. Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https:// pttttps://axxii.ortt org. abs 09388
Feng Yao, Liyuan Liu, Dinghuai Zhang, Chengyu Dong, Jingbo Shang, and Jianfeng Gao. Your efficient RL framework secretly brings you off-policy RL training, 2025. URL https: Accessed through social media reference.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint 2503v:2003.14476 2025.
Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan. What's behind PPO's collapse in long-cot? value optimization holds the secret. arXiv preprint arXiv:2503.0149,, 2025.
Yu Yue, Yufeng Yuan, Qiying Yu. Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05//6 18, 2025.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agar- wal. Generative verifiers: Reward modeling next-token prediction, 2025a. URL https:
7/
2408 15240
Ruiqi Zhang, Daman Arora, Song Mei, and Andrea Zanette. SPEED-RL: Faster training of reason- ing models via online curriculum learning, 2025b. URL/ttppss//arvvvvorr//bb///
09016
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy op- timization, 2025a. URL/https: /ttttp::///rxiv.../ www.wii.rrraaaaee wwgg/aasss...oooooo.e 2250718077
Haizhong Zheng, Yang Zhou, Brian R. Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, and Beidi Chen. Act only when it pays: Efficient reinforcement learning for LLM reasoning via selective rollouts, 2025b. URLpttts: wwtp::/wwwwg//a/////e/a// 02177
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 EXTENDED RELATED WORK
wave of recent work has applied Reinforcement Learning (RL) to improve the reasoning abili- ties of large language models (LLMs): often achieving state-of-the-ar results on challenging tasks (OpenAl 2024, Guo et al., 2025 Seed et al., 2025] Carbonneaux et al. 2025). OpenAl's ol se- ries of models established that large-scale RL can substantially enhance long-horizon reasoning, but did not release any details on how these models were trained. Deepseek R1 (and RI-Zero) (Guo) jet al. 2025) provided the first comprehensive study on training tigh-performing and long Chain- of Thouht (CoT) models primarily via RL, documenting emergent behaviours under extended RL without any reliance on reward models (Lightman et al.) 2023) Monte Carlo Tree Search (MCTS) (Xie et al. 2024).
The earliest widely referenced RLVR (verifiable-reward) algorithm underlying this wave of reason- ing development is Group Relative Policy Optimization (GRPO), introduced in Shao et al. (2024). GRPO is critic-free, group-relative policy gradient with PPO-style clipping that replaces learned value baseline with group baselines to reduce computational cost and stabilize credit assignment for long CoTs. While GRPO catalyzed rapid progress, subsequent work document its limitations (token-level clipping. model collapse risks) and motivate different group- or sequence- level vari-
ants Yu et al. 2025
Yue et al. 2025 Hu al. 2025b; Zheng et al., 2025a).
Yu et al. (2025) propose the Decoupled clip and Dynamic Sampling Policy Optimization (DAPO), where they decouple Clow and Czigh clipping in the GRPO objective and do Clip-Higher for Chigh to avoid entropy collapse. Furthermore, they do dynamic sampling of prompts in given batch to avoid samples with zero variance (or advantage) which contribute zero policy gradients. Finally, they employ token-level loss aggregation unlike GRPO, which uses sample-level loss averaging. With these modifications, they are able to surpass the vanilla GRPO bascline while avoiding entropy collapse in the RL training. In parallel, Yue et al. (2025) develop VAPO; value-augmented PPO tailored for long CoT' with strong stability and outperforming value-free baselines like GRPO and DAPO. They combine value pre-training and decoupled Generalized Advantage Estimation (GAE) from VC-PPO (Yuan et al.] 2025). loss objective modifications from DAPO, and propose length- adaptive GAE come up with an open recipe, VAPO, that has been used train large MoE models in Seed et al. (2025). Similarly, other technical report like Magistral (Rastogi et al. 2025), Kimi- k1.5 (Kimi Team al.) 20256), Minimax-01 01 (Li al. (025a) detail various details on their RL training recipes, but don't 't share extensive experiments on why their design choices are better than the baselines.
A.2 RL FOR LLMS: GRPO AND DAPO
Group Relative Policy Optimization (GRPO) GRPO (Shao al. 2024) adapts PPO Schulman ct al. (2017) for LLM fine-tuning with verifiable rewards. For given prompt æ, the old policy gen (Qota) generates G candidate completions [ each assigned a scalar reward rg. To empha-
size relative quality within the group, rewards are normalized as
A mean(fr)) std (cd(fr)),) 1=11)) +
(5)
Each completion yı of length 1y; contributes at the token level through ratios
pi,(()) Terain (yiee Ø. Me, 0) Tgen (Vi,t 36, Di, Oata)
(6)
The GRPO objective averages across both completions and tokens:
G
ludl min (p),(0)A,, clip(p.. + }
JGRPO (0) = EE p>.D {ur 1 gont leel
(7)
Thus GRPO preserves token-level policy ratios as in PPO, while using sequence-level, group- normalized advantages to stabilize learning under sparse rewards.
15

---

Published as a conference paper at ICLR 2026
Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) DAPO (Yu et al. extends GRPO with two key modifications. First, it replaces symmetric clipping with asymmer- ric clipping, using distinct thresholds for upward and downward deviations: clipasym (p, cp, 1 1 Et), where and e+ are hyper-parameters.
2025
Second, DAPO changes the aggregation scheme to operate at the prompi level. For given prompt ~ D. the old policy produces G completions {ui)en with advantages {A,} (Equation ) Let
T = |% denote the total number of tokens across all completions. With token-level ratios as in Equation The DAPO surrogate objective is
w ΣΣ T
1
min (p.)(())A clipasym (P.)))
(f) = =EaD (ucher (a.
(8)
This prompt-level normalization ensures that each token contributes equally to the prompt's loss, regardless of the number or length of its sampled completions. DAPO also introduces dynamically dropping O-variance prompts from the batch during training and filling the batch with more prompts until the batch is full. We skip that change here since its effect is similar to having larger batch size.
A.3 TRAINING SETUP
Datasets For small-scale SFT. we use a curated data mix of reasoning traces. We filter this dataset by removing trivial prompts, discarding solution traces exceeding 12k tokens, and decontaminating with AIME 2024/2025 (AoPS) 2025) and MATH-500 (Hendryeks et al., 202T) benchmarks. For the RL stage, we use the Polaris-53K dataset (An et al., (225)) for most of our runs; additionally using the Deepcoder dataset (Luo et al. 2025) for runs with both math and code.
Supervised Fine-tuning We SFT using batch size of 2M tokens, max sequence length of 12288, and a learning rate of 3 x 10 using the Adam W optimizer (Loshchilov & Hutter 2019) on 32 H100 GPU nodes for approximately 4 epochs and 32B tokens in total.
Reinforcement Learning We allocate 14k generation budget during RL training, where 12k to- kens are allocated to the intermediate reasoning ("thinking"), followed by 2k tokens for the final solution and answer. We sample 48 prompts in each batch, each with 16 generations per prompt. Thus, we get the total batch size as 768 completions per gradient update step. The rewards are given 11 to correct and incorrect traces respectively. We use constant learning rate of 5 10-7, AdamW optimizer (Loshchilov & Hutter 2019) with e = 10-15, weight decay of 0.01 (default in AdamW), and a linear warmup of 100 steps. The lower is avoid gradient clipping (epsilon underflow) (Wortsman et al. 2023).
We automated checkers like Sympy (Meurer al. 2017) or Math-Verifyl for assessing the correctness of the final answer for math problems after stripping out the thinking trace ..chink>...</hiite
We use a custom code execution environment for coding problems involving unit tests and desired outputs.
We used 80 Nvidia GB200 GPU for single run, with compute budget ranging from 3.5-4K GPU hours for establishing different design choices in Section 3.2 16K for the leave-one-our experi- ments (Section 4. and finally 100K GPU hours for our larger scale runs (Section 5 We adopt generator-traine split between GPUs. For 80 GPU experiments, we set 64 of those generators, responsible for the generation of reasoning trace using the optimized inference codebase. The re- maining 16 GPUs act trainers, which receive generated trajectories, perform policy updates, and periodically broadcast updated parameters back to the generators.
Training Regimen All experiments are conducted on the RL for reasoning domain, where the model produces a thinking trace enclosed with special tokens (<think> final solution. Unless noted, training uses a sequence length of 16, 384 tokens: 12, 288 for thinking, 2, 048 for the solution, and an additional 2, 048 for the input prompt. We adopt the 12, 288 thinking
</think>) and
16

---

Published as a conference paper at ICLR 2026
budget for faster iteration, and show in Section that SCALERL extrapolations remain predictive when training with larger thinking budgets (32, 768). For math RL experiments, we use the Polaris- 53K dataset (An et al. 2025) with batch size of 768 (48 prompts with 16 generations each). In our setup, scaling RI. compute corresponds to running multiple epochs over the training prompts.
A.4 WHAT CURVE TO FIT?
Pre-training curves are usually fit using power-law equation (Li ct al. 2025b Kaplan et al. 2020 Muennighoff et al. 2025). which in our case would model performance as Rc A
D/C - Co. where D is constant, and Co marks the compute threshold beyond which the law holds. Intu- itively, this implies that each multiplicative increase in compute yields a constant proportional gain in performance. For RL for several reasons. First, for bounded metrics such as accuracy or reward, sigmoidal curves provide better predictive fits (Ruan et al. 2024 Srivastava al.. 2022); ; observe the same, with accurate extrapolation to higher compute (Figure []. Second, power laws are unbounded at low compute and are typically fit only beyond threshold Co. In RL, where total training spans far fewer steps (c.g., only ~75 evaluation points to fit only in Figure T. discarding early points a lot further reduces the already limited data available for fitting. Third, empirically, sigmoidal fits are substantially more robust and stable than power-law fits. Concretely, consider the 100k GPU-hour run the 8B dense model shown Figure When we fit power-law curve between 1.5k-50k GPU hours, predicts an asymptotic performance of A = 1.0, which is clearly incorrect the actual curve saturates near 0.65. In contrast, the sigmoidal fit yiclds an accurate prediction of = 0.645. Moreover, the power- law fit is highly sensitive to the chosen fitting regime: fitting over (5k, 50k) GPU hours instead gives A 0.74, while the sigmoidal fit remains robust and still predicts = 0.645. Power-law mod- els only recover the correct asymptote when fitted exclusively in the high-compute regime (e.g., 30k-60k GPU hours). However, our goal is to predict large-scale performance from lower-compute regimes, where such long runs unavailable.
however, find sigmoidal fit (eq. (I) more appropriate
Given these considerations, we use the sigmoidal form throughout our analysis. Intuitively, a sig- moidal curve captures saturating returns grows slowly in the low-compute regime, accelerates sharply through a mid-range of efficient scaling, and then saturates at high compute as it approaches a finite performance ceiling.
One thing to note is that at high compute regime, sigmoidal curve behaves same as power-law. Concretely, we can have the following approximation of sigmoidal curve:
A Ro
Rc Ro
(sigmoidal curve from eq.
1 1 (Cmia/ C)
B mid C
Rg Ru + (A Ro)
1
(For C >> Cmrid high compute regime)
AA- Ro) C
A
mid
C
D
= A
B
where above D = (A - Ro) CB mid- And this is the same form of power-law mentioned at the start of
this section.
A.5 FITTING SCALING CURVES
We fit the sigmoid-law equation in Equation T to the mean reward on our held-out validation set. This set consists of 1, 000 prompts held out from the Polaris-53k (An et al., (025) math dataset, with 16 generations sampled every evaluation step performed at 100 steps intervals.
Directly fitting all three parameters {A,, B, Cmid) } is challenging. Instead, we perform grid search over A E {0.450, 0.455, 0.460.... 0.800) and Civid E [100, 40000] (searching over 100 linearly separated values), and for each candidate A, Cmid fit B. The best fit (measured by sum of squared residuals) across this grid is selected as the final curve. We use SciPy's curve fit with default initialization; varying the initialization strategies produced identical results. To enable future studies
17

---

Published as a conference paper at ICLR 2026
Analysing Error Margin across different Runs
Scout LOO No FP32
0.74
*/+/*
training points
training points
Pass rate (log scale]
0.62
x/>/x
extended training points
Pass rate (log scale)
0.70
</x
extended training points
fitted curve
fitted
0.58
-
extrapolated
0.06
extrapolsted curve
0.54
0.62
0.58
0.46
Sca JeR-1
ScaleRL-2
8: 1.97, A:
B: 1.92,
A: 0.610
0.54
1 MoE B: 1.66, A: 0.710
178x16
ScaleRL-3
8: 2.20,
A: 0.595
ScaleRL 178x16 Mot tp32 1.81, 8.708
0.38
0.50
8
2
d
64
GPU Hours 10a (log scale) (a)
GPU Hours 10 (log scale)
(b)
Figure 7: (a) Variance in scaling fits. We train 3 independent runs of SCALERL to measure vari- ance. We observe a +.002 error margin for asymptotic performance A. (b) FP32 LOO on Scout:
Comparing SCALERL on Scout with and without FP32 precision fix at the LM Head. SCALERL performs better with the FP32 fix.
to fit compute- perforraneee RL scaling curves, promise to release code repository after the acceptance of the paper.
To estimate the error margin of our fits, trained three independent SCALERL runs with batch size of 768 and generation length of 14k (as used in Section 4 shown in Figure 7a We found that the fit values of A varied by most +0.015, suggesting 0.02 reasonable error margin on the estimates of asymptotic performance. Estimating the error margin for the fitted value B is difficult, as different algorithms with different A values can have different error margins for B. However, for the purosse of comparing algorithms,, we can safely deduce that if two methods achieve similar A values (within 0.02), the one with higher B when refit done with the average of A values is at least good in terms of efficient scalability.
A.6 COMPARING ALGORITHMS
Consistent with observations in large-scale pre-training, where the loss exhibits sharp initial drop before settling into a predictable power-law decay (Li al. 2023b), we observe a similar two-phase behavior in RL. The mean reward increases rapidly, almost linearly, during the ~firtt epoch (~ 1k steps, or ~ 1.5k GPU Hours for most runs), after which the curve follows sigmoidal-law behavior (see Figure 12 to the "sigmoid" like curve). Our sigmoidal-law fits are applied this latter portion of the training curve.
Unlike pre-training, our main goal which algorithms and design choices scale reliably, and to design algorithm that exhibits predictive nature. Achieving highly robust fits typically requires very large runs with hundreds or thousands of evaluation points, which is impractical in our setting for two reasons. First, running all ablations such scale would be computationally prohibitive. Second, many RL algorithms we compare are themselves not scalable to such extreme budgets: they often saturate much earlier or even degrade with more compute due instability. For example, our baseline method (Section 3.2) destabilizes beyond 3500 GPU-hours, since overlong generation truncations exceed 10% of generations reducing the effective batch size. More discussion on this is in Section A6
predict the performance of a fixed recipe, but identify
As we ablate across different axes in Section 3.2 at higher compute. Some ablated variants can scale further, e.g.. ~ 5k GPU hours for e = 0.26 in DAPO, ~ 6k GPU hours with the FP32 precision fix, and 7k GPU hours for CISPO. Once we combine the best design choices, we obtain a stable and scalable recipe, which allows us to run leave-one-out (L.O0) experiments for 1600 GPU hours per run.
we discover design choices that improve stability
18

---

Published as a conference paper at ICLR 2026
A.7 ROBUSTNESS OF FITS
One may wonder how robust our fitted curves are. We address a few relevant points below:
changing the fitting regime (e.g,, including or excluding the initial 1.5k GPU-hour range) yields similar predictable results. For instance, in the 100k GPU-hour run on the 8B dense model, fitting over (1.5k, 50k) gives
For stable and scalable experiments, including all runs from
0.645, while (0, 100k) gives = 1.56, A 0.655, (0, 50k) gets B = 1.7, A = 0.645, and (5k, 50k) gives B = 1.67, A = 0.645. Across these regimes, parameter values remain within the expected error margin (Section
1.70,
We nonetheless skip the low-compute regime because early training phases, especially in less stable setups from Section 3.2, often plateau prematurely or deviate from the sigmoidal trend due to transient instabilities (see Appendix AA.6] A.16). Excluding this region allows the fit to focus on the mid-to-hip compute range where saturation behavior is clearer and more consistent.
The 1.5k GPU-hour threshold is heuristic chosen empirically: it approximately corre- sponds to one epoch for most experiments in Section 3.2 Larger cutoffs reduced the num- ber of fitting points, while smaller ones often introduced noise. We found 1.5k GPU hours to provide the best balance between fit stability and sample coverage, consistent with prac- tices of skipping low-FL.OPs regime in pre-training scaling analyses and fitting (Li et al., 2025b).
A.8 INTERPRETING SIGMOIDAL CURVES
Figure presented an example fit illustrating the influence of parameters A, B, and Cmid- Here, extend this with additional illustrations: Figure 8al Figure [8]] and Figure 9a vary B, Cuid. and A respectively, while keeping the other parameters fixed. We observe that and Cmid primarily affect the efficiency of scaling, whereas A determines the asymptotic performance achievable large compute. In Figure 9b we see a case of two runs where one is much more efficient, hence shows initial promising gains, but converges to lower asymptote, while the other progresses more slowly yet ultimately surpasses it due to higher A. In practice, scaling strategies should prioritize design choices that raise the asymptotic ceiling A, and only then optimize for efficiency parameters such as B Gmid-
Scallig
Scallng sffect Gea
Expectadd Rloward R.
Paymptti roward A
Akyreptatic rewar Al
Coay 536.
12t6
caneaate at 50% (aliigga
Commute (log soaee
Compute log scalei
Figure 8: Keeping all parameters same and only changing (a) B, (b) Omid. Both these parameters modulate the efficiency of the training run.
A.9 FORWARD AND LOO ABLATIONS
We show additional results for Section 3.2 in Figures 10a 10b We also plot the pass rate vs compute leave one out experiments from Section in Figure
A.
ARE THE DESIGN CHOICES WORTH IT?
In Section 3.2
certain design choices alter asymptotic performance, such as loss type (Figure44b) and FP32 precision (Figure 4c). However, in our LOO experiments with SCALERL (Figure 5) these components appear less critical individually (last column the figure). This raises the question of whether certain design choices can be safely left at their "default" values.
19

---

Published as a conference paper at ICLR 2026
Scaling fl: Oftect af
Scaling ni tettal Asymptotic Reward
Poymesst ronard (ml
Aoymenot revrd Infai
ranquen
Compute scalel
Compuse dog
(a)
(b)
Figure 9: (a) Keeping all parameters same and only changing A. yet reach a higher asymptote. When designing scalable methods, one should prioritize choices that raise the asymptotic ceiling A, since the ultimate goal is maximizing performance scale.
(b) A design choice can be less efficient
Loss Agg Aggregation Techniques
Advantage Normalization Techniques
0.54
0.54
Pass rate (log scale)
Pass rate (log scale)
0.50
0.50
0.46
prompt avg: B: 2.98, 0.535
0.46
no-norm: 3.69, 0.525
sample avg: B: 4.55, A: 0.520
batch-lv/nnomm: B: 3.15. A: 0.530
token avg: B: 2.04, A: 0.535
prompt-lvinnomm: B: 4.55, A: 0.520
GPU Hours 10a (log scale) (a)
102 (log scale)
(b)
Figure 10: Comparing (a) loss ageregation, (b) different advantage normalization techniques.
We argue the answer is no. Even when choice seems redundant in the combined recipe, it can still provide stability or robustness that can become decisive in other regimes. For example, while the FP32 precision fix makes little difference with dense 8B trained with SCALERL (Figure 5). provides large gains in GRPO/DAPO-style losses by mitigating numerical instabilities. This indi- cates that its benefits extend beyond the specific SCALERL configuration study. To further test this, we ran leave-one-out experiment on the Scout 17 MoE and observed that FP32 precision improves overall scalability (Figure 7bp.
A similar case arises with the loss type. In Figure 5 reverting to DAPO yields similar asymptotic performance to CISPO within SCALERL. Nonetheless, as discuss in Appendix A.18 CISPO is markedly more robust to the choice of IS-clipping parameter mmax, reducing the sensitivity of training hyperparameter tuning. Moreover, it's also more efficient than DAPO, as seen LOO experiment (B 2.01 vs B = 1.77). This justifies preferring CISPO, even if a carefully tuned DAPO variant can perform similar asymptotically.
In summary, even when individual design choices appear redundant within the combined recipe, they often enhance training stability, robustness, or efficiency ways that generalize across models and setups. SCALERL retains such components not just for marginal gains in a specific configuration, but because they address recurring sources of instability and variance that arise across reinforcement learning regimes.
A.11 CONTROLLING GENERATION LENGTH
One common concern in reasoning RL is to control exploding generation lengths, which harms both training efficiency and stability (Appendix 16) We consider two approaches: (a) interruptions, used in works like GLM-4.1V (GLM-V Team et al. 2025). ) and Qwen3 (Yang al., (025) and (b) length penalties, used in works like DAPO Yu et al. (025), Kimi (Kimi Team et al. 2025b), Magistral (Rastogi al. 2025). and Minimax-Mi (MiniMax et al. 2025).
20

---

Published as a conference paper at ICLR 2026
O-Variance Filtering
No Positive Resampling vs Uniform Sampling
0.54
Pass rate (log scale)
Pass rate (log scale)
0.50
0.46
0.46
batch: B: 4.55, 0.520
No-Positive-Resammling B: 3.85, A: 0.525
effec batch: B: 2.58, A: 0.540
Sniforr-SSappiin 4.55, 0.520
2 GPU Hours x 10 (log scale)
GPU Hours x 10? (log scale)
(a)
(b)
Figure 11: (a) "Zero" variance filtering: We filter out "zero" variance (accuracy 0 or 1) samples in batch since they contribute zero policy gradient and find it achieves higher asymptote, and (b) Adaptive prompt sampling: Filtering out prompts with pass rate 2 0.9 in subsequent epochs results higher asymptotic performance.
647
1007 ring solts
04)
0.00
shaat DE 19 poits
Nidd pnve
2-60, 0-014
concicate
11 1.00
1ax pla
2.39
6.253
3-03
19099
31
1.62
0.085
0.2
042
6.42
chu hours x 102 (log scalo)
dnu Hours x 162 rlog scale)
GPU Hours 10p (log stale
(a)
(b)
(c)
Figure 12: Comparison of different leave-one-out strategies using 16k GPU-hours budget. loo-plan refers to using prompt level advantage normalization, loo-Ip means using length penalty, loo-batch refers to using the entire batch without any O-variance prompts filtering. loo-8op refers using PPO-offpolicy-8, loo-fp32 means not using FP32 precision fix, loo-savg means using sample average loss aggregation, loo-dapo means using DAPO loss function instead of CISPO. Table Figure the values of Cmin addition to A and B. We notice that all methods have similar values of A (within +0002 emror margin range). Hence, all methods scale well, but affect efficiency parameters B and Cmiid-
Interruptions forcibly stop generation by appending a marker phrase such as "Okay, time is up. Let me stop thinking and formulate a final answer </think>" signaling the model terminate its reasoning and produce final answer. In our setup, the interruptions tokens are placed randomly in between [10k; 12k] token length, to induce generalization to different generation lengths.
Length penalties instead reshape the reward. Following DAPO Yu 2025), we penalize overly long completions with a tolerance interval Teguche:
Limxx
1, -1,
- 1y|
Riength (y) = clip
Leache
This penalty added only to the correct traces, discouraging excessively long generations. In the length penalty experiment, set Lmax 14k tokens and Leache 2k tokens.
In Section 4 we compare length penalty and interruption a scale of 66 GPU-Hours. We find that replacing interruption with length penalty in our final SCALERL. recipe does not improve perfor- mance.
A.12 PIPELINERL.
Using the baseline setup, we ablated the off- off-colicy parameter in PipelineRL (Figure 13). Both 4 and off- off-policyness performed equally well, and we adopt 8 as the default setting when updating the baseline in Section 3.1
21

---

Published as a conference paper at ICLR 2026
Comparing PipelineRL max off-policy
0.58
8-offpolicy: B: 4.55, A: 0.520 4-offpolicy: B: 4.14, A: 0.520 12-offpolicy: B: 4.22, A: 0.490
0.54
0.50
0.46
1
2
4
GPU Hours 103 (log scale)
Figure 13: Different off-policy runs with PipelineRI.
Why does PipelineRL consistently outperform the classic PPO-off- off--ffllyccy approach (Sections 3.1 []d We attribute this to its closer alignment with on-policy training. In PPO-off-policy, gener- ation and training proceed in alternating phases: the trainer operates strictly on batches that are as off-policy the chosen parameter k, making updates based on stale rollouts. In contrast, Pipelin- passed to the trainer;
eRL operates in a streaming fashion. As soon as a batch is available, model update is ready, it is shared back the generators, who immediately use it includidnng in the continuation of partially generated traces. This tight feedback loop keeps training closer to the on-policy regime, reducing the mismatch between generator and trainer distri- butions.
likewise,
Moreover, since there is not much waiting in trainers and generators, it helps in training being more efficient as well.
A.13 ENTROPY CURVES: SCALING BATCH SIZE
We tracked entropy on the held-out validation set throughout training. Across all experi- ments Seents-Spanning variations in batch size, number of tasks, generation length, and model scale-we observed consistent overall decrease in entropy.
An interesting finding that entropy may always offer predictive insight into the performance, as proposed by some recent works like Cui et al. (2025). In Section A.13 we plot entropy for SCALERL runs with batch sizes 768 and 2048. Despite the 2048-batch size achieving much stronger downstream performance at every stage (Figure 17a). both runs followed nearly identical entropy trajectories per step (Section A.13]. This highlights an important point although entropy is sometimes used as proxy for exploration, simply maintaining higher entropy does not translate into better generalization. Instead, larger batches reduced effective exploration similar smaller batches, per step, yet still yiclded substantially better performance underscoring batch size as an important decisive factor.
Overall, our findings suggest that while entropy decreases consistently during training, it is not necessarily reliable predictor of downstream performance. This observation reinforces the need to focus algorithmic and scaling choices (e.g., batch size, off-policy method) in adddition to entropy dynamics when aiming for improved performance, both on training distribution as well as downstream task distribution.
22

---

Published as a conference paper at ICLR 2026
Entropy across training run
0.30
Sc666 ScaleRL-bs2048
0.25
Entropy
0.20
0.15
1000
2000
3000
4000 Steps
5000
6000
7000
Figure 14: Comparing entropy of large and smaller batch size runs across training steps.
MME244, Scaling Number of Generations
Scaling Number of Generations
0.65
0.67
*
training points
ScalcRL-8gen -8gen
Pass rate (log Scale)
0.63
0.59
titted curve
training points
Pass rate (log scale)
0.57
ScaleRL- -2gen
SealeRL-16ger
extrapolated curve
ScaleRL- -32gen
0.55
0.51
0.49
0.47
Lol -8gen L
8: 2.44, A: 0.585
1.92, A: 8.610
0.43
ScaLeRL S4gge
2.22, A: 0.590
0.41
ScaLeRL- -2ggen
B: 2.07, A: 0.595
0.39
8 GPU Hours 10 (log scale)
16
GPU Hours 10 (log scale)
(a)
Figure 15: Scaling to (a) different number of generations per prompt, (b) Downstream performance of different number of generations per prompt
14 SCALING ON MULTIPLEE AXES
We provide the remaining scaling to different axes figure here in Figure 15 and Figure I6 and the corresponding downstream evaluation in Figure 18 We also provide the value of A, B, Caid in T
Experiment
Cmid
B
A
SCALERL
2542
1.92
0.610
SCALERL-332
11272
1.89
0.645
SCALERL-8gen
2542
2.44
0.585
SCALERL--24gm
3054
2.22
0.590
SCALERL-32gen
2936
2.07
0.595
SCALERL-Scoul
4242
1.65
0.710
SCALERL-bs512
2818
1.77
0.605
SCALERL-bs2044
10909
1.70
0.645
-cat math curve
2896
2.05
0.595
SCALERL-math+code code curve
1675
1.09
0.615
Table 1: Cmidd B, and values for the large scale runs in Section 5
23

---

Published as a conference paper at ICLR 2026
Scaling Number of Tasks
0.67
training points
0.63
x|xIX
extended training points
Pass rate (log scale)
0.59
fitted curve
extrapolated curve
0.55
0.51
0.47
Multi task, code
B: 1.09, A: 0.615
0.43
Multi task, math B: 2.05, A: 0.595
Math
B: 1.92, A: 0.610
0.39
2
16
GPU Hours x 10 (log scale)
Figure 16: SCALERL scales predictably on math and code. We report both the code and math validation set performance on the joint math+code RL run; along with the math only SCALERL run as a reference. These results demonstrate that our sigmoidal compute compute-performanee relationship holds across task mixtures, and that SCALERL's scalability generalizes beyond single domain training.
AIME-24, Scaling Batch Size
AIME-24, Scaling Generation Length
Pass rate (log scale)
0.65
ScateRL-bss bs512
ScateRL-bs/i
Pass rate (log scale)
0.65
Sca -4k
ScaLeRL 14k
ScaleRL. RB22eRL-bs2048
0.57
0.49
0.49
0.41
0.41
8
16
32
64
128
8
16
32
64
GPU Hours x 10 (log scale)
GPU Hours 10 (log scale)
(a)
(b)
Figure 17: Scaling RL batch size and generation length. larger batch size and generation length are slower in training but settles at higher asymptote. These show an inverse trend initially where smaller values seem better at lower compute budget, but reach higher asymptotic performance at larger scale.
A.15 DOWNSTREAM PERFORMANCE
In Figure T and 18 we report a representative set downstream evaluation curves. These include SCALERR runs with batch sizes (512, 768, 2048). long-context training run with 32k gen- cration length, the large-model (Scout) training run, a multi-task run (math + code), and different number of generations per prompt (with fixed batch size) run. For each setting plot performance against compute. Moreover, we see downstream performance better for experiments like larger batch sizes, longer generation length, and large model size mirroring similar order for validation set curves.
A.16 TRUNCATIONS AND TRAINING INSTABILITIES
Across our experiments we found that training instabilities were often linked to truncations. As gen- eration length grew, many RL runs exhibited fluctuating truncation rates that sometimes increased over training. At batch size 768, we observed that truncations in the range of 10-15% typically destabilized training, with performance degrading and not recovering without intervention. Exam-
24

---

Published as a conference paper at ICLR 2026
AIME-24, Scaling Number of Tasks
LiveCodeBench, Scaling Number of Tasks
Multi task, math
0.65
Pass rate (log scale)
Math
Pass rate (log scale)
0.39
0.57
0.37
0.41
Multi task, code
0.35
8 16
2
GPU Hours 10 (log scale)
GPU Hours 102 (log scale)
(a)
(b)
Figure 18: Downstream performance of (a) different number of generations per prompt, on AIME, (b) LiveCodeBench (Jan-June 2025) performance on math+code run, (c) AIME-24 performance on math+code run
ples include the extended GRPO run in Figure 2 where instability correlated with rising truncation rates, and the updated baseline used in Section
By contrast, SCALERL runs were more stable. On the 8B model, truncations remained below 5% for over 90% of training. At batch size 2048, truncations were slightly higher, occasionally approaching ~ 7%. This increase was largely attributable to longer average generation lengths observed during training. which naturally raise the chance of exceeding the budget. Nevertheless, because the effective batch size (after excluding truncated samples) remained large, training stability was preserved. Intuitively, larger generation length budget should help reduce truncations. Training with 34k generation length (batch 768) remained stable truncations briefly spiked to ~ 4% but quickly fell below 2%.
Larger models were even more robust. On the Scout rum, truncations remained consistently below 2%. and for > 90% of training steps were under 1%. This likely reflects both the inherent ability of larger models to regulate generation length and their stronger instruction-followin ability, which made interruption signals more effective.
Overall, we suggest practitioners monitor truncation rates closely. Our findings indicate that high truncation rates are reliable warning signal of instability, while larger models, higher generation budgets, and careful design choices (as in SCALERL) substantially mitigate this risk.
A.17 COMPARING PREVALENT METHODS
In Figure 2 we compared some popular training recipes with SCALERL. We briefly describe these existing recipes here.
DeepSeek (GRPO) This recipe mostly follows the DeepSeck Guo ct al. 2025 work. We use GRPO as the loss function (Section[A.2) A with Cmin = Cmaz 0.2, sample average loss aggregation, and PPO-offpolicy-8 algorithm. We saw the training became unstable post 6k GPU Hours due to truncations (Section A.16).
Qwen2.5 (DAPO) This recipe follows DAPO Yu al. (205)). It includes the DAPO loss function (Appendix A.2 with Cmin 0.2, man 0.26 (Appendix[A.Ie) 1.. This recipe uses PPO-offpolicy- 8, and prompt average loss aggregation. The only change from the original DAPO paper (Yu et al.) 2025) was regarding dynamically filling in the batch. Specifically DAPO drops O-variance prompts and samples more prompts until the batch full. In our codebase, this was not efficent because for PfO-offpolicy algorithm, had generators pre-decide that each generator will generate rollouts for #prompts/ #gmeeratoes. Therefore, specific generator had more O-variance prompts, it sampled further prompts complete its share of #prompes generators This could lead to other generators being
25

---

Published as a conference paper at ICLR 2026
stalled and overall slowdown. Hence, to get around this issue, rather kept larger batch size 1280 (80 prompts, 16 generations each), and dropped varinnce prompts from the batch. We noted that post-dropping, the effective batch was still greater than 768, what used for SCALERL. Therefore, if at all, we gave some advantage to the DAPO recipe.
Magistral This refers to the recipe used in Rastogi al. (2025). It includes similar recipe as DAPO with the main difference being PipelineRL used as the off oof-policy algorithm.
MiniMax This refers the recipe used in MiniMax al. (2025). It uses CISPO loss. FP32 precision fix at the LM head, PPO-offpolicy algorithm, and prompt average. Similar to DAPO, drops O-variance prompts well and hence give larger batch size of 1280 as well.
A.18 LOSS TYPE- STABILITY AND ROBUSTNESS
As discussed below, GRPO/DAPO-style losses are highly sensitive to the choice of clipping ratio hyperparameter Grax-- In contrast, CISPO and GSPO show far greater robustness. For example, in Appendix 18.8.2 varying <max for CISPO between {4, 5, 8} produced no significant differences in performance. For GSPO, the 10-4 clipping scale used in the original paper (Zheng al. (025a) did not work well in our setting. We therefore ablated across broader scales and found that once the correct order of magnitude was identified (e.g.. 4 10-3 and higher), performance was stable and largely insensitive to fine-graing grained changes (e.g., f4 10 1x5x003)
Effect of max GRPO/DAPO-like loss
CISPO Upper Clipping Ratio clip 4: 2.14, A: 0.595
0.62
Pass rate (log scale)
Pass rate (log scale)
0.58
clip 8: 2.56. A: 0.590
0.50
Emas
0.27:
0.54
Emat
0.26: B: 2.34, A: 0.530
Emax
0.25: B: 4.55, A: 0.520
0.46
0.50
0.23: B: 1.96, A: 0,510
Emas
0.22:
A: 0.500
Emax
0.20: B: 3.19, A: 0.500
0.46
2
4
GPU Hours 10 (log scale)
GPU Hours x 10 (log scale)
(b)
Figure 19:
Ca)mprring upper clipping ratio of DAPO loss function. Change of emaa fundamen-
tally changes the asymptotic performance value A. (b) CISPO clipping ratio ablations
A.18.1 DAPO CLIPPING RATIOS
In this section, we analyze the role of the clipping threshold 6max in DAPO Loss Function (eq. The hyper-parameter sensitivity of Eman has been observed in prior work, for example, GRPO typ- ically sets 6max 0.2, while DAPO uses 0.28. However, beyond tuning sensitivity, we find that max directly alters the scaling behavior of the algorithm. As émax increases, the terminal reward A increases until an optimal range is reached, after which A decreases again. This is striking effect: unlike many hyper-paramette that merely shifts the convergence speed, 6max governs the asymptotic error itself.
A.18.2 CISPO CLIPPING RATIOS
We ablate the higher clipping ratio for CISPO, keeping the lower clipping ratio fixed at 0 (Fig- ure 19b Across wide range of values, we find little difference in performance, indicating that CISPO is largely insensitive to this hyperparameter. This robustness mirrors our findings for GSPO (Section 1.33.)), and stands in contrast DAPO/GRPO-style objectives, which highly sensi- tive to the exact choice clipping threshold. Such stability under hyperparameter variation makes CISPO strong candidate for default use in large-scale training.
26

---

Published as a conference paper at ICLR 2026
A.18.3 GSPO ABLATIONS
We ablate the clipping-ratio scale used in GSPO, as shown in Figure 20a The default 10 scale as given in the GSPO paper Zheng et al. (2025a) does not scale the best for our 8B model. The 10 3 scale performs as well as, or better than, alternatives (Figure 20a Given this scale, we further 10 10-3 and found x5x 10-3) yielded slightly better
varied the upper clipping ratio in {4 x 10 fit (Figure 20b
An important observation is that GSPO is quite robust to the choice of clipping ratio. Once the correct scale identified, most nearby values or even larger scale perform similarly. This robustness contrasts sharply with DAPO-style losses, which are highly sensitive to the exact value of the higher clipping ratio, as noted in Section 3.2
GSPO Clipping Ratio Scale
GSPO 10-1 Scale Clipping Ratio
0.58
0.58
Poss rate (log scale)
Pass rate (log scale)
0.54
0.54
2
0.50
0.50
1ppo_4_33___0
gspo 0.585
9.20_4.3...2.::342.
0.565
35po_5_3_c_3:888 1.79, 0.590
0.46
0.575
0.46
g5p0 g5po_4_3...33 3: 2.37. 0.585
4
8
2
4
GPU Hours 10 (log scale)
GPU Hours x 102 (log scale)
(a)
(b)
Figure 20: (a) GSPO Scale comparison. gspo x y e in the legend means an upper and lower threshold of far x 10-* and x 10-*) respectively. (b) With 10 scale, we found similar perfor- mance for both 4_3 and 5_3_e_3, with latter performing slightly better.
A.18.4 GSPO vs CISPO
Despite hyperparameter robustness, encountered stability issues with GSPO. On multiple occa- sions, GSPO runs diverged mid- traitraining, leading to sudden drops performance. For 8B models, restarting from stable checkpoint allowed recovery, but this strategy failed larger models such as Scout, where instability persisted despite repeated resetting to a stable checkpoint. While we checked to the best of our ability for any implementation bugs, we did not find one.
Overall, while all three loss families can be competitive under tuned settings, CISPO offers the best balance stability and robustness hyperparameters, making it our recommended choice.
A. ANALYZING FP32 PRECISION FIX
we introduced using FP32 at the LM Head as a way to decrease trainer-generator numerical mismatch. Here, we provide some additional experiments and analysis to support how this fix affects the training. First, we notice in Figure 21 that using the FP32 fix, the importance sampling gets more accurate (closer to 1, and hence log of it close 0) at the Oth step of the training. We also provide scatter plot in Figure 22 as alternate way to visualize this. Second, Figure 23 we log the clipping fraction, and the max and min value of importance sampling ratio during the training
In Section 3.2
of baseline with and without FP32 precision fix, and also the effect of using the fix on only either trainer or generator. We notice that while having FP32 precision fix on either the trainer generator gives some benefit, having both gives the most accurate IS ratios.
A. SAMPLE EFFICIENCY AND SYSTEM THROUGHPUT
We analyze some of our ablations to understand the gains got from sample efficiency vs system throughput. First, in Figure 24 we plot the tokens processed per step by ScaleRI. using PipelineRI.,
27

---

Published as a conference paper at ICLR 2026
base
fp32 last layer (both)
0.4
0.4
log p_ p_train log p Pegen
0.2
0.2
0.0
0.0
-0.2
-0.2
-0.4
-0.4
-10.0
-7.5
-5.0
-2.5
0.0
-10.0
-7.5
-5.0
-2.5
0.0
log p_gen
log p_gen
(b)
Figure 21: FP32 fix at the LM Head reduces the mismatch in probabilities of trainers and generators. We plot the log of importance sampling vs log probability of the generator (a) without the FP32 fix, (b) with the FP32 fix. Note that the above figures are the Oth step of training, hence should ideally have values of 0. Experiment conducted on the 8B Dense model.
Without FP32
With FP32 fix
-1
-1
-2
log p train
log p train
-2
-4
-3
-5
-4
-6
correlation-0.9974
correlation-O.9955
-7
-5
-6
-2
-5
-2
log P_gen
log p_gen
(a)
(b)
Figure 22: Scatter plot of lof og probabilities with and without using FP32 precision fix. This logged at step=0, hence the expected behavior is to have all the points on x=y line. We notice that with the fix, values are much more aligned, with higher correlation value as well.
vs LOO-PPO-OOO PPO-Off-PPlicy-- K. We notice that the sample efficiency of both the methods are similar, and the performance gains per step is similar as well. Although, we noticed in Figure [ Sop) that PipelineRL is more efficient (has larger value of efficiency parameter "b"). This means that most the gains come from system throughput, and not an inherent sample efficiency.
The other two interesting comparisons where expect sample efficiency to be affected are: (1) Using No Positive Resampling. Here, since we filter out the easy prompts, one may expect gen- erations to be longer the batch of no-so-easy prompts, and hence more tokens processed. But
28

---

Published as a conference paper at ICLR 2026
No TP32
0.80
Clio fraction
0.003
Max IS Ratio
1.7
treee gesertee
0.75
F732
F732
pererdtr
1.6
0.002
1152
Min
0.65
0.000
1.3
100
200
303
400
500
100
200
300
400
500
100
200
300
400
500
Step
Step
Step
(b)
(c)
We compare not having FP32 fix, having FP32 fix on only the trainer, the generator, and on both trainer and generator. (a) The fraction of tokens clipped, (b) Maximum IS ratio in the batch, (c) Minimum IS ratio in the batch. We notice that having on either of just the trainer or generator makes IS ratio more accurate than having on none, but still leads less accurate IS values compred to having on both. Using FP32 precision fox both makes the ratio much more accurate, with the maximum and minimum values being closer and having the least clipping.
Figure 23:
9
Tokens Processed Step
HipelineRL
PPO-Offpolicy-K
8
0.60
7
Pass rate
0.55
6
0.50
5
0.45
PipelineRL
PPO-Offpolicy-K
4
0.40
0
2000 4000
6000
2000
4000
6000
Gradient Update Step (a)
Gradient Update Step
(b)
Figure 24: (a) We compare the tokens generated and processed per step by ScaleRL (which uses PipelineRL) and Offpolicy-K, while rest of the recipe being same as ScaleRL. We notice that the #tokens processed per gradient update step approximately same for both the methods.
The latter means PipelineRL replaced with PPO-
(b) We also plot per step performance of these two methods as well, and observe similar per step gains, with PipelineRL per- forming slightly better. From these figures, deduce that the efficiency benefits of PipelineRL comes mainly because of system level improvements and not inherent sample efficiency
we notice in Figure 25a that's not necessarily the case, and the tokens processed roughly same as if one uses uniform sampling (2) Model Size. As we increase model size, expect the average generation length to be smaller for given prompt. This is because better model may require less thinking solve the prompt. We observe that this is indeed the in Figure 25b
A.21 ADDITIONAL DOWNSTREAM EVALUATION PLOTS
We discussed in Section and Appendix A. some of the downstream performances of ScaleRL on AIME-24. 24. Here, we present two additional plots, evaluating ScaleRL on 8B Dense and 17Bx16 MoE model on MATH-500 and AIME-25 datasets. We notice that similar to validation curves (Figure 6 the downstream performance improves in the expected way roughly following a similar sigmoidal curve.
29

---

Published as a conference paper at ICLR 2026
Tokens Processed Stee
Tokens Processed Step
2.25
2.00
1.75
No Positive Resample Uniform Sampling
1.50
ScaleRL-8B Dense
1.25
ScaleRL-17Bx16 MoE
2000
4000
6000
1000
2000
3000
4000
Step
Step
(a)
(b)
Figure 25: (a) We compare the tokens generated and processed per step by ScaleRL. (which uses No Positive Resampling) and pling replaced with Uniform Sampling, while rest of the recipe being same ScaleRL. We notice that the #tokens processed per step is approximately same for both the methods. (b) We do similar analysis using ScaleRL but on different model sizes 8B Dense and 17Bx16 MoE. Since a more capable model needs less thinking to solve given prompt, we notice that 't's more sample efficient.
The latter means No Positive Resam-
AIME-25, Scaling Compute
0.49
MATH 500, Scaling Model Size
0.45
Pass rate (log scale)
Pass rate (log scale)
0.41
0.915
0.37
0.875
0.33
0.29
ScaleRL- 8B Dense ScaleRL - MoE
0.835
ScaleRL-8B Dense
ScaleRL-178a0 MoE
0.25
0.795
7k
8k
16k
6dk 128k
1
2
4 8
16
32 64
GPU Hours 103 (log scale)
GPU Hours x 10a (log scale)
(a)
(b)
Figure 26: We evaluate ScaleRL 8B Dense and 17Bx16 MoE, (a) AIME-25 and (b) MATH- 500 tasks. We notice that similar to AIME-24 (Figure [T), performance improves the expected manner.
30


Reviews:
[{"review_id": "z22e5SWt5U", "reviewer": "Reviewer_zx5H", "summary": "This paper establishes a scaling law for RLVR training by running experiments on various RL algorithms with different implementation practices. The scaling law categories two key perspectives, asymptomatic reward gain, which defines the upper bound of the training process, and compute efficiency, which quantifies the performance gain versus used compute in RL training. The scaling law allows for predicting performance after RL training up to 100k GPU hours based on 40k GPU hours. The authors also investigate several practices in RLVR training, including loss aggregation, loss function, advantage normalization, dynamic fitlering, FP32 for policy head. Combining the best practice, the authors provide a recipe named ScaleRL that outperforms baselines in terms of both final performance and training efficiency.", "questions": "1. Is \"400,000 GPU hours\" in the intro a typo? According to Figure 1, datapoints up to 100k GPU hours are predicted based on datapoints under 40k GPU hours.\n2.  How could the scaling law used in practice? In the LOO experiments, the authors fit the parameters A&B in two ways, one is to fit both parameters simultaneously, while the other one is to fit B with a fixed A. In practice, suppose we run two different trials with different configurations, how to compare the two parameters if both A&B are fitted? Also, if we only fit B with a fixed A, how to determine A before at least one complete trial has been completed?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed scaling law is shown through extensive experiments.\n2. The ScaleRL recipe considers a list of different techniques, which would provide valuable insights to the community.\n3. Section 4 on scaling training compute across different axes reveals an interesting conclusion that configurations that are less efficient under limited compute may achieve better results given a larger compute.", "weaknesses": "1. In Sec 4, the large scale experiment on model scale makes a comparison between an 8B dense model and a large 17Bx16 MoE model. It would be better to include model scale experiment on comparing model sizes among dense models,", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XX3jgp0ImV", "reviewer": "Reviewer_Cv6x", "summary": "In this paper, the authors provide a comprehensive study of the scaling laws for large-scale distributed RL training. Based on the data collected as a result of this study, they propose and analyze a sigmoidal model, ScaleRL, that reasonably interpolates the training data and, crucially, closely extrapolates to additional points to predict the expected performance given a larger training budget. The authors consider the impact of several design choices when implementing RL, including the loss function, loss aggregation, and fp32 precision for logits. An extensive ablation study illustrates the interplay between several design methods and differentiates between substantial versus marginal impact on the peak performance of the training model. The authors provide several plots that illustrate the impact of a large number of design decisions on the training dynamics and use this data as a means to justify the design decisions that define the ScaleRL recipe to yield the best result.", "questions": "- Out of curiosity, all the graphs start at 1K total GPU hours. Before that threshold, are the graphs uninformative or too similar to provide any interesting information?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- This is one of the most extensive large-scale RL studies I've had the pleasure to read. In particular, the ablation study illustrated in Figure 4 really highlights the trade-offs the authors considered during the course of this study.\n- Given the emphasis and impact of RL as a key component of modern LLM pre-training this study will be particularly useful for researchers attempting to refine foundational models.\n- ScaleRL gives a clear baseline recipe for researchers to use when beginning the RL training process and sheds additional light on how to scale RL and the relationship between several tradeoffs.\n- This study provides a useful analogy to the heavily studied scaling laws for pre-training.\n- I appreciate the fundamental view that a better understanding of the interplay between existing RL training hyperparameters may be more insightful than proposing yet another loss function or base RL algorithm.\n- Comparison with", "weaknesses": "- Because the focus of the paper revolves around the comprehensive study of the different training parameters to consider for RL training it does not propose any novel methods, other than the sigmoidal model produced as a result of the collected data.\n- The RL experiments are mainly conducted on a single model with 8B parameters and the model may not fit as closely as the structure and number of parameters are changed. However, this is offset somewhat by the experiments conducted on the larger 17B Llama-4 model and illustrated in Figure 5(b).\n- It is still quite difficult to accurately interpret the leave-one-out experiments because of the complex interplay that may exist between several design decisions acting jointly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b5TR9vM2Vo", "reviewer": "Reviewer_JrWQ", "summary": "This paper studies how RL post‑training for LLMs scales with compute and proposes a practical recipe, ScaleRL, that is claimed to scale predictably and stably to very large budgets. The central methodological contribution is to fit a sigmoidal compute‑performance curve for pass rate (reward) vs. compute,. Using this framework, the paper ablates many design choices, assembles ScaleRL (PipelineRL with off‑policyness, CISPO loss with truncated IS, prompt‑level loss averaging, batch‑level advantage normalization, FP32 logits, zero‑variance prompt filtering, and \"No‑Positive‑Resampling\" curriculum), and claims that early‑compute fits extrapolate well to much larger runs. A major  result ofthe paper is that an 8B model trained for 100k GPU‑hours follows the curve predicted from the first 50k GPU‑hours, and that ScaleRL attains a higher fitted asymptote than several contemporary recipes. The paper further explores scaling along axes such as context length, batch size, and model scale (including a 17B×16 MoE), and reports downstream gains on AIME‑25.", "questions": "Can you report the effective tokens generated/consumed (actor and learner sides) for each method so we can decouple throughput from sample efficiency?\n\nFig. 3c shows a large asymptote jump from 0.52 to 0.61 with FP32 logits. Can you disentangle numerical‑mismatch reduction from any implicit regularization this might induce (e.g., via more accurate IS ratios)? Ablating only generator or only trainer in FP32 would help.\n\nDid you compare sigmoid to Gompertz/Richards/saturating power‑law fits, and evaluate with AIC/BIC or hold‑out residuals?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I like that the paper reframes RL post‑training through a predictive, saturating scaling law for in‑distribution pass rate vs. compute, and leverages it to make early‑compute forecasts, which is an angle that is underexplored relative to pre‑training scaling laws. Also, the study spans ~400k GPU‑hours of ablations and 100k GPU‑hours long runs, which is unusually extensive for open academic RL studies in LLMs. Additionally, the predictive claim is backed by several \"fit‑then‑extend\" tests: e.g., fitting up to 8k hours and extrapolating to 16k for LOO runs, and fitting to 50k and matching 100k on the main 8B run. The main recipe of the paper, ScaleRL, seems to be a very stable and practical idea.", "weaknesses": "Limited task breadth and generalization evidence:\n\nMost scaling fits are on verifiable math prompts drawn from Polaris‑53K; downstream evaluation focuses largely on AIME‑25. with some math+code multi‑tasking in the appendix. This leaves open how well the sigmoid fits and the ScaleRL recipe transfer to diverse RL setups (e.g., tool‑use, planning, safety‑constrained tasks, preference‑model rewards). A broader suite (beyond math/code) would strengthen the generality claim.\n\nCompute as \"GPU‑hours\" conflates sample efficiency and throughput differences:\n\nThe x‑axis is GPU‑hours rather than effective tokens / updates / actor‑generated tokens. Because PipelineRL increases hardware utilization, improvements in \"compute efficiency\" may partially reflect systems throughput rather than intrinsic sample efficiency. Reporting both (hardware time and tokenized data processed) would make cross‑recipe comparisons more interpretable.\n\nMisc. Notes:\n\nI think that the Sigmoid choice is empirically motivated but not stress‑tested against alternatives. I see this as a necessary experiment.\n\nThe text sometimes describes “pass rate vs. log(compute)” but plots label compute in GPU‑hours on a log scale.\n\nSome scope of the comparisons is missing. For example, value‑augmented baselines (e.g., VAPO/VC‑PPO‑style methods cited in App. A.1) are not included in the main scaling comparisons", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r9gszSNHjT", "reviewer": "Reviewer_abjx", "summary": "The paper proposed a methodological breakthrough. The first systematic study of RL scaling laws. This paper aims to address a core pain point in the field of RL for LLMs: the lack of a scientific framework capable of predicting how RL performance scales with the investment of computational resources. Focus on the problem of unpredictability, research bottelneck on large-scale experiments and algorithm decision makeing difficulty. It show the system investigation of the RL post-training for LLMs. It introduces the concept of scaling laws from the pre-training domain into RL. Acknowledging that RL performance is bounded. The framework provides a new, quantitative language for evaluating and comparing different RL methods. ScaleRL itself is not a algorithmic invention. Instead, through rigorous empirical study, it combines the most effective existing components from the community. The authors propose a two-pronged solution, establish a predictive computation-performance fitting framework based on a sigmodial curve to model the relationship between reward and computation. Futuremore, propose a best-practice recipe scaleRL integrates a series of technical components.", "questions": "None.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Significant computational cost savings: The framework enables researchers to evaluate the potential of different RL methods using a small computational budget\n2. Reduced risk in large-scale training: Using ScaleRL recipe, researcher can better predict the LLM training performance before large training.\n3. The provided open-source training recipe on 100,000 GPU hours, which can achieves high performance ceiling, is valuable for community.", "weaknesses": "1. Lack of generalization research. The authors also mention it in paper. It hard to predict the generalization performance. Also the experiments focus mainly on math, it is hard to predict whether the findings can be generalized on more difficult, long horizon task, with different training data mixtures, under different architectures. \n2. It doesn’t proposed the new RL algorithm, but a comprehensive large scale experiments on current existing  most effective algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)