Title: The Art of Scaling Reinforcement Learning Compute for LLMs
Primary area: foundation or frontier models, including LLMs
Keywords: Scaling, LLMs, Reasoning

Abstract: While the training compute for reinforcement learning (RL) for LLMs is massively increasing, the field is still lacking predictive scaling methodologies for RL comparable to those established for pre-training. This gap is increasingly consequential given recent large scale RL efforts for reasoning-centric post-training. We present the first open, large-compute, systematic study of RL scaling for LLMs. We fit sigmoidal compute-performance curves for RL post-training and ablate a wide range of common design choices. We observe: (1) Not all recipes yield similar asymptotic  performance; (2) details such as loss aggregation, normalization, curriculum, and precision handling primarily modulate compute efficiency without materially shifting the asymptote; (3) Stable and scalable recipes exhibit predictive performance behavior as a function of compute, akin to established recipes in pre-training. Combining these insights, we propose a ``best-practice'' recipe, \textbf{ScaleRL}, and demonstrate its effectiveness by successfully scaling and predicting RL training performance on up to 100,000 GPU-hours, based on 400,000 GPU-hours of total experiments. Our study provides a principled foundation for predictive RL scaling laws in the LLM era, and a stable, scalable recipe.

TL;DR: We study compute scaling properties of RL methods on LLMs