Title: AREAL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: distributed system
Abstract: Reinforcement learning (RL) has become a trending paradigm for training large language models (LLMs), particularly for reasoning tasks. Effective RL for LLMs requires massive parallelization and poses an urgent need for efficient training systems. Most existing large-scale RL systems for LLMs are synchronous by alternating generation and training in a batch setting, where the rollouts in each training batch are generated by the same (or latest) model. This stabilizes RL training but suffers from severe system-level inefficiency. Generation must wait until the longest output in the batch is completed before model update, resulting in GPU underutilization. We present AReaL, a fully asynchronous RL  system that completely decouples generation from training. Rollout workers in AReaL continuously generate new outputs without waiting, while training workers update the model whenever a batch of data is collected. AReaL also incorporates a collection of system-level optimizations, leading to substantially higher GPU utilization. To stabilize RL training, AReaL balances the workload of rollout and training workers to control data staleness, and adopts a staleness-enhanced PPO variant to better handle outdated training samples. Extensive experiments on math and code reasoning benchmarks show that AReaL achieves up to 2.77x training speedup compared to synchronous systems with the same number of GPUs and matched or even improved final performance. The code of AReaL is available at https://github.com/inclusionAI/AReaL/.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present AReaL, a fully asynchronous system design enabling efficient RL for reasoning models. This system uses a “streaming”-like approach to LRM sampling that accounts for the higher & more heterogeneous inference workloads that come with reasoning. The authors stress test the asynchronous tolerance of standard PPO under their system and design a modified, more tolerant version of the learning objective accordingly. Experiments are conducted on the competitive math and programming settings.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The paper is well-written. The figures are relatively clean and easy-to-follow.\n- The motivation for the method is compelling and the results are strong, though somewhat limited in scope. A new, fast, and open-sourced framework for RL is a valuable contribution to the community.\n\n---\n\n### Weaknesses\n\n**Limited scope of experimentation**\n\nExperiments with AReaL are limited to (1) PPO and (2) Qwen models. \n\n(1) While PPO is perhaps the most standard evaluation choice, variants of PPO like GRPO and alternative algorithms like RLOO are also popular choices for training reasoning models in the community. It would be valuable to see (a) what the differences in asynchronous tolerance and (b) the AReaL tradeoffs between speedup vs stability & performance gain look like across algorithms with different variants of advantage bootstrapping.\n\n(2) Many works have reported that reasoning-style RL on Qwen models behaves very differently to models from other families [1, 2]. Given this, I would be interested to see at least the experiments in Figure 5 (a) and (b) extended to another model family. For example, it is non-obvious to me that the degree of stability to off-policy data may not also be a somewhat model-specific property.\n\n\n[1] Shao, R., Li, S. S., Xin, R., Geng, S., Wang, Y., Oh, S., ... & Zettlemoyer, L. (2025). Spurious Rewards: Rethinking Training Signals in RLVR. arXiv preprint arXiv:2506.10947.\n\n[2] Gandhi, K., Chakravarthy, A., Singh, A., Lile, N., & Goodman, N. D. (2025). Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01307.\n\nQUESTIONS:\n**Suggestions**\n\n1. Extending the experiments in figures 5(a) and (b) to a few other extremely popular RL algorithm variants (e.g. GRPO & RLOO), as well as at least one model series other than Qwen would improve the experimental soundness of the work.\n\n2. Nit: Many of the figures in the paper use different fonts, font size, and color schemes. While this is a minor detail, it is distracting for a reader.\n\n\n**Questions**\n\n> However, all these systems still follow a batched generation setting, where all the samples 50 within a training batch are from the same model version.\n\nHow exactly does AReaL completely avoid this? From what I understood, there is still micro-batching at the device level during generation -- could you please expound upon this point?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTraining large AI models for complex reasoning tasks is often slow because the system has to constantly switch between generating text and then training on that text, leaving powerful GPUs idle. This paper introduces AREAL, an approach that gets around this bottleneck by letting the generation and training processes run completely separately and at the same time. While this creates a challenge of the model learning from slightly \"stale\" data, AREAL incorporates a smart version of PPO to effectively handle these outdated samples. Ultimately, this asynchronous system provides a significant speed-up in training time, all while achieving comparable on reasoning benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- **Targets a real bottleneck** – proposes a fully asynchronous actor-learner pipeline that keeps GPUs busy by overlapping generation, reward computation and training.  \n- **Extensive empirical study** – evaluates 1.5 B, 7 B, 14 B and 32 B-parameter models on four math datasets (AIME 24/25, AMC 23, MATH 500), one code dataset (LiveCodeBench), scales to 512 GPUs, and includes ablations on staleness, batching and interruptible generation.  \n- **Clear efficiency gains** – achieves 2.6-2.8 × higher training throughput than synchronous baselines while approximately maintaining performance.\n\n### Weaknesses\n- **Narrow coding coverage** – Table 1 reports results only on LiveCodeBench; broader code tasks remain untested.  \n- **Heavy hardware demand** – experiments rely on clusters of many GPUs, with no guidance for small-scale academic setups (< 16 GPUs) where the staleness/throughput trade-off may differ.\n\nQUESTIONS:\nThis approach is clearly effective at a large scale. Could you discuss on how practical it would be to do research on these type of methods for smaller-scale experiments (4-8 GPUs)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces AREAL, a fully asynchronous reinforcement learning (RL) system designed to improve the training efficiency of large reasoning models (LRMs), such as language models used for math and code reasoning tasks. Unlike existing synchronous RL frameworks that alternate between generation and training phases and suffer from GPU underutilization, AREAL decouples these two phases entirely. Generation and training run in parallel across dedicated GPU clusters, enabled by interruptible rollouts and a staleness-aware design.\nTo address the challenges introduced by data staleness and inconsistent model versions in an asynchronous setting, the authors propose algorithmic innovations including a decoupled PPO objective and bounded staleness control, ensuring training stability even when using partially outdated rollouts. The system further incorporates engineering optimizations like dynamic micro-batch allocation and parallelized reward computation.\nExtensive experiments on math and coding benchmarks with Qwen models up to 32B parameters demonstrate that AREAL achieves up to 2.8× speedup over state-of-the-art synchronous RL systems while matching or surpassing performance in terms of task accuracy. The results highlight the promise of asynchronous RL systems for scalable and efficient LLM training.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- RL is an important and attractive field in the community. The paper could be very appealing to the community as it introduces a fully asynchronous RL system for LLM training, addressing key limitations in synchronous systems such as GPU underutilization and poor scalability. \n- AREAL is a well-rounded design in both senses of algorithm and system. The system design is well-motivated and the novels algorithms are also invented to solve the data staleness problem in a pretty natural and straightforward way.\n- The empirical results of AREAL is strong. It achieves 2x throughput improvement while maintaining or even improving the performance on math and coding datasets.\n\n**Weaknesses:**\n- Despite the strong system and algorithmic contributions, the paper does not release any code. Given the paper's claim to be a infra contribution by offering a new paradigm for RL training, the absence of code will limits reproducibility and may hinder broader adoption and feedback from the open-source community.\n- All experiments are conducted using only the Qwen family of models. This raises concerns about the generalizability of AREAL across different model architectures or training stacks (e.g., LLaMA, Gemma, Mistral).\n- AREAL currently only supports SGLang as the generation backend, which may restrict its usability in broader settings. Since many research and industry users rely on frameworks like vLLM for generation, the system’s current reliance on SGLang may act as a barrier to integration and adoption.\n- Some system-level design choices, such as the fixed ratio between inference and training, are based on heuristics without dynamic tuning or adaptation. A more adaptive approach could further improve throughput and robustness in varied deployment scenarios.\n\nQUESTIONS:\n- A minor point at line 252. Section 8 -> Appendix E.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces AReaL, an asynchronous reinforcement learning system designed to efficiently train large language models (LLMs) on reasoning tasks. Unlike traditional RL systems that operate synchronously, forcing model training to wait for all generated outputs to complete, AReaL fully decouples generation from training. Rollout workers continuously produce new outputs without blocking, while training workers update the model as soon as data is ready, significantly improving GPU utilization. AReaL is a fully asynchronous RL system for training large language models on reasoning tasks. It decouples generation and training to improve GPU utilization and efficiency. With staleness-aware PPO and system optimizations, AReaL achieves up to 2.57× speedup over synchronous baselines while maintaining or improving performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1. The introduction of a decoupled PPO objective and staleness-aware training effectively tackles theoretical issues such as policy mismatch and data staleness—common bottlenecks in asynchronous RL. The paper provides formal analysis and empirical ablation to validate their approach.\n\n2. AREAL is extensively evaluated on multiple benchmarks (AIME24, LiveCodeBench, MATH 500) using models up to 32B parameters. It consistently matches or exceeds prior methods while significantly reducing training time. The scalability results show near-linear throughput improvements up to 512 GPUs.\n\n\nWeakness:\n\nAs a reviewer not familiar with system design, I didn’t identify any clear weaknesses in this paper. Please refer to my question for further clarification.\n\nQUESTIONS:\n1.The strength of this paper is evident—AReaL significantly accelerates RL training. However, I noticed that many data points for VeRL are missing or marked with “-”. Since the comparison with VeRL is important (despite AReaL’s clear speed advantage), it would be helpful if the authors could include results across multiple random seeds. This would provide a clearer picture of how VeRL’s final performance compares to AReaL.\n\n2.Additionally, from the reported results, it seems that VeRL occasionally outperforms AReaL. Does this imply that training on the most recent data may yield better model performance than using stale samples, at least within the current implementation?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}