AREAL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
Authors: Wei Fu
Jiaxuan Gao
Xujie Shen
Chen Zhu
Zhiyu Mei
Chuyi He
Shusheng Xu
Guo Wei
Jun Mei
WANG JIASHU
Tongkai Yang
Binhang Yuan
Yi Wu
Venue: NeurIPS 2025 poster ()
Keywords: distributed system
Primary area: infrastructure

Abstract: Reinforcement learning (RL) has become a trending paradigm for training large language models (LLMs), particularly for reasoning tasks. Effective RL for LLMs requires massive parallelization and poses an urgent need for efficient training systems. Most existing large-scale RL systems for LLMs are synchronous by alternating generation and training in a batch setting, where the rollouts in each training batch are generated by the same (or latest) model. This stabilizes RL training but suffers from severe system-level inefficiency. Generation must wait until the longest output in the batch is completed before model update, resulting in GPU underutilization. We present AReaL, a fully asynchronous RL  system that completely decouples generation from training. Rollout workers in AReaL continuously generate new outputs without waiting, while training workers update the model whenever a batch of data is collected. AReaL also incorporates a collection of system-level optimizations, leading to substantially higher GPU utilization. To stabilize RL training, AReaL balances the workload of rollout and training workers to control data staleness, and adopts a staleness-enhanced PPO variant to better handle outdated training samples. Extensive experiments on math and code reasoning benchmarks show that AReaL achieves up to 2.77x training speedup compared to synchronous systems with the same number of GPUs and matched or even improved final performance. The code of AReaL is available at https://github.com/inclusionAI/AReaL/.

AREAL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
Wei Fu 12 Jiaxuan Gao', Xujie Shen', Chen Zhu?, Zhiyu Mei Chuyi He?, Shusheng Xu Guo Wei Jun Mei Jiashu Wang", Tongkai Yang", Binhang Yuan', Yi Wu
III,, Tsinghua University, 2 Ant Group, 3 HKUST fusrth17@mail. com, jxwuyi@gmail.. com
Abstract
Reinforcement learning (RL) has become a trending paradigm for training large language models (LLMs), particularly for reasoning tasks. Effective RL for LLMs requires massive parallelization and poses urgent need for efficient training systems. Most existing large-scale RL systems for LLMs are synchronous by alternating generation and training in batch setting, where the rollouts in each training batch are generated by the same (or latest) model. This stabilizes RI. training but suffers from severe system-level inefficiency. Generation must wait until the longest output in the batch is completed before model update, resulting in GPU underutilization. We present AReaL, fully asyıchronous RL system that completely decouples generation from training. Rollout workers in AReaL continuously generate new outputs without waiting, while training workers update the model whenever batch of data is collected. AReaL also incorporates a collection of system-level optimizations, leading to substantially higher GPU utilization. To stabilize RL training. AReaL balances the workload of rollout and training workers to control data staleness, and adopts a staleness enhanced PPO variant to better handle outdated training samples. Extensive experiments on math and code reasoning benchmarks show that AReal. achieves up 2.77 x training speedup compared to synchronous systems with the same number of GPUs and matched or even improved final performance. The code of AREAL available at bttts://githhr cwwm feom/incluse AnclusionAAAARRal.
staleness-enhancee
1 Introduction
Reinforcement learning (RL) has emerged as a new scaling paradigm for enhancing the capabilities of large language models (LLMs) by enabling thinking abilities (52). Given a prompt, RL allows LLM to generate thinking tokens before outputting a final answer, enabling test-fime scaling (7) These thinking LLMs are named Large Reasoning Models (LRMs) and have been shown have particularly strong capabilities on challenging reasoning problems, such as math 5 20
coding
[5
logic puzzles
2 34
and agentic tasks 2 57]
Effective RL training often requires massive parallelization to derive a large batch rollouts for sufficient exploration, which is the key to obtaining optimal model performance. For example, popular RL algorithms, such as PPO 42 and GRPO [3], often require an effective training batch
of thousands of outputs (60] 61 53 Moreover, LRM can generate tens of thousands of thinking tokens for each input prompt (6) further posing an urgent need for an efficient training system to run RL training large scale.
"This work was supported by Ant Group Research Intern Program
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

However, developing an efficient large-scale RL system is challenging. An RL system needs to frequently switch between LLM generation and training, which can introduce significant system overhead without careful optimizations. For LRMs, the output length of the training model varies significantly for different prompts throughout the RI process, which results in an ever-changing workload for both generation and training. This characteristic often triggers idle time in high- performance hardware, leading to a waste of computation. Furthermore, classical large-scale RL algorithms like PPO or GRPO typically require on-policy training data, i.e., samples generated by the latest model, to ensure the best model performance, which poses additional system challenges.
Consequently, most existing large-scale RL systems are designed in a fully synchronous manner [27 [1 44 by strictly alternating between LLM generation and training, which ensures that the LLM is always trained on the latest outputs for the best practical performance. In such a synchronous design, the generation step must wait until the finish of the longest output within a batch. Due to the varying output lengths for LRMs, a synchronous RL system suffers from severe training inefficiency. Very recently, there have also been attempts to explore parallel generation and training 30 24
These works use outputs generated from a previous model version to update the current model. For the best performance, the model version used for rollout generation is limited to only one or two steps older. However, all these systems still follow a batched generation setting, where all the samples within a training batch are from the same model version. Accordingly, the issue of system inefficiency during the generation phase still remains unaddressed.
To fundamentally resolve the issues in system design, we develop AREAL, a fully Asynchronous RL training system for LRMs that completely decouples generation from training without hurting the final performance. AREAL runs L.M generation in a streaming manner, where each rollout worker continuously generates new outputs without waiting, leading to high GPU utilization. Meanwhile, the trainer workers in AREAL run parallel model updates whenever a training batch is obtained from the rollout workers. Once the model is updated, we synchronize the model weights in each rollout worker. such an asynchronous design, each training batch of A AREAL may contain samples generated by different model versions. Therefore, AREAL incorporates modified objective of the PPO algorithm, which can leverage samples generated from much older model versions without any performance drop. AREAL also conducts data filtering process to ensure the staleness of training sample is well controlled. In addition, AREAL also introduces several system-level optimizations, including interruptible rollout workers, dynamic batching for variable-length outputs, and parallel reward service, which further improve the overall training throughput.
We evaluate AREAL on challenging mathematical reasoning and code generation tasks using models up to 32B parameters. Compared to state-of-the-ar synchronous systems, AREAL achieves up to 2.57x higher training throughput and linear scaling efficiency up to 512 GPUs. Crucially, this acceleration even comes with improved solution accuracy on these tasks, illustrating that RREAL delivers significant efficiency gains without sacrificing (and indeed enhancing) model performance.
sectionRelated Work
RL for LLMs Reinforcement learning (RL) has emerged as the predominant paradigm for enhancing
the reasoning capabilities of Large Language Models (LLMs) [1 32). Existing RL approaches
typically focus on tasks with well-defined reward functions, including mathematical reasoning (9).
coding 14 15), scientific problem solving [39 36). and tool use [57]. During training, models learn to reason by progressively extending the length of chain-of-thought trajectories (52 6 Recent open-source initiatives have demonstrated significant success in improving model capabilities through smaller distilled models [24 25 Our work builds upon this research direction, distinguishing itself
from preference- bassed RLHF 33 and zero-shot reasoning approaches (60] 61 [2
that attempt to
acquire reasoning skills from preitrai models without task-specific fine-tuning.
combined with cor- responding algorithmic innovations 7 [6]. has achieved remarkable success in game applica- tions [1 Although similar asynchronous approaches have been explored for LLM training, they typically focus on short-context settings (30
Asynchronous RL The decoupled asynchronous RL architecture 21
40) (e.g.. RLHF) or one/two-step generation-training overlap 24 48). Our work extends these studies and provides a more flexible trade-off between staleness and training speed, as we will show in Section[] In contrast to concurrent work [64] that maximizes system-level efficiency, we adopt an algorithm-system co-design approach that provides both an expressive system and a practical algorithm implementation. Our interruptible generation technique conceptually similar to partial rollout [17 in synchronous RL systems. Instead of

---

setting fixed length budget, AREAL dynamically interrupts generation while maintaining consistent training batch sizes through buffering, thus preserving the stability of PPO. Compared with prior methods 40 30 our algorithmic innovation in the asynchronous setting can endure higher data staleness and remains compatible with interruptible generation.
LLM Training and Inference Our work focuses on dense transformer models [50] The RL training primarily consists of generation (inference) and training phases. Generation involves auto- regressive decoding. which requires efficient KV cache management [63 18 and optimized decod- ing kernels [58]. Training requires careful orchestration of data, tensor, and pipeline parallelism
strategies [88 46 62). While conventional synchronous systems execute generation and training sequentially on the same hardware resources, they require different optimal parallclization strategies. Recent work has proposed context switching [19] [7] or weight resharding 45] (7] techniques to address this mismatch. AREAL advances beyond synchronous RI. systems by decoupling generation and training, completely eliminating resharding overhead from the critical training path.
2 Background
2.1 Preliminaries about RL Training
RL Formulation and PPO We formulate our problem within the Markov Decision Process (MDP) framework (37], defined by the tuple AS, P: 7; H). Here, S represents the state space, A the action space. P the transition model, r S x A + R the reward function, the discount factor, and the horizon. The LRM implements parameterized policy Th S + A where each action a, E A corresponds to a text token from the vocabulary. The state St E S consists of question 81 q followed by previously generated response tokens (a1,.., at-1), with deterministic transitions Si41 concat(s;, 0z). Given question distribution D, we optimize the objective:
H Σ
((0) -EEgDD, am
7 pe-'r(se, at)
(I)
Following common practice 6 [25 we use a rule-based reward function that only provides non-zero feedback on the final action, indicating answer correctness, and set 1. We optimize this objective using Proximal Policy Optimization (PPO)
42
JPPo (f) = E Daa.Daaamaaaaaaa..
Σ
min
a.), (u, 1- c, +,1 A(s,, ai))
where ue (0)
(asll denotes the importance ratio and Å(st, ar) represents the estimated 33 we divide the global batch into advantage 41 Following standard practices in RL
xa(aels))
minibatches for sequential parameter updates
Distributed Systems for LRM Training Our work focuses on enhancing reasoning capabilities for LRMs after Supervised Fine- TuninT (SFT), distinct from approaches that incentivize reasoning in pre-trained base models [6] LRMs after SFT produce long reasoning sequences (c.g., 32K tokens) and usually require large global batch sizes (e.g., 128 prompts with 16 responses each) for stable RL training 66 25 24 60 61). In synchronous RL systems, two phases are iteratively exccuted: generation (rollout) and training. The generation phase uses the latest model parameters to produce multiple reasoning traces for each query in the training batch. The training phase then updates the model parameters based on the generated trajectories. These phases execute iteratively the same GPUs.
2.2 Motivation for Asynchronous RL System
We identify two essential limitations in synchronous RL systems:
Inference devices are underutilized. As shown in Figure (left), generation must wait for the longest sequence to complete before training can begin. This leads to non-uniform decoding length across GPUs, which underutilizes GPU compute resources.
This differs from gradient accumulation. which performs single update across minibatches.
3

---

Trahing
Generatio
Load Weigtt
1
GPU
1-8
9-16
17/24
GPU
GPU
1-8
9-16
17-24
dPU4
GPU 4
Figure 1: Execution timeline a synchronous (left) and a one-step overlap (right) RL system showing underutilized inference devices.
Reward Service
Generation
Keward
Traiing
Intoruptible
Trainer Worker
Rolloul Worker
GPU Job
Rollout Controller
Agprepate poate Bitton
Rep Butfer
Trainer Worker
Intemupt Signal
Interupibbee
Full Batch
(Update Weights]
Trainer Worker
Service
Prompt
Trajecooy
Interuatiió
Trainor Worker
ServerLoad
Figure 2: The AREAL architecture featuring asynchronous generation and training components.
Scalability is poor in synchronous RL systems. Synchronous systems distribute generation across all devices, reducing the Per-GPU decoding batch size. This pushes the decoding process into memory-10-boune regime
28 where additional devices fail to improve throughput.
System Architecture
The limitations identified in Section 2.2 motivate our design of system that fully decouples genera- tion and training across separate GPU clusters. This system should be hardware-efficient scalable, and equipped with the flexibility for customized RL workflow. We implement these principles in AREAL, an asynchronous RL system specifically designed for efficient large-scale L.RM training.
3.1
System Overview
Figure presents the architecture and data flow of AREAL. The system comprises 4 components:
Interruptible Rollout Worker handles two types of requests: (1) The generate request generates responses given prompts. (2) The podate_weighte request interrupts all ongoing generations and loads parameters of new versions. Upon the interruption, the rollout workers discard KV caches computed by old weights, and re-compute them using the new weights. Afterwards, the rollout workers continue to decode the unfinished sequences until the next interruption or termination. We emphasize that such interruptions and in in-flight weight updates would result in trajectories composed of segments produced by different model versions. This introduces a novel algorithmic challenge. which will be addressed in Section! 4
Interruptible Generation
Training
Generation
Load Weight
GPU2
KV Cache Recompute
Intermppted Sequence
GPU3
Time
3/c
Batch
Batch
Batch
Figure 3: Illustration of generation management in AREAL. Vertical lines show the ready time for the next step training. Blue crosses show the interrupted requests when new parameters arrive.

---

Reward Service evaluates the accuracy of the responses generated by the model. For example, in the coding task, this service extracts the code and executes unit tests verify its accuracy.
Trainer Workers continuously sample from the replay buffer, accumulating data until reaching the configured training batch size. They then perform PPO updates and store the resulting parameters in distributed storage. To ensure data freshness, data from the replay buffer is used only once.
serves as critical bridge between the rollout workers, reward service, and the model workers. During the training process, reads data from the dataset and invokes the rollout worker's generate request. The received response is then sent to the reward service to obtain the reward. The trajectory, along with the reward, is stored in the replay buffer, waiting to be trained by the model worker. After the model worker updates the parameters, the controller calls the rollout worker's s update_weight. We illustrate the generation and training management in Figure 3 This asynchronous pipeline ensures continuous full utilization of both generation and training resources.
Rollout Controller
3.2 Algorithmic Challenges
While the asynchronous system design offers significant acceleration through improved device utilization. it introduces several technical challenges that require algorithmic considerations.
Data Staleness Due to the asynchronous nature AREAL,, each training batch contains data from multiple prior policy versions. Prior works asynchronous RL training systems have demonstrated that such staleness can degrade learning performance in both RLHF (30] and game environments 2 Data staleness leads to a distribution gap between the training data and the latest model. In asyn- chronous RL training for LRMs, this issue could be even more severe for long trajectories due to extended decoding time.
Inconsistent Policy Versions As discussed in Section 3.1 the generated trajectories may involve segments produced by different policy versions. This inconsistency fundamentally violates the formulation of standard PPO in Eq. 2 that assumes all actions are generated by a single policy Tol-
In the following section, we detail our technical innovations for overcoming these challenges while preserving the efficiency advantages of an asynchronous system.
Addressing the Algorithmic Challenges in AREAL
4.1
Staleness-Aware Training
To avoid the performance drop due to training on data with extremely high staleness, we introduce a hyperparameter 7) representing the maximum permitted staleness each training batch for staleness- aware training. In particular, when = 0, our system degenerates to synchronous RL with all training samples generated by the current policy. We implement the staleness control in our system by dynamically controlling the throughput of the generation requests sent to the generation servers. Given the current policy version i, the total number of generated trajectories N,, and the training batch size B for each training step, enforce the following formula whenever submitting new generation requests:
I(N, - 1)
We also prioritize older trajectories from the data buffer to form a training batch. our system implementation, the rollout controller tracks both the generated samples N, and policy version i from the parameter server. rejects new generation requests that may violate the staleness constraint.
Note that this rate-limiting protocol is a simple yet effective design choice in practice. However, when 1 is too small, the generation throughput can slowed down when some extremely long trajectories are being generated. Therefore, we empirically suggest adopting a large saleness-contro parameter n for the best system throughput. This system-wide practice also motivates us apply an enhanced algorithm that can make effective use of more stale data for RL training.
4.2 Decoupled PPO Objective
We apply decoupled PPO objective 110 that disentangles the behavior policy and the proximal policy. The behavior policy Tbelav represents the policy used for sampling trajectories and the
5

---

proximal policy Tprex is proximal policy serving recent target to regularize the update of IT. By applying importance sampling on the sampled trajectories, we derive a decoupled PPO objective suitable for asynchronous RI. training:
Importaance Ratio
[()) B. Thebba
min( Tbehay
A
prox
clip(
1 1-c,1++)A.
T bebaa
T Tprx
Importance Ratio
Trust Region Center
(4)
H
BarrD ae ag
Σ
Arrox min prox clip (0), (1,
(5)
Tbehav
where uy 1epoox
[s()) is the importance ratio with respect to the proximal policy. We omit Tprox (ar (s))
the state-action terms for conciseness.
The main difference between the asynchronous PPO objective in Equation S and the standard one in Equation 2 lies in the proximal policy xprox for regularizing the model update. In asynchronous PPO training, using the behavior policy as the proximal policy will pull the latest policy тę towards the old-version and low-quality policies, thus slowing down model improvements. By employing recent policy as the proximal policy, model updates happen within the trust region around the high-quality proximal policy Trrxx. thus stabilizing training.
The decoupled PPO objective in Equation 5 provides a natural benefit: relaxes the requirement that all data within one training batch should be generated with single policy. This property crucial for maintaining algorithmic correctness when combining interruptible generation with policy updates. We claim that the inconsistent policy versions across trajectory maintain equivalence a single behavior policy bebaav (See Section for the proof.)
Proposition 1. For any sequence (g, (q.,,,...,, , an) generated by policies (mo...., T8+4) where ma4 produces tokens (at,)..., Ut; 11), where 1 = to < .... < li+1 = H, there exists a behavior policy Tbehay such that the interrupted generation is equivalent sampling entirely from Tbehay.
Practical Remark While Hilton al. [T0 maintains an exponential moving average of parameters for Trrox. this approach is prohibitively expensive for LRMs. Consequently, we simply use the parameters before each model update step as Tprox- Equation 5lis implemented by recomputing token probabilities upon the arrival of the global batch in each training step.
5 Implementation
We implement AREAL using Python and PyTorch [35] built upon the ReaLHF 27 framework. Our
system combines SGLang [63] v0.4.6 for generation serving with Megatron-Core 46 v0.11.0 as
the training backend, managed by SLURM (59 for resource scheduling. To maximize throughput for both generation and training phases, we implement several key system-level optimizations that address critical bottlenecks in the pipeline.
AREAL decouples GPU computation from CPU operations, including rule-based reward computation (such as string matching for math problems or unit test exccution for code) and TCP-based data transfer. By executing these operations in separate threads and pipelining the workflow, we overlap reward computation and data transfer with subsequent generation requests. We use asyncio coroutines to concurrently run multiple requests in the rollout worker avoid mutual blocking waits.
To handle training with variable-length sequences, we employ a padding-free sequence packing strategy coupled with dynamic allocation algorithm. The algorithm balances token distribution across micro-batches under fixed memory constraints (see Algorithm[1] [ This approach maximizes passes.
GPU memory utilization while minimizing the number of required

---

Experiments
Our evaluation comprises three components: (1) comprehensive comparisons against state-of of-the- art open-source frameworks across model sizes, (2) wrrong-scaling analysis with varying compute resources, and (3) ablation studies validating our design choices.
6.1 Experiment Setup
We evaluate AREAL on challenging math and coding tasks. We employ the distilled Qwen2 model series [54] (55 from DeepSeek-R1 [6] as base models (i.e., R1-Distilled- spanning from 1.5B to 32B parameters. For each task-model combination,
train for fixed number of PPO updates and evaluate the final checkpoint. Our evaluation of mathematical tasks follows the Qwen evaluation
protocol [56 13). while coding models are assessed on LiveCodeBench (8/1/24-2/1/25) [4] using
the official protocol. Unless otherwise specified, we set the maximum staleness 7j = 4 for coding
and 17 8 for math, and adopt the training configurations used in Section 6.2 with additional hyperparameters detailed in Appendix B
We conduct experiments on an H800 GPU cluster comprising 64 nodes, each equipped with 8 GPUs. The cluster features NVLink for intra-node connectivity and RoCE with 3.2Tbps bandwidth for inder-node communication. To ensure rapid convergence, we allocate minimum of 16 nodes as baseline pod configuration for complete experiments. We scale the number nodes proportionally with model size, ultimately utilizing 48 nodes for training our largest 32B parameter model. This scaling strategy enables us to run experiments of varying sizes in parallel while maintaining efficient resource utilization.
For AREAL, we maintain a fixed ratio between inference and training devices, allocating three- quarters of the devices for inference. This configuration was selected over an equal 50-50 partition based on our early experiments, where the 75-25 partition demonstrated higher training throughput. Although we adopt this ratio as a heuristic configuration, we emphasize that the optimal partition may vary across different settings and could potentially benefit from dynamic adjustment during training, as discussed in Section 7
6.2 End-to-End Comparison
We establish two state-of-the baselines using synchronous RL systems: DeepScaleR (25 for mathematical reasoning with 1.5B model, and DeepCoder [24] for code generation with 14B model, both trained using verl [45). For larger 7B and 32B models where comparable baselines are unavailable, we performed controlled experiments by training from scratch using synchronous variant of AREAL. After training, we evaluate on the challenging AIME24 benchmark for math models and the LiveCodeBench [14] benchmark for coding models. Evaluation results additional benchmarks are presented in Appendix
Our main results are shown in Table I Since the code for obtaining previous SOTA models can be out-of-date, we measure the throughput and estimate the training hours using the latest verl code for fair comparison. ARFAL consistently matches or exceeds baseline performance while achieving significant speedups without performance degradation. In particular, the end-to-end training time can be reduced by x compared with synchronous systems.
6.3 Scalability
We compare the scalability of AREAL with verl [45), the statc-of-the-arr synchronous RL system, across different model sizes and context lengths. We select the minimum number of GPUs when verl does not encounter the OOM issue for 7B models and 32k context length, then we proportionally adjust the number of GPUs according to the model size. We measure the effective throughput for training, defined as the rate of consuming generated tokens during PPO updates, after proper warmup steps. Figure 4 presents the results for context lengths of 16k and 32k. Here, context length refers to the sum of prompt length and generated length, with the maximum prompt length capped at 1k.
Across all settings, AREAL demonstrates an approximate linear scaling trend with increased device count, while the synchronous system typically fails to scale effectively. AREAL'S throughput surpasses the baseline in most settings, and could achieve at most 2.5x speedup. We note that for
7

---

Table End-to-End Performance Comparison. We evaluate the AIME24 benchmark for math and LiveCodeBench (8/1/24-2/1/25) for coding. We limit the maximum generation length to 32K tokens and sample responses per question, reporting the average pass@) accuracy. * represents the best known reproducible results obtained via RI, as cited from DeepScaler [25] and DeepCoder [24
respectively. AReal. achieves comparable performance with 2x fewer training hours.
Model
AIME24 f
# Nodes
PPO Steps
Training Hours 4
1.5B basemodel
29.3
w/ VeRL.
43.1*
16
250
33.6
w/ Ayea.AReal.
42.0
16
250
41.0
w/ AReaL (ours)
42.2
16
250
14.8
7B basemodel
54.3
w/ VeRL
24
250
52.1
w/ Syec.AReaL
63.0
24
250
57.7
w/ AReaL (ours)
63.1
24
250
25.4
Model
LiveCodeBench
# Nodes
PPO Steps
Training Hours |
14B basemodel
53.4
w/ VeRL.
57.9*
32
80
44.4
w/ SRea.AReal.
56.7
32
80
48.8
w/ AReal. (ours)
58.1
32
80
21.9
32B basemodel
57.4
w/ VeRL.
48
60
46.4
w/ Sync.AReaL
61.2
48
60
51.1
w/ AReaL (ours)
61.0
48
60
31.1
smaller context lengths, the advantage AREAL can be smaller because the generation throughput cannot match the pace of training throughput. Although many sequences are generated, they are not effectively consumed by the training process. Additionally, AREAL is more robust with longer generation lengths due asynchronous and interruptible generation. The generation of long responses can be fully hidden in the critical path, so extending generation length does not drastically affect the effective training throughput of AREAL.
6.4 Algorithm Ablations
We conduct ablation studies validate algorithmic innovations in Section 4 by training 1.5B LRM on math tasks. We follow the basic experiment setting of DeepScaleR and then gradually
Model= 1.5B ctx= 16384
Model=7B, ctx=16384
Model=32B, cbx=16384
235k
175k
37k
155k
112k
29k
verl
Throughput (token/s)
102k
Bnear
71k
23k
67k
46k
18k
44k
29k
15k
29k
19k
12k
64
128
256
64
128
512
256
512
M.del=1.5B,
ctx= 32768
Model=7B, ctx=32768
Model=32B, ctx=32768
241k
157k
37k
155k
103k
32k
100k
68k
28k
64k
44k
24k
41k
29k
21k
27k
19k
16k:
32
64
128
256
64
128
256
512
256
512
Number GPUs
Figure The strong scaling trend. Dotted lines indicate ideal linear scaling. verl consistently encounters OOM with 32k context length and the 32B model so the data points are missing.

---

naive PPO
+Decoupled Objective
Maximum Stalemnss
4+
10-
-40
100 300
s5o
Training Stop
Thaining Stop
Effective Training Throughout (k tokensts)
(a) Learning curves with naive PPO.
(b) Learning curves with
(c) Effective training throughput.
Figure 5: Ablation studies the decoupled PPO objective and staleness control with a 1.5B model on math reasoning tasks. Both algorithmic choices are essential. With moderate staleness value and the decoupled objective, training progress can be accelerated by over 2x while maintaining final evaluation performance.
Table 2: Evaluation scores when varying data staleness, comparing performance with and without the decoupled objective. Numbers within +1 of the oracle score underlined.
AIME24
AIME25
AMC23
MATH 500
MtaeStale.
W/o
With
W/o
With
W/o
With
W/o
With
0 (Oracle)
42.0
32.9
84.4
89.2
41.8
42.1
30.7
31.9
83.3
85.2
89.9
89.8
40.0
41.8
32.1
32.5
82.3
84.3
89.6
89.6
23.3
42.2
23.1
32.0
58.5
85.1
66.9
89.5
35.7
41.0
27.8
31.1
81.2
82.9
87.8
89.2
16
35.8
38.7
26.2
32.5
78.4
83.2
87.4
89.1
0
34.0
36.9
26.9
29.9
79.4
81.0
87.1
88.1
increase the 1) value for ablation purposes. Specifically, we vary the maximum allowed staleness 7 and compare configurations with and without the decoupled PPO objective. Figures 5a and 5b show the learning curves after 250 training steps. Table presents the corresponding final evaluation performances across multiple mathematical reasoning benchmarks. We follow the common practice of PPO and perform multiple mini-batch updates within each training step. We emphasize that constrains the training batch staleness regarding training steps.
Figure 5a demonstrates that naive PPO fails to match the performance of the synchronous RL oracle (1.c., the performance when 17 0). Even slight staleness significantly degrade final performance due to the improper clipping center and policy changes during interruptible generation. Furthermore, increasing data staleness consistently degrades learning performance, aligning with observations from prior work in other domains [] However, as shown by comparing Figure 5b) and Figure 5al the decoupled PPO objective substantially improves training stability when handling stale data, consistent with findings from [10 in game domains. In addition, we observe that even with the decoupled objective, unbounded staleness (maximum staleness 20) still results in inferior performance compared to the zero-staleness oracle. When properly constrained, moderate staleness (c.g., 7) has minimal impact on final performance while significantly accelerating training through the asynchronous pipeline, as demonstrated in Figure 5c and Table our approach of combining controlled staleness with the decoupled PPO objective for efficient asynchronous RL training.
2 These results validate
6.5 System Ablations
Dynamic Microbatch Allocation We investigate the effectiveness of dynamic batching by compar- ing PPO training throughput against standard micro-batching strategy. The standard micro-batching strategy can result in multiple long sequences being assigned the same micro-batch, thus usually requiring sufficiently large number of micro-batches prevent out-of memory errors. In our

---

Throughpul (TFLO) PPLPP per GPU)
Dynamic vs. Normal Batching
Average Throughput (tokens/s)
Interruptible Generation
300K
Dynamic
interruptible
500
454.7
Normal
231k
non-interruptible
100
404.4
387.7
200K
300
303.1
283.0
130k
111k
200
100K
100
1B node)
7B nodes) Model Size
32B nodes)
1.5B
7B
Model Size
(a) Ablation study of dynamic micro-batch allocation.
(b) Ablation study of interruptible generation.
Figure Ablation studies on system optimizations.
experimental setup, we configured 32 hiiro-bbtcches for the standard setting and established a token budget of 32,768 per micro-batch for the dynamic batching approach. As demonstrated in Figure 66a dynamic batching yields an average of 30% throughput improvements across various model sizes.
Interruptible Generation We ablate interruptible generation and present the resulting generation throughput in Figure [b) Without interruptible generation, the controller must wait for the longest response. particular, interruptible generation leads a 12% and 17% throughput increase for 1.5B and 7B models respectively on nodes, which validates our architectural design choice.
7 Conclusion
This paper introduces AREAL, fully asynchronous system designed for efficient large-scale re- inforcement learning (RL) training. The AREAL architecture provides both the flexibility and expressiveness required for implementing asynchronous algorithms. Building upon this foundation, we contribute several algorithmic innovations, including staleness-aware training and decoupled PPO objectíve, which enable efficient and stable PPO training in asynchronous environments. Our experimental results demonstrate AREAL's superior hardware efficiency, sample efficiency, and scalability compared to existing synchronous RL systems. This work provides a starting point for reliably scaling RL training. We hope that it can enable future advances in large-scale Al systems that push the boundaries of machine intelligence further.
References
[1] B. R. Bartoldson, S. Venkatraman, J. Diffenderfer, M. Jain, T. Ben-Nun, S. Lee, M. Kim, J. Obando-Ceron Ceron, Bengio, and B. Kailkhura. Trajectory balance with asynchrony: Decoupling exploration and learning for fast, scalable LLM post- traiinga CoRR, abs/2503. 18929, 2025. doi: 00.48550/AXXI.25555 18929. URL /tttts:///ot...... org/10 48550/ arriv. 2503. 18929
[2] C. Berner, G. Brockman, B. Chan, V. Cheung, Debiak, C. Dennison. D. Farhi, Q. Fis- cher, S. Hashme, C. Hesse, R. Józefowicz, S. Gray, C. Olsson, J. Pachocki, M. Petrov, H. P. de Oliveira Pinto, J. Raiman, T. Salimans, J. Schlatter, J. Schneider, S. Sidor, I. Sutskever, J. Tangg, F. Wolski, and S. Zhang. Dota 2 with large scale deep reinforcement learning. CoRR, abs/1912.06680, 2019. URL http:://t:aaa http:v//axivvvv//m www.rxvv.orrgg. org abs 0abb//992.06600
[3] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, Brockman, A. Ray, Puri, G. Krueger, M. Petrov, Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, Tillet, F. P. Such, Cummings, M. Plappert, Chantzis, Barnes, A. Herbert- Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, Achiam, V. Misra, Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish,
10

---

I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021. Utt.ttps://arai..rrg http:/awwwii.r..raaa../o abtp//////221/.. 03374
[4] Z. Chen, A. May, R. Svirschevski, Y. Huang, M. Ryabinin, Z. Jia, and B. Chen. Se- quoia: Scalable and robust speculative decoding. In A. Globerson, L. Mackey, D. Bel- grave, A. Fan. U. Paquet. J. Tomozak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 129531-12956; Curran Associates, Inc., 2024. URL
wwwwwee..w..pppaeeeeeeee wwppaper.pppppeeee -f wiwes//apaaieee httpppppppep2//////p pdf
[5] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems. CoRR, abs/21 10. 14168, 2021. httpsttp: :ttpps://ane :/tari//aaii//// wtwaaaiwwwgaaaaaaaaaaaaaaa abs///1/ 14168
[6]
DeepSeck-AI, D. Guo, D. Yang. H. Zhang, Song. R. Zhang, R. Xu, Q. Zhu, S. Ma, Wang. X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, 7. Gou, Z. Shao, 7. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang. H. Bao, H. Xu, H. Wang.
H. Ding, H. Xin, H. Gao, H. Qu, H. Li, J. Guo, J. Li, Wang. J. Chen, J. Yuan, J. Qiu, J. Li,
L. Cai, J. Ni, J. Liang, J. Chen, K. Dong, K. Hu, K. Gao, K. Guan, K. Huang, K. Yu, L. Wang, L. Zhang, L. Zhao, L. Wang. L. Zhang, L. Xu, L. Xia, M. Zhang. M. Zhang, M. Tang, M. Li,
M. Wang, M. Li, N. Tian, P. Huang, P. Zhang, Q. Wang. Q. Chen, Q. Du, R. Ge. R. Zhang, R. Pan, R. Wang, R. J. Chen, R. L. Jin, R. Chen, S. Lu, S. Zhou, S. Chen, S. Ye, S. Wang, S. Yu, S. Zhou, S. Pan, and S. S. Li. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcementt learning. CoRR, abs/2501.1294., 2025. doi: 1044550/ARRXI.2501..229 URL hRtphtts:://dot org/ 10. 085.44555///// arxvvv 2501. 12948
[7] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, S. Legg, and K. Kavukcuoglu. IMPALA: scalable distributed deep-rl with impor- tance weighted actor-learner architectures. In G. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmäs- san, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1406-1415, PMLR, 2018. URL http::/ espeholt18a. html
mlr. press/v80/
[8] L. Espeholt, R. Marinier, P. Stanczyk, K. Wang, and M. Michalski. SEED RL: scalable and efficient deep-rl with accelerated central inference. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
URL httptt:://peerreeeeeee
[9] D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the MATH dataset. In J. Vanschoren and Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. cct.paap/e//22///ahh
URL httpp://aattattt---bbaaaaaaaaaaa proceeeeerrsoooooeoeooo.....
heurips.... httpg/geeeerrrppppp/////////1110
be92a30ccoob77666222111013333333 http:///6000c1.11.aa...aaaa..a.. Abstract- round2. 2htold
[10] J. Hilton, K. Cobbe, and J. Schulman. Batch size- iivarinare for policy optimization. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA,, USA, November 28 Decem-
ber 9, 2022, 2022. URL http::/ nips cc/ Aee---bsstrrccc Cowferennce..oooooooooooooooooce html
cttpps.ccppppprr_ppp pwppr_apppiiiee fileef//ieee//eee httpe/ffiesssppeeeeeeeeea http:/pppp///// http//22222hhhaaaa
[11] J. Hu, x. Wu, W. Wang, Xianyu, D. Zhang, and Y. Cao. Openrlhf: An easy-to-use, scalable and righ-pefformacc RLHF framework. CoRR, abs/2405.11143, 2024. doi: 10.48550/ARXIV. 2405.1114 1143.
hattps:://ae org/10 48.0..855/////// arxiv. 2405 11143
[12] J. Hu, Y. Zhang, Q. Han, D. Jiang, X. Zhang, and H. Shum. Open-reasoner-zerro An open
source approach to scaling up reinforcement learning on the base model. CoRR, abs/2503..4990. 2025. doi: 1448550/AARXXV225002222290 URL https:/t::
10. 48850//rr.8555///.... 2503
24290
[13) B. Hui, J. Yang, 7.. Cui, Yang. D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Dang, A. Yang, R. Men, F. Huang, X. Ren, X. Ren, J. Zhou, and J. Lin. Qwen2.5-coder technical report.
11

---

CoRR, abs/2409.12186, 2024. doi: 10.48500ARXXI.240092222 URL httph:://oli org/
10. 1..44850/arriix 2409. 12186
[14]
N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar- K. Sen, and I. Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.ntt, 2025. URI. htttt::////p:eteeeeret
net/forruniid=chfJJJC333
[15] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Ovennevie net/ wow.ooiddnn
2024.
[16] S. Kapturowski, G. Ostrovski, J. Quan, R. Munos, and W. Dabney, Recurrent experience replay in distributed reinforcement learning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. [KLLpttps://pperrvviee wwt./neeeeedddrrl//eoeo 11yTTjAqv
[17] KimiTeam, A. Du, B. Gao, B. Xing, C. Jiang. C. Chen, C. Li, C. Xiao, C. Du, C. Liao, C. Tang, C. Wang. D. Zhang, E. Yuan, E. Lu, F. Tang, F. Sung, G. Wei, G. Lai, H. Guo, H. Zhu, H. Ding, H. Hu, H. Yang, H. Zhang, H. Yao, H. Zhao, H. Lu, H. Li, H. Yu, H. Gao, H. Zheng, H. Yuan, J. Chen, J. Guo, J. Su, J. Wang. Zhao, J. Zhang, J. Liu, J. Yan, J. Wu, L. Shi, L. Ye, L. Yu, M. Dong, N. Zhang, N. Ma. Q. Pan, Q. Gong, S. Liu, S. Ma, S. Wei, S. Cao. S. Huang,
T. Jiang, W. Gao, W. Xiong, W. He, W. Huang, W. Wu, W. He, X. Wei, X. Jia, X. Wu, X. Xu,
X. Zu, X. Zhou, X. Pan, Y. Charles, Y. Li, Y. Hu, Y. Liu, Y. Chen, Y. Wang. Y. Liu, Y. Qin,
Y. Liu, Y. Yang.
Bao, Y. Du, Wu, Y. Wang, Z. Zhou, Z. Wang, Z. Li, Z. Zhu, Z. Zhang, Z. Wang, Z. Yang. Z. Huang, Z. Huang, Z. Xu, and Z. Yang. Kimi k1.5: Scaling reinforcement learning with llms. CORR, abs/2501. 11..501...2999 12599, 2025. doi: 10.45550/ARXIV2550...15
URL
https://doi www.lo..ggg..o.. 48550. arxiv www.iii..11...... 12599
[18] W. Kwon, 2. Li, S. Zhuang, Y. Sheng, L.. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with pagedattention. In J. Flinn, M. I. Seltzer, P. Druschel, A. Kaufmann, and J. Mace, editors, Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023, pages 61 1-626. ACM, 2023. doi: 10.1145/5000066361131666 URL https /145/3600006666 3613666
[19] K. Lei, Y. Jin, M. Zhai, K. Huang, H. Ye, and J. Zhai. PUZZLE: efficiently aligning large language models through weight context switch. In S. Bagchi and Y. Zhang, editors, Proceedings of the 2024 USENIX Annual Technical Conference, USENIX ATC 2024, Santa Clara, CA. USA, July 10-12. 2024, pages 127-140. USENIX Association, 2024. URL https //www. usenix. org/ htt:/oonfeeeeeeee wwwrreenee/ac//////e/////// wwtc24//eeseetatiiooooo
[20] J. LI. E. Beeching, L. Tunstall, B. Lipkin. R. Soletskyi, S. C. Huang. K. Rasul, L. Yu, A. Jiang, 7. Shen, Z. Qin, B. Dong, Zhou, Y. Fleureau, G. Lample, and S. Polu. Numina- math. hhttts://hugging:e
thub. co AI www.umiiaaaaaatoaoaaaa (https: thub com/ datasst.adattettaataaa pdf)
project- pesocttnmmiiaaaaeeee aimaaammoppppa prtooo-roorsss--pee httpgg/esspppeeee////o//////rppp 2024.
[21] E. Liang, R. Liaw, R. Nishihara, P. Moritz, R. Fox, K. Goldberg, J. Gonzalez, M. I. Jor- dan, and Stoica. Rllib: Abstractions for distributed reinforcement learning. In J. G. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmssmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 3059-3068. PMLR, 2018. URL http://prroceedings. mlr www...pressvv//// matass/vv0/1aaag11.. www.nnaannnnn.oo/ htmll
[22] B. Y. Lin, R. L. Bras, K. Richardson, A. Sabharwal, R. Poovendran, P. Clark, and Y. Choi. Zebralogic: On the scaling limits of Iims for logical reasoning, 2025. URL https:
org/abs/ 01100)
[23] X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang. S. Zhang, X. Deng, A. Zeng, Z. Du, C. Zhang, S. Shen, Zhang, Y. Su, H. Sun, M. Huang, Y. Dong, and J. Tang Agentbench: Evaluating llms as agents. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. httsstts:://
12

---

[24] M. Luo, S. Tan, R. Huang, A. Patel, A. Ariyak, Q. Wu, X. Shi, R. Xin, C. Cai, M. Weber, et al. Deepcoder: A fully open-source 14b coder at o3-mini level, 2025.
[25] M. Luo, S. Tan, J. Wong, X. Shi, W. Y. Tang, M. Roongta, C. Cai, J. Luo,
L. E. Li, R. A. Popa, and I. Stoica. Deepscaler: Surpassing ol ol-preview with
a 1.5b model by scaling rl.
https///prett--raaioo Syaalling RL-19961922:44400bbbbbbaa
notion. site/
DeepScaleR- SeaeieeesSrrassinn--aae Previee with-a-1-553
2025. Notion Blog.
[26] Mei, W. Fu, J. Gao, G. Wang, H. Zhang, and Y. Wu. SRI:: scaling distributed reinforcement
learning to over ten thousand cores. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openrevieez mett/forumidddlajnliii
[27] Z. Mei, W. Fu, K. Li, G. Wang. H. Zhang, and Y. Wu. Realhf: Optimized RLHF training for large language models through parameter reallocation. CoRR, abs/2406. 140522006.14088, 2024. doi: 10.48550/ARXIV.24006 140888 http:t::::///// //doi..:////.// org/10. www.g//...85.///// 48550/arXiv..../////// 2406. 14088
[28] X. Miao, G. Oliaro, Z. Zhang, X. Cheng, Z. Wang, Z. Zhang, R. Y. Y. Wong, A. Zhu, L. Yang. X. Shi, C. Shi, Z. Chen, D. Arfeen, R. Abhyankar, and Z. Jia. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference Architectural Support for Programming Languages and Operating Systems, Volume 3, ASPLOS '24, page 932-949, New York. NY, USA, 2024. Association for Computing Machinery. ISBN 9798400703867.
doi: 10. 10.11456620066.3611135.
/tt.:tttt////o.... org/ 10. 1w1111161666666... /05166666666666633533333 3651335
[29] N. Muennighoff, Z. Yang, W. Shi, I.. Li, I.. Fei-Fei, H. Hajishirzi, I.. Zettlemooyr, P. Liang,
E. J. Candès, and T. Hashimoto. sl: Simple test-time scaling. CoRR, abs/2501. 2025. doi:
10. 00445500ARXXI.250... 19393.. Jtttts:::://doi. org/ 4.0.0088550/araaa arrosv/arxvvv... 2501. 19393
[30] M. Noukhovitch, S. Huang, Xhonneux, A. Hosseini, R. Agarwal, and A. C. Courville. Asynchromoous RLHF: faster and more efficient off-policy RL for language models. CoRR,
abs/2410.18252. 2024. doi: 10.48550/ARXXV.2410.0 18252. URL /tttp:://alt.org//
48550/arXiv 2410. 18252
[31] OpenAl, Sep 2024.
URL
http:://penai.com/ineeee
learning- re-sen with with-1lms/
[32] OpenAI, Apr 2025.
URL
https://openaa.commineee
introducing o3- and 04-mini/
[33] L. Ouyangg J. Wu, X. Jiang. D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022. New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL httphhttpp/ppppps..ip.... cc/ happrr/pprr.... walles/paper/200 htmll
[34] J. Pan, J. Zhang, X. Wang, L. Yuan, H. Peng, and A. Suhr. Tinyzero. https:/ggttub..om//iiii Pan/TinyZero, 2025. Accessed: 2025-01-24.
[35] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, I.. Antiga, A. Desmaison, A. Köpf, E. Z. Yang. Z. Vito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An im- perative style, high-performance deep learning library. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. B. Fox, and R. Garnett, editors, Advances in Neu- Neural Information Pro- cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 8024 8035. 2019. URL https: https::/prrocedinngg wwwpiiiiinnnn..uriin wwwwips.cc/ppaeeeepe wwtpc//pppppe/1/////es.s. b (ttp:/88eeef9922220 wtt.///992222220220772/77 hwwp://77774-Aattattttoo
ral Information Processing Systems 32: Annual Conference
[36] L. Phan, A. Gatti, Z. Han, N. Li, Hu, H. Zhang, S. Shi, M. Choi, A. Agrawal, A. Chopra, A. Khoja, R. Kim, J. Hausenloy, O. Zhang. M. Mazeika, D. Anderson, T. Nguyen, M. Mahmood, Feng, S. Y. Feng, H. Zhao, M. Yu, V. Gangal, C. Zou, Z. Wang, P. Wang, P. Kumar,
O. Pokutnyi, R. Gerbicz, S. Popov, J. Levin, M. Kazakov, J. Schmitt, G. Galgon, A. Sanchez,
13

---

Y. Lee, W. Yeadon, S. Sauers, M. Roth. C. Agu. S. Riis, F. Giska, S. Utpala, Z. Giboney, G. M. Goshu, J. of Are Xavier. S. Crowson, M. M. Naiya, N. Burns, L. Finke, Z. Cheng. H. Park, F. Fournier-Facio, J. Wydallis, M. Nandor, A. Singh, T. Gehrunger, J. Cai, B. McCarty, D. Duclosel, J. Nam, J. Zampese, R. G. Hoerr, A. Bacho, G. A. Loume, A. Galal, H. Cao, A. C. Garretson, D. Sileo, Q. Ren, D. Cojoc, P. Arkhipov, U. Qazi, L. Li, S. Motwani, C. S. de Witt, E. Taylor, J. Veith, E. Singer, T. D. Hartman, P. Rissone, J. Jin, J. W. L. Shi, C. G. Willcocks, J. Robinson, A. Mikov, A. Prabhu, L. Tang, X. Alapont, J. L. Uro, K. Zhou, E, de Oliveira Santos, A. P. Maksimov, E. Vendrow, K. Zenitani, J. Guillod, Y. Li, J. Vendrow, V. Kuchkin, and N. Ze- An. Humanity's last exam. CoRR, abs/2501. 14249944...449, 2025. doi: 10.48550/ARXIV25500.1 URL hRtppss:://doi. org/ 10. 1..00.8550/rr111. 2501 14249
[37] M. L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley Series in Probability and Statistics. Wiley, 1994. ISBN 978-0-47161977-2. doi: 10.1002/ 9780470316887,
[ttLlttts:///odoo /ttp::////ooooo//////// 10. 10.110029978888 000000//7770440318888777777
[38] S. Rajbhandari. J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlania, Georgia, USA, November 9-19, 2020, page 20. IEEE/ACM, 2020. doi: N..109/SC441405.2220.oo
https::t: or0 1109/
SC41405. 2020..020000000 00024
[39] D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman. GPQA: graduate-level pooggeeproofffooo q&a benchmark. CoRR, abs/2311 2023. doi: 2AB/://AAXXX2311..1100
URL https: : //doi. org/10.....0..o 400.00.8855//88 arriv. 2311. 12022
[40] N. L.. Roux, M. G. Bellemare, J. Lebensold, A. Bergeron, J. Greaves, A. Fréchette, C. Pelletier, E. Thiff,, S. Tóth, and S. Work. Tapered off off-policy REINFORCE: stable and efficient reinforcement learning for 1lms. CoRR, abs/2503. 14286, 2025, doi: 10.48550/ARXIV. 1428.1 URL http:: otg/10 48850/ 2503. 14286
[41] J. Schulman, P. Moritz, S. Levine, M. I. Jordan, and P. Abbeel. High-dimension control using generalized advantage estimation. In Y. Bengio and Y. LeCun, editors, 4th International Conference Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
continuous
/tap:::///xxv....// www.vvorg/aaaa wttgg/bs///t 1506
02438
[42] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CoRR, abs//177.0033 (00477, 2017. URL http: http://xxiv..oo/ hetg/abs/1700 06347
[43] Z. Shao, Wang, Q. Zhu, Xu, J. Song, M. Zhang, K. Li, Y. Wu, and D. Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300. 2024. doi: 0..45500/ARXIV.44022 03300
TtRL.ttpss:///
org/10..//// 4884048555555 2402
[44] G. Shen, Z. Wang. O. Delalleau, J. Zeng, Y. Dong, D. Egert, S. Sun, J. J. Zhang, S. Jain, A. Taghibakhshi, M. S. Ausin, A. Aithal, and O. Kuchaiev. Nemo-aligner: Scalable toolkit for efficient model alignment. CoRR, abs/2405.01 .01144181.. 2024. doi: 10448550/ARXIV.24550011 URL httl:ttts://dol. org/ 10. 48550/ arXiv. 2405 2405.01481
[45] G. Sheng, C. Zhang, Z. Ye. X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu. Hybridflow: A flexible and efficient RLHF framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys 2025, Rotterdam, The Netherlands, 30 March 2025 3 April 2025, pages 1279-1297, ACM, 2025. doi: 10. 10.1145/368903 1.3666075. URL https: //doi 20g/10.10 1145/ 01445/5669331... 01189031.3690075
[46] M. Shoeybi. M. Patwary, R. Puri, P. LeGresley. J. Casper, and B. Catanzaro. Megatron-Im: Train- ing multi-billion parameter language models using model parallelism. CoRR, 0iss/999,08055. 2019. URL http::/ttt:tttt
org/ aws//b/9. 08053
[47] C. V. Snell, J. Lee, K. Xu, and A. Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025.
WtRllhttts////pnneee wttp:/ewiiiinneeeeeeeeeeeeee nwt/ffruummm 4d4FWAw2t222.
[48] Team, S. Jaghouar, J. Mattern, J. M. Ong, J. Straube, M. Basra, A. Pazdera, K. Thaman, M. D. Ferrante, F. Gabriel, F. Obeid, K. Erdem, M. Keiblinger, and J. Hagemann. Intellect-2:
14

---

A reasoning model trained through globally decentralized reinforcement learning, 2025, URL
https://arxiv. httpp:::/aaxiv.org/aaaa./ abs/ wtwpa//www..rr//////....... 07291
[49] P. I. Team, S. Jaghouar, J. Mattern, J. M. Ong, J. Straube, M. Basra, A. Pazdera, K. Thaman, M. D. Ferrante, F. Gabriel, F. Obeid, K. Erdem, M. Keiblinger, and J. Hagemann. Intellect-2: A reasoning model trained through globally decentralized reinforcement learning, 2025. URL
https://arxiv orgg abs 2t00//555500029111
[50] A. Vaswani, N. Shazeer, Parmar, Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and Polo- sukhin. Attention is all you need. In I. Guyon, U. von I. Luxburg, S. Bengio, H. M. Wallach, R. Fer- gus, S. V. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA, pages 5998-6008, 2017. URU/attps://proceediiii Cc/ pc/paper//e /017 hwsh/ahhhaaaaaaa
neur neurips
[51]
O. Vinyals. I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, Danihelka, A. Huang. L. Sifre,
T. Cai, J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden, Y. Sulsky, J. Molloy, T. L. Paine. Ç. Gülgehre, Z. Wang. T. Pfaff, Wu, R. Ring,
D. Yogatama, D. Wünsch, K. McKinney, O. Smith, T. Schaul, T. P. Lillicrap, K. Kavukcuoglu, D. Hassabis, C. Apps, and Silver. Grandmaster level in starcraft II using multi-agent reinforcement learning. Nat., 575(7782)3350-3555 2019. doi: 011038/541586-01--1111 WRL/http:://dat http:p:://doi.or/1.... 1t.g////.//0 1038/ 00.000///41188 s41586 -11---19----11 1724-2
[52] Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou. Chain-of-thouugh prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Anmual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- Decem- ber 9, 2022. 2022. URL http::/ wwspaaerrsiiips..omm cc/ wwwprppppr.apeeeeeeeeeeeeeeee _f files htppp/appp//// /eaperr/022/// 2022 2022/hash/ 9d5609613524ecf4F watocfff11aff000 1fafff
Abstract--ttttttteee Conf html
[53] H. Xin, D. Guo, Z. Shao, Z. Ren, Q. Zhu, B. Liu, C. Ruan, W. Li, and X. Liang. Deepseek-prover Advancing theorem proving in ilms through large-scale synthetic data, CoRR, abs/2405.14333 2024. doi: 10.48550/RXXV..4005. 14333. Utthttts::/
org/ 10. 48550/arXiz 2405
14333
[54] A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Yang, J. Xu, J. Zhou,
J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. Wang, R. Peng, R. Men, R. Gao, R. Lin, S. Wang, S. Bai, S. Tan, T. Zhu, T. Li, T. Liu,
W. Ge, X. Deng, X. Zhou, X. Ren, X. Zhang, X. Wei, X. Ren, X. Liu, Y. Fan, Y. Yao,
Y. Zhang. Y. Wan, Y. Chu, Y. Liu, Z. Cui, Z. Zhang, Z. Guo, and Z. Fan. Qwen2 technical report. CoRR, abs/2407. 106771,..667, 2024. doi: 10.48550/ARXIV.24077. 10671. URL. https
//doi.org/ 1....48550/ar//.
2407.10671 10671
[55] A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei,
H. Lin, J. Yang, Tu, J. Zhang, J. Yang, J. Yang, Zhou, Lin, K. Dang, K. Lu, K. Bao, K. Yang. L. Yu, M. Li, M. Xue, P. Zhang. Q. Zhu, R. Men, R. Lin, T. Li, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu, Qwen2.5 technical report. CoRR, abs/2412.15115. 2024. doi: 10.4850/ARXXV.241..5511
URL
https://doi. /ttt::://doooor/10...
118500/aXii:
2412 15115
[56] A. Yang, B. Zhang, B. Hui, B. Gao, B. Yu, Li, D. Liu, J. Tu, J. Zhou, J. Lin, K. Lu, M. Xue, R. Lin, T. Liu, X. Ren, and Z. Zhang. Owen2.5-mate technical report: Toward mathematical expert model via self meel--improveeent. CORR, abs/2409. 12122, 2024. doi: 10.48550/ARXIV.2400.
12122. URL htt.::tts://aae/
org/ 10. 48550/ arxiv. 2409 12122
[57] S. Yao, N. Shinn, P. Razavi, and K. Narasimhan. b-bench: A benchmark for tool- age-tggent interaction in woald domains. CoRR, abs/2406. 2024. doi: 10.48550/ARXIV.24000
1 12045. https///::///o: http//di..oo/g//////// org/10. 2.0.44555/arXii1. 2406. 12045
[58] Z. Ye, L. Chen, R. Lai, W. Lin, Y. Zhang, Wang, T. Chen, B. Kasikci, V. Grover, A. Kr- ishnamurthy, and L. Ceze. Flashinfer: Efficient and customizable attention engine for LLM inference serving. CoRR, 0///250100100. 2025. doi: 10.48550/ARXXI225501.000
URL
https: :/dps: 2rg/10.co 48550/arXiv arxiv 25011.0000 01005
15

---

[59] A. B. Yoo, M. A. Jette, and M. Grondona. SLURM: simple linux utility for resource management. In G. Feitelson, Rudolph, and U. Schwiegelshohn, editors, Job Scheduling Strategies for Parallel Processing, 9th International Workshop, JSSPP 2003, Seatlle, WA, USA, June 24, 2003, Revised Pupers, volume 2862 Lecture Notes in Computer Science, pages 44 60. Springer, URLphttps: //wi. org/10. 10. 10070009669877
2003. doi: 10. 10.1007/10968987..
[60] Q. Yu, Z. Zhang. R. Zhu, Y. Yuan, X. Zuo. Y. Yue. T. Fan, G. Liu, L. Liu, X. Liu, H. Lin,
Z. Lin, B. Ma. G. Sheng. Y. Tong, C. Zhang, M. Zhang, W. Zhang, Zhu, J. Zhu, J. Chen,
J. Chen, C. Wang, H. Yu, W. Dai, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang,
L. Yan, M. Qiao, Y. Wu, and M. Wang. DAPO: an open-source LLM reinforcement learning system at scale. CoRR, abs/2503, 14476, 2025. doi: 10.48550/ARXIV25500.1 14476 URL Intps: //dts: org/10. 48550/ arXiv 2503. 14476
[61] Y. Yue, Y. Yuan, Q. Yu, X. Zuo, Zhu, W. Xu, J. Chen, C. Wang, T. Fan, Du, X. Wei, X. Yu,
G. Liu, J. Liu, L. Liu, H. Lin, Z. Lin, B. Ma, C. Zhang, M. Zhang, W. Zhang, H. Zhu, R. Zhang,
X. Liu, M. Wang, Wu, and L. Yan. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks, 2025. URL/htts::
/ttpps:/aarxin org/ /bs/ 05118
[62] Y. Zhao, A. Gu, R. Varma, L. Luo, C. Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer, A. Desmaison, C. Balioglu, P. Damania, B. Nguyen, G. Chauhan, Y. Hao, A. Mathews, and S. Li. Pytorch FSDP: experiences on scaling fully sharded data paral- URL
lel. Proc. VLDB Endow., (6(12):3384-3886)) 2023. doi: 10.
https://www wwwpbwwwwwwwwwwwwwwwwwww.......w www.orgg.o......... prggpvldb/vollnn www.oo/31111-----11
pdf
[63] L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, Cao, C. Kozyrakis, I. Stoica, J. Gonzalez, C. W. Barrett, and Y. Sheng. Sglang: Efficient execution of structured language model programs. In A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL http::/papprrr wwsppprrs.iips.oo wwwpppprppppcp.ccppcccc files/paper Conference html
2024/hash/72400 2224442228..131111066ffff
[64] Y. Zhong, Zhang, X. Song. Hu, Jin, B. Wu, N. Chen, Chen, Y. Zhou, C. Wan, H. Zhou, Y. Jiang, Y. Zhu, and D. Jiang. Streamrl: Scalable, heterogeneous, and elastic rl for ulms with disaggregated stream generation, 2025. URL
2b55//44..22....... 2504. 15930
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main scope is in Section 2 and the main contribution is in Section 4 The
abstract summarizes them.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Appendix E
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

Justification:
All assumptions have been illustrated in Proposition | and Section 2
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have discussed the model, dataset, and hyper-parameters we use Ap- pendix B and Section 6
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification: We have included code in the supplementary material. Datasets and models we used are all open-sourced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /pips.ccc/pulic//uuu guwde//ddee/ee/ Cote/ooodeeeieseeeeeeee hwtpbbmmsssioooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Appendix B
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We did include error bars for large-scale end-to-end experiments because they are expensive to run. We present results within single trial under the same fixed random seed across different settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
19

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
In Appendix B and Section 6
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: htte::://eerrreee httpp///pppppptttttcccccedeaaaaa
Answer: [Yes]
Justification: N/A
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification:
This paper aims to optimize training system, which has limited social
impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper uses datasets and models used by prior works.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The original sources all properly cited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
23

---

A
Reproducibility
The code of AREAL is available at wtttt:s://// base models in our experiments are taken from the open-source community (see Appendix B). used fixed random seed of 1 across all experiments.
httttts://gtthhii http://gtttb..... www.//nocdlssinnA//RRee./ Datascts and
B Implementation Details
B.1 PPO Details
We disable the critic model and the reference model in PPO. The advantage estimation parameter in GAE and the RL discount factor Y are fixed 1. The reward is at the final token if the answer is correct and -5 otherwise. We additionally adopt advantage normalization across the global batch stabilize the training. Other learning related hyperparameters and configurations can be found in Table 3
Table 3: Training configurations and hyperparameters.
Training Configuration
Batch size (number of prompts)
512
Random seed
PPO Parameters
PPO Minibatches
4
Clipping
0.2
Advantage normalization
True
Discount factor 7
1.0
GAE A
1.0
Optimizer Parameters
Optimizer
Adam
Learning rate
2.0 10
Weight decay
0.05
B1
0.9
B2
0.95
Adam c
1 x 10-5
Gradient norm clipping
1.0
Learning rate scheduler
constant
Warmup steps proportion
0.001
Precision Parameters
Parameter dtype
fp16
KV cache dtype
fp16
Gradient dtype
fp32
Optimizer state dtype
fp32
Generation Parameters
Answers per prompt
16
Temperature
1.0
Top-p
1.0
Top-k
-1
Max prompt length
1024
Min generation length
0
Max generation length
27648

---

B.2 Dataset Details
For the math task, we used the open-source data from DeepScaleR [25] For code training, we used the dataset released by DeepCoder [24]. All compared methods use the same dataset.
B.3 Dynamic Batching
Algorithm 1 Dynamic Batching
Require: Sequence lengths S {si, $2, ..,............................. ,p)), maximum micro-batch capacity C, minimum
number of micro-batte sicro-batches kmin
Ensure: Balanced partition of sequences into micro-batches with total length S C
1: Sort S in descending order
2: batches
3: for all S do
4: if' [batches] < kmin or no existing batch can fit then
5: Create new micro-batch containing sequence i batches.append
6:
7: else
8:
Find all batches that can accommodate 9: Select the micro-bated with fewest sequences
end end for
11:
return batches
The dynamic batching algorithm is shown in Algorithm
B.4 Baselines
In our experiments, we use the lastest version (main branch of verl repository, May 2025) of verl 45) to evaluate the training throughput in Figure 4 and the training hours in Table 1 For most of the results, we use SGLang [63] v0.4.6 as generation backend and pytorch FSDP [62] as training backend. In few cases where SGLang raises errors (experiments with 32B models 64 nodes), we use vLLM 18 v0.8.4 as a substitution.
Additional Results
C.1
Additional Evaluation Results
We evaluate the models trained with AReaL on more math and coding benchmarks, and list the results in Table 4 and Table 5 respectively.
Table 4: Results on math benchmarks.
Model
AIME24
AIME25
AMC23
MATH 500
1.5B basemodel
29.3
24.4
71.0
84.3
w/ Syne. AReaL
42.0
32.9
84.4
89.2
w/ AReal.
42.2
32.0
85.1
89.5
7B basemodel
54.3
41.7
89.5
92.8
Syne. AReaL
63.0
50.0
93.2
94.2
w/ AReaL
63.1
47.3
93.6
94.3
C.2 Generalization Across Model Architectures
We conducted additional experiments using the long-CoT model based on Ilama 3.1 8B. We matched the experimental configuration with the Qwen 7B math model from Table I and the results presented in Table
model, which is
25

---

Table 5: Results on coding benchmarks.
Model
LiveCodeBench v5
Codeforces
CodeContests
14B base model
53.4
1801/95.89
32.0
Syne. AReaL 14B
56.7
1845/96.44 4%
37.0
AReaL 14B (ours)
58.1
1840/96.3%
35.9
32B base model
57.4
1839/96.3%
34.3
Sync. AReaL 32B
61.2
1911/96.9%
36.3
ARcaL 32B (ours)
61.0
1889/96.7%
36.5
Table 6: Generalization results DeepSeck-Diittlled-LLlam across math benchmarks.
Model
AIME24
AMC23
MATH500
AIME25
DeepSeck-Distilld--llnm
50.4
84.2
89.1
23.3
AREAL Fine-Tumed 7 =4
58.4
92.3
92.2
42.6
AREAL Fine-Tuned 7 =8
57.2
91.5
91.9
41.6
The results demonstrate that AReaL generalizes effectively across different model families.
C.3 Staleness Staleness-Throughhpee Throughput
Trade-off with Small-Scale Academic Setups
We conducted series of experiments using the SaasSekk-Distiled--ORA context length and batch size 64 x 16 on GPUs, testing various staleness values. The following table shows the experimental results. We observe that our preliminary conclusions from the large-scale setting (Table generally align with findings using fewer GPUs.
model with 8k
Table 7: Staleness Staleness-throughhurr trade-off small-scale academic setup.
Model
AIME24
AIME25
AMC23
MATH500
Throughput
29.3
24.4
71.0
84.3
AREAL Tenne-Tune/// =0
31.7
26.1
78.9
86.7
27.
AREAL
Fine-Tuned =1
32.6
26.4
76.6
86.4
47.8k
AREAL Fine- Tune-TueedPPa -2
32.4
26.7
76.0
86.6
47.8k
A AREAL Fine-Tuned f) -4
34.1
28.1
75.5
86.9
49.0k
A AREAL Fine-Tuned -8
29.9
23.2
76.1
86.1
51.5k
AREAL Fine-Tuned !) =16 32.8
25.9
78.1
86.3
52.0k
C.4 Staleness- Throughs--hroughhutt Trade-off with Different RL Algorithms
We conducted additional experiments using the RLOO advantage bootstrapping method. We trained 1.5B models in local 8-GPU setting with 8k context length and various staleness values. The evaluation results in Aable that RLOO exhibits slightly better tolerance to asynchronous training compared to vanilla PPO.
Table 8: Staleness-thooughee trade-off using RLOO algorithm.
Model
AIME24
AIME25
AMC23
MATH500
Throughput
DeepSeek-Distileee 1w5n-1.5B
29.3
24.4
71.0
84.3
RLOO =0
32.4
29.2
79.2
87.3
27. Ik
RLOO =1
32.9
26.0
76.4
87.7
47.8k
RLOO -2
34.1
28.0
81.1
86.9
47.8k
RLOO =4
32.9
27.9
76.0
87.0
49.0k
RLOO 88
31.5
28.1
78.0
87.4
51.5k
RLOO 7) 16
32.7
27.7
78.4
87.4
52.0k
26

---

These results highlight an important direction for future research. AREAL modifies the PPO/GRPO algorithm because the importance sampling term naturally supports asynchronous off- off-policy training. Beyond PPO-based workflows, it would be valuable to investigate the asynchronous tolerance of REINFORCF-liK and other off-policy algorithms.
D Proof of Proposition
Proposition 1. For any sequence (g, .q..a..... aH) generated by policies (Te...., TT+k) where no+s produces tokens (at,)...,, at,.1), where 1 = to <... < tk+1 = H, there exists a behavior policy Tbehav such that the interrupted generation is equivalent sampling entirely from Tbehav-
Proof. For question g, let Sz(q) denote states encountered at step / by the sequence of policies. Since S (q) Sty (4) for # J, can construct:
(-+) arbitrary
if st ty+1 and E (g)
Tbehav (-|s)
otherwise
E Limitations and Future Work
Our work presents several limitations that suggest directions for future research. First, the ratio between inference and training devices could be further optimized for specific training setups. Additionally, this ratio might benefit from dynamic adjustment during training, particularly as context lengths typically increase when fine-tuning pre-trained base models. While focused evaluation on single-step mathematical and coding tasks, the AREAL architecture is not inherently limited to these domains. We leave the exploration of multi-turn interactions and agentic scenarios future work.
27


Reviews:
[{"review_id": "ST95aJmxGg", "reviewer": "Reviewer_u73n", "summary": "This paper introduces AReaL, an asynchronous reinforcement learning system designed to efficiently train large language models (LLMs) on reasoning tasks. Unlike traditional RL systems that operate synchronously, forcing model training to wait for all generated outputs to complete, AReaL fully decouples generation from training. Rollout workers continuously produce new outputs without blocking, while training workers update the model as soon as data is ready, significantly improving GPU utilization. AReaL is a fully asynchronous RL system for training large language models on reasoning tasks. It decouples generation and training to improve GPU utilization and efficiency. With staleness-aware PPO and system optimizations, AReaL achieves up to 2.57× speedup over synchronous baselines while maintaining or improving performance.", "questions": "1.The strength of this paper is evident—AReaL significantly accelerates RL training. However, I noticed that many data points for VeRL are missing or marked with “-”. Since the comparison with VeRL is important (despite AReaL’s clear speed advantage), it would be helpful if the authors could include results across multiple random seeds. This would provide a clearer picture of how VeRL’s final performance compares to AReaL.\n\n2.Additionally, from the reported results, it seems that VeRL occasionally outperforms AReaL. Does this imply that training on the most recent data may yield better model performance than using stale samples, at least within the current implementation?", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n1. The introduction of a decoupled PPO objective and staleness-aware training effectively tackles theoretical issues such as policy mismatch and data staleness—common bottlenecks in asynchronous RL. The paper provides formal analysis and empirical ablation to validate their approach.\n\n2. AREAL is extensively evaluated on multiple benchmarks (AIME24, LiveCodeBench, MATH 500) using models up to 32B parameters. It consistently matches or exceeds prior methods while significantly reducing training time. The scalability results show near-linear throughput improvements up to 512 GPUs.\n\n\nWeakness:\n\nAs a reviewer not familiar with system design, I didn’t identify any clear weaknesses in this paper. Please refer to my question for further clarification.", "was_revised": true, "final_justification": "I would like to keep my current positive rating of the paper, and encourage the authors to incorporate our discussions into the manuscript.", "rebuttal": "We sincerely thank the reviewer for their supportive comments\\! We hope the following responses address your concerns:  \n\n> I noticed that many data points for VeRL are missing or marked with \"-\" in Table 1  \n\nWe acknowledge that despite our best efforts to tune VeRL before paper submission, our VeRL reproduction did not achieve comparable results to AReaL in some cases. Consequently, in the draft, we included results from third-party forks of VeRL, such as DeepScaleR\\[1\\] and DeepCoder\\[2\\], which typically incorporate customized modifications to improve training stability.  \n\nCurrently, we are re-running experiments with the latest version of VeRL to address these missing data points. However, given the computational expense of these experiments (e.g., the 7B math model requires over 10,000 GPU hours to complete), we have not been able to produce valid results during the rebuttal period. We will make every effort to include these amended results in the camera-ready version.  \n\n\\[1\\] [https://huggingface.co/agentica-org/DeepScaleR-1.5B-Preview](https://huggingface.co/agentica-org/DeepScaleR-1.5B-Preview)  \n\n\\[2\\] [https://www.together.ai/blog/deepcoder](https://www.together.ai/blog/deepcoder)   \n\n> It would be helpful if the authors could include results across multiple random seeds. This would provide a clearer picture of how VeRL's final performance compares to AReaL.  \n\nWe completely agree that including multiple random seeds would strengthen the robustness of our results. In the current draft, AReaL uses a fixed random seed of 1 for all training experiments. The evaluation protocol samples 32 responses with different random seeds and reports the average accuracy, following common practices adopted in the community \\[3\\].  \n\nWe plan to re-run experiments with multiple random seeds to test AReaL's robustness once we have available computational resources. We commit to updating these results in the camera-ready version.  \n\n\\[3\\] https://github.com/QwenLM/Qwen2.5-Math  \n\n> VeRL occasionally outperforms AReaL. Does this imply that training on the most recent data may yield better model performance than using stale samples, at least within the current implementation?  \n\nWe agree with the reviewer about this observation. In general, training with more recent data does yield better performance. However, the algorithmic modifications in AReaL increase tolerance to staleness, enabling us to achieve comparable performance even with staler data.  \n\nWe note that the end-to-end comparison between VeRL and AReaL may not clearly showcase the effect of data staleness due to numerous implementation differences, such as the use of log-probabilities produced by the inference engine and different training/inference backends. Each of these factors can introduce numerical issues that influence final performance. Based on Table 1, we conclude that VeRL and AReaL demonstrate comparable algorithmic performance.  \n\nRegarding the effect of data staleness, Table 2 presents a more controlled ablation study. The results show that staleness levels of 1, 2, and 4 all achieve similar final performance to the synchronous oracle (staleness=0) across multiple evaluation benchmarks. These results demonstrate AReaL's increased tolerance to data staleness."}, {"review_id": "XF9NxMUg7F", "reviewer": "Reviewer_9Jfg", "summary": "This paper introduces AREAL, a fully asynchronous reinforcement learning (RL) system designed to improve the training efficiency of large reasoning models (LRMs), such as language models used for math and code reasoning tasks. Unlike existing synchronous RL frameworks that alternate between generation and training phases and suffer from GPU underutilization, AREAL decouples these two phases entirely. Generation and training run in parallel across dedicated GPU clusters, enabled by interruptible rollouts and a staleness-aware design.\nTo address the challenges introduced by data staleness and inconsistent model versions in an asynchronous setting, the authors propose algorithmic innovations including a decoupled PPO objective and bounded staleness control, ensuring training stability even when using partially outdated rollouts. The system further incorporates engineering optimizations like dynamic micro-batch allocation and parallelized reward computation.\nExtensive experiments on math and coding benchmarks with Qwen models up to 32B parameters demonstrate that AREAL achieves up to 2.8× speedup over state-of-the-art synchronous RL systems while matching or surpassing performance in terms of task accuracy. The results highlight the promise of asynchronous RL systems for scalable and efficient LLM training.", "questions": "- A minor point at line 252. Section 8 -> Appendix E.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- RL is an important and attractive field in the community. The paper could be very appealing to the community as it introduces a fully asynchronous RL system for LLM training, addressing key limitations in synchronous systems such as GPU underutilization and poor scalability. \n- AREAL is a well-rounded design in both senses of algorithm and system. The system design is well-motivated and the novels algorithms are also invented to solve the data staleness problem in a pretty natural and straightforward way.\n- The empirical results of AREAL is strong. It achieves 2x throughput improvement while maintaining or even improving the performance on math and coding datasets.\n\n**Weaknesses:**\n- Despite the strong system and algorithmic contributions, the paper does not release any code. Given the paper's claim to be a infra contribution by offering a new paradigm for RL training, the absence of code will limits reproducibility and may hinder broader adoption and feedback from the open-source community.\n- All experiments are conducted using only the Qwen family of models. This raises concerns about the generalizability of AREAL across different model architectures or training stacks (e.g., LLaMA, Gemma, Mistral).\n- AREAL currently only supports SGLang as the generation backend, which may restrict its usability in broader settings. Since many research and industry users rely on frameworks like vLLM for generation, the system’s current reliance on SGLang may act as a barrier to integration and adoption.\n- Some system-level design choices, such as the fixed ratio between inference and training, are based on heuristics without dynamic tuning or adaptation. A more adaptive approach could further improve throughput and robustness in varied deployment scenarios.", "was_revised": true, "final_justification": "My concerns are largely resolved during the discussion stage. \nThis is a technically solid paper and could be impactful to open-source communities. The authors are also committed to develop more features to make this work adaptable to prevalent frameworks such as vLLM.", "rebuttal": "Thank you for the detailed and insightful review\\! We would like to address your concerns with the following responses:  \n\n> The paper does not release any code.  \n\nWe have provided an anonymous link in Appendix A. We apologize that we cannot repost the link in this rebuttal response, as it is strictly prohibited by the rebuttal guidelines.  \n\n> Concerns about the generalizability of AReaL across different model architectures.  \n\nDuring the rebuttal period, we conducted additional experiments using the DeepSeek-Distilled-Llama-8B model, which is a long-CoT model based on Llama 3.1 8B. We matched the experimental configuration with the Qwen 7B math model from Table 1, and the results are presented in the following table:\n\n| model | aime24 | amc23 | math500 | aime25 |\n| :---- | :---- | :---- | :---- | :---- |\n| base model | 50.4 | 84.2 | 89.1 | 23.3 |\n| AReaL staleness=4 | 58.4 | 92.3 | 92.2 | 42.6 |\n| AReaL staleness=8 | 57.2 | 91.5 | 91.9 | 41.6 |\n\nThe results demonstrate that AReaL generalizes effectively across different model families.  \n\n> The system's current reliance on SGLang may act as a barrier to integration and adoption.  \n\nWe primarily incorporated and modified SGLang to implement the interruptible rollout workers, but AReaL is not restricted to specific inference and training backends. We commit to supporting additional backends in the future, including PyTorch FSDP for trainer and vLLM for rollout inference worker. This expansion is currently part of our ongoing internal development efforts and will be uploaded to an open-source repository in the future.  \n\n> Some system-level design choices, such as the fixed ratio between inference and training, are based on heuristics without dynamic tuning or adaptation. A more adaptive approach could further improve throughput and robustness in varied deployment scenarios.  \n\nThe reviewer's observation is absolutely correct\\! We currently use a heuristically tuned inference-to-training ratio designed to saturate training GPUs. This ratio could be further optimized (decreased) based on specific training configurations and would benefit significantly from dynamic adjustment during training, particularly when context lengths change during RL training.  \nWe will discuss these limitations in the appendix and consider these important ideas for future work. Thank you again for this insightful suggestion\\!"}, {"review_id": "hJfhMj43H8", "reviewer": "Reviewer_EQw4", "summary": "Training large AI models for complex reasoning tasks is often slow because the system has to constantly switch between generating text and then training on that text, leaving powerful GPUs idle. This paper introduces AREAL, an approach that gets around this bottleneck by letting the generation and training processes run completely separately and at the same time. While this creates a challenge of the model learning from slightly \"stale\" data, AREAL incorporates a smart version of PPO to effectively handle these outdated samples. Ultimately, this asynchronous system provides a significant speed-up in training time, all while achieving comparable on reasoning benchmarks.", "questions": "This approach is clearly effective at a large scale. Could you discuss on how practical it would be to do research on these type of methods for smaller-scale experiments (4-8 GPUs)?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- **Targets a real bottleneck** – proposes a fully asynchronous actor-learner pipeline that keeps GPUs busy by overlapping generation, reward computation and training.  \n- **Extensive empirical study** – evaluates 1.5 B, 7 B, 14 B and 32 B-parameter models on four math datasets (AIME 24/25, AMC 23, MATH 500), one code dataset (LiveCodeBench), scales to 512 GPUs, and includes ablations on staleness, batching and interruptible generation.  \n- **Clear efficiency gains** – achieves 2.6-2.8 × higher training throughput than synchronous baselines while approximately maintaining performance.\n\n### Weaknesses\n- **Narrow coding coverage** – Table 1 reports results only on LiveCodeBench; broader code tasks remain untested.  \n- **Heavy hardware demand** – experiments rely on clusters of many GPUs, with no guidance for small-scale academic setups (< 16 GPUs) where the staleness/throughput trade-off may differ.", "was_revised": true, "final_justification": "The authors have clarified my points. After reading other reviewer, i think the paper is ready for acceptance.", "rebuttal": "We appreciate the reviewer's detailed and constructive feedback\\! We would like to address your concerns one by one in the following responses:  \n\n> Table 1 reports results only on LiveCodeBench.  \n\nDuring the rebuttal period, we conduct additional evaluations in the coding domain and present the results below. We will include these results in the appendix of the camera-ready version.\n\n| Model | LiveCodeBench v5 | Codeforces | CodeContests |\n| :---- | :---- | :---- | :---- |\n| 14B base model | 53.4 | 1801/95.8% | 32.0 |\n| Sync. AReaL 14B | 56.7 | 1845/96.4% | 37.0 |\n| AReaL 14B (ours) | 58.1 | 1840/96.3% | 35.9 |\n| 32B base model | 57.4 | 1839/96.3% | 34.3 |\n| Sync. AReaL 32B | 61.2 | 1911/96.9% | 36.3 |\n| AReaL 32B (ours) | 61.0 | 1889/96.7% | 36.5 |\n\n> Experiments rely on clusters of many GPUs, with no guidance for small-scale academic setups (\\< 16 GPUs) where the staleness/throughput trade-off may differ.  \n\nThank you for this important observation\\! We conducted a series of experiments using the Qwen 1.5B distilled model with 8k context length and batch size 64×16 on 8 GPUs, testing various staleness values. The following table shows the experimental results. We observe that our preliminary conclusions from the large-scale setting (Table 2\\) generally align with findings using fewer GPUs.\n\n| model | AIME24 | AIME25 | AMC23 | MATH 500 | Throughput |\n| :---- | :---- | :---- | :---- | :---- | :---- | \n| 1.5B base | 29.3  | 24.4  | 71.0 | 84.3 | - |\n| staleness=0 | 31.7 | 26.1 | 78.9 | 86.7 | 27.1k |\n| staleness=1 | 32.6 | 26.4 | 76.6 | 86.4 | 47.8k |\n| staleness=2 | 32.4 | 26.7 | 76.0 | 86.6 | 47.8k |\n| staleness=4 | 34.1 | 28.1 | 75.5 | 86.9 | 49.0k |\n| staleness=8 | 29.9 | 23.2 | 76.1 | 86.1 | 51.5k |\n| staleness=16 | 32.8 | 25.9 | 78.1 | 86.3 | 52.0k |\n\nWe note that fine-tuning strong reasoning models with RL is typically resource-intensive \\[1\\], so training with 8 GPUs has to use smaller context lengths, and the final performance may not match that obtained from large-scale distributed training. We consider improving the efficiency of asynchronous RL training for smaller setups an important future direction.  \n\n\\[1\\] https://github.com/giterinhub/DeepScaleR-1.5B-Preview  \n\n> Could you discuss how practical it would be to conduct research on these types of methods for smaller-scale experiments (4-8 GPUs)?  \n\nAReaL provides benefits for long CoT and agentic settings regardless of experimental scale. When trajectory length variance is high, synchronous systems can easily encounter resource underutilization issues while waiting for the longest rollout to complete. AReaL eliminates this idle time through continuous batching and rollout interruption, thereby improving overall throughput (Figure 3).  \n\nResearchers may have concerns about the usability and customizability of asynchronous systems. However, the algorithmic implementation of AReaL does not differ significantly from synchronous counterparts. Users simply need to switch between synchronous and asynchronous rollout methods. The algorithm orchestration code can remain identical for both synchronous and asynchronous RL:  \n\n```  \ndataloader = StatefulDataloader(dataset)  \nfor step in range(max_steps):  \n    if async_training:  \n        batch = rollout.prepare_batch(dataloader, workflow=workflow)  \n    else:  \n        data = next(dataloader)  \n        batch = rollout.rollout_batch(data, workflow=workflow)  \n    # Prepare training inputs  \n    adv_batch = actor.compute_advantages_and_returns(batch)  \n    batch['advantages'] = adv_batch['advantages']  \n    # Execute PPO update  \n    stats = actor.ppo_update(batch)  \n```  \n\nThis design ensures that researchers can easily adopt AReaL in their existing workflows with minimal code modifications."}, {"review_id": "75wCTvlwkd", "reviewer": "Reviewer_ETFe", "summary": "The authors present AReaL, a fully asynchronous system design enabling efficient RL for reasoning models. This system uses a “streaming”-like approach to LRM sampling that accounts for the higher & more heterogeneous inference workloads that come with reasoning. The authors stress test the asynchronous tolerance of standard PPO under their system and design a modified, more tolerant version of the learning objective accordingly. Experiments are conducted on the competitive math and programming settings.", "questions": "**Suggestions**\n\n1. Extending the experiments in figures 5(a) and (b) to a few other extremely popular RL algorithm variants (e.g. GRPO & RLOO), as well as at least one model series other than Qwen would improve the experimental soundness of the work.\n\n2. Nit: Many of the figures in the paper use different fonts, font size, and color schemes. While this is a minor detail, it is distracting for a reader.\n\n\n**Questions**\n\n> However, all these systems still follow a batched generation setting, where all the samples 50 within a training batch are from the same model version.\n\nHow exactly does AReaL completely avoid this? From what I understood, there is still micro-batching at the device level during generation -- could you please expound upon this point?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- The paper is well-written. The figures are relatively clean and easy-to-follow.\n- The motivation for the method is compelling and the results are strong, though somewhat limited in scope. A new, fast, and open-sourced framework for RL is a valuable contribution to the community.\n\n---\n\n### Weaknesses\n\n**Limited scope of experimentation**\n\nExperiments with AReaL are limited to (1) PPO and (2) Qwen models. \n\n(1) While PPO is perhaps the most standard evaluation choice, variants of PPO like GRPO and alternative algorithms like RLOO are also popular choices for training reasoning models in the community. It would be valuable to see (a) what the differences in asynchronous tolerance and (b) the AReaL tradeoffs between speedup vs stability & performance gain look like across algorithms with different variants of advantage bootstrapping.\n\n(2) Many works have reported that reasoning-style RL on Qwen models behaves very differently to models from other families [1, 2]. Given this, I would be interested to see at least the experiments in Figure 5 (a) and (b) extended to another model family. For example, it is non-obvious to me that the degree of stability to off-policy data may not also be a somewhat model-specific property.\n\n\n[1] Shao, R., Li, S. S., Xin, R., Geng, S., Wang, Y., Oh, S., ... & Zettlemoyer, L. (2025). Spurious Rewards: Rethinking Training Signals in RLVR. arXiv preprint arXiv:2506.10947.\n\n[2] Gandhi, K., Chakravarthy, A., Singh, A., Lile, N., & Goodman, N. D. (2025). Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01307.", "was_revised": true, "final_justification": "In their rebuttal, the authors:\n\n(1) Expanded asynchronous tolerance evaluation with their framework from just PPO to RLOO, a popular alternative to training reasoning models.\n(2) Added an experiment on asynchronous tolerance with a Llama model, expanding from Qwen-only evaluations.\n\nI believe these additional experiments (especially (1)) strengthen the soundnesses of the work and its impact for the community. \n\nOverall, reviewers agree that the work has good quality, clarity, significance, and originality. Thus, I raise my score to \"accept\" -- this is a good paper.", "rebuttal": "We sincerely appreciate your thorough and insightful review\\! We would like to address your concerns with the following responses: \n\n> It would be valuable to see (a) what the differences in asynchronous tolerance and (b) the AReaL tradeoffs between speedup vs stability & performance gain look like across algorithms with different variants of advantage bootstrapping.  \n\nWe conducted additional experiments using the RLOO advantage bootstrapping method. Due to the tight rebuttal timeline, we ran 1.5B models in a local 8 GPU setting with 8k context length and various staleness values. The following evaluation results demonstrate that RLOO exhibits slightly better asynchronous tolerance to vanilla PPO.\n\n| model | AIME24 | AIME25 | AMC23 | MATH 500 | Throughput |\n| :---- | :---- | :---- | :---- | :---- | :---- | \n| 1.5B base | 29.3  | 24.4  | 71.0 | 84.3 | - |\n| RLOO staleness=0 | 32.4 | 29.2 | 79.2 | 87.3 |27.1k |\n| RLOO staleness=1 | 32.9 | 26.0 | 76.4 | 87.7 | 47.8k |\n| RLOO staleness=2 | 34.1 | 28.0 | 81.1 | 86.9 | 47.8k \n| RLOO staleness=4 | 32.9 | 27.9 | 76.0 | 87.0 | 49.0k |\n| RLOO staleness=8 | 31.5 | 28.1 | 78.0 | 87.4 | 51.5k |\n| RLOO staleness=16 | 32.7 | 27.7 | 78.4 | 87.4 | 52.0k |\n\nThe reviewer raises an important direction for future research. AReaL modifies the PPO/GRPO algorithm because the importance sampling term naturally supports asynchronous off-policy training. Beyond PPO-based workflows, it would be valuable to investigate asynchronous tolerance for REINFORCE-like and other off-policy algorithms. We consider this an important avenue for future work.  \n\n> Many works have reported that reasoning-style RL on Qwen models behaves very differently to models from other families \\[1, 2\\]. Given this, I would be interested to see at least the experiments in Figure 5 (a) and (b) extended to another model family.  \n\nThank you for this valuable suggestion\\! We should clarify that there exists a paradigm difference: references \\[1,2\\] mainly investigate reasoning patterns starting from ***base or instruction-tuned*** models, while AReaL focuses on RL fine-tuning of ***long CoT models*** (e.g., distilled from OpenAI-o1/DeepSeek-R1). After distillation, the reasoning pattern should be less affected by the base model. To validate this point, we conducted additional experiments with DeepSeek-R1-Distill-Llama-8B in the math setting. The following results demonstrate that AReaL's effectiveness transfers across different model families:\n\n| model | aime24 | amc23 | math500 | aime25 |\n| :---- | :---- | :---- | :---- | :---- |\n| base model | 50.4 | 84.2 | 89.1 | 23.3 |\n| AReaL staleness=4 | 58.4 | 92.3 | 92.2 | 42.6 |\n| AReaL staleness=8 | 57.2 | 91.5 | 91.9 | 41.6 |\n\nWe primarily used the Qwen family for experiments because: (1) it offers a range of model sizes (from 1.5B to 32B), which is convenient for both ablation studies (Figure 5\\) and scalability validation of AReaL, and (2) the Qwen family provides readily available distilled long CoT models for RL training (i.e., DeepSeek-R1-Distill-Qwen). While running ablation studies on other model families (e.g., Llama 8B) is more computationally expensive, we are prepared to conduct these experiments when additional computational resources become available and will include these results in the camera-ready version.  \n\n> How exactly does AReaL completely avoid batching?  \n\nAReaL avoids generation batching within the RL pipeline, but the inference engine will automatically handle request batching on GPU. To be more specific, instead of calling batched generation with `engine.generate`, AReaL asynchronously sends individual generation requests to a remote HTTP server. The server (e.g., vLLM or SGLang) handles request batching according to its internal scheduler logic and executes batched computation on the GPU, then returns HTTP responses individually to the RL pipeline. Therefore, from the perspective of the RL pipeline, AReaL completely avoids batching during rollout while still benefiting from the server-side batching computations."}]

Author rebuttal:

Meta-review: All reviewers recommend clear acceptance, highlighting the paper’s solid technical contributions, strong empirical validation, and clear impact on the LLM community. 

Several concerns raised by the reviewers were adequately addressed in the rebuttal, including:
- Comparison with VeRL (reviewer `u73n`)
- Model family dependence (reviewer `9Jfg`, `ETFe`)
- Coding coverage and hardward demand (reviewer `EQw4`)
- Unclear technical details (reviewer `ETFe`)

The authors are recommended to include these experiments/ablations in the revision and release the code to benefit the community.
Given the uniformly positive reviews and its contributions to both research and practice, I recommend acceptance.
Decision: Accept (poster)