AReaL: A Large-Scale Asynchronous Reinforcement Learning System for
  Language Reasoning
Authors: name: Wei Fu, name: Jiaxuan Gao, name: Xujie Shen, name: Chen Zhu, name: Zhiyu Mei, name: Chuyi He, name: Shusheng Xu, name: Guo Wei, name: Jun Mei, name: Jiashu Wang, name: Tongkai Yang, name: Binhang Yuan, name: Yi Wu

Reinforcement learning (RL) has become a trending paradigm for training large
language models (LLMs), particularly for reasoning tasks. Effective RL for LLMs
requires massive parallelization and poses an urgent need for efficient
training systems. Most existing large-scale RL systems for LLMs are synchronous
by alternating generation and training in a batch setting, where the rollouts
in each training batch are generated by the same (or latest) model. This
stabilizes RL training but suffers from severe system-level inefficiency.
Generation must wait until the longest output in the batch is completed before
model update, resulting in GPU underutilization. We present AReaL, a
fully asynchronous RL system that completely decouples generation from
training. Rollout workers in AReaL continuously generate new outputs without
waiting, while training workers update the model whenever a batch of data is
collected. AReaL also incorporates a collection of system-level optimizations,
leading to substantially higher GPU utilization. To stabilize RL training,
AReaL balances the workload of rollout and training workers to control data
staleness, and adopts a staleness-enhanced PPO variant to better handle
outdated training samples. Extensive experiments on math and code reasoning
benchmarks show that AReaL achieves up to 2.57times training
speedup compared to the best synchronous systems with the same number of GPUs
and matched or even improved final performance. The code of AReaL is available
at https://github.com/inclusionAI/AReaL/.