ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learning
Authors: name: Jie Xiao, name: Meng Chen, name: Qingnan Ren, name: Jingwei Song, name: Jiaqi Huang, name: Yangshen Deng, name: Chris Tong, name: Wanyi Chen, name: Suli Wang, name: Ziqian Bi, name: Shuo Lu, name: Yiqun Duan, name: Xu Wang, name: Rymon Yu, name: Ween Yang, name: Lynn Ai, name: Eric Yang, name: Bill Shi, name: Song Jingwei
Organization: Gradient

Reinforcement learning (RL) is a critical stage in post-training large language models (LLMs), involving repeated interaction between rollout generation, reward evaluation, and centralized learning. Distributing rollout execution offers opportunities to leverage more cost-efficient inference resources, but introduces challenges in wide-area coordination and policy dissemination. We present ECHO-2, a distributed RL framework for post-training with remote inference workers and non-negligible dissemination latency. ECHO-2 combines centralized learning with distributed rollouts and treats bounded policy staleness as a user-controlled parameter, enabling rollout generation, dissemination, and training to overlap. We introduce an overlap-based capacity model that relates training time, dissemination latency, and rollout throughput, yielding a practical provisioning rule for sustaining learner utilization. To mitigate dissemination bottlenecks and lower cost, ECHO-2 employs peer-assisted pipelined broadcast and cost-aware activation of heterogeneous workers. Experiments on GRPO post-training of 4B and 8B models under real wide-area bandwidth regimes show that ECHO-2 significantly improves cost efficiency while preserving RL reward comparable to strong baselines.