Sharing is Caring: Efficient LM Post-Training with Collective RL
  Experience Sharing
Authors: name: Jeffrey Amico, name: Gabriel Passamani Andrade, name: John Donaghy, name: Ben Fielding, name: Tristin Forbus, name: Harry Grieve, name: Semih Kara, name: Jari Kolehmainen, name: Yihua Lou, name: Christopher Nies, name: Edward Phillip Flores Nuño, name: Diogo Ortega, name: Shikhar Rastogi, name: Austin Virts, name: Matthew J. Wright
Organization: Gensyn

Post-training language models (LMs) with reinforcement learning (RL) can
enhance their complex reasoning capabilities without supervised fine-tuning, as
demonstrated by DeepSeek-R1-Zero. However, effectively utilizing RL for LMs
requires significant parallelization to scale-up inference, which introduces
non-trivial technical challenges (e.g. latency, memory, and reliability)
alongside ever-growing financial costs. We present Swarm sAmpling Policy
Optimization (SAPO), a fully decentralized and asynchronous RL post-training
algorithm. SAPO is designed for decentralized networks of heterogenous compute
nodes, where each node manages its own policy model(s) while "sharing" rollouts
with others in the network; no explicit assumptions about latency, model
homogeneity, or hardware are required and nodes can operate in silo if desired.
As a result, the algorithm avoids common bottlenecks in scaling RL
post-training while also allowing (and even encouraging) new possibilities. By
sampling rollouts "shared" across the network, it enables "Aha moments" to
propagate, thereby bootstrapping the learning process. In this paper we show
SAPO achieved cumulative reward gains of up to 94% in controlled experiments.
We also share insights from tests on a network with thousands of nodes
contributed by Gensyn community members running the algorithm on diverse
hardware and models during an open-source demo.