Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm:
  Demystifying Some Myths About GRPO and Its Friends
Authors: name: Chaorui Yao, name: Yanxi Chen, name: Yuchang Sun, name: Yushuo Chen, name: Wenhao Zhang, name: Xuchen Pan, name: Yaliang Li, name: Bolin Ding

Off-policy reinforcement learning (RL) for large language models (LLMs) is
attracting growing interest, driven by practical constraints in real-world
applications, the complexity of LLM-RL infrastructure, and the need for further
innovations of RL methodologies. While classic REINFORCE and its modern
variants like Group Relative Policy Optimization (GRPO) are typically regarded
as on-policy algorithms with limited tolerance of off-policyness, we present in
this work a first-principles derivation for group-relative REINFORCE without
assuming a specific training data distribution, showing that it admits a native
off-policy interpretation. This perspective yields two general principles for
adapting REINFORCE to off-policy settings: regularizing policy updates, and
actively shaping the data distribution. Our analysis demystifies some myths
about the roles of importance sampling and clipping in GRPO, unifies and
reinterprets two recent algorithms -- Online Policy Mirror Descent (OPMD) and
Asymmetric REINFORCE (AsymRE) -- as regularized forms of the REINFORCE loss,
and offers theoretical justification for seemingly heuristic data-weighting
strategies. Our findings lead to actionable insights that are validated with
extensive empirical studies, and open up new opportunities for principled
algorithm design in off-policy RL for LLMs. Source code for this work is
available at
https://github.com/modelscope/Trinity-RFT/tree/main/examples/rec_gsm8k.