Title: Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: LLM post-training; off-policy RLVR

Abstract: Traditional on-policy Reinforcement Learning Fine-tuning (RFT) frameworks suffer from experience waste and reward homogeneity, which directly hinders learning efficiency on difficult samples during large language models post-training. In this paper, we introduce Batch Adaptation Policy Optimization (BAPO), an off-policy RFT framework to improve the data efficiency in large language models post-training. It dynamically selects training batches by re-evaluating historically difficult samples and reusing high-quality ones, while holding a lower bound guarantee for policy improvement. Extensive experiments further demonstrate that BAPO achieves an average 12.5\% improvement over GRPO across mathematics, planning, and visual reasoning tasks. Crucially, BAPO successfully resolves 40.7\% of problems that base models consistently fail to solve.

TL;DR: We introduce Batch Adaptation Policy Optimization (BAPO), an off-policy Reinforcement Learning (RL) framework to improve the data efficiency in large language models post-training.