Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning
Authors: Xu Wan
Yansheng Wang
Wenqi Huang
Mingyang Sun
Venue: ICLR 2026 Poster (2026)
Keywords: LLM post-training; off-policy RLVR
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We introduce Batch Adaptation Policy Optimization (BAPO), an off-policy Reinforcement Learning (RL) framework to improve the data efficiency in large language models post-training.

Abstract: Traditional on-policy Reinforcement Learning with Verifiable Rewards (RLVR) frameworks suffer from experience waste and reward homogeneity, which directly hinders learning efficiency on difficult samples during large language models post-training. In this paper, we introduce Batch Adaptation Policy Optimization (BAPO), an off-policy RLVR framework to improve the data efficiency in large language models post-training. It dynamically selects training batches by re-evaluating historically difficult samples and reusing high-quality ones, while holding a lower bound guarantee for policy improvement. Extensive experiments further demonstrate that BAPO achieves an average 12.5\% improvement over GRPO across mathematics, planning, and visual reasoning tasks. Crucially, BAPO successfully resolves 40.7\% of problems that base models consistently fail to solve.

Published as a conference paper at ICLR 2026
BUFFER MATTERS: UNLEASHING THE POWER OF OFF- POLICY REINFORCEMENT LEARNING IN LARGE LAN- GUAGE MODEL REASONING
Xu Wan Yansheng Wang*, Wenqi Huang Mingyang Sun Zhejiang University + Bytedance Seed Roboties China Southern Power Grid
Peking University
ABSTRACT
Traditional on-policy Reinforcement Learning with Verifiable Rewards (RLVR) frameworks suffer from experience waste and reward homogeneity, which di- rectly hinders learning efficiency on difficult samples during large language mod- els post-training. In this paper, introduce Batch Adaptation Policy Optimiza- tion (BAPO), an off-policy RIVR framework to improve the data efficiency large language models post-training. dynamically selects training batches by re- evaluating historically difficult samples and reusing high-quality ones, while hold- ing lower bound guarantee for policy improvement. Extensive experiments fur- ther demonstrate that BAPO achieves an average 12.5% improvement over GRPO across mathematics, planning, and visual reasoning tasks. Crucially, BAPO suc- cessfully resolves 40.7% of problems that base models consistently fail to solve.
INTRODUCTION
Reinforcement Learning from Human Feedback (RLHF) has emerged transformative paradigm for aligning Large Language Models (LLMs) with human preferences and improving their per- formance on complex reasoning tasks (Ouyang al., 2022; Bai al., 2022). A significant recent evolution is Reinforcement Learning with Verifiable Rewards (RLVR) (Lambert et al., 2024), which replaces costly neural reward models with deterministic verification functions for more efficient and reliable training (Guo et al., 2025). Numerous on-policy RL optimization methods, particularly Group Relative Policy Optimization (GRPO) (Shao et al., 2024), and its variants like Dynamic Sampling Policy Optimization (DAPO) (Yu et al., 2025), Group Sequence Policy Optimization (GSPO) (Zheng et al., 2025), have demonstrated remarkable success in LLM post-training scenar- ios, achieving exceptional performance mathematical reasoning, code generation, and various downstream applications (Yang et al., 2025; Chen et al., 2025a; Shen et al., 2025).
Although with lower bound guarantees of policy improvement theoretically (Mroueh, 2025), existing RL post-training frameworks still face significant ef- ficiency challenges in practice. As shown in Fig- ure 1, models after GRPO post-training struggle handle difficult samples, especially those with zero accuracy in the initial rollout group. The reasons are twofold: (1) Homogeneous rewards: Recent in- vestigations (Hong et al., 2025; Simoni et al., 2025)
Number of Prompts
400
3350
(DeepSeek
highllihht accunacy group
300
200-
198
96
61
62
67
Number of Prompts
After GRPO Epoch)
300
271
200
reveal that samples at both extremes of difficulty of- fer minimal benefit for post- tostt-raaiing policy improve- ment. This arises because advantage estimation in most GRPO-bbsed methods relies heavily on rel- ative reward diversity within each group. Conse- quently, when intra-group rewards are identical, the lower bound guarantee for policy improvement col-
100
51
7/8
Accuracy Bins
Figure Tracking the sample counts across accuracy groups of the mathematical dataset before and after GRPO post-training.
"Coresponding Author

---

Published as a conference paper at ICLR 2026
lapses (Zhang al., 2025; Mroueh et al., 2025), resulting in negligible effective gradient contribu- tions (Liu et al., 2025; Yu et al., 2025). (2) Waste experience: Given the sensitivity of policy improvement group reward variance, uneven difficulty distributions yield significantly fewer high-quality samples than the configured batch size implies. Crucially, since these methods are pri- marily on-policy and lack experience replay, each rollout group is consumed only once, leading to substantial waste of valuable training data (Sun et al., 2025; Li et al., 2025).
A straightforward solution is to adopt off-policy rather than on-policy training paradigms, which has been established in traditional RL tasks as a viable solution to increase sample efficiency and diversity in the training batch (Queeney al., 2021; Hilton et al., 2022; Meng al., 2023). However, naively applying sample-reusing schemes to RL frameworks may exacerbate instability during LLM post-training, leading to entropy collapse, and ultimately performance degradation (Yu al., 2025; He et al., 2025; Chen et al., 2025c).
Thus, to systematically exploring the utility of stale off-policy experience RLVR post-training, we incorporates multiple off-policy strategies into on-policy RLVR framework to dissect effective pathways for historical data utilization. The main contributions of this paper are as follows:
(1) We propose a difficulty-aware experience replay mechanism practical solution for efficient off-policy data utilization. Unlike the simple mixing of the buffer's data and online data, we ac- tively re-evaluate historical hard prompts to drive exploration while directly reusing high-quality trajectories with a dynamic quality threshold.
(2) Theoretically, we prove that under certain assumptions, the proposed adaptive construction mechanism mitigates the homogeneous reward issue via adaptive batch construction and KL- constrained updates.
(3) By integrating it into multiple reasoning tasks with different LLM backbones, we validate the proposed Batch Adaptation Policy Optimization (BAPO) method achieves better convergence and yields greater improvements on solving difficult samples compared to existing on-policy and off- policy RLVR frameworks.
2 RELATED WORK
2.1 ON-POLICY RL POST-TRAINING FRAMEWORK
We first review the concept of on-policy RLVR, where the core objective is to optimize an LLM policy to maximize the outcome response reward. Let X represent the input prompts, and E y' denote responses generated by the LLM policy тy. The terminal reward r(z, y) E {0. determined by deterministic verification function (Lambert al., 2024; Guo al., 2025). Following the setting of GRPO (Shao et al., 2024), the objective is formulated as:
G
min (pre(0)A (lip(0) 1, +c) - (TK (1)
1=1
T=1
where G {v1 1/2 vG) represents G-size group of responses sampled from no, (-1z) for each
input x; (0) is the probability ratio xo xo(ulle
Te.u (M(l,)" between current policy and old policy Tea for the i-th responses' t-th token, E limits the magnitude of policy updates; and Dxr. constrains the policy Ty from deviating too far from a reference policy Tee.- Crucially, A., denotes the estimated advantage of response y4 for input x, which derived from the standardization of rewards using the statistical properties of group G. For the i-th response yı E G with reward r; = r(æ, yi), the estimated advantage is:
A. 10-mmenn( ((re))
/ (tre)( +
where mean (fre)) and sta? (fre)) are the empirical mean and variance of rewards in group G. re- spectively.
To enhance the practical efficiency of GRPO, a series of improved on on-policy frameworks has been proposed. For instance, DAPO (Yu et al., 2025) sets distinct clipping ranges flow and Ehigh. and
2

---

Published as a conference paper at ICLR 2026
employs dynamic sampling strategy to ensure Å., * 0. However, it consumes approximately four times the number of rollouts (Qu et al., 2025) compared to GRPO. Meanwhile, GSPO (Zheng et al,, 2025) abandons the token-level ratio (i, and shifts to the sequence level s,))), which has been validated to maintain more stable training, particularly Mixture-of-Expers. (MoE) architectures.
While the details of these methods vary, they all adhere to the on-policy framework for sampling and updates: the inference server is updated in synchronization with the trainer parameters, and the sampling strategy follows the uuse-onceanndddiscar" principle throughout the training process.
2.2 OFF-POLICY RL POST-TRAINING FRAMEWORK
In contrast, shown in Figure 2, off-policy RL post-training frameworks operate under distinct paradigm, characterized by two core components: off-policy rollout for generating responses and off off-policy training for constructing the training batch, detailed below.
Off-policy Rollout avoids exclusive reliance the current training pol- icy for generation, instead leverag- ing past policies external guid- ance. For example, AReaL (Fu al., 2025) employs fully asyn- chronous architecture that decouples generation from training. allowing rollout workers to use past policy. Mroueh et al. (Mroueh et al., 2025) fix the rollout policy the vLLM inference server for multiple itera- tions to ensure stable sample gener- ation. LUFFY (Yan et al., 2025) in- corporates traces from stronger exter- nal policies to enhance reasoning ca- pabilities beyond the model's initial limits.
Throw out samples after train
Batch Inputs
On-policy Rollout
On-policy Training
Same policy
(a) On-pollcy Framework
Select important samples
Historical samples
DOO
Batch Inputs
Off polppolicy Rollout
Off-palicy Training
Select
Different policy---eeee
(b) Off-policy RLVR Framework
Figure
The overview the (a) on-policy and (b) off- policy RL Post-training framework
Off-policy Training uses replay buffers manage samples from historical policies with varying activation strategies. ARPO (Lu et al., 2025) dynamically samples non-zero reward samples from the buffer only when current batches contain all-zero rewards. DOTS (Sun al., 2025) maintains a FIFO buffer that consistently reuses recent valid rollouts. RePO (Li et al., 2025) mixes buffer samples with on-policy samples using diverse retrieval strategies. ReMix (Liang et al., 2025) blends samples at fixed ratios while increasing the update-to-danl ratio for efficiency. ReLIFT (Ma et al., 2025) stores high-quality solutions to challenging problems
buffer and refines them through interleaved supervised fine-tuning. Kimi K1.5 (Team al., 2025) stores both complete and partial trajectories to reduce temporal correlations while maintaining computational efficiency.
However, most off-policy RLVR methods ignore the policy stability of experiences. Samples en- tering the buffer at different training steps may exhibit varying policy distributions. These discrep- ancies introduce excessive noise into policy learning, which in turn exacerbates training instability. More importantly, simply reusing historical samples may even hinder the policy's improvement. The high-accuracy historical samples may cause the model overly focus existing reasoning paths with high advantages, suppressing the model's exploration capability and resulting in premature convergence suboptimal solutions (Cui et al., 2025).
3 METHOD
In this section, we detail the core components of BAPO, particularly the adaptive construction strat- egy for the training batch, and provide theoretical guarantee for the training stability BAPO's policy update. Figure provides overview of the off-policy rollout and training workflow.
3

---

Published as a conference paper at ICLR 2026
Pr
Pig
Rollout Policy
M
Current Policy
F+
( E (c,,
Prepere prompls
Rollout G Reeporiee
Get (n, G) Rewards
j [0,ci]
Biac
Fresh samples
Buss
Fresh
Re-evaluated
Samp
High-quality
(a) Off-policy Rollout
(b) Off-policy Training
Figure 3: The workflow of (a) off-policy rollout and (b) off off-policy training in our RLVR framework
3.1 FORMAL DEFINITIONS
We first formalize our training objective La (TT) a combination of online rollout-derived and historical buffer-derived contributions:
La (TO) E( (a) A(z, v)
B (a Pas (0) A(x, v) B (x
(3)
Contribution from fresh samples
Coutribution from historical samples
where (x. y) ~ a refers to filtered online samples from the rollout policy a = me._. with e > representing the delay timesteps. (x, B denotes historical samples from the replay buffer B. The importance sampling ratios are defined as Pa = ze(vlz) for the online rollout samples and
oggla)
xe(y(æ) for buffer samples, og is the historical rollout policies that generated the buffer.
Pas
as(yln)
Each entity in the buffer B is formally defined as:
B = (w: {xt, 1 (rag) (as(ur,) (r) [m))
(4)
where u, is the unique identifier of each prompt, {- (1 (rey) represent the set of prompts, generated responses, and corresponding rewards, respectively. (a (= is the rollout pol- icy's probability, which stored for calculating Pas when reusing, and [B] is the buffer size.
3.2 ADAPTIVE TRAINING BATCH CONSTRUCTION
The core of off off-policy RLVR lies how to integrate historical experiences with online samples, to maintain non-homogeneous rewards and an appropriate difficulty distribution in each training step. For BAPO, we introduce a filter function ((x) in Definition 3.1 that decomposes the data selection criteria for each training step's batch into three parts.
Definition 3.1 (Training Batch Filtering Function). Define (,,((z) = By (+12) [r(2, (r) v)] as the
expected reward under policy π for input x. The training batch indicator function 1:X' + {0, 00 is formulated as:
T(x:) (4) (tag
+ (ansS
(5)
Fresh
Improved Historical Diflicult
Historical High-quality
where a denotes the delayed rollout policy and a denotes the policy associated with buffer samples. The function selects samples based on three criteria, yielding subsets X,, X2 and X'3 respectively.
Next, we explain the selection principles for [(x) and derive three categories of samples, namely X1, X2, and X3, which obtained from these three conditions, respectively.
(1) Filtered Fresh Samples (X1). To prevent gradient vanishing and maintain training stability, we filter the online rollout batch exclude samples with zero variance. Specifically, retain fresh

---

Published as a conference paper at ICLR 2026
samples where the group mean reward satisfies (a. l 1. While other filtering strate- gies (e.g., Gaussian sampling or uniform sampling) can applied, we find that simple truncation sufficient for effective learning. A detailed discussion and comparison of different online filtering functions are provided in Appendix A.3.
(2) Improved Historical Difficult Samples (222). Samples exhibiting extremely low group mean rewards, where (2) & [0, ci), present significant challenges to the current policy and typically yield negligible policy improvement. However, as the model evolves, these historically difficult queries may eventually become tractable for a successor policy. To harness this, we periodically re-g re-generate responses using the current policy Te, every m training steps and construct the subset X/ based on the observable improvement.
Let Boad B denote the buffer for difficult samples. To manage the computational overhead asso- ciated with the re-evaluation process, we limit the buffer capacity |Beall to be equal to the training batch size. A First-In-First Li--iirtt-Ou (FIFO) mechanism employed to automatically discard outdated samples when the buffer reaches capacity. X2 is formulated as:
X2 =
{ (z, v) (z, y) E Brad y ~ Te, (: x), < (r),,(()) < <1}
(6)
where y' represents the response generated by no, and we specifically select samples that show improvement such that e < (TTo.-(()) <
(3) Reused Historical High-quality Samples (At).. To prevent underfilled batches caused by the scarcity of 2 and X2, we maintain a FIFO auxiliary buffer Bligh S B. To mitigate training insta- bility from stale data, Bhigh is restricted to high-quality trajectories from the three most recent steps. The subset X3 is randomly sampled to fill the remaining capacity:
X3 (Brgh, min (BBigh), B | - 1x22))
where the configured training batch size and S(., denotes the random sampling of elements. Furthermore, to progressively master increasingly difficult tasks, we employ linear mapping to shift the historical "high-quality" from easier to harder instances, scaling in accordance with the global average performance Tior
C = Fiot (c,
c + C.,, i 2,3
(8)
3.3 THEORETICAL ANALYSIS
In this section, we further provide theoretical analysis in Theorem 3.2 to establish BAPO's training stability based on (Mroueh et al., 2025)'s theorem. We show that, under certain assumptions, our constructed adaptive batches can consistently maintain guaranteed bounded policy improvement.
Theorem 3.2 (Policy Improvement Lower Bound with Adaptive Training Batch). Assume re- wards are bounded: 0 r < 1. Let no, be the current policy, on To.-. be the delayed rollout policy, 022 = no, the current policy for re-evaluation, 0z = ag be the buffer policy distribution, and (xx) be the filtering function partitioning samples into X,. X2, and Nz.
Suppose cı, C2, 03 E =0,11 with 02 < C3, and the following TV distance constraints hold:
(le)) TV(To, (11x), (18((=))))) 83
81 Vr X1
(9)
Vo X's
(10)
where 81, 83 > 0 are sufficiently small such that the variance lower bounds remain positive.
Then, for the policy update objective in Equation 3, the expected policy improvement over filtered samples satisfies:
Ex-px (T(x) - (+1x)) >
Σ La(To,
where:
[ x)) = E Ey-no (m) r(x,))
Li(To, az) Erex, (To, (mo(-)2:) 2K, TV(T0(+2)) ax(+)=)) (7V os(())))
5

---

Published as a conference paper at ICLR 2026
with Las (To (T6(+)))) (as.mc())
1
(J(T)(-12)) [(as(-lz))) The constants are:
1 1
G-1 +e G
K1
(11)
G-1
G +E
1 (1(119))
K2 =
(12)
Ve - cc)
1 y (1
+
K3
(13)
/min(22( 02), ca ca)) +
More importantly, we highlight several properties from this theorem:
Bounded Stability. All constants K1. K2, and K3 are finite positive values, which guarantee that the training process remains numerically stable and theoretically bounded.
Off-policy Tolerance. The stability of trust-region methods inherently constrain the magnitude of single-step policy updates. Consequently, the divergence between the current policy To, and the delayed rollout policy remains bounded over short intervals. Furthermore, the strict FIFO mechanism with limited buffer capacity ensures that only samples from recent policies are retained, thereby maintaining policy consistency within the training batch.
4 EXPERIMENTAL SETUP
To comprehensively evaluate the effectiveness of our off-policy RLVR framework, we conduct extensive experiments across different tasks and backbones, following the experimental setup de- scribed in (Qu et al., 2025).
First, we select three representative reasoning tasks, as detailed below:
Mathematics. Following prior work (Luo et al., 2025), we use the DeepSeek R1 Distilled 1.5B (Guo et al., 2025) and Qwen3 8B (Yang et al., 2025) as the base model, and conducted post-training on the CeepScallR-Preveeww Dataset (Aggarwal & Welleck, 2025), which contains 40 thousand question- answer pairs sourced from several mathematics competitions. Evaluation is performed on multiple mathematics benchmarks, including AIME24, AMC23, MATH500 (Hendrycks et al., 2021), Min- erva Math (Minerva) (Lewkowycz et al., 2022), and OlympiadBench (Olympiad) et al. (2024).
Planning. We choose Qwen2.5 Math 1.5B and 7B (Yang al., 2024) the backbone, and adopted the Countdown Number Game as the specific task. For training, we used a 10,000-problem subset of the Countdown-34 dataset, where each problem provides 3-4 source numbers. Evaluation was conducted on two variants: Countdown-3ter
(CD-34) test set using a 200-problem held-out split, and the more challenging Countdown-4 (CD-4) test set with 200 problems that consistently provide four source numbers (Chen et al., 2025b).
Visual Geometry. We train Qwen2.5 VL and 7B (Bai et al., 2025) on the 2,101-problem training split of the Geometry3K dataset (Lu et al., 2021), where each problem consists of geometric diagram paired with a natural language question requiring spatial and logical reasoning. Evaluation was performed on the official 300-problem validation split (Geo-3K val) and 601 601-problem test split of GKKmetry3K (Geo-3K test).
Besides, we select several on-policy and off-policy RLVR frameworks as baselines:
On-policy. We select GRPO (Shao et al., 2024), DAPO (Yu et al., 2025), and MoPPS (Qu et al., 2025) representative on-policy RLVR methods. GRPO is the first to integrate group-relative advantage estimation into the RLVR framework, while DAPO further improves training stability and efficiency. MoPPS incorporates difficulty-aware prediction into prompt selection.
Off-policy. We compare our approach with three representative off-policy methods: GRPO (o = 5) (Mroueh et al., 2025), RePO (Li et al., 2025), and Remix- GRPO (Liang et al., 2025). Specifically, GRPO (v = 5) delays the rollout policy with a frequency of 5, whereas RePO and Remix-GRPO adopt diverse replay strategies retrieve off-policy samples from replay buffer.

---

Published as a conference paper at ICLR 2026
Implementation Details. All comparative experiments memory based on the Verl framework (Sheng et al., 2025). Identical parameters were used to ensure fair comparison, with specific details in Appendix A.7.
8 A100 GPUs with 80GB
5 RESULTS ANALYSIS
5.1 MAIN RESULTS
We evaluate BAPO across three reasoning tasks demonstrate its broad applicability. Experimental results show that BAPO consistently outperforms existing baselines throughout training (Figure 4) and testing (Figure 12). Notably, in mathematical tasks, the GRPO bascline exhibits severe training instability, as evidenced by significant oscillations in its early-stage training curve. This is attributed the high variance in problem difficulty within the DeepScalerR dataset. Under the same settings, BAPO achieves smoother convergence and higher reward bounds.
In Tables BAPO achieves an average 12.5% accuracy improvement over baselines. Crucially, while DAPO approaches BAPO's performance in some metrics, it requires approximately 2.5x more rollouts (as visualized in Figure 9), imposing substantial computational burden.
Mathmetical Tesk
Planning Task
Visual Geometry Task
0.60
0.7
DAPO GRPO
BAPO
0.6
0.55
GRPO
(T)
GRPO
Train Rewards
Rewards
0.5
0.5
0.50
0.4
0.45
0.3
0.3
0.40
0.2
0.0
0
100
200
300
100
20
40
60
80
100
50
100
150
200
Training Steps
Training Steps
Training Steps
Figure 4: Training Curves of Reward Changes for mathematies, planning, and geometry tasks using DeepSeek Distilled Qwen 1.5B, Qwen2.5 Math 1.5B, and Qwen2.5 3B, respectively.
Table 1: Comprehensive Evaluation Results. '+' indicates fine-tuning via the corresponding method. Accuracy averaged over 32 runs. The bold value denotes the top result, and the underlined value denotes the second-top result.
(a) Mathematics Benchmarks
Method
AIME24
AMC
MATH500
Minerva.
Olympiad.
Arg.
Rollouts
Type
DeepSeek Qwen 1.5B
62.90
+CRPO (Guo ct al., 2025)
30.73
67.47
85,40
28.95
45.33
$1.58
677k
+DAPO (Yu et 2025)
35.73
70.08
86.05
30.70
48.48
$4,20
1921k
on
+MoPPS (Qu et al., 2025)
33.33
65.29
84.94
28.88
45.93
$1.67
737k
GRPO (v 5) (Mroueh et al., 2025)
30.49
65.09
86.72
28.16
46.18
$1.57
677k
off
+RePO (Li al., 2025)
30.42
64.76
83.75
28.33
45.44
50.54
677k
off
-Remix-GRPO (Liang et al., 2025)
33.33
65.06
84.60
26.10
43.55
50.53
off
+BAPO (Ours)
38.54
72.74
733k
off
(b) Planning Visual Geometry Benchmarks
Method
CD-34
CD-4
Avg
Method
Geo-3K(val)
Geo-3K(test)
Avg
Qwen2.5 Math 1,58
1.12
0.37
0.75
Qwen2.5 VL. 3B
14.77
19.18
16.98
+GRPO (Guo et al., 2025)
62.94
35.88
49,41
+GRPO (Guo et al., 2025)
36.44
43.12
39.78
+DAPO (Yu et al., 2025)
70.56
45.87
58.22
+DAPO (Yu et al., 2025)
40.11
45.18
42.65
+BAPO w/o X, (Ours)
60.31
35.31
47.81
+BAPO w/o X'2 (Ours)
30.57
36.92
33.75
+BAPO Xs (Ours)
64.43
51.59
+BAPO w/o. X' (Ours)
32.22
39.79
+BAPO (Ours)
73.00
47.50
60.25
+BAPO (Ours)
40.11
46,33
43.22
Qwen2.5 Math 7B
2.68
0.94
1.81
Qwen2.5 VL. 7B
30.40
36.10
33.25
+GPOO (Guo al., 2025)
70.75
50.25
60.50
+GRPO (Guo 2025)
40.79
47.15
43.97
+DAPO (Yu et al,, 2025)
78.25
57.43
68.09
+DAPO (Yu et al., 2025)
40.87
47.02
43.95
79.13
+BAPO (Ours)
41.89
48.77
metiollss performaacc from correspunding pojer.

---

Published as a conference paper at ICLR 2026
5.2 MECHANISM ANALYSIS
To deeply investigate whether BAPO's success stems from sensitive hyperparameter tuning its core batch reconstruction mechanism, we conducted both Minimalist Verification and Hyperpa- rameter Robustness experiments.
Off-policy Components OOff-policy Hyperparameters
The performance gains of BAPO primarily stem from the structural logic of its off- components rather than specific hyperparameter settings. The framework remains effective even under rigid, parameter-free conditions.
BAPO GAPO
BAPO (mini GAPO
BAPO DMPO
0.40
0.35
0.35
0.35
0.30
0.30
0.30
0.23
0.25
0.15
0.15
0.10
0.10
0.10
0.8
0.05
(ntr
0.05
0.00
0.00
Traniing
Figure 5: Test Curves of Group Accuracy Changes on AIME for different RLVR methods based on Qwen3 8B. Left: Standard BAPO vs. GRPO. Medium: BAPO (mini test) vs. GRPO. Right: Standard BAPO vs. DAPO.
Minimalist Verification. To validate the theoretical implications of Theorem 3.2 without relying on hyperparameter engineering, specifically avoiding the tuning of thresholds Cı c2 C3 and update frequencies, we devised a "Mini-test" experiment. We trained Qwen3 8B on the mathematics task under 4K length constraints using stripped-down, parameter-free BAPO logic for constructing training batch:
Xi: We apply strictly standard zero-advantage filtering, removing only the prompts where all G responses are entirely correct or entirely wrong.
X:: We replay historical all-wrong samples (x), 0). These correspond exactly the difficult cases discarded by X1, creating closed-loop system that recovers waste data without requiring difficulty threshold c₁.
Xx: Instead of dynamic accuracy range, we reuse historical samples with exactly 50% accuracy. As formally proven in Proposition A.3, samples with accuracy (a, 1/ maximize the reward variance, thereby providing the theoretical maximum potential for single-step policy improvement ((ee) ).T.))
The results in Figure 5 demonstrate that even in the hyperparameter-free "Mini-test", BAPO main- tains clear advantage over GRPO. This confirms that the structural introduction X2 and X drives the performance, not the specific tuning of e values.
Component Efficacy. To evaluate the contribution of re-evaluated difficult samples X2 and reused high-quality samples Xz. we conduct ablation studies shown in Table 1 and Figure 6 (Column 2). Both components are essential: removing X' causes ~21% performance drop, underscoring the importance of explicitly targeting difficult samples.
Hyperparameter Robustness. We further evaluate the sensitivity of BAPO to its key hyperparame- ters: rollout delay v. re-rollout frequency m, and difficulty thresholds. Frequency (v, m): As shown in Figure 6 (Column 1), performance remains stable within reasonable ranges (e.g., 0 5, m = 5). Extreme delays only degrade performance when policy divergence becomes excessive, aligning with our theoretical analysis regarding the trust region. Difficulty Thresholds (C2, ca): While our adap- tive boundary mechanism yields the best convergence, Figure 6 (Column 3) shows that using fixed
8

---

Published as a conference paper at ICLR 2026
ranges still significantly outperforms baselines. This indicates that the presence of diverse historical data is more critical than the precise values of the thresholds.
CD-34 Test Set
CD-4 Test Set
CD-4 4 Test Set
0.40
0.6
0.35
Test Accuracy (1)
0.5
Accuracy (T)
0.30
0.3
0.3
0.2
BAPO (v=5.mm5)
BAPO
BAPO (c1=0/8)
mv-
0.3
0.10
(c1-1/8)
0.1
BAPO (v=20
BAPO (wlo X3)
0.05
BAPO (c1=2/8)
0.0
0.00
0
20
40
60
80
100
120
0
20
40
60
80
100
120
20
40
60
80
100
120
Training Steps
Training Steps
Training Steps
CD-34 Test Set
Geo3K Test Set
1D-34 Test Set
0.6
Test Accuracy
0.4
Accuracy
0.35
0.5
0.4
0.3
0.30
0.3
BAPO (v=5mm=1)
0.25
BAPO
0.2
BAPO (adap c2 c3)
BAPO 5.-5
(w/o X2)
BAPO (c2-4/8, (3-518)
0.1
BAPO 5v=
0.20
BAPO
(w/o
0.1
BAPO (c2-1/8, c3-2/8)
0.0
0
20
40
60
80
100
120
25
50
75
100
125
150 175 200
20
40
60
80
100
120
Training Steps
Training Steps
Training Steps
Figure 6: Ablation Studies for BAPO. The first column presents ablations on frequency-related hyperparameters (m, v). The second column shows ablations on buffer subsets (22, Kz). The third column compares fixed vs. adaptive difficulty thresholds.
5.3 DETAILED ANALYSIS
We analyze BAPO's internal mechanisms below. For extended analysis on training dynamics, com- putation, and visualization, please refer to Appendices A.4, A.5 and A.6.
Tracking Difficult Samples. We visualize the training dynamics in Figure 7. BAPO exhibits superior capability to "unlock" difficult problems: after 3 epochs, BAPO successfully improves 31 % of the samples that were initially unsolvable (0/8 accuracy), compared only 19 % for GRPO.
Base (Step
GRPO (Step 471)
BAPO (Step 471)
Number of Proenpts
i:e
350
283
308
350
232
000
190
203
302
390
337
108
100
45
62
67
51
58
58
76
45
Accuracy Bins
Accuracy Bins
Accuracy Bins
Figure 7: Tracking changes in the Number of Different Accuracy Bins on the DeepScalerR training subset. Special attention paid to the reduction of bad samples (red bars).
Sample Distribution & Efficiency. To uncover the source of BAPO's efficiency, we analyze the dynamic batch construction in Figure 8 alongside the rollout costs in Figure 9.
As observed in Figure 8, the assembled training batch size frequently fluctuates below the maxi- mum configured capacity. This reduction in backward propagation load effectively offsets the com- putational overhead caused by off-policy re-evaluation and log probbabbbyly re-computation. Con- sequently, detailed in Table. 2, BAPO maintains training speed comparable to GRPO while requiring significantly fewer rollouts than DAPO, achieving a superior trade-off between conver- gence performance and computational cost.

---

Published as a conference paper at ICLR 2026
X1 Samples
X2 Samples
X3 Samples
Total Samples
700
1000
000
1500
1500
500
400
300
400
500
200
500
Plarning
100
Geometry
0
50
100
156
200
250
200
0
50
100
150
200
250
300
30
100
150
200
250
300
50
100
150
200
250 300
Treining
Triining
Steps
Figure& 8: Dynamic Sample Distribution. The composition of BAPO's 12, M3 and the total samples compared to fixed GRPO batch size (Red line).
Mathematical
Planning
Vision Tark
3400
DAFO BARO
D2P0
800
SAPO
Rooheat (eetche
BAPO
0.0
Reduced 2:8X rollouts
200
Reduced 2.5x rellouts
Reduced 1.8X rollouts
200
Cumu
800
Curmuatiion
150
200
100
Cummaet
100
50
50
100
150
230
250
300
350
20
40
60
30
100
120
140
40
63
100
120
140
Training Stops
Taining Stops
Traiiing Steps
Figure 9: Cumulative Rollout Batches Comparison between BAPO and DAPO. The maximum rollout time for DAPO is set to 4.
Efficient Batch Adaptation
BAPO maintains training efficiency comparable to GRPO. While the periodic re-evaluation of X₂ introduces additional generation overhead, this cost is effectively offset by reduced training samples, particularly during the initial stages of training.
CONCLUSION
In this paper, we propose BAPO, an off-policy RIVR framework for LLM post-training. It aims utilize historical training data better and thereby improve training efficiency. Specifically, appropriately delay the rollout policy to stabilize the policy discrepancies of buffer samples. More importantly, we construct training batches by re-evaluating difficult samples and reusing historical high-quality ones, thereby enhancing the efficiency of post-training. We validate the strong adapt- ability of the BAPO framework through experiments on three distinct reasoning tasks using different LLM backbones, and the results demonstrate that BAPO significantly outperforms baselines in both convergence performance and training efficiency. Nevertheless, exploring how to adapt BAPO to large models with MoE architectures. well to agentic RL frameworks, remains significant challenge.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported by the National Natural Science Foundation of China under Grant 7257 1007 and Grant 72555533777577,,958881,,,,, and by Beijing Nova Program (No. 20250484850).
ETHICS STATEMENT
All authors of this study strictly adhere to the ICLR code of ethics. Our research does not involve any potential conflicts of interest or sponsorship issues. We have carefully considered and addressed concerns related to discrimination, bias, and faireess in our methodology. The study raises no pri- vacy or security concerns, maintains full legal compliance, and upholds the highest standards of research integrity. All experimental procedures and data handling practices follow established ethi- cal guidelines for machine learning research.
REPRODUCIBILITY STATEMENT
To ensure full reproducibility of results, we provide comprehensive implementation details of the proposed BAPO training algorithm the supplementary materials. All experimental settings, hyperparameters, and dataset specifications tions, complete proof's and clear explanations of all assumptions are included in the appendix. Code and data will be made available upon acceptance facilitate replication of findings.
clearly documented. For our theoretical contribu-
THE USE OF LARGE LANGUAGE MODELS
In this research, employed LLMs solely language editing tools to improve the clarity and read- ability of our manuscript. LLMs were used for grammar checking, style refinement, and language polishing purposes only. All core research ideas, experimental design, analysis, and conclusions entirely the original work of the authors. The use of LLMs did not contribute to the conceptual or technical content of this study.
REFERENCES
Pranjal Aggarwal and Sean Welleck. L1: Controlling how long reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503. 04697, 2025.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Yuntao Bai. Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 22X4v:220.00566. 2022.
Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng. Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning A survey of long chain-of- thought for reasoning large language models. arXiv preprint arXiv:2503. 09567, 2025a.
Xiaoyin Chen, Jiarui Lu, Minsu Kim, Dinghuai Zhang, Jian Tang, Alexandre Piché, Nicolas Gontier, Yoshua Bengio, and Ehsan Kamalloo. Self-evolving curriculum for Ilm reasoning, arXiv preprint arXiv:2505. 14970 /4970, 2025b.
Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catan- zaro, and Wei Ping. Acereason-nemontee forcement learning, arXiv preprint arXiv:2505. 16400, 2025c.
Advancing math and code reasoning through rein-
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo. Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.2266 2025.

---

Published as a conference paper at ICLR 2026
Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang. et al. Areal: A large-scale asynchronous reinforcement learning system for language reasoning. arXiv preprint arXiv:2505.24298, 2025.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprini arXiv:2501.1294 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint 27Xi::2402.14008,, 2024.
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, et al. Skywork open reasoner technical report. arXiv preprint wrXiv:2505.22312. 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Erie Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03877 2021.
Jacob Hilton, Karl Cobbe, and John Schulman. Batch size-invariance for policy optimization. Ad- vances Neural Information Processing Systems, 35:17086- 2022.
Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, et al. Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning, arXiv preprint arXiv:2507.0 2025.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint
1...22211.11122 2024.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, el. Solving quantitative reasoning problems with language models. Advances neural information processing systems, 35:3843-3883 -3554, 2022.
Siheng Li, Zhanhui Zhou, Wai Lam, Chao Yang. and Chaochao Lu. Repo: Replay-enhanced policy optimization. arXiv preprint arXiv:2506.03340. 2025.
Jing Liang, Hongyao Tang, Yi Ma, Jinyi Liu, Yan Zheng, Shuyue Hu, Lei Bai, and Jianye Hao. Squeeze the soaked sponge: Efficient off-policy reinforcement finetuning for large language model. arXiv preprint arXiv:2507.06m 006992, 2025.
Ziru Liu, Cheng Gong. Xinyu Fu, Yaofang Liu, Ran Chen, Shoubo Hu, Suiyun Zhang, Rui Liu, Qingfu Zhang, and Dandan Tu. Ghpo: Adaptive guidance for stable and efficient Ilm reinforce- ment learning. arXiv preprint arXiv:2507. 10628, 2025.
Fanbin Lu, Zhisheng Zhong, Shu Liu, Chi-Wing Fu, and Jiaya Jia. Arpo: End-to-end policy opti- mization for gui agents with experience replay. arXiv preprint 252.:22005.6682, 2025.
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 6774 6786, 2021.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, et al. Deepscaler: Surpassing ol ol-preview with a 1.5 model by scaling rl. Notion Blog, 2025.
Lu Ma, Hao Liang, Meiyi Qiang, Lexiang Tang. Xiaochen Ma, Zhen Hao Wong, Junbo Niu, Chengyu Shen, Rumming He, Bin Cui, et al. Learning what reinforcement learning can'tt In- terleaved online fine-tuning for hardest questions. arXiv preprint ar arXiv:2506.07522 2025.
12

---

Published as a conference paper at ICLR 2026
Wenjia Meng, Qian Zheng, Gang Pan, and Yilong Yin. Off- Off-policy proximal policy optimization. In 2023.
Proceedings of the AAAI Conference on Artificial Intellignnee volume 37, pp.
Youssef Mroueh. Reinforcement learning with verifiable rewards: Grpo's effective loss, dynamics, and success amplification. arXiv preprint arXiv:2503.066339 2025.
Youssef Mroueh, Nicolas Dupuis, Brian Belgodere, Apoorva Nitsure, Mattia Rigotti, Kristjan Gree- newald, Jiri Navratil, Jerret Ross, and Jesus Rios. Revisiting group relative policy optimization: Insights into on-policy and off-policy training. arXiv preprint arXiv:2505.22257, 2025.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35:
27730-27744, 2022.
Yun Qu, Qi Cheems Wang, Yixiu Mao, Vincent Tao Hu, and Xiangyang Ji. Can prompt diffi- culty be online predicted for accelerating rl finetuning of reasoning models? arXiv preprint arXiv:2507.0460 2025.
James Queeney, Yannis Paschalidis, and Christos G Cassandras. Generalized proximal policy op- timization with sample reuse. Advances in Neural Information Processing Systems, 34:1 1909- 11919, 2021.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-rl: A stable and generalizable rl-style large vision-language model. arXiv preprint arXiv:2504.0760
2025.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp. 1279-1297, 2025.
Marco Simoni, Aleksandar Fontana, Giulio Rossolini, and Andrea Saracino. Gtpo: Trajectory- based policy optimization in large language models, 2025. URL 2508. 03772.
Yifan Sun, Jingyan Shen, Yibin Wang, Tianyu Chen, Zhendong Wang, Mingyuan Zhou, and Huan Zhang. Improving data efficiency for IIm reinforcement fine-tuning through difficulty-targeted online data selection and rollout replay. arXiv preprint arXiv:2506.0522 053350660036,,... 2025.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with 1lms. arXiv preprint arXiv:2501.
2025.
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang- Learning to reason under off-policy guidance. arXiv preprint arXiv:2504. 101004.1944 14945, 2025.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvemment arXiv preprint 2rXiv:2409.1211 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 093888, 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu. Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv: 14476, 2025.
13

---

Published as a conference paper at ICLR 2026
Xiaojiang Zhang, Jinghui Wang. Zifei Cheng, Wenhao Zhuang, Zheng Lin, Minglei Zhang, Shaojie Wang, Yinghan Cui, Chao Wang. Junyi Peng, et al. Srpo: A cross-domain implementation of large-scale reinforcement learning Ilm. arXiv preprint 1.ti::2504.122860 2025.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang. Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 GLOSSARY OF TERMS AND NOTATIONS
Term
Definition
C1 02, C3
Thresholds for classifying historical samples by difficulty (group mean reward).
X2, X3
Subsets of training batch: fresh, re-evaluated difficult, and historical high-quality samples.
m
Re-evaluation frequency for historically difficult samples. Delay steps for updating the rollout policy.
G
Group size, number of responses generated per prompt during rollout. Replay buffer storing historical samples.
B
AL.
Estimated advantage for token in response i. Clipping parameter in PPO-style objectives. Coefficient for KL penalty in the objective function. Filter function for constructing BAPO's training batch. Kullback-Leible divergence, used to constrain policy deviation. Rollout policy for BAPO, which synchronizes to my every v steps. LLM policy parameterized by 6.
B
(xx)
DKL.
a
wo
Trof
Reference policy (e.g., initial pre-trained model).
p(0)
Importance sampling ratio: [a(y)>))
Tus(y/=)*
r(z, y)
Reward function, we to binary (0/1) based on correctness. Expected reward under policy π for input æ. We approximate this value using the mean of r(x,, y) corresponding to G responses y generated by the rollout policy a for each prompt Standard deviation of rewards under policy a for input x, with smooth-
(a)(())
(x))
ing s.
[(T(-1x))
Expected reward of policy for input x: (- [r(x; y)]-
(1Har(I) 0²)
A sampling method that assigns weights online rollouts based normal distribution centered at μ with standard deviation a,, used to filter samples by their group mean reward (a.r(x).
A.2 THEORETICAL ANALYSIS
Lemma A.1 Ceattorovicc-Ruuennsie duality of total variation distance). The Kantorovich- Rubinstein duality (variational representation) of the total variation distance is follows:
TV(m1, m2) = 2T
sup {Ez~m, [s(2)] - (0(3))) gEśL
(14)
where Gr. = (g: RR 11 L).
Theorem A.2 (Policy Improvement Lower Bound with Adaptive Training Batch), Assume re- wards are bounded: S r S 1. Let no, be the current policy, 01 To._, be the delayed rollout policy, 0:2 T0, be the current policy for re-evaluation, 023 = og be the buffer policy distribution, and ((x)) be the filtering function partitioning samples into X1, X2, and Xz.
Suppose C1, C22 03 E (0, 1) with o2 < <Ca. and the following TV distance constraints hold:
TO. TV(To, (112), (1x=) S 81 VI E Xi
(15)
TV(me, (11x), (=(()=))) < 83 Va X3 where 81, 6z > are sufficiently small such that the variance lower bounds remain positive.
(16)
Then, for the policy update objective in Equation 3, the expected policy improvement over filtered samples satisfies:
Ez-px (T=))((()((()) - [[To, (-1x)))] 2 Σ Li(To,
15

---

Published as a conference paper at ICLR 2026
where:
Li(TO, a4) = RREX, [L,, (TO(-+2)) - 2K, TV(TO(1Z)) a((+))) - 27V(Ta (-1z), (-(l))
with (T.(TT(-)2)) = (o) (J(TT(-1)) ((ar(+1x))) The constants are:
1 -
G-1 +E
G
K1
(17)
G-1 +E 6²
K2 1 (1-
(18)
(1
1 /min(c2(1-ca))eee c3)) +E
K3
(19)
min(c221 c2), ca c3)) +
Proof. We prove the bound by analyzing each filtered sample set separately, applying off-policy policy improvement bounds tailored to the reference distribution used in each region.
Step 1: Core inequality for off-policy samples. For any x such that [(x) = 1, we establish the fundamental inequality:
](T((11)) - [[To, (-1z)) 2 La, (To - 2K, (V(T((11))..(111))
(20)
2To, (+1z),.((-Iz:)
(21)
where K; (e (2) is constant that depends on the variance of rewards in each filtered subset.
First, we expand the advantage objective. By definition:
Lax (-m(-)=)) = E (-)x)
(a(aar))
Aa; (st, v)
(22)
a.(ulæ))
To(ylar) (2r,y) - (2))( a.(ylz) (a.ne())
= (y)o)(())
(23)
1
(a.r.e()) (J(zo(+1=) -(s(((1)))
(24)
Next, we establish the key algebraic identity relating La, (-1x)) to ((Fo(l)))) - (11x))
Lo, (TT(- - (J (T(T(())))) - (-1=)))
(25)
1- (a) (J(TO--1=)) -(as(+l))) + (TT, - ((a((-l=)))
(26)
Jagrr, (x))
Application of Sentorovvcc-Ruueenste duality: For bounded rewards with |||| = 1, the Sectorovich-Rubenstee duality Lemma A.1 provides:
(=oo(=))) ((s((-))) 2- (V)
(27)
[Ta, (-1z)) - ((a)(+ < TV (- az(())))
(28)
Since < r < 1, we have (a)ne (x) < 1, ensuring K; = 1-0m bounds yields the desired inequality.
2 0. Combining these
Step 2: Analysis for Xi (Filtered fresh samples). For x E X1, samples are generated by the delayed rollout policy an To.., and selected via Gaussian sampling with group-level accuracy (x) E ( 2 ....., G-1), excluding extremes (0, 1}.
Variance analysis on discrete set: For the variance function f(p) = p(1 p) over the discrete
set ( &,..., G-1 ), the minimum value occurs at the boundary points p = &
1 or G-1 both
C
yielding f(p) CT
Therefore:
G -
, (x) (x)
(29)
G
16

---

Published as a conference paper at ICLR 2026
Thus: (x) 2 V C + yielding:
1 -
G-1 G +E
K1
Step 3: Analysis for X2 (Re-evaluated difficult samples). For x 22 samples are generated by the current policy 0g = ng, through re-evaluation of historically difficult samples. The selection criterion ensures that historically difficult samples (Han.r(z) S (1) now achieve improved perfor- mance (c1 < ( 1) under the current policy.
Since these samples are directly generated by ne,, we have 02 Ta. and the constraint ( (x) < provides a natural lower bound, yielding:
aag, (x) = Haz (z) - (az,( > c1 c1)
(30)
Therefore: (2) > Va(11 (1) te, giving us:
K₂ 1- c)) (9(1-a) +E
Step 4: Analysis for X3 (Historical high-quality samples). For x Ns, samples are generated by historical buffer policies ag = ag with (x) E [c2,cs].
Since (x) Mas.r (z)) achieves its minimum at the endpoints of the interval (c2, cy]:
(a) (x) 2 min(e2(1 (1)) ea))
(31)
Therefore: Dagine(a) (x) 2 V /min(c2211 (g) e. yielding:
K3 1- m -1)
min(c2(1 c2), c3)) +
Step 5: Combining the results. Taking expectations over æ px and applying the indicator function decomposition:
Ex-ox (()))
(32)
E Rxsox (eex (J(TO(( ((ma
(33)
3
2
Exex, [L,, (To(-(2) 2K, TV(T)(11)) - (1))
(34)
as)
(35)
All constants K1, K2, K3 are finite, since denominators strictly positive by construction and numerators are bounded by under Cı, C2, c3 E (0, 1), completing the proof.
Proposition A.3. For binary reward tasks where r(x, v) (0, the contribution to the policy improvement lower bound is maximized when the expected group reward of the sample is = 0.5.
Proof. Recalling Theorem 3.2, the lower bound for policy improvement specific data distribu- tion involves the constant K, which scales the penalty for policy divergence. The tightness this bound is governed by the standard deviation of the rewards (x)
17

---

Published as a conference paper at ICLR 2026
Due to advantage standardization Á 1. the effective step size in the advantage estimation and consequently the gradient magnitude is proportional to the inverse of the standard deviation. How- ever, in the context of the lower bound analysis in Theorem 3.2, the stability constant K is defined as:
(()) 1 1-0(1)
(36)
(J))
where smaller K indicates tighter bound and thus larger guaranteed improvement step. For a binary reward function r E {0,1}, the reward distribution follows a Bernoulli distribution with parameter p(x)) B(rlz]. The variance is given by:
x² 4(1 1 11)
(37)
To find the y that maximizes variance, we take the derivative with respect to jc:
d (u x²) = -2p du
(38)
Setting the derivative to zero:
1 - 24 = / 0.5
(39)
Since the second derivative d²
2 < 0, this is global maximum.
du
At μ = 0.5, the variance is maximized (o² = 0.25, 0.5). This corresponds to the state of max- imum entropy, where the model is most "uncertain" about the outcome. Training on these samples provides the strongest gradient signal for distinguishing between correct and incorrect reasoning paths, effectively maximizing the information gain per step. Conversely, as μ 0 or μ 1, a + O, causing the advantage estimates to numerical instability or the gradient signal to vanish. Therefore, selecting samples with μ = 0.5 theoretically offers the most efficient learning signal and the most favorable stability bound.
A.3 ONLINE FILTER MECHANISM ANALYSIS
To investigate the impact of fresh sample selection on training stability and convergence, conduct an ablation study using Qwen3 8B with a 4K response length limit. We compare three distinct filtering strategies for the online component (X²):
I اال This effec- tively removes only the zero-advantage samples (all-correct or all-incorrect) that contribute minimal gradients.
Mode 1 (Range Filter): It retains samples with group mean rewards E
Mode 2 (Gaussian Filter): A difficulty-weightel strategy that prioritizes samples with high vari- ance (accuracy near 0.5) using a Gaussian distribution, thereby reducing the proportion of extremely easy or hard samples.
Mode 3 (Uniform Filter): A baseline that randomly selects 60% of the fresh samples regardless of their quality. This ratio was chosen match the approximate data retention rates of Mode 1 and Mode 2 (approximately 40%-60%) for a fair comparison of data volume.
The Value of Quality over Randomness. As illustrated in Figure 10, the uniform filter mechanism exhibits severe instability, characterized by exploding gradient norms and complete collapse in performance after 150 steps. Since this strategy blindly includes all-wrong samples (where μ = O), the model is forced to update based on low-quality, zero-advantage signals. Suppressing the token probabilities of incorrect responses without a corresponding positive signal introduces significant noise and uncertainty, ultimately destabilizing the policy. This failure highlights that the quality of the training batch, particularly the exclusion of aero-advantage noise, is crucial.
Convergence Speed and Final Performance. The Gaussian filter demonstrates faster convergence in the early stages. By focusing heavily on samples with the highest variance (accuracy ss 0.5), provides the steepest learning signal initially. However, its final convergence performance is lower
18

---

Published as a conference paper at ICLR 2026
Mean@8
Grad Norm
0.3
0.8
Mean@8 (T)
Grad Norm
0.2
w/ range filter
0.1
w/ gaussian filter
0.2
w/ uniform filter
50
100
150
200
250
300
50
100
150
200
250
300
Training Steps
Training Steps
Figure 10: Ablation on Online Filtering Strategies. Comparison of Range Filter, Gaussian Filter, and Uniform Filter training stability (Grad Norm) and performance (Mean @8). The star symbol indicates the best checkpoint for BAPO.
than that of the range filter. hypothesize that the Gaussian filter restricts sample diversity by ag- gressively filtering out samples that slightly easier or harder but still informative. In contrast, the range filter retains a broader spectrum valid samples. While learns slightly slower initially, maintains rich distribution of training data, preventing premature plateauing and ultimately achiev- ing the highest asymptotic performance.
A.4 TRAINING DYNAMICS AND TEST CURVES
As illustrated in Figure 11 and Figure 12, present detailed training dynamics and test curves for the Planning and Vision Geometry tasks. The results indicate that both BAPO and DAPO consistently outperform GRPO in terms of training rewards. Interestingly, BAPO exhibits higher entropy. reflecting better exploration capability compared to other algorithms, which also results in longer response lengths.
Planing Task Train Rewards
Planing Task Actor Entropy
Planing Task
Response Length
0.7
0.25
1400
Rewards
0.6
Acter Emtrosy
0.20
Responsee Length
1200
0.5
0.4
0.15
0.3
BAPO
800
0.2
0.1
DAPO
GRPO
0.05
600
0.0
20
40
6C
80
100
20
40
100
20
80
100
Training Steps
Training Steps
Training Steps
Vision Geometry Task Train Rewards
Vision Geometry Task Actor Entrogy
Vision Geometry Task Response Length
0.6
1.2
600
0.5
Acter Entropy
1.0
0.8
Resparsse Length
500
0.4
0.6
450
0.3
BAРO
400
DAPO
350
GRPO
300
20
00
100
120 140
160
20
100 Training Steps
00
160
60
100 80
160
Training Steps
Training Steps
Figure 11: Training Dynamics during BAPO, GRPO, and DAPO post- cost-training, including training rewards, training entropy, and response lengths.
19

---

Published as a conference paper at ICLR 2026
AMC Set
CD-4 Test Set
Geo 3K Test Set
0.74
BAPO
0.4
BAPO GRPO
BAPD GRPO
0.72
Teat Accuracy [T]
0.40
Test Accuracy
0.70
0.3
0.35
0.68
0.2
0.06
0.1
0.64
0.0
0.20
100
200
300
400
20
60
100
120
50
100
150
200
250
Training Steps
Training Steps
Training Steps
Figure 12: Test Curves of Group Accuracy Changes for mathematics, planning, and geometry tasks among AMC, CD-4 test set, and Geo-3K test set, respectively.
A.5 COMPUTATION ANALYSIS
From Table 2, we observe that BAPO's computational overhead correlates with the number of sam- ples requiring re-evaluation and the actual training batch size. For the Planning task, BAPO (w/o X2) achieves the fastest training time by eliminating bad case re-evaluation, but this comes at the cost of reduced performance. For the Mathematics task. the high number of bad cases (as shown by the 0/8 accuracy samples in Figure 7) means that under our re-evaluation frequency setting of m = 5, infer- ence time exceeds that of GRPO. However, this additional time investment proves valuable, yielding better bad-case handling rates and overall test performance, shown in Figure 4 and Table ??. We plan to explore lower re-evaluation frequencies to assess the performance trade-offs.
BAPO (c2 0.375, C3 0.5) runs significantly faster than BAPO (c2 0, 0.25) due to the
larger historical data volume the latter configuration. This causes BAPO (02 0, = 0.25) to maintain larger effective batch size than BAPO (c2 0.375, C3 0.5). Training logs also confirm this observation: BAPO (c2 = 0, c3 = 0.25) consistently utilizes 100% of the configured batch size (equivalent to on-policy methods' batch size), while BAPO (c2 0.375, c3 0.5) operates at approximately 70% capacity.
Table 2: Computational Overhead Analysis. "Batch size" (a, b) represents the sample batch size and train mini batch size b. "Time" is measured in total training time (d=days, h=hours, m=minutes) on A100 GPUs.
Tasks
Methods
Batch Size
Num Epoch
Time
GRPO
(256, 64)
1d 16h 58m
Mathematics
DAPO
(256, 64)
2d 15h 30m
BAPO
(256, 64)
3
Id 22h 37m
GRPO
(256, 64)
3
3h 47m
DAPO
(256, 64)
35m
BAPO
(256, 64)
3
3h 23m
Planning
BAPO (w/o X²2)
(256, 64)
2h 38m
BAPO (w/o Xx)
(256, 64)
3
3h 4m
BAPO (C2 0, - 0.25)
(256, 64)
54m
BAPO (Cg 0.375, 0.5)
(256, 64)
3
3h 4m
GRPO
(256, 64)
30
7h 55m
DAPO
(256, 64)
30
12h 19m
Visual Geometry
BAPO
(256, 64)
30
5h 50m
BAPO (w/o X2)
(256, 64)
30
3h 42m
BAPO (w/o X3))
(256, 64)
30
4h 31m
A.6 VISUALIZATION
We present additional visualization details, including the sample accuracy tracking for the Count- down and Geometry3K datasets, shown in Figure 13. Meanwhile, visualize the source of
20

---

Published as a conference paper at ICLR 2026
(Step
GRPD 117)
117)
BB2
468
396
Number of Promotts
628
205
202
see
124
70
38
30
34
Accuracy Bins
Accuracy Bins
Accuracy Bins
Base (Step 0)
GRPO (Step 240)
BAPO (Step 240)
20e
136
Number of Pronugts
124
35
22
14
40
22
32
09
09
Accuracy Blnss
Accuracy Bins
Accuracy Bins
Figure 13: Tracking changes in the Number of Different Accuracy Bins on the Countdown (up- per) and Geometry3K training sets (lower) for the baseline model, GRPO, and BAPO method. Special attention is paid to the change in the number of bad samples (red bars) that the base model fails to handle.
100
DeepScalert Train
100
CD-34 54 Train
Train
Proporsion (%)
Dote Source Proporion (%))
80
30
x
X1
X2
100
200
300
400
08
20
60
80
100
1.00
150
200
250
300
Raining
Taining Staps
Taining
Figure 14: Batch Distribution Visualization of X1, X2, X3 for Mathematics, Planning, and Visual Geometry Tasks (left to right) during BAPO's training.
21

---

Published as a conference paper at ICLR 2026
Trans Stap
Tan 35p
Ske 350
Tou Ste-AT)
Cuannll
Siny
Figure 15: Accuracy Migration Matrix Analysis. We track fixed subset of 1,000 randomly selected prompts from the training set and visualize their movement between accuracy bins (0/8 to 8/8) at Steps 0, 150, 300, and 471 (the last step). The y-axis represents the initial accuracy bin Step O, while the x-axis represents the current accuracy bin. The scarcity of samples in the lower triangle demonstrates that performance degradation rare.
samples in each training batch and their respective proportions during the training process, as illus- trated in Figure 12. It can be observed that approximately 40-60% of the actual training samples for BAPO come from online samples Xi, while the remaining samples are derived from X₂ or Az.
Stability of Historical High-Quality Samples. potential concern regarding the reuse of historical high-quality samples (X's in Eq. 5) is the assumption of policy samples that were high-quality under past policy remain valid for the current policy. To address this, we visualize the evolution of sample difficulty in Figure 15 by tracking the accuracy migration of training subset.
whether
The heatmaps in Figure 15 reveal a distinct pattern: the mass is concentrated along the diagonal (performance maintenance) and the upper triangle (performance improvement). Crucially, the pro- portion of samples exhibiting significant performance degradation (migrating to the lower triangle) negligible. For example, samples that initially achieved 8/8 accuracy predominantly remain in the high-accuracy bins throughout the training process, with minimal regression to lower bins. This empirical evidence demonstrates that high-quality reasoning paths learned by RL are robust and re- sistant to forgetting. Consequently, historical high-quality samples stored in the buffer likely remain high-quality under the current policy, validating the consistency of the N' data source.
A.7 HYPERPARAMETER SETTING Hyperparmeters
The major hyperparameter choices are shown in Table 3.
Table 3: Hyperparameter Configuration for BAPO Framework on Mathematics Task. For plan- ning and visual geometry tasks, some parameters differ slightly; specific configuration scripts provided in our code repository.
Parameter
Value
Parameter
Value
Parameter
Value
Rollout Configuration
Top-p Group size (G)
Top-k
-1
Temperature
8
Max prompt length
2048 256
Max response length
8192
Rollout workers
8
Sample batch size
Seed
42
Training Configuration
Learning rate
lc-6
Train mini batch size
GAE lambda
1.0
Training epochs
3
KL coefficient (3)
0.001
Entropy coefficient
0.001
Off-policy Configuration
Cj threshold
1/8
C2 range
11/8,
ca range
2/8
Buffer size (B)
256
Rollout delay (v)
5
Re-evaluation freq (m)
5
Gaussian std (x)
0.2
Gaussian mean (μ))
0.5
Max re-evaluate prompts
128
22

---

Published as a conference paper at ICLR 2026
Reward Function To evaluate the impact of method, we adopt simple reward function as below. All training experiments employ the same reward function.
1, y is correct r(z, 4)
otherwise
Datasets and Benchmarks benchmarks categorized into mathematical, planning and vision geometry reasoning benchmarks as described in Table 4.
To evaluate the models above, use three training datasets and eight
Table 4: Datasets and Benchmarks used in this study.
Dataset
#Train
#Test
Task Type
Domain
License
Source
Training Datasets
DEEPSCALER- PREVIEW
P.SV-RVEEW 40,000
Math reasoning
Mathematics
Apache 2.0
Link Link
COUNTDOWX-TASKS--IH
49.000
Logic reasoning
Planning Visual Gcometry
Apache 2.0
GEOMETRY3K
2.100
Visual reasoning
Apache 2.0
Link
Test Benchmarks
AIME24
30
Math competition
Mathematics
MIT Apache 2.0
Link
AMC
83
Math competition
Mathematics
Link
MATH500
500
Math reasoning
Mathematics
Link
MINERVA
272
Math reasoning
Mathematics
Apache 2.0
Link
OLYMPIAD
674
Math competition
Mathematics
Apache 2.0
Link
COUNTDOWN- TASKS-
3TO4
200*
Logic reasoning
Planning
Apache 2.0
Link
SHINTDOWN-TASKS:
200'
Logic reasoning
Planning Apache 2.0
Link
GEOMETRY3K
901
Visual reasoning
Visual Geometry
Apache 2.0
Link
"We only use random subset of this benchmazk faster ablation stadies.
A.8 ALGORITHM
Algorithm presents the proposed BAPO, which can be seamlessly integrated with any GRPO-like
RLVR algorithm.
Algorithm Batch Adaptation Policy Optimization (BAPO)
Require: Policy Ta,, buffer B = 0, thresholds C1, C3. delay steps v, re-evaluate frequency m
for 1 to T do
2: // Off-policy Rollout Phase
3: if t mod u then
4: Synchronize rollout policy's parameter with trainer: a
5: end if
6:
Using rollout policy to generate G responses {u, [u for each question
7:
Compute log probabilities a(yla (z) and rewards for constructing the online batch Xon
8:
Store samples into buffer Bhad + { (x, y, ox x), E Xon Harr (x) S ci
9:
Store samples into buffer Bughh < {xx, y, a ((y(x),)) E Xun c2 (x, < cs)
10:
// Off- policy Training Phase
X1 online filter on Xoo with (x) ( (اl) (Filtered Fresh Samples)
12:
A/ +
if mod 0 then
14:
Re-evaluate Bond with ne, to get X2 using Equation 6 (Re-evaluated Difficult Samples)
end if
16:
X3s + Sample from ((x, v)
Bligh (x) E [C2, cal} (Historical High-quality Samples)
17:
Final batch + x, U U3
18:
Compute advantages and update critic/actor with final_batch
19:
Add D, to buffer B
20:
end for
23

---

Published as a conference paper at ICLR 2026
A.9 GENERALIZATION ANALYSIS
To demonstrate the algorithmic generalizability of our framework, we extended the Batch Adap- tation paradigm Proximal Policy Optimization (PPO), denoted BA-PPO. In this experiment, both the Actor and Critic networks were initialized with the Qwen3-4B backbone and trained on the DeepScaleR dataset with maximum response length 4K tokens. We maintained consistency with the foundational BAPO configuration by applying standard aero-advantage filtering for Ai (removing only all-correct and all-wrong groups), utilizing the initial BAPO values for thresholds C1 C2 C3 and setting the buffer size to 64.
AIME Mean@8
0.35
BA-PPO PPO
0.30
0.25
Reward (T)
0.20
0.15
0.10
0.05
50
100
250
300
Training Steps
Figure 16: Generalization to Actor-Critic Algorithms (BA-PPO). Performance comparison be- tween standard PPO (orange triangles) and BA-PPO (purple circles) the AIME 2024 benchmark using Qwen3-4B. The star (+) marks the peak performance BA-PPO (0.325).
As illustrated in Figure 16, BA-PPO achieved remarkable performance gain of +5.5 the AIME 2024 benchmark compared to the standard PPO baseline. This result further confirms that the core principle of dynamic batch construction is effective only for GRPO but also functions robust, algorithm-agnostti enhancement for actor-critic methods.
24


Reviews:
[{"review_id": "sOwdSaqN76", "reviewer": "Reviewer_g1rK", "summary": "The paper introduces Batch Adaptation Policy Optimization (BAPO), an off-policy reinforcement learning fine-tuning framework for LLM reasoning tasks. It incorporates two key off-policy designs into standard RL post-training framework: (1) during rollout process, it delays the update of the inference server to mitigate rollout policy instability that may arise from rapid distribution shifts (a technique from prior work), and (2) during training process, BAPO dynamicaly constructs training batches that combine online generated samples, re-evaluated difficult samples, and historical high-quality samples. Experiments on mathematical reasoning benchmarks demonstrate that BAPO achieves higher accuracy compared to baseline methods.", "questions": "- In the computation analysis results (Line 457), it is counter-intuitive that BAPO is not slower than GRPO, which requires clarification. Given identical training batch sizes, the generation cost before online rollout filtering should be similar to that of GRPO (assuming the pre-filtering sample size is the same, which is not described in the paper). Moreover, constructing $|\\mathcal{X}_2|$ requires periodic large-scale computation of the current policy probabilities over all historically difficult samples. And during training, off-policy samples also require additional probability computation under the current policy.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The ablation study is comprehensive, including ablations on $\\mathcal{X}_2, \\mathcal{X}_3$, delay steps (v), re-rollout frequency (m), and difficulty thresholds $c_1,c_2,c_3$.\n- Experiments span multiple reasoning domains, demonstrating versatility.\n- The paper structure is clear.", "weaknesses": "- Overall, the method appears to be a collection of empirical tricks rather than a broadly generalizable or methodologically novel approach. Specifically, The method introduces a large number of hyperparameters and custom design choices—such as $c_1, c_2, c_3$,  online sample mean ($\\mu$) and standard deviation ($\\sigma$), re-rollout frequency (m), linear mapping function, rollout delay steps (v), and the proportion of different sample types, which greatly limit its plug-and-play usability.\n- The overall method design is highly heuristic, with limited justification on its rational. For example, in Eq. (5), the use of “historical high-quality samples” assumes that samples that were once high-quality remain so under the current policy. Although the method designs a linear mapping function to determine $(c_2, c_3)$ based on the buffer’s average performance, this approach is coarse and lacks solid reasoning to ensure that the selected samples are indeed high-quality under the updated policy.\n- In the $\\mathcal{X}_2$ filtering phase, the method requires periodically re-generating responses for all historically difficult samples using the current policy, leading to substantial computational overhead—especially when the dataset contains numerous difficult samples (a common scenerio). Similarly, in the $\\mathcal{X}_1$ filtering phase, not all online rollouts are utilized, resulting in additional inefficiency.\n- The paper lacks details on the quantities of $|\\mathcal{X}_1|, |\\mathcal{X}_2|, |\\mathcal{X}_3|$, which are crucial. Specifically, for $\\mathcal{X}_1$, how many online rollouts are retained out of the total generated? For $\\mathcal{X}_2$, since the number of such samples is varies considerably, how are the sample sizes distributed and balanced across the three sample types?\n- The theoretical proofs rely on assumptions of small total variation distance, but these assumptions are not sufficiently validated. In particular, assuming $\\mathrm{TV}(\\pi_{\\theta_t}(\\cdot | x), \\alpha_B(\\cdot | x)) \\le \\delta_3$ with sufficiently small $\\delta_3$ could severely limit the improvement of $\\pi_{\\theta_t}$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k8sf5tRNgI", "reviewer": "Reviewer_SdcC", "summary": "The paper proposes a general sample optimization algorithm that ingeniously integrates both on-policy and off-policy data to train the policy model in a targeted manner, adaptively selecting samples based on their learning difficulty assessed from rollouts.", "questions": "- BAPO is specifically designed for GRPO-like group-based advantage estimation, where homogeneous rewards within a group can lead to zero gradients. It remains unclear whether this framework generalizes to other on-policy RL algorithms such as PPO or Reinforce, which does not suffer from the same gradient vanishing issues.\n- Does BAPO incorporate any mechanisms to encourage exploration beyond the existing experience pool, or is performance ultimately bounded by the policy's rollout coverage?\n- BAPO w/o X3 shows a substantial performance gap compared to full BAPO. Since X3 only provides historical high-quality samples that have already been used, could you explain the source of this large gap? Is this primarily due to the mismatch in effective batch sizes (as you configured BAPO w/o X3 to retain all fresh samples X1), or does the historical data contribute beyond simply maintaining batch size consistency?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper clearly identifies the key limitations of existing on-policy RLVR methods (experience waste and reward homogeneity) and proposes a well-motivated off-policy framework with intuitive design principles that align with RL fundamentals.\n- The paper provides rigorous theoretical analysis (Theorem 3.2) proving that the adaptive batch construction mechanism maintains a lower bound guarantee for policy improvement, ensuring training stability while leveraging off-policy data.\n- The evaluation across diverse reasoning tasks and model backbones demonstrates consistent improvements over baselines, with thorough ablation studies confirming the effectiveness of the proposed components.", "weaknesses": "- While BAPO effectively improves learning efficiency through adaptive sample selection, it primarily reorganizes existing experiences rather than fundamentally expanding the model's exploration space. This may limit its ability to solve problems that consistently fail under the current policy distribution.\n- The paper primarily compares rollout counts, but lacks detailed analysis of the actual computational overhead, particularly the costs of forward passes (e.g., computing log probabilities for importance sampling ratios) and backward passes during training. Since BAPO reuses historical samples that do not require new rollouts but still incur training costs, a comprehensive breakdown of these components would better clarify the true efficiency gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oplgJAW739", "reviewer": "Reviewer_Upr9", "summary": "This paper proposes a novel off-policy reinforcement learning framework called BAPO, which effectively addresses the issues of experience waste and reward homogeneity in traditional on-policy methods through an adaptive batch construction strategy that dynamically re-evaluates historically difficult samples and reuses high-quality ones. The paper provides a theoretical lower-bound guarantee and demonstrates outstanding performance and data efficiency across a series of mathematical, planning, and visual geometry reasoning tasks. Overall, the paper features a well-structured organization, well-justified motivations, and comprehensive experiments. However, several key design aspects in the methodology section are insufficiently explained, which affects the reproducibility and persuasiveness of the work.", "questions": "Please refer to the “Weakness” section for related questions.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1.\tThis paper proposes a method that systematically integrates the concept of off-policy reinforcement learning into the reinforcement learning with verifiable rewards for large language models. The method establishes a learning framework different from the traditional on-policy training paradigm by introducing experience replay and importance sampling mechanisms.\n\n2.\tThe core innovation of the method lies in the design of a dynamic re-evaluation mechanism, which can reassess historical data according to the model's current capability level. This mechanism transforms previously underutilized training samples into valuable training data, thereby improving data efficiency.\n\n3.\tThe adaptive batch construction strategy proposed in this paper manages the difficulty distribution of training data in a principled manner. By dynamically adjusting the proportion of samples of different difficulties, the method effectively avoids the problem of reward homogeneity that may occur during training.\n\n4. Experiments conducted in the three major reasoning domains of mathematics, planning, and visual geometry, based on models of different scales, show that the method consistently outperforms a series of representative on-policy and off-policy baselines in comprehensive benchmark tests, empirically demonstrating its effectiveness.", "weaknesses": "1. **On the necessity of the Gaussian sampling in the $\\mathcal{X}_1$ module.** The paper does not validate the performance impact of removing Gaussian sampling through ablation experiments, nor does it compare other sampling strategies (such as uniform sampling or difficulty-based sampling). Readers cannot determine whether this module is indispensable or merely a redundant design that increases methodological complexity. Therefore, it is recommended to supplement ablation experiments on Gaussian sampling and clarify the advantages of the Gaussian sampling strategy.\n\n2. **Insufficient explanation of the $c_1/c_2/c_3$ threshold settings.** The values of $c_1/c_2/c_3$ in the paper lack adequate theoretical or experimental justification. Thresholds such as $c_1=0.125$, $c_2 \\in [0,0.375]$, and $c_3 \\in [0.25,0.5]$ appear arbitrary, with no explanation of their design inspiration or statistical basis. Moreover, although it is mentioned that $c_2$ and $c_3$ are based on \"linear mapping,\" the specific mapping formula is not provided, making it impossible for readers to reproduce the adaptive process.\n\n3. **Insufficient summarization of the principles behind the $c_1/c_2/c_3$ threshold values.** While the existing ablation experiments effectively demonstrate that \"dynamic adjustment is useful,\" which is highly important, readers cannot derive a clear and transferable parameter design paradigm from them. The current ablation experiments resemble \"effect validation\" rather than \"exploration of principles.\"\n\n4. **Suggestion on terminology standardization and conceptual clarity.** The term \"RFT\" in the abstract is inconsistent with \"RLVR\" in the main text. Given that the method heavily relies on verifiable rewards, unifying the terminology to \"RLVR\" would enhance conceptual clarity and professionalism.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZJPayLOGrr", "reviewer": "Reviewer_x1w9", "summary": "This paper introduces Batch Adaptation Policy Optimization (BAPO), an off-policy reinforcement learning framework to improve the data efficiency of fine-tuning large language models on reasoning tasks. The core problem BAPO addresses is the failure of on-policy methods to learn effectively from difficult samples. BAPO constructs training batches by combining three data sources. It filters fresh rollouts to focus on moderately difficult samples. It also periodically re-evaluates historically difficult samples with the current policy to find newly solved ones. Finally, it reuses high-quality historical samples to ensure batch stability. Experiments show that BAPO improves performance over existing baselines.", "questions": "1. Regarding the adaptation of thresholds c2 and c3 mentioned in lines 248 to 251, could the authors provide the exact form of the linear mapping function used? How sensitive is the model's final performance to the specific design of this function?\n\n2. Could the authors elaborate more on the novelty of BAPO in comparison to established methods? Specifically, what are the key distinctions in how BAPO re-evaluates and samples difficult historical data that make it particularly suitable for LLM reasoning tasks?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed batch construction method is well-motivated. Instead of simply mixing old and new data, it treats historical samples differently based on their difficulty. Re-evaluating hard examples and reusing successful ones is a more nuanced approach than standard experience replay and directly targets the stated problem of learning from difficult instances.\n\n2. The evaluation in the paper covers three distinct reasoning domains (mathematics, planning, and visual geometry) using several modern LLM backbones. The comparison against a range of both on-policy and off-policy baselines is thorough, and the inclusion of detailed ablation studies strengthens the authors' claims about their design choices.\n\n3. The work provides a solid theoretical foundation for the proposed method. The inclusion of a policy improvement lower bound in Theorem 3.2 offers a formal guarantee of training stability under certain conditions. This theoretical analysis elevates the work beyond a purely empirical contribution and builds confidence in the method's robustness.", "weaknesses": "1. The novelty of the core components could be overstated. The individual ideas of using a replay buffer, re-evaluating samples, and filtering data based on difficulty have precedents in the broader reinforcement learning literature, such as Prioritized Experience Replay. The main contribution lies in the specific combination of these techniques for LLM reasoning with verifiable rewards. The authors should further explain the novelty of this paper. \n\n2. The mechanism for adapting the high-quality sample thresholds c2 and c3 is not clearly defined. The paper states that a linear mapping function is used based on the buffer's average performance, but the details of this function are omitted. This is a critical hyperparameter setting that could significantly impact performance, and its heuristic nature is not fully explored or justified with an ablation study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)