BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping
Authors: Zhiheng Xi
Xin Guo
Yang Nan
Enyu Zhou
Junrui Shen
Wenxiang Chen
Jiaqi Liu
Jixuan Huang
Xun Deng
Zhihao Zhang
Honglin Guo
Zhikai Lei
Miao Zheng
Guoteng Wang
Peng Sun
Rui Zheng
Hang Yan
Tao Gui
Qi Zhang
Xuanjing Huang
Venue: ICLR 2026 Poster (2026)
Keywords: RLVR
LLM Reasoning
Primary area: foundation or frontier models, including LLMs
TL;DR: This paper reveals the imbalanced optimization and the Entropy-CLip Rule in off-policy RL for LLMs, and proposes BAlanced Policy Optimization with Adaptive Clipping (BAPO) to stabilize RL optimization.

Abstract: Reinforcement learning (RL) has recently become the core paradigm for aligning and strengthening large language models (LLMs). Yet, applying RL in off-policy settings—where stale data from past policies are used for training—improves sample efficiency, but remains challenging: policy entropy declines sharply, optimization often becomes unstable and may even collapse. Through theoretical and empirical analysis, we identify two key insights: (i) an imbalance in optimization, where negative-advantage samples dominate the policy gradient, suppressing useful behaviors and risking gradient explosions; and (ii) the derived Entropy-Clip Rule, which reveals that the fixed clipping mechanism in PPO-like objectives systematically blocks entropy-increasing updates, thereby driving the policy toward over-exploitation at the expense of exploration. Building on these insights, we propose BAlanced Policy Optimization with Adaptive Clipping (BAPO), a simple yet effective method that dynamically adjusts clipping bounds to adaptively re-balance positive and negative contributions, preserve entropy, and stabilize RL optimization. Across diverse off-policy scenarios—including sample replay and partial rollout—BAPO achieves fast, stable, and data-efficient training. On AIME 2024 and AIME 2025 benchmarks, our 7B BAPO model surpasses open-source counterparts such as SkyWork-OR1-7B, while our 32B BAPO model not only achieves state-of-the-art results among models of the same scale but also outperforms leading proprietary systems like o3-mini and Gemini-2.5-Flash-Thinking.

Published as a conference paper at ICLR 2026
BАPО: STABILIZING OFF-POLICY REINFORCEMENT LEARNING FOR LLMS VIA BALANCED POLICY OPTIMIZATION WITH ADAPTIVE CLIPPING
Zhiheng Xi-xt, Xin Guo-*, Yang Nan', Enyu Junu'u
Junrui Shen',
Wenxiang Chen', Jiaqi Liu', Jixuan Huang', Zhihao Zhang', Honglin Guo', Xun Deng², Zhikai Lei?, Miao Zheng,, Guoteng Wang", Shuo Zhang", Peng Sun?, Rui Zheng', Hang Yan², Tao 1,3 Qi Zhang' .Xuanjing Huang'
Fudan University Shanghaai Qiji Zhifeng Co., Ltd. SShnngaii Innovation Institute zhxi22 @m. fudan. edu.en, ftqui, q2 fudan. edu. cn
ABSTRACT
Reinforcement learning (RL) has recently become the core paradigm for aligning and strengthening large language models (LLMs). Yet, applying RL in off-policy settings- where stale data from past policies used for training-improves sam- ple efficiency, but remains challenging; policy entropy declines sharply, optimiza- often becomes unstable and may even collapse. Through thcoretical and em- pirical analysis, we identify two key insights: (i) imbalance in optimization, where negative-advantage samples dominate the policy gradient, suppressing use- ful behaviors and risking gradient explosions; and (ii) the derived Entropy-Clip Rule, which reveals that the fixed clipping mechanism PPO-like objectives sys- tematically blocks entropy- entropy-increasine increasing updates, thereby driving the policy toward over-exploitation at the expense of exploration. Building on these insights, we propose BAlanced Policy Optimization with Adaptive Clipping (BAPO), sim- ple yet effective method that dynamically adjusts clipping bounds to adaptively re-balance positive and negative contributions, preserve entropy, and stabilize RI. optimization. Across diverse off--y partial AilouttBAAPO achieves fast, stable, and data-cfficient training. On AIME 2024 and AIME 2025 benchmarks, our 7B BAPO model surpasses open-source counterparts such as Sky Work-ORI-77 while our 32B BAPO model not only achieves state-of-thed results among models of the same scale but also outper- forms leading proprietary systems like o3-mini and Gemini-2. Momnini-2.5-Pe 2.5-FFash- maashh-Thiiiing...
cecturoos-includie sample replay and
100
Despöeck-RI
ORI--1
Our--BAPO-318
Light-R1-7BBB
Qwee3-325
s3-misi-metlia
$2.2
19.4
10.3
Accuracy Percentile (%)
76
78.
72.9
73.3
70-
$1.9
60
54.5
50
442
40
34.4
AIME 2024
AIME 2025
AIME 2024
AIME 2025
Figure 1: Performance of BAlanced Policy Optimization with Adaptive Clipping (BAPO).
Equal contribution. contribution..Coreppoooo 'Corresponding authors.

---

Published as a conference paper at ICLR 2026
Training Reward
Entropy
Clip Fraction
Grad Norm
0,4
0.04
0.2
0.3
100
200
100
200
0.00
100
200
1.00
200
Step
Step
Step
Step
Staleness-O
Staleness-2X
Stalemess-4x
Staleness-BB
Figure 2: Preliminary results with different data staleness. As the staleness increases, the model suffers from unstable optimization, decreasing entropy, and even a sudden collapse in training.
INTRODUCTION
Reinforcement learning (RL) has become pivotal paradigm for optimizing large language models (LLMs) (Zhang al. 2025). delivering significant improvements in complex tasks such as reasoning Jaech et al.] [2244] Guo et al. 2025a). coding (Anthropic (025), and agentic decision-making (Bai et al. Xi et al., 2025). Among RI methods, off-policy -here the rollout policy (behavior policy) differs from the training policy (target policy)-emergee emerges particularly promising (Roux cct al. 2025b; Arnal et al.] 2025). offers high sample efficiency and tolerance to data staleness, making it well-suited for extremely long-horie hong-horizon and challenging scenarios, while also aligning more naturally with features in modern Al infrastructures such as partial rollout (Team et al. 2025
Fu et al. 2025).
However, applying off-policy RL to LLMs introduces substantial challenges (Yu et al. 2025 Arnal
al. 2025). As shown in Figure 2 increasing data staleness leads to unstable optimization, ploding gradient and even collapse. Meanwhile, policy entropy declines sharply, reflecting reduced exploratory capacity and bias toward over-exploitation. By contrast, on-policy training- rollout and target policies coincide rrmmain stable across metrics, consistent with prior studies
Tang et al. 2024 Roux al. 2025b Arnal al. 2025).
To understand the instability of off- training, we conduct a comprehensive theoretical and em- pirical analysis to reveal two key insights. We first demonstrate an imbalance in optimization: pol- icy updates are often dominated by regative-advantage samples, producing excessive penalization signals that suppress even neutral or correct actions and may cause gradient explosions (Gülçehree) al. 2023). We then derive and empirically validate the Entropy-Clip Rule in the widely-used PPO Schulman et al., (017) and GRPO (Shao et al.) 2024)) showing that the clipping mecha- nism PPO-like objectives blocks many wwwprobabilitty positive tokens while over-penalizing low-probability negatives. This systematically excludes antropy-increasing updates, sharpens the output distribution, and drives policies toward over-exploitation at the cost of exploration.
Based on these insights, we propose BAlanced Policy Optimization with Adaptive Clipping (BAPO), new method for stable and effective off-policy RI. BAPO dynamically adjusts the clip- ping bounds to re-balance positive and negative contributions for each update step, incorporate low- probability positives while filtering excessive negatives, and preserve policy achrevig better balance between exploration and exploitation. An overview of our approach is illustrated on the right side Figure
Experiments across diverse off policy scenarios scenarios-including sample replay, partial rollout, and vary- ing degrees of staleness on base models such PrepSeek-RI-Distilll L--Distill-Qwen--- (Guo et al. (025a) and OctoThinker- Wwwihnnn---laa...... 2---B.L--- Zero (Wang et al.) 2025b) show that BAPO consistently yiclds significant improvements. Our 7B model achieves scores of 70.8 on AIME24 and 62.5 on AIME25, surpassing open-source counterparts such as Sky OR1-B 1-7B (He et al. 2025). Moreover, our 32B model reaches 87.1 on AIME24 and 80.0 on AIME25, outperforming both comparably scaled open-source models like Qwen3-32B (Yang al. 2025a) and leading proprictary systems including (OpenAl 2025) and Gemini-2.5 Flas5- Thinking (Comanici et al. 2025).
Our contributions are summarized follows:
We identify and analyze two key insights behind instability in off-policy RL for LLMs: the im- balanced optimization and the Entropy-Clip Rule. ($3)
2

---

Published as a conference paper at ICLR 2026
GRPO: Sharpen distribution Entropy (t) 1
BAPO: Smooth distribution Entropy (t)
Probability
Probability
Negative samples]
[Positive samples]
[Negative samples]
[Positive samples]
token; (optimized)
token; (clipped)
token, (uptimized)
token, (clipped)
Clow
Chiph
Cuee
11
Gugs
b+
Before Optimization
D. After GRPO
E. After BAPO
Figure 3: An illustration of proposed BAPO. (Left) Baseline methods like GRPO use symmet- ric fixed clipping bounds, reinforeing high-pne sroobbbbabylityy positive tokens while penalizing excessive low-probability negatives, leading to sharp distributions and entropy collapse. (Right) BAPO dy- namically adjusts the clipping bounds Clow and (high based on the loss contributions from positive tokens. It excludes overly negative tokens A maintain smoother distribution and incorporates previously clipped positive tokens A to preserve entropy balance.
We propose BAPO, a new RL algorithm that dynamically adjusts clipping bounds to balance positive and negative signals, preserving entropy for exploration, and stabilizing training. ($
We validate BAPO across multiple backbones, model scales, and off-policy settings, showing that it achieves stable optimization and competitive results with proprictary systems ($5)
2
PRELIMINARIES
Policy gradient. In the field of LLM RL (Trng al. 2024 Jaech al. 2024 Xi et al. 2024b
policy gradient- based (PG) algorithms (Williams) 1992) are widely used. Specifically. given an input prompt æ, LLM my sequentially generates token response y (y (,),..,,7))
To (b((x = I1 To (Vi æ y-r))
(1)
Given training dataset D = {x1 .... ÆND and reward function R, the RL objective is to maximize the expected reward:
(() EDDD yr-mo(-la) [R( y)]
PG algorithms then leverage gradient ascent to optimize the policy with the following gradient:
T
V0) ( y-mo(-la
Va log (ol yyd) A
(3)
where A₁ denotes the estimated advantage at time step I, i.e., how much better action is than the expected action under the current policy.
Importance sampling and PPO objective. To improve sample efficiency and adapt to modern infrastructure, mainstream RL algorithms for LLMs typically adopt PPO-like surrogate objective (Schulman al. 2017):
PPO (0) = EEND, y Toatoon (11a))
minn(r: Ar, clip(r,, +s) A.]]
(4)
where Tt xxllvlo..ve.
the importance weight that corrects for the distribution mismatch,
estimating the expected advantage of tokens generated by the behavior policy Te under the target policy To. The clipping mechanism in PPO serves to implicitly enforce a trust region between the
Our code available http::t/p////..... h//p::///// htt///gttttutt com wac.///ooooyy//aA/

---

Published as a conference paper at ICLR 2026
Loss Contribution
Token Proportion
Advantage =
Advantage >
0.65
0.55
Probabiity
Negative Tokes
Postive Tokmn
0:45
30
100
150
50
100
150
p4,02: (82-04) 14ttal (SS,3a RA Importance Sampling Weight
pa, LAE-s LiBt Laiiaa Importance Sampling Weight
Step
Step
Figure 4: Contribution positive and negative tokens to the volicy-gradiens loss and their pro- portion of tokens during training.
Figure 5: Relationship between token probabil- ity and importance sampling weight.
behavior and target policies, preventing overly large policy updates that could destabilize training. The hyperparameter ε (0, 1) determines the width of this clipping interval.
We then analyze data with positive and negative advantages respectively. The policy gradient can then be expressed as:
V JPO- E(T(A, 0) T(re < 11 A, (y)
V log (v))
positive tokens
(5)
+ T(Ar < 0) T(r, 1- e) A (v)) V log no (1:)
negative tokens
where I represents the indicator function.
3 MOTIVATION: IMBALANCED OPTIMIZATION AND ENTROPY-CLIE RULE
In this section, we first conduct preliminary experiments to show the influence of data staleness on the RL optimization process. Next, we perform in-depth empirical and theoretical analysis reveal the underlying mechanisms and provide new insights.
Training instability with data staleness. We perform experiments under different levels of data staleness using the popular GRPO algorithm. Results in Figure [i that, compared to on on-policy training. off- off-policy RL typically suffers from instability, and entropy decreases rapidly, reflecting reduced exploratory capacity (He et al.) 2025). As staleness increases, the entropy decline becomes more severe and larger number of tokens are clipped; meanwhile, training becomes more unstable. In the following paragraphs, attempt explain this phenomenon from different perspectives and summarize the motivation behind our method.
Excessive negative samples lead to imbalanced optimization. Within the PPO-like objective for policy updates, we analyze tokens with positive and negative advantages separately, shown in Equa- tion 55) Empirical results in Figure - reveal pronounced imbalance: positive samples constitute minority both in number and in their con- tribution to the policy-gradient loss. We attribute this skew to two main factors: (i) the model tends to generate longer trajectories on difficult queries, thereby producing more tokens in negative samples (Figure G): and (ii) in early stages of training, the model has not yet acquired sufficient capability, resulting in higher proportion of negative sam- ples. This observation may help explain the effectiveness of certain curriculum-basee approaches (Xi et al. 2024a Yuan et al. 2025).
Average Response Length
7000
Negative
6000
Positive
5000
50 Step
100
Figure 6: Average model response training.
length during
In the RL training of LLMs, reinforcing positive samples is often more efficient for driving per- formance gains than attempting to "suppress" the vast number of negative samples (Gülçehre al.) 2023 Zhu et al. 2025). To this end, prior work has proposed amplifying positive signals through the clip-higher technique (Yu et al. 2025). However, merely enlarging the clipping upper bound does not mitigate the influence of negative data, thus failing to prevent them from dominating the opti- mization process. Moreover, shown in Equation 5 the accumulation of low-probability negative
4

---

Published as a conference paper at ICLR 2026
tokens (i.e., (p)))) + 0. driving the log term toward -oc) may trigger gradient explosion, further
destabilizing training Yang et al. 2025c).
The Entropy-Clip Rule exposes insufficient entropy promotion in optimization, leading to en- tropy collapse. Theoretically, we derives Equation (see Appendix for detailed derivations) for PPO surrogate objective to reveal the factors that influence the policy entropy (Roux et al. 2025b
(H(TE) -1 Cov m- [log (o vcc). x(y)) + C]
(6)
where C' is a constant, and
1, if A. 0 <1+e
(y)
or A: < 0 &r > 1
(7)
0, otherwise.
We observe that changes in policy entropy are driven by the influence of unclipped tokens, which is determined by the covariance between their log probabilities and advantages. We term this as the Entropy-Clij Rule. The left side of Figure illustrates how the optimization of different tokens in- fluences the probability distribution, thereby affecting entropy. The Entropy-Clij Rule theoretically explains the following statement: Specifically, updating the policy with positive geggpprbbability tokens (high advantage, high probability) and negative low-probability tokens (low advantage, low probability) sharpens the distribution and consequently reduces entropy. Conversely, updating the policy with negative prob-brobability tokens and positive low-probability tokens smooths the distri- bution, resulting an increase in entropy (dctailed proofs are available in Appendix (..42)..
Empirically, our statistical analysis on token probabilities and their importance sampling (IS) weights further clarifies this dynamic. As shown Figure [ we find that tokens with either very high or very low IS weights tend to have low probabilities. However, in standard algorithms with symmetric clipping bounds (e.g., [0.8,1.2]) a majority of positive, low-probability tokens are pre- vented from contributing to the optimization. This systematic exclusion of entropy-increasing up- dates causes a continuous decline in entropy, ultimately crippling the model's exploratory capacity and resulting in a performance bottleneck.
Summary of motivation. Based on the above analysis, we can summarize two main motivations: (1) to balance the contributions of positive and negative tokens while preventing gradient explosion, and (2) to preserve policy entropy for sustaining exploration and preventing collapse.
METHODOLOGY
4.1 VALIDATION EXPERIMENT: ASYMMETRIC CLIPPING
The main idea of our method is to stabilize the training and maintain exploration ability of the policy by asymmetrically adjusting the trust re- gion for positive and negative tokens, i.e., ad- justing Quw and Chigh.
Training Reward
Entropy
0.6
0.5
0.4
0.2
0.4
We then conduct preliminary validation exper- iments to examine whether asymmetrically ad- justing the clipping range could influence the training dynamics. The results, shown in Figure 7 together with Figure reveal that increasing the upper bound Chigh (which introduces more low-probability positive tokens to policy up- dates) improves performance while counteract- ing the downward trend entropy, albeit at a rapid pace. contrast, relaxing the lower bound Clw (which introduces more low-probability negative tokens to policy updates) not only degrades
0.0
50
100
150
100
150
Step
Step
Clip=[0.8.1.5]
Clip=[0.8,1.2]
Clip=[0.5,1.2]
Figure 7: Training dynamics of asymmetric clip- ping experiments.
5

---

Published as a conference paper at ICLR 2026
Algorithm 1: BAPO
Input: Initialized LLM policy Te. training dataset reward function R. staleness E, movable
range of clipping bounds (a b and [a b+], step size of upper bound ò1, step size of lower bound 62. positive token contribution threshold Po
for step 8 1...S do
Procedure Sample and filter out responses
Update the old LLM policy B TT
Sample the s-th batch D from D
Sample G responses {ye [ ~ Tm (. (2), where x E 'Ds
Compute reward and advantage for each y: based reward function R: for staleness = O...E do
Procedure Dynamically adjusting the clipping bounds Chigh and Clow
Initialize clipping bounds Gow = a and Chigh aat;
while the positive token contribution p Po and Gaw + d2 S b do
11
if Chigh ô1 b+ then
Ohigh f Onigh + 81
else
Clow Clow + 82
16
end
17
end
18
Procedure Update the LLM policy nh
Update the LLM policy Te by maximizing the following objective: JBAPO (0)
E ByroTeaaao (.Ix) Σr [min(r:- Ar, clip(rr,, Gow, Chigh) A..]]
20
21
end
22 end
performance but also accelerates entropy collapse. These findings confirm the effectiveness of en- tropy control through asymmetric clipping. Nevertheless, this approach remains relatively rigid and manually specified, providing limited flexibility and adaptation.
4.2 BAPO: BALANCED POLICY OPTIMIZATION WITH ADAPTIVE CLIPPING
To this end, propose BAlanced Policy Optimization with Adaptive Clipping (BAPO). new method to achieve stable, fast RL optimization for LLMs. The core insight of BAPO lies its adap- tive clipping mechanism, which dynamically adjusts the clipping bounds Chigh and Cow to regulate the positive contribution to the policy loss and maintain a balance in entropy throughout RL training. Formally, for each update with batch, our goal is to find pair of Chigh and Clow that satisfy:
EA,20 Teataa (3re min(r. Aes clip(r., O, Chigh) Ae) A+)]
2 Po
(8)
EA, Te (yx) min(r: Ar, clip(r:, Clow, 9high)
where Po is the target contribution of positive signals to the policy gradient loss. Specifically, BAPO gradually increases Chigh and Clw with step sizes of ō1 and 8g, respectively, until the condition in Equation 8 is met. We present an overview of BAPO Figure summarize in Algorithm !
Overall, BAPO offers several significant benefits. First, by dynamically adjusting Chigh and Clow for each step, we can increase the contribution of positive tokens to the policy-gradiens loss while pre- venting negative tokens from excessively dominating the optimization objective. Second, based on our earlier analysis of the relationship between IS weights and token probabilities in Figure BAPO incorporates more low-probability positive tokens and filters out more low-probability negative to- kens, both of which contribute to maintaining entropy. Third, by setting the target contribution from positive tokens, BAPO prevents uncontrolled entropy growth, avoids situations where positive to- kens overwhelm the loss, and mitigates tail degradation where the model overfits to easy problems but fails to handle more challenging (Ding et al. 20256).

---

Published as a conference paper at ICLR 2026
Training Reward
Entropy
Positive Loss Contribution
Gmd Norm
0.6-
100
0.2
0.4
9.1-
0.42
Bealin:
0.40-
0.3g
130
Step
Step
Step
Step
Figure 8: Training dynamics of BAPO.
Prob vs Weight vs Intropy
Wright vs Probobilitt
Prob ve Entropy
1.0
sith Percmilke
2.5
Eattopy
2.0
Probubillty
6o
Entragy
1.0
0.5
09
li 0g Probability
Weight
6.00
0.0
28
25
Lo
0.33
0.75
Weight
3.00
1.25
1.50
0..0
0.2
Probability
0.4
0.5
0.5
1.0
Figure 9: Relationship among token probabilities, importance sampling weights, and entropy.
4.3 ANALYSIS
Stable and fast training of BAPO. As shown in Figure g BAPO enables a more stable opti- mization process, characterized by rapidly increasing training rewards, greater contributions from positive tokens, steady gradient normalization, and stable policy rntropy-resulting in an improved balance between exploration and exploitation.
Clip Bound
We further visualize the adjustment process of the clipping bounds in BAPO. As shown in Figure TO. the averaged upper and lower clipping bounds both fluctuate during training, confirming that BAPO dynamically adjusts the clipping for both types of data and adaptively balances their contributions to the loss. In contrast to ap- proaches such as DAPO (Yu et al. 2025) or the asymmetric clipping in Section[4.] which rely on empirical tuning, BAPO eliminates the need for complex manual hyperparameter tuning, making simple yet effective.
Clip-Htigh-Bund
Clip-Low- Bound
100
150
Step
Figure 10: Clipping bounds.
Effectiveness of BAPO across different stal- eness. We conduct experiments using the RI- Distill model (Guo et al. 2025a) on the WkrWork-
Ascuracy Pereentle (%))
39.35
20
ORI-RL dataset (He et al. 2025), with a maxi- mum sequence length of 32k. The results in Fig-
show that under different data staleness, our method consistently outperforms both the base- line and the clip-higher approach, demonstrating its superiority.
AIME 2024
AIME 2024
AIME 2025
AIME 2025 (Staleness 4)
(Stallness 2)
(Staleness 4)
(Staleness 2)
Figure 11: Results with different data staleness.
The working mechanism of BAPO and its connection to prior work. To better understand the working mechanism of BAPO, we present the relationship among token probabilities, IS weights, and entropy during training in Figure 9 We find that as IS weights deviate further from 1, the corresponding token probabilities decrease, and such low-probability tokens often exhibit higher entropy. Based on this observation, we explain how BAPO relates to prior work. For example, Clip-Higher in Yu et al. 2025) sets the clipping upper bound to 1.28, thereby including more low- probability positive tokens in training, which stabilizes entropy while balancing the contributions of positive and negative tokens. Similarly, Wang et (2025)) retain only the top 20% highest-entropy tokens for training, ensuring stable entropy throughout optimization and preserving the model's
7

---

Published as a conference paper at ICLR 2026
Table Main evaluation results.
Model
Model Size
AIME 2024
AIME 2025
Average
100B Models and Proprictary Models
Qwen3 Swe53-235B-- 2-----32222 A22BB Yang et al
2025a
235B
85.7
81.5
83.6
DeepSeek-FF (uo ct al. 2025a
671B
79.8
70.0
74.9
DeepSeek-R1 52328
2025u
671B
91.4
87.5
89.5
Olneiiun (Jacch al. 2024
83.3
79.0
81.2
o3-mininediop OOeenAl 2025
79.6
76.7
78.2
03-minisigt
OpenAi 2025
87.3
86.5
86.9
Gemi mi-2.00 Oplash Bhinking Google 2024
73.3
Gemini-2 5plash Sptahh-WBinkian Whinking-5522
2025
82.3
72.0
77.2
JOB 100B Scale Models
Owen3-30B-A-3 Yang al. 2025a
30B
61.3
sl-Disttills 1atr1111-weeer -wwe--ee----- 52B 52B (Goo 2025a
32B
72.6
54.9
63.8
QwQ-32B Qwen 2025
79.5
72.4
Qwen3-32B Yang
2025a
32B
81.4
77.2
3kyykakkOR1133/
HC al 2025
32B
82.2
73.3
77.8
4P-Math-32RRg
32B
84.4
78.1
81.3
3P-Kath-
1arbkatthh-BB..c0 Bawo
32B
84.6
78.8
81.7
32-Math-32BBu
32B
87.1
80.0
83.5
Models
R1-Distil1-Qwes--> Guoo
2025a
54.2
Light-R1-7B-DS Wen at 2025
7B
59.1
44.2
51.7
L.kea--bobaRLL- (u a 2025
7B
61.9
48.3
55.1
AceReasen-Nemo
ron- Chen cf
2025c
7B
69.0
53.6
61.3
Sty8crk-0R1-77
He
70.2
3P-Math-78utt
7B
66.9
59.0
62.9
37-Kath-7Bumm
7B
69.2
59.2
64.2
32 32-Math--BBB BBaAo
7B
70.8
62.5
66.7
exploratory capability, and the target entropy technique which aligns with our motivation.
He et al. 2025) plays similar role,
5
EXPERIMENTS AND DISCUSSION
5.1 EXPERIMENTAL SETUPS
Datasets and Models. We use SkyWork-OR1-RL-DD (He et al. 2025
RL dataset, it is widely adopted and of high quality. For evaluation, we employ both the AIME 2024 and the newly released AIME 2025 {IME 2025) benchmarks. Our experiments cover range of backbone mod- els, including DeepSeek RI-Distiilli -wnll--ween-ene 7B, DeepSeek- -R1 -tteeeeeee-----tiiiiiiee---e---- (Guo et al. 2025a and Oct OctoThinker-Llana, addition, we incorporate two our own supervised fine-tuning (SFT) models. BP-Math 73 and BP-Math-3255 which are derived from Qwen2. 5-Mathh (Yang et al. 2024) through fine-tuning.
Qwen 32B 2-3B-Long-Zere (Wang al. 20256). In
Implementation details. We leverage GRPO the basis for BAPO. Both our preliminary and validation experiments are conducted using seepseek-R1-Distill- -1 with the max- imum response length set to 8k, learning rate to 10-6 and temperature 0.6. For main results on BP-Math models, we set the maximum response length to 64k to align with the SFT setting. To introduce staleness,
adopt multiple strategies, including experience reuse through _poch Schulman et al., 2017) and the modern partial rollouts (Team et al. 2025 Fu et al. 2022)) ). For BAPO, set the target contribution Po 0.4, the movable range a 0.6. b 0.9, b+ = 3.0, and the step size 61 0.05, &y = 0.02. These hyperparameters are not finely tuned, as they already demonstrate strong empirical performance. For evaluation, we report results averaged over 16 rollouts.
1.2.
Baselines. We include a variety of commercial and open-source models of different scales base- lines, shown in Table and report their performance extracted from prior work. In addition, we compare different training approaches, including SFT and GRPO.
5.2
MAIN RESULTS
The main results shown in Figure I and Table
8

---

Published as a conference paper at ICLR 2026
Significant performance improvements across models of varying sizes. For strong SFT mod- els. GRPO provides only marginal benefits for instance, it improves performance by just 0.2 and 0.7 points on AIME24 and AIME25 with the BP-Math-322 model. In contrast, BAPO delivers substantial gains across models of different scales. Specifically, with the - model, BAPO outperforms SFT by 2.7 and 1.9 points AIME24 and AIME25, respectively; with the BP 1e-Matth777 7B model, it achieves even larger improvements of 3.9 and 3.5 points.
SOTA performance over open-source models of comparable sizes and competitive results against proprietary models. Compared to open-source models of similar sizes, our BAPO- trained models achieve state-of-the-art (SOTA) performance. For instance, among 32B models, BP- Math-32Baaa outperforms Qwen - by 5.7 and 7.1 points on AIME24 and AIME25, respec- tively, and surpasses SkyWork-OR1- 32B by 4.9 and 6.7 points. Among 7B models, BP-Math- BBAPPO also delivers a notable 7. 7.9-point improvement over SkyWork- OR1 73 on AIME25.
Moreover, BP-Mathh32Beare even outperforms some larger-scale models-for example, sur- passes DeepSeek-R1 by 7.3 and 10.0 points on AIME24 and AIME25, respectively-whit achieving performance comparable to o3-mini. Notably, even the smaller -Math-7Bosse yields results on par with Gemini- 1-00-Faah-TThiie of our approach against commercial models.
underscoring the competitiveness
5.3 DISCUSSION
Partial rollout. To speed up rollouts in LLM reinforcement learning, modern AI infrastruc- tures have introduced several techniques, with partial rollout being particularly noteworthy (Team et al. 2025; Fu et al.) 2025). In this ap- proach, long trajectories are split into segments: when a rollout exceeds a fixed token budget, the unfinished portion is stored in replay buffer and resumed in later iterations instead of being regenerated from seratch. While this improves training efficiency, it also introduces off-policy leamning since different parts of the same trajectory may come from multiple outdated policies. We evaluate BAPO under this setting, as shown in Figure [2] Compared to the baseline GRPO, BAPO exhibits greater robustness such off off-policy infrastructures and achieves more stable optimization.
Training Reward
Entropy
0.5
0.3
0.5-
0.2
0.1
50
100 Step
150
200
50
100 Step
150
200
Budget-4k
RAPO, Rielget-2
Baseling, Budget 4k
Baseline, Budget-24
Figure 12: Training dynamics with partial rollout.
Results OctoThinker-Llamaas L-te:ikeeLLaaaaa-3--LLLLLLan --3-:..----------eer
Table 2: Performance of Llama-based models.
In addition to the DeepSeck-RI-Dissee twwnn, we also conducted experiments on Llama-based models (Wang iet al. 2025b). As shown in Table 2 and Figure 13 in
Method
AIME 2024
MATH
Appendix B our method achieves more competitive results and exhibits greater stability in training dynamics.
GRFO
2.5%
2.9%
58.4%
BAPO
5.4%
5.8%
66.0%
Generalization across diverse reasoning domains. To further validate the universality and ro- bustness of BAPO beyond standard mathematical benchmarks, we expanded our evaluation scope to cover broader spectrum of reasoning tasks. Experiments were conducted using DeepSeek-RII Distill-Qwen-775 We configured the maximum sequence length to 16k for the logical reasoning task to accommodate long-context few-shot examples, while maintaining and scientific tasks. As shown in Table and Figure [4 in Appendix C BAPO consistently delivers substantial performance gains across all tested domains.
8k limit for mathematical
Comparison with other algorithms. Beyond the standard GRPO baseline, we extended our eval- uation to systematically compare BAPO with advanced variants sharing similar motivations, includ- ing TOPR Roux et al. 22055)), DAPO Yu al. (2025). and DCPO Yang et al. (2025b). All methods were implemented and evaluated using the identical 3P-Math-77 backbone and BAPO consistently outperforms these strong baselines presented in Table This superior performance validates that, among current advanced variants of GRPO, our adaptive clipping strategy offers the most effective solution for stabilizing training and maximizing reasoning capabilities.

---

Published as a conference paper at ICLR 2026
Table 3: Comparison of performance across different domains. Results includes AMC 2023 and OlympiadBench He et al. (2024) for advanced mathematical reasoning, ARC-AGI Chollet (2019) for logical reasoning, and GPQA-Diamond Rein et al. (024) for scientific question answering.
Domain
Training Set
Test Set
Original
GRPO
BАPO
Improvement
AMC 2023
73.3
82.5
92.5
+19.2
Math
Skywork Math
OlympiadBench
33.6
54.2
58.4
+25,4
Logical
Enigmata
2025a
ARC-AGI
0
1.4
3.2
+3.2
General
General-Reasonee Ma et al. 2025
GPQA-Diamond
42.4
45.7
51.3
+8.9
Robustness to hyperparamteters To demonstrate the robustness BAPO and assess its sensi- tivity to hyperparameter settings, conducted comprehensive ablation studies the BP-Math-7B model. We specifically examined three key components of our algorithm: (1) the positive token con- tribution threshold Po. (2) the movable ranges of clipping bounds [a_.b. b-] and [a.. b+]. and (3) the adjustment step sizes 81 and 62- The results are detailed in Appendix [ Overall these experiments demonstrate that BAPO does not require intricate hyperparameter tuning. Its key advantage lies in the ability to automatically adapt clipping values within broad search spaces, achieving superior performance using simple heuristic settings.
RELATED WORK
Recent landmark models, like OpenAl ol (Jaech et al. 2024). DeepSeek-RI (Guo al. 2025a) Gemini 2.5 (Comanici et al., 2025), QwQ [Qwen] 2025). have demonstrated that menemnteemnnt
learning can effectively enable long tfain-of-thought reasoning LLMs (Shao et al. 2024 Zhang
et al. 2025 Cai et al. 2025 Chen et al. (0226)). Mainstream algorithms include PPO (Shhulman
al. 2017 and GRPO (Shao al. 2024): PPO constrains updates via a clipping- surrogate objective, while GRPO enhances long-horizon reasoning through group-based rewards.
Despite the remarkable success of RL for LLMs, ensuring stability and efficiency in optimization remains a major challenge (Yu et al., 2025 Cui et al. (205)) Recent studies have sought to better un- derstand the underlying mechanisms RL and proposed new methods to achieve a balance (Zheng
et al. 2025 Wang et al.) 2025at Yang et al. 2025b 2026 Ding al. 2025a Guo al. 2025b Xi
ct al. 2024c). For example, DAPO Yu et al. (2025) introduces techniques such as Clip-Higher and
dynamic sampling to raise the performance ceiling; Wang al. (2025a) explore optimizing only a small subset of high-entropy tokens for improved efficiency. He et al. 2025), Cui et al. (2025),
and other works (Zheng al. (025 Cheng et al. 2025 Liu al.) 2025 systematically investigate how to maintain entropy stability during training, thereby preserving the model's exploration abil- ity. For off-policy Roux al. (0255)) and Arnal et (2025) introduce asymmetric clipping mechanisms. The most similar to our work is DCPO (Yang et al. (0255), which adjusts token-level clipping based on token prior probabilities. However, our approach takes holistic optimization perspective: we observe the imbalance in loss contributions and derive the Entropy-Clip Rule for the PPO objective, enabling dynamic control over global clipping bounds. We further validate the effectiveness of our method through larger-scale experiments.
7 CONCLUSION
In this paper, we begin by analyzing the impact of data staleness on model training through both empirical and theoretical studies. We reveal the imbalance between positive and negative samples in RL optimization, and derive as well as empirically validate the Entropy-Clip Rule for PPO-like objectives. Building on these insights, we propose BAPO, which dynamically adjusts the clipping bounds to balance positive and negative samples while preserving the model's 's exploratory capability during training. We conduct extensive experiments across different models and settings to validate method. We hope work provides key insights for the L.M RI. community.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
The authors wish thank the anonymous reviewers for their helpful comments. This work was partially funded by Shanghai Municipal Science and Technology Major Project 2025SHZD2X025G07, National Natural Science Foundation of China (No.62476061, 62576 106. 62376061), Shanghai Rising-Star Program (23QA1400200), and Natural Science Foundation of Shanghai (232R1403500)
ETHICS STATEMENT
This research introduces an RL methodology designed to augment reasoning capabilities. However, we recognize that it may inadvertently strengthen other capabilities, including those with potential for malicious use. We firmly state that this work is intended for ethical and constructive purposes. Users of this method bear the full responsibility for ensuring it is applied in safe, fair, and harmless manner. Any misuse of this method is strictly against the intent of the authors.
REPRODUCIBILITY STATEMENT
We have describe our method and the hyperparameters in $ and $ To support reproducibility, we will open-source our code. The datasets used for RL experiments are already publicly available.
REFERENCES
AIME Aime problems and solution, 2025. URL http:///os///rrrpppooooooloo/oo/
wi php/ATME.
andss_and____ P
Anthropic. Claude code, 2025.
URL http:://eos..ntttrpcc.co//////
claude-code
Charles Arnal, Gaëtan Narozniak, Vivien Cabannes, Yunhao Tang, Julia Kempe, and Rémi Munos. Asymmetric REINFORCE for off policy reinforcement learning: Balancing positive and negative rewards. CoRR, abs/22506.255000 2025. doi: 10448550ARXX1.2500005222 URL https://
doi www.orrg/10.. 411.048550/rra www 2506 20520
Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, Yichen Feng, Kelin Fu, Bofei Gao, Hongcheng Gao, Peizhong Gao, Tong Gao, Xinran Gu, Longyu Guan, Haiqing Guo, Jianhang Guo, Hao Hu, Xiaoru Hao, Tianhong He, Weiran He, Wenyang He, Chao Hong, Yangyang Hu, Zhenxing Hu, Weixiao Huang, Zhiqi Huang, Zihao Huang. Tao Jiang. Zhejun Jiang, Xinyi Jin, Yongsheng Kang, Guokun Lai, Cheng Li, Fang Li, Haoyang Li, Ming Li, Wentao Li, Yanhao Li, Yiwei Li, Zhaowei Li, Zheming Li, Hongzhan Lin, Xiaohan Lin, Zongyu Lin, Chengyin Liu, Chenyu Liu, Hongzhang Liu. Jingyuan Liu, Junqi Liu, Liang Liu, Shaowei Liu, T. Y. Liu, Tianwei Liu, Weizhou Liu, Yangyang Liu, Yibo Liu, Yiping Liu, Yue Liu, Zhengying Liu, Enzhe Lu, Lijun Lu, Shengling Ma, Xinyu Ma, Yingwei Ma, Shaoguang Mao, Jie Mei, Xin Men, Yibo Miao, Siyuan Pan. Yebo Peng, Ruoyu Qin, Bowen Qu, Zeyu Shang, Lidong Shi. Shengyuan Shi, Feifan Song, Jianlin Su. Zhengyuan Su, Xinjie Sun, Flood Sung, Heyi Tang, Jiawen Tao, Qifeng Teng, Chensi Wang, Dinglu Wang, Feng Wang, and Haiming Wang. Kimi K2: open agentic intelligence. CoRR, 0lls22577.205344 2025. doi: 10.48550/ARXIV.550..000 205344 htRt::tts:://do org/ 10. 005548550//0 http//aaxiii.......... 20534
Yuxuan Cai, Chen, Qiaoling Chen, Yuyang Ding, Liwen Fan, Wenjie Fu, Yufei Gao, Honglin Guo, Pinxue Guo, Zhenhua Han, et al. Nex-nl: Agentic models trained via unified ecosystem for large-scale environment construction. arXiv preprint arXiv:25/2.04987, 2025.
Jiangjie Chen, Qianyu He, Siyu Yuan, Aili Chen, Zhicheng Cai, Weinan Dai, Hongli Yu, Qiying Yu, Xuefeng Li, Jiaze Chen, et al. Enigmata: Scaling logical reasoning in large language models with synthetic verifiable puzzles. arXiv preprint ar0iv:2505. /99000999999.... 2025a.

---

Published as a conference paper at ICLR 2026
Wenxiang Chen, Wei He. Zhiheng Xi. Honglin Guo, Boyang Hong. Jiazheng Zhang. Nijun Li, Tao Gui, Yun Li, Qi Zhang, et al. Better process supervision with bi-directional rewarding signals. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 14471-14485, 2025b.
Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catan- zaro, and Wei Ping. Acereason-nemmotron: Advancing math and code reasoning through rein- forcement learning. CoRR, abs/2505.164400, 2025c. doi: 10.4850/ARXI22555..666 /tt.///...550//20
URL
16400
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. CoRR, abs/2506. 14758, 2025. doi: 1044850/ARXIV.5506. 14758. URL /tte:///oo...gg1...888////////
14758
François Chollet. On the measure of intelligence. arXiv preprint arXiv: 19t1:/99111110 19100547911.015,,2200
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inder- jit S. Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpek- tor, Nan-Jiang Jiang, Krishna Haridasan, Ahmed Omran, Nikunj Saunshi, Dara Bahri, Gaurav Mishra, Eric Chu, Toby Boyd, Brad Hekman, Aaron Parisi, Chaoyi Zhang, Kornraphop Kaw- intiranon, Tania Bedrax-Weiss Oliver Wang. Ya Xu, Ollie Purkiss, Uri Mendlovic, Ilai Deu- tel, Nam Nguyen, Adam Langley, Flip Korn, Lucia Rossazza, Alexandre Ramé, Sagar Wagh- mare, Helen Miller, Nathan Byrd, Ashrith Sheshan, Raia Hadsell Sangnie Bhardwaj, Pawel Janus, Tero Rissa, Dan Horgan, Sharon Silver, Ayzaan Wahid, Sergey Brin, Yves Raimond, Klemen Kloboves, Cindy Wang, Nitesh Bharadwaj Gundavarapu, Ilia Shumailov, Bo Wang, Mantas Pajarskas, Joe Heyward, Martin Nikoltchev, Maciej Kula, Hao Zhou, Zachary Garrett, Sushant Kafle, Sercan Arik, Ankita Goel, Mingyao Yang, Jiho Park, Koji Kojima, Parsa Mah- moudich, Koray Kavukcuoglu, Grace Chen, Doug Fritz, Anton Bulyenov. Sudeshna Roy, Dim- itris Paparas, Hadar Shemtov, Bo-Juen Chen, Robin Strudel, David Reitter, Aurko Roy, An- drey Vlasov, Changwan Ryu, Chas Leichner, Haichuan Yang, Zelda Mariet, Denis Vnukov, Tim Sohn, Amy Stuart, Wei Liang, Minmin Chen, Praynaa Rawlani, Christy Koh, JD Co- Reyes, Guangda Lai, Praseem Banzal, Dimitrios Vytiniotis, Jieru Mei, and Mu Cai. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next genera- tion agentic capabilities. CoRR, abs/2507.006611, 2025. doi: 10448550//RXX..2500..666 ltttps://dee /ww:i//oiiiioo org 10. 48540/5550000 http://wwwwv.......... 06261
URL
252..66261
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng. Lei Bai, Wanli Ouyang. Yu Cheng, Bowen Zhou, and Ning Ding. The entropy mechanism of reinforcement learning for reason- ing language models. CoRR, abs/2505.22617, 2025. doi: 10.48550AAXXW.25002226611 URL
https: wrg//o................ooo....... 48540/5550000
Yiwen Ding, Zhiheng Xi, Wei He, Lizhuoyuan Lizhuoyuan, Yitao Zhai, Shi Xiaowei, Xunliang Cai, Tao Gui, Qi Zhang, and Xuan-Jing Huang. Mitigating tail narrowing in I1m self-improvement via socratic-guided sampling. In Proceedings of the 2025 Conference of the Nations of the Ameri- cas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 10627-10646, 2025a.
Yiwen Ding, Zhiheng Xi, Wei He, Lizhuoyuan Lizhuoyuan, Yitao Zhai, Shi Xiaowei, Xunliang Cai, Tao Gui, Qi Zhang, and Xuanjing Huang. Mitigating tail narrowing in LLM self-improvement via socratic-guided sampling. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceed- ings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2025 Volume 1: Long Papers, Albuquerque, New Mexico, USA, April 29 May 4, 2025, pp. 10627-10646. Associa- tion for Computational Linguistics, 2025b. doi: ottp:////.....00..11... wtt.oo//gg...1111////1///1.111 wttp:/22222aaa111o1ogoe.
URI.
Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, Tongkai Yang. Binhang Yuan, and Yi Wu. Areal: A large-scale asynchronous reinforcement learning system for language reasoning. CoRR, abs/2505. 1ttg//11.885550
2025. doi: 10. 2505. 24298
48550/ 24298. htt.st:s::
www./rrr...... 48550 rxi w.e.ii..55....0.
12

---

Published as a conference paper at ICLR 2026
Google. Introducing gemini 2.0:
new ai model for the agentic era, De- google-gemina wtw.-aaaappatteeeeeeeeeeee 2024/
cember 2024. URL https:::p//bll
Çaglar Gülçehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahem, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas. Reinforced self self-training (rest) for language modeling. CoRR, abs/2308 08998, 2023, doi: 10.48550ARXXIV22000 08998, SoLhttp:: /oo... org/ 10. 188550/rr11n 2308 08998
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhi- hong Shao, Zhuoshu Li, Ziyi Gao. Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen. Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang. Hanwei Xu, Honghui Ding, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jingchang Chen, Jingyang Yuan, Jinhao Tu, Junjie Qiu, Junlong J. L. Cai, Jíaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaichao You, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingxu Zhou, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan. S. S. Li, Shuang Zhou, Shaoqing Wu, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang. Xiaokang Chen, Xiaotao Nie. Xin Cheng, Xin Liu, Xin Xie, Xingehao Liu, Xinyu Yang. Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song, Xinyi Zhou, Xianzu Wang. Xinxia Shan, Y. K. Li, Y. Q. Wang, X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo. Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunx- ian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseck-rl incentivizes reasoning in 1lms through re- inforcement learning. Nature, 645(8081):6633-633 638, 2025a. doi: 1100033844158--025-044422222 URL https:
Xin Guo, Zhiheng Xi, Yiwen Ding, Yitao Zhai, Xiaowei Shi, Xunliang Cai, Tao Gui, Qi Zhang, and Xuanjing Huang. Counteracting matthew effect in self-improvement of Ivlms through head-tail re-balancing, arXiv preprint arXiv:2510.2641 2025b.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-leve bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1; Long Pupers), pp- 3828-388 3850, 2024.
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reasoner technical report. CoRR, abs/2505..21111 2025. doi: 10.48550/ARXIV.25055 22312. URL https: /doii wrw.:0..g111000 40550//..455///a// 22555.2331
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett,
13

---

Published as a conference paper at ICLR 2026
Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, An- drey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghor- bani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lu- garesi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer. Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Ed- mund Wong, Elizabeth Proehl, Enoch Cheung. Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclerc, Hadi Salman, Haiming Bao, Hao Sheng, Hart Andrin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung. Ian Kivlichan, Ian O'Connell, Ian Osband, Ignasi Clavera Gilaberte, and Ilge Akkaya. Openai ol system card. CoRR, abs/2412. 167200 2024. doi: 1040550/ARXXV.241..6655 URL 16720
www.oo/g......5//33.. http:::/8ll.orr//00.45550
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. CoRR, abs/2503. 20783, 2025. doi: 10448550/ARRXIV25503.222 20783
URL httpt:h//ooiii http://iiioogg1..433388 httpo/o0r//.....5////a..... 2503.
Xueguang Ma. Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. General-reasoner: Advancing Ilm reasoning across all domains. arXiv preprint arXiv:2505.14652 2025.
OpenAL. Openai 03-mini system card, 2025.
URL. htttp::/ccd..ppett .wmm
03 m ni aad--yytte---ar--ffaba p.
Qwen. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https:
github. wttpb/g///oologgooooooo 32b/
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a bench- mark. First Conference on Language Modeling, 2024.
Nicolas Le Roux, Mare G Bellemare, Jonathan Lebensold, Arnad Bergeron, Joshua Greaves, Alex Fréchette, Carolyne Pelletier, Eric Thibodeau-Laufer Sándor Toth, and Sam Work. Ta- pered off-policy reinforce: Stable and efficient reinforcement learning for llms. arXiv preprint arXiv:2503. 14286, 2025a.
Nicolas Le Roux, Mare G. Bellemare, Jonathan Lebensold, Arnaud Bergeron, Joshua Greaves, Alexandre Fréchette, Carolyne Pelletier, Eric Thibodcau-Laufer, Sándor Töth, and Sam Work. Tapered off-policy REINFORCE: stable and efficient reinforcement learning for 1lms. CoRR, abs/2503. 142866 2025b. doi: 10448500ARXXVV2500.4422 14282.66.....6....
48550/arxiv 2503. 14286
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 77770644777 2017. URL httpp://arxiv.orr//tttt
7707.06344
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Mingchuan Zhang, Y. K. Li, Y. Wu. and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300, 2024. doi: 10.45500ARXXI..402..333 https://doi /0t0//..400//////rr..... arxiv 2402 03300
URL
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, and Will Dabney. Understanding the performance gap between online and offline alignment algorithms. CoRR, abs/2405.08448, 2024. doi: 10.48550/ARXIV.24000 088488 URL https: /ooi www/10...1000 48550/4555aa.. 2405. 08448
14

---

Published as a conference paper at ICLR 2026
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuming Tang. Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao, Huabin Zheng, Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang, Jie Zhao, Jin Zhang. Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma, Shupeng Wei, Sihan Cao, Siying Huang, Tao Jiang, Weihao Gao, Weimin Xiong. Weiran He, Weixiao Huang, Wenhao Wu, Wenyang He, Xianghui Wei, Xianqing Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, Xinyu Zhou, Xuehai Pan, Y. Charles, Yang Li, Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang. Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang. Yiping Bao, Yulun Du, Yuxin Wu, Yuzhi Wang, Zaida Zhou, Zhaoji Wang. Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang, Zhilin Yang. Zhiqi Huang, Zihao Huang, Ziyao Xu, and Zonghan Yang. Kimi k1.5: Scaling reinforcement learning with 1lms. CoRR, abs/2501.1259,, 2025. doi: 10. 48550/ARXIV
htR.httts:// //oii www.el.rrg11.1000 wttp://.1.40///..1..ooooo 2501 12599
Luong Quoe Trung, Xinbo Zhang, Zhamming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Rea- soning with reinforced fine-tuning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 7601-7614. As- sociation for Computational Linguistics, 2024. doi: 10. 10.11853/V1/22 122-22222..2LON......
URL
petpss//doo. 165.:/..000....1111/////////11./
1101110.186//// 2024 acl.-acc--- 1all--ong Long. 410
Shenzhi Wang, Le Yu. Chang Gao, Chujic Zheng. Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang. Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoming. CoRR, abs/2506.01939, 2025a. doi: 10. 48550/ARXIV.2506011933 URL
org. 48550 arxiv. 2506. 01939
Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Mid-training incentivizes reinforcement learning scaling. CoRR, 2ttp/2500.205512. 2025b. doi: 10.48550/ARXVV25566
20512. URL
2501.
Liang Wen, Yunke Cai, Fenrui Xiao, Xin Hc. Qi An. Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, Haosheng Zou, Yongchao Deng, Shousheng Jia, and Xiangzheng Zhang. Light-rl: Curriculum sft, DPO and RL for long COT from scratch and beyond. CoRR, abs/2503.10460, 2025. doi: 1044850/ARXXI.2553..00 10460 URL httts:tt////oit org./1...../1..... 18550/arXiv.2555 10460]
Ro J. Williams. Simple statistical fradient-following algllooithh.ggritnnmmtt for conneciionisttreiiforcee.e learning. Mach. Learn,, 252,,-256, 1992. doi: 10. 10.1007//FF00992669 URL /tttp:://oo..
org/ 10.1007//700992550
Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senjie Jin, Rui Zheng, Wei He, Yiwen Ding, Shichun Liu, Xin Guo, Junzhe Wang, Honglin Guo, Wei Shen, Xiaoran Fan. Yuhao Zhou, Shihan Dou, Xiao Wang, Xinbo Zhang, Peng Sun, Tao Gui, Qi Zhang, and Xuanjing Huang. Training large language models for reasoning through reverse curriculum reinforcement learning. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024a. URLhttps: /pwp//peeeeeeeeeeeeeeeeeeee net/ forum?id=t82y3 fmRtk]
Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senjie Jin, Rui Zheng, Wei He, Yiwen Ding, Shichun Liu, Xin Guo, Junzhe Wang, et al. Training large language models for reasoning through reverse curriculum reinforcement learning. arXiv preprint arXiv:2402.05808 2024b.
Zhiheng Xi, Dingwen Yang, Jixuan Huang, Jiafu Tang, Guanyu Li, Yiwen Ding, Wei He, Boyang Hong, Shihan Do. Wenyu Zhan, et al. Enhancing 11m reasoning via critique models with test-time and training-times time supervision. arXiv preprint arXiv:2411.16579, 2024c.
Zhiheng Xi, Jixuan Huang, Chenyang Liao, Baodai Huang. Honglin Guo, Jiaqi Liu, Rui Zheng, Jun- jie Ye, Jiazheng Zhang, Wenxiang Chen, et al. Agentgym-rl: Training Ilm agents for morizoo decision making through multi-tumn reinforcement learning. arXiv preprint DXXi:2509008755 2025.
15

---

Published as a conference paper at ICLR 2026
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-mate technical report: Toward mathematical expert model via self-improvemmen. CoRR, abs/2409. 11tp//299.12222,,, 2024. doi: 00.48550/RRIV.2499.. 12122. 12122
URL htt.hhttss:///oi... http:///oooorg///////// 48550/
Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng. Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jian Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng. Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. CoRR, abs/2505.09388. 2025a. doi: 10.48550/ARXI.25500 008.09888..
URL https: org. 48540/5535/88 www.v..iiii...... 2505. 09388
Shihui Yang, Chengfeng Dou, Peidong Guo, Kai Lu, Qiang Ju, Fei Deng, and Rihui Xin. Depo: Dynamic clipping policy optimization. arXiv preprint (XXx::25090023333 2025b.
Yuming Yang. Mingyoung Lai, Wanxu Zhao, Xiaoran Fan, Zhiheng Xi. Mingqi Wu, Chiyue Huang, Jun Zhao, Haijun Lv, Jian Tong, et al. Which reasoning trajectories teach students to reason better? a simple metric of informative alignment. arXiv preprint arXiv:2601. 14249, 2026.
Zhihe Yang, Xufang Luo, Zilong Wang, Dongqi Han, Zhiyuan He, Dongsheng Li, and Yunjian Xu. Do not let low-probability tokens over-dominate in RL for 1lms. CoRR, abs/2505. 12292, 2025c. doi: 10.48550/ARXIV.25000 12929. URL http:::tp///.. //doi. org/ 4..50..8555/aaxaa 2505. 12929
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng. Yuxuan Tong, Chi Zhang. Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-
Ma. Ya- Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: an open- source LLM reinforcement learning system scale. CoRR, abs/2503. 2025. doi: 10. 48550/ARXIV 14476.
URL httht:tt: www.orgg11... 012110..855000 0055550/axii10 2503. 2403..44767 14476)
Ruifeng Yuan, Chenghao Xiao, Sicong Leng, Jianyu Wang. Long Li, Weiwen Xu, Hou Pong Chan, Deli Zhao, Tingyang Xu, Zhongyu Wei, Hao Zhang, and Yu Rong. Vl-cogito: Progressive curricu- lum reinforcement learning for advanced multimodal reasoning. CoRR, abs/2507.2260) 2025. doi: 10,48550/ARXIV.2507.21 22607
URL httphhtt:///e //do:i///oi../ 1ttto/oo///.....5//r////ia/ 2507.
Kaiyan Zhang, Yuxin Zuo, Bingxiang He, Youbang Sun, Runze Liu, Che Jiang, Yuchen Fan, Kai Tian, Guoli Jia, Pengfei Li, et al. A survey of reinforcement learning for large reasoning models. arXiv preprint wtw::2559008227 2025.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence pol- icy optimization. CoRR, abs/2507.18071, 2025. doi: 10-8550AARXIV2507..80000 https: http://ooo.ogg///// 0tt///1..8855000 48550/ 28071
URL
Xuckai Zhu, Daixuan Cheng, Dinghuai Zhang, Hengli Li, Kaiyan Zhang, Che Jiang, Youbang Sun, Ermo Hua, Yuxin Zuo, Xingtai Lv, et al. Flowrl: Matching reward distributions for Ilm reasoning. arXiv preprint arXiv:2509, 1500, 2025.
A THE USE OF LARGE LANGUAGE MODELS
LLMs are utilized in this manuscript for partial grammatical checks and language polishing. The authors are fully responsible for the final content.
16

---

Published as a conference paper at ICLR 2026
B PERFORMANCE ON OCTOTHINKER-LLAMA
We illustrate the training dynamics on Octo Thinker-Llamie in Figure 13 Since Llama family models behave badly RL training, we choose the model after mid-training Wang al. 2025b) show the robustness of BAPO. We can find that BAPO provides consistent and significant improvement training. For training details, we set the low bound as 0.8-0.9, high bound as 1.2-2.0, and target positive loss contribution as 0.45.
Training Reward
Entropy
Positive Loss Contribution
0.375
0.20
0.2.
0.350
0.15
0.20
BAPO
0.325
0.10
Baseline
0.300
50
100
150
50
100
150
0
50
100
150
Step
Step
Step
Figure 13: Training dynamics of LccoT
C PERFORMANCE ON GENERAL REASONER DATASET
We illustrate the training dynamics of 11-Distill-Qwen225--2 trained on the General Reasoner Dataset Ma al. (2025). which spans a wide range of domains such as physics, chemistry, finance, and electronics. Figure 14 shows that BAPO effectively stabilizes the training process and yields substantial gains across these diverse domains.
Entropy
Grad Norm
Clip High Bound
10
hyr MM/hyp My M
2.5
5
0.2
2.0
50
100
150
0
50
100
150
100
150
Step
Step
Step
Toothr
BATO
Clip-
Figure 14: Training dynamics on General-Reasome
D COMPARISON WITH OTHER ALGORITHM
Here, we present the results comparing BAPO with other algorithms.
Table 4: Performance comparison different RL variants on the BP-Math-7B backbone. BAPO achieves consistent improvements over SFT and other state-of-the-art baselines.
Method
AIME 24
AIME 25
AVG
SFT
66.9
59.0
62.9
GRPO
69.2
59.2
64.2
DAPO
68.3
60.8
64.6
TOPR
68.8
60.4
64.6
DCPO
68.8
57.9
63.4
BAPO (Ours)
70.8
62.5
66.7
17

---

Published as a conference paper at ICLR 2026
E ROBUSTNESS TO HYPERPARAMETERS
1. Positive Token Ratio We varied Po around the default value of 0.4. Results in Table 5 show minimal fluctuation performance, suggesting the method is robust long as reasonable proportion of positive signal is maintained.
Table 5: Ablation studies positive token contribution threshold po-
Po
AIME 2024
AIME 2025
Average
0.4 (default)
70.8
62.5
66.7
0.45
69.6
63.8
66.7
0.50
71.3
60.8
66.1
Clipping Ranges: We tested various intervals for the lower and upper constraints. As shown in T BAPO consistently achieves high scores, proving its ability to find effective clipping points automatically.
Table 6: Ablation studies movable range of clipping bounds.
[a_,b_]
[a+, b4]
AIME 2024
AIME 2025
Average
[0.6, 0.9) (default)
(..2,3.0] (default)
70.8
62.5
66.7
00.5, 0.9)
[1.2, 3.0]
70.8
60.8
65.8
0.6, 0.8
1.2, 3.0
71.3
59.6
65.5
0.6, 0.9
[1.0, 3.0
71.3
62.5
66.9
0.6, 0.9)
[1.2, +00
72.1
60.4
66.3
3. Step Sizes d: We analyzed the impact of varying the step sizes used for updating bounds. Table confirms that the algorithm is insensitive to minor variations in 8, and d2.
Table 7: Ablation studies on step sizes 81 and dz.
81
82
AIME 2024
AIME 2025
Average
0.05 (default)
0.02 (default)
70.8
62.5
66.7
0.02
0.02
71.3
62.5
66.9
0.05
0.05
70.8
62.5
66.7
DERIVATION FROM EQUATION 4 TO EQUATION 5
F.I RESTATING THE PPO OBJECTIVE (EQUATION
In large-scale reinforcement learning for language models, the PPO surrogate objective widely used:
PPD = EzED, y gmatee (-12)
Σ min(r: Ar, clip(r,, 1-
s, 1
A1)
(9)
where (e(y)1z.9cc). ".)))))* range.
A, denotes the advantage function, and e (0,1) controls the clipping
F.2 DECOMPOSING THE PPO Loss STRUCTURE
Fixing partieular time step f, we can rewrite:
PPO (0)
Σ
Trollou (ye Ir, Bee)
min(re A,, clip(r,, 1+ 1+s) -))))..
(10)
18

---

Published as a conference paper at ICLR 2026
Note that when ri lies within the clipping interval, clip(re...)) re. and thus the two terms equal. When r, exceeds the interval, the second term becomes a constant, resulting in zero gradient express the objective
with respect to A. Mcanwhile, observe that 7, Frlloou Tg. Therefore, in piecewise form based on the sign of A,
(0) = Σ I(A, 0} Irr S +e) (v( xz V.M) A
(11)
+1A < 0} 1rr 2 - 8} no (ye a. Vee))) Ar).
In this way, the expectation form is converted into summation over sampled pairs (æ, ye), and the behavioral policy term is eliminated, paving the way for the gradient derivation in the next section.
F.3 TAKING THE GRADIENT WITH RESPECT TO PARAMETERS (EQUATION
5
For the PPO objective in Equation 11 when differentiating with respect to 0, only To( (,,ee)
depends on 0. Thus we have:
V PPO
Σ
T(A 0) I[r, S + A. VoTolW x, Vet)
(12)
+HA 0) 1(r, 2 - =} A. VoTo(ye x, v<12
Using the log-likellne log-likelihood identity Ven = ne V log Te. the above becomes:
VoJePo = Σ [[ 0} Ifr, - 1 + E} (v x. x. A. Vo To(u Ix, vr,
(13)
+1AA 0} 1rr, N 2} (M x. y<t) Ar V log (V Ir, y<t)
This corresponds precisely to Equation in the PPO derivation: When A, > and ri 1 1+++ + s, when A, < 0 and > - the sample contributes an effective gradient; outside the clipping range, the gradient is suppressed to zero, enforcing smooth policy updates.
G PROOFS OF EQUATION 6
G.1 EXPLANATIONS FOR ALL VARIABLES AND EXPRESSIONS
All notation used in the following justification, including variables and expressions, is provided with detailed explanations Table
G.2 PREPARATION: REWRITE THE PPO DERIVATIVES
To facilitate the justification of the propositions below, rewrite the PPO loss function in the following form:
PPO Σ
To(U)) [(r(y)) < +1 A(y:) V log (
A(m))s
positive tokeus
Σ Re(vi) 1r(()) > 1- A(wc) V log To (v()))
A(v:)<0
negative tokens
where
Toe (yl σ. yEE)
Te TV( = (v x y<t) r(ye)
A(y:) A(yi2 ()t)
To (yr x,, yet)
19

---

Published as a conference paper at ICLR 2026
Table Notation used in justification below.
Category
Symbol
Meaning
The policy parameterized by 0
Boolious
The standard sampling policy
Given prompt
Variables
A T-token response generated by when given ær
The t-th token of
Learning rale
mo( æ. Mcce)
Probability of generating token under policy me given input @ and previous tokens y-rr
Probability generating token under standard sampling policy Mapottout given inpul æ and previous tokens yet
A(-la, yet)
The measurement of how much better(or worse) selecting token compared to the expected value under current policy, given and y<t
H(-12,,
The information entropy of policy given and yet
(a(
The expected covariance of a(gi)) and bige) over y sampled from the policy no, given a and yet
Expressions
I(a = b)
Indicator function equals if
0 otherwise
(.
The expected cumulative reward obtained by taking token given input æ and previous tokens under policy Te
(x)
The expected return the new taking token given imput æ and previous tokens under policy xg
quantity representing the cumulative weight of sequence y given input under policy we, reflecting contribution to policy taken at current optimization step
Vo J(8)
The gradient
policy taken with respect to the logit parameter yy. representing how the policy me should adjusted token de given input
G.3 PROOFS THE MAIN PROPOSITIONS
The following derivation is inspired by the proof framework in Cui et al. (2025). While the original work focuses mainly the basic gradient formulation of naive REINFORCE provide heuristic explanation, our study advances this approach by deriving the gradient expression specific to the PPO objective. This refinement offers specific, intuitive yet theoretical account of how policy entropy is intrinsically shaped by the interaction between token-level advantages and their sampling probabilities.
G.3.1 PRECLAIMS
Proofs of these three lemmas below are available in Cui et al. (0255)
Lemma 1. Let the actor policy no be a tabular softmax policy, the difference of information entropy given prompt x. between consecutive steps k and l satisfies
H(m k+1 xr, yet) H(mm DCe)
Cov y ( (log a (ye),
29.00
k+1
=
20

---

Published as a conference paper at ICLR 2026
Lemma 2 (Derivative of softmax function).
i log no(vi) [(: = 1/3 (o(
00y
Lemma (Expectation of Advantage function given prompt 2:).
B (y ( (ye)]
= Egg (x ( (ye, x) - V (2))
= F (QV),. - By [(a)]
((x))
0
G.3.2 PRINCIPLE PROPOSITIONS
Proposition 1: Assume the actor policy тo follows a tabular softmax policy and is optimized via the PPO objective, the difference of Zy.= between two consecutive steps and k+1 satisfies
1++ k zy
= 11-1 - [A(ye)
X + Cj
where
1. if A(v)) > 0 & r(ve) < +
Xv:)
A(y:) < 0 & r(y:)
otherwise
and C includes all clauses irrelevant to yt-
It is worth noting that X' (3+) = 0 if and only if yv. is clipped.
Proof. In tabular softmax policy, each trajectory-prompt pair (y, x>) is associated with an individual logit parameter žy.æ 8 Through gradient backtracking, zy.æ is updated via 2A+1 V Ve According to the loss function of PPO, we have
"y.
2y. 17
1++1
1) V (P)
z.
=y.
= 17 B A()))20
[Ir())) < 1 5} Ve log ( A(y))
+ 17 Ey moo A(v/)<0
[T(((()) > E} Vo To))) A
= E (px, Von. log (o(1 A(v
77 1 [U(r(z() 1 +e Ven.. log Tog/)()() A A(v/)20
1) Ey (o [T((())) < - 8) V Vog.. (ogro(v)) A
A(
1
(8)
We first perform the derivation the term marked as 1
21

---

Published as a conference paper at ICLR 2026
Ô log To(3)) A(vh))
1 = Byg 110.
y0,,m
Lomma
(Te(
(113/ -yr - =o(2))) A(vil
1/4
=1 no (Be)
(1 - (-To(Ue)) A((e)) - Σ To(u/) A(v))
Ve
= - NO (V)e)
A(ye) - Σ [vol A(v))
1/2
Lemana 3
To())) [A(y+) - 0]
= (v)) A(ve)
To keep the presentation concise, we provide only the resulting derivations of Term 2 and 3 as the detailed steps follow similarly to those for Term ①
2
1) To (v)))) A(2a)
(1 -1- x(x))
(V(e))
Σ
[T(((v() > 1 ++ mo(()) A(x()
A
Ty(y))
Σ
[r(())) =3 No (1(3 A(00)
By substituting the results of the above derivation into Clause (8), we observe that:
(8) = ! (2 +
= 17 To(()) { {A(V)) (v:)
Σ [Tr(v/) + E) mo(y') A(y()
A6
(r(()) 1-
=} To(y)) A(v/)]
A
By grouping all elements unrelated to y into C, we are able to successfully establish our proposition.
Building on Proposition 1, we establish the relationship between policy entropy and the covariance of specific tokens, which is stated as Proposition 2 below.
Let the actor policy ny be tabular softmax policy, and no is updated via PPO objective, the difference of information entropy given prompt x and trajectory part yer between two consecutive steps and k+1 satisfies
Proposition 2 (Equation 6)
H( k+ |x, V<t)
H(=ā ar, y<t) s -1 Cov ya Cov (Hawee)) (log T (wt). A(w) x + CC)
Proof. Leveraging the conclusions of Lemma and Proposition 1, we find that, under policy opti- mization and iteration via the PPO algorithm, the following relationship is satisfied:
.+1 k y.
= (A(y.) (v:) + C).
22

---

Published as a conference paper at ICLR 2026
Applying this into Lemma have
H( k+1 11, - H( x. yCt) -1 Cov, y1 ( (pawe))) (log T8 (we). A(v)) ((a) + O).
G.4 ANALYSIS
G.4.1 DIRECT ANALYSIS: WILY VARYING & ALTERS ENTROPY?
We begin by examining the covariance of the clipped token, denoted
Based on the observation stated above, the contribution of a to the entropy can be expressed as:
- (a) Cov (a), C*) = 0,
which indicates that only the retained tokens contribute to the overall entropy.
In other words, manipulate the number of tokens that can contribute the entropy by altering the parameter
G.4.2 ADVANCED ANALYSIS: WHICH TYPE OF TOKENS MATTER MOST FOR ENTROPY?
To understand how individual tokens contribute to the overall entropy, we first revisit the Proposition G.3.2 established above. In this section, we provide more precise definition of tokens with low/high probabilities and advantages. It should be noted that in the analysis experiment (Figure ( we adopt the naive REINFORCE algorithm without clipping. Consequently, tokens with high or low advantages are defined according to the sign of their advantage values, i.e., O for high advantage and < 0 for low advantage.
H(mb/+1pp (<t) H(T x. UKt)) Fo -1 Cov wow (log (w:), A(w)) (((v)) - C)
T
(vp yee) (log क (yp) E w (maee)) flg ~3 (v:))
=
(A(w)) X(x))) B (A())) Xx(x)))
where T is the size of the dictionary.
For convenience, denote Ey, as ( As only retained tokens contribute to the en- tropy, foeus only tokens that are not elipped. We begin by making the following simplification:
Ey, (A(y:) (y)) E Ea (A(ys) 0) Byrrenaa (A(y:) 1) Ey (A(ys))
So for a selected token ye- its contribution the overall entropy can expressed as:
-n (g.)) (log (v.) Ey, (log (v:))) (A(y.) - Eyounss Al (Fretained) )))
Next, we analyze how different types of tokens contribute to the overall entropy. To avoid ambiguity, first give strict definitions that distinguish between tokens with high/low probabilities and tokens with high/low advantages.
Definition 1. For a token ys. we classify it as follows:
High advantage: if
A(ya) E Experinn A ((Hretined)
Otherwise, it called low advantage.
23

---

Published as a conference paper at ICLR 2026
High probability: if
To(ye) > exp(Ey, (log (o())))))
Otherwise, called low probability.
Secondly, we present two propositions that directly follow from the above definitions.
Proposition 3. For token V/s. we have
0, if ys is high-advantage token,
A(y.) mmmans A (Hrreaiined)
0, if Ys is low-advantage token.
Proposition 4. For token yss have
O, if ys is high-probability token,
To (0(3)) (log To (y+) Ey, (log To (y(i))))
O, if ya is low-loopo low-probability token.
Proof. Let us denote
C Ey (log (y:))),
which is independent of Vas and let x no (y))) As TT(y) < 1 for every y C < 0.
Consider the function
f(x) x- (log(z) C).
Figure 15 illustrates the behavior of this function.
-C
0
ec-1
eC
1 x
-
Figure 15: Graph of the function f(z) = x(log æ - C).
The proposition follows directly from the properties of f(x) as observed in the figure.
Due to the propositions given above, we have the table below:
AH(y.) = -1 ( ) (log (v))) Ey, (log To (A(ys) E Youmne A( (Fretained)
Table 9: Influence of token characteristics on AH(Y)). The "prob" denotes the probability no (yx), and the "adv" represents the advantage A(y.).
Token properties
AH(yy)) (-17
high prob, high ady high prob, low ady low prob, high adv low prob, low adv
0
0
0
0
24

---

Published as a conference paper at ICLR 2026
It should be noted that token ys decreases the entropy if AH(y.) O, and increases it otherwise.
Therefore, we observe that tokens which are positive with high probabilities and high advantages, negative with low probabilities and low advantages, contribute to a reduction in the overall en- tropy. Conversely, positive tokens with high probabilities but low advantages, and negative tokens with high probabilities but low advantages, contribute to increase in the overall entropy. This observation justifies the statement made in the main part of the thesis.
25


Reviews:
[{"review_id": "rSkzPIoaCJ", "reviewer": "Reviewer_mQ9j", "summary": "This paper identifies two issues with standard PPO-like RL algorithms when applied to stale data (e.g., due to sample replay or partial rollouts):\n(1) optimization imbalance, where negative-advantage samples dominate the policy gradient;\n(2) systematic suppression of entropy-increasing updates, which limits exploration.\nTo address these issues, the authors propose BAPO, a method that adaptively adjusts the clipping range to ensure that the proportion of loss contributed by positive-advantage samples remains above a pre-specified threshold.\nEmpirical results indicate that BAPO-trained models outperform many existing models on the AIME 2024/2025 benchmarks.", "questions": "Questions about Eq. (5), which rewrites the standard PPO-style policy gradient:\n* For both terms on the right-hand side, should the leading $\\pi_{\\theta}(y_t)$ terms be replaced with the probability ratio $r_t$? \nIf so, how does this affect the theoretical analysis about entropy later in this work, e.g., Eq. (6)?\n* On the left-hand side, please make explicit that $\\nabla J^{PPO}$ is defined for a specific $x, y_t$, in contrast to the loss $J^{PPO}$ in Eq. (4).\n\n\nQuestions about Eq. (8): \n* Why does the left-hand side contain $\\pi_{rollout}$ terms? \n* Is there a guarantee that the numerator should be smaller than the denominator?\nNote that the denominator sums both positive terms (same as numerator) and negative terms before taking the absolute value.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* This work provides a sensible and relevant analysis of issues commonly encountered in practice.\n* The proposed method is reasonable, practical, and straightforward to implement.\n* Strong model performance on AIME 2024 / 2025 benchmarks.", "weaknesses": "- **Limited evaluation scope.**\nEvaluation are restricted to AIME 2024/2025, each containing only 30 problems. Broader evaluation on more diverse and widely used benchmarks would strengthen the claims.\n\n- **Attribution of improvements is unclear.** \nComparisons are made with prior models trained under substantially different setups, notably in terms of training data. This paper lacks detail on the SFT stage (before RL), making it difficult to determine whether performance gains of the final models stem primarily from BAPO or from the SFT data. Surprisingly, the SFT models (BP-Math-7B/32B-SFT) already outperform baseline models in most cases, as shown in Table 1, suggesting that SFT data might play a major role.\n\n- **Hyperparameter tuning.**\nBAPO introduces 7 new hyperparameters (Line 272). The authors claim these are not finely tuned and that BAPO reduces manual tuning compared to methods like DAPO (for which the upper and lower clip ranges are tunable).\nHowever, two different hyperparameter sets are used for the main experiments in Section 5 and the additional experiments in Appendix B, suggesting that some setting-specific tuning is still required. \nMoreover, Figure 8 shows that the adaptive clip range (determined by BAPO) remains rather stable throughout, raising the question of whether well-tuned GRPO/DAPO could achieve comparable results.\n\n- **Potential exaggeration of instability issues.** \nSome claims and empirical results may overstate the instability caused by data staleness or off-policy RL. In fact, splitting a batch into 4 or 8 mini-batches (corresponding to staleness 4 or 8, if I understand correctly) has been common practice and worked well with standard GRPO/DAPO algorithms in broad settings.\n\n- **Clarify issues about the analysis.**\nSee questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RHx0QFAXNE", "reviewer": "Reviewer_CZZd", "summary": "The paper proposed a new RL loss for llm training which stablize off policy training", "questions": "For a real LLM RL training, it might need to combine multiple tasks. but they might have different extent of imbalance. How will the algorithm tackle that?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Entropy clipping rule is a good contribution\n* the result shows strong improvement", "weaknesses": "* hyper parameter complexity\n* eval task diversity is very limited (only math)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WwfdKxGkKu", "reviewer": "Reviewer_drrk", "summary": "The paper tackles instability in RL for LLMs with respect to a fundamental challenge in off-policy training: data staleness or distribution shift, i.e., the distribution of rollout policy differs from the distribution of the optimization target policy. The paper proposes that the instability mainly comes from two sources: i) imbalanced positive and negative samples, and ii) lack of exploration. To address both issues from empirical practice, the authors propose BAPO, which dynamically adjusts lower/upper importance sampling weight clip bounds to balance the negative and positive tokens. Experiments on reasoning tasks show competitive improvement over open baselines without clip-bound adjustment.", "questions": "See Weaknesses.\n\nSome minor question:\n\n1. From Figure 8, the fluctuation of both bounds across time is very minor compared to the asymmetry between higher and lower bounds. It seems we could also use a stationary asymmetric clip constant across time. Why would practitioners resort to dynamic clipping, which needs to compute the clip constant every step, versus a stationary asymmetric clip? Why are the authors in favor of the first approach?\n\n2. In (8), why do you use $c_{low} = 0$ in the numerator?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. The empirical performance with an asymmetric clip constant is promising.\n2. Some intuition-level justification from the policy gradient contribution perspective and the entropy dynamic perspective is given to pave the motivation.\n3. The paper is quite well-written and easy to follow.", "weaknesses": "I'm somewhat skeptical of both perspectives on the training instability discussed in the paper:\n\n1. From the policy gradient perspective, e.g., Eq. (3), in general RL, both positive and negative samples are important in estimating the value function gradient. Taking out either part will result in biased estimation and harm the performance improvement. On the other hand, if a random minibatch contains only negative samples, the policy can still be improved in expectation. So, the dominance of negative samples does not necessarily prevent learning and lead to training instability or failure.\n\n2. The decrease of entropy also does not directly imply instability of training. The entropy decreases as the policy converges to a deterministic policy, which may or may not be the optimal solution. Therefore, entropy collapse should be expected for all RL training. If the policy converges to the optimal policy, the entropy also collapses, but we should not witness instability or sub-optimality. The instability could come from other places, such as the distribution shift or simply the high variance of using importance sampling. Whether to preserve or decrease entropy is the classic trade-off in exploration and exploitation, which this paper oversimplifies.\n\n3. Essentially, the problem studied in this paper is somewhat artificial and stems from the clip-trick instead of the RL problem itself. In the ideal case, one should clip outlier events symmetrically over the probability measure such that the clipped estimation is still an unbiased estimator of the genuine policy gradient, but symmetric clipping on numerical values fails to coincide, since the irregular distribution of advantages in practice, and potentially depends on the state as well. These shortcomings have been quite well-studied in classic winsorization literature with application to RL, and, quite surprisingly to me, the paper does not discuss any of them. Therefore, I do not see much ideological contribution in the field. See below:\n\n\n4. It seems all the downsides and limitations of PPO-clip mentioned in the paper, i.e., one single trust region for all states, lack of exploration due to fast entropy decrease, could be naturally solved with PPO-KL or TRPO, i.e., posing the trust region on policy instead of the action of each state, and using the trust region to prevent fast entropy decrease. It is very unclear why the authors do not discuss this approach. The paper should include a comprehensive discussion (from theory to empirical evaluation) between dynamic clipping and the policy trust region approaches, to convince the value of their proposed approach. \n\nOrenstein, Robust Importance Sampling with Adaptive Winsorization, 2018\n\nVehtari et al., Pareto Smoothed Importance Sampling, 2015\n\nSu et al., Doubly robust off-policy evaluation with shrinkage, 2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V2KbmhXHiE", "reviewer": "Reviewer_Zvwr", "summary": "The paper proposes a new adaptive and asymmetric clipping for PPO algorithm applied to LLM. Indeed the authors first observe that a common symmetrical clipping $clip(r_t, 1-\\epsilon,1+\\epsilon)$ for some $\\epsilon > 0$ implies a decrease of policy's entropy with its eventual collapse, which prevents from further exploration. They argue that it happens because of misbalance between positive and negative tokens. The final algorithm BAPO adaptively tunes the clipping bounds to include more low probability positive tokens, and potentially reducing the number of negative tokens that are retained for further update steps. This approach is theoretically motivated and is further validated on AIME 2024 and AIME 2025 benchmarks, showing the consistent improvement over GRPO and SFT, demonstrating even some competitive performance compared to some proprietary models.\n\nOverall, the paper is easy to read, providing new interesting results, but it can benefit from more comparisons with other prior assymetric clipping mechanisms. Therefore, I recommend weak accept.", "questions": "- Is there a computational overhead of computing BAPO? How big?\n- How does BAPO perform for longer training, does it only delay the entropy collapse or the entropy is expected to stabilise at some small non-zero value?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is nicely written and in a concise manner. BAPO represents an easy to implement and flexible clipping mechanism for finetuning LLM with RL. It is well motivated, with some theoretical insights, and its advantage over a manual symmetrical clipping is evident. Their models trained with BAPO achieve on-par results on AIME 2024 and 2025 compared to some proprietary algorithms.", "weaknesses": "Despite that the paper mentions other works that consider asymmetric clipping mechanisms, like TOPR, DAPO, and even dynamic clipping — DRPO, there is no systematic comparison between different methods for fixed original models like BP-Math-7B or BP-Math-32B. The only baselines that the authors are comparing to are GRPO, SFT and a single fixed asymmetric clipping with strictly predefined bounds. \n\nMinor:\n\n- L141: provide the specific function for A_t that you use\n- Figure11: why not checking [0.8, 1.5] as in figure 7?\n- L841: provide a more precise definition for $z_{y,x}$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)