PROS: Towards Compute-Efficient RLVR via Rollout Prefix Reuse
Authors: Baizhou Huang
Xiaojun Wan
Venue: ICLR 2026 Poster (2026)
Keywords: RLVR
reasoning
Primary area: foundation or frontier models, including LLMs

Abstract: Large reasoning models (LRMs) trained with *Reinforcement Learning with Verifiable Rewards* (RLVR) have achieved remarkable progress on complex reasoning tasks. However, RLVR heavily relies on on-policy rollout generation, whose cost grows rapidly with rollout length and model size, eventually becoming the training bottleneck. Our empirical analysis reveals that independent rollouts for the same query often share similar early steps, indicating substantial redundancy. To address this, we propose **Pros** (**P**refix **R**euse for **O**n-policy **S**ampling), a paradigm that reuses promising prefixes of historical rollouts in RLVR training. **Pros** appends these self-generated partial rollouts to the original queries to form *Augmented Queries*, which are then used as regular training inputs in subsequent iterations, thereby reducing redundant computation. To select training batch from augmented queries, **Pros** adopts a hierarchical Bayesian model to estimate their pass rates and prioritize those with the highest reward uncertainty. Experiments across diverse settings show that **Pros** consistently improves training efficiency and achieves higher accuracy than strong baselines. These results highlight **Pros** as a practical path toward scalable and compute-efficient RLVR.

Published as a conference paper at ICLR 2026
PROS: TOWARDS COMPUTE-EFrICIENT RLVR VIA ROLLOUT PREFIX REUSE
Baizhou Huang& & Xiiaojun Wan Wangxuan Institute of Computer Technology, Peking University (hbz19 9, wanxi jojunj jojjunekkuu.... @ew.e.du.......eeee.. on
ABSTRACT
Large reasoning models (LRMs) trained with Reinforcement Learning with Veri- fiable Rewards (RLVR) have achieved remarkable progress on complex reasoning tasks. However, RLVR heavily relies on on on-policy rollout generation, whose cost grows rapidly rapidly with rollout length and model size, eventually becoming the training bottleneck. Our empirical analysis reveals that independent rollouts for the same query often share similar early steps, indicating substantial redundancy. To ad- dress this, we propose PROS (Prefix Reuse for On-policy Sampling), a paradigm that reuses promising prefixes of historical rollouts in RLVR training. PROS ap- pends these self-generated partial rollouts to the original queries to form Aug- mented Queries, which are then used as regular training inputs in subsequent it- erations, thereby reducing redundant computation. To select training batch from augmented queries, PROS adopts a hierarchical Bayesian model to estimate their pass rates and prioritize those with the highest reward uncertainty. Experiments across diverse settings show that PROS consistently improves training efficiency and achieves higher accuracy than strong baselines. These results highlight PROS as a practical path toward scalable and compute-efficien RLVR.
INTRODUCTION
Large reasoning models (LRMs) have achieved remarkable progress on complex tasks, especially in code generation and mathematical problem sol in some cases even surpassing average human performance (OpenAl et al.] 2024 Shao et al. 2024] DeepSeek-Al et al., 2025 Team et al. 20256 Lambert et al.) 2025). A key driver behind these advances is Reinforcement Learning with Verifiable Rewards (RLVR), where an LRM is optimized as a policy that generates chain-of-thouge thouhhss (CoTs) (Wei al. 2023) trajectories and receives binary rewards on final answers from deterministic ver- ifiers. This paradigm enables models to improve reasoning abilities via supervision from verifiable outcomes, offering scalable path for learning from self-exploration.
Despite its promise, the RLVR paradigm is constrained by heavy reliance on on-policy rollout gen- eration. Each training iteration of RLVR algorithm consists of four steps: Select queries from given dataset as a training batch, Generate several rollouts based on selected queries with the policy, Verify these rollouts with a deterministic verifier to obtain binary rewards, and Update parameters of the policy to increase the likelihood of high- actions via policy-gradiens methods such as PPO (Schulman et al. 2017) or GRPO (Shao et al., 2024). As models tackle increasingly complex problems, the length of their Mlain-of-thougee of online rollout generation escalates and soon dominates the overall training time, making one of the principal obstacles to further scaling RLVR.
(CoT's) grows accordingly. Consequently, the cost
Our empirical analysis reveals that, for a given query, independently sampled reasoning trajectories can be naturally organized into tree structure like branching search over the answer space as shown in Figure []] Although their final answers may diverge, the early steps often share similar lines of reasoning. This observation indicates substantial redundancy: by reusing the prefixes of historical rollouts in the previous iteration, we can avoid repeatedly generating these near-duplicate initial steps and thereby saving significant amount of computation.
Based on this observation, we introduce PROS (Prefix Reuse for On-policy Sampling), a paradigm designed to make RLVR more compute-efficient for further scaling. In each training iteration, PROS

---

Published as a conference paper at ICLR 2026
Query
Rollout Step
Rollout Step 2
Iteration 10
Losing positions toloww:
Alcc and Boo play the tollowiin game A stack of tokens liss before them. The players tabe turns with Alice going first, On each tum, the player removes etheri takken or toliens from the stack Whoever romoves the lest
Iteration 20
Aralyze lonar
O(wir),
((sse)
Iteration 30
3(wūz), 40uiin].a S(use)
Analyee mure Lanae: x 6(wix). a [ a 2(oos
Answer:
404
X
reletion:
Iteration 40
Suppose indicnts is losing postiors 1(6) -Mfs 1)
Whitee cod:
pecudo
rumber integers
equal
Iteration
Find the palleen:
Answer: 809
there wosts stretngy that guar guarantees that Boo will win the game regardless of Allee's play.
position
Aralyze
((wis) n 1(lse), 2(wis) M(inoe.. 4(Jone) S(win)
Iteration 6D
Analyte more canme:
G(lase) 10400) 2(lose)
Figure I: Rollouts for a query from different training iterations can naturally be organized into a tree structure, where their reasoning trajectories are highly similar in early steps and gradually diverge.
identifies high-quality prefixes of generated rollouts using readily available signals such token- level entropy or value estimation from a critic. These rollout prefixes are then appended to their original queries construct Augmented Queries. An augmented query preserves the task semantics of the original query while adding partial reasoning steps produced by the policy itself. It can be fed back into subsequent training iterations as a prompt, shortening on-policy generated reasoning steps required and thus reducing overall training compute. Beyond efficiency, PROS also enables a form of cross-iteration search pruning: it allows the policy exploit those high-quality initial steps while avoiding useless early dead ends, thereby steering subsequent reasoning towards promising directions. This reallocates training compute to meaningful exploration in later reasoning steps, enabling the policy to acquire richer reasoning patterns.
As training progresses, more augmented queries are generated, and the dataset expands into a two- layer tree-structured hierarchy: each original query serves as parent, and its derived augmented queries serve as children. A central challenge is selecting which augmented queries to train on. Bae al. (2025) give principle that queries with the highest reward variance exhibit the greatest learnability and provide the most informative training signals. Motivated by this, we instantiate a hierarchical Bayesian model over augmented queries, estimate per-query pass rate from historical
reward statistics and prioritize those queries with high reward uncertainty (i.e. pass rate close to 0.5)
(Hon al. 2022a
Essentially, PROS can be seamlessly integrated into most policy-gradient RLVR algorithms as a plugin. With the main algorithm logic unchanged, it only adds two additional steps, augmented query construction (Section 3) and augmented query selection (Section 4 both of which incur negligible computational overhead. In summary, this paper makes the following contributions:
We propose PROS, a simple and general training paradigm that reuses high-value histor- ical prefixes Augmented Queries, reducing redundant generation and enhance policy exploitation.
We further introduce a selection mechanism over all augmented queries that targets at the most valuable one for training, which implemented via pass rate estimation by hierar- chical Bayesian model.
PROS consistently outperforms strong baselines across diverse settings, achieving the best performance cost trade-offs. Integrated as plugin, it further raise the upperbound of both PPO and GRPO, yielding an average improvements of +3.96 and +6.21 points on AIME24 and AMC23, respectively.
2 TRAINING BOTTLENECK ANALYSIS
RLVR Basics We begin by reviewing the process of Reinforcement Learning with Verifiable Re- wards (RLVR) for training large reasoning models. For simplicity, we describe the setting with batch size 1, and denote the policy by иe- In each training iteration, query g is selected from a training dataset, and na autoregressively produces a chain of thought y = (y1,...,97 condi- tioned on q. A deterministic verifier serves as environment mapping (4, v) binary reward
2

---

Published as a conference paper at ICLR 2026
(a)
(b)
(c)
0.675
Fraction Generation Time
0.650
Normalized Edit Distance
0.8
0.625
0.600
0.6
Rouge-1
0.7
0.575
0.4
0.6
0.550
0.2
0.5
0.4
0.500
0.3
0.475
2k
4k
6k
8k
16k
50
100
200
s60
1000
50
100
200
sóo
1000
Length
Length
Length
Figure 2: (a) Fraction of rollout generation time versus different max rollout length under fixed batch size and hardware. (b) Pairwise normalized edit distance versus prefix window size averaged 64 rollouts per query. Higher value indicates lower similarity. (c) Self-rouge versus prefix window size averaged over 64 rollouts per query. Higher value indicates higher similarity.
r R(q.y) E {0, 1]. indicating whether the final answer matches the ground truth or satis- fies a task-specific specification (e.g. passing all unit tests for code generation). The objective of RLVR finetuning to optimize the policy parameters maximize the expected rewards, i.e. maxg bg R(g. y)] which is commonly implemented via policy-gradient methods like PPO or GRPO.
Generation Share. As both the scale of LRMs and the length of CoTs grow, on-policy rollout generation gradually dominates the training time RLVR algorithms. We perform preliminary measurement to quantify the generation share (i.e. the fraction of wall-clock time spent in rollout generation) under varying max rollout length Lmax- As shown in Figure (a). the generation share rises rapidly as the maximum generation length increases. Currently, the scale of LRMs have already approached hundreds of billions, with context lengths more than hundreds of thousands tokens. This trend highlights scalability issue, underscoring the need for methods alleviate generation costs.
Redundancy in Early Reasoning Steps. We empirically observe strong similarity at the begin- ning different rollouts for the same query across training iterations. Their reasoning trajectories naturally unfold like branching search: the early steps share similar setup, while later steps grad- ually diverge for different reasoning directions. specific case is presented in Figure ! To fur- ther quantify this effect, sample 64 independent rollouts per query the DAPO-Trail dataset Yuu et al.] 2025) and measure pairwise similarity on truncated prefixes of different lengths. Con- cretely, we compute the average of two pairwise similarity metrics: (i) normalized edit distance (i.e. EditDist/ TotalLen), and (ii) Rouge-L (Lin 2004). As shown in Figure 2 shorter prefixes yield markedly higher similarity, indicating substantial redundancy repeatedly generating near- duplicate initial reasoning steps.
Wasted Computation in Early Dead Ends. A second source of redundancy also follows from the tree structure of reasoning trajectories. Correct solution trajectories are sparse in the reasoning space. Naive multi-sampling may therefore start on branches that are truly dead ends. Subsequently, their remaining suffix contribute little training value. For example, in Figure [ rollouts in iterations 10 to 30 make a mistake at the first step and still consume long suffixes. This unnecessary dead-end cost compounds model scale and sequence length grow.
AUGMENTED QUERY CONSTRUCTION
3.1 OVERVIEW
Building on the observations from Section 2 we introduce PROS (Prefix Reuse for On-policy Sampling), a paradigm designed mitigate redundant generation and wasted computation in RLVR.
3

---

Published as a conference paper at ICLR 2026
(A) PROS modules
(B) RLVR standard pipeline
Promiing Pigft Merrifaatton
Treving Batch fiwendd
(4) Update
(5) Construct AQ
connect quer
denathy busti"
Creca
plcl
(3) Verify
Vartly the arower
Dattust Froe Strustvve ourry to dence
Traksig Rarch Roliomt (r)
catly" (
edtirote poss oie soch sugmented quary
(2) Generate
ot on-splky rolouti
LLM
Alsr-lote dsteoaatinn
Praiing Eetth (t') atth 0.5)
(1) Select AQ
e't9, - 06)
Original Queryy Augmented Query
0.9
O6
welect cuery with poor chou ae
e'th - 96)
Hollout
05
Figure 3: Overview of PROS. The right-hand side follows standard RLVR algorithms, while the left-hand side is the augmented query modules introduced by PROS. (1) Estimate the pass rate 0 of each query and select those with highest uncertainty; (2) Generate rollouts {y for selected queries }
J: (3) Verify rollouts against ground truths: (4) Update the policy with rollouts rewards: (5) Construct augmented queries by appending high-quality rollout prefixes to original queries (e.g. 4 concal(g", y (")) for future reuse.
Concretely, in each training iteration, after the rollouts are generated and verified, we identify high- quality prefixes within them. We then append these valuable prefixes to their original queries cre- ate new Augmented Queries (AQ), which are then added to the training set as new query instances for subsequent training iterations. Over time, this augmentation process expands the training data into a two-layer tree-structured dataset: each original query serves as a parent node, and its derived augmented queries become child nodes.
During training. augmented queries are treated equivalently as normal queries. The policy generates continual reasoning trajectories conditioned on both the historical prefix and the original query. The reinforcement learning update then proceeds over the collected continuations as in standard RLVR. The only difference that both credit assignment and gradient update are conducted only on the newly generated continuations without the reused historical prefixes. This way, the policy is reinforced for how it continues from the prefix, preventing overfitting to the previously generated reasoning steps. Figure 3 illustrates the overall PROS pipeline.
Notably, an augmented query preserves the task semantics of original query, only providing unverified partial reasoning steps generated by the policy itself. It does not reveal any final answer and retains the on- on-policy nature of the RLVR algorithm, in contrast to off policy experience replay
methods (Fedus et al. 2020; Schaul al. 2016 Liang et al. 2021
3.2 PROMISING PREFIX IDENTIFICATION
A crucial question in PROS is identifying which rollout prefixes are worth reusing. We employ heuristic strategy based on both policy uncertainty and value signals, along with a length-based constraint, to construct augmented queries that are both informative and efficient.
We consider two signals to estimate a prefix's potential: uncertainty- signal. As the uncertainty signal, we use the token-level entropy along the rollout, which serves as a proxy for uncertainty. A higher entropy indicates that the policy was less certain about the next steps, suggesting the prefix lies in an unexplored and informative region of the reasoning space. Such prefixes are candidates for reuse because they encourage exploration and mitigate over-confidence (Cui al., 2025 Wang al. 2025). Using entropy is almost free without extra computational overhead, since the RLVR algorithm already requires a forward pass to compute the log- probbabiiyy every timestep.
signal and value-based
4

---

Published as a conference paper at ICLR 2026
Besides entropy, we also leverage the readily available value function learned by critic model for those actor-critic methods (c.g. PPO). Prefixes with higher predicted value are more likely to lead to correct final answer. So reusing them can help to exploit and focus computational costs promising reasoning directions.
In practice, reuse the prefix y/: [-] = [vo...... V/- with the highest token-level entropy from each correct rollout y, i.e. t" = arg maX Entropy(#(-Ib/))
9)). When the underlying RLVR algorithm is actor-critic, we additionally use the value function as filter, restricting to the top 10% of timesteps with the highest predicted values. In addition to these signals, we also impose sim- ple length constraints to seek balance between exploitation and exploration. Specifically, we limit the range of t" E [HT, (T). Although coarse, this constraint ensures that the reused prefix yields a non-trivial reduction generation time, while leaving sufficient room for the policy exploration without giving away too much of the solution.
4
AUGMENTED QUERY SELECTION
4.1 OVERVIEW
As the augmented query dataset grows continuously across iterations, selecting valuable training batch from a large amounts of AQs is non-trivial. Prior studies suggest that train- ing is most effective when queries fall into an intermediate dif- ficulty range rather than being trivially easy or impossibly hard (Yu et al. 2025 Razin et al., 2025 Vygotsky 1978). Further- more, Bae et al. 2025) theoretically demonstrates that queries with rollout pass rate close to 0.5 exhibit the strongest learn- ability signal. Both Yu et al. (2025) Bad et al. (2255) adopt an online filtering mechanism, removing queries whose pass rates are too high too low after on-policy rollout generation. However, such online filtering requires costly rollout genera- tion for every query. To reduce this overhead, we instead es- timate pass rates of augmented queries from historical obser- vations on rewards via Bayesian inference, and we prioritize those queries with estimated pass rates near 0.5.
Ben(Sigmaid((.))
(enn(Sggmdd((
N (V
V (s parent
VIES,
Figure 4: Overview of graphical model. indicates reward of each query, following a Bernoulli controlled by log odds v). iy; of each augmented query i is conditionally Gaussian cen- tered its parent's 1y.. Sy indicates the children set node j
As stated before, the augmented query dataset forms a tree-structured hierarchy, where each original query serves parent while its derived augmented queries serve as children. To leverage the relationship between original query and its derived augmented queries, we initialize a two-layer logit-normal Bayesian model for pass rate estimation (Hong et al. 2022a b). Let pass rate 0 sigmoid($)) = + exp(-t))-- we suppose the log-odds of each parent par follows U'par ~ V(H,+²) and the log-odds of its children is conditionally Gaussian, vi | Upar NV 2)). Each time query is used for generating a rollout, binary reward r ~ Bern(O) is then observed, indicating whether the rollout passes verification. The graphical model is presented in Figure
This formulation encodes the inductive bias that augmented query solves the same underlying problem as its parent but begins from an intermediate reasoning state, so child's 's pass rate should be correlated with that of its parent. The hierarchical structure enables information sharing between related queries and improves statistical efficiency.
Building the Bayesian model above, we can infer the posterior distribution of latent log-odds by incorporating historical reward observations. Posterior samples { are then drawn for each query, and the corresponding pass rates 0 = a (v²) estimated for training batch selection.
4.2 BAYESIAN INFERENCE WITH PÓLYA-GAMMA AUGMENTATION
In order to sample from the joint posterior, we derive the full conditional distributions for both parents and children, which allows Gibbs sampling. Specifically at iteration t, let H, = (n, 8,) denote the reward history of query i, where n; the number of times the query being selected and s; the number of success times of its corresponding rollouts. Suppase the parent set of nodes is F, and the children set of node j is Sy. By Bayes' rule, the joint posterior admits hierarchical chain
5

---

Published as a conference paper at ICLR 2026
factorization:
H( & II JEl
p(
x
H
x
x
p (HH, (p:)
prior of parent
likelihood parent
of child
likelihood of child
key challenge in posterior calculation above is the non-conjugacy of the Bernoulli likelihood and Gaussian prior. We address this by introducing Pólya-Gammy (PG) auxiliary variables following Polson al. (2013); Dumitrascu al. (2018), which render the calculation tractable.
We first derive the conditional posterior p( H;, par) of a child i, given its parent Upar and history H, (n,, s;).
oxple - (HHp)))))(() (pap)) )2 / exp(x1)
whyh ppo |ni, O)ww;
where K. 2 and PG(n,, 0). The identity follows directly from Polson et al. 2001 Theorem 1). By introducing an auxiliary variable wi. the conditional posterior p(vy H,, Upper Wa) becomes Gaussian and thus tractable, given that the prior p( t/s par) ) is Gaussian. In particular, we have the following proposition:
Proposition 4.1. Given the log-odds of parent Upr and the history H; (1) 8;).
1
Wa H..,H,-PG(n.th 4/s l0,, H4g U/r ~ (m,,V) where V o wwi - V- 22 +x)
Similarly, we introduce a PG variable wpar for each parent tpar and derive its conditional posterior with the same augmentation method:
Proposition 4.2. Given the log-odds of all children (wa Spor and the history Hpar = (Mpar. Spor) )
Wpar ppprr par
PG (G Wpar)
U'er Wwarr Hpars {wh Bpar N(m, V)
where V 2r-2 + Wpor
S Sparlo
2)-1 m V (T ( "E the Kiprr)
Proposition and together define simple Gibbs sampler for the joint posterior p([t), {w) H) from which we can efficiently sample v and estimate pass rates for all augmented queries. The overall estimation process is presented Algorithm 1. Detailed proofs in this section are presented in Appendix B
Algorithm Pass Rate Estimation for Augmented Query
Input:
Log-odds of all queries from previous iteration (w) History of all queries {n,,)); Number of Gibbs sweeps G; Model Hyperparameters J,T,0
to
// One Gibbs sweep foreach parent CF do
foreach child e do
// Posterior sampling based on Proposition
sample PPmpl,
( (o - V- 2 Spar +k), sample 4i wim.(m
end
Posterior sampling ng based on Proposition 4
sample Wpar~ PGr ):ar); (t +2 /p Kpar), sample ((r,)
m 2
end end
4.3 EXPONENTIAL FORGETTING FOR NON-STATIONARY ENVIRONMENT
As training progresses, the policy evolves continuously, which leads to gradual shifts the pass rate of queries. Therefore, introduce exponential forgetting mechanism. At each iteration,

---

Published as a conference paper at ICLR 2026
Table Comparison between PROS and other baselines under different settings. #Tokens denotes the average number of tokens generated per iteration (in millions), and #Time denotes the average GPU wall-clock time per iteration (in minutes).
DAPO-Train
AIME-OId
AIME24
AMC23
#Tokens
#Time
AIME24
AMC23
#Tokens
#Time
PPO
28.23
73.63
10.24
9.17
31.35
57.85
4.39
6.06
w/ Dynamic Sampling
28.85
66.69
11.07
17.62
33.96
69.21
9.10
40.04
w/ Experience Replay
30.10
73.02
8.93
8,98
31.35
57.85
3.61
5.88
Priority Sampling
31.87
73.25
11.35
9.76
31.46
60.82
4.15
5.91
w/ Pros-ablation
31.35
73.93
7.92
9.29
31.87
65.09
6.93
8.89
w/ PROS
33.23
78.20
8.49
9.77
34.27
65.62
5.00
8.09
GRPO
29.58
73.40
9.27
6.58
31.04
59.98
3.58
3.77
w/ Dynamic Sampling
31.15
76.68
11.16
19.45
29.58
68.90
9.22
45.38
w/ Experience Replay
29.90
74.62
7.60
6.36
31.15
59.98
3.14
4.14
Priority Sampling
30.73
73.70
6.87
33.44
60.14
3.88
4.33
PROS-ablation
30.52
75.91
7.00
33.44
63.41
5.88
6.26
w/ PROS
34.17
78.28
8.46
7.57
34.38
67.53
5.84
6.69
Table 2: Comparison between PROS and other baselines Qwen3-4B trained with AIME-Old.
Table Ablation study on hyperparameters by varying a and X. PROS consistently surpasses Vanilla (59.98 on AMC23; 31.04 on AIME24).
AIME24 AMC23 #Time
0.99 (default)
0.3 (default)
PPO
21.25
52.59
5.83
AMC23
AIME24
AMC23 AIME24
w/ dynamic
25.73
61.19
10.83
w/ replay
23.75
53.73
5.81
0.10
64.25
34.79
0.95
65,62
35,94
w/ prior
24.90
60.44
5.86
0.20
62.42
36.98
0.99
67.53
34.38
w/ PROS
27.40
62.12
6.21
0.30
67.53
34.38
0.995
66.77
33.44
scale down the historical statistics s; and n, of every augmented query by forgetting factor E (0, 1) before incorporating new observations. This exponential forgetting ensures that more recent rewards exert greater influence on posterior updates, allowing the sampler to adapt to policy improvements.
5 EXPERIMENTS
We conduct main experiments the Qwen3-8B model, trained using PPO (Schulman et al.) 2017) and GRPO (Shao et al., 2024). For evaluation, we report Pass@ / on two benchmarks: ATME 2024 and AMC 2023. To mitigate variance, we report averages over 32 and independent runs on these datasets, respectively, following Hochlehnert et al. (2025).
Training configuration. We adopt math reasoning datasets training corpora: DAPO-Trait is a large and diverse corpus covering broad domains of math problemst Yu et al. (025): AIME Old consists of all AIME problems prior 2024, which is more relevant the AIME24 benchmark. We train for 400 and 300 iterations on these datasets, respectively. At each iteration, we generate rollouts per query. The batch size 512 with mini-batch size 64, yielding gradient updates per PPO epoch. The maximum response length is set to 6144 tokens.
Baselines. We compare our proposed PROS against several strong baselines: (1) Vanilla stan- dard PPO/GRPO training algorithm. (2) Dynamic Sampling (Yu et al.) 2025) constructs training batches by filtering queries whose on-policy rollouts are either correct or all incorrect. (3) Pri- ority Sampling (Team et al. 2025b a) tracks the historical pass rate Ô each query and samples proportionally to 1 B. (4) Experience Replay augment PPO/GRPO with replay buffer with replay ratio set to 1 1/8. We also apply truncated importance sampling following ACER (Wang al.) (207)). (5) PRos-abi ablation: a variant of PROS with query being randomly sampled, isolating the effect of proposed augmented query selection mechanism. Additional details provided in Appendix C

---

Published as a conference paper at ICLR 2026
DAPO-Train
AIME-Old
650
Sumpling Paarly
680
65
AMC23 (Pas3@11
60
350
523
30
20
30
40
50
60
70
80
16
30
40
50
78
GRRP
55.
66
30
20 30 60
50
60
10
15
20
25
35
40
GPU Wall-clock (hour)
Figure 5: Training efficiency comparison among different methods. The z-axis denotes the training time (GPU hours), and the y-axis reports the best performance achieved up to that time.
MAIN RESULTS
Overall performance. Table [ summarizes the comparison between PROS and other baselines. PROS delivers consistent improvements across all four settings. Under PPO trained on DAPO- Train, it outperforms the vanilla baseline by +5 and achieves the best performance on both AIME24 and AMC23, Comparable gains are observed under GRPO. These results validate our hypothesis: reusing promising prefixes enhances exploitation, thereby improving overall performance. In con- trast, Experience Expseiexe Replay improves efficiency but yields performance comparable to vanilla. Dynamic Sampling achieves strong results, particularly on AMC23, but requires 2-4x more wall-clock time than PROS due to repeated rejection sampling. By comparison, PROS attains strong performance with only modest computational overhead. Most of the extra cost comes from its length scaling behavior (see below), which yields longer CoTs. To isolate the influence of augmented query sam- pling, we compare PROS with the PROS-ablation variant. The results show that prefix reuse alone already improves compute efficiency and achieves better performance. Moreover, incorporating the proposed augmented query selection mechanism consistently provides additional gains. We also provide additional experiments Qwen3-4B in Table which exhibit similar trends.
Performance-Cost trade-offs. To further assess the compute efficiency of different methods, we analyze the performance growth with respect GPU wall-clock time in Figure It is shown that PROS consistently exhibits better efficiency under different settings. Compared to dynamic sam- pling, which achieves competitive performance but prohibitive cost, PROS provides a significantly more favorable balance between efficiency and performance.
Influence of Hyper-parameetrss We conduct an ablation study evaluate the robustness of PROS with respect to key hyperparameters. In the Bayesian model (Section 4.1), the child prior db Upar (e o²) introduces variance parameter x² that controls similarity between parent and child nodes, while the temporal decay factor À governs the rate of forgetting in pass rate estima- tion. We train PROS with GRPO on the AIME-Old dataset, varying both a and X. Table reports @ass@1 1 results, and Figure 7 shows the estimation error of pass rates. The results show that PROS consistently brings improvements to vanilla GRPO across hyperparameter settings, and the pass rate estimation remains robust, becoming increasingly accurate training proceeds.
Length scaling. A key property of RLVR algorithms is their ability to benefit from increased rea- soning lengths, which enables improved test-time scaling (Snell al. 2024). We present the length scaling trends of different methods Figure (6) Across all settings, the rollout length PROS continues to scale up the training goes. By contrast, vanilla PPO/GRPO, well their experi-
8

---

Published as a conference paper at ICLR 2026
Training Rollout Length
3500
3500
3030
2500
2500
2000
Alkapalinca teply
1500
1500
1000
50
100
150
200
250
300
350
400
50
100
150
200
250
300
#Traiiing Iteration
Figure 6: Length scaling trends of different methods under different settings.
Estimation Error of Rate
0.2
0.3
0.995
0.20
0.20
0.15
0.15
50
100
150
200
250
300
50
100
150
200
250
300
#Training Iteration
Figure 7: Average pass rate estimation error in PROS with different hyperparameters.
ence replay and prioritized sampling variants, suffer from length collapse the AIME-Old dataset, limiting their ability to further exploitation.
7 RELATED WORK
Our proposed method focuses on improving efficiency of rollout generation in RLVR. In traditional reinforcement learning, a common approach to reduce the cost of on-policy rollout generation is experience replay (Fedus al., 2020: Schaul et al., 2016 Liang et al. (021). For instance, PPO typically reuses the same batch of rollouts for multiple training epochs. However, He et al. (2025) show that, due to the complexity of large language models, repeatedly reusing the same data quickly leads to overfitting and entropy collapse, which prevents RLVR from scaling effectively in the long run. Another line of related work reduces task difficulty by providing partial reference solutions as hints during training, which requires costly human annotation(Xi et al. contrast, our approach derives partial solutions directly from the model own past rollouts, removing the need for external reference solutions in complex reasoning settings. Thirdly, our augmented query selection also draws inspiration from curriculum learning (Soviany et al. 2022 Narvekar al. 2020; WWan al. 2021), by dynamically selecting training queries whose difficulty best
2024 Liu et al. 2025a b). In
matches the current policy. Several concurrent studies also investigate online data selection in RLVR
improve performance (Sun et al. 2025 Bae et al. 22055] Zheng et al., [2025] Qu al. 2025). Our
method differs in that it is specifically designed for the hierarchical augmented dataset.
8 CONCLUSION
We presented PROS, a prefix-reuse paradigm for RLVR training that constructs augmented queries from historical rollout and leverages hicrarchical Bayesian model for uncertainty-awe aware selection. Experiments demonstrate its consistent improvements in both efficiency and accuracy compared to strong baselines, highlighting the promise of PRos for compute-efficient training of LRMs.
While effective, our current design of augmented queries still simple, relying on entropy and value signals that may introduce bias. Future work could investigate richer prefix identification, explore adaptive integration with other forms of uncertainty estimation, and extend prefix to broader domains beyond mathematical reasoning.
principled criteria for

---

Published as a conference paper at ICLR 2026
9 ACKNOWLEDGEMENT
This work was supported by Beijing Natural Science Foundation (L253001), Key Laboratory of Sci- ence, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Tech- nology) and National Engineering Research Center of New Electronic Publishing Technologies. We appreciate the anonymous reviewers for their helpful comments. Xiaojun Wan the corresponding author.
ETHICS STATEMENT
This work focuses improving the compute efficiency of reinforcement learning with verifiable rewards (RLVR) for large reasoning models. All experiments are conducted on publicly available datasets (AIME, AMC, DAPO) that do not contain sensitive personal information. Our approach does not introduce new data collection, and we adhere to the original licenses and intended use of these datasets.
We believe this work raises no additional ethical concerns beyond those already inherent in the study of large language models and reinforcement learning. We emphasize that our contributions are intended solely for research, and caution should be exercised when transferring these methods real-world applications.
REPRODUCIBILITY STATEMENT
All experiments in this paper are conducted on publicly available datasets (AIME, AMC, and DAPO), which can be readily accessed and downloaded. Complete proofs of the theoretical proposi- tions are provided Appendix B Detailed descriptions of experimental settings, including training configurations, hyperparameters, and baseline implementations, are provided in Appendix C to fa- cilitate replication. The full source code repository together with reproducibility scripts is included in the supplementary materials.
REFERENCES
Sanghwan Bae, Jiwoo Hong. Min Young Lee, Hanbyul Kim, Jeong Yeon Nam, and Donghyun Kwak. Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning, April 2025. URL
03380 arXiv:2504.03366 [cs].
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. The Entropy Mechanism Reinforcement Learning for Reasoning Language Models, May 2025, URL htttp://arxi..orggge wwtp:////.aabbb....//////// arXiv:2505.22617 [cs].
DeepSeek-AL, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai. Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao. Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li. Jiawei Wang, Jingchang Chen. Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan. S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei. Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong
10

---

Published as a conference paper at ICLR 2026
Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang. Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha. Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song. Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. DeepSeek-R1: Incentivizing Reasoning Capability in LLMS via Reinforcement Learning, January 2025. URL (tttttt://aan http://...ggaa///550.1.. arXiv:2501.122 12948 [es].
Bianca Dumitrascu, Karen Feng, and Barbara E Engelhardt. Pg-ts: Improved thompson sampling for logistic contextual bandits, May 2018.
arXiv:1805.0744 074558 [stat.ML].
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay, July 2020. URL
httpp::/aaat //axxiv org/ otg/abs/////oooooooooogoo/...ear 2007. 067000 arXiv:2007.06700 [cs.LG].
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li. Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou, Skywork open reasoner technical report, May 2025. https:
//arxiv.org// /200. 22312 arXiv:2505.22312 [cs.LG]]
Andreas Hochlehnert, Hardik Bhatnagar, Vishaal Udandarao, Samuel Albanie, Ameya Prabhu, and Matthias Bethge. A Sober Look Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility, April 2025. URL arXiv.2504.07086 [cs] version: 1.
Joey Hong, Branislav Kveton, Sumeet Katariya, Manzil Zaheer, and Mohammad Ghavamzadeh. Deep hierarchy in bandits, Feb 2022a. URL
arXiv:2202.01454 [cs.LG].
Joey Hong, Branislav Kveton, Manzil Zaheer, and Mohammad Ghavamzadeh. Hierarchical bayesian bandits, 2022b. URL wttptttts//// http://////axvv.r///bb.ss
06929.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Ma- lik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Ha- jishirzi. Tulu 3: Pushing fronticrs in open language model post-training, November 2025. URL https: /ww.vvvvvvv/..////////.a//...av. / abs 241 15124 arXiv:2411 1. 15124 [cs.CL].
Xingxing Liang, Yang Ma. Yanghe Feng, and Zhong Liu. Ptr-ppo: Proximal policy optimization with prioritized trajectory replay, December 2021.
03798 arXiv:21 12.03798 [cs.L.G].
Chin-Yes Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcclona, Spain, July 2004. Association for Computational Linguis-
tics. /wR.//ttpps./aalaathhlllle wwtpll/gy....g/0000/o 10033
Mingyang Liu, Gabriele Farina, and Asuman Ozdaglar. Uft: Unifying supervised and re-
inforcement fine-tuning, May 2025a. URL /ttppp/s//axxvv.o////bbsss/
arXiv:2505.1 16984 [cs.I.G].
Ziru Liu, Cheng Gong, Xinyu Fu, Yaofang Liu, Ran Chen, Shoubo Hu, Suiyun Zhang, Rui Liu, Qingfu Zhang, and Dandan Tu. Ghpo: Adaptive guidance for stable and efficient Ilm reinforce- ment learning, 2025b. TtR.https:://// wttp:/awwww.....//////////../...
10688
11

---

Published as a conference paper at ICLR 2026
Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey, March 2020. URL httth://aaxiiii org/ absa 04960 urXiv:2003.04960 [cs.LG].
OpenAI, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alee Helyar, Aleksander Madry, Alex Beutel, Alex Careey, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghor- bani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lugaresi. Cary Bassin, Cary Hudson, Chak Ming Li. Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse. Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong, Eliz- abeth Proehl, Enoch Cheung, Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclere, Hadi Salman, Haiming Bao, Hao Sheng, Hart An- drin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman. Hyung Won Chung, Ian Kivlichan, Ian O' O'Connell, Ian Osband, Ignasi Clavera Gilaberte, Ilge Akkaya, Ilya Kostrikov, Ilya Sutskever, Irina Kofman, Jakub Pachocki, James Lennon, Jason Wei, Jean Harb, Jerry Twore, Jiacheng Feng, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joaquin Quiñonero Candela, Joe Palermo, Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga, Julie Wang, Kai Chen. Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang. Leo Liu, Leon Maksin, Leyton Ho, Liam Fedus, Lilian Weng, Linden Li, Lindsay McCallum, Lindsey Held, Lorenz Kuhn, Lukas Kon- draciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglekar, Mark Chen, Marko Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yatbaz, Melody Y. Guan, Mengyuan Xu. Mengyuan Yan, Mia Glaese, Mianna Chen, Michael Lampe, Michael Malek, Michele Wang, Michelle Fradin, Mike McClay, Mikhail Pavlov, Miles Wang. Mingxuan Wang, Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowdhury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofrr Nachum, Oleg Boiko, Oleg Murk, Olivia Watkins, Patrick Chao, Paul Ashbourne, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes, Raz Gaon, Reah Reimar Leike, Renny Hwang, Rhythm Garg, Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan Greene, Saachi Jain. Sam Altman, Sam Toizer, Sam Toyer, Samuel Miserendino, Sandhini Agarwal, Santiago Hernandez. Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Hu, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuan Zhang, Siyuan Fu, Spencer Papay, Steph Lin, Suchir Balaji, Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang, Taylor Gordon, Ted Sanders, Tejal Patwardhan, Thibault Sottiaux, Thomas Degry. Thomas Dim- son, Tianhao Zheng, Timur Garipov, Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peterson, Tyna Eloundou, Valerie Qi, Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong, Vlad Fomen, Weiyi Zheng, Wenda Zhou, Wes McCabe, Wojeiech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen He, Yuchen Zhang, Yunyun Wang. Zheng Shao, and Zhuohan Li. Openai ol system card, December 2024. URL htttps///arrdv..oo/////////1/11 LtXiv:2412.1677 [cs.AI].
Miyara,
Nicholas G. Polson, James G. Scott, and Jesse Windle. Bayesian inference for logistic models using polya-gamma latent variables, May 2013. URL arXiv: MEE.
Yun Qu, Qi Cheems Wang, Yixiu Mao, Vincent Tao Hu, and Xiangyang Ji. Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?, July 2025. URL http::/ http::////xxvv..g///bbb /207 03432 arXiv:2507.0466 [cs].
Noam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei, Jason D. Lee, and Sanjeev Arora. What makes a reward model a good teacher? an optimization perspective, March 2025. URL http:: 15477 arXiv:2503.15477 [cs.LG].
15477
12

---

Published as a conference paper at ICLR 2026
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience play, November 2016. URL htttps://axxxi..oo abs/ 0555525521 arXiv:1511.05955 [cs.LG].
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal pol- icy optimization algorithms, July 2017. URL ptttpa://arxxv..or//abb//////
arXiv:1707.06347 [es.LG].
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of math- ematical reasoning in open language models, February 2024. URL https: httts:://rxxy>: /wwx/vaxvvvvo.///eea/eaaa org/
abs /oss/2402.c0 03300 arXiv:2402.03300 [cs.CL].
Charlie Snell, Jachoon Lec, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters, August 2024. URL httpps://axxiv.
org/abs// 2408. 03314 arXiv:2408. T0W::240.003144 [cs.LG].
Petru Soviany. Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. Curriculum learning: A survey, January 2022. URL
10382, arXiv:2101.10382 [cs. [G).
Yifan Sun, Jingyan Shen, Yibin Wang, Tianyu Chen, Zhendong Wang, Mingyuan Zhou, and Huan Zhang. Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty- targeted Online Data Selection and Rollout Replay, June 2025. URL http:/ttt///xiii// org/
abs/ 05316 2Xiv:2506.00316 [cs].
GLM-V Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang. Yan Wang, Yean Cheng, Zehai He, Zhe Su. Zhen Yang. Ziyang Pan, Aohan Zeng, Baoxu Wang. Bin Chen, Boyan Shi, Changyu Pang, Chenhui Zhang. Da Yin, Fan Yang, Guoqing Chen, Jiazheng Xu, Jiale Zhu, Jiali Chen, Jing Chen, Jinhao Chen, Jinghao Lin, Jinjiang Wang, Junjie Chen, Leqi Lei, Letian Gong, Leyi Pan, Mingdao Liu, Mingde Xu, Mingzhi Zhang, Qinkai Zheng. Sheng Yang, Shi Zhong, Shiyu Huang, Shuyuan Zhao, Siyan Xue, Shangqin Tu, Shengbiao Meng, Tianshu Zhang, Tianwei Luo, Tianxiang Hao, Tianyu Tong, Wenkai Li, Wei Jia, Xiao Liu, Xiaohan Zhang, Xin Lyu, Xinyue Fan, Xuancheng Huang, Yanling Wang. Yadong Xue, Yanfeng Wang. Yanzi Wang, Yifan An, Yifan Du Yiming Shi, Yiheng Huang, Yilin Niu, Yuan Wang. Yuanchang Yue, Yuchen Li, Yutao Zhang, Yuting Wang, Yu Wang, Yuxuan Zhang, Zhao Xue, Zhenyu Hou, Zhengxiao Du, Zihan Wang, Peng Zhang, Debing Liu, Bin Xu, Juanzi Li, Minlie Huang, Yuxiao Dong, and Jie Tang. Glm-4.5v and glm-4. glm-4.1v-thinke. v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning, 2025a. URL http::tp/://. /tttpps//arxivvi org/ abs abtpa/as.//////// 2200/257.000 htt.b///2200000005
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuming Tang. Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang. Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao, Huabin Zheng, Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang, Jie Zhao, Jin Zhang, Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma, Shupeng Wei, Sihan Cao, Siying Huang, Tao Jiang, Weihao Gao, Weimin Xiong, Weiran He, Weixiao Huang, Wenhao Wu, Wenyang He, Xianghui Wei, Xianqing Jia, Xingzhe Wu, Xinran Xu. Xinxing Zu, Xinyu Zhou, Xuehai Pan, Y. Charles, Yang Li. Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang, Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang, Yiping Bao, Yulun Du, Yuxin Wu, Yuzhi Wang, Zaida Zhou, Zhaoji Wang, Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Ziyao Xu, and Zonghan Yang. Kimi kl.5: Scaling Reinforcement Learning with LLMs, January 2025b. URL http: aarxiv.orq/abl. 2tt..a/5.................. arXiv:2501.12599 [cs] version: 1.
L. S. Vygotsky. Mind in Society: Development of Higher Psychological Processes. Harvard Uni- versity Press, 1978. ISBN 9780674576285. URL http: www.::wwwww.............. jstor. wttp:////orgsaabbb////bb
ctv]E9v24]
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng. Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen
13

---

Published as a conference paper at ICLR 2026
Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 Rule: Eigh-Entrpp Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning, June 2025. URL /ttt:///rxiv. org/ 2506. 01939 arXiv:2506.01939 [cs].
Xin Wang, Yudong Chen, and Wenwu Zhu. A survey curriculum learning, October 2021. URL https://arxixs httpps//aaxxi.oorr//a////// wtwp:///////ooooo////oooooooooo. 13166 arXiv:2010. 13166 [cs.LG]]
Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay, November 2017. URL
01224 1rXiv:1611.01222 [cs.LG]]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URLhttpa /ttp:////axxvv.o/. wtt..o/////b///2..11999..
abs /bwp//2/......
Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senjie Jin, Rui Zheng, Wei He, Yiwen Ding, Shichun Liu, Xin Guo, Junzhe Wang, Honglin Guo, Wei Shen, Xiaoran Fan, Yuhao Zhou, Shihan Dou, Xiao Wang, Xinbo Zhang, Peng Sun, Tao Gui, Qi Zhang, and Xuanjing Huang. Training large lan- guage models for reasoning through reverse curriculum reinforcement learning, February 2024. org. abs 2402. 05808 arXiv:2402.0588 058008 [cs.Al].
URL
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan. Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source II reinforcement learning system at scale, March 2025. URL https: //axxiv.org/abs///2.....
14..11441 14476 1rXvv [cs.LG].
Yu Yue, Yufeng Yuan, Qiying Yu. Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang. Tian Fan, Zhengyin Du, Xiangpeng Wei, Xiangyu Yu, Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang, Xin Liu, Mingxuan Wang, Yonghui Wu, and Lin Yan. VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks, April 2025. URL http:/// wrxiv.org/aat abrg//bs//5tt 2504. 05118 MXXv:2504.05111 [es].
Haizhong Zheng, Yang Zhou, Brian R. Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, and Beidi Chen. Act Only When It Pays: Efficient Reinforcement Learning for LLM Rea- soning via Selective Rollouts, June 2025. URL httt::///axiii wttp//awwwi......aaaaoo/o aasgg///b/// abs/2506 02177 arXiv:2506.02117 [es].
A THE USE OF LARGE LANGUAGE MODELS
Large language models (LLMs) were only used for editing and polishing the text of this paper, in order to improve clarity and fluency of presentation. They were not used for generating ideas, conducting experiments, analyzing results, or writing technical content.
B PROOFS OF SECTION 4.2
B. PROBLEM SETUP
We first review the problem and our proposed Bayesian model. Selection the AQ tree can be casted as a hierarchical multi-armed bandit: each parent arm corresponds to an original query and its children correspond to its derived augmented queries. Each arm (i.e. query) inherently has success rate 0. Pulling an arm yields a binary reward r Bern(0), indicating whether one generated rollout from the query correct.
We propose a two-layer logit-normal Bayesian model for success rate estimation. Let success rate 0 (1) = (1 (1+0ol exp( (op(-)))-1 -1 We suppose the log-odds of each parent por follows 4/par Ni,, ?2) and the log-odds of its children i is conditionally Gaussian, s/; ppar N (W o²)
14

---

Published as a conference paper at ICLR 2026
The objective is to leverage the historical observations rewards to derive the joint posterior ditioned history, from which and then estimate their success rates sigmoid(11).
sample log-odds parameter v of both parents and children
Specifically at iteration t, let H; = (n,, s;) denote the reward history of query i, where ny is the number of the query being selected and s; the number of successes of its corresponding rollouts. Suppose the father set of nodes is F, and the children set of node j is Sy. By Bayes' rule, the joint posterior admits hierarchical chain factorization:
H( & II jes
p()))) prior of parent
x
x
x
(I)
Likelihood of parent
RES
prior of child
likelihood of child
In the following, will apply Pólya-Gamma augmentation to render the posterior sampling tractable by sample from an augmented posterior p({t), (W) which has single Gibbs sampler.
B.2 PROOFS OF PROPOSITION 4.1
Polson et al. (2013) gives the following lemma about PG distribution:
Lemma B.1. (Pólya-Gamma Gamma identity) Let p(w) denote the density of the random variable PG(b, 0), 0. Then the following integral identity holds for all a E R.
exp(1b) (1+ exp())) 2- expp(a - b/22(b) 100 exp(-wbb
/2) (2)p(w)ddw
We first derive the conditional posterior p(v H4, parr of a child i, given its parent upor and history H, = (n, st).
p(sbs H4, Upar))
xpp s/s (ppr)) pp( Hi) (/)))
p(s/sl/ppr)) p(88, nelvbe)
exp(vx)
=p(1hslupar)
(1 exp(e))))m
fo exp(1ap,
wi²? )prolwy mn O)du;
=p(ssslppar) n
where x = 2 and wi ~ PG(n,, 0). The identity follows directly from Lemma B.1 Now that have the following identities:
wiyh )pro(us|ne,00 0)
p(yhi, wil Hi, s/prr) x pl (/1///// par))
exp(1<//
exp(
2iv?
PPG (wi/114 0)
p(wekhh He paar)
proluilm;, (4)
000
exp(
|PpC(wi 0)dw;
The second identity follows from the definition of Pólya-Gamma probability density. The full con- ditional posterior of th then follows:
ppar)//pr //pp/ HH)) oc p(tv/ (ppr)) exp(xabs wath? p(u/ Hy, upar, wi) = p(abe ws Hi, ppar
2
Given that the prior u lrpr also a Gaussian:
N (1 0²), 's full conditional posterior p(ub Hi Upar Wi) is
the Hi, par Wi (m, V), where V --2 wi
V 2 + x)
And the proof of Proposition 4.1 ends.
15

---

Published as a conference paper at ICLR 2026
B.3 PROOFS OF PROPOSITION 4.2
Similarly, we can also derive parents' conditional posterions p(Vparl Mpar, faha fa ( by introducing PG variables:
p(v H par Epar) (xp(()par)) p(spar Wpar)
II AE
plub (par)
pl (((relbpprr) 2parpar Apar ( exp(Kipar Vpar
Wpar 2/2
p(Pppr))
II Sarr
par
ppr (upar Mpar Odddr
2
Ryar where Kipar = Spar 2 war ~ (nprr 0). The identity also follows from Lemma
B.1
Based
on this, easily derive the following conditional posteriors:
p(vra Wpar) exp t/er Wper 2 par
p(upprr par, Baar)
p( ((ppar))
II
2
KESpar
PPG (wpar (ppar:0)
wpar Vpar
plwpprr Wpr, Haar,Huucc Hpor) exp(
JPPG (Wpar Mpar.
Prra(wpue Mparr Wpar)
So exp(
Hpur V PPC (Wpar (par.
1/22
KrSpar ((pprrlwprrtte par, (wh ( (aar) x ((iipa))
II
exp V
W par par
2
Given that prior Upar N(,, 72), valuppar also Gaussian:
N (1hpar )²), Upar's 's full conditional posterior is
Upar laipar, H Hpars
{1x Spar N(m,
where +utpar Spar 1- m1 V (T-2 240 + +0-2 2x U/h + ). And the proof Propo-
sition 4.2 lnds.
C EXPERIMENTAL SETTINGS
The main experiments are conduct on PPO and GRPO with implementation in ver At each train- ing iteration, generate 8 rollouts per query with temperature set to 1.0. We use batch size of 512 and a mini-batch size of 64, yielding gradient updates per PPO epoch. The maximum response length is set to 6144 tokens. We adopt the clip-higher strategy (Chigh 0.28) and the overlong reward shaping (Leache = 1024) introduced by Yu et al. (202)), but do not apply addi- tional KL regularization length-adaptive GAE proposed in VAPO (Yue et al. (025). We also conduct value pretraining for twenty iterations following VAPO. For dynamic sampling baseline, we reuse the implementation code in verl. For prioritized sampling, we tracks the number of rollouts being generated for each query and the number of success times within these rollouts to calculate the pass rates. Similar to our proposed PROS, we also adopt a exponential decay of X 0.9. For experience replay, we only reuse the rollouts from the last iteration avoid large gap. specific, we use x rollouts from the replay buffer, while the rest of (512- 64) x 8 rollouts are generated by the current policy. The truncation threshold for truncated importance sampling is set to 10 following Wang et al. (2017). For PROS, adopt μ 0, 1.5 to ensure discounting factor À = 0.99 and the variance of children prior o = 0.3 by default. We also ap- ply diversity regularization in the selection stage that each query appears at most any consequential K training iterations. To ensure a fair comparison with respect to training effi- ciency and GPU wall-clock, apply identical engineering hyperparameters to all methods, such as appu_mmorr..uill. www.yutiiiitiiiin. for inference engine, max.token. wereeeeeeeenee www.pee..ppp....e for dynamic batching, etc.
entropy loss. Specifically for PPO, we adopt the decoupled-GAB and
uniform prior of pass rate 8. The exponential
time within
httts://gi coml..com/vee wwm/vol cwt.:///oo/oooolginoeeooooe
16

---

Published as a conference paper at ICLR 2026
Table 4: Comparison of @2 between PROS and other baselines of Qwen3-8B trained on DAPO-Train.
AIME24
AMC23
#Tokens
#Time
PPO
57.67
90.03
10.24
9.17
w/ Dynamic Sampling
63.89
90.69
11.07
17.62
w/ Experience Replay
63.15
90.05
8.93
8.98
w/ Priority Sampling
64.55
91.41
11.35
9.76
PROS aros-ablation
62.64
92.13
7.92
9.29
w/ PROS
66.05
90.87
8.49
9.77
GRPO
59.14
90.62
9.27
6,58
w/ Dynamic Sampling
63.82
91.09
11.16
19.45
w/ Experience Replay
63.48
91.05
7.60
6,88
w/ Priority Sampling
64.77
91.02
9.55
6.87
w/ PROS-ablation
65.44
91.24
7.87
7.00
PROS
65.01
90.54
8.46
7.57
Table 5: More ablations of PROS with Qwen3-4B trained on AIME-Old.
AIME24
AMC23
#Time
21.25
$2.59
5.83
w/ dynamic
25.73
61.19
10.83
w/ replay
23.75
53.73
5.81
w/ prior
24.90
60.44
5.86
w/ PROS (value-hased prefix identification)
24.27
65.32
5.97
w/ PROS (entropy-based prefix identification)
24.68
63.79
6.33
PROS (reward variance selection)
25.72
61.81
6.07
w/ PROS
27.40
62.12
6,21
D ADDITIONAL EXPERIMENTS
Here we present the additional experiments required by reviewers.
Table presents the Pass @ 32 comparison between PROS and other baselines.
additional ablation studies on PROS. The entropy- based prefix identification variant selects the prefix with the highest token-level entropy (which is the same selection heuristic used in GRPO w/ PROS) for future reuse. The value-based prefix identification variant selects the prefix with the highest value according to the critic. Both of these variants outperform the baseline by a large margin, demonstrating the effectiveness and generalization of our proposed prefix reuse mechanism. The reward variance selection variant replaces the proposed Bayesian-based selection mechanism with an e-greedy method, which greedily selects augmented queries with the highest reward variance from history, while still maintaining an mechanism ensure exploration.
17


Reviews:
[{"review_id": "v4pRC24Pmp", "reviewer": "Reviewer_cmZx", "summary": "The paper targets the **on-policy rollout bottleneck** in RL with verifiable rewards (RLVR) for reasoning: many on-policy rollouts for the same query share long identical prefixes, so generation time is wasted regenerating early steps. The proposed method, **PROS**, (1) identifies “promising” prefixes in past *correct* rollouts using readily available signals (token-level entropy and critic value), appends them to the original query to create **Augmented Queries (AQs)**, and then trains *only on the newly generated continuation*—preserving on-policy updates while saving generation compute; and (2) runs a **hierarchical Bayesian selector** over the growing parent→children (query→AQ) tree to prefer items with uncertain pass rates (≈0.5), sharing strength between a parent query and its AQs.   \nPROS integrates as a plugin to PPO/GRPO and is evaluated primarily on math benchmarks (AIME24, AMC23), reporting Pass@1 and compute trade-offs; Dynamic Sampling, Prioritized Sampling, and Experience Replay serve as baselines.\n\nLLM usage disclosure: I used a large language model to reorganize wording and improve clarity in this review. The judgments expressed are my own.", "questions": "1. **Pass@k and budget matching.** Please report **Pass@k** (e.g., k=8/32/128) in addition to Pass@1, with matched budgets, to separate exploration breadth from sampling reweighting gains. (Current results focus on Pass@1. )\n\n2. **Convergence & final comparison.** Provide **longer-horizon** training curves and **final** metrics comparing PROS to **Dynamic Sampling** and other baselines under **identical compute budgets**. Check if methods like Dynamic Sampling eventually catches up or surpasses PROS (as hinted on some setups) and discuss whether PROS’s early gains diminish over time. \n\n3. **Hard-subset analysis.** Report performance on the **hardest part** of items and discuss whether mid-difficulty targeting (pass rate ≈0.5) reduces exposure to very hard queries (pass rate ≈0). (Selector targets ≈0.5 region. )\n\n4. **Prefix heuristics ablation.** Ablate the **prefix-picking** rule: (a) value-only, (b) entropy-only, (c) random correct-prefix; include efficiency and accuracy. (Heuristic defined in Sec. 3.2. )\n\n5.  **Generality to Other Domains:** Do you have any results applying PROS to a non-mathematical RLVR task, such as code generation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* **Originality/Significance:** The paper addresses a critical RLVR bottleneck. Its novelty lies in integrating prefix reuse (which *creates* a structured dataset) with a Bayesian selector *tailored* to that hierarchical structure.\n* **Methodology:** The method is sound. Applying gradients only to the new continuation preserves the on-policy nature of RLVR. The hierarchical Bayesian selector with exponential forgetting is a principled way to share statistics and adapt to the non-stationary policy.\n* **Empirical Quality:** Experiments use strong, relevant baselines like Dynamic Sampling. Figure 5 shows PROS achieves a better performance-cost trade-off, outperforming Dynamic Sampling while being 2-4x faster in wall-clock time. An ablation confirms the selector's value over random sampling.\n* **Clarity:** The paper is well-written, using Figures 1 and 3 to effectively illustrate the core concepts of prefix redundancy and the PROS pipeline.", "weaknesses": "* **Limited Evaluation Metric (Pass@1):** The evaluation relies exclusively on Pass@1. This metric fails to show if the model is learning *diverse* solutions. Pass@K (K=32/128, etc.) is a feasible and important metric that is missing.* **Unclear Final Capability vs. Efficiency:** The paper's central claim is a better performance-cost trade-off, and it excels at showing faster initial learning (Figure 5). However, the core methodological trade-off against baselines like Dynamic Sampling (DS) is not fully resolved.\n* **Risk of \"Hard-Tail\" Starvation:** PROS explicitly targets queries with high reward uncertainty (pass rate $\\approx 0.5$). This is a known efficiency strategy, but it risks starving the model of exposure to the hardest problems (pass rate $\\approx 0$). In contrast, DS is *specifically designed* to focus on this hard-tail. The experiments, which are limited in training time, do not establish whether PROS's efficiency-first approach can match the final performance ceiling of a capability-first method like DS, given a matched (and sufficient) compute budget. The paper is missing a clear convergence-point comparison.\n* **Un-ablated Prefix Heuristics:** The prefix identification rule (entropy + value) is purely heuristic . The paper provides **no ablation** to justify this choice over simpler ones (e.g., value-only). The main \"PROS-ablation\" only tests the *selector* (Component 2), not the *prefix identification* (Component 1).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g1c29vabWc", "reviewer": "Reviewer_yLPZ", "summary": "The paper tackles a bottleneck in RLVR: the high cost of on-policy rollouts. The authors observe that many rollouts for the same prompt share identical or near-identical prefixes. PROs leverages this redundancy by reusing verified prefixes from previous rollouts constructing augmented queries that begin from these prefixes, and then sampling only the continuation (suffix). This idea is known before. See below. \nThe authors talk about a hierarchical Bayesian selection scheme to identify which prefixes are most promising to reuse, prioritizing uncertain or under-explored augmented queries. That was interesting and to my knowledge novel. \n\n Comparison: \n\nSPEC-RL (Liu et al., 2025) also reuses verified rollouts but does so at a coarser granularity, caching *entire* rollouts or long segments. PROS’s is finer-grained.  \n AR3PO (Zhang et al., 2025) proposes *response reuse* (retraining on previously verified answers) but does not address redundancy within rollouts themselves. PROS directly targets intra-trajectory redundancy.\nTreePO as the name suggests (MAP/ByteDance, 2025) exploits shared prefixes structurally through tree-based rollout planning.  \n\nThere are earlier works going to Kearns et al I believe. But overall,  PROS represents a clear step beyond existing efficiency methods by focusing on *within-trajectory reuse*, introducing a principled selection mechanism, and maintaining on-policy validity.\n\nDespite minor concerns about exploration and theoretical framing, the contribution is substantial, the experiments convincing, and the writing clear. PROS is likely to influence future work on efficient LLM-based reinforcement learning.", "questions": "none", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Clear motivation and empirical grounding: The observation that many rollouts share prefixes is both intuitive and empirically demonstrated.\nGenearlity: The prefix-reuse mechanism is orthogonal to the underlying policy architecture or reward type.\nSoundness:  The Bayesian selection framework is well justified and avoids trivial over-reuse of easy prefixes.\nImpact: Implementation is simple and yields immediate compute savings in RLVR systems—highly relevant for scaling large-model training.\n\n-", "weaknesses": "Exploration bias By reusing existing prefixes, PROS may over-concentrate training around known reasoning paths, potentially limiting discovery of novel strategies. While the Bayesian selector mitigates this, empirical diversity metrics would strengthen the case.\n \nMinimal theoretical framing: The paper remains largely empirical; a more formal analysis of sample complexity improvement or convergence would enhance rigor.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xSHO1r63uu", "reviewer": "Reviewer_NRn5", "summary": "The manuscript proposes to improve the generation phase of Reinforcement Learning by not generating each complete trajectory separately but instead identifies promising prefixes for a given problem and then only generates additional trajectories based on those prefixes. The author call their approach PROS (Prefix Reuse for On-policy Sampling) and the combination of problem and trajectory prefix Augmented Queries. These new queries can then be added to the training dataset in subsequent training steps and can be treated like regular problems, however with less redundant computation necessary.\n\nPrefixes are chosen based on uncertainty, for which token-level entropy is used, as well as value function offered by the critic model, if available in the given setup. To select Augmented Queries, PROS uses a hierarchical Bayesian model to estimate their pass rates and focuses on queries, where reward uncertainty is the highest to improve exploration. Additionally optimization techniques such as exponential forgetting of rewards are employed, so that more recent Augmented Queries are selected, giving more weight to the improved policy.", "questions": "* Figure 2a: I did not understand what fraction is shown in that subfigure. Of the whole policy optimization algorithm?\n* What is the computational overhead for the selection and reward estimation?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* address a relevant topic, i.e. how to improve the efficiency of the generation phase of the RL pipeline, and is also timely\n* manuscript is mostly well written with the idea being clearly presented and the theoretical underpinnings derived well\n* reasonable evaluation, which includes a similar reasonable ablation study", "weaknesses": "The evaluation is reasonable (two datasets, three baselines and a little bit of ablation, which is also reasonable), however it is also somewhat limited, since only a single, small, model is evaluated. Some additional ablation study on an ever smaller model is provided. Additionally the computational environment is not discussed even though metrics such as wall clock time are reported. \nThe analysis is at times spotty, i.e. not comprehensive enough, for example one of the motivations was to address redundancy in the generation (at least the last part of Section 2 as well as the first part of the related work section give that impression), but the token cost increase (at least for AIME-Old).\nSome notation issues, such as using certain symbols twice or inconsistent parameter order, muddy the otherwise clear writing.\n\nminor issues:\n* introduction, page 1:\n  * first paragraph: reference for CoT is missing\n  * second paragraph: \"PPO(Schulman\" and \"GRPO(Shao\" - missing whitespace\n* chapter 2:\n  * page 3, first paragraph: \"And the objective\" - language, a sentence should not start with \"And\"\n  * page 3, Redundancy in Early Reasoning Steps: \"on DAPO-Train dataset\" - language: add a \"the\" before DAPO-Train?\n  * Figure 2b/c: It might be worth discussing the two metrics a little bit, since they seem to go into different directions - normalized edit distance: smaller values indicate higher similarities, where it is the opposite for ROUGE-L?\n    * also ROUGE-L (text) is written inconsistently (vs. Rouge-L in Figure 2c)\n* 3.1, last paragraph: \"remains the on-policy nature of RLVR algorithm\" - language: sounds strange to me, maybe \"retains the on-policy nature of the RLVR algorithm\"\n* Figure 3:\n  * You might want to careful with the notation. In Section 2 $y_i$ is an individual thought whereas here it represents a whole trajectory/rollout.\n  * \"Promising prefix identification\" - the other titles of the rounded rectangles are capitalized\n* 3.2, page 5, first paragraph: \"leverage the readily available value function learned by critic model for those actor-critic methods\"\n  * you claim that PROS can be easily be plugged into GRPO, but GRPO does not have a critic models, but derives its value function differently: in the subsequent paragraph this is presented more clearly\n* 3.2, last paragraph:\n  * \"with highest token-level\" - language: add \"the\" before \"highest\"?\n  * maybe explain what $H$ is; here it is probably the log-likelihood, but later it is used for the history (in 4.2)\n* Figure 4: What is $\\mathbb{S}_j$? Probably the children of parent j. (After reading further it is discussed in 4.2, so after the figure is displayed.\n* 4.2:\n  * Proposition 4.1: \"V)where\" in equation, at least a whitespace is missing, maybe even an additional comma\n  * sometimes the History $H$ is $(n, s)$ (at the beginning, Algorithm 1) and sometimes the other way around (for the propositions); similar issues in Appendix B.2\n* 5.1: title is probably not necessary, if Section 5 has not other subsections\n* references:\n  * cited differently than other arXiv references (can only be surmised from the URL); additionally consider capitalizing the titles to be consistent, especially abbreviations and proper names:\n    * [Dumitrascu et al. 2018]\n    * [Fedus et al. 2020]\n    * [He at al. 2025]\n    * [Hong et al. 2022a/b]\n    * [Lambert et al. 2025]\n    * [Liang et al. 2021]\n    * [Liu et al. 2025a/b]\n    * [Narvekar et al. 2020]\n    * [OpenAI et al. 2024]\n    * [Polson et al. 2013]\n    * [Razin et al. 2025]\n    * [Schaul et al. 2016]\n    * [Schulman et al. 2017]\n    * [Shao et al. 2024]\n    * [Snell et al. 2024]\n    * [Soviany et al. 2022]\n    * [Wang et al. 2021]\n    * [Wang et al. 2017]\n    * [Xi et al. 2024]\n    * [Yu et al. 2025]\n  * consider capitalizing the titles to be consistent, especially abbreviations and proper names:\n    * [V Team et al. 2025b]\n* B.3:\n  * title: It is probably Proposition 4.2 instead of 4.1\n  * page 16:\n    * line 810: unnecessary whitespace at the beginning of the line\n    * equation cuts into the margin\n    * line 826: comma should probably be at the end of the previous line, after the equation\n* C:\n  * uses PROST instead PROS\n  * line 843: \"from replay buffer\" - language: add a \"the\" before \"replay\"; same line: \"by current policy\" - add a \"the\" before \"current\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rZKBjI1mxg", "reviewer": "Reviewer_NFuG", "summary": "This paper introduces PROS, an RLHF training paradigm that reuses correct-prone trajectories to augment raw prompts, thereby improving exploration toward correct responses. PROS employs a Bayesian model to prioritize training on augmented prompts with high uncertainty, via token-level or progress-based values. Experimental results show that PROS achieves faster convergence and higher accuracy compared to baseline methods.", "questions": "Thank you for submitting your work. The paper is clearly presented and provides a reasonable solution to improve RLHF efficiency. However, the contribution would be stronger with more comprehensive experiments and deeper analysis. Below are specific comments and questions:\n\n- Q1: How does PROS perform on other datasets, such as the code generation task or MATH500? Using only AIME24 and AMC23 limits the generality of the results. \n\n- Q2: Reusing correct-prone trajectories may improve sample efficiency but risks reducing response diversity. Can the authors discuss potential strategies to mitigate this effect?\n\n- Q3: The ablation study on Bayesian Inference (BI) for prompt selection is appreciated. Could the paper also compare simpler alternatives (e.g., variance-based scoring) to justify whether BI offers sufficient benefit to warrant its added complexity?\n\n- Q4: In Figure 5(a), some baselines (e.g., Vanilla PPO) appear to have been early-stopped before convergence. Could the paper clarify the stopping criteria?\n\n- Q5: Tables 2 and 3 include useful hyperparameter studies. It would be better also to show how the baseline methods perform under the same hyperparameter variations for a fairer comparison.\n\nReference\n\n[1] TreeRPO: Tree Relative Policy Optimization, arXiv:2506.05183.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ It tackles an important problem in RLHF.\n+ The design is conceptually clear and practical to implement.\n+ The paper is overall well-written.\n+ Evaluations demonstrate good speedups and accuracy gains.", "weaknesses": "- Evaluations are limited to only two datasets and miss some ablation studies. \n- The methodological novelty is moderate; the core ideas resemble existing works such as TreeRPO[1].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)