Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models
Authors: Yixiu Mao
Yun Qu
Cheems Wang
Heming Zou
Xiangyang Ji
Venue: ICLR 2026 Poster (2026)
Keywords: large reasoning model
reinforcement learning finetuning
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose Dynamics-Predictive Sampling (DPS) for RLVR, which infers prompt learning dynamics online to select informative prompts before rollout, accelerating RL finetuning of large reasoning models without the need for rollout-intensive filtering.

Abstract: Reinforcement learning (RL) finetuning has become a key technique for enhancing the reasoning abilities of large language models (LLMs). However, its effectiveness critically depends on the selection of training data. Recent advances underscore the importance of online prompt selection methods, which typically concentrate training on partially solved or moderately challenging examples under the current policy, thereby yielding more effective model updates. While significantly accelerating RL finetuning in terms of training steps, they also incur substantial computational overhead by requiring extensive LLM rollouts over large candidate batches to identify informative samples, an expense that can outweigh the finetuning process itself. To address this challenge, this work proposes Dynamics-Predictive Sampling (DPS), which online predicts and selects informative prompts by inferring their learning dynamics prior to costly rollouts. Specifically, we introduce a new perspective by modeling each prompt's solving progress during RL finetuning as a dynamical system, where the extent of solving is represented as the state and the transition is characterized by a hidden Markov model. Using historical rollout reward signals, we perform online Bayesian inference to estimate evolving state distributions, and the inference outcome provides a predictive prior for efficient prompt selection without rollout-intensive filtering. Empirical results across diverse reasoning tasks, including mathematics, planning, and visual geometry, demonstrate that DPS substantially reduces redundant rollouts, accelerates the training process, and achieves superior reasoning performance.

Published as a conference paper at ICLR 2026
SAMPLING FOR ACTIVE RL FINETUNING OF LARGE REASONING MODELS
Yixiu Mao, Yun Qu, Wang Heming Zou, Xiangyang Ji"
Department of Automation, Tsinghua University
(myx21, qy22, zouhm24 4) @ma i 1s t.s.ssinngg tasinghuaa...... edu.cn
cheemswang@mai 1. ts tsinghua wwwiinguaaed..c.. edu.cn, xyji@tsinghua.
ABSTRACT
Reinforcement learning (RL) finetuning has become key technique for enhane- ing the reasoning abilities of large language models (LLMs). However, its effec- tiveness critically depends on the selection of training data. Recent advances un- derscore the importance of online prompt selection methods, which typically con- centrate training on partially solved or moderately challenging examples under the current policy, thereby yielding more effective model updates. While significantly accelerating RL finetuning in terms of training steps, they also incur substantial computational overhead by requiring extensive L.M rollouts over large candi- date batches to identify informative samples, an expense that can outweigh the finetuning process itself. To address this challenge, this work proposes Dynamies- Predictive Sampling (DPS), which online predicts and selects informative prompts by inferring their learning dynamics prior costly rollouts. Specifically, in- troduce a new perspective by modeling each prompt's solving progress during RL finetuning as dynamical system, where the extent of solving is represented as the state and the transition is characterized by a hidden Markov model. Using historical rollout reward signals, we perform online Bayesian inference to esti- mate evolving state distributions, and the inference outcome provides predictive prior for efficient prompt selection without rollout-intensive filtering. Empirical results across diverse reasoning tasks, including mathematics, planning, and visual geometry, demonstrate that DPS substantially reduces redundant rollouts, acceler- ates the training process, and achieves superior reasoning performance. Our code is available at
INTRODUCTION
Reinforcement learning (RL) finetuming has emerged crucial technique enhance the reason- ing capabilities of large language models (LLMs) (Lightman et al., 2023; Jacch et al., 2024; Guo et al., 2025; Team et al., 2025). These finetuned models, often referred to large reasoning mod- els (LRMss), generate chain-of-thoughts (CoT's) to perform multi-step structured inference and have achieved remarkable progress across wide range of Enowledge-intensivy applications, including scientific question answering (He et al., 2024), symbolic mathematics (Luo et al., 2025b), logical deduction (Xie al., 2025), and program synthesis (Luo al., 2025a).
While RL finetuning has demonstrated substantial progress, its effectiveness depends heavily on the quality of training data (Guo et al., 2025; Yang et al., 2024b), prompting increasing attention to data curation (Wen et al., 2025; Hu et al., 2025). A common practice is to perform offline data filtering, in which prompts are ranked or selected prior to training using static heuristics such as estimated difficulty. domain balance, or diversity (Ye et al., 2025; Li et al., 2025; Wang et al., 2025b). Al- though beneficial, this approach fails to adapt to the model's evolving competence during training. To improve adaptivity, recent work has explored online prompt selection strategies that dynamically adjust to the model's evolving behavior. These methods typically operate on per-step or per-epoch basis, selecting informative prompts that provide stronger training signals (Yu et al., 2025; Zhang
"Corresponding authors.

---

Published as a conference paper at ICLR 2026
Fully Unaolived
Partially Solved =
Fully Solved
Curpent-step Prediction
Tup-B Selection
Observation
Transition Update
[Next-step Prediction
LLLM Rollout
+pp
Prompt
Taprice
ft Tupoo
Tuptrer Peol
L.LM Roliout
988
$ P1
raguet
Prompt
Tidd
Pr+1
Not Evalusted
#2 14
Prompt m3
Prior as Posterior
P
PaPa
Tatelee
Step r
Step Posterior
Step
Figure 1: Penamics-Predictiin Sampling (DPS) framework. DPS models each prompt's solving progress in RL finetuning dynamical system, treating solving extent as the state with transitions characterized by hidden Markov model. By employing lightweight inference, it predicts and se- lects informative (partially solved) prompts online, without requiring rollout-intensive filtering.
et al., 2025; Cui et al., 2025). representative state-of-the-art (SoTA) approach is Dynamic Sam- pling (DS) (Yu al., 2025), which expands candidate prompt batches, generates multiple responses per prompt, discards uninformative prompts with consistent rewards, and uses the retained subset for finetuning. This strategy improves training sample quality and significantly accelerates RL fine- tuning in terms of training steps. However, for reasoning- iintinn-inttnsvvee tasks, generating responses with long CoTs is computationally expensive. As result, DS incurs substantial overhead from extensive LLM generation on enlarged batches, which in practice often outweighs the cost of finetuning itself.
This work aims preserve the adaptivity of online prompt selection while avoiding redundant roll- outs. To this end, we propose Synamics-Predicttoo Sampling (DPS), which online predicts informa- tive prompts by inferring their learning dynamics. Specifically, we introduce a new perspective by formalizing each prompt's solving progress during RL finetuning dynamical system. The solv- ing extent each prompt is treated as the state of the system, while the distribution these states evolves as LRM updates. Technically, this process is instantiated which serves as a tractable tool for tracking the prompt-solving dynamics. Given the constructed dynamical system, we perform online Bayesian inference to estimate the evolving state distributions from historical rollout reward signals. The inference outcome offers predictive prior for adaptive prompt selection, thereby improving sample efficiency without rollout-intensive filtering.
hidden Markov model (HMM),
Empirically, we evaluate the proposed DPS across diverse reasoning downstream tasks, including competition-level mathematics, numerical planning, and visual geometry. The results demonstrate that DPS can accurately predict prompts' evolving solving states and consistently select higher proportion of informative samples compared baseline methods. Leveraging this capability, DPS substantially accelerates RL finetuning, achieving performance comparable or even superior to the oracle rollout-intensive strategy DS with significantly fewer rollouts.
PRELIMINARY
RL Finetuning for LRMs. Given a prompt r sampled from a dataset and response y generated from the model's policy no (ylT). the objective of RL. finetuning is to maximize the expected return:
max O
EYWD, y--m((-Ir) [F(T, v)],
(1)
where the reward function r(r, 8) typically verifies the correctness of responses, with binary signals commonly used in domains such as mathematics (i.e., for correct and () for incorrect).
Group Relative Policy Optimization (GRPO). To solve the above optimization problem, a num- ber of policy gradient methods have been proposed. GRPO (Shao et al., 2024) is recent and widely adopted variant that eliminates the need for explicit value function estimation, making it particularly suitable for finetuning L.LMs. Formally, for arbitrary prompt and its corresponding k sampled
2

---

Published as a conference paper at ICLR 2026
responses {ur 1 GRPO maximizes the following objective:
(RPOO(0) = B, M/ (
(o [T Tew (3/ T)
A7. clip (v] (T) Tew (3[TT) e, 1 1 1i) -B (Tol(att))))
where the clipped policy ratio prevents from deviating excessively from the previous policy Te..> while the regularization coefficient / penalizes divergence from fixed reference model Tret- GRPO employs a group-based normalization scheme estimate the advantages A7:
r(T, v!) (een(((((r,)])) A
(2)
std((r(T, 05) 1/)))=))))
This strategy significantly reduces training complexity and has demonstrated strong empirical per- formance across diverse LLM reasoning tasks (Shao et al., 2024; Guo et al., 2025).
Dynamic Sampling for Online Prompt Selection. In RL finetuning of LLMs, training examples contribute unequally to policy improvement. When the model consistently answers a problem either correctly or incorrectly, a phenomenon frequently observed during training (Zhang et al., 2025), the reward provides limited optimization signals (Chen al., 2025; Yu al., 2025). For algorithms such as GRPO, this situation causes the normalized advantages to vanish, effectively halting optimization.
To mitigate this issue, online prompt selection strategies are proposed to dynamically curate prompts under specific rules (Zhang et al., 2025; Yu et al., 2025). A representative SoTA method is Dynamic Sampling (DS) (Yu et al., 2025). At each training step f, DS rolls out with larger, randomly sampled candidate prompt batch B,, and discards uninformative prompts with identical rewards across the responses, forming the final training batch By:
B: = {T E B, std ((r(T,B7)) ()))) > >0}-
(3)
Despite its effectiveness, DS introduces significant computational overhead due to repeated LLM rollouts and evaluations over the enlarged candidate batch. In many cases, the candidate batch is several times larger than the final batch, resulting in proportional increase in LLM generation costs. This burden is particularly pronounced in reasoning tasks requiring long CoT generation.
For extended discussions related work, refer the reader Appendix A.
DYNAMICS-PREDICTIO SAMPLING FOR ACTIVE RL FINETUNING
This section formalizes the prompt-solving progress dynamical system, develops an inference strategy for solving extent prediction, and proposes an efficient pipeline for online prompt selection.
3.1 GENERATIVE MODELING OF PROMPT-SOLVINE DYNAMICS
Problem Formulation. Prior research has revealed the existence of prompt-solving states for ef- ficient policy optimization. Specifically, History Resampling (HR) (Zhang et al., 2025) categorizes prompts into fully solved ones and others, whereas DS (Yu al., 2025) distinguishes partially solved prompts from the rest. Both theoretical analyses and empirical findings (Bae et al., 2025; Chen et al., 2025) suggest that prompts yielding both successful and failed responses they provide stronger gradient signals for updates. In light of this, this work defines an implicit state 2. {1, 3) for each prompt D. indicating its rollout outcome training step
more informative, as
State (fully unsolved): All responses are incorrect, Σx= r(T, v:) = 0:
State (partially solved): Some responses are correct and some incorrect, Σt_ r(T,
State 3 (fully solved): All responses correct, Σk_ ((T.) k.
According to prior work (Bae et al., 2025; Chen et al., 2025), State 2 prompts are the most informa- tive and therefore should be prioritized during training. However, each training step, the solving
3

---

Published as a conference paper at ICLR 2026
state of any given prompt is unknown prior to rollout and evaluation. In the batch training setting, solving states are only observed intermittently, when certain prompts are selected for rollout. Con- sequently, each prompt yields an intermittent observation sequence, with the observation of prompt r at step t. denoted as V (where yi" = Ø if no observation made). Our objective is to estimate the filtered prior belief of the solving state at step before observation, denoted by n4 n.
14g
p.prior
= (
via-1), Vi E (1,2,3)
Prompt Solving as Dynamical Systems. We formalize the evolution of each prompt's solving state using Hidden Markov Model (HMM), which captures how the LLM's ability solve given prompt evolves during training. For clarity, we omit the superscript in this section and Section 3.2, describing the generative and inference process for single prompt, which applies all others.
Formally, the initial solving state 21 is drawn from a categorical prior potior A In the absence of prior knowledge, we adopt uniform distribution:
1 (ategorical (x?"", 11
prior
(5)
3+
Subsequent states evolve according to a Markov process with a 4 R3x3 where entry ( j) represents the probability of transitioning from state state i:
transition matrix
2x -t-1 Categorical ( zt-1)): 1)),
((i,j)
P(=+ ž = =),, E o(i..) 1. (6)
At each timestep, if the prompt is selected for training, the observation Ye reveals the current state exactly; otherwise, the state remains unobserved. This yields degenerate emission model:
plya 21) = [5(y), 21).
{1.
if ye E (1,2.3) if yt Ø0
where -(:-)) denotes the Kronecker delta function. Assigning emission probability to missing observations preserves marginal consistency while imposing constraint 24 Putting these com- ponents together, the solving progress for each prompt can be represented as dynamical system. Specifically, the joint distribution over states 21:7 and observations /1.r factorizes
p(ZE., (1:)) = p(z1) I( 2,, D) II p(ue 2 t=1
where the transition matrix & is treated as a random variable, This formulation specifies the under-
lying generative process, thereby enabling subsequent Bayesian inference over the solving states.
3.2 ONLINE INFERENCE AND TRANSITION LEARNING
We perform online Bayesian inference to track the solving states for a given prompt during training. The procedure follows three-stage pipeline at each training step t: (i) update the prior prior to posterior 44 using the observation y: if available, otherwise setting the posterior to the prior; (ii) y is observed, refine the transition model; and (iii) propagate the posterior forward through the transition model to generate the next-step prior p
Observation Update.
ye is observed, Bayes' rule updates the prior jl prior the posterior p1
p(Ue 21 = i) 11. prior
8(y1. i) 11 prior
14
post
if E (1,2,3}.
(9)
Σn P(ve 21 k)
prior 14 (k)
Σe (Be, k) p prior (k)
If is unobserved, the Bayesian update defaults to y. pu without new evidence.
Transition Update. We place independent Dirichlet priors on the columns of transition matrix:
#e(:J) Dirichlet(a (1.,), o,, (2. (3.j)), Vj (1,2,3).
(10)
where ar (1. specify the distribution over the transition probabilities. We initialize the transition matrix with uninformative prior by setting (i j) As observations arrive sequentially, the
4

---

Published as a conference paper at ICLR 2026
parameters an (i. are updated online. Specifically, when yt is observed at step Bayesian update applied to j) using the soft transition statistics:
i j) = a (i, + E(i,)),
(11)
where je denotes the posterior transition pseudo-count:
5( P(z-11 J, = i y1a), if E (1,
(12)
This update rule follows from the conjugacy between the Dirichlet and Categorical distributions. Observing transition from state j to i adds one pseudo-count to the corresponding parameters of the Dirichlet prior. As the transition is uncertain, the expected contribution given by &(i,,) By the Markov property and the conditional independence of observations given states, we obtain:
p/ (i) & (i. p(u: 2t =)
=(=-1=ja = = il v1.) = Ey ( (i) (2', ply: 21 =1=²²
(13)
with derivations deferred to Appendix C. Using the deterministic emission model in Eq. (7), and setting & when ys unobserved (so the Bayesian update defaults to the prior), & simplifies to:
p (j) 4 (2.
if ye+
(l =
Eg y 1 (i') - +1-1(1,5"" 0,
(14)
otherwise.
Non-stationary Extension. The standard Bayesian HMM assumes stationary transition dynamics. However, prompt-solving states in LRMs may evolve non-stationarily due to the complex learning process. To accommodate changing transition dynamics, we propose a lightweight extension that applies an exponentially decayed Dirichlet posterior update to the transition model:
(i(i.)) = À (t.)) + (1 - A) ao(i.)) Ee(i.)) À (0, 1).
(15)
This mechanism introduces forgetting by emphasizing recent transition statistics while gradually discounting outdated patterns. Smaller values of À yield faster adaptation to evolving dynamics. The prior op serves as a regularizer: it prevents collapse when recent evidence is sparse and also enables the encoding of domain knowledge about plausible transition structures.
Next-state Prediction. After the observation and transition updates at step I, we use the posterior belief 14 and the inferred transition matrix o to form the predictive prior for the next step:
M M++1
post D. f1
i.e., M ンラ 4 (i.j) 14 (j).
(16)
This prior 1441 represents our forecast of the prompt-solving state at training step + 1 before its observation, and serves as the initial belief for the subsequent inference iteration. Unlike classical (Baum et al., 1972)), which require access to
prior
HMM smoothing methods (e.g., full trajectories, our approach updates both the state belief and transition posterior in an online man- ner. Moreover, the computational cost of this inference framework is typically negligible compared to response rollout or model finetuning, as involves only very low-dimensional matrix operations.
3.3 PROMPT SAMPLING WITH PREDICTED DYNAMICS
The central goal of modeling prompt-solving dynamics is to online predict which prompts should be prioritized for training at each step, before conducting costly rollouts. Given the predictive solving- state belief μt r-prior = P2 31:1-1) ) for each prompt т. we prioritize prompts according to their predicted probability of being partially solved (State 2), denoted μu, prii (2). Crucially, rely on the prior belief μ47 p.prior rather than the posterior μ.ū T.ppost since selection must occur before outcomes at step / are observed via rollouts. Formally, the B prompts with the highest State 2 probabilities are selected to constitute the training batch at step t:
B = Top ((TEDIA) p.prior (o (2)))
(17)
We note that while the Top-B selection strategy is purely exploitative, it exploits objective (i.e., 14, p.prior) that already incorporates degree of exploration via the non-stationary decay mechanism.
5

---

Published as a conference paper at ICLR 2026
Algorithm 1: Dynamiics-Predictiv Sampling (DPS) for Active RI. Finetuning
Input: Prompt dataset D; Dirichlet prior ag; Initial state belief ! Batch size B; Decay ratio X; Large language model тo; Total training steps T.
Output: Finetuned large reasoning model Te.
for 1 do
// Se Select. most likely informative prompts for training
Sample batch of prompts B, t Top3 ( D p papior (2) (2))):
foreach r E B, do
L Generate k responses using To and evaluate to obtain of E (1,2,3); Update the LLM ma using trajectories from B, with RL algorithm; // Update solving-stats beliefs and transition dynamics foreach T D do
if yi observed (i.e., r E B₂) then
Compute posterior belief 44, pepest via Bayes' rule by Eq. (9);
Compute posterior transition pseudo-count &7 by Eq. (14);
Update Dirichlet transition posterior: of = 0 1 (1 A) ap + +
else
Set posterior belief 14; p--post the prior belief 14 Tprior
Decay Dirichlet transition posterior: af = X of + A) og:
Generate prior belief 1441 for the next step by Eq. (16);
Overall Algorithm. Integrating these components, we present the complete algorithm DPS in Algorithm 1, with framework overview shown in Fig. 1. A detailed analysis of the time complexity of DPS and its implicit connection to curriculum learning is provided in Appendix B.
4 EXPERIMENTS
In this section, we conduct several experiments to examine the validity of DPS. Appendices D, E, and F provide implementation details, additional results, and data examples, respectively.
4.1 EXPERIMENTAL SETUP
Tasks. We evaluate DPS across three challenging reasoning domains, training separate models on their respective datasets: competition-level mathematics (MATH dataset (Hendrycks et al., 2021)), numerical planning (Countdown dataset (Pan al., 2025)), and visual geometric reasoning (Geom- etry3k dataset (Lu et al., 2021; Hiyouga, 2025)). To further assess its generality. we test a range of large language and multi-modal models that vary capacity and architecture. Models are finetuned with the GRPO algorithm within the verl framework (Sheng et al., 2024) and evaluated by average @ass@1 accuracy over completions per prompt. Details of the training datasets, test benchmarks, and base models reported in Appendix D, with illustrative data examples in Appendix F.
Baselines. We compare against three sampling strategies: (i) Uniform Sampling (US): the default strategy that randomly selects prompts without preference. (ii) Dynamic Sampling (DS); compute- intensive oracle approach that oversamples and filters prompts using rollout feedback (Yu et al., 2025). Here, "oracle" refers sampling batch of all partially solved prompts, instead of achieving the best performance by training on sampled prompts. (iii) History Resampling (HR): an heuristic method that excludes prompts from the dataset they yield all correct responses in the current epoch (Zhang et al., 2025), effectively treating the fully solved state as absorbing at the epoch level.
4.2 PREDICTION ACCURACY OF SOLMING STATES
key component of DPS is online prediction each prompt's 's solving state, which enables adaptive prioritization of partially solved examples during training. We evaluate the accuracy of this predic-

---

Published as a conference paper at ICLR 2026
Effective Sample Ratio (DPS)
Effective Sample Ratio (IIR)
Effective Sample Ratio (US)
Prediction Accuracy (DPS)
Class2 Fl (DPS)
Class2 Recall (DPS)
Class2 Precision (DPS)
Math 1.5B
Countdown 3B
Geometry 3B
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
100
200
300
20
40
60
80
100
120
40
60
80
100
120
Step
Step
Step
Math
Countdown 7B
Geometry 7B
1.0
1.0
0.8
0.8
0.8
0.6
0.4
0.2
100
60
100
120
Step
Step
Step
Figure 2: Proportion of partially solved prompts (Effective Sample Ratio) within sampled batches under different data sampling strategies, along with prediction metrics of DPS.
CM Stop 10
Stepp
at Stop 10
CMit Stop
Stop
Skep 50
18
17922 Teer
E32
24
150
14
Thad Toud
175
100
Thea 1791
200
17
110
Tuad
200
463
-59
180
Brod2 Pred3
Proll: Prodl
Proll Predl
Predl Prod]
Predl Prol)
Pred2 Pred3
(a) Math 1.5B
(b) Countdown 3B
(c) Geometry 3B
Figure 3: Confusion Matrix (CM) for DPS predictions different training steps across tasks.
tion mechanism by treating it as an online classification task. In Fig. 2, overall prediction accuracy is reported to assess general performance across the three classes, while precision, recall, and F1 score additionally reported for Class 2 (partially solved), the state most critical for training efficiency. Throughout training, the predictor maintains high overall accuracy and achieves strong precision and recall for Class 2. Fig. 2 also shows the proportion of partially solved prompts in sampled batches. Compared with US and HR, DPS consistently yields a significantly higher concentration of such prompts, reaching approximately 90% in many tasks.
To further illustrate predictive behavior, Fig. visualizes confusion matrices over training steps, where each cell gives the raw count for each (true, predicted) label pairs. Additional visualizations on more steps are deferred to Fig. 9. As training progresses, diagonal entries strengthen while off- diagonal errors diminish, showing improved discriminability. Notably, the center cell grows more prominent in both predictions and ground truth, indicating that the predictor increasingly emphasizes partially solved prompts. We also report the number of fully solved and unsolved prompts in batches across tasks in Fig. 8. Overall, these results demonstrate that DPS reliably tracks solving progress through lightweight inference and concentrates training on informative prompts.
4.3 RL FINETUNING EFFICIENCY AND PERFORMANCE
Training Progress. Fig. 4 presents the training curves of different sampling methods across tasks and models, where performance is tracked on AIME24 for MATH and on the respective test sets for Countdown and Geometry. DPS exhibits substantially faster policy improvement than US and HR and reaches higher final performance, benefiting from reliable prediction and a greater proportion of informative samples. In contrast, US and HR suffer degradation on MATH, likely due to entropy collapse (Liu et al., 2025a) arising from too few effective samples per batch. We attribute HR's 's less favorable performance to two factors: (i) its epoch-level absorbing transition assumption is overly rigid, limiting adaptability during training: and (ii) it only filters out fully solved prompts, which are often rare in carly and middle stages of training. Relative to the oracle DS baseline, DPS achieves comparable overall performance across tasks and even slightly surpasses MATH. This advan-

---

Published as a conference paper at ICLR 2026
DPS (Ours)
HR
US
DS (Oracle)
Math 1.5B
Countdown 3B
Geometry
34
45
32
Test Accuracy (%)
Test Accupt
30
Teat Accuracy
40
28
60
26
35
30
22
20
40
25
0
100
200
300
20
40
60
80
100
120
20
40
60
80
100
120
Step
Step
Step
Math
Countdown
Geometry
50
Test Test Accuracy (%)
48
Tost Aocurecuyy (%)
80
Test Test Accuracy (%)
48
46
75
-6
44
70
65
42
40
60
40
0
40
80 Step
100
120
40
60
80
100
120
20
40
60
Step
Stop
Figure 4: Training curves of different methods across reasoning tasks with varying model sizes. The
curves in Math are smoothed with window size of DS serves high-resource oracle baseline.
Table I:
Evaluation across mathematics benchmarks. '+' represents finetuning with the method.
Method
AIME24
AMC23
MATH500
Minerva.
Olympiad.
Avg.
Rollouts]
Runtime!
RI-Distill-1.5B
18.33
51.73
76.64
23.83
35.31
41.17
+US
26.46
63.18
82.78
27.46
43.00
48.57
737k
27h
+HR
28.13
64.61
82.88
27.37
43.15
49.23
737k
28h
+DS (Oracle)
31.88
67.32
84.79
29.18
46.83
52.00
2933k
89h
+DPS (Ours)
32.71
67.77
84.95
29.09
46.11
52.13
737k
32h
RI-Distill-7B
37.71
68.45
86.94
34.74
46.94
54,95
+US
45.83
73.57
89.06
37.68
50.42
59.31
287k
30h
+HR
46.46
75.98
90.01
37.94
51.50
60.38
287k
36h
+DS (Oracle)
49.79
90.96
37.89
54.45
62.42
1147k
73h
+DPS (Ours)
51.04
80.35
91.13
37.82
55.32
63.13
287k
39h
tage may stem from differences in sampling criteria: while DS samples randomly from evaluated partially solved prompts, DPS consistently selects the top-B prompts with the highest predicted probability of being partially solved, which might be more beneficial for policy improvement.
Generalization Performance. We evaluate the trained models over multiple challenging benchmarks to assess their generalization capabilities. Table 1 re- ports results for models trained on MATH, evaluated AIME24, AMC23, MATH500, MinervaMath, and OlympiadBench. The models are also evaluated on general reasoning benchmarks, including ARC-c and MMLU-Pro, with results provided in Table 4. Ta- ble 2 presents evaluations on Countdown, where mod- els trained on a subset of the Countdown-34 dataset are tested both the held-out split (CD-34) and harder variant Countdown-4 (CD-4). Table 3 shows evalu- ations on Geometry. Across tasks, DPS consistently outperforms US and HR, while matching exceeding DS in generalization performance.
Table 2: Evaluation Countdown.
Method
CD-34
CD-4
Rollouts
Qwen2.5-3B
+US +HR
69.87
39.42
246k
70.19
42,10
246k
+DS (Oracle)
74.95
47.67
1141k
+DPS (Ours)
74.27
47.78
246k
Qwen2.5-7B
+US
77.84
53.27
246k
+HR
78.15
54.54
246k
+DS (Oracle)
81.26
60.77
1006k
+DPS (Ours)
81.15
59.61
246k
Rollout and Runtime Efficiency. We also compare methods in terms rollout usage and runtime. Tables 1, 2 and 3 report the total number of rollouts, while Fig. 11 plots the model performance as a function of rollout counts. The results demonstrate that DPS achieves strong performance with significantly fewer rollouts than DS, typically using less than 30% of DS's rollout budget to match or exceed its results. Moreover, as shown in Table 1, DPS incurs substantially lower runtime than DS when trained the standard MATH dataset, generally using about half of DS's runtime. While
8

---

Published as a conference paper at ICLR 2026
- ===1
0= 0 -
+===1
- = == =
Geometry 3B
Geometry 3B
Countdown 3B
Countdown 3B
Predictoon Accunacy (%)
Production Accancy (%)
Test Score (96)
Fool Score (%)
80
65
70
73
38
60
60
36
70
50
55
80
120
50
Step
Step
Step
Step
Figure 5: Performance and prediction accuracy of DPS under varying non-stationary decay ratios A.
if state-2 state-3
state-5
state-7
state-2 state=3
statc-5 state-?
Geumetry 3B
Elloctive Sample itatio (96
Geometry 3B
Countdown 3B
Elfeetive Sample Itatio (%)
Countdown 3D
Tost Scare (%)
Tes Soore (%)
70
80
60
50
38
20
40
60
80 Step
100
120
50
75
100
125
60
120
25
50
125
Step
Step
Step
Figure 6: Performance and effective sample ratios of DPS under different solving-state partitions.
DPS exhibits slightly longer runtime than US and HR, this difference is not due to its prediction and selection operations, which are negligible in our experiments. Instead, it arises from longer response generations associated with higher performance, as illustrated in Fig. 13.
Computational Scaling Behavior. We further examine how the computational cost of different operations, including LLM training, LLM generation, and DPS sampling and updates, scales with both dataset size and LLM size. Detailed results and analyses are provided in Appendix E.6.
4.4 ABLATION STUDY
Effects of Non-stationary Decay. The non-stationary decay ratio A [0, 1] controls the extent to which older observations are gradually discounted. As shown in Fig. 5, DPS maintains strong performance over wide range of X across tasks. Notably, removing non-stationary decay (i.e., = 1, which assigns equal weight to all past observations) results in decline in both performance and prediction accuracy. This suggests that the solving-state dynamics is indeed non-stationary and that adaptation to recent observations is crucial. Conversely, setting À = O, which relies solely on the most recent feedback while discarding all past information, also leads to degraded performance and reduced prediction accuracy. A moderate decay ratio strikes balance, allowing the model to remain responsive to recent trends while retaining sufficient historical context for robust estimation.
Effects of Different Solving-State Partitions. We examine the impact of coarser or finer parti- tions of solving states, With two states, prompts are divided into partially solved versus all others. With more than three states, the success rate interval [0. 1] is uniformly partitioned, and prompts predicted to lie near 0.5 accuracy are prioritized, prior work (Bae et al., 2025) suggests these yield the most informative signals. Fig. presents performance and effective sample ratios under different partitions, where the latter is still defined as the proportion of partially solved prompts in each batch. Overall, both metrics decline under either coarser or finer partitions. We attribute this to two factors: (i) coarse partitions that merge fully unsolved and fully solved prompts obscure their distinct dynamics, making transitions harder to model; and (ii) finer partitions distribute limited training observations across more states, resulting sparsity and reduced prediction reliability.
Effects of Transition Priors. The transition prior ag allows flexible incorporation of domain- specific knowledge about plausible transition patterns. The effects of different transition priors on prediction accuracy and training efficiency are analyzed in Appendix E.4.

---

Published as a conference paper at ICLR 2026
DPS wi w
DPS A 0.9 0.9
DPS w/ w 0.7
DPS w/A 0.5
US
Sacppe
Suaple Counts: Min
Sumple Counts.
Prodiction Accuncy
20
15
60
40
Ransom Procppts
Diffiandi Prienpte
2
20
Under-ampled Prompts
50
100
25
(b) Figure 7: (a) Statistics of sample counts across the dataset on the Countdown 3B task. (b) DPS prediction accuracies for different types of prompts on the Countdown 3B task.
(a)
Sensitivity to Response Group Size. We evaluate DPS and US with response group sizes i e {4,8, 8, 16} Countdown. Figs. 15 and 16 present learning curves of test accuracy, effective sample ratio, and DPS prediction accuracy. DPS consistently outperforms US in both performance and effective ratio, with the largest gap that a policy produces a mix of correct and incorrect responses for a given prompt (with probability 1 -pp (1--7 f for success rate p), making US less likely to sample effective prompts, reflected by its very low effective ratio at l = 4. DPS mitigates this by actively selecting effective prompts.
= 4. This is because smaller & reduces the probability
4.5 ADDITIONAL ANALYSIS
Exploration in DPS. Infrequently sampled prompts may have relatively inaccurate state predic- tions, potentially creating a negative feedback loop where they are sampled even less. However, the non-stationary decay in DPS (Eq. (15)) implicitly encourages exploration that mitigates this risk. As the transition posterior decays, predictions for sampled prompts drift toward a uniform distribution; when clearly informative prompts become scarce, these prompts naturally revisited and updated. Fig. 7(a) supports this effect: a smaller decay parameter À yields more uniform sample counts, with lower variance and higher minimum across the dataset.
Prediction Accuracy on Representative Prompts. To examine prediction quality on representa- tive prompts, after each training step, we evaluated three sets: (a) 256 prompts with the fewest past sample counts ounder-sampled prompts), (b) 256 prompts with the highest DPS-estimated probabil- ity of being fully unsolved (difficult prompts), and (c) 256 randomly sampled prompts. Fig. 7(b) shows their prediction accuracies on the Countdown 3B task. Difficult prompts achieve even higher prediction accuracy than random ones, likely because they tend to exhibit simpler or more stable state distributions and transitions, making them easier to predict even with limited observations. Under-sampled prompts typically show slightly lower accuracy, but the gap is small. We conjecture that a similar explanation applies: under-sampled prompts are often confidently predicted to be in State 1 or 3, and thus may largely consist of very hard or very easy problems, which are gener- ally easier to infer. In this sense, the prompts most susceptible to estimation error under infrequent sampling are often those whose states are inherently easier predict. This provides an additional perspective on how DPS mitigates the impact of sparse observations.
5 CONCLUSION AND LIMITATIONS
This work models each prompt's solving progress during RL finetuning as a dynamical system, rep- resenting the solving extent as the state and characterizing its transition with a hidden Markov model. lightweight inference strategy developed to online predict and select informative prompts with- out rollout-intensive filtering. Empirical results across diverse reasoning tasks demonstrate that DPS reduces redundant rollouts, accelerates training, and achieves superior reasoning performance.
A limitation of this work lies in its reliance on correctness-based rewards to define solving states. Nevertheless, the DPS framework naturally extends to more complex reward structures, such as dense praeess-based rewards, by partitioning cumulative return intervals. Furthermore, the use of the straightforward top-k selection strategy may not be optimal. Future work will explore more sophisticated criteria, such entropy-basee prioritization of uncertain samples.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (NSFC) with the Number # 62306326 and the National Key R&D Program of China under Grant 201 2018AA0012201.. We thank all reviewers for their constructive feedback on this work.
ETHICS STATEMENT
This work adheres the ICLR Code of Ethics. All experiments publicly available datasets, and no private, sensitive, or human-subject data are involved. The proposed methods focus on improving training efficiency and do introduce additional ethical risks beyond standard LLM finetuning. We follow dataset licenses and ensure no privacy, safety, or fairness concerns arise.
REPRODUCIBILITY STATEMENT
All theoretical derivations are provided in Appendix C. Full experimental details, including datasets, benchmarks, model configurations, evaluation metrics, RL finetuning procedures, sampling method implementations, and hyperparameters, are provided in Appendix D. All datasets used are public, and we are committed to releasing the complete code to support reproduction.
REFERENCES
Sanghwan Bae, Jiwoo Hong, Min Young Lee, Hanbyul Kim, Jeong Yeon Nam, and Donghyun Kwak. Online difficulty filtering for reasoning oriented reinforcement learning, arXiv preprint 055::25004.03380, 2025.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Leonard E Baum et al. An inequality and associated maximization technique in statistical estimation for probabilistic functions of markov processes. Inequalities, 3(1):1-8, 1972.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference machine learning. pp. 41 41-48, 2009.
Xiaoyin Chen, Jiarui Lu, Minsu Kim, Dinghuai Zhang, Jian Tang, Alexandre Piché, Nicolas Gontier, Yoshua Bengio, and Ehsan Kamalloo. Self-evolving curriculum for Ilm reasoning. arXiv preprint arXiv:2505.14970 2025.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint urXiv:2501.1716.1 2025.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. 2018.
arXiv preprint 1803..18003.00
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.12773. 2023.
Quy-Anh Dang and Chris Ngo. Reinforcement learning for reasoning small Nms:: What works and what doesn't. 't arXiv preprint arXiv:2503. 162/9, 2025.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.0788 2024.

---

Published as a conference paper at ICLR 2026
Mehdi Fatemi, Banafsheh Rafice, Mingjie Tang, and Kartik Talamadupula. Concise reasoning via reinforcement learning. arXiv preprint arXiv:2504.05183 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14006. 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.0388 2021.
Hiyouga. Geometry3K: A large-scale multi-modal geometry reasoning dataset. https:// dat htt.:/ac...../ttatttttss///////e petpu//ggggeeemttr.....
httgg/aa........aaaasse////o///s 2025.
Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang- Thinkprune: Pruning long Chain-of-thougee of Ilms via reinforcement learning. arXiv preprint arXiv:2504.01290 2025.
Jian Hu. Reinforce++: simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262 2025.
Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zerro An open source approach scaling up reinforcement learning on the base model. arXiv preprint 1xXiv:2503.242000 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412. 2024.
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking potential for Ilm reasoning through refined credit assignment. arXiv preprint arXiv:2410.01679, 2024.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances Neural Information Processing Systems, 35:3843-3851 2022.
Xuefeng Li, Haoyang Zou, and Pengfei Liu. Limr: Less is more for rl scaling. arXiv preprint arXiv:2502. 1886, 2025.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2023.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong- Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprini 25X5:22505.2486. 2025a.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi. Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding ri-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025b.
12

---

Published as a conference paper at ICLR 2026
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang. Xiaodan Liang, and Song-Chus Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. In The Joint Conference the 59th Annual Meeting of the Association for Computationnal Linguis fics and the Ithh International Joint Conference on Natural Language Processing (ACI-IJCNLP 2021), 2021.
Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weher, et al. Deepcoder: A fully open-source 14b coder at 03-mini level. Notion Blog, 2025a
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, et al. Deepscaler: Surpassing o ol-preview with 1.5 b model by scaling rl. Notion Blog. 2025b.
Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, et al. Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning. CoRR, 2025.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35:
27730-27744, 2022.
Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. Tinyzero. https://eihuu..ccm/aaiiiiPpii SatiiaayiiaaaTTyyeeeee Teny 2025. Accessed: 2025-01-24.
Yun Qu, Qi Wang, Yixiu Mao, Vincent Tao Hu, Björn Ommer, and Xiangyang Ji. Can prompt difficulty be online predicted for accelerating rl finetuning of reasoning models? arXiv preprint arXiv:2507.04632, 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728-53744 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXiv:177700633 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.0338 2024.
Qianli Shen, Daoyuan Chen, Yilun Huang, Zhenqing Ling, Yaliang Li, Bolin Ding, and Jingren Zhou. Bots: A unified framework for bayesian online task selection in Ilm reinforcement finetun- ing. arXiv preprint arXiv:2510.26311 2025.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 19266, 2024.
Zhiqing Sun, Sheng Shen. Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309. 14525, 2023.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXi::2501. 2025.
Qi Wang, Zehao Xiao, Yixiu Mao, Yun Qu, Jiayi Shen, Yiqin Lv, and Xiangyang Ji. Model predic- tive task sampling for efficient and robust adaptation. arXiv preprint arXiv: 2501. 10099, 2025a.
Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with training example. arXiv preprint arXiv:2504.2057/ 2025h.
13

---

Published as a conference paper at ICLR 2026
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu pro: A more robust and challenging multi- task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266-95290 2024.
Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, et al. Light-rl: Curriculum sft, dpo and rl for long cot from scratch and beyond. arXiv preprint arXiv:2503. 10460, 2025.
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong. Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Ilm reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2502.14776. 2025.
Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-ol: Let vision language models reason step-by-step. arXiv preprint arXiv:241 10440, 2024.
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under offlcpoicy guidance. arXiv preprint arXiv:2504. 14945, 2025.
Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv e-prints, pp- arXiv-2412. 2024a.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint 2xXi:22499.1222.. 2024b.
Yixin Ye. Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint 2tXiv:2502.03387 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint S.4i::250031444777 2025.
Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan. What's behind ppo's collapse in long-cot? value optimization holds the secret. arXiv preprint arXiv:2503.01491, 2025.
Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint #xXiv22504.05/16 2025.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian Hc. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint ar arxiv:2503. 18892, 2025.
Xiaojiang Zhang, Jinghui Wang. Zifei Cheng. Wenhao Zhuang, Zheng Lin, Minglei Zhang, Shaojie Wang, Yinghan Cui, Chao Wang, Junyi Peng, al. Srpo: A cross-domain implementation of large-scale reinforcement learning on 1lm. arXiv preprint arXiv:2504. 14286, 2025.
Haizhong Zheng, Yang Zhou, Brian R Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, and Beidi Chen. Act only when it pays: Efficient reinforcement learning for I1m reasoning via selective rollouts. arXiv preprint 2XXx:25566001777,, 2025.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006-5501,, 2023.
14

---

Published as a conference paper at ICLR 2026
A RELATED WORK
RL for LLM Optimization. Reinforcement learning (RL) has become a pivotal technique for adapting large language models (LLMs) to complex tasks and desired behaviors. In particular. Re- inforcement Learning with Human Feedback (RLHF) has proven effective for aligning LLMs with human preferences and safety constraints (Ouyang et al., 2022: Dong et al., 2024; Rafailov et al., 2023; Dai et al., 2023; Sun et al., 2023; Sheng et al., 2024). In domains where reward signals are verifiable, such as mathematics, code generation, and symbolic planning, Reinforcement Learning with Verifiable Rewards (RLVR) has been shown to substantially enhance the reasoning capacity of LLMs (Jaech et al., 2024; Shao et al., 2024; Team et al., 2025; Chu et al., 2025; Guo et al., 2025). From an algorithmic perspective, Proximal Policy Optimization (PPO) (Schulman et al., 2017), foundational policy gradient method in RL, is directly applicable to LLM finetuning. More recently, Group Relative Policy Optimization (GRPO) (Shao et al., 2024) eliminates the computational over- head of PPO's value network by introducing a lightweight, group-normalized advantage estimator, and has rapidly become one of the most widely used RL finetuning algorithms. Subsequent refine- ments have focused on mitigating gradient bias, reducing training instability, and lowering compu- tational cost (Yuan et al., 2025; Yue et al., 2025; Liu al., 2025b; Yu et al., 2025; Kazemnejad et al., 2024; Hu, 2025). On the application side, substantial efforts have extend RL finetuning to broader task domains and increasingly large-scale models (Luo et al., 2025b; Dang & Ngo, 2025; Luo al., 2025a; Zeng et al., 2025; Meng et al., 2025; Xu et al., 2024). At the same time, infrastructure- advances have developed scalable frameworks for distributed and compute-efficient RL training tai- lored to LLMs (Sheng et al., 2024; Hu et al., 2025).
Data Selection for RL Finetuning. A growing body of work emphasizes that the effectiveness of RL finetuning critically depends on the quality of training data (Guo al., 2025; Yang et al., 2024b), which has motivated growing interest in data curation as driver of efficient learning (Hu et al., 2025; Wen et al., 2025; Wang et al., 2025a). A common approach offline data filtering, which ranks or selects prompts prior to training based on static heuristics such as estimated difficulty, domain balance, or diversity (Ye et al., 2025; Li et al., 2025; Zhou et al., 2023; Wen et al., 2025; Hu et al., 2025; Yang et al., 2024b; Fatemi et al., 2025; Wang et al., 2025b). While beneficial, this approach introduces preprocessing overhead for ranking or clustering and, more importantly, fails to adapt to the model's evolving competence during training. To address this limitation, recent work has investigated online selection strategies that dynamically choose prompts in response to the model's isse current behavior (Yu et al., 2025; Zhang et al., 2025). One class of methods performs per-step selection, either by filtering out uninformative prompts (Yu et al., 2025; Liu et al., 2025a; Cui et al., 2025; Meng et al., 2025) or by focusing on examples of intermediate difficulty (Bae al., 2025). While these strategies improve the quality of training samples, they remain hindered by the high computational cost of inttenttittesivee filtering or by limited accuracy difficulty estimation. Alternative approaches adopt per-epoch data selection, updating the sample set periodically (Zhang et al., 2025; Zheng et al., 2025). However, these methods typically rely on coarse heuristics or empirical trends observed over epochs, which limits their responsiveness and often introduces high estimation error. Concurrently with this work, some studies (Qu et al., 2025; Shen et al., 2025) are motivated by a similar predict-then-sample principle, in which intermediate-diffien prioritized before rollout. These methods typically model each prompt's 's success rate as a latent variable and formulate prompt selection as Bernoulli bandit problem. However, this approach is better suited to settings with relatively stable success rates and is less flexible than DPS in capturing complex dynamics underlying continual model evolution. Moreover, under sparse observations, lacks a reliable mechanism for extrapolation over unobserved intervals. Our approach formalizes prompt-solving progress as a dynamical system and introduces a tractable inference strategy for step-wise prompt selection with negligible computational overhead, achieving accurate prediction, fast convergence, and superior performance under a low rollout budget.
prompts
B DISCUSSIONS
Time Complexity. We analyze the time complexity of Uniform Sampling (US), DS (Yu et al., 2025), and DPS. DS repeatedly samples candidate prompts, performs LLM rollouts, and discards those that fail to meet predefined constraints until [B] prompts are retained. Let Pkeep denote the expected probability that sampled prompt is retained in DS, Cim the expected cost for generating
15

---

Published as a conference paper at ICLR 2026
and evaluating & LLM rollouts per prompt, Cpred the expected cost of inference per prompt in DPS, and Cropk the expected cost of top-k selection over the dataset in DPS.
The expected time complexity for prompt selection and evaluation per step is: O (IB)Cim) for for DS, and O (ID)C + Clopt + [B|/Cmm) for DPS. Since our method involves only very low-dimensional matrix operations (Cpred; Cropk < Cim), it holds that O (12) Cpee + Ctapk + [3] (il) ~ O (lm)) Therefore, DPS significantly reduces computational overhead compared to DS while typically adding negligible cost relative to the default US. The pre- diction and selection overhead in DPS scales approximately linearly with the dataset size [P]. For existing popular datasets, this overhead is negligible. However, for potential extremely large datasets where the cost may become non-trivial, one can approximate the full-dataset updates and selection using a randomly sampled candidate subset B (BB) [B]
US, O (I Pheep [
(D]) at each step.
Implicit Curriculum Learning. Beyond maximizing learning signals, this selection strategy in- duces an implicit form of curriculum learning (Bengio et al., 2009). Early in training, prompts with high State 2 probability are typically easier ones, for which the model begins show partial success. As training progresses and the model improves, these prompts may transition to the fully solved state (State 3) and are no longer selected. Conversely, harder prompts that were initially always incorrect (State 1) may begin to yield partially correct responses, making them eligible for sampling.
This mechanism creates a self-paced progression from easier to harder prompts: beginning with tractable examples to bootstrap learning, then gradually shifting to more challenging cases as model capacity grows. Moreover, by targeting prompts in the partially solved regime, the method avoids both trivial and unsolvable cases, which provide little training benefit and may waste resources. Crucially, this adaptive curriculum is not manually curated but emerges naturally from the method, providing principled and scalable alternative to handcrafted curricula.
C PROOF AND DERIVATION
Derivation of the Transition Update. observed emissions y/2 E {1, 2,3 as:
The posterior transition pseudo-count (t, is defined for
{(i,)) P(2-11 = i Via), if (1,2,3).
(18)
The joint posterior distribution can be expressed as:
P = J, 22 i y1.1) P(2,-1 j. = i, Via)
(19)
P(y::)
Using the Markov property 2, V1---1 | 2 and the conditional independence of observations Ye 1 11 2e, the numerator factorizes as:
P(2-1 = j, = i, v1:)) P( 2-1 j) P(2 = i| 21-1 j) P(ye 2 1). (20)
Substituting into the posterior expression yields:
P = j. 2 v1+) ((V1:111; = 1) P(zz = i 2c-1 = j) Plys 2t =
(21)
P(yn:)
P(--1 = 11 P(( = =i| 12-1 = 1) PP(y I | =)
(22)
P(y 91-011)
(23)
Using the notation 1 (j) := P(2-11 = j V1--1). 1.-1(i.)) = P(2 = i 2--1 j). and
P(ye 2 i) := P(y 24 i), we obtain:
post (1) T (i.j) p(U: 1 2e = i)
PP = j. = i v1:e) =
(24)
P( | 11t-11)
16

---

Published as a conference paper at ICLR 2026
The normalizing denominator P( 11:1-1) can be obtained by marginalization:
P(y (1.4-1) = _
u past (4)) Pe Pe-1('',
Σ
p(ua 2$ = 1').
(25)
3'
Therefore, the full expression becomes:
post (j) 4 (8,J) p(yh 1 2t =
Mi-1
P(=--1 j, = v1.1)
(26)
Ey M (j') Ev 1 1 (i', ply, 2t = 8)
Under the deterministic emission model in Equation (7), we have p(ye zt = - 8(yr, i) for
Ve E (1, where å denotes the Kronecker delta function. Substituting gives:
14-1 (i) @ (i. 8(ye,
P( = j. = y1:e)
ye E (1,2,3). (27)
Eg P4-1 (i) Pr 22-1(0,3)) d'
pust
Note that 8(ye, 1') non-zero only when yt. so the inner sum over i' reduces (2-1 Thus, the expression simplifies to:
post (j) 1.-1(i.5)
if yr, Vr [1,2,3)
P(2-11 = j,2 = (1:)) =
E, (j") 4 10,1((
post
(28)
if + ye Ve [1,2,3}.
Setting & 0 for unobserved y/ so that the Bayesian update in Equations (11) and (15) defaults to the prior without new evidence, the expression of & simplifies to:
Mi (j) (i-)(i
if yt.
(ii,))
Σ (i)) -1-1111,7)
(29)
otherwise.
D EXPERIMENTAL DETAILS
D.I DETAILS OF TASKS AND MODELS
We evaluate DPS across three distinct and challenging reasoning domains: competition-level math- ematics, numerical planning, and visual geometric reasoning. To verify its broad applicability, we experiment with a range of large language and multi-modal models with varying capacities and chitectures. We adopt the popular GRPO algorithm implemented within the verl framework (Sheng al., 2024) to fine-tune models. Evaluation based on average pass@l accuracy computed over 16 independent completions per example. Training datasets, test benchmarks, and base models in each domain are detailed follows, with illustrative data examples provided Appendix F.
D.1.1 MATHEMATICS
Training Dataset. For mathematies, train large reasoning models the training split of MATH dataset (Hendrycks et al., 2021), consisting of 7,500 problems designed to reflect competition-level difficulty. Specifically, use the Hugging Face release from https:/ // buggingface co/ ht/ddaaattttDiiii htwaaaatt//iiiiaaaaaannnngeeeeoo with prior work (Sheng al., 2024).
MMAHHE/AATH---A HitH--lghheell.... consistent
Test Benchmarks. We assess performance across diverse mathematics benchmarks including AIME24, AMC23, MATH500 (Lightman et al., 2023), Minerva Math (Lewkowyez et al., 2022), and OlympiadBench (He et al., 2024), with all the datasets obtained from DeepScaler (Luo et al., 2025b). . In particular, AIME24 is used to monitor training progress and plot the training curves. We additionally evaluate the trained models et al., 2018) and MMLU-Pro (Wang et al., 2024). We follow the evaluation setup in Yan et al. (2025) and adopt PRIME's prompt template for evaluation.
general reasoning benchmarks, including ARC-c (Clark
17

---

Published as a conference paper at ICLR 2026
Base Models. Following prior work (Luo al., 2025b), two base models from DeepSeek (Guo et al., 2025) are used: Depppeex-al---iaaiii-oovrrr-r_Le from Hugging Face repository https::
-app-eeee-----iiii1------------- from https: htttps://hugginggaaceee co/
and beepsekkal/Deggse---l-------
D.1.2 NUMERICAL PLANNING
the Countdown Number Game, where agents must construct the target number using basic operations over given number set (Pan al., 2025). Training is carried out on 2,000-item subset of the complete Countdown-34 dataset at Hugging Face repository https:/ haapss:/hugginggeee hww.ona.......aaaaaaaeeoee wwt/atasetss/Jaa//aaaaaaaoaa
Training Dataset. For arithmetic planning,
Countdown- Tasks-3to4.
Test Benchmarks. Models are evaluated on two benchmarks: (i) CD-34, containing 512 held-out problems from Countdown-34; (ii) CD-4, including 512 problems from Countdown-4, a harder generalization version that operates 4 numbers, accessible at https: / h/wugginggafccccc datasets SaiiissSiaaaa Jiayi-Paaaa PaanConnnooonr Sowunndooo-aasss--ae In particular, CD-34 is used to monitor training progress and plot the training curves.
co/
Base Models. Following prior work Chen et al. (2025), we test with two base models from /wwnn2...
Qwen (Yang al., 2024a): Qwen2. 5-3B from https: httpl:::///uggingaceee 5B and Qwen2. 5-7B from https: htt.mttts///uugiiieeeeeeee htt.g/ggggggaccceee/e////e/e/eee co Owen/ 1oven2.5-720
D.1.3 VISUAL GEOMETRY
Training Dataset. Visual geometry experiments leverage the training split of the Geome- try3k dataset (Lu et al., 2021; Hiyouga, 2025), accessible from https: pettps://bugee http///hggggggaceeeeeeeo co/ datasets /wwyyugaa/aa///e//aaa/aaa htt///gometttgg//// questions, requiring both image understanding and symbolic reasoning.
The dataset comprises 2, 20101 diagram-based geometry
Test Benchmark. We evaluate trained models on the benchmark test comprising 601 visual reasoning problems.
Base Models. For visual geometric reasoning, we adopt two vision-language models from Qwen (Bai et al., 2025): Owen2. ---V--3--Instuuee from https: http::s:///hggnggaaeeeeee co/Qwen//weene Qwen2. 5 V-VL- wonnssuuuttt and 5--.::22------------------ from https: http:///ceeeeo////////ee/ee/e/// Qwte///en///// Qwen2. 5- -B-IIIIruc.
//buggingface. co
D.2 IMPLEMENTATION DETAILS
RL Finetuning Implementations. Our method and all sampling baselines shared the same RL finetuning implementations, detailed follows. We adopt the popular GRPO algorithm (Shao al., 2024) implemented within the verl framework (Sheng et al., 2024) to fine-tune models. Evaluation is based on average pass@l accuracy computed over 16 independent completions per prompt sampled with temperature 0.6 and nucleus sampling parameter top-p = 0.95, following the setup of Luo al. (2025b). For each training step, we generate & 8 responses per prompt under temperature 1.0 and top-p = 1.0 to compute advantage estimates and finetune models. An entropy regularization term with weight 0.001 is introduced, consistent with Luo al. (2025b). Models is optimized with Adam (Kingma & Ba, 2014), using constant learning rate of 1e-6 momentum parameters (0.9, 0.999), no warm-up, and weight decay of 0.01. We further adopt the Clip-Higher scheme in DAPO (Yu al., 2025), which employs asymmetric clipping bounds, Elow 0.2 and Ehigh 0.28.
Task-specific training configurations are as follows: batch size is set to 256 for MATH (mini-batch 128) and Countdown (mini-batch 64), and to 512 for Geometry3k (mini-batch 256). The maximum output length is set to 8192 tokens for MATH and 1024 tokens for Countdown and Geometry3k. The KL-divergence penalty is omitted in actor loss for MATH and Countdown, following (Yu al., 2025), but preserved Geometry3k to maintain stable optimization, with a coefficient of 0.01 for 3B models and 0.03 for 7B models. For MATH, we use binary reward function that assigns a reward
18

---

Published as a conference paper at ICLR 2026
of 1 for a correct answer and O otherwise, following the default setup in verl (Sheng et al., 2024), while for Countdown and Geometry3k, we include a format bonus 0.1 in the reward function if the response is incorrect but with correct formatting, following the setup in Pan et al. (2025).
All experiments are executed on NVIDIA A100 GPUs with 80GB memory.
Sampling Method Implementations. For Dynamic Sampling (DS) (Yu et al., 2025), we directly use the implementation from verl (Sheng et al., 2024), where prompts with zero variance in rewards are filtered out at each training step. For History Resampling (HR) (Zhang et al., 2025), we imple- ment it within the verl framework by excluding prompts from the training dataset if they yield all correct responses in the current epoch. For DPS, we initialize the state belief n 11 (i) = 1/3 and set the Dirichlet prior as (i.j) = 1, assuming no prior knowledge about both the initial prompt- solving states and transition probabilities. Thus, the only hyperparameter that requires tuning the non-stationary decay ratio X, which is set to 0.7 for MATH and 0.5 for Countdown and Geometry3k.
DPS (Ours)
HR
DPS (Ours)
HR
# Fully Unsolved per Batch
# Fully Solved per Batch
Fully Unsolved per Batch
Fully Solved per Butch
120
100
150
100
175
80
125
150
60
100
60
100
40
40
75
50
20
25
20
25
0
0
1000
200 Step
300
100
2000
300
0
50
100
50
100
Stup
Step
Step
(a) Math 1.5B
(b) Math 7B
DPS (Ours)
HR
US
DPS (Ours)
HR
US
250
Fully Unsolved per Batch
Fully Solved per Batch
#
Fully Unsolved per Batch
200
Fully Solved per Batch
140
140
175
200
120
1.20
150
mmm
150
100
100
M
125
100
100
60
60
75
50
40
40
50
20
20
25
0
0
0
25
50
75 Step
100
125
50
75
100
125
25
75
100
125
25
50
75 Step
100
125
Step
Step
(c) Countdown 3B
(d) Countdown 7B
DPS (Ours)
HR
US
DPS (Ours)
HR
US
# Fully Unsolved per Batch
Fully Solved per Batch
# Fully Unsolved per Batch
Fully Solved per Butch
300
120
200
2.50
100
200
80
1.50
80
150
60
100
60
100
40
40
50
20
20
0
0
0
25
50
75 Stop
100
125
0
25
50
75 Stop
100
125
0
20
40
60
80
0
20
40 Step
60
80
Stop
(e) Geometry 3B
(1) Geometry 7B
Figure& 8: Number of ineffective prompts (fully unsolved or fully solved) in training batches sampled with different strategies across tasks.
19

---

Published as a conference paper at ICLR 2026
CM at Siop
CM Step 5
CM Step to
CM Step 50
CM Step 100
CM as Step 300
Toun Thur! Thar
Touc Trucl Trucl
Trud Trui
Teacl Tracl Traet
Touc Truck
137
207
199
220
30
Prod] Prui2 Pred3
Prudt tredd Profl
Predi Phatl Prods
Pred2 Prod?
Pred1 Pred? Prods
Prodi Prrdt Phod3
(a) Math
1,5B
CM at Stop
CM Step 5
CM Step to
CM ot Step 50
CM Stop 2001
CM an Step I4D
Trxet Trse Treal
Tiee
Trud That
Irsel (nel
2000
Tose Trse
300
Predit Pred3
Prodl Pheil
Predl Prod3
Predh Prod?
Predi Prod2
Prodi Podd3
(b) Math 7B
CM at Skep
CMii Stup S
CM at Step to
CM at Step 50
CM Step 100
CM an Step 120
150
352
Tons Tracl
190
Trud Toa2
37
Trud
Prodi Predi Pred3
Pred2
Prodi Preds
Pred2
Predi Preds
Prodi Predi Pred3
(c) Countdown
at Stop
CM Stop 5
CM Step
CM Step 50
CM Stop 2001
CM at Step 120
Ind Inet Inee
H198
-300
Trxt TIR2
7:4
-00
125
Trd
Pusdl Pred2t Pred3
Prodl Brel2
Prefl Prdd2 PredS
Pred2 Pied
Prell Pied2 Freds
Piell Predz Pas13
(d) Countdown
Stop
CM at Step
Step
S1sp330
al Step
at Step 180
253
Tous Thxr Thel
46
62
13
10
21
Tracl Tnsc Tnne
14
10
Truc Tiec
240
200
-20
Tou2
422
423
17
-xee
100
Profl Prudt Prid
Prodt tred2 Proft
Pradi Proft Pruct
Proti Pruc2 Proft
Prod1 Prod2 Prndi
Profi Proi Prud3
(c) Gcometry 3B
Stop
CM at Step
Step
Step
al Step
ai Step
374
TONE Towt
21
41
-00
14
-xc
10
26
-100
14
-X00
32
Trus Twee
29R
0
36
384
200
17
413
9
420
14
420
23
392
16
Tows
21
190
Proft Frod Pred3
Prodt Bred2 Frodt
Predl Prott Prwcs
Proft Prot2 Profs
Pred1 Pret2 Prods
Proft Prod Pru33
(f) Geometry 7B
Figure 9: Confusion Matrix (CM) for DPS prediction at different training steps across tasks.
E EXTENDED EXPERIMENTAL RESULTS
E.
ADDITIONAL PREDICTION RESULTS
A key component of DPS is the real-time prediction of each prompt's solving state, which enables adaptive prioritization of partially solved examples during training. We evaluate the accuracy of this prediction mechanism by treating it as a dynamic classification task. This section provides additional analysis to complement Section 4.2.
20

---

Published as a conference paper at ICLR 2026
Number of Fully Solved and Unsolved Prompts. Figure 8 reports the number of fully solved and fully unsolved prompts in batches across tasks. The results show that DPS consistently and significantly yields fewer fully solved and fully unsolved prompts than US across all tasks. In addition, HR treats the fully solved state as absorbing, which much stricter than that of DPS. As result, HR produces the fewest fully solved prompts across tasks but also the largest number of fully unsolved prompts. Overall, this leads to substantially lower effective sample ratio for HR compared to DPS, as shown Figure 2.
Confusion Matrix. Figure 9 visualizes confusion matrices over training steps across tasks, where each cell shows the count for each (true, predicted) label pair. As training progresses, diagonal entries strengthen while off-diagonal errors diminish, indicating improved discriminability. Notably, the center cell becomes increasingly prominent in both predictions and ground truth, suggesting that the predictor places greater emphasis on the target region.
Overall, these results demonstrate that DPS reliably tracks solving progress through lightweight inference and concentrates training on desired prompts.
E.2 ADDITIONAL EVALUATION RESULTS
Evaluation Geometry. Table 3 shows evaluations the Geometry task, where models trained and tested on the respective official Geometry3k datasets. The results show that DPS out- performs US and HR under the same rollout budget. On the other hand, DPS matches DS while requiring significantly fewer rollouts, making more scalable in practical settings.
Table 3: Evaluation results Geometry.
Qwen2.5-VL "VL-3B-Instruu
Qwen2.5-VLe E-VL-7B-Innstruc
Method
Test Score
Rollouts
Test Score
Rollouts
US
40.69
492k
46.22
328k
HR
40.44
492k
46.52
328k
DS (Oracle)
44.33
1262k
48.11
782k
DPS (Ours)
44.47
492k
47.78
328k
Evaluation General Reasoning Benchmarks. We additionally evaluate the MATH-trained models on general reasoning benchmarks, including ARC-c (Clark et al., 2018) and MMLU- Pro (Wang et al., 2024). We follow the evaluation setup in Yan al. (2025) and adopt PRIME's prompt template for evaluation. The results are provided Table 4. On these general (OOD) rea- soning tasks, DPS also shows consistent improvements over the baseline methods.
Table 4: Evaluation on general reasoning benchmarks for models trained on the MATH dataset. Performance is measured by Pass @ accuracy with maximum response length of 8k tokens. '+' represents finetuning with the method.
Method
ARC-c
MMLU-Pro
Avg. f
Rollouts +
Runtime +
RI-Distill- 1.5BB
41.81
21.02
31.42
+US
43.17
21.24
32.21
737k
27h
+HR
42.83
21.03
31.93
737k
28h
+DS (Oracle)
44.88
23.25
34.07
2933k
89h
+DPS (Ours)
46.16
23.41
34.79
737k
32h
RI-Distill-7B
74.32
50.44
62.38
+US
75.09
50,59
62.84
287k
30h
+HR
74.57
51.56
63.07
287k
36h
+DS (Oracle)
77.05
51.43
64.24
1147k
73h
+DPS (Ours)
78.67
52.37
65.52
287k
39h
21

---

Published as a conference paper at ICLR 2026
Evaluation with the Llama Model. Beyond Qwen-series models, further train Llama-3.2-3B Instruct on Countdown to evaluate different sampling methods. Figure 10 compares the resulting test accuracies and effective sample ratios. The results show that, with Llama-3.2-3B 2-----InnBtuutt, DPS also performs comparably to DS and surpasses HR and US both test accuracy and effective sample ratios, with even larger relative gains than those observed with the Qwen models.
DPS (Ours)
US
(Oracle)
Countdown Llama3B
Countdown Llama3B
Countdown Llama3B +DPS
Effective Sample Ratio (%)
75
80
Test Score (%)
70
0.8
0.6
60
55
40
0.4
Accuracy
Class2
20
Class2 Recall
45
Class2 Precision
40
20
40
60
80 Step
100
120
0
25
50
75 Step
100
125
20
60 Step
100
120
(a) Comparisons performance and effective sample ratios
(b) DPS prediction metrics
Figure 10: Comparisons of different sampling methods using the additional model
Instruct. (a) Performance and effective sample ratios. (b) Prediction metrics of DPS.
Evaluation with Extended Response Length. We further explore generalization by conducting out-of-distribution study: MATH models, trained with a maximum response length of 8k, tested under an extended 32k response budget. The results are reported in Table 5. DPS not only continues to surpass US and HR. but also slightly outperforms DS, showing clear advantages from the increased response length. These results highlight the scalability and generalization capacity of DPS large-context settings.
Table 5: Evaluation across mathematics benchmarks under maximum response length of 32k. '+' represents finetuning with the method. Evaluation is based on average Pass@) accuracy over 16 responses per prompt.
Method
AIME24
AMC23
MATH500
Minerva.
Olympiad.
Avg.
Rollouts
Runtime 1
R1-Distill-I.s5
28.12
61.67
83.18
26.54
43.33
48.57
+US
31.46
67.70
84.22
27.94
45.06
51.28
737k
27h
+HR
30.42
66.49
84.30
27.53
45.06
50.76
737k
28h
+DS (Oracle)
32,92
69.95
86.44
30.26
49.66
53.85
2933k
89h
+DPS (Ours)
37.92
71.16
48.32
54.48
737k
32h
E.3 ROLLOUT EFFICIENCY
Figure demonstrates that DPS and DS significantly accelerate RL finetuning over US and HR in terms of training steps. Yet, such comparisons overlook the cost of LLM rollout inference, which often exceeds finetuning itself. Because DS depends on oversampling, it rolls out a larger batch of prompts per training step, which substantially increases LLM inference overhead. Figure plots performance against rollout numbers during training. The results show that DPS reaches strong performance with far fewer rollouts than DS, typically requiring less than 30% of DS's rollout budget to match or surpass its results.
E.4 EFFECTS OF TRANSITION PRIORS
Our approach allows flexible incorporation of inductive bias by modifying the Dirichlet prior over the transition matrix. While the default configuration uses uninformative prior av(i,) 1 for all (i.j), many real-world scenarios may exhibit structural regularities their solving dynamics. This section investigates how certain priors affect prediction accuracy and training efficiency.
22

---

Published as a conference paper at ICLR 2026
DPS (Ours)
HR
US
DS (Oracle)
Math 1.5B
Countdown 3B
Geometry
34
75
70
30
40
28
35
24
Test
50
Test
30
22
25
1.0
2.5
0.0
1.2
Rollouts (x10%)
Rollouts (x10%)
# Rolloous (x10%)
Math 7B
Countdown 7B
Goometry 7B
50
48
Text Accuracy (96)
48
Accuracy (%)
46
46
Tes Accuracy
44
44
42
Tost
63
42
40
40
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
#:
Rollouts (x10%)
Rollouts (x10%)
# Rollouts (x10%)
Figure 11: Training curves over the number of rollouts generated by LLM during training.
UninSormative
Stahility
Local
Progres
Uninfommative +
Stahility
Local Progress
Geometry
Geometry
Countdown 3B
Courtdown 3B
Prodiction Acarncy (99)
90
75
Prediction Accuriicy (96)
44
70
42
80
Test (%)
65
60
40
20
60
38
36
60
35
20
50
34
40
60
100
120
25
50
75 Step
100
125
40
60
80 Step
100
120
25
50
75
100
125
Step
Figure 12: Performance and prediction accuracy of DPS under different transition priors.
We evaluate several representative priors, each encoding a different structural assumption: (i) Sta- bility prior (stability
Assigns larger pseudo-counts to self-transitions (ap(i, = 1. (i, 0.5 for i + f), which suppresses frequent state changes and reflects belief that solving states tend persist across steps. (ii) Progress prior Canti-regression): Sets lower pseudo-counts for regression transitions (ao(i,)) = 0.5 for < j), imposing a preference against regressing from solved state to less solved one. (iii) Local prior (local-transition): Sets (o(;, 0 for j ---i > 1, suppressing long-range transitions while retaining flexibility for adjacent-state updates. This encodes an assumption of smooth, gradual evolution solving dynamics.
These priors are evaluated under identical training settings, with results shown in Figure 12. We report both task performance and prediction accuracy. We find that certain structured priors can lead to slight improvements over the uninformative baseline, particularly during early training stages where data limited (see Countdown for example). As training progresses and more data becomes available, the advantages of structural priors diminish with degraded prediction accuracy, due to potential mismatch between the prior's bias and the actual dynamics. This highlights the tradeoff between introducing prior structure and maintaining long-term flexibility.
E.5 RESPONSE AND PROMPT LENGTH
Response Length. Response length has been identified strong correlate of reasoning abil- ity (Yu et al., 2025). Figure 13 illustrates how different strategies influence this metric during MATH training. The average response length of DPS initially aligns with US and HR but quickly
23

---

Published as a conference paper at ICLR 2026
increases, following trajectory similar to DS. This also suggests that DPS rapidly learns the under- lying prompt-solving dynamics. Both DPS and DS generate responses that are consistently longer than those from US. Longer outputs provide opportunities for deeper exploration and enable the model to engage in more complex reasoning processes, which may partly explain the observed per- formance gap (Yu et al., 2025). Notably, in the MATH 7B setting. HR exhibits sharp increase in response length during later training stages. We attribute this to HR's ' rigid exclusion rule: once prompt fully solved some epoch, is permanently removed, even if errors may occur later. Un- der the stronger 7B model, this removes too many relatively easy problems, substantially raising the average difficulty of the remaining set. Faced with unsolvable inputs, the model tends to generate excessively long responses, often approaching the length limit (Hou et al., 2025).
(Ours)
HR
DS (Oracle)
Math 1.5B
Math 7B
6000
6000
Average Response Lengih
Average Response Lengh
5000
5000
4000
4000
3000
3000
2000
2000
1000
1000
350
Step
Step
Figure 13: Average response length in the sampled batch during MATH training.
Prompt Length. Figure 14 tracks the average length of sampled prompts throughout MATH train- ing. Compared with US, all of DPS, HR, and DS tend to select longer prompts, and the average prompt length increases slightly as training progresses. This trend can be explained as follows. Since DS and DPS target partially solved prompts, improvements in trained model competence shift the training batches toward more difficult prompts, which are statistically often longer. Likewise, HR's i exclusion of already fully solved examples leaves progressively harder pool of prompts, also corresponding to greater length on average.
DPS (Ours)
HR
US
DS (Oracle)
Math 1,5
Math 7B
140
150
130
Average Prompe Length
140
130
120
Average Prompe
120
110
Average
100
80
Stop
Step
Figure 14: Average prompt length in the sampled batch during MATH training.
E.6 EMPIRICAL ANALYSIS ON COMPUTATIONAL SCALING BEHAVIOR
This section conducts experiments to examine how the computational cost of different operations scales with both dataset size and LLM size.
(1) Computational scaling with dataset size. We construct pseudo-datased (with arbitrary size (D)) systematically evaluate the cost of DPS sampling and updates. Specifically, at each step
24

---

Published as a conference paper at ICLR 2026
T-prior randomly generate [D] transition posterior matrices aj and belief vectors H4 popt and '4 corresponding to all [D] pseudo-samples. In the sampling stage, we perform top- B selection on 14 (. in the HMM-update stage, we assign random observations to the batch of B samples and apply independent HMM updates to all [D] samples. For comparison, the per-step costs of LLM training and generation, which are independent of dataset size, are obtained by finetuning the 7B model on MATH. Table 6 reports the per-step costs of different operations for dataset sizes ranging from 10a to 107. The runtime and memory usage of DPS scale approximately linearly with dataset size. yet even for a very large dataset of size [D] = 10", DPS requires only 2.4s of runtime and 0.9 GiB of memory, while consuming GPU memory. In contrast, LLM training and generation together require about 1100s of runtime and 600 GiB of GPU memory.
Given its linear scaling, the computational overhead of DPS could become nog-negligible at suf- ficiently large scale ((D) > 10"), though such dataset sizes are beyond typical practical settings. For these cases, Appendix B also discusses scheme that approximates the full-dataset updates and selection using a randomly sampled candidate subset B satisfying B [B] < (D).
Table 6: Computational cost of different operations across varying dataset sizes, measured by per- step runtime and memory usage during the finetuning of DeepSeek- R1 R1-Distill--Wws Ow (8 A100 GPUs, batch size 256). The results for LLM training and generation are evaluated on the MATH dataset, while those for DPS are obtained on datusess that emulate large-scale scenarios.
LLM train
LLM generation
DPS (sample update)
Dataset size
any
any 520 6000 (GPU)
10 (MATH)
10
106
107
Runtime (8)
580
0.0005+0.002
0.004+0.02
0.06+0.2
0.6+1.8
Memory (GiB)
600 (GPU)
R=0.0009
es.00
ss0.9
#00.9
(2) Computational scaling with LLM size. The cost of LLM training and generation scales with model size. In particular, the additional rollout cost of DS also grows with LLM size, whereas DPS, as rollout-free alternative DS, incurs no such dependence. Table 7 compares the per-step costs of different operations for 1.5B and 7B models. The total runtime of LLM training and generation increases from roughly 370s to I 100ss as the model size increases from 1.5B 7B. At the 7B scale, the additional overhead introduced by DS versus DPS is approximately 1500s vs. 0.003s. Therefore, the advantage of DPS can become increasingly significant as L.M size grows.
Table 7: Computational cost of different operations across varying LLM sizes, measured by per- step runtime for finetuning the MATH dataset (8 A100 GPUs, batch size 256). The 1.5B and 7B models refer to DeepSeek-E R1 Distill-Qwen- and DeepSeek-Be -1 -isttill-tweenn
respectively.
LLM train
LLM generation
DS sample (baseline)
DPS (sample update)
Model size
1.5B
7B
1.5B
7B
1.5B 2 3x200
7B 2: 3x520
any 0.0005+0.002
Runtime (s)
170
580
200
520
E.7 SENSITIVITY ANALYSIS ON THE RESPONSE GROUP SIZE
We evaluate DPS and US under different response group sizes A: E [4,8, 16) on the Countdown 3B task. Figures 15 and 16 present the learning curves of test accuracy, effective sample ratio, and DPS prediction accuracy. The results show that DPS consistently outperforms US with both higher performance and effective sample ratios, and the advantage of DPS is the most pronounced when k = 4. For US the performance with k = 4 drops substantially compared to A = 8 and 16 (falling to less than half), whereas DPS exhibits only a slight decrease (about 4%). In particular, for k 4, the test accuracy of DPS is more than twice that of US.
We attribute this to the fact that for smaller k the probability that the same policy produces a mix of correct and incorrect responses for the same prompt becomes lower (given a fixed success rate p, the probability of generating mixed responses is 1 pt (1 (1-p)*). Hence, with a smaller k. the default US is much less likely to sample effective prompts (reflected in the extremely low effective sample ratio of US at & = 4 in Figure 16). This creates greater potential for improvement when using DPS, which actively selects effective prompts. On the other hand, smaller k may lead to
25

---

Published as a conference paper at ICLR 2026
more frequent state transitions and make the underlying dynamics harder estimate. Nevertheless, as shown in Figure 16, DPS maintains high prediction accuracy at the -mall-yet-raccticl value of k 4, leading to substantial performance gains. Consequently, in scenarios where the response group size is constrained, such as under limited training resources, applying DPS is likely to be particularly advantageous.
DPS (Ours)
US
Responses: &
Responses:
Responses:
75
75
70
Test Accurac (%)
Test Accuncy (%)
65
65
50
55
55
30
45
45
100
80
120
Step
Step
Step
Figure 15: Performance of DPS and Uniform Sampling (US) under different response group sizes on the Countdown 3B task.
Effective Sample Ratio (DPS)
Effective Sample Ratio (US)
Prodiction Accuracy (DPS)
Responses: k
Responses: -
# Responses: K
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
20
40
100
120
100
120
20
80
100
120
Step
Step
Step
Figure 16: Effective sample ratios and prediction accuracies under different respone group sizes on the Countdown 3B task.
E.8 ENTROPY REGULARIZED SELECTION SCHEME
Introducing exploration into sample selection could potentially improve the model's robustness. To this end, we test a variant, DPS+Entropy, that explicitly balances exploitation and exploration by combining the entropy of the predicted distribution with the State-2 probability for Top-B sam- pling. We conduct experiments on Countdown and tune the entropy regularization coefficient in (0.01, 0.1, 1, 10). The training curves are shown Figure 17. DPS+Entropy performs best when the coefficient is 0.1, but it does not yield noticeably greater improvement over DPS in either test accuracy or effective sample ratio. We provide further analysis below.
While the Top-B selection strategy is purely exploitative, it exploits an objective (i.e., the predicted probability) that already incorporates degree of exploration. The non-stationary decay mechanism in DPS (Fq. (15)), although originally designed to accommodate non-stationary dynamics, also implicitly introduces exploration. gradually decays the transition posterior and drifts the predicted states of under-sampled prompts (i.c.. those predicted to be in State or 3) toward more uniform distribution, increasing their likelihood of being selected and updated. This behavior is supported by Figure 7(a), which shows that smaller decay ratio À leads more uniform sample counts, with lower variance and higher minimum across the dataset. Hence, the additional entropy term partly overlaps with this built-in exploration effect, which may account for the limited improvement.
We also note that the specific choice of selection criterion is not the primary focus of this work. Once the state distribution is predicted, any selection criterion, such softmax selection entropy-based
26

---

Published as a conference paper at ICLR 2026
sampling, can be applied. DPS adopts simple criterion and introduces few hyperparameters as possible (with only X) while already achieving strong performance.
DPS (Ours)
DPS+ Entropy0.01
DPS+ Enttrop0. + D
DPS+ DPS+oottopy10 -
US
Performance
Effective Sample Ratio
DPS Prediction Accuracy
75
Effecsive Sample Ratio (%)
Prediction Accuracy (%6)
70
80
80
65
60
60
60
40
40
55
20
120
Step
Step
Step
Figure 17: Evaluation of the entropy regularized selection scheme for DPS on Countdown 3B task.
E.9 COMPARISON WITH ADDITIONAL BASELINES
Simple Non-probabilistic Heuristic. We implemented simple predictive baseline, denoted Var+EMA, that tracks exponential moving average of the reward variance for each prompt and samples the prompts with the Top-B values across the dataset. We conduct experiments on Count- down with Var+EMA, tuning the EMA smoothing factor in (0,0.1, 0.1, 0.5, 0.9) and choosing 0.5 yields relatively better performance. The comparative results in Figure 18 show that DPS outper- forms Var+EMA with higher test accuracy and effective sample ratios. The following analyzes the necessity and advantages of the HMM framework over this simple predictive heuristic. (i) Dynam- ies estimation. Var+EMA implicitly assumes that the solving extent of each prompt tends to persist across steps, which resembles maintaining a fixed, stability-promotte Therefore, this heuristic less flexible than DPS capturing more complex underlying dynamics that may arise in practice. (ii) State prediction. Due to the infrequent sampling of given prompt, its reward-variance observations are unavailable on most steps. Under this setting, Var+EMA lacks re- liable mechanism to extrapolate and predict variance during these unobserved intervals. In contrast, a core advantage of the HMM framework is its ability to model state transitions and, crucially, to extrapolate under missing observations. Regarding hyperparameters, DPS uses only parameter, the non-stationary decay ratio X, whereas Var+EMA uses an EMA smoothing factor.
transition model in DPS.
DPS (Ours)
Var+EMA
Diversity
US
Countdown Llama3B
Countdown Llama3B
Effective Sample Ratio (%)
80
Test Score (%)
70
60
60
55
40
20
45
20
40
80
100
120
25
50
75
100
125
Step
Step
Figure 18: Comparison with additional baselines in terms of performance and effective sample ratio.
Diversity-based Sampling. We also implement a baseline that performs active sampling based on batch-level sample diversity. Specifically, we first pre-sample candidate batch that is times larger than the actual training batch, and embed each prompt into a 1024-dimensional vector using WordLlama. We then iteratively select the candidate whose embedding maximizes the cumulative pairwise L2 distance to previously selected samples, thereby greedily constructing a batch with high dispersion in the embedding space. We evaluate this variant Countdown and tune the candidate
27

---

Published as a conference paper at ICLR 2026
batch size multiplier E {2, 8). ultimately selecting 4 as it yields slightly better perfor- mance. As shown in Figure 18, diversity-batt bressity-based sampling offers only marginal improvements over US, and both its test accuracy and effective sample ratio remain far below those of DPS.
E.10 PRELIMINARY EXPLORATION OF EXTENSIONS TO CONTINUOUS PROCESS REWARDS
This section first discusses the main challenges of applying active sampling in prwcess-reward set- tings, and then presents a preliminary exploration of extending DPS to continuous process rewards.
Our focus on binary rewards reflects their practical prevalence and their well-understood connec- tion with sample informativeness, which enables principled sampling strategies. contrast, how process rewards relate to informativeness remains unclear in the field. To our knowledge, existing methods that incorporate process rewards still rely on binary outcome rewards when applying active sampling; for instance, PRIME (Cui et al., 2025) uses process rewards for RL finetuning but applies an accuracy-based sampling filter as in DS. Thus, a key open challenge in process-reward settings to first establish a meaningful link between process rewards and sample informativeness, which would enable DPS or other sampling strategies to be applied in a principled way.
We conduct preliminary investigation of applying DPS continuous process rewards based on a simple hypothesis: prompts whose average trajectory returns fall into an intermediate range may be more informative. Specifically, we compute return for each response by summing its process rewards, and then categorize each prompt's 's average return into one of three intervals defined by two boundaries, aiming prioritize prompts in the middle interval. Using PRIME (Cui al., 2025) as the testbed, we explore two DPS variants. The first uses fixed boundaries: since PRIME augments outcome rewards with small implicit process rewards, we simply set the boundaries to 0 and 1. The second uses dynamic, quantile-based boundaries, estimated from observed returns using quantiles 0.2 and 0.8, and updated via an exponential moving average (smoothing factor 0.9). As shown in Figure 19, the synamic-boundar Countdown and also increases the proportion of partially solved prompts in training batches. This improvement is likely due the ability of dynamic boundaries mitigate potential issues such as interval mismatches and sparse observations that may arise under fixed boundaries. We leave the development of more refined rewrsss--wwrd--bbaa active sampling strategies for future work.
variant outperforms both the fixed-boundary variant and US
DPS+Dynamic Boundaries
DPS-Fixed Boundaries
Performance
Partially Solved Ratio
60
70
Partially Solved Ratio
Test Score (%)
60
40
50
30
40
10
20
60
80
100
120
0
20
60 Step
80
100
120
Step
Figure 19: Evaluation in precess reward setting the Countdown 3B task. Sampling strategies are applied to the PRM-based method PRIME (response group it = 4, base RL algorithm RLOO).
F DATA EXAMPLES
We provide below the illustrative data examples for each of the tasks in experiments. Prompt templates for MATH and Geometry3k are drawn from verl (Sheng et al., 2024), whereas Countdown employs the template in Pan et al. (2025).
28

---

Published as a conference paper at ICLR 2026
MATH Data Example
Prompt:
Given a prime p and an integer a, we say that a primitive root (mod p) if the set {a, a², aa, ..... a²' ] contains exactly one element congruent to each of 1, 2, ......p (mod p).
For example, 2 is a primitive root (mod 5) because {2, 22,22,2, ( = (2,4,3,1) (mod 5), and this list contains every residue from to 4 exactly once.
However, 4 is not primitive root (mod 5) because {4, 42, 43, = (4,1,4,, (mod 5), and this list does not contain every residue from to exactly once.
What is the Let's s think step by step and output the final answer within [boxed[}.
of all integers in the set 21, 3, 4, 5, 6) that primitive roots (mod 7)2
Answer:
8
Countdown Data Example
Prompt:
A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer.
User: Using the numbers [63, 95, 96], create an equation that equals 64. You can use basic arithmetic operations (+,-, *, /) and each number can only be used once. Show your work in <think>> < /think>> tags. And return the final answer in <answer>> < //anwwr>> tags, for example <answer> (1 2)/3 </answer>.
Assistant: Let me solve this step by step.
<think>
Geometry3k Data Example
Prompt:
W
S U
In ARST, Z the centroid and RZ 18. Find ZV. You FIRST think about the reasoning process as internal monologue and then provide the final an- swer. The reasoning process MUST BE enclosed within <think> < /think> tags. The final answer MUST BE put in \boxed().
Answer:
9
G STATEMENT ON LLM USAGE
This work was completed without any substantive contribution of large language models (LLMMs). The authors used L.MMs exclusively for post-writing refinement. All core aspects of this work, including research idcation, methodology development, theoretical derivation, code implementation, experiments exccution, and results analysis, were conceived and conducted solely by the authors.
29


Reviews:
[{"review_id": "OjMIH4i4qY", "reviewer": "Reviewer_4vBK", "summary": "The paper introduces Dynamics-Predictive Sampling (DPS), an online prompt-selection framework that enhances reinforcement-learning (RL) finetuning of large reasoning models (LRMs). The core motivation is that dynamic sampling strategies (e.g., Dynamic Sampling (DS), History Resampling (HR)) improve RL finetuning efficiency by prioritizing informative prompts—typically those that are only partially solved—but at the cost of significant rollout overhead.\nDPS aims to predict which prompts are likely to be informative before performing expensive LLM rollouts. It models each prompt’s solving progress as a Hidden Markov Model (HMM) whose latent states correspond to unsolved, partially solved, and fully solved stages. Using Bayesian updates of transition and emission probabilities, DPS infers prompt-specific state distributions online and selects prompts with the highest predicted probability of being partially solved.\n\nThe proposed method is lightweight and scalable, requiring only low-dimensional updates per prompt. Experiments on mathematical reasoning (MATH, AIME24, AMC23), numerical planning (Countdown), and visual geometry (Geometry3K) demonstrate that DPS.", "questions": "None", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper offers a new dynamical-systems perspective on data selection in RL finetuning, bridging prompt-level reward evolution with state-space modeling.\n\nDPS eliminates the need for redundant LLM rollouts by predicting informative prompts, yielding 2–3× runtime reduction and 70 % rollout savings while maintaining accuracy parity with DAPO.\n\nComprehensive experiments across three reasoning domains and multiple model scales. Evaluation includes accuracy curves, confusion-matrix analyses of prediction reliability, and ablation studies (non-stationary decay λ, number of states, prior α₀) demonstrating robustness and interpretability.", "weaknesses": "Reward formulation dependency:\n\nThe approach currently depends on binary correctness rewards, which are straightforward in math-style benchmarks but not directly transferable to open-ended or process-based domains (e.g., code synthesis with partial correctness). The authors acknowledge this and suggest extending DPS to dense or step-wise rewards.\n\nSimplistic selection criterion:\n\nDPS uses top-B selection on predicted State 2 probabilities. More nuanced criteria (e.g., entropy-based uncertainty, expected information gain) could further enhance exploration-exploitation balance.\n\nScalability in very large datasets:\n\nWhile the complexity analysis asserts negligible overhead, maintaining per-prompt HMM parameters might become costly when |D| ≫ 10⁶ unless sampling approximations are used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ciH5hj9DJ0", "reviewer": "Reviewer_Nhzv", "summary": "This paper introduces Dynamics-Predictive Sampling (DPS), a method that predicts the informativeness of training samples by modeling their learning progression as a Hidden Markov Model (HMM), thereby replacing costly LLM rollouts for sample selection. The method's primary strength is its computational efficiency, shifting the overhead from LLM inference to lightweight matrix operations. This claim is well-supported by comprehensive experiments across diverse reasoning tasks.\nHowever, its limitations warrant attention. First, its three-state model is tailored for binary rewards, and its generality for tasks with denser reward signals (e.g., from process supervision) is unclear. Second, by maintaining independent dynamics for each prompt, the method may suffer from data sparsity issues, leading to unreliable estimates for infrequently sampled prompts and thus impacting its robustness.", "questions": "- The state observation depends on generating k responses per prompt, creating a trade-off between cost and observation noise. The paper uses k=8 but does not include a sensitivity analysis for this hyperparameter. An analysis or discussion on the impact of varying k—and how effectively the HMM can mitigate noise from smaller k values—would provide a clearer picture of the method's robustness.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The core contribution lies in shifting the sample selection overhead from LLM inference costs, which are proportional to batch size, to computationally negligible matrix operations. Experiments convincingly show that DPS can match or exceed the performance of baselines with substantially lower computational resources, a critical factor for the practical adoption of RL fine-tuning.\n- The study features a comprehensive experimental design across diverse and complex reasoning tasks (mathematics, planning, visual geometry), with thorough comparisons against key baselines (e.g., Uniform Sampling, Dynamic Sampling). The inclusion of extensive ablation studies enhances the credibility of the methodological design and provides good support for the core arguments.", "weaknesses": "- The proposed three-state model (unsolved, partially solved, solved) is highly effective for binary reward settings. It would be beneficial to discuss the framework's extensibility to tasks with denser reward signals, such as continuous scores from process supervision. Addressing questions like how the state space could be partitioned (e.g., fixed vs. dynamic boundaries) would provide valuable insight into the method's potential applicability to a wider range of problems.\n- The model maintains independent transition dynamics for each prompt. This raises a concern about data sparsity: for infrequently sampled prompts, the estimated dynamics may be unreliable, which could compromise the accuracy of state predictions. It would strengthen the paper to explain how this potential issue is handled. Additionally, an analysis of what happens to persistently under-sampled, difficult problems—and whether their learning dynamics risk stagnation—would be a valuable addition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hTqx701Pcq", "reviewer": "Reviewer_xSRN", "summary": "This paper proposes Dynamics-Predictive Sampling (DPS), a novel method to reduce the high computational cost of online prompt selection during the reinforcement learning (RL) finetuning of large reasoning models. The core problem is that existing methods, like Dynamic Sampling (DS), must perform extensive and costly LLM rollouts on large candidate batches to identify informative, \"partially solved\" training examples. DPS avoids this by modeling each prompt's solving progress as a dynamical system, specifically a hidden Markov model (HMM). This HMM tracks the prompt's latent \"solving state\" $z_{t}^{\\tau}$ (e.g., fully unsolved, partially solved, or fully solved). Using historical reward signals, DPS performs online Bayesian inference to efficiently compute a predictive prior $\\mu_{t}^{\\tau,prior}(2)$, estimating the probability that a prompt is in the most informative \"partially solved\" state before any rollouts are generated. The main contributions are this HMM-based predictive framework and empirical results showing DPS substantially reduces redundant rollouts (e.g., achieving strong performance with less than 30% of DS's rollout budget), accelerates training, and achieves comparable or superior reasoning performance to costly oracle methods across diverse mathematics, planning, and visual geometry tasks.", "questions": "Could you provide a justification for choosing an HMM framework over simpler, non-probabilistic heuristics? For instance, did you experiment with tracking a simple exponential moving average of reward variance for each prompt to identify \"partially solved\" items, and how would its performance and efficiency compare to DPS?\n\nThe Top-B selection strategy is greedy, exploiting prompts known to be in State 2. Did you consider alternative selection strategies that incorporate exploration, such as prioritizing prompts with high entropy in their predicted state distribution (i.e., high uncertainty), and how might that impact learning stability and final performance?\n\nThe definition of the solving states (unsolved, partial, solved) seems highly dependent on the number of samples, k=8. How does the model's performance and the HMM's state estimation change with different values of k (e.g., k=4 or k=16)?\n\nYou briefly mention that the framework could extend to dense or process-based rewards. Could you elaborate on how you would concretely map a continuous reward signal onto the discrete 3-state system? For example, would you use fixed thresholds, and how would you prevent this from re-introducing the observation-sparsity issues you noted in the ablation study on finer-grained partitions?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper's primary strength is its direct and effective focus on improving the computational efficiency of active prompt selection for RL finetuning. The work is presented with good clarity, first identifying the high cost of existing online sampling methods (like DS) that rely on extensive rollouts, and then proposing a clear, lightweight alternative. In terms of originality, the paper offers a practical methodological refinement; rather than introducing a new sampling goal, it iteratively improves the mechanism by using an HMM to predict a prompt's solving state, thereby avoiding the need for costly pre-evaluation. The quality of this contribution is substantiated by a solid empirical evaluation across several reasoning tasks. The results consistently show that the proposed DPS method achieves performance comparable to the compute-intensive DS baseline while operating with a significantly reduced rollout budget, as detailed in the tables and efficiency graphs. The work's significance is, therefore, practical: it provides a viable method to achieve the benefits of dynamic data curation without the prohibitive computational overhead, making active sampling a more feasible option for practitioners.", "weaknesses": "* The necessity of the HMM framework is not fully justified, as the paper omits comparisons to simpler predictive baselines. A non-probabilistic heuristic, such as tracking an exponential moving average of reward variance for each prompt, might also identify \"partially solved\" items with similar efficiency gains and less modeling overhead.\n* The claim of \"negligible\" computational overhead is only validated on relatively small datasets (e.g., 7.5k for MATH). The method's costs scale linearly with the total dataset size |D|, as it maintains and updates a separate HMM for every prompt, which could become a practical bottleneck for datasets with millions of examples.\n* The Top-B selection strategy is purely exploitative, as it only samples prompts with the highest predicted probability of being in State 2. This greedy approach ignores sampling based on uncertainty (e.g., high entropy across the HMM state belief), which could be crucial for efficiently detecting state transitions and improving the model's robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "epKcQlsgdr", "reviewer": "Reviewer_joQK", "summary": "This paper presents an online prompt selection method to improve the efficiency and the effectiveness of RLVR. The main motivation is to keep the informative prompts for training (those that yield both correct and wrong responses for the current policy). Instead of following previous work such as DAPO which may need sampling a big number of rollouts to obtain responses that have different rewards, this paper proposes a method named DPS to perform online Bayesian inference to estimate the state distributions. Three states represent, for a single prompt: all questions are incorrect, all are correct, and the rest. The prompts with the highest probabilities in the \"partially solved\" state will be selected for training.\nFor evaluation, the authors compared with several selection strategies, such as uniform sampling, history resampling (discard all prompts once the responses are all correct), and the \"oracle\" DS used in algorithms such as DAPO. The results on several reasoning benchmarks show the effectiveness and efficiency.", "questions": "other suggestions: \n\n+ The models are Qwen-series. The authors are suggested to test the performance on a model in other model families.\n+ The authors may need to clarify that \"oracle\" actually refers to finding all prompts that have both successful and failed responses, instead of a method that can achieve the best performance by training on sampled prompts.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Compared to DS, this method achieves comparable performance yet with significantly fewer rollouts.\n+ This paper addresses a key problem in RLVR about how to effectively select data.", "weaknesses": "+ In each training step, the state belief for each prompt in D is updated, which makes it less efficient than baseline sampling methods. This may be a problem when data scales, although the training data used in this work is usually relatively small-scale.\n+ The authors did not show the method's sensitivity to the number of rollouts per prompt. It is unclear whether this method is still stable with a smaller number of rollouts.\n+ The baselines are relatively naive and rule-based. More advanced methods, such as entropy/diversity/gradient-based selection, should be considered.\n+ The training data and evaluation benchmarks are primarily math-related. The authors may consider general-domain tasks for evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)