BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning
Authors: Qianli Shen
Daoyuan Chen
Yilun Huang
Zhenqing Ling
Yaliang Li
Bolin Ding
Jingren Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement finetuning (RFT)
Large Language Models (LLMs)
Online task selection
Bayesian inference
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose BOTS, a Bayesian framework for online task selection in LLM finetuning that integrates explicit and implicit evidence with posterior sampling, achieving efficient and effective training.

Abstract: Reinforcement finetuning (RFT) is a key technique for aligning Large Language Models (LLMs) with human preferences and enhancing reasoning, yet its effectiveness is highly sensitive to which tasks are explored during training. Uniform task sampling is inefficient, wasting computation on tasks that are either trivial or unsolvable, while existing task selection methods often suffer from high rollout costs, poor adaptivity, or incomplete evidence. We introduce \textbf{BOTS}, a unified framework for \textbf{B}ayesian \textbf{O}nline \textbf{T}ask \textbf{S}election in LLM reinforcement finetuning. Grounded in Bayesian inference, BOTS adaptively maintains posterior estimates of task difficulty as the model evolves. It jointly incorporates \emph{explicit evidence} from direct evaluations of selected tasks and \emph{implicit evidence} inferred from these evaluations for unselected tasks, with Thompson sampling ensuring a principled balance between exploration and exploitation for task selection. To make implicit evidence practical, we instantiate it with an ultra-light interpolation-based plug-in that estimates difficulties of unevaluated tasks without extra rollouts, adding negligible overhead. Empirically, across diverse domains and LLM scales, BOTS consistently improves data efficiency and performance over baselines and ablations, providing a practical and extensible solution for dynamic task selection in RFT.

Published as a conference paper at ICLR 2026
BOTS: A UNIFIED FRAMEWORK FOR BAYESIAN ON- LINE TASK SELECTION IN LLM REINFORCEMENT FINETUNING
Qianli Shen"
Daoyuan Chen
Yilun Huang
Zhenqing Ling
Yaliang Li
Bolin Ding!
Jingren Zhou
Alibaba Group
ABSTRACT
Reinforcement finetuning (RFT) is key technique for aligning Large Language Models (LLMs) with human preferences and enhancing reasoning, yet its effec- tiveness is highly sensitive to which tasks are explored during training. Uniform task sampling is inefficient, wasting computation on tasks that are either trivial or unsolvable, while existing task selection methods often suffer from high roll- out costs, poor adaptivity, or incomplete evidence. We introduce BOTS, unified framework for Bayesian Online Task Selection LLM reinforcement finetuning. Grounded in Bayesian inference, BOTS adaptively maintains posterior estimates of task difficulty as the model evolves. jointly incorporates explicit evidence from direct evaluations of selected tasks and implicit evidence inferred from these evaluations for unselected tasks, with Thompson sampling ensuring a principled balance between exploration and exploitation for task selection. To make implicit evidence practical, we instantiate it with an ultra-light interpolation- based plug-in that estimates difficulties of tasks without extra rollouts, adding negligible over- head. Empirically, across diverse domains and LLM scales, BOTS consistently improves data efficiency and performance over baselines and ablations, providing a practical and extensible solution for dynamic task selection in RFT
INTRODUCTION
Reinforcement finetuning (RFT) has become a key technique for aligning Large Language Models (LLMs) with human preferences and enhancing their reasoning capabilities (Jacch et al. 2024 Guo
al. 2025 Luo al. 2025 Hu et al. 2025 Zeng et al. 2025). However, the effectiveness of RFT
is highly sensitive to task selection (Parashar et al. 2025 [hen et al.) 2025] Zhu et al.) 2025 Wen
jet al. 2025 Li et al. 2025a). Naively training on a static, uniformly sampled dataset inefficient: the model spends excessive computation on tasks that are either already mastered (too easy) or beyond reach (too hard) (Yu et al., 2025; Bae et al. 2025 Chen et al.) (0255).. This inefficiency not only inflates training costs but also destabilizes optimization by reducing the effective batch size. The central challenge, therefore, is to dynamically select tasks of "just right" difficulty to maximize learning efficiency the model's capability evolves.
Existing methods to this challenge face several limitations. Offline task selection (Parashar et al,, 2025 Shen et al. 2023 Zhu et al. 2025 Wen al.) 2025 Li et al. 2025a), which pre-schedules tasks from easy to hard, is too rigid and does not adapt to the evolving trajectory of the model. In re- sponse, few online selection methods have been proposed, aiming adaptively choose tasks based on model's 's current capability. Core challenge of these methods lies in the tradeoff between the com- putational cost of collecting information and the accuracy of the resulting performance estimates. We argue that existing solutions are not sufficiently efficient: some expend excessive computation on information gathering, undermining efficiency, while others fail fully exploit collected infor- mation, leading to suboptimal selection. On one hand, wversamplnng-basee methods (Yu et al.) (025)
Bae et al. 2025) find suitable tasks by rolling out oversized batches, introducing substantial extra
"qqual contribution
"Corresponding Authors. Email to (shenqianli. sql, yaliang. li, bolin.ding @alibaba-inc.com wwtp:///wwwneeeeeee/e/etetteeeee
Our code is released at

---

Published as a conference paper at ICLR 2026
Prier
Extimation
Close to pru0.57
Rein forcementt F Finetuning
Per Stap
Perfonsance
ky turr
BOTS Selection
M
Netwoor.8)
by
suppliag
Betwln) Ab
Qwen
Randor Selaction
Betweed AP)
VVVV
BUTS Vanii1o
Training Staps
Interpolatton--lase Taplictt Feidence [Fec
X
Posturior
Update 3.2)
XVXX
SPP Book Redel
Gurnect Nodel X
Strragg Hoas
Eqpilett avioe for silecied take
X
1M0 4
XVVX XXXX
trotect Rdtme inilaced
XVVV XVVX
Implsctt Eviderce
Overview of the BOTS framework. BOTS operates in a continuous loop of task selec- tion, model training, and posterior updating. (1) Selection: Thompson sampling from the posterior beliefs selects a batch of tasks whose estimated success probabilities are near target difficulty (e.g.. p- = 0.5). (2) Training & Evidence Collection: The LLM is finetuned, yielding direct suc- cess/failure counts (explicit evidence) for the selected batch. For unselected tasks, predicted counts (implicit evidence) are produced by a plug-in; in Section 3.3 we introduce an ultra-lightweight interpolation- variant with negligible overhead. (3) Posterior Updating: Explicit and implicit evidence are fused using our generalized Bayesian update rule (Section 3.2). The BOTS framework
Figure 1:
and the Interpolation- -bsed Implicit Evidence are summarized as Algorithm |~ 2 in Appendix C
cost. On the other hand, non-oversampling approaches typically rely on a single source of infor- mation-either leveraging historical evaluations as explicit evidence (Chen al. 2025b Qu al., 2025) or exploiting inter-task correlations as implicit evidence (Sun et al.) (2025). 5. Our empirical results reveal a clear complementarity. explicit evidence provides stable and accurate task-difficulty estimates but suffers from a slow warm-up when historical evaluations are scarce in early training, whereas implicit evidence quickly guides early-stage selection yet becomes less reliable in later stages. These findings indicate that relying solely on one type of evidence leaves information un- derutilized and leads to suboptimal task selection. Therefore, a principled framework to fusee these complementary evidence sources is essential for robust and efficient online task selection.
In this work, we introduce BOTS, the first unified and extensible framework for Bayesian Online Task Selection in LLM reinforeement finetuning. BOTS recasts online task selection as a princi- pled Bayesian inference problem over the model's evolving capabilities. By doing so, it naturally addresses the core challenges of non-stationarity and partial observability, featuring with three key design elements: (1) Bayesian foundation: Grounded in Bayesian inference, the framework nat- urally adapts to the evolving capability of the model, allowing task difficulty to be continuously re-estimated. (2) Integration of two evidence sources: Tunable update rules jointly incorporate explicit evidence from direct evaluations and implicit evidence inferred from related tasks, leverag- ing their complementary strengths. (3)hhompson sampling: Task selection is guided by posterior sampling, ensuring principled balance between exploration and exploitation.
For implicit evidence, we further instantiate the framework with an extremely efficient interpolation- based plugin that estimates the difficulty of unevaluated tasks without additional rollouts, making the overhead negligible. We demonstrate empirically, across diverse domains and model scales, that our method significantly improves data efficiency and model performance over baselines and ablations, offering practical, effective, and extensible solution for online task selection in RFT.
2 RELATED WORKS
The impact of task difficulty RFT of LLMs has become active research topic. Inspired by the seminal idea of curriculum learning (Bengio et al. 2009), researchers have proposed various strategies for selecting appropriate tasks in L.M RFT. For instance, Parashar et al. (2025); Shen
2

---

Published as a conference paper at ICLR 2026
et al.
2025) advocate scheduling tasks from easy hard, enabling LLMs to gradually acquire reasoning skills. Similar ideas have been extended to multi-modal LLMs (Zhu et al. 2025; Wen
et al. 2025 Li et al. 2025a), though these methods mainly focus offline task selection following
an easy-to-hard trajectory.
More recently, online selection strategies have emerged, often targeting tasks of moderate difficulty. One line of work adopts sampling-bats saalling--adddd--e task filtering, where tasks with consistently trivial rewards (all zeros ones) are considered uninformative and down-weighted filtered (Yu et al. 2025 Bae et al. 2025). While effective, these methods require additional rollouts, incurring non-trivial overhead. To avoid extra rollouts, several works attempt to predict task passing rates without di- rect rollouts. Chen et al.] (2025b) formulate task selection as a non-stationary multi armed bandit problem, treating each problem category (e.g., difficulty level or type) as an arm and using abso- lute advantage as reward proxy, with posterior estimation based on historical outcomes, Qu et al. (025) extend this framework to the task level. Although these methods eliminate extra rollouts, they rely solely on direct evaluations and overlook cross-task relationships. In contrast, [unn et al. (2225) propose evaluating a small set of reference tasks and predicting the passing rates of others using attention-inspire kernel over embeddings. However, this approach still requires additional rollouts for the reference set and discards historical evaluation information.
More comprehensive discussion and comparison related works are provided in Appendix B
BAYESIAN ONLINE TASK SELECTION
3.1 PRELIMINARIES: MODELING TASK DIFFICULTY
A task T (Q, R) is defined as tuple consisting of query Q expressed in natural language, and a reward function R that maps any natural language response O to binary reward R(Q, O) {0, 1), which is common in domains like math, coding such that indicates correct and 0 indicates incorrect. Consider RFT a parameterized language model MMg. which maps a query Q to a response O, a task T follows a Bernoulli distribution Be Bernoulll (PO.T), where P0,T = Mo-M(-) R(O, T) denotes the model's success probability T. With slight abuse of notation, we denote the reward distribution for a given model and task as R(-IT; 0) ;= Bernoull. i (po.T). Since we focus on online task selection over a fixed set of tasks, we simplify the notation by letting p denote pa.. Th and R. denote R(:IT>; 0z), whenever the context is clear. All notations are summarized in Table
of N tasks (T*) The binary reward obtained by executing the model on
3.2 CORE MECHANISM: FUSING EVIDENCE IN UNIFIED POSTERIOR
Our goal is to estimate the success probability o of the online-adapted model on k-th task T* For efficiency, direct evaluations are only performed after a task selected. As statistical evidence, at time step t, obtain online samples rf.z. iid. Rt (-) for each selected task T in the training batch B, where n corresponds to the number of rollouts per task.
A natural way model the estimation is via Beta distribution, Beta .88). where the posterior parameters o and 3 represent the accumulated counts of successes and failures, respectively, for model lr on task T The problem then reduces designing online adaptation rules for of and 3k. We propose the following online adaptation rules: Given a batch of direct evaluation results
B. ((TB.())
Iin
) we define the adaptation rules
af+1 (1 (1- + Aaa (1- s +pst,
B4,1 -1- + + s: + f#. (1)
where oxo, 3o denote the prior parameter set for the Beta distribution, and the coefficient À E [0, 1] discounts historical information by interpolating the counts with the prior (Raj Kalyani 2017):
Σ TH
1
h Σ
TW-
A: Σ +
KEB.
KEB.
denote the explicit success and failure counts from direct evaluations, by slightly abusing the notation I E B, represent task * received direct evaluation at time step t. Notice when direct evaluation
3

---

Published as a conference paper at ICLR 2026
results not available (A: & Bd), st st 0: and
5 = S + l[kk & Br) p(k. Be) n. + B.] (1, n.
(3)
Here, E [0. 1] balances the contributions explicit and implicit evidence, & and ft coincide with i and f when direct evaluation results are available for task Tk. and otherwise represent the pseudo success and failure counts. These are derived from an estimator p(k: B,), which uses inter- task relationships
infer difficulty for tasks framework places no restrictions on the specific form p(kk, B,), while in Sec. 3.3 we introduce a lightweight interpolation based instance to produce the pseudo counts. Additionally, manage the equivalent total sample size-and hence the uncertainty of the estimate-the pseudo sample size is ensured to satisfy 8 + ft n.
present in the current evaluation batch Br. Our
The following proposition indicates that the update Equation ( preserves the Beta family as the (generalized) posterior for a Bernoulli parameter under a cempered/prior-mminng
update.
Proposition Lei p (0, 1) be the Bernoulli success probubility at time I. Suppose the current belief is ne (p) = Beta(p | 04,, Bc), and let wo(p) = Beta(p cro, Po) be buse prior. Given counts (St, fi) and pseudo counts (5t; with St, ft, St, fo 2 0, define the generalized-Bie Bayes update
(p)
Te(p)²~> To(D)>
[p (1-1))'T 1-p
(p'(LLD))
(4)
prior mixing/ / discounting
tempered explicis likethhood
tempered implicis evidence
with À E (0, 1) and p E [0. 1]. Then T is exactly Beta( Be+1) with
0t++1 = (1 A)a + Aoo + (1 p)8t + p8e,
Bet = (0 1)3, + A80 +1 -P)se ps..
The proof is placed in Appendix B.T
3.3 ULTRA-LIGHT INTERPOLATION PLUG-IN FOR IMPLICIT EVIDENCE
Given batch of online evaluation results B, = [t] Tim ))== we aim estimate the passing rate Pp¹ for any task To using an estimator p(B,, It). this work, we adopt an ultra-lighweiight interpolation-basee estimator minimize additional computational overhead for online task selec- tion. Notably, the adaptation rules in Sec. 3.2 place no restrictions on the specific form of p(B,, k:).
Assume that for each task T have empirical success rates ph and /h from reference models of distinct capability (weak vs. strong). Define the average empirical success rates of the current, weak, and strong models on B,
Dt (B.)
1 B2
r P (B,)
Σ
Pwr (re((Bi)
KEB, n
B.
1B,
kEB,
&
We estimate the relative capability coefficient of the current model as pe(B.) = (pict (B.) Fesl (Bc)) (17e
- n (B.)), which locates the current model between the weak and strong reference models on the batch B3 (we assume prof > pref: otherwise one may add a small to the denominator). To reduce variance from stochastic rollouts, we maintain a momentum version of the coefficient j4, 7/4-1 + (1 y)p.
Finally, the passing rate of the current model on task T is obtained via linear interpolation between the weak and strong references, followed by clipping to [0, 1]:
p(k, B.) clip (B.) + (1 - (B.)) p., 0, 1).
(5)
3.4 THOMPSON SAMPLING FOR TASK SELECTION
Having established task difficulty posterior Beta(pt , over the success probability of each task Th we now turn to the crucial step of selecting tasks for the next training batch.
The first question is: at what difficulty level does the current model benefit most from training? Prior works (Chen et al. 2025b Sun et al.) 2025) show that, under binary rewards, tasks with suc- cess probability around 0.5 most informative for learning they lead to gradients with larger
4

---

Published as a conference paper at ICLR 2026
expected magnitude than tasks with success probability close to 0 or 1. We define the utility of task as the absolute deviation of its posterior mean from a target success probability p E (0, 1), with p" 0.5 the canonical choice.
Given this target success probability, the problem of online task selection naturally reduces to a non-stationary bandit problem. The central challenge is the tradeoff between exploitation and ploration for task selection: the model must decide whether to select tasks with high-confidence estimates close to the target rate in order to maximize immediate utility, or select tasks with high uncertainty to gather information that may improve future decisions. A purely exploitative strat- egy might select tasks whose posterior mean P is closest to p². but this risks overlooking tasks whose difficulty is currently uncertain but potentially optimal. To naturally balance exploration and exploitation, we employ Thompson Sampling [Tommsoon] 1933), strategy renowned for both its empirical effectiveness and theoretical guarantees. More specifically, at each selection step we perform the following: (1) Posterior Sampling: Draw sample of the passing rate from its current posterior distribution Pk: Beta(a,, ) for each task T in the pool. (2) Selection: Select tasks with the highest estimated utilities {ύg =- - lp - p J to form the training batch B This procedure elegantly prioritizes tasks likely be near the target difficulty, while the inherent variance in sampling from the posterior ensures that tasks with higher uncertainty are naturally explored.
3.5 HYPERPARAMETERS
The hyperparameters À and p play distinct but complementary roles in shaping both the difficulty estimation process and the uncertainty profile that drives Thompson Sampling- based task selection. Their effects can be summarized along two axes:
Impact on difficulty estimation. The parameter À controls the balance between adaptivity and stability in Bayesian belief updating. A larger À places more weight on recent evaluations, enabling the posterior to rapidly adapt to the model's evolving capability. However, this increased adaptivity also makes the estimate more sensitive to noise in the most recent observations, reducing stability. Conversely, smaller À assigns greater weight to accumulated history, reducing variance in the estimated probability of success. This leads to more stable but less responsive difficulty estimates. A toy example illustrating these effects is provided in Figure
The parameter P governs how explicit and implicit evidence are fused. A larger biases the update toward implicit evidence, causing tasks without direct evaluations to inherit stronger difficulty sig- nals inferred from tasks with direct evaluations. A smaller p. in contrast, favors explicit evidence, making the posterior rely primarily on direct observations and thus behave more conservatively when a task has not been recently evaluated.
Impact on selection-time uncertainty. Proposition 2 characterizes how the hyperparameters and p jointly determine the effective sample size controlling the confidence of the estimated probability success.
a + 3g of each task's 's Beta posterior, thereby
Proposition 2. Let n4 as Br. Suppose the updates follow Equation E [0, 1), we have
with X E (0,
lim inf n4 = Tig + fn,
sum = Ylu + F.. 71+00
The proof is given in Appendix B.2 In terms of balancing exploration toward tasks whose current success probability estimates deviate from the target and exploitation of tasks estimated to be near the target success probability, À controls the overall scale of the effective sample size n, a, + B,, which directly dictates the variance of the posterior distribution. A larger À accelerates discounting of older evidence, reducing ni and increasing posterior uncertainty. Under Thompson Sampling, this results in more exploratory behavior, particularly toward tasks whose current success-probabiln timates deviate from the target difficulty. smaller À enlarges ny, decreasing posterior variance and nudging the strategy toward exploitation of tasks estimated to be near the target success probability.
The parameter primarily affects the effective sample sizes of tasks that have nor been directly evaluated at the current step. A larger p increases these tasks' effective sample sizes via stronger pseudo-count updates, reducing their posterior uncertainty and making them less likely to be ex- plored. A smaller p decreases their effective sample sizes, increasing posterior variance and allow-
5

---

Published as a conference paper at ICLR 2026
ing tasks with difficulty estimates farther from the target to be selected, thereby improving long-rum exploration.
3.6 PRACTICAL CONSIDERATIONS
Hyperparameters. We recommend using À = 0.1 and p = 0.1 as the default configuration for BOTS. This setting consistently performs well across all model scales and task domains evaluated in Section 4 For new applications, practitioners may start from this default and subsequently adjust the hyperparameters using the insights provided in Section 3.5
Computational Cost of Interpolation-based Task Difficulty Estimation. The main cost of the interpolation-basee estimator (Section 3.3) comes from evaluating reference models on all tasks. Fortunately, such evaluations are increasingly common in modern RI. datasets- example, the GURU dataset (Cheng et al. 2025) provides Qwen2.5-7B-Insteee
(Yang et al.,, 2024) and Qwen3- 30B-A3B (Yang et al.) 2025) scores as difficulty tags for offline filtering. Our estimator simply reuses these tags for online task selection. In settings where reference-model evaluations are not available, enabling implicit evidence requires one-time rollout cost to compute them. Given the substantial gains that implicit evidence brings, this introduces a natural tradeoff. Unlike oversampling- -aasee online methods, this cost does recur during training and amortizes well when training multiple models or checkpoints. Additional practical benefits this computation are discussed in Appendix F.3 Once the reference scores available, the online over- head of the interpolation based estimator is negligible (see Appendix G.I) since no extra rollouts are required and all updates reduce simple vector operations.
Choice of Reference Models. When dataset provides pre-computed base model evaluation tags, as in GURU, we recommend directly using these tags as reference-model signals. This avoids any additional rollout cost, even though it may introduce extrapolation (e.g., when both reference models are stronger or weaker than the training model). Our empirical results in Appendix G.2 show that while extrapolation computational savings generally worthwhile. When multiple reference-mode pairs are available, suggest choosing models with clear capability gap, which provides stronger discrimination for interpolating task difficulty. For instance, GURU's choice of Dwwen.55-7B-Instruu and Qwen3-30B: A3B yields informative difficulty signals across wide range. Finally, although the interpolation- based estimator supports extrapolation in principle, our results show that avoiding extrapolation produces more accurate implicit evidence and slightly stronger overall BOTS performance.
reduce the accuracy of implicit evidence, BOTS remains robust, making the
EXPERIMENTS
We begin with Section 4.1 which introduces datasets, reinforcement finetuning protocols, evaluation metries, and computational cost. S.2tion and Section 4.3 analyze the impact of hyperparameters (p). (A) on task selection respectively. Section 4.4 compares BOTS with competitive basclines across model scales and domains, while Section 4.5 summarizes additional experiments provided in the Appendix
4.1 SETUPS
We conduct experiments GURU (Cheng et al., 2025). a well-curated cross-domain RL dataset. Each subset is deduplicated, verified, and filtered. We use its math, code, and logic subsets (excluding the Zebra Puzzle due to its non-binary reward). Detailed information about the used datasets is provided Appendix (for training) and Appendix [.2 (for evaluation).
Dataset.
and Qwen2.5-7BB Key hyperparameters include a learning rate of le-6, 16 rollouts per task, and a temperature of 1.0. Comprehensive training details and used RL algorithm are provided in Appendix D33 Dn5
RFT Setting.
We adopt GRPO (Shao et al. 2024),
1...2.5--1.B-Insiii
Evaluation. We report the following metrics to evaluate our framework and its ablations, with formal definitions given in Appendix D.4

---

Published as a conference paper at ICLR 2026
Effective Task Ratio (ETR). It evaluates tasks selection. The fraction of sampled tasks whose empirical success rate, estimated from n = 16 independent rollouts, falls strictly within the (0. 1) range. A higher ETR indicates a more efficient task selection that successfully filters out tasks that are either already mastered (p = 1) or currently unsolvable (p- =0) 0).
Soine-to-Baseliig (TTB). It measures training acceleration relative the random baseline to achieve a specific performance. Let the baseline start from performance Pinit and reach the best performance Phett within the training window. For a target fraction E {50%, 75%, 100%), we define the target performance as P- Pinit + (P P. TTB(T) is the ratio of steps required by method to reach P. compared to the baseline. For example, if the baseline starts 0.1 and reaches 0.3 by step 100, then the 50% target is 0.2. If the baseline first reaches 0.2 at step 40 while another method reaches it at step 30, then (50() = 30/40 0.75. By definition, the baseline has TTB = 1; smaller values indicate greater acceleration.
Best-so-far (BSF). measures the performance gain relative to the random baseline under fixed budget. Within specific ratio (25% 50%, 100%) of total training steps, BSF is the ratio between a method's best-so-far performance and the baseline's 's best-so-far performance. For example, at step 50 (total steps 100), if the baseline's 's best is 0.4 and method's best is 0.6, then BSF(50%) 0.6/0.4 1.5. The baseline always has BSF = 1; larger values indicate greater gains.
Computational Overhead. We note that BOTS introduces negligible train-time additional compu- tation, with overhead measured at < 0.2% total training time (see analysis Section 3.3 and empirical results in Appendix [G.1 1. Thus, in the main results, report TTB and BSF, in terms of training steps rather than wall-clock time, as their difference is practically insignificant.
4.2 ANALYZING THE IMPACT OF p
In Section 3.5
we analyzed the role of the hyperparameter À and its influence BOTS. In this section, we empirically validate these analyses. We investigate the role of p in task selection by varying p {0.0, 0.05, 0.1, 0.2, 0.5, 1.0), while keeping other hyperparameters fixed at their default values (A 0.1, posterior sampling enabled).
Effective Batlo
0.35
3.00
03.15
01.00
0.39
0435
3.000
Ro
Figure 2: Qwwen2.5-1.5B-nntrru on Math with X = 0.1, E (0.0, 0.05, 0.2, 1.0). Ratio of sampled training tasks (measured over 16 rollouts) with passing rates: strictly between and (L1), strictly greater than 0 (L2), and strictly less than 1 (L3), along with aggregated performance (MATH500, AMC22 and AIME24), plotted against training steps. All results are averaged over 3 random seeds, and 95% confidence intervals reported. More detailed results are placed in Appendix G.6
Analysis and Takeaways. Our central finding is that implicit evidence crucial for rapid cold- starts, while explicit evidence is vital for long-term accuracy. A principled fusion of both is key to effective task selection. Relying solely on implicit evidence (p 1) leads to error accumulation, while ignoring it (p 0) suffers from severe data sparsity in carly training, especially on large-scale datasets. This conclusion is supported by the following observations from Figure 2 and Table [
(1) Implicit evidence provides an essential early boost. In the initial training phase, all settings with p > demonstrate a sharp increase in the Effective Task Ratio (ETR) over the random baseline, primarily by filtering out unsolvable tasks (p 0). In contrast, the p 0 setting, which relies solely on sparse explicit feedback, behaves almost identically to random sampling. This highlights the critical role of implicit evidence in overcoming the cold-start problem.
7

---

Published as a conference paper at ICLR 2026
Agonggand
Targrt
p
1.08
0.95
0.99
0.26
2.18
p= 0.0
0.23
135
1.24
0.35
01.22
1.11
1.85
1,12
EL66
0/11
1.20
1.85
0.78
0.88
0.64
1.08
1.09
p- 0.1
0.85
0.65
0.78
133
1.86
0.86
0.68
0.74
1.10
1.16
1.06
0.56
0.85
0.50
1.20
3.25
1.25
0.85
0..6
0.72
1.06
1.32
p= IL2
0.76
8.1
0.16
110
118
1/24
0.87
0.62
0.70
1385
1.29
1.04
1.00
0.71
0,47
1/22
1.63
1.11
0.28
0.63
0.69
1,02
1,10
100
0.5
6.82
048
0.56
100
1.04
1,31
0.78
0.61
114
1.16
1.00
0.30
0.50
0:67
1.66
1.24
1.16
0.79
0.66
0.89
1,09
1.00
1.00
1.06
0.6
0.55
1.4
1.13
1.00
0.96
0.48
1.2
1.11
1.22
0.75
0.89
0.99
1.07
Table I: Qwen2.5-1.5B-Instrre TTB, notation "-" indicates that the target performance is never achieved within the evaluation win- dow. The best and second best results are marked accordingly.
on Math with À = p E {0.0, 0.05, 0.1, 0.2, 1.0). For
(2) Over-reliance on implicit evidence degrades long-term performance. As training progresses, settings with a large p (e.g.. 0.5, 1.0) show a declining ETR. This driven by an increased tendency to select tasks whose success probability is 1, as shown in the curve of < 1 ratio. Consequently, the performance is negatively affected, as seen in the aggregated TTB and BSF metrics where large values underperform.
(3) A small positive achieves the best of both worlds. A E f0.05, 0.1, 0.1, strikes an opti- mal balance. It leverages implicit evidence for initial acceleration while allowing more accurate, accumulating explicit evidence to dominate in the long run. This fusion leads to robust performance gains throughout entire training process, achieving best overall TTB and BSF scores.
4.3 ANALYZING THE IMPACT OF X
In Section 3.5 we analyzed the role of the hyperparameter p and its influence on BOTS. In this section, we empirically validate these analyses. We investigate the role X in task selection by varying À E (0.0, 0.05, 0.1, 0.2, 0.5, 1.0), while keeping other hyperparameters fixed at their default values (p = 0.1, posterior sampling enabled).
Prfective Tasc Ratto (FTR)
NO Ratio
<1 Ratio
Matt Agg Performance
1.00
0.55
0.50
0.15
Figure 3: Qwwe2.55-1.B-IIsttru on Math with À E {0.0, 0.05, 0.1, 0.2, 0.5, 1.0), 0.1. Ratio of sampled training tasks (measured over 16 rollouts) with passing rates: strictly between 0 and (L.1), strictly greater than 0 (I.2), and strictly less than 1 (I.3), along with aggregated performance (MATH500, AMC23, and AIME24), plotted against training steps. All results are averaged over 3 random seeds, and 95% contidence intervals are reported. More detailed results are placed in Appendix G.7
Analysis and Takeaways. The hyperparameter À governs (i) the adaptivity of difficulty estimation under non-stationary learning process, and (ii) the lection. Our key takeaway that both of these aspects are crucial for strong long-term performance. An overly small À prevents the system from adapting to the model's 's improving capability, whereas an overly large À injects excessive randomness into task selection. This conclusion is supported by the following observations from Figure 3 and Table 2
balance during task se-
(1) Small X fails to track capability improvement. Settings with small À (e.g., 0.0, 0.05) exhibit declining, especially < Ratio, in the mid-to-late training phase. This is because their preference to historical evaluations makes them slow to update the difficulty of tasks, causing ineffective tasks to be wastefully selected. This ultimately limits late-stage performance gains.
8

---

Published as a conference paper at ICLR 2026
Agonggand
Targrt
1.00
0.35
0:97
0.91
0.60
1.05
1.01
0.37
0.54
1,43
2.87
0.66
1,11
0.35
0.67
8:96
1.55
1.00
1.04
1882
1:11
1.01
1.34
1152
0:31
0.9
1:23
1.25
0.83
0.:00
0.89
1.02
1.00
A-011
0.85
0/66
1.788
135
1.86
0.86
0.68
0.74
8.10
1.16
1.06
0.56
0.85
0.90
1.20
3.25
1.25
0.35
11.6
0.73
1.06
1.32
A 112
0.83
0.76
1.22
146
100
101
0.90
0.59
0.65
1884
1.13
1.66
0.89
0.47
1.21
8.28
1.21
0.3
0.71
0.85
1.05
1.10
114
A-0.
0.32
0,44
0.97
0.57
1.04
1.00
1,81
1,30
0.51
105
0.99
1.06
0.56
0.90
0.62
1,05
1,87
1.21
0.55
0.58
0.75
0.59
1,88
Ln0
0.99
0.9
1.18
1.01
0.99
1.00
0.91
0.78
1.00
0.54
1.05
0.96
0.96
1.00
1.02
0.99
Table 2: Qwen2.55-1.BB-Insttree on Math with À E {0.0, 0.05, 0.1, 0.5, 1.0]. = 0.1. For TTB, notation "-" indicates that the target performance is never achieved within the evaluation win- dow. The best and second best results are marked accordingly.
(2) Large À induces near-random task selection. Conversely, large À produce task selection be- havior that is almost fully random. This arises because large À aggressively discounts historical evidence, yielding very small effective sample sizes and thus high posterior uncertainty, which in turn drives overly exploratory Thompson sampling behavior.
(3) Moderate À achieves the best trade-off. A moderate À E {0.05, 0.1, strikes a balance: the difficulty estimates remain sufficiently adaptive while the uncertainty stays controlled, enabling a healthy -xxraaiioneeploitaation balance. This leads to strong and consistent performance gains across the entire training process, achieving the best TTB and BSF results.
4.4 PERFORMANCE COMPARISON ACROSS MODELS AND DOMAINS
Domain
Math
Code
Logic
Metrie
BSF (T)
TTB
(1)
TTB
Target Friction
50%
75%
100%
25%
50%
1009E
50%
75%%
100%
25%
50%
100%
50%
75%%
100%
25%
50%
100%
Random
1,00
1.00
1.00
1.00
1.00
1.00
1,00
1.00
1.00
1.00
1,00
1,00
1.00
1.00
1.00
1.00
1.00
1.00
Offline
0.22
1.09
1.11
1.04
1.03
1.09
1.36
0.67
BUTS- MoPPS
0.96
1.09
1.04
BOTS-DOTS
0.70
0.70
0.73
1.08
1.08
1.01
0.67
0.65
0.79
1.17
1.09
1.02
0.66
1.32
1.28
0.92
0.93
BOTS
0.85
0.66
0.72
1.06
1.12
1.05
0.58
0.90
0.77
1.17
1.08
1.03
0.85
0.94
1.05
1.19
1.01
1.00
Table 3: BOTS-Qwen2.5.. Inntstrrut Across Domains. The recommended setting outperforms both out-of-framewond and within-framework baselines, achieving 10 first-place and 6 second-place place
finishes out of 18 reported metrics. Full results are in Appendix G.8
G.10
Domain
Math
Code
Logic
Metrie
TTB (J)
BSF
TTB (L)
BSF (1)
'TR (1)
BSF
Target Fraction
50%
75%
100%
25%
50%
100%
50%
75%
100%
25%
50%
100%
50%
75%
100%
25%
50%
100%
Random
1,00
1.00
1.00
1.000
1.001
1.00)
1.00
1.00)
1.00
1.00)
1,00
1,00
1.00)
1.00
1.00
1,00
1.00
1.00
Offline
0.91
0.73
0.76
0.99
1.00
1.05
0.97
1.24
0.99
0.99
0.99
1.06
1.23
0.95
0.96
0.95
1.04
ROPSS
1.07
0.70
0.94
1.04
0.84
0.69
0.84
1.04
0.92
0.69
1.07
1.00
BOUS-DOTS
0.83
0.76
1,02
0.91
1.08
BOTS
0.86
0.77
0.63
0.99
1.04
1.04
0,82
0.79
1.06
1.04
1.01
1,00
0.79
0.77
0.50
1.03
1.01
1.00
Table 4: BOTS-Qwen2.5-78 Across Domains. of ff-frmeeoork and within-framework baselines, achieving 6 first-place and 8 second-place finishes out of 18 reported metrics. Full results are in Appendix G.8 G.10
The recommended setting outperforms both out-
We conduct extended experiments across model sizes (1.5B and 7B) and task domains (math, code, logic). We compare BOTS under the default setting (A = 0.1, p 0.1, posterior sampling enabled) against four baselines. Two are col-of-framewort baselines: (i) Random, where tasks are uniformly sampled, and (ii) Offline, where tasks are ranked from easy to hard based on success probabilities of Qwen2.5-77 2ree2.5-7BBInsstrucct Instruct and Qwen3-30B-A2
(for tie- ne-bbraking). Two are within-framewee lines: (iii) BOTS-MoPPS. with À 0.0, p 0.0 and posterior sampling enabled, which reduces our framework to MoPPS (Qu et al. 2025) and thus enables direct evaluation of a purely explicit- evidence strategy; and (iv) BOTS-DOTS, with X= 1.0, p 1.0 and posterior sampling disabled,
base-
9

---

Published as a conference paper at ICLR 2026
serving as a proxy inspired by DOTS (Sun et al. of relying almost entirely on implicit evidence without corrective feedback. Implementation details and further discussion of these baselines are provided in Appendix[Da.
2025 which evaluates the long-term effectiveness
Analysis and Takeaways. The default BOTS setting consistently outperforms both out-of- framework and within-framework baselines, validating the principle of fusing explicit and im- plicit evidence. Moreover, BOTS-DOTS emerges as the strongest baseline, confirming that our interpolation-basee implicit evidence provides useful guidance for task selection. These conclu- sions are supported by the following observations from Table and Table
(1) BOTS achieves the best overall performance. For the 1.5B model, BOTS obtains 10 first-place and 6 second-place fimishes out of 18 reported metrics. with notable 28% acceleration (TTB 0.72) the math domain. For the 7B model, BOTS secures 6 first-place and second-plas finishes, including remarkable 50% acceleration (TTB(100%) = 0.50) in the logic domain.
(2) BOTS-DOTS ranks second. For the 1.5B model, second-place finishes. For the 7B model, it achieves 5 first-place and 1 second-place finishes, out- performing the remaining baselines.
achieves 6 first-place and
In summary, these results demonstrate that BOTS not only delivers consistent gains across different domains and model scales, but also achieves substantial acceleration in training efficiency. The su- periority of BOTS over both BOTS- and BOTS-DOTS highlights the necessity of combining explicit and implicit evidence, while the strength of BOTS-DOTS underscores the practical value of our interpolation-basee implicit evidence. Together, these findings establish BOTS as a principled, effective, and scalable solution for online task selection in LLM RFT.
4.5 OVERVIEW OF ADDITIONAL EXPERIMENTS
We provide extended experiments in Appendix G for deeper understanding of BOTS: (1) A wall- clock breakdown (Appendix G.1) shows task selection adds less than 0.2% overhead. (2) Evaluation of the interpolation-baat based implitict evidence with various combination of reference models (Ap- pendix G.2) confirms its effectiveness and robustness, and provides insights for practice. (3) An ablation on Thompson sampling (Appendix G.3) shows posterior sampling yields more stable se- lection. (4) A fine-grained analysis of selected task dynamics (Appendix G.4) illustrates how BOTS shifts computation away from trivial (p = 1) and impossible (p- = 0) tasks toward the informa- tive mid-difficulty region. (5) Comparison against an external baseline Dynamic Sampling Yu et al. (222)) (Appendix G.5) further demonstrates the superiority of BOTS. (6) The extended experimen- tal results in Appendix G.6.G..00) 1. provide additional details for the experiments in this section.
CONCLUSION AND DISCUSSION
We introduced BOTS, unified Bayesian framework for online task selection in LLM reinforce- ment finetuning. BOTS formulates task difficulty estimation as Bayesian belief updating, fusing two complementary evidence sources: stable but sparse explicit evidence from direct rollouts and dense but less precise implicit evidence from inter-task relationships. Instantiated with a lightweight interpolator and Thompson sampling, BOTS achieves adaptive and efficient online data selection with negligible overhead. Experiments show consistent gains in data efficiency and model perfor- mance across domains and model scales, surpassing methods that rely on a single evidence source. We envision BOTS as a practical foundation for dynamic, model-aware data selection, advancing efficient and effective LLM training.
This work opens several promising directions. First, we focused mainly on binary-reward tasks; extending and validating BOTS in non-binary reward settings is an important next step. Second, BOTS currently uses fixed update rules, though results show that different values of X and p work best at different training stages. Developing adaptive update strategies that adjust to training dynamics would further improve robustness. Third, while lightweight interpolator efficiently provides implicit evidence, designing stronger plug-in alternatives and systematically studying the trade-off between predictive accuracy and computational cost remain open challenges. A more detailed discussion of these directions is provided in Appendix E.4
10

---

Published as a conference paper at ICLR 2026
Ethics Statement All authors have read and adhered to the ICLR 2026 Code of Ethics. Our search focuses on the algorithmic efficiency of reinforcement finetuning for Large Language Models and does not involve human subjects, animal experiments, the processing of personally identifi- able information. The datasets used in our experiments are publicly available and established bench- marks within the research community; all software, datasets, and frameworks utilized governed by the permissive Apache-2.0 open-source license. Our method aims to make Al research more sustainable and accessible, and do not foresee any direct negative societal impacts or ethical concerns arising from our proposed methodology. The authors declare no conflict of interest.
Reproducibility Statement We are committed to ensuring the full reproducibilityy of our research. To facilitate this, we will release source code, which includes the implementation of the BOTS framework and the scripts required to replicate all experiments presented in this paper. The appendix contains comprehensive description of our experimental setup, detailing all model configurations, dataset processing steps, and hyperparameter settings.

---

Published as a conference paper at ICLR 2026
REFERENCES
Alon Albalak, Duy Phung. Nathan Lile, Rafael Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, and Nick Haber. Big-math: large-scale, high-quality math dataset for reinforcement learning in language models. CoRR, abs/2502.17387, 2025.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang. Carrie Cai, Michael Terry. Quoe Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108. 07732, 2021.
Sanghwan Bae, Jiwoo Hong, Min Young Lee, Hanbyul Kim, JeongYeon Nam, and Donghyun Kwak. Online difficulty filtering for reasoning oriented reinforcement learning. arXiv preprint 25X::22004.033800 2025.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
Minghan Chen, Guikun Chen, Wenguan Wang. and Yi Yang. SEED-GRPO: semantic entropy en- hanced GRPO for uncertainty-awant policy optimization. CoRR, abs/2505. 12346, 2025a.
Xiaoyin Chen, Jiarui Lu, Minsu Kim, Dinghuai Zhang, Jian Tang, Alexandre Piché, Nicolas Gontier, Yoshua Bengio, and Ehsan Kamalloo. Self-evolving curriculum for Ilm reasoning, arXiv preprint 14X1: 2025b.
Zhoujum Cheng, Shibo Hao, Tianyang Liu, Fan Zhou, Yutao Xie, Feng Yao, Yuexin Bian, Yonghao Zhuang, Nilabjo Dey, Yuheng Zha, et al. Revisiting reinforcement learning for Ulm reasoning from cross-domain perspective. arXiv preprint arXiv:2506. /4965, 2025.
Francois Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. Arc prize 2024: Technical report. arXiv preprint arXiv:24 /2. 12.04404, 04604,
2024.
Francois Chollet, Mike Knoop, Gregory Kamradt, Bryan Landers, and Henry Pinkard. Arc-agi-2: A new challenge for frontier ai reasoning systems. arXiv preprint 11X1, 2025.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng. Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. The entropy mechanism of reinforcement learning for reasoning language models. CoRR, 0as/25555226111 2025.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO: model alignment as prospect theoretic optimization. CoRR, abs/2402.01306 2024.
Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. In The Thirteenth International Conference on Learning Representations, 2025.
Daya Guo, Dejian Yang. Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501. /2948. 2025.
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, et al. Skywork open reasoner technical report. arXiv preprint arXiv:2505.22311 2025a.
Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al. Deepmath-1033 103k: A large-scale, challenging, de- contaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504. / 1456, 2025b.
12

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021.
[ttttsss:///axii org/ wws.///..... 03874
Jiwoo Hong, Noah Lee, and James Thome. ORPO: monolithie preference optimization without reference model. In EMNLP, pp. 11170-11189 Association for Computational Linguistics, 2024.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zerr:: An open source approach to scaling up reinforcement learning the base model. arXiv preprint arXiv:2503.24299, 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.1672 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.0797. 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems princi- ples, pp. 611-626, 2023.
Kaixin Li. Verified taco problems. vertiverifie 2024.
Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. Superfiltering: Weak-to-strong data filtering for fast iestruction-tunnng ACL (1), pp. 14255-14273, Association for Computational Linguistics, 2024a.
Shicheng Li, Lei Li, Kun Ouyang. Shuhuai Ren, Yuanxin Liu, Yuanxing Zhang, Fuzheng Zhang, Lingpeng Kong, Liu, and Xu Sun. Temple: Temporal preference learning video Ilms difficulty scheduling and pre-sft alignment. arXiv preprint arXiv:2503. 16929, 2025a.
Wen-Ding Li, Keya Hu, Carter Larsen, Yuqing Wu, Simon Alford, Caleb Woo, Spencer M Dunn, Hao Tang, Michelangelo Naim, Dat Nguyen, al. Combining induction and transduction for abstract reasoning. arXiv preprint arXiv: 241 1244/1.0222.. 2024b.
Yu Li, Zhuoshi Pan, Honglin Lin, Mengyuan Sun, Conghui He, and Lijun Wu. Can one domain help others? a data-centric study on multi-domain reasoning via reinforcement learning. arXiv preprint arXiv:2507. 175112 2025b.
Zhenqing Ling, Daoyuan Chen, Liuyi Yao, Qianli Shen, Yaliang Li, and Ying Shen. Diversity as reward: Fine-tuning LLMs on a mixture of undeernnndee data. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for align- ment? A comprehensive study of automatic data selection in instruction tuning. ICLR. Open- Review.net, 2024.
Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. #instag: Instruction tagging for analyzing supervised fine-tuning of large language models. In ICLR. OpenReview.net, 2024.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, et al. Deepscaler: Surpassing ol -repiev with a 1.5 model by scaling rl. Notion Blog, 2025.
MAA.
American
invitational
mathematics examination
aime.
In
American Invitational 2024.
Mathematics Examination
AIME 2024,
Febru-
ary 2024.
URL pttps://mma..o math nt-competiiionss..
american Dalccnnnnviaatinnaa
Accessed: 2025-
09-09.
13

---

Published as a conference paper at ICLR 2026
Justus Mattem, Sami Jaghouar, Manveer Basra, Jannik Straube, Matthew Di Ferrante, Felix Gabriel, Jack Min Ong, Vincent Weisser, and Johannes Hagemann. Synthetic-1: Two mil- lion collaboratively generated reasoning traces from deepseek-rl, 2025. URL https:/tt///tt
iemeintelleen aa/bli/bbll og/ wttp:a//.......tteettii...e setytthetic---eee lasse
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. In NeurlPS, 2024.
OpenAl. GPT-4 technical report. CoRR, abs/2303.08774. 2023.
Xuchen Pan, Yanxi Chen, Yushuo Chen, Yuchang Sun, Daoyuan Chen, Wenhao Zhang, Yuexiang Xie, Yilun Huang, Yilei Zhang, Dawei Gao, Yaliang Li, Bolin Ding, and Jingren Zhou, Trinity- rft: pereral-purpose and unified framework for reinforcement fine-tuning of large language models. CoRR, 0bs/2505517826 2025.
Shubham Parashar, Shurui Gui, Xiner Li, Hongyi Ling, Sushil Vemuri, Blake Olson, Eric Li, Yu Zhang, James Caverlee, Dileep Kalathil, et al. Curriculum reinforcement learning from easy to hard tasks improves Ilm reasoning. arXiv preprint arXiv: 20X.:25506.0662,, 2025.
Benjamin Pikus, Pratyush Ranjan Tiwari, and Burton Ye. Hard examples are all you need: Maxi- mizing GRPO post-training under annotation budgets. CoRR, abs/2508. 14094, 2025.
Yun Qu, Qi Wang, Yixiu Mao, Vincent Tao Hu, Bjöm Ommer, and Xiangyang Ji. Can prompt difficulty be online predicted for accelerating rl finetuning of reasoning models? arXiv preprint arXiv:2507.04632. 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In NeurlPS, 2023.
Vishnu Raj and Sheetal Kalyani. Taming non-stationary bandits: A bayesian approach. arXiv preprint 1tXiv:1707.09720 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, 0ax/1707.06344 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.003300, 2024.
Wei Shen, Jiangbo Pei, Yi Peng. Xuchen Song, Yang Liu, Jian Peng. Haofeng Sun, Yunzhuo Hao, Peiyu Wang. Jianhao Zhang, et al. Skywork-rIv3 technical report. arXiv preprint arXiv:2507.06167 2025.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rihf framework. In Proceedings of the Twentieth European Conference Computer Systems, pp. 1279-1297, 2025.
Yifan Sun, Jingyan Shen, Yibin Wang, Tianyu Chen, Zhendong Wang, Mingyuan Zhou, and Huan Zhang. Improving data efficiency for Ilm reinforcement fine-tuning through difficulty- targeted online data selection and rollout replay. arXiv preprint arXivv2506.05316, 2025.
William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):255-29,, 1933.
Maryam Toloubidokhti, Yubo Ye, Ryan Missel, Xiajun Jiang, Nilesh Kumar, Ruby Shrestha, and Linwei Wang. Dats: Difficulty-wware task sampler for meta-learning physics-informed neural networks. In The Twelfth International Conference on Learning Representations, 2023.
QC Wang. Xiao, Y Mao, Qu, Shen, Y Lv, and Ji. Model predictive task sampling for efficient and robust adaptation, 2025b. URL https:/arxii. org/abs/2501. 1039, 2025a.
14

---

Published as a conference paper at ICLR 2026
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang. Zhenru Zhang, Yuqiong Liu, An Yang. Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoning. CoRR, abs/2506.01932/ 2025b.
Zhenting Wang, Guofeng Cui, Yu-Jhe Li, Kun Wan, and Wentian Zhao.
Dump: Auto- arXiv preprint
mated distribution-levar curriculum learning for rl-based Ilm post-training. arXiv:2504.09710, 2025c.
Cheng Wen, Tingwei Guo, Shuaijiang Zhao, Wei Zou, and Xiangang Li SARI Structured audio reasoning via curriculum-guidee reinforcement learning. arXiv preprint arXiv:2504. 15900, 2025.
Yunhui Xia, Wei Shen, Yan Wang, Jason Klein Liu, Huifeng Sun, Siyue Wu, Jian Hu, and Xiaolong Xu. Leetcodedataset: A temporal dataset for robust evaluation and efficient training of code llms. arXiv preprint arXiv:2504. 14655, 2025.
Zhe Xu, Daoyuan Chen, Zhenqing Ling, Yaliang Li, and Ying Shen. MindGYM: What matters in question synthesis for thinking-centen fine-tuning? In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025.
Zhongwen Xu and Zihan Ding. Single-stream policy optimization, 2025. URL httpt:///axxii
argg/bbs2500 2509. 13232
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. 2025.
Qwen An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang. Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yi-Chao Zhang, Yunyang Wan, Yuqi Liu, Zeyu Cui, Zhenru Zhang. Zihan Qiu, Shanghaoran Quan, and Zekun Wang. Qwen2.5 technical report. ArXiv, abs/2412.1511s 2024. URL hRLhstts:://pp semanticscholas wwt.aaarrrr///oooouaaaauuu 22: 274859421
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. LIMO: less is more for reasoning. CoRR, abs/2502.03387, 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan. Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint 1xXX::2550.14470 2025.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- 700: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint ar Xiv::25033 18892, 2025.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy op- timization. CoRR, abs/2507. 18071, 2025.
Chenglin Zhu, Tao Zhang, Chong Li, Mingan Lin, Zenan Zhou, and Jian Xie. Eduflow: Advanc- ing milms' problem-solee solving proficiency through multi-stage, culti-perspective critique. arXiv preprint arXiv:2507.09374 2025.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
Notation Summary
17
Proofs
18
B.1 Proof for Proposition
18
B.2 Proof for Proposition
18
Pseudo Codes
19
D Implementational Details
20
D.I Training Data
20
D.2
Evaluation Benchmarks
20
D.3
Trainining Details
20
D..
Formal Definitions of Metries
................................
21
D.5
Policy Optimization Algorithm: GRPO
22
D.6
Baseline Details
22
More Related Works
23
Extended Discussion
24
A toy example illustrating the role of À in difficulty estimation
24
Attention-Based Adaptive Difficulty Estimate
24
F3
Interpolation-Basee Implicit Evidence
24
Discussion on Future Works
26
F4.1
Generalization to Other Reward Distributions
26
F44.2
Self-Adaptive Update Rules
27
F4.3
Alternative Plug-In for Implicit Evidence
27
Additional Empirical Results
28
G.I
Computational Overhead
28
G.2
Interpolation-Basee Implicit Evidence: Empirical Results
................................
28
G.3
Sampling from Posterior
30
G.4
Dynamics of Selected Tasks
................................
G.5
External Baseline: Dynamic Sampling
33
G.6
Extended Experimental Results: of
34 ................................
G. 7
Additional Experimental Results: Impact of X
35
G.8
Extended Experimental Results: Math
36
G.9
.9
Extended Experimental Results: Code
38
G. 10
Extended Experimental Results: Logic
40
Usage of Large Language Models
41
16

---

The target success rate for optimal learning (e.g., 0.5). Published as a conference paper at ICLR 2026
A
NOTATION SUMMARY
For ease of reading and reference, we present the symbols used in this paper Table 5
Symbol
Description
General & Problem Setup
Te, T
The k-th task, or generic task.
Q,O
A query (input) and outcome (output) of a task.
R(-)
A binary reward function, mapping an outcome to {0, 1].
Mg
The language model parameterized by 0. Success probability of model MMg on task T.
Pe.T
t
Index for the training step iteration.
pt
Shorthand for Po,,T success probability task k at step f. Shorthand for the reward distribution (Bernoulli) of task at step t.
R
N
Total number of tasks in the pool.
n
Number of rollouts per task in a batch.
B.
The batch of tasks selected for training step
Bayesian Estimation Framework
ar. .8k
Parameters of the Beta posterior for task it at step f. Parameters of the base prior Beta distribution for task k.
ag.Pi 96
n
Equivalent sample size for task & at step f, n a + 3/.
Equivalent sample size of the prior for task k, n = og + 38
A
Discounting factor in [0, 1] for historical information.
Balance coefficient in [0. 1] for explicit vs. implicit evidence. The i-th binary reward obtained for task k.
Explicit success and failure counts for task at step f. Pseudo success and failure counts from implicit evidence.
Interpolation-based Implicit Evidence
Phn ph pt pret or
Pre-computed success rates of a weak and strong reference model on task Avg. success rates of current, weak, and strong models reference batch. Relative capability coefficient of the current model at step l.
Pt.
Momentum-updee updated relative capability coefficient.
Momentum coefficient for updating Pt.
p(k, B.)
Estimated success probability for task k using implicit evidence.
Task Selection (Thompson Sampling)
p²
Pk
Success rate sampled from the posterior Beta(af pt) Utility of a task, computed as Ipk p=].
Evaluation Metrics
Target fraction (e.g., 0.5, 0.75) for defining performance milestone. Initial and best performance of the bascline method.
Pinit: Pbest
P.
Target performance, Pimi T (Pbea P.n)).
TM
Training step (hitting time) for method . to first reach performance P,.
8
Budget ratio (e.g., 0.25, 0.5) for evaluating Best-so-far (BSF).
Generalizations & Alternatives (Appendix)
K(..))
Kernel function between two tasks. Temperature parameter for the kernel function.
7
Natural parameter of an exponential family.
A(n)
Sufficient statistic and log log-partition function of an exponential family. Hyperparameters of the conjugate prior for an exponential family.
X.V
Table 5: Symbol Notation
17

---

Published as a conference paper at ICLR 2026
B PROOFS
B.1 PROOF FOR PROPOSITION
Proposition Let p (0,1) be the Bernoulli success probability at time t. Suppose the current belief is Te(p)= = Beta(p | an, Bc), and let To(p) = Beta(p ou, Po) be a base prior. Given counts
(8t, Ji) and pseudo counts (5t, St) with Jt, St, Se 2 0. define the generalized-Bute Bayees update
Tp)(())
Tp(p) To(D)*
(1-) 1-p x (1'
(4)
prior mixing discounting
lempered explicit likelthood
tempered implicir evidence
with À 1) and p E 1]. Then M241 is exactly Beta(act1, with
a++1 (1 -)aa Aαo + (1 p))sppps-0 + pst.
Be+1 = (1 A)3k A80 + (1 p) + pfc.
Proof. Write the Beta densities (up to normalization) (p) p1 p) and (P)
pou-1(1 p) Raising these to powers and multiplying by the (tempered) likelihood terms in equation yields
(p) & - (a ()) (1 pi (1 -
Collecting exponents shows that T++11 is Beta with parameters exactly as stated (add +1 back to exponents), completing the proof.
B.2 PROOF FOR PROPOSITION
Proposition 2. Let n, Or, Br. Suppose the updates follow Equation 1 with À E (0, P E [0, we huve
lim inf nu no Sn,
lim Simsupre The n1 I.. t-+00
Proof. We drop the task index for clarity. From Equation and Equation 3}.
M++1 (1- X)me Ano
(> + fi) p(3; + f() (1 A)m, + Ano + (1 p)n I[E + pn,
where &, is the event that the task receives direct evaluation at time (so s; + fr = n iff & holds;
otherwise 8t + ft = 0). Unrolling the linear recurrence for 2
T4 = (1 --) + (1 - ( + pm + (1 - ([
[-1 (1 Σ
=np + A
pm (1 (1 - 2) + (1 p)n
A) 118
==0
where we used Σ Et11( A)
2)a Because [[E.] E {0, 1}.
t-1
+-1 X) 1[ $ (1-
(1 A)*
0
< Σ
A
w
0
0=0
Multiplying by (1 - p)n and adding the common term 7o + pn (1 - (1 - X) gives the uniform bounds
x
no + - (1 (1- - (1-(1- < n, 1o N (1 -1 (1- 1)
Since (1 A) + 0 exponentially, the lim inf lim sup statements follow immediately.
Moreover, these bounds are tight: [[,,] = (never directly evaluated), then n, no + en (1 (1 ) and lim n4 = no + pn if [[E = 1 (always directly evaluated), then n4 = no + x
[1 - (1 >)r) and lime = no + F.
18

---

Published as a conference paper at ICLR 2026
C
PSEUDO CODES
Algorithm BOTS: Bayesian Online Task Selection
Require: Task set {T I==1> task batch size B, target success probability p', implicit evidence
constructor ( :), LLM parameter 0., RFT trainer RFT(;;))
Ensure: Trained model parameters 0
1: Initialize Cto. Sb.
If not specified, axo = 1. 3o
2: for 0 T do
3:
if T'homsson Sampling then Sample it Beta(a B+), =1, L....,N
4:
5: else
6: 11
-5 - LI,...,
7:
end
8:
Select batch of tasks (TB. K with top B utilities {úx := - 1px - p*1}.
9:
RFT with the selected tasks: 0e+ RFT(O.. (TB, 11)))))
10:
Collect the online evaluation results of the selected tasks {rim Ji=1'
8.1d
Compute explicit evidence f Jmm with {ri:n [
11:
1 J==1-
>> Eq.
12:
Compute implicit evidence {(5 f())
k= with p, frim Ji=1
Eq.
13:
Update posterior a4-1, Br+1, with X,
> Eq.
14:
end for
15:
return 0T
Algorithm Interpolation based Implicit Evidence Estimator
Require: Task index k, task batch {TB() b online evaluation results 18 efficient y.
J==1,
momentum co-
B Ensure: Estimated success rate p(k, T M==1) for TE.
1: Retrieve success rates of weak/strong reference models {pe Bli [ ) & Pre-computed. , J
2: Compute the average empirical success rates of the current, weak, and strong models B,
B
pref
B[
prof (B)
1
(B)(B)
1
Σ
BM
B
72
B
B
3: Estimate the relative capability coefficient of the current model as
MBB) (77e2 (B) p (B)) / (F. (B) Fre((()) (B))
4: Momentum estimation: ji 70
(B).
5: Retirve success rates of weak/strong reference models on Ts: (pt. p!)).
6: Linear interpolation with clipping: i(k, (TB[), rim 880 B
1) clip u + (1 (D) ph, 0, 1).
B86 7: return p(k, (TB(g) 1nm )
19

---

Published as a conference paper at ICLR 2026
D IMPLEMENTATIONAL DETAILS
D.1 TRAINING DATA
The training data for our experiments is sourced from GURU (Cheng et al. 2025). a well-curated, cross-domain RL dataset covering mathematics, code, logic, science, Simulation, and tabular tasks. Each subset has been rigorously deduplicated, verified, and filtered. Our training utilizes the math, code, and logic subsets, with the Zebra Puzzle excluded from the logic portion due its non-binary reward structure.
Mathematics (54.4k) The Mathematics subset comprises data from Skywork ORI (He et al. 2025a). DAPO (Yu et al. 2025), and DeepScaler (Luo et al. 2025). For all math problems, models are instructed to provide the final answer within [boxed environment.
Code (18.1k) The Code subset includes programming challenges from LeetCode (Xia et al.) 2025) TACO-verified 2024), PrimeIntellect (Mattern al., 2025), and LiveCodeBench (Jain)
al. 2024). For PrimeIntellect and LiveCodeBench, incorporates pre-filtered versions provided
by DeepCode
Logic (5.0k) The Logic subset is composed of several benchmarks designed to test structured rea- soning. It includes Ordering Puzzles (relational ordering). Graph Puzzles (implicit graph traver- sal), the public training splits of ARC-AGI (Chollet et al.) 2024) and ARC-AGI-2 (Chollet et al.) 2023) (abstract grid transformations), and a sampee from BARC (Li et al. 2024b) (synthetic ARC-style tasks). For these tasks, predictions are extracted from <answer> tags for reward calcu- lation via exact match.
D.2 EVALUATION BENCHMARKS
Followed by GURU (Cheng et al. 2025), the evaluation suite consists of a set of established bench-
marks rigorously assess the model's 's performance in a zero-shot setting across the same domains.
Math We evaluate mathematical reasoning on AIME24 (MAA 2024) and MATH500 (Hendrycks) et al. 202T), which cover wide range of competition-leve math problems.
Code Programming capabilities are assessed using HumanEval (Chen et al. 2021). MBPP (Austin
al. 2)21), and subset of LiveCodeBench (Jain al. 2024). These benchmarks span from basic function generation to complex algorithmic challenges.
Logic Logical and abstract reasoning measured using Ordering Puzzles for general reasoning, and ARC-AGI (Chollet et al. 2024) for grid-based abstract reasoning.
D.3 TRAININING DETAILS
The RFT setup follows standard practices (Cheng et al. 2025 Qu et al., 2025: Sun et al. 2025
employing GRPO (Shao al. 2024) as the RL algorithm (detailed in Appendix D.5 All experi- ments are conducted on verl framework (Sheng et al. 2025) with 8 NVIDIA A 100 (80GB) GPUs. We utilize PyTorch's 's FSDP for distributed training, with vLLM (Kwon al., 2023) employed to accelerate the response generation during the rollout phase. The actor model is optimized using learning rate of x 10-6 and weight decay of 0. We apply constant learning rate warmup for the first 10 steps and use gradient clipping with a maximum norm of 1.0. For the GRPO algorithm, we set the clipping ratio e to 0.2 and generate 16 rollouts with a temperature of 1.0. The maximum prompt and response lengths are set 4,096 and 8,192 tokens, respectively.
The training configurations are tailored for two LLMs: Qwen2..5-1.BB-Insttue and Qwen2.5- 7B Yang et al., 2024), across three reasoning domains: Math, Code, and Logic. The primary differences across settings lie in batch sizes and memory optimization strategies to accommodate different model sizes. Specifically, the 7B model experiments utilize larger training batch size (512wl vs. 256) and GRPO mini-hatch size (64 vs. 32). To the larger model on the same hardware,
httpp::/ http:://ww..coghhrrr//////o/oo
20

---

Published as a conference paper at ICLR 2026
enable FSDP's CPU offloading for all 7B model experiments. Key hyperparameters for all six experimental settings are summarized in 6
Hyperparameter
Lath-Math Math
7B-Math
1.5B-Code
7B-Code
L.5B-Logic
7B-Lagic
Base Model
Qwen2.5-1 Swon2.5-1.9B-Inst 1.Sss.
Qwen2,5-7B
Swea2.5-1.5B-Immr
Qwen2.5-7B
Qwen2.5-1 5--5-...s.
Qwen2.5-7B
Optimizer
Adam
AdamW
Adam
Adamm
Adam
Adam
Learning Rate
10=5
x 10-
10-6
x 10
10-1
x 10
Weight Decuy
D.1
0.1
0.1
0.1
0.1
0.1
Global Steps
100
100
70
70
100
100
Batch Size (Prompts)
256
512
256
512
256
512
GRPO Mini batch Size
32
32
64
64
Rollouts per Prompt (r)
16
16
16
16
16
GRPO Clip Ratio (f)
0.2
0.2
0.2
0.2
0.2
Rolloat Temperature
1.0
1.0
1.0
1.0
1,0
1.0
Max Prompt Length
4,096
4,096
4,096
4,096
4,096
4,096
Max Response Length
8,192 Disabled
8.192 Enabled
8,192 Disabled
8,192 Enabled
8,192 Disabled
8,192
FSDP CPU Offload
Enabled
Table 6: Key hyperparameters for RFT across different models and domains. "Inst." is an abbrevia- tion for Instruct.
D.4 FORMAL DEFINITIONS OF METRICS
We provide formal definitions of the key metrics used in our experiments:
(TTB) and Best-so-far (BSF).
Hime-to-Baseling (TTB). We define Time-to -aa-elli (TTB) as metric measure the acceler- ation of a method relative to the random baseline. Let the random baseline start from performance Pinit and reach its best performance Phest within a fixed training window K steps. For a target frac- tion (50%, 75%, 100%), the target performance is defined as the corresponding interpolation between the initial and best performance:
P. Pimi T. (Prest Pim)-
Denote by TM the (possibly interpolated) training step at which method M first achieves perfor- mance Pr. Then the TTB of method M
TTBM(T)
Thaseline
By definition, TTBoseline 1. Smaller values of TTB indicate that method reaches the target improvement faster, and thus achieves greater acceleration relative to the baseline.
Example. Suppose the baseline starts at performance Pinit 0.1 and reaches Phess = 0.3 within 100
training steps. The total improvement is therefore 0.2. For 50%, the target performance is
P00% = 0.1 + 0.5 x (0.3 0.1) = 0.2.
If the baseline first reaches 0.2 step 40, then Thaseline earlier, at step 30, then Tar = 30, and thus
40. another method M reaches 0.2
30
TTB (50%)
0.75.
40
This indicates that method M achieves the relative improvement (50% of the baseline's maxi- mum gain) using only 75% of the training steps required by the baseline, reflecting 25% accelera- tion.
Best-so-far (BSF). We define Best-so-far (BSF) to measure the relative performance gain of method against the random baseline under the same training budget. Let the total training window be T steps. At budget ratio B (0, 1]. corresponding to step t = [@T]. denote by bost (t) the best performance achieved by method M up to step f, and by bestrana(t) the best performance achieved by the random baseline up the same step. Then the BSF of method M is defined as
BSF (MM) = best (t)
best (t)
21

---

Published as a conference paper at ICLR 2026
By definition, BSFF(rand) = 1. Larger values indicate that the method has achieved stronger absolute performance under the same budget, i.c., it delivers better best-so-far outcomes than the baseline, not just relative improvement.
Example. Suppose the total training window is T 100 steps, and we consider = 0.5 (t = 50). the random baseline achieves bostrand (50) = 0.4, while another method M achieves best (50) 0.6, then
0.6
BSF 50% (M)
0.4 = 1.5.
This means that by step 50, method M has achieved best-so-far performance 1.5 times that of the random baseline.
D.5 POLICY OPTIMIZATION ALGORITHM: GRPO
The policy is optimized using Group Relative Policy Optimization (GRPO) (Shao et al. 2024), policy gradient method that operates without a learned value function. GRPO, the advantage for given response is computed by normalizing its reward against the statistics of a group of candidate responses sampled for the same prompt.
Specifically, for each prompt z, set of G responses Y = fv...... yG is sampled from the policy Th. After obtaining the reward R(n,)) for each response, the group-relative advantage A(yx) is defined as:
R(yk) by
A(ya)
(6)
QY
where jy and oy represent the mean and standard deviation of the rewards {R(y))...... R(yg)) for the group.
This advantage is then incorporated into a clipped surrogate objective function to update the policy parameters 0:
LGRPO (0) = Eyyrreu [min (r))))(())) clip(r e, ((
(7)
Here, rv())= [e(ua denotes the probability ratio between the current and old policies. The
(p(a))
clip function constrains this ratio within the interval [] E, 1 + e). limiting the magnitude of policy updates during optimization.
D.6 BASELINE DETAILS
We detail the configurations of the baselines and ablations used for comparison.
Random:
Tasks are sampled uniformly at random from the entire training pool. This represents no-curriculum scenario and serves the fundamental baseline for measuring performance gains.
Offline Baseline: Tasks are pre-sorted once from easy to hard based on the success rates of external models Biven2.5-7BIInsreee with Dwven-33B-A33 for bebbekkking..... This baseline represents static curriculum and is used to benchmark our adaptive method against fixed task sequence.
Setting from MoPPS (Qu et al. 2025): This ablation uses 0.0, p = 0.0 with posterior sampling, relying solely on explicit evidence for task selection. Under this configuration, our framework reduces exactly to the setting studied in Qu et al. (2025). enabling a direct evaluation of purely seabiiit-evidence-baeeee strategy. Note that Qu al. (0025) did not study the code or logic domains, nor did they provide principles for determining X (1 in BOTS). We therefore reuse their hyperparameter choice for math (A 1, i.e., > in BOTS) when evaluating other domains. This comparison fair since no hyperparameter tuning performed for either MoPPS BOTS.
Proxy for DOTS (Sun et al. 2025): This ablation uses X = p.0 1.0 with posterior sampling disabled, so task selection is driven entirely by implicit evidence from our estimator. It serves as proxy for the approach in Sun et al. 2025), though our construction of pseudo-counts differs, see Appendix F2 for more details. This baseline allows us to evaluate the long-term efficacy of a strategy that relies almost exclusively implicit evidence without direct corrective feedback. We
22

---

Published as a conference paper at ICLR 2026
note that Sun et al. (2025) additionally explored reusing historical trajectories, which lies beyond the scope of this paper. To ensure clarity, we isolate such tricks from DOTS and focus solely on the online task selection component.
E MORE RELATED WORKS
Reinforcement Finetuning for LLMs Reinforcement Finetuning (RFT) has become pivotal technique for aligning LLMs with human values and enhancing their capabilities in complex rea- soning tasks (OpenAl 2023] Guo et al.] (025 Zeng al., 2025 He et al.) 2025a). Initial RLHF methods using PPO (Schulman et al. (017)) have been complemented by preference optimization techniques like DPO (Rafailov al. 2023) and its variants such as KTO (Ethayarajh et al. 2024),
ORPO (Hong et al.) (024) and SimPO (Meng et al. 2024). More recently. attention has shifted to irrection-sapliing based finetuning, popularized by frameworks like GPRO (Shao al.] 2024) for its effectiveness in verifiable reasoning. This has spurred variants like DAPO (Yu et al.] 2025 and
GSPO (Zheng et al. (025), with further work focusing on algorithmic refinements (Cui al. 2025]
Wang et al. (025b) and reward shaping (Chen et al. 2025a Pan et al. 2025). Despite these algo-
rithmic advances, the data curriculum, that how tasks are selected and presented, remains a critical bottleneck for RFT efficiency.
The Role of Data Curriculum The importance of data difficulty in RFT is increasingly recog- nized, leading the creation of highly challenging benchmarks (Albalak et al. 2025 He et al., 2025b Gao et al. 2025). Beyond dataset creation, data selection and curriculum design has also become a central topic investigation. For Supervised Finetuning (SFT), variety of data selee- tion and curriculum strategies have been widely studied based on quality (Liu et al. 2024
Li et al.,
2024a), diversity (Ling al.) [0255] Lu et al., 2024), or pre-assessed difficulty (Xu et al. 2025 Ye al. 2025). However, such static methods are ill-suited for the dynamic nature of RFT. This has motivated a focus on dynamic curricula that adapt to the model's evolving capabilities, typically
by leveraging the notion of task difficulty (Cheng et al. 2025 Pikus al. 2025
Li al. 2025b
Toloubidokhti et al. 2023 Wang et al. 2025a).
Task Selection Strategies for RFT Existing task selection strategies for RFT vary in their ap- proach to leveraging task difficulty. An early line of work, directly inspired by curriculum learning, employs offfine curricula that schedule tasks along a fixed easy-to-har hard trajectory (Parashar al., 2025 Shen et al., (2025: Zhu et al., 2025 Wen et al. 2025 Li et al. (025a). While simple and intu- itive, these methods are non non-adaptive and cannot respond the model's real-time learning progress. To address this, online selection strategies have emerged. A straightforward approach is sampling- based task filtering, which uses extra rollouts evaluate and discard tasks that are too easy or too hard, thereby incurring significant computational overhead (Yu et al. 2025 Bae et al.) (025). To avoid this cost, recent works attempt to predict task success rates. Some frame task selection as a non-stationary multi-armed bandit (MAB) problem, but often treat tasks independent arms, thus overlooking cross-tase task relationships (Chen et al., [0255]] Qu et al. (025). Others use a small ref- erence set predict the difficulty of other tasks via similarity kernels, but this still requires extra rollouts and discards valuable historical information (Sun et al. 2025 Wang et al. 2025c).
Positioning Our Work. The current landscape reveals a clear need for a task selection method that Our framework, BOTS, designed to fill this gap. introduces a unified Bayesian framework that fully online and adaptive. Critically, BOTS is the first to jointly incorporate explicit evidence from direct evaluations and implicit evidence inferred from related tasks, without requiring any additional model rollouts. BOTS provides principled, low- overhehee,,, and effective solution for dynamic task selection in RFT.
is simultancously adaptive, computationally efficient, and information-compee
23

---

Published as a conference paper at ICLR 2026
F EXTENDED DISCUSSION
F.1 A TOY EXAMPLE ILLUSTRATING THE ROLE OF IN DIFFICULTY ESTIMATION
Toy Example Illustrating the Rale in Difficulty Estimation
Teith
D22
D
11.8
0.7
11.
05-
016
Figure 4: A toy example illustrating the role of À in difficulty estimation. Ground Truth simulates a continuously evolving success probability, while Observations add Gaussian noise to mimic esti- mation errors from explicit/implicit evidence. We plot the posterior means obtained by BOTS 50 update steps for three settings of À {0.1, 0.2, 0.5}. A larger À (e.g., 0.5) reacts quickly to rapid changes in the underlying difficulty but is highly sensitive to noise. smaller À (e.g., 0.1) yields smoother and more stable estimates but lags behind when the true difficulty increases rapidly. The intermediate setting À 0.2 balances the two behaviors, offering better stability than À 0.5 and better adaptivity than = 0.1.
F.2 ATTENTION-
BAASED BASED ADAPTIVE DIFFICULTY ESTIMATE
Given batch of online evaluation results B, = (TBB, (i), 1am J= we need to estimate s and ft for each task T Besides the interplation bbaeee estimate discussed in 3.3 another straightforward idea is to construct a kernel-based estimator:
8, |.
B =
CT,,, Tx)
KTT, Tx) E KC Tr)
St.
f.
(8)
Bi K(T Ta)
1=1
For example, (unn et (2025) proposed to use an attention- kernel structure:
exp (Embd(T), EmbddT'))
K(T, T')
(9)
However, the limitations of such approaches are two-folds: (1) Due to its convex structure, the kernel estimator does not support extrapolation. Specifically, given a batch of tasks with passing rates bounded within [Pmin, Pmax]. the estimated passing rate for any task is also restricted to this range. (2) Its effectiveness depends heavily on the quality of the task embeddings, which often require additional training, may be high-dimensional. and thus introduce negligiiee storage overhead.
F.3 INTERPOLATION-BASID IMPLICIT EVIDENCE
When linear interpolation is exact and why it becomes imperfect practice.
Assumption (Linear Interpolation Assumption). Let p(T), pie (T), and pt (T) denote the suc- cess probability of the current model, the weak reference model, and the strong reference model, respectively. We assume that all tasks share a common interpolation coefficient, i.e. for any pair of tasks T .T2,
- pfof (T') (T!)
p(T)) - - (T2)
(T2) pf (T2)
We denote the (task-invarians interpolation coefficient by n²"
24

---

Published as a conference paper at ICLR 2026
This assumption states that the current model's success probabilities lie the same affine transfor- mation of the weak and strong reference models; hence linear interpolation is perfect estimator of task difficulty.
Proposition 3. Consider a butch of tasks B = {T*
k== with ground-trath success probabilities
(p(T*), P. (T [] For any task 'T & B, define the linear interpolation estimator
J====
((T) p(B) ps (T) + (1 ((BB) P (T).
where
1
E P (T)) K Ek Pmi (T*))
(BB) = R
I
E p (Tk) K
Ex= Pio (Tk) p
K
Then - =(T for all T B and only if Assumplion I hholds.
Proof. (If) Assume Assumption holds. Let 1e¹ be the common interpolation coefficient for all tasks. Then for every T B,
p(T*) = n² p. (Th) (
(1- 4²" "") (T)
Taking averages over B, obtain
1
1 E p(T 4²
1
ps (Th) + (1 v())
ref (T
K
K
Rearranging gives ((B) 1t*
Now for any T & B, Assumption implies
(T)) p"p,"
p. ret (T) (1 1 p (T).
Since (BB) = ,,,, the estimator satisfies
((T) = p(T).
Thus linear interpolation is perfect under the assumption.
(Only If) Suppose ((T) p(T) for all T & B. In particular, choose two arbitrary tasks T
B. By construction of p(-), we have
p = (u p. (T) + (1 (a(B)) pi (T)), i= ==1,2 1,2.
Solving for p(B) in each equation yields
P(T') p/ (Tl) p (T prc (T') M(B)
P(T2) pro (T2) ps (T2) piu (T2)
Thus the interpolation ratio is identical for all tasks, which is precisely Assumption pletes the proof.
I This com-
However, in practice, different tasks do not progress at the same learning pace, and thus Assump- tion [ may not hold. This mismatch naturally introduces estimation error into the linear interpola- bion-based difficulty estimator. Nevertheless, our empirical results show that even with imperfect estimates, the interpolation-based implicit evidence is sufficiently informative to substantially im- prove training efficiency within BOTS. more fine- mnn-grained theoretical analysis of interpolation error under richer and more realistic assumptions is an interesting direction for future work.
Extrapolation Capability. Unlike kernel-based estimators (Sun et al. 2025) (see Appendix F22 for a brief introduction), which are confined to the convex hull of observed tasks, our interpolation- based estimator naturally supports extrapolation. For example, if the reference pair consists of a strong and stronger model, the capability coefficient μ4 (B,) defined in Section 3.3 may fall outside [0, 1]. yielding extrapolated predictions. The clipping step in Equation (5) ensures these estimates remain within the feasible range [0, 1).
25

---

Published as a conference paper at ICLR 2026
Additional Practical Benefits of Evaluating Reference Models We highlight two additional practical benefits of evaluating reference models: (1) as demonstrated in GURU, reference- evaluations can directly support offline difficult--wwar filtering, removing tasks that are too trivial or too hard hefore training even begins; and (2) these evaluations can also assist RI. algorithms that rely on online difficulty estimation, such SPO (Xu & Ding 2025).
Discussion on the Potential Limitations. The simplicity of our interpolation-busee approach, while a key strength, also entails two potential limitations. (i) Expressive power: The linear inter- polation assumes linear relationship between model's global capability and its per-task success rate. While our empirical results suggest this is a powerful approximation (see Appendix G.2 for validation), the true learning dynamies of LLMs may be more complex. (ii) Distributional shift in capability estimation: The capability coefficient 14, is estimated on the selected batch B, which not uniform sample from the task pool. This introduces bias, as the model is likely to per- form better on this adaptively chosen batch than on the entire dataset. Consequently, be might be an overestimate of the model's true global capability. However, we argue this bias not fatal: the primary role of Pr is to track the progression of the model's ss capability, and even a biased estimate can provide a valuable monotonic signal for this purpose, see Section 4.4 for empirical validation. (iii) Potential Failure Mode: If task exhibits identical success probabilities under both reference models, then the interpolation collapses to that shared value, making the estimated difficulty entirely independent of the current model's capability.
F.4 DISCUSSION ON FUTURE WORKS
F.4.1
GENERALIZATION TO OTHER REWARD DISTRIBUTIONS
While our main exposition focuses on binary rewards with Beta- Bernoulli model, the core princi- ples of BOTS extend naturally to any reward distribution within the exponential family that admits a conjugate prior. This generality makes BOTS a versatile blueprint for online task selection algo- rithms.
The General Framework. Let the reward r for a task follow a distribution from a one- ane-parameter exponential family: f(r [1) = exp())()))) ((fr)-AA(p)) A()),, where 17 is the natural parameter and T(r) is the sufficient statistic. The conjugate prior for takes the form p(n v) x exp(xn - -A)))) where (x. v) are hyperparameters. Here, X can be seen as a pseudo-sum of sufficient statistics from prior observations, and as pseudo-count of those observations.
Our generalized Bayesian update rule from Eq. equation 4 can be directly mapped to this setting. The update for the posterior hyperparameters (Xt, 1/)) becomes:
Xx+1 (1 A)x+ + Axo + (1 D)Tesplicit + p implicits
(10)
Vt+1 = (1 A)v, + Ano + (1 - p)M + primpllcitss
(11)
where (xo, vo) are the base prior's parameters. For Nesplicit direct observations (re), the explicit evidence is Texplici = Exeri (r.)). For implicit evidence, we assume a pscudo-observation of size Timplici with an estimated total sufficient statistic Timplica- This structure precisely mirrors our update for the Beta parameters, preserving conjugacy across iterations.
Revisiting the Bernoulli Case. For Bernoulli reward r E {0, 1} with success probability p. the natural parameter is the logit 7 log(p/(1 (1- (1) and the sufficient statistic is T() r. The conjugate (eta(a, (3) prior corresponds to hyperparameters X = a - and v = a + 3 - 2. With n rollouts, Texplicit &t (success counts), Mexplict n. Timplicit 3, and Namplica n. Plugging these into Equation|
and transforming back to (a, 3) parameters precisely recovers our update rule in Equation i This confirms that our proposed update is a specific instance of this general principle.
Example: Gaussian Rewards. Consider continuous reward like a score from a powerful critic model, modeled as R ~ N(,, o²) with known variance o². The conjugate prior for the mean is Gaussian, μ ~ N(pp, o.). In the exponential family form, n n/o² and T(r) prior hyperparameters are Xo = Ho and 1/- = a² 0²/oß- The BOTS update would apply directly to (x), where T Texplict is the sum of observed rewards and T Timlliit is an estimated sum from the
r. The
26

---

Published as a conference paper at ICLR 2026
interpolator. The posterior for μ remains Gaussian, allowing for Thompson sampling by drawing sample of the mean j and selecting tasks whose fi is closest to some target score μr"..
Example: Categorical Rewards. For tasks with K discrete outcomes (e.g. multi-level ratings), the reward is a one-hot vector, and the distribution is categorical. The conjugate prior is the Dirichlet distribution, multivariate generalization of the Beta. BOTS would maintain vector of K Dirichlet parameters (a1.... ax) for each task. The update rules would apply component-wis to each aj based on explicit and implicit counts for that outcome.
This generality significantly broadens the applicability of our framework beyond binary suc- cess/failure tasks. provides principled and extensible blueprint for difficulty-ave online task selection across wide spectrum of RFT problems involving diverse reward structures.
F.4.2 SELF- ADAPTIVE UPDATE RULES
In the main paper, we set the belief update coefficients À and p as fixed hyperparameters. How- ever, our empirical study (Section 4.3 and Section 4.2) shows that different settings benefit different training stages: smaller À accelerates adaptation in early training but may cause instability later, while moderate p effectively leverages implicit evidence early on but can reduce accuracy later stages. A natural extension is to design self-adaptive update rules that automatically adjust À and according to training dynamics example, by monitoring posterior uncertainty, validation per- formance, or the variance of estimated success probabilities. Such adaptive schemes would allow BOTS to dynamically balance exploration and exploitation, potentially improving robustness across diverse tasks and model scales.
F.4.3 ALTERNATIVE PLUG-IN FOR IMPLICIT EVIDENCE
Our interpolation- based estimator provides an extremely lightweight way to generate implicit evi- dence without additional rollouts, but it not the only possible choice. In fact, also comes with an inherent limitation, discussions in Appendix
F.3 More expressive alternatives could be explored,
such as kernel-based predictors (Sun et al. 2025). task-embedding regressors, or small auxiliary models trained jointly with the main model. These alternatives may improve predictive accuracy, especially when the reference models poorly bracket the training model's 's capability. However, they also introduce trade-off: richer implicit evidence often requires higher computational and age costs. Systematically characterizing this trade-off-between accuracy and efficiency in implicit open research question and promising direction for future work.
27

---

Published as a conference paper at ICLR 2026
G ADDITIONAL EMPIRICAL RESULTS
We provide extended experiments in this section for a deeper understanding of BOTS: (1) wall- clock breakdown (Appendix G.I) shows task selection adds less than 0.2% overhead. (2) Evaluation of the interpolation-basee pendix (.2) confirms its effectiveness and robustness, and provides insights for practice. (3) An ablation on Thompson sampling (Appendix (i.3) shows posterior sampling yields more stable se- lection. (4) A fine-grained analysis of selected task dynamics (Appendix G.4) illustrates how BOTS shifts computation away from trivial (p = 1) and impossible (p = 0) tasks toward the informa- tive mid-difficulty region. (5) The extended experimental results in Appendix G.6.G.II provide additional details for the experiments this section.
implitict evidence with various combination of reference models (Ap-
G.10
G.1 COMPUTATIONAL OVERHEAD
We examine the computational overhead introduced by task selection. The breakdown of wall-clock time across training phases is shown in Figure
Qwen2.5-1.58-Instrul
Qwen2.5-7B
task selection (0.2%)
task selection (0.1%)
reward (2.3%)
reward (0.9%)
log prob (13.3%)
log prob (21.8%)
model update (22 5%)
gener alion (34.4%)
generation (61.7%)
model update (42.8%)
Figure 5: Wall-clock time breakdown across training phases for Qwen2.5-1.5B-Insttee (Left) and Qwen2.5-7B (Right) GURU-Math, trained on 8 A100 GPUs. Runtime is averaged over the first 100 training steps. The cost of task selection-ineludingg posterior sampling, index sorting, and distribution parameter updates-is is negligible compared to overall training.
As illustrated, the dominant arises from generation and model updates, which together account for more than 75% of runtime. By contrast, the overhead of task selection-includingg posterior sampling, index sorting, and distribution parameter updates-is negligible (0.2% or less). Impor- tantly, unlike generation and model updates, this cost does not increase with model size. Overall, our Bayesian framework and the chosen practical instantiation remain extremely lightweight, adding almost no extra burden to training.
G.2 INTERPOLATION-BAAT BASED IMPLICIT EVIDENCE: EMPIRICAL. RESULTS
To empirically validate our interpolation-baseee implicit evidence estimator, we assess its predictive quality against the evolving empirical success probabilities of the training model. We examine the trajectory under vanilla training, i.c., uniformly sampling tasks for training. At each step, compare the predictions from our interpolation-base estimator with the ground-truth online task success probabilities.
Evaluation Metrics. Two metrics are used: (i) Pearson Correlation, measuring the linear rela- tionship between estimated and empirical difficulties, and (ii) ROC AUC, evaluating the ability to distinguish effective tasks (success strictly between and 1) from ineffective tasks (success equal to 0 or 1).
Training Models. Evaluations are conducted throughout training for two models of different scales: Owen2.5-1.5B--Inttee and Qwen2.5-7B.
Reference Models. Reference models are selected among three models: Qwen2.5-1.5B-Insst -nIttruutt (1.5B), Qwen2.5-7BBInnttuul (7B), and Qwen3-30B-A3B (30B). Besides the default choice of ref-
28

---

Published as a conference paper at ICLR 2026
erence models (7B x 30B) used in Section 4 1.5B x 7B and 1.5B x 30B.
additionally include two reference- model pairs:
The resulting Pearson Correlation and ROC AUC curves are shown in Figure 6
Qwen2.5-1 Swenn2.511.B-innsmnuct 5.55--nsttuct, reference models: 78, 308
Qwen2.5-78. reference models: 78, 308
0.2
Porrson Correar
ROC AIC
0.4
0.3
00
35
Steps
Steps
Qwen2.55 5-1.5.111---00 1--IStuuttt reference 1.50,
Qwen2255778 models:
Comeaatinn
Pesron Comertton
0.3
08
Staps
Stops
Qwen2.5-1. 1.rem2.5-1..00 5-1.558-nsstttcct reference models: 1.58, 308
Qwen2.5-78. reference models: 1.58, 308
1.3
1.
Cormantion
Pertonn Commattion
RDC
Pearton Cormeation (Raw)
Pearton Cormiation (moothed)
ROC (Ram)
ROC (Groothed))
Figure 6: Performance of the interpolation-basee implicit evidence estimator during training of Swen225-1.5B-Instrrus (Left) and Qwen2.5-7B (Right), under different choices of reference model pairs: 7B & 30B (Top), 1.5B & 7B (Middle), and 1.5B & 30B (Bottom). Predictive quality is measured by Pearson Correlation and
Observations. The consistently positive correlation and ROC AUC above 0.5 demonstrate that interpolation-batt based implicit evidence effectively captures task difficulty, even in the the extrapolation-heary setting, where the pair (7B, 3 is used to predict task success probabilities for 1.rem2.5-1..BB-nnttuutt All other reference-modde combinations maintain even stronger predic- tive quality throughout training. A mild performance drop observed in the {1.5B, 30B) - 7B setting, which the only non-extrapolative scenario that both reference models differ substantially in capability from the training model. Additionally, we observe both Pearson Correlation and ROC AUC decline later stages in several cases, indicating that implicit evidence becomes less infor- mative for difficulty prediction as the model matures. This highlights the necessity of incorporating direct evaluations as explicit evidence to maintain accurate estimation.
Analysis and Takeaways. These results indicate that choosing well-bracketed reference mod- els SSs-partiuularly avoiding strong extrapolation regimes and, when possible, including a reference model whose capability is close to the training model--ubstantiall improves the quality of im- plicit evidence within BOTS.
Next, evaluate how different choices of reference models affect the performance of BOTS. We apply three reference model pairs to BOTS and train Qwen2.5-1155-Innss on the GURU-Math dataset. The results are presented in Figure
29

---

Published as a conference paper at ICLR 2026
Ratto
030
Stuat
Soppe
MSOO
ANC23 sugg4
avgg2
Staps
MATH500 avagl (smouthed)
AMC23 ava24 (smosched)
AIME24 axcej (smoothuc.)
0340
0.5
(11
29
Bandors
reforence medels:7b,
model:l 5b.
BCTS, reference models:1.5l,
Figure 7: Qwen2.5-1.5BIIntrruee on Math with various combinations of reference models. Ratios of sampled training tasks (measured with 16 rollouts) whose empirical success probabilities are strictly between 0 and 1 (LU), strictly greater than 0 (MU), and strictly less than 1 (RU), together with performance curves averaged over random runs with 95% confidence intervals on MATH500 (LM), AMC23 (MM), and AIME24 (RM), plotted against training steps. The bottom row shows smoothed versions (running average, window size 3) of the curves in the middle row.
Observations. We observe that the reference model pairs with stronger implicit evidence estima- tion quality (Figure () namely 1.5B 7B and 1.5B x 30B achieve clearly better ETR and more efficient training curves compared to the {7B 30B) pair. Even though 7B 30B performs slightly worse than the other combinations, it still delivers performance that significantly surpasses the random baseline.
Analysis and Takeaways. Overall, BOTS exhibits low sensitivity to the choice of reference mod- els: even imperfect implicit-evidence estimates (e.g., under extrapolation) still yield substantial training improvements. At the same time, better-chosen reference models produce higher-quality implicit evidence, which in turn further strengthens training efficiency and end performance.
G.3 SAMPLING FROM POSTERIOR
We now investigate the impact of posterior sampling. As discussed in Section pling naturally balances exploration and exploitation bandit-style problems. Without sampling, tasks with the closest estimated success rates are greedily selected for training. which risks over- exploitation and insufficient exploration.
3.4
posterior sam-
To examine this effect, compare default setting (A 0.1, 0.1) with posterior sampling enabled versus disabled. The valid ratio metrics and benchmark performance metrics are reported in Figure & and Table G.3
30

---

Published as a conference paper at ICLR 2026
Effective Task Radio (ETR)
>0 Ratio
<1 Ratio
Math Aggregatee Performance
540
0.-
0.24
Random
AHO3, prb.i sarrpherfata
Figure 8: Qwen2.5-1.5BIInttte on Math. Ratio of sampled training tasks (measured over 16 rollouts) with passing rates: strictly between O and 1, strictly greater than O, and strictly less than 1, along with aggregated performance (MATH500 and AIME24), plotted against training steps.
Benchmark
MATHS00
AIMF24
Math Aggrepated Performance
(1)
BSP
TIB
BSF(1)
(1)
(1)
Method (1), (-2)
50%
75%
1009%
25%
50%
100M6
50%
75%
100%
25%
50%
100%
50%
75%
100%
25%
50%
100%
1.00
1.00
= 0.I.p = 0.1, sample-Trae
0.89
0.49
0.57
1.13
1.0
105
0.51
1.00
1.00
1.25
1.75
1.00
0.89
0.56
0.64
1.12
1.07
1.05
A 1.1.p sample=Fulse
0.54
103
0.81
1.07
Table 7: Qwen2.5-1.5B-Insttreee on Math. TTB (lower better) and BSF (higher better) evaluated on MATH500, AIME24, and aggregated performance. For TTB, notation "-" indicates the the target performance never achieved within the evaluation window. The best and second best results are marked accordingly.
Observations. When posterior sampling is disabled, the valid ratio exhibits faster and higher boost in the early phase due to the removal of randomness, but fluctuations appear as training pro- gresses. In contrast, enabling posterior sampling yields smoother valid ratio trajectory over time. Notably, these differences in valid ratio do not translate into significant differences in benchmark performance: both settings outperform the random baseline and achieve very similar performance levels. Given the improved stability of the valid ratio, recommend enabling posterior sampling, though this conclusion is less pronounced compared to the effects of À and p. We leave a larger-scale study for future work to obtain more reliable evidence.
G.4 DYNAMICS OF SELECTED TASKS
In addition to reporting the Effective Task Ratio (ETR), which reflects the proportion of selected tasks with success probabilities strictly between 0 and 1, conduct finer-grained analysis to capture more detailed dynamics. Specifically, we visualize the distribution of success probabilities for selected tasks along the training trajectory, for both 1rren2.-1.55B-Insttee and Qwen2.5-7B models. This analysis complements ETR by revealing how the quality of selected tasks evolves beyond the binary cffectivee/nefffcctiee distinction.
We use a heatmap where the x-axis represents training steps, the y-axis represents the empirical success rate (discretized from 0/16 to 16/16), and the color intensity indicates the proportion of tasks sampled at that success rate. This allows us to compare the distributional dynamics of BOTS against the random baseline.
The resulting heatmaps in Figure reveal starkly different behaviors. The random baseline (Top) exhibits largely static distribution, with a persistent, high-density band at the 0/16 success rate, indicating continuous wasted computation unsolvable tasks. In contrast, BOTS (Bottom) demon- strates a highly dynamic curriculum. The initial concentration of tasks at the 0/16 success rate diminishes rapidly. Concurrently, the sampling density shifts upward, concentrating in the interme- diate difficulty range.
This visualization directly illustrates how BOTS actively filters out overly easy hard tasks and focuses computational resources on the most informative ones, which explains the superior Effective Task Ratio and overall performance gains observed previously.
31

---

Published as a conference paper at ICLR 2026
Qwen2.5
3826
35236
1526
13776
9.36
06
8838
6666
423
2016
8/38
04
100
Step
Qwen2.5
16-66
15406
13238
13:16-
9.36
06
1836
666
4/38
00
Figure 9: Heatmap visualizing the distribution of sampled task success rates over training steps for Random sampling (Top) and BOTS (Bottom) on Qwen2.5-1.5B Instruct.
Qwen2.5-78, Random
15216
1336-
0.8
13:38
1026
003
676
4:3
203
02
1/16
Step
Qwen2.5-78 BOTS
15:16
13:16
0.2
13:36
1071
0336
030
518
303
02
1/16
Step
Figure 10: Heatmap visualizing the distribution of sampled task success rates over training steps for Random sampling (Top) and BOTS (Bottom) on Qwen2.5-7B.
The analysis on the 7B model, shown in Figure 10] reinforces our findings. Consistent with the 1.5B results, BOTS (Bottom) rapidly diminishes sampling of unsolvable tasks (0/16 success rate) and progressively shifts its focus to the intermediate difficulty range. However, interesting phe- nomenon emerges in the later training stages due to the 7B model's stronger capability. For the random baseline (Top), high-density band appears at the 16/16 success rate, indicating that sig- nificant computation is wasted on tasks the model has already mastered. stark contrast, BOTS effectively avoids this region, maintaining broad distribution across the intermediate success rates.
32

---

Published as a conference paper at ICLR 2026
This demonstrates BOTS's advanced adaptivity: it not only filters out tasks that too hard but also dynamically avoids those that become too easy, thereby maximizing learning efficiency throughout the entire training process.
G.5 EXTERNAL BASELINE: DYNAMIC SAMPLING
NATT5500
ANC2 2V994
NINE24 avg0
0.50
3.25
0.50
3.45
8.533
3.15
5.20
3.0
0.33
4.40
305
1.50
0.
200
300
6.90
100
avpg1 (smoothed)
sugg4 (smosthed)
AJNE24 ave 32 (smoothec)
larye
Dynamic samping
Figure 11: Qween.55-1.B-IInsttue on Math. Performance curves averaged over 3 random runs with 95% confidence intervals on MATH500 (LU), AMC23 (MU), and AIME24 (RU), plotted against training steps. The bottom row shows smoothed versions (running average, window size 3) of the curves in the upper row.
We conduct additional experiments comparing BOTS with Dynamic Sampling (Yu al. 2025). an oversampling-ball biasee online task selection strategy. Results are shown in Figure I1 For Dynamic Sampling, we follow the original setup and train for one full epoch (55 steps, 54.4K tasks). For reference, both BOTS and the random baseline consume 25.6K tasks over 100 training steps (batch size 256). Although the performance curves are plotted step-wise for consistency, note that each step of Dynamic Sampling incurs roughly 2.5x the time cost of BOTS or random baseline step.
important to
Observations. Interestingly, while Dynamic Sampling consistently outperforms the random base- line, it slightly underperforms BOTS even under a step-wise comparison despite achieving the ideal effective task ratio (ETR=
Analysis and Takeaways. A likely explanation is that Dynamic Sampling only filters out invalid tasks (success probability 0 or 1) but does not differentiate among the remaining ones. In contrast, BOTS explicitly prioritizes tasks whose success probabilities lie near the ideal difficulty (e.g., 0.5), which are most beneficial for learning. Consequently, BOTS achieves both higher step-wise effi- ciency and lower computational cost. Overall, in the studied setting, BOTS outperforms Dynamic Sampling on both axes.
33

---

Published as a conference paper at ICLR 2026
G.6 EXTENDED EXPERIMENTAL RESULTS: IMPACT P
Steas
Steas
Shee
MUST11500
AMC23
NINE2
MATHH00 wugg1 (smontheed)
AMF2' sugB4 (veonthed)
AIMPP4 wayeh 12 (ymontheee)
2.a
1.50
3.2
1.30
1.30
1.55
6.35
3.1
312
0.11
110
2.00
27
Figure 12: 1rren.5-1.5B-Instruee Math with À 0.1, p E {0.0, 0.05, 0.2, 0.5, 1.0] Ratios of sampled training tasks (measured with rollouts) whose empirical success probabilities are strictly between 0 and 1 (LU), strictly greater than 0 (MU), and strictly less than 1 (RU), together with performance curves averaged over random runs with 95% confidence intervals on MATH500 (LM), AMC23 (MM), and AIME24 (RM), plotted against training steps. The bottom row shows smoothed versions (running average, window size 3) of the curves in the middle row.
34

---

Published as a conference paper at ICLR 2026
G.7
ADDITIONAL EXPERIMENTAL RESULTS: IMPACT OF
Pflective Task Ratio (FTR)
20 Ratio
Ratto
1.00
6372
0.50
6.4
0.00
MMM
1.80
200
ave
augg24
augg232
1.43
200
030
0.0
630
Steat
Stags
Stepe
MATH500
(smoochedl)
AIME24 (smosthhec)
S182s
Fandom - 0.0.0 0,066 2-0.1
1-0
2.0055
1.00
Figure 13: Qwen2.55-1.B--nstruuee on Math with À E {0.0, 0.05, 0.1, 0.2, 0.5, 1.0), 0.1. Ra- tios of sampled training tasks (measured with 16 rollouts) whose empirical success probabilities are strictly between 0 and 1 (LU), strictly greater than (MU), and strictly less than (RU), together with performance curves averaged over random runs with 95% confidence intervals on MATH500 (LM), AMC23 (MM), and AIME24 (RM), plotted against training steps. The bottom row shows smoothed versions (running average, window size 3) of the curves in the middle row.
35

---

Published as a conference paper at ICLR 2026
G.8
EXTENDED EXPERIMENTAL RESULTS: MATH
Pflective Task Ratio (FTR)
20 Ratio
Ratio
6.72
0.50
6.4
0.00
1.00
1.60
200
ave
1.30
augg24
augg232
0.40
2.0
030
0.0
690
Staat
Stags
Stepe
MATH500
(smoochedl)
AIME24 (smootthec))
S1823
Random
Offiine
3OTS-MUPPS
80T5-20TS
BOTS
Figure 14: Qwen225-115B-Insstrue on Math. Ratios of sampled training tasks (measured with 16 rollouts) whose empirical success probabilities are strictly between O and 1 (LU), strictly greater than (MU), and strictly less than 1 (RU), together with performance curves averaged over 3 random runs with 95% confidence intervals MATH500 (LM), AMC23 (MM), and AIME24 (RM), plotted against training steps. The bottom row shows smoothed versions (running average, window size 3) of the curves in the middle row.
Bexhhaak
Agproganed Perforence
Target
1,02
1.66
0.27
0.66
MOPPs
9:59
0.98
0.97
1.36
0.73
0:51
0.00
1.05
BOTS DOTS
0.72
0.56
0.74
1.12
1.07
1,01
8.75
0.76
0.66
1,03
1.08
1.01
0L58
1.23
0.64
1.18
0.96
1.24
0.70
0.70
0.73
1.08
1.06
BOTS
0.85
0.65
0.78
1.0
100
1,86
2.56
0.24
110
1.16
1.66
0.56
0.5
0.50
1.20
2,25
1.3
035
0.06
0.73
1.66
1.12
1,05
Table 8: TTB and BSF evaluated on Math with Qwen2.55-15B--nntrrue For TTB, notation <.. indicates that the target performance is never achieved within the evaluation window. The best and second best results are marked accordingly.
36

---

Published as a conference paper at ICLR 2026
Effective
Ratio
1.0
0.6
0.4
07-
40
100
20
60
103
20
40
60
100
Training Steps
Training Stees
Training Steps
Random
Omtine
AnD p.0.0
- A.0.1, po111
A=1.0. p=1.0
Figure 15: Qwen2.5-7B Math. Ratio of sampled training tasks with different passing rates training steps.
MATH500 Accuracy
AIME24 Accuracy
Aggregated Performance
0.20
0.50
0.7
0.15
0.40
0.30
0.00
0.20
Random
Offine
sample-Thoe
1m011 pro.1. sample-rTue
2m:1.. po..0. somple-FFato
Figure 16: Qwen2.5-7B on Math. Performance on downstream math benchmarks (MATH500, AIME24) and their aggregation.
Benelmaak
MATH500
AIMEZ4
Metric
TTB (1)
BSF (1)
TTB (1)
BSF (1)
TTB (3)
BSF (1)
Merhod (J). ( !)
50%
75%
100%
25%
50%
100%S
50%
75%
100%
25%
50%
100%
50%
75%
100%:
25%
50%%
100%
Offine
0,96
0.70
0,79
1,02
1.01
0.86
0.74
0.88
1.00
1.05
0.91
0.73
1.00
1.05
BOTS MoPPS
1.10
1.10
1.35
0.96
1.00
1.00
1.14
0.64
0.85
1.19
1.10
1.07
1.15
0.70
0.94
1.04
1.04
BOTS
0.91
0.74
0.93
1,01
1,02
1.02
0,79
0.63
0.94
0.97
1.11
0.86
0.77
0.63
0.99
1.04
1.04
Table 9: Qwen2.5-7B on Math. TTB (lower better) and BSF (higher better) evaluated down-
stream math benchmarks.
37

---

Published as a conference paper at ICLR 2026
G.9
EXTENDED EXPERIMENTAL RESULTS: CODE
Effective Task Ratio (ETR)
Ratio
Ratio
1.3
10
0.0
0.9
0.0
0.4
0.0
0.0
Training Stegrs
Faining Steps
Taiiinng Stepp
Random
Offilne
20300,
samole-True
sample-True
2= p=1.0. somple--Fatse
Figure 17: -wen2.5--.55-Insstit on Code. Ratio of sampled training tasks with different passing rates.
Hummnntval Accuracy
HBPP Accuracy
LieCoddBeench Accuracy
Code Apgregated Performance
8:0
0.45
8.5-
8.55
0.35
14
8.44
0.30
20
35
43
30
3103
$mcken
Figure 18: Qwen2.5-1.B--nstrrun on Code. Performance on downstream code benchmarks (Hu- manEval, MBPP, LiveCodeBench) and their aggregation.
Baxdumak
HassaEErl
LiveCodBerrch
Merio
TTR
ISP
TSSF
25%
Bundon
1.00
1.01
1.00
1.50
L(D
1,00
1.00
1.00
LIMI
1.20
1.50
1.00
1.00
1.0)
1.00
1.00
1.01
1.00
1.00
1.00
1/03
Oifline
B:9
0.81
1.17
1m
0.63
1.99
1.17
0.99
1.09
Table 10: Qwen2.5-1.5h 1ren2.5-1.5B-Immn --Sttuuttt on Code. TTB and BSF evaluated on HumanEval, MBPP, Live- CodeBench, and aggregated performance.
38

---

Published as a conference paper at ICLR 2026
Valid
Ratio
Ratio
1.0
041
2.4
0.7
10
20
50
60
33
30
40
20
30
60
70
Training Steps
Training Steps
Training Steps
Random
Ofline
AnO.O, p-0.0. Timpleu
Auo. p.o.l. sample-True
- Am1.O, p=1.., stttle-Fatoe
Figure 19: Qwen2.5-7B on Code. Ratio of sampled training tasks with different passing rates.
08
Iuumantvall Accuracy
NUPP Accuracy
0:5
LiveCedeteench Accuracy
0.00
Code Apgregaced Performance
0.8
0.20
0.10
6:7%1
04-
0:00
0.45
15
8.5
0.
33
20
96
2.
35
8.0
9.2
$p5
2x2.5 pH0lA
pm11). Valoooaii
Figure 20: Qwen2.5-7B on Code. Performance on downstream code benchmarks and aggregation.
LovtUbektlench
Aggropard
BSF
(T)
(T)
Mathed (1). (-4)
3055
100%:
92%
10075
1005
1,03
1.00
1.0)
1801
1.00
3.13
[L:9
S0PS.MPPP
0:51
0.54
844
188
1.65
1.0
0.83
1.21
0.96
0.59
0.02
0.56
0.38
0.56
0.54
0.58
1.44
1.07
BOTS DOTS
857
0.80
0.73
183
0.73
0.68
1.00
0.95
1.30
0.56
0.54
0.36
1L.66
1.02
1/14
0.33
100
6,9
082
1X0L
Table 11: Qwen2.5-7B on Code. TTB and BSF evaluated on downstream code benchmarks.
39

---

Published as a conference paper at ICLR 2026
G.10
EXTENDED EXPERIMENTAL RESULTS: LOGIC
Effective Task Ratio (ETR)
>-0 Ratio
<1 Ratio
1.3
10
1.
0.0
0.-
2.-
6.1
60
0.0
20
102
0.0
Training Steps
Training Steas
Training Steps
Random
Offinne
2=3.0,
samole-Thue
tample-True
2= p=1.0.
Figure 21: Qwen2.5-1.5B- -.5tstruutt on Logic. Ratio of sampled training tasks with different passing rates over training steps.
Ordering Puzzle Accuracy
ARCAGI Accuracy
Logic Aggregated Performance
0.25
0.15
0.05
0.000
p.0.0,
Figure 22: Qwwnn.551.5B-Insttree on Logic. Performance on downstream logic benchmarks (Or-
dering Puzzle, ARCAGI) and their aggregation.
Benchmark
Ordering Puzzle
ARCAGI
Aggregated
Metric
(1)
(1)
(1)
(1)
Method (J). 95
50%
75%
1009
25%
50%
100%
50%
75%
1005:
25%
50%
100%
50%%
75%
100%
25%
50%
100%
Random
1.00
1.00
1,00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
Offine
BOTS-MoPPS
0.87
1.02
1.12
0.98
0.98
1,23
1.04
0.78
1.00
1.00
1.00
0.87
1.04
1.13
0.96
0.96
BOTS-DOTS
0.65
1.51
1.27
0.93
0.94
1,00
0.87
1.17
0.78
0.89
0.66
1.32
1.28
0.92
0.93
Table 12: Q.wen2.5-1.5B-Instttee on Logic. TTB and BSF evaluated on downstream logic bench- marks.
40

---

Published as a conference paper at ICLR 2026
Effective
Ratio
1.0
0.2
0.4
6A-
0.4
40
00
100
20
60
503
20
60
100
Training Steps
Training Steess
Training Steps
Random
Offine
AcO.0, p.O.D. sample-True
2=011 p-0.1. sample-True
- 2= p=1.0. somple--Fass
Figure 23: Qwen2.5-7B on Logic. Ratio of sampled training tasks with different passing rates training steps.
Ordering Puzzle Accuracy
ARCAGI Accuracy
Logic Aggregated Performance
0.03
0.3
0.03
0.3
0.01
0.2
0.00
Random
Offine
Am:.0.
sample-rhoe
1m011 pro.1, sample-rTre
km.10. somple-Fatoe
Figure 24: Qwen2.5-7B on Logic. Performance on downstream logic benchmarks and their aggre- gation.
Ordering Puzzle
ARCAGI
Aggregated
Metric
TTB
(1)
BSF (1)
TTB (1)
BSF (1)
TTB (3)
BSF (1)
Merhod (J). ( !)
50%
75%
100%
25%
50%
100%
50%
75%
100%
25%
50%
100%
50%
75%
100%:
25%
50%
100%
1.00
Offine
1,07
1.21
1.69
0.98
1.05
1.13
1,51
0.71
1.00
1.06
1.23
0.95
1.01
BOTS MoPPS
0.75
0.89
0.67
1.12
1.00
1.05
0.84
1.47
0.71
1.12
0.55
0.75
0.92
0.09
1.07
1.00
1.05
BOTS
0.80
0.77
0.89
1,01
1.00
1.00
0.72
0.88
1.12
0.79
0.78
0.30
1.03
1.01
1.00
Table 13: Qwen2.5-7B Logic. TTB and evaluated on downstream logic benchmarks.
H USAGE OF LARGE LANGUAGE MODELS
We employed Large Language Models solely for the purpose of polishing the writing in this manuscript. Their function was limited tasks such as correcting grammatical errors, rephras- ing sentences to enhance clarity and flow, and ensuring the consistent use of terminology. The LLMs had no role in the ideation of the research, the development of the BOTS framework, the experimental design, or the analysis of results.
41


Reviews:
[{"review_id": "uEQsZs5r9A", "reviewer": "Reviewer_e5tW", "summary": "This paper studies the problem of online task selection in reinforcement finetuning (RFT) for Large Language Models (LLMs). The authors introduce BOTS, a unified Bayesian framework that aims to actively select tasks for each step that are of appropriate difficulty at the current stage. BOTS models task difficulty using a Beta distribution posterior and adaptively updates its beliefs by fusing two sources of information: explicit evidence from direct evaluations of selected tasks and implicit evidence from unselected tasks. The framework includes a lightweight interpolation-based method to generate implicit evidence with negligible overhead. Task selection is performed via Thompson sampling to balance exploration and exploitation. Experiments across various domains and model scales show that BOTS improves data efficiency and model performance compared to random selection.", "questions": "1. This work seems to be an extension of MoPPS [4], with new consideration on the balance of explicit and implicit evidence. But I don't understand why it can additionally capture \"cross-task relationship\". In my understanding, BOTS follows the same Thompson Sampling strategy as MoPPS, where different tasks are all treated independently. I believe BOTS aims to synchronizingly update tasks' difficulty that are not sampled at the current stage, to get a more precise estimation of these tasks for later sampling. Please correct me if I'm wrong.\n2. How are $\\bar{p}^k_w$ and $\\bar{p}^k_s$ computed? I'm confused according to the description in the main text. From Appendix E.3, it seems that it directly uses precomputed results, then it may have large errors since training details are quite different. Moreover, I think the ablation of reference model choices is necessary and is not adequate for future work. I wonder if better reference model choices can indicate better Pearson correlation and ROC AUC?\n3. Why is the result of \"Random\" not reported in Figure 15/16?\n\n[4] Can prompt difficulty be online predicted for accelerating RL finetuning of reasoning models?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The studied online task selection problem is crucial for the LLM reinforcement finetuning problem, and BOTS's formulation of it as a unified Bayesian inference problem is elegant and well-suited for the task.\n2. BOTS avoids the need for additional model rollouts during training, and the computational overhead is shown to be negligible ($\\le 0.2\\%$) while consistently showing performance gains over random selection, making the framework efficient and effective in adoption.\n3. The idea of trying to balance explicit and implicit evidence is practical for stabilizing the effective sample size $n_t$, as proved in Proposition 2. And the authors show it is beneficial for cold start in Section 4.2.", "weaknesses": "1. Although quite efficient, the linear interpolation function in BOTS may be weak as an instantiation of $\\tilde{p}(k, \\mathcal{B}_t)$. Results in Figure 5 show that the Pearson correlation between estimated and empirical difficulties is weak, and the ROC AUC is also not good enough (especially for Qwen2.5-1.5B-Instruct). And there is no theoretical analysis to justify why a linear interpolation alone can adequately capture the complex information needed for task difficulty estimation.\n2. The training time is not reported in the main results. Although BOTS generates the same number of rollouts and trains for the same number of steps compared to random sampling, the actively selected tasks usually generate longer rollouts that actually need more computational resources. Thus, a fairer comparison should let random sampling generate more rollouts per task to have a similar training time as BOTS.\n3. The comparisons to prior work DOTS [1] are done via \"within-framework\" ablations (BOTS-DOTS) rather than against their original implementations (Section 4.4, Appendix B.6). While this is sufficient for case studies within the BOTS framework, it does not fairly compare the performance between BOTS and DOTS. A comparison between BOTS and the original DOTS is needed. Additionally, more related baselines should be included for comparison to highlight BOTS's advantage (e.g., Dynamic Sampling [2], SPEED [3]).\n4. The framework mainly studies two important hyperparameters, $\\lambda$ and $\\rho$. The ablation studies show that the model's performance is sensitive to these choices (Section 4.2, 4.3). The paper does not provide a clear methodology for setting these hyperparameters beyond empirical sweeps, which could be computationally expensive for new applications. Additionally, the momentum coefficient $\\gamma$ is a crucial hyperparameter for interpolation, but the impact of it is not discussed in the paper.\n5. It seems BOTS mainly works on math problems; there's no significant performance gain compared to random sampling in other cases, especially on experiments using Qwen2.5-7B on Code and Logic. The generalization is somewhat limited.\n6. Minor suggestions: providing an algorithm pseudocode can help readers understand the workflow of BOTS more quickly.\n\n[1] Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay.\n\n[2] DAPO: An Open-Source LLM Reinforcement Learning System at Scale.\n\n[3] SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mCV70Q0qqD", "reviewer": "Reviewer_kBRo", "summary": "The paper proposes a unified framework for Bayesian Online Task Selection in LLM reinforcement finetuning, named BOTS. The method jointly incorporates explicit evidence the selected tasks and implicit evidence from unselected tasks, with Thompson sampling to pick tasks which is close to a target difficulty. The method's effectiveness is empirically demonstrated on several benchmarks.", "questions": "In table 1, it seems that the proposed method is sensitive to the hyperparameter values and there is no consistent conclusion. Is there any common principle to set the hyperparameters according to the task?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n\n2. The proposed idea is interesting and novel.\n\n3. The empirical results demonstrates the effectiveness of the proposed method.", "weaknesses": "1. The proposed method is only evaluated on binary rewards. It is not clear whether it is applicable on more complex tasks.\n\n2. The comparisons are internal. Lack of comparisons with other alternatives.\n\n3. There are some related work on task selection are missed:\n\n[1] DATS: Difficulty-Aware Task Sampler for Meta-Learning Physics-Informed Neural Networks.\n\n[2] Model predictive task sampling for efficient and robust adaptation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "di4xqsqc4M", "reviewer": "Reviewer_u4yu", "summary": "The paper proposes BOTS, a Bayesian framework for online task selection in reinforcement finetuning (RFT). It maintains posterior estimates of task difficulty and uses Thompson sampling to select tasks near a target difficulty (in practice, $p^*=0.5$).  BOTS aims to balance data efficiency and adaptability by combining explicit evidence from direct rollouts and implicit evidence inferred through an interpolation-based plug-in. Experiments on the GURU benchmark with Qwen2.5 models demonstrate improved sample efficiency compared to uniform random and other baselines.", "questions": "- Were any statistical significance tests (e.g., across seeds) conducted for the reported improvements? Confidence intervals or variance would strengthen claims of robustness.\n- Since $\\lambda$ and $\\rho$ control forgetting and evidence fusion and seem to balance early and long-term performance, have you considered adaptive or schedule-based versions that evolve over training rather than remaining fixed?\n- Correct me if I'm wrong, but there seems to be a minor inconsistency in Section 3.4: the paper defines utility as the distance $\\| p_k - p^* \\|$ from the target difficulty but states that tasks with the highest utility are selected, which appears opposite to the stated goal of favoring tasks difficulty near $p^*$.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Provides a principled Bayesian formulation of online task selection that generalizes prior methods.\n- Fuses explicit and implicit evidence to improve stability and cold-start performance.\n- Includes ablations on $\\lambda$ (forgetting) and $\\rho$ (evidence fusion), providing good insight into the method’s behavior.\n- The interpolation-based implicit evidence is computationally efficient (<0.2% overall overhead).", "weaknesses": "- Experiments are limited to two LLM scales (1.5B and 7B) of Qwen2.5 on a single benchmark (GURU), raising concerns about generalizability.\n- The reported improvements over baselines, appear modest and sometimes inconsistent across domains and target fractions and no confidence intervals or statistical tests are provided.\n- The method assumes access to reference models for implicit evidence, the practicality of this in new domains is not fully addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XWAlOIdJHh", "reviewer": "Reviewer_UaYb", "summary": "This paper proposes a principled Bayesian framework to improve task selection efficiency during RFT of large language models. Instead of uniformly sampling tasks, which wastes computation on trivial or unsolvable ones, BOTS maintains Bayesian posteriors over task difficulties that evolve with model capability. It fuses explicit evidence (direct evaluation results) and implicit evidence (inferred task difficulty via inter-task relationships) using generalized Bayesian updates, and employs Thompson sampling to balance exploration and exploitation. The authors further introduce an ultra-light interpolation plug-in that estimates unselected task difficulty without extra rollouts. Experiments across math, code, and logic domains, using Qwen models of different scales, show that BOTS consistently improves data efficiency and training speed with negligible computational overhead.", "questions": "1. Regarding the interpolation-based implicit evidence plug-in in Section 3.3: (1) What happens if there are multiple candidate weak and strong models? Would the resulting interpolations be consistent across different reference pairs?\n(2) I saw the empirical results in the appendix, but the reported correlation coefficients (around 0.5) seem rather low. Could you elaborate on why this level of correlation is sufficient to justify the estimator’s reliability?\n(3) If part of a dataset is either too easy or too hard—so that both weak and strong models perform similarly high or low—would the interpolation still behave well? I understand you could in principle extrapolate, but is there any empirical evidence showing this works reliably?\n(4) Also, while you claim efficiency due to pre-computed references in existing datasets, would new datasets still require additional one-time rollouts for the reference models? It would be helpful to discuss this limitation explicitly.\n\n2. In Section 4.2, you state that implicit evidence “fails to recognize when tasks are fully mastered.” Could you provide further explanation or empirical evidence to support this claim?\n\n3. (Minor) For the BSF metric, why do you report results at 25% and 50% of total training steps? Wouldn’t BSF at 100% be sufficient to summarize the final performance? \n\n4. (Minor) I suggest adding a complete pseudocode of the full algorithm in the appendix. This would help readers clearly follow the overall logic and implementation flow.\n\n5. (Optional) Including error bars for key results—at least for the 1.5B model—would make the experimental analysis more convincing, if computational resources permit.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1. The paper presents an interesting and original idea of combining both explicit and implicit evidence for online task selection, and unifies them under a principled Bayesian framework.\n\n2. The interpolation-based implicit evidence plug-in is novel and practical, enabling efficient estimation without additional rollouts.\n\n3. The writing is very clear and well-organized, with a solid and detailed appendix that strengthens the paper’s technical quality.\n\n4. The experimental section is thorough: it clearly introduces evaluation metrics, systematically analyzes the effects of ρ and λ, and then presents comprehensive main results demonstrating certain improvements.", "weaknesses": "1. A major issue lies in the analysis of λ. The statement at line 244 that “smaller values accelerate forgetting and increase posterior uncertainty” appears inconsistent with Equation (1) and Proposition 2. Based on the update rule, smaller λ should slow down forgetting and reduce posterior uncertainty due to larger effective sample size. Moreover, line 372 later describes λ → 0 as “long memory,” which directly contradicts the earlier statement. The paper should carefully revisit this part.\n\nRelatedly, the use of terms such as forgetting, exploration, and exploitation is somewhat confusing. It is not always clear what exactly is being “forgotten” or what dimensions are being “explored” versus “exploited,” especially since these terms may differ from their usage in reinforcement learning. Clarifying these concepts would improve readability and conceptual precision.\n\nOverall, I suggest the authors carefully revisit the analysis of how λ and ρ affect the algorithm. The current explanations are confusing and inconsistent, making it difficult to assess whether the paper’s claims about their impact are correct.\n\n2. On the experimental side, the evaluation on math benchmarks is limited. Only MATH500 and AIME24 are used, which seems insufficient to demonstrate generality. Moreover, AIME24 may be too difficult for small models, leading to unstable results. For example, as shown in the appendix figures, the 1.5B model’s AIME24 accuracy remains near zero throughout training, making it difficult to draw meaningful conclusions from such results.\n\n---\n\nI would be open to adjusting my rating depending on how the authors address these issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mc1u4elHHv", "reviewer": "Reviewer_SYAn", "summary": "The paper proposes a unified Bayesian online task selection framework for reinforcement fine-tuning of large language models. Specifically, it builds on Bayesian inference using a Beta distribution to model the policy’s success rate, incorporating both explicit evidence from batch reward signals and implicit evidence from inter-task relationships to update the parameters. The authors provide theoretical justification for the posterior inference and analyze how parameters influence the learning dynamics. Empirically, they conduct extensive experiments demonstrating the effectiveness of the proposed method and the impact of its key hyperparameters in terms of multiple metrics, including effective task ratio, time-to-baseline, and best-so-far across domains and models.", "questions": "1. Overall, based on Figure 3, when $\\lambda$ is small, the sampled $p$ from the posterior estimation tends to be smaller than the ground truth, whereas when $\\lambda$ is large, the sampled $p$ tends to be larger than the ground truth. Could you provide a consistent explanation for this behavior from both the mean and variance estimation perspectives?\n2. I appreciate that the authors discuss the selection of the reference model and the potential bias in estimating $\\mu_B$ in the appendix. How does the choice of reference model affect empirical performance? Is it robust? Is there any guidance or heuristic for selecting it in practice?\n3. How is the prior distribution (i.e., $\\alpha_0$ and $\\beta_0$) for each sample initialized?\n4. Could you report the correlation between the interpolated average success rate and the ground-truth average success rate across training rounds to demonstrate the accuracy of the implicit evidence estimation?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method is technically sound and principled. Incorporating both explicit and implicit evidence within recursive Bayesian inference framework for RL online task selection is well-motivated.\n2. The paper provides both theoretical and empirical justifications for the key hyperparameters $\\lambda$ and $\\rho$, which is commendable.\n3. The experimental analysis is thorough, and the metric design is reasonable, showing consistent gains across multiple domains and model scales.", "weaknesses": "1. Given that the recursive Bayesian framework has been proposed in prior work, and that the proposed estimation of implicit evidence via interpolation with the reference models is relatively simple, the overall contribution beyond prior studies [1][2] appears incremental.\n2. A major issue in the paper is the discussion of prior mixing or temporal discounting, i.e., the interpretation of $\\lambda$, which is confusing.\n- First, it is unclear why interpolation with the prior distribution $\\pi_0$ is introduced in the Bayesian sequential update. While such interpolation might be beneficial for non-stationary processes, the paper should provide appropriate references or theoretical justification.\n- Several statements regarding $\\lambda$ are inconsistent and potentially contradictory, raising concerns about possible implementation errors.\n(1) In Proposition 2 (lines 239–242), the effective sample size $n_t$ depends inversely on $\\lambda$, implying that $n_t$ increases as $\\lambda$ decreases. This contradicts the claim in lines 1139–1140 that smaller $\\lambda$ leads to a smaller effective sample size and higher variance.\n(2) In Section 4.3, lines 350–352 state that smaller $\\lambda$ emphasizes recent evaluations while larger $\\lambda$ focuses on accumulated history. However, lines 373–375 claim that $\\lambda \\rightarrow 0$ corresponds to short-term memory and $\\lambda \\rightarrow 1$ to long-term memory. These descriptions are unclear and inconsistent.\n- Given the known distributional form, it would be helpful to include toy examples illustrating the effects of $\\lambda$ and observation noise to clarify its role.\n3. In general, the analysis would benefit from separating the discussion of uncertainty estimation and success-rate estimation, rather than conflating them. The paper should more clearly define and distinguish terms such as “memory,” “forgetting,” and “exploration.”\n4. The argument for low computational cost of the interpolation method in Section 3.3 is not convincing. The method requires reference labels from two models across the entire training dataset, which limits its scalability and undermines its practicality for new models and domains.\n\n[1] Qu, Yun, et al. \"Can prompt difficulty be online predicted for accelerating rl finetuning of reasoning models?.\" arXiv preprint arXiv:2507.04632 (2025).\n[2] Sun, Yifan, et al. \"Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay.\" arXiv preprint arXiv:2506.05316 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)