Bootstrapping Task Spaces for Self-Improvement
Authors: Minqi Jiang
Andrei Lupu
Yoram Bachrach
Venue: Accepted by TMLR ()

Abstract: Progress in many task domains emerges from repeated revisions to previous solution attempts. Training agents that can reliably self-improve over such sequences at inference-time is a natural target for reinforcement learning (RL), yet the naive approach assumes a fixed maximum iteration depth, which can be both costly and arbitrary. We present Exploratory Iteration (ExIt), a family of autocurriculum RL methods that directly exploits the recurrent structure of self-improvement tasks to train LLMs to perform multi-step self-improvement at inference-time while only training on the most informative single-step iterations. ExIt grows a task space by selectively sampling the most informative intermediate, partial histories encountered during an episode for continued iteration, treating these starting points as new self-iteration task instances to train a self-improvement policy. ExIt can further pair with explicit exploration mechanisms to sustain greater task diversity. Across several domains, encompassing competition math, multi-turn tool-use, and machine learning engineering, we demonstrate that ExIt strategies, starting from either a single or many task instances, can produce policies exhibiting strong inference-time self-improvement on held-out task instances, and the ability to iterate towards higher performance over a step budget extending beyond the average iteration depth encountered during training.

Published in Transactions on Machine Learning Research (02/2026)
Bootstrapping Task Spaces for Self-Improvement
Minqi Jiang
msju@eela.com com
Meta Superintelligence Labs
Andrei Lupu
Meta Superinted enperintelligence Labs University of Oxford
Yoram Bachrach
Meta Superintelligence Labs
Reviewed on OpenReview: https: // openreview. net/ forun? id= k2VsgUa06X
Abstract
Progress in many task domains emerges from repeated revisions to previous solution at- tempts. Training agents that can reliably self-improve over such sequences at inference-time is natural target for reinforcement learning (RI), yet the naive approach assumes a fixed maximum iteration depth, which can be both costly and arbitrary. We present Exploratory Iteration (ExIT), a family autocurriculum RL methods that directly exploits the recurrent structure of improvepro tasks to train LLMs to perform multi-step self-immprvement at inference-time while only training on the most informative single-step itcrations. ExIT grows task space by selectively sampling the most. informative intermediate, partial his- tories encountered during episode for continued iteration, treating these starting points as new self-iteration task instances to train a self-improvement policy. ExIT can further pair with explicit exploration mechanisms to sustain greater task diversity. Across several domains, encompassing competition math, multi-turn tool-use, and machine learning engi- neering, we demonstrate that ExIT strategies, starting from either a single or many task instances, can produce policies exhibiting strong inference-time self-improvement on held- out (ask instances, and the ability to iterate towards higher performance over step budget extending beyond the average iteration depth encountered during training.
1 Introduction
When given more time to think, humans can often reason toward better solutions difficult problems. Recently, reinforcement learning (RI) task-specific rewards has been shown to induce similar reasoning capabilities in pretrained large language models (LLMs) (Zelikman et al. 2022 Jaech et al. 2024 Guo al. 2025), whereby the model learns to output extended chains-of-thought (CoTs) exhibiting reasoning behaviors that can iteratively self-improve a solution, via tactics like correctness verification and backtracking. This approach has been applied in both verifiable domains, such as math and coding, where an oracle function determining the correctness of any given solution can be explicitly defined (Lambert al. (024)) and more recently extended to non-uerifiable domains like crcative writing (Gurung & Lapatal [2025] Zhou et al., 2025). Training such reasoning models via RL can be challenging: Empirically, eliciting useful reasoning patterns is dependent patterns (Gandhi et al. 2025), and thus supervised fine-tuning stage using carefully-curated reasoning traces may be necessary. As CoTs lengthen with multiple reasoning steps, response generation takes longer to complete, slowing down training and potentially exhausting context length and memory.
how likely it is for the base model to generate Co'ss already exhibiting these
In this work, we consider a complementary approach to improving an LLM's capacity for self-improvement inference time, based on training the LLM to perform K-step self-im improvemntt In this problem setting, the LLM is presented with initial solution and has budget of K steps to reach the best-performing

---

Published in Transactions on Machine Learning Research (02/2026)
solution, with each subsequent step starting from the response produced in the previous step, along with an optional feedback signal communicating the quality of that solution.
Naively, K-step self-improvement training may be performed by randomizing over the step budget K, with the reward for each step corresponding to either the absolute quality of the solution produced that step or the relative improvement with respect to the previous solution. However, this approach both overly restrictive and costly, requiring commitment to a maximum self- self-immrovement depth of K beforehand, while also increasing inference costs by an additional factor of (K + 1)/2 on average, when maintaining fixed number of training task instances. Moreover, this approach does not extend naturally to the multi-turn task setting, where cach turn may be allotted budget of K-steps for self-improvement before arriving at a final response for that step. Lastly, RL fine-tuning based on verified rewards can reduce output diversity, making it harder to find effective improvements to previous solutions (Wu et al. 2025
Our approach avoids these issucs by training for K-step self-improvement using only singlo-step self- improvement transitions. The starting point is the output of a previous turn, selected via prioritized sampling using score that quantifies the learning potential of reattempting that turn given its exact history. Each training task comprises the original instruction, the sampled turn's previous output, and (in multi-turn set- tings) the remaining turn history up to that point. By teaching the LLM to iterate on its own solutions, our method can be viewed as self-generated data augmentation over the task space. To maintain diversity during iteration, we study two exploration mechanisms: (1) a variant of e-greedy exploration where, with probability e, the task switches from self-improverment to self-divergence, aiming for a valid but substantially different approach; and (2) multiplicatively scaling the group- wise advantage of each rollout by its distance from the group centroid in an embedding space. Because any step endpoint can seed new iteration task, step-level and task-level exploration act in the same task space, blurring the line between classic step-level exploration and task-level autocurriulla. We refer to our approach as Exploratory Iteration (ExIT).
We demonstrate the effectiveness of EXIT variants across a variety of domains including competition math, multi-turn tool-use, and machine learning engineering tasks based on previous Kaggle competitions where policies trained via ExIr variants exhibit. improved K-step self-inprovement at inference-tine, with K extending beyond the typical improvement depth seen during training. Further, we observe that ExIT induces emergent autocurricula over various task complexity metrics while naturally increasing the diversity of the training lask instances, leading to improved performance even before sell-improvement.
2 Background
2.1 Self- Self-improvement decision process
We consider the K-step self-iniprovement setting in a task space M defining a set of task instances m, each corresponding to a multi-turn POMDP with a timestep budget of Tm For clarity and consistency with recent usage for LLM-based fasks, we refer to the timesteps of the underlying POMDP as turns and reserve step to refer to self-iteration step, where the LLM generates modified version of previous response.
For task instance m, at every turn t, for 1 S t S Tm, the LLM no with parameters 0, receives the task prompt (also denoted as m for convenience) and outputs response containing a solution to the request ye, followed by & 2 0 additional steps of self-iteration. Each iteration step acts on the latest solution iterate to produce the next iterate, vÊ where we define vf = yt. so the improve step with index corresponds to the initial response attempt for that turn. At cach itcration, π also observes the remaining history, y composed of the sequence of the last iterates per turn, up to the last. fully-iterated turn ( 1- 1. Optionally, associated feedback signals ot and e e.g. execution traces in a coding task, can be provided to π for each observed itcrate. For convenience, let the history Tf = (vf v et, ck..) Then,
yt t+l ~ me(-l>p,,
(1)
Let the final history be K Tim = Am Further, let G be a function mapping to a real scalar measuring the quality of the responses. In this work G is the undiscounted sum of rewards ri the end of each turn in f. We then seek the optimal policy parameters maximizing the quality of the solution represented by P,
2

---

Published in Transactions on Machine Learning Research (02/2026)
arg max B Bm- [((fm)]
(2)
0 mo
Let us define a self-improvement task as a tuple comprising an imitial task m, a partial history r², and a self- improvement budget K, with the objective of maximizing the quality of the final solution iterate Im reached via K improvement steps per turn starting from TÝ. We further define recurrent task a task for which the output of one instance can serve the input parameter define another instance. Self-mmprovement tasks are recurrent tasks.
2.2 Group-Relative Policy Optimization
We search for the optimal LLM parameters in Equation i using Group-Relative Policy Optimiza- tion (GRPO) (Shao et al. 2024). GRPO is an online policy-gradient algorithm that optimizes clipped advantage objective based on that used in PPO. Rather than computing the baseline term using a learned value ionction as in PPO (Schulman et al. 2017), GRPO uses a group of G Monte-Carlo rollouts, {o) for each initial prompt m to estimate this baseline. We use GRPO for its strong empirical performance and reduced resource requirements compared to PPO from dropping the learned value function, typically a second copy of the LLM. Let r be the vector of returns for each rollout, where r; is the return for rollout og. Within each group, the advantage Ai, for token index t in rollout og
A, T; mean(r)
(3)
std(r)
where the same advantage is assigned to every completion token index t. GRPO makes use of KL regular- ization term constraining to a reference policy 0ret, initialized with the initial parameters, and updated to the latest 0 with a - a decay factor every M training iterations. The full GRPO objective is then
lod|
1
J(0) EmnM, (Tu))
min Aia, clip(pa,, 1- 1 1+ Ar,] DKI. Trod]
(4)
G
Oi
where Fold is the set of parameters used to generate the rollouts, > 0 is the clip threshold, B > 0 is the KL coefficient, and the importance sampling coefficient for og at token index is
To ( m, Oi,ct) Pi.
(5)
Told Oi. m, 0i,ct)
3 Task diversity improves learnability
A key observation motivating EXIT is that the learnability of individual tasks can be improved by training on additional tasks with related struc- cannot directly
Single task
+2 tasks
Normalized score
ture. In Figure we see learn the ML.E-bench task random-actss-o-ppiza, while simultancously training on two additional MLEEbench tasks enables learning. Cross- task transfer is popularly exploited by methods like domain randomiza- tion (Toiin et al. 2017 Mehta et al. 2020), where training distribution over varied task instances is sampled to improve task diversity, and cur-
200
400
600
800
riculum learning methods, which actively sample the lask space. The specific approach used for sampling task instances defines an exploration stratogy over the task space. The recurrence in self-improvement tasks introduces a natural source of related tasks, as each solution iterate itself serve task instance.
Training iterations
Figure 1: Simultancously training with other tasks makes random- acts-of- acts-of-pizza learnable.
3

---

Published in Transactions on Machine Learning Research (02/2026)
Update start set with
Improve
0.8
0.8
0.8
S = 0.8
Sample initial trajectories from start
Episode
Initial
request
Turn
Turn
$ 0.1
Episode
Episode
Turn 3
Turn
0.3
0.3
Diverge
Overview of EXIT strategics. Each episode samples new task (at turn 0) or seleets a partial turn history from a previous episode as starting point for self-iteration (either sell-improvement divergence). Partial histories sampled by prioritizing those that led to higher GRPO group return variance.
Figure 2:
Our approach, ExIT, exploits this fact by actively exploring across the history of solution iterates, both by actively sampling promising previous iterates for further itcration and by promoting iteration steps to output novel solutions. In this way; ExTT seeks to produce more robust. self-improvemmnt abilities via cross-task transfer across potentially open-ended set of self-generated starting solutions.
4 Exploratory Iteration (Exlt)
We now define family of exploration methods, which we refer to as Exploratory Iteration (ExIT) strategies. These methods directly exploit the recur- rent structure of self-imprrvement tasks to actively generate and sample new task instances. Impor- tantly, EXIT strategies seek to train the model for K-step self-imprvvement using only single-turn self- improvement tasks. This simplification is achieved by dynamically sampling starting solution ilerates from running buffer of the most informative so- lution itcrates so far reached during training, al- lowing the single-step iteration tasks to accumulate to an effectively arbitrary depth during training. By decomposing extended self-imprveement chains into one-step iteration tasks, this approach enables GRPO to credit-assign dense reward to each indi- vidual self-improvement step, while naturally presenting each self self-immpoveeent task instance in an autocur- riculum of improvement steps of progressively greater depth.
A Algorithm 1: Exploratory Iteration (ExIT)
Init policy and an empty task buffer B of size N while training do
if [B] 2 Bmin and random() < then
Sample instance (m,, k; of.) ~
Sample partial history nk for turn t7 S
Use instance me' <- (m, K + 1,
else
Sample base task m' ~ M end
Self-iterate from m' to produce m+
Evaluate score S for group rollouts of m and update using these rollouts Update B with m and mí with score S
end
Concretely, new task instances are generated via two kinds transformations, each acting on a task history of up k self-iteration steps on the response for turn for task instance m. Here, 7f corresponds to the history ending with the initial response for turn & (without self-improvement)
Sample task m' with turn history from the self- improvemenn task buffer B. Then, sample random turn prefix containing the first t- < f turns from тr, ending with the k'th itcrate of the response for turn 1_ Use this partial history 7K to create a new self-improvement task m = (m, k2' + +,,7())
Selection:
Given a turn history TA, take a self-iteration step from the last-turn response, resulting in an k+1 extended, complete history T Expansion corresponds sclf-improvement task m (m, 1, 7/
Expansion:
Let A = G(+ G(T ) be the difference in quality measure between succossive iterates. Under EXIT, the reward for self-iteration steps is max (0, A/A./Amann A/Amax); where Amax the largest possible improvement from
4

---

Published in Transactions on Machine Learning Research (02/2026)
the previous iterate. The reward for base tasks remains the same. While GRPO already normalizes reward scales, we apply this explicit normalization to ensure variance-based learnability scores remain comparable.
At the start of each training episode, the task instance is first sampled via selection over the task buffer with probability p if the task buffer is above a minimum size, and otherwise uniformly at random from the base task space M. The one-step rollout from the sampled task instance then naturally performs expansion, and we subsequently update the task buffer with the new self-improvement instance resulting from expansion.
In practice, when updating the task buffer with new instance with history Tf, we precompute and insert all instances corresponding to the per-turn partial histories of r. that can be sampled via selection, when performing this buffer update. To limit the size of the task buffer, we take the simple approach of assigning a priority score to each lask instance and keeping only the top N such partial histories with the highest score. In this work, we use a learnability score, discussed in Section [.1 We describe extensions of the expansion step to promote exploration pseudocode for our approach, simplified for the case of single rollout per training iteration, is provided in Algorithm I Our self-iteration prompt templates are provided in Appendix
Section 4.2 The general ExIT approach is depicted in Figure / and the
4.1 Selecting for learning potential
To enforce a finite capacity for the lask bulfer and focus training on only the most useful task instances, we keep only the top N task instances based on learning potential, measured by the variance of the final quality measure (e.g. total return) of cach rollout in a GRPO group: 5 var(r), where the quality measure is rescaled to ensure all values are in the range [0, 1]. The variance metric extends the learnability measure studied as a prioritized sampling metric for binary success outcomes {Vinyals et al.) 2019; Rutherford et al., 2024) to general scalar outcomes. In deterministic task instances (such as all tasks considered in this work), high variance in outcomes indicates the policy often succeeds and fails, and thus the potential for improvement (i.e. learning). We leave modifications of variance-based learnability metrics for stochastic environments to future work. As this lcarnability metric is simply function of GRPO's group rollouts, using it for prioritized sampling of training instances results in a eewpvvlleeeiialenl...oo autocurriculum method.
When updating the buffer with a self-iteration task m based on expanding a previous instance m', we do not have a Icarnability score for m In practice, we initialize m with the same score as the empirical score for m', with the assumption that higher-variance starting points will tend to lead to subsequent high- variance points and vice versa. For example, if the policy can always succeed on self-iteratio task m', its learnability will be 0, which is the expected score assignment for m*, which starts from the correct solution.
If the task buffer is at capacity (i.e. |B] = N); lask instance is only inserted into the buffer if its score is greater than or equal to the instance with the lowest score in the buffer, which is replaced by the inserted instance. When sampling training instances from the buffer, the i-th instance in the buffer with score S., for < N, is assigned a probability mass equal to exp(S,x)/ ) exp(S,k)) where k is the inverse temperature.
4.2 Expanding for divergence
As EXIT accumulates new self-improvement tasks based on the model's previous responses, the resulting task diversity strongly depends on the diversity of the model's outputs. To counteract RI.'s tendency to reduce output diversity, we incorporate directly diversity-seeking components:
Divergent improvements. With probability Pdiv, self-iteration step becomes relf-divergenee step, whereby the policy is prompted to improve on a previous solution while diverging significantly from it (See Appendix B for the self-divergence prompt). We find divergence steps to induce meaningfully different responses from the model, resulting in increased task-space coverage when integrated into an ExIr strategy.
Multiplicative diversity bonus. Given an embedding space and a GRPO rollout group, compute a diversity score d, for the solution produced by the i-th rollout as the solution's embedding distance to the group centroid embedding e, normalized by the range of distances within the group:
5

---

Published in Transactions on Machine Learning Research (02/2026)
e - 2
d; (
(6)
ming (lee ell)
Following the approach in Chung et al. group-relative advantage defined in Equation 3 thereby relatively upweighting more successful trajectorics
(025)) , these diversity scores serve as coefficients that multiply each
and downweighring por-performing trajectories by how much they diverge relative to the group.
5 Experiment setting and results
We investigate the effectiveness of ExTT at inducing inference-time self-improvement behavior in several task domains, which together test the benefits of ExIT in improving test-time self-improvement in a variety of inference scttings. These domains include mathematical reasoning (a standard, singlo-turn sctting), tool-use (a multi-turn setting), and machine learning engineering lasks based on real Kaggle competitions (a setting where the LLM is typically run within a search scaffold).
5.1 Task domains
Competition math. We train on a randomly-sampled 1280-problem subset of an open dataset of compe-
tition math problems (Luo al. 2025 Li et al. 2024) and evaluate models on several held-out test sets via math-verify (Kydlččkk] 2025|: MATH500 (Hendrycks et al. 2021), AMC12 2022/2023, AIME 2024/2025, the Minerva test split (Lewkowycz et al., 2022), and the text-only samples from the OlympiadBench test split (He et al. 2024). We specifically make use of this smaller training subset (epoched after every 10 GRPO iterations) in order to Lest. the impact of ExIT in augmenting the training set with additional task instances.
Multi-turn function calling. The multi-turn celli--turn-bar base split of the Berkeley Function Calling Leaderboard (BFCLv3) benchmark (Patil et al. (025)) provides a set of 200 tasks simulating multi-turn user interactions that require the LLM assistant to make a series of function calls based on API modules spanning domains including travel booking, stock price analysis, and file system management. We split these tasks into equal- sized train and test splits of 100 task instances, using stratified sampling to ensure each split shares similar fraction of instances of different total turn counts and API sets. Following the modified environment logic in /Zhuang et al. (2025), a trajectory is considered correct if both 1) its sequence of function calls leads to the final ground-trute state, and 2) the ground-trute sequence of function calls is a subset of the sequence. During RL fine-tuning, this correctness check serves as the verified reward function. This task allows us to evaluate ExIr's impact on a natively multi-turn task.
ML engincering. Coding naturally benefits from inference-time seff-improvement, where it takes the form of debugging and optimization of previously outpul code solutions. We evaluate ExIr's impact on improving coding performance in MLE-bench (Chan et al. 2024), based on historical Kaggle competitions, by adapting this benchmark into an RL environment. The statc-of-tho-art solutions in MLE-bench roly on a search scaffold- a program that repeatedly calls the LLM in order to arrive at a final solution, by iterating on intermediate outputs (Jiang et al., [2025] Liu et al. 2025a; Toledo et al. 2025 Zhao et al. (025b)). We make use of this more challenging domain as a testbed for the ability of ExlT strategies improve model performance within scarch scaffold for code iteration (with error traces execution feedback). We train three lasks: At test time, we run the model within a greedy-scarch scaffold on held-out tasks: jigaaw-toxiccoommntt clasifiicttinn-challnngg
and random-acts-of-piie
and deriallcactus-ideetffcclllll
We provide additional experiment details and our choice of hyperparameters in Appendix
5.2 Baselines and ablations
In each task domain, we compare models trained via EXIT and its ablations against the original model before and after standard GRPO fine-tuning, via a heavily-modified fork of open-instruct (Wng et al., 2023). Specifically, we look at ablated variants of the "full" ExIT strategy: using only the learnability curriculum (i.e. only the selection transformation), additionally using self-improvement steps (IMPROVE),
6

---

Published in Transactions on Machine Learning Research (02/2026)
Method
Math
All (%)
A16
Tool-use (%)
A
MLE-bench (%)
Ai6
Base model
17.410.7
-0.4
60.540.4
+0.1
4.2+1.
+2.4
GRPO
18.7+0.4
+1.1
73.5+1.5
-0.5
48.0+0.7
+9.1
+ curriculum
18.8+0.77
+0.9
75.41.22
+1.0
53.0+9..7
+11.9
+ self-improve step (IMPROVE)
19.6#0.7
+1.2
75.5+1.2
+3.4
47.8+2.5
+9.4
+ divergent step (DIVERGE)
20.1+0.77
+1.6
76.822.2
+0.6
57.3+7.3
+10.1
+ diversity bonus (FULL ExIr)
20.4100.5
+.2.0
76.411.0
+1.2
58.6.12.5
+8.4
Table 1: Evaluation of ExIT against its ablations and the GRPO bascline held-out task instances. Math results are averaged over all test splits. All results are the mean and std of performance after 16 improvement steps over 3 training runs, alongside the net percentage of improvement (Ax) between initial response and final response after K self-improvement steps.
additionally using self-divergence sleps (DIVERGE), and the full method additionally using the embedding- based diversity bonus (full ExIT). For the math domain, we use 1-t---...------tttttte (Meta Al, 2024). its baseline performance leaves considerable room for improvement. We compute embeddings for the diversity bonus using a BERT model fine-tuned on an academic math dataset (Devlin et al. 2019 Steinfeldi & Mihaljević, 2024). For the more challenging tool-use and ML engineering domains, we usee the best model at the 7B parameter scale from the Qwen2.5 series (Team 2024 for each task. For tool-use, we fine-tune Qwen2.5-775 Insttruct,,, and for ML enginecring, both tool-use responses and code solutions using the 400M parameter CodeXEmbed model (Liu el al. 2024
(Guo et al. 2025). We embed
5.3 Self-improvement at inference time
Initial model
GRPO
Curriculum
Improve
Diverge
Exit
Math (mean acc. over all evals)
Tool-use (first turn acc)
Tool-use (total return)
0.20
0.75
1.6
average@@
1.5
0.19
0.70
1.4
0.65
1.3
0.60
0
4
8
16
0
2
4
0
improvement steps)
improvement steps)
improvement steps)
Figure 3: Left: Mean accuracy on all math test splits. Center: Mean first-turn accuracy on the multi-turn tool-use test split. Right: Mean total task return the multi-turn tool-use test split. Results are avg@8 values across 3 training runs per method.
We measure the ability of a model to iteratively self-improve its responses at inference-time, with budget of K self-imprrvement steps. For each turn 1, at self-improvement step 0 < h S K, the model is prompted to provide an improved response given the original request (i.e. from the user) and the model's latest response iterates from all previous turns up to and inchuding the current turn. The self-improvement results in Table D and Figures 3 show that across task domains, ExIr strategies directly using exploration mochanisms (DIVERGE's prompt-based exploration and the full ExIT stratogy's 's use of an exploration bonus) consistently improve the model's ability to self-improve its solutions, with respect to the original model and after fine-tuning with GRPO. Notably, ExIT strategies also result in higher initial solution quality, while the curriculum-only bascline tends to achieve comparable performance with standard GRPO, indicating that augmented task-set diversity via selection and expansion transforms can result in more robust policies. We provide example self-iteration steps encountered during training in Appendix D
In Figure 4 we see that across the held-out math splits, IMPROVE, DIVERGE, and the full ExIT method all lead to a higher net number of successfully corrected solutions over 16 self-improvement steps, accumulating over 170 successful corrections across all held-out problems. On the majority of test splits (all except. ATME,
7

---

Published in Transactions on Machine Learning Research (02/2026)
Llama-3. L 2-3B-lnstruct
GRPO
Curriculum
Improve
Diverge
Exit
MATH500
AMC12 2022
AMC12 2023
AIME 2024
150
150
100
10
225
# corrections
75
12
4
16
16
12
16
AIME 2025
Minerva
OlympiadBench
1.0
25
0.5
-20
0.0
-30
1 12
16
12 4
8
16
12
16
12
8 16
improvement steps (k)
Figure 4: Net corrections across held-out math splits, computed over 8 samples per problem per checkpoint, averaged over training runs per method (and an equivalent # samples/ problem for Llama-3.2 3.2-3B- 22-3B-Instruct)
camples/problem
2025, where all methods fail to mcaningfully self-improve), EXIT and DIVERGE continue to net additional corrections even after 16 self-improvement steps, despite the mean self-iteration depth remaining under two steps throughout training (see Figure
In the multi-turn tool-use domain, we find EXIT stratogy policies lead greater performance, both before and after sell-improvement steps compared to GRPO and the initial model, which both achieve worse performance after self-iterating over K 4 steps. Notably, not always correspond to the ranking total return the latter function of all-turn accuracy with the curriculum-only bascline performing worse than the GRPO bascline in all-turn return, while performing better on first-lurn accuracy. We see that the additional exploration mechanisms employed by DIVERGE and the full EXIT method enable the curriculum- based approach to exceed the performance of GRPO.
that performance ranking first-turn accuracy does
In MLE-bench, we find that both DIVERGE and the full EXIT method lead to the highest normalized performance across train and test. tasks, while IMPROVE and the curriculum-only baseline attain similar performance the GRPO baseline. This result highlights how even standard GRPO fine-tuning set of MLE-bench tasks can transfer to held-out tasks. Our approach then further improves the quality of solutions discovered by the fine-tuned model within the multi-step greedy-search scaffold. This result shows that ExIT can be an efficient approach for fine-tuning LLMs for use extended multi-step scaffolds at test-time, by producing and lcarning from purely single-step iterations at training time.
small
Appendix C further reports how success rate evolves per task split for the math and bench domains.
5.4 Emergent complexity during training
By selectively sampling task instances with the highest group return variance, ExIT strategies induce an adaptive curriculum of task instances that prioritizes those that are most learnable for the model at any time, without need for an externally-speciffed task ordering. Figures 5 and G show the curriculum-driven evolution of various task complexity metrics that intuitively align with task difficulty. Across domains, we find ExIT strategies tend to lead to greater complexity metrics than the curriculun-only baseline, with the direct exploration variants (DIVERGE and full ExIT) most often achieving the highest values. In the math domain, see that the mean number of ground-truth solution steps of sampled instances increases over time, indicating the emergent curriculum naturally drives towards more challenging problems over the course of training. Similarly, in the multi-turn tool-use domain, the mean number of ground-truth tool- call steps of sampled task instances increases over time. Here we also see that the mean starting turn of replay trajectory progressively incrcases during training, indicating that as the model bocomes more adept handling tool calling earlier steps, the frontier of learnability moves deeper into the solution sequence. While ExTr variants also increase sample depth- the number of self-iteaatio steps already taken
8

---

Published in Transactions on Machine Learning Research (02/2026)
Initial model
GRPO
Curriculum
Improve
Diverge
Exit
MLE-bench (training dynamics)
MLE-bench competitions
0.6
Sample from iteration
0.5
Sample depth
400
0.4
4-
0.3
200
0.2
2
0.1
400
600
800
200 400 600
800
0.0
0
14 k (# improvement steps)
4
8
10
12
16
Training iterations
Figure 5: Left: Emergent curriculum over the sampled history's 's recency and starting depth during ML.F-
bench training. Right: Normalized MLEEbench scores achieved by each method over all train and test tasks via increasing greedy-search budgets (mean over 3 training runs).
Curriculum
Improve
Diverge
Exit
Competition
Tool-use
gold solution steps
17
Sumple from iteration
Sample from from Itaration
20
2.0
Sample depth
1.8
1.6
Step
20
1.4
10-
1.4
4.0
12-
13
1.0
1.0
30
100
50
100
100
200
200
400
200
460
Training iterations
Training iterations
Figure 6: The evolution various task instance properties during training in the math domain (left) and the multi-turn tool-use domain (right). Results are based on mean and std over 3 training runs.
on the starting response-men more than the curriculum-only baseline, we find that the rank ordering among methods is inconsistent. This may be due how self-iteration does not guarantee subsequent iterates deviate substantially from previous iterates in terms of difficulty for further improvement.
5.5 Exploratory iteration increases task diversity
We now analyze the diversity of task instances sampled by each method during training. Figure shows the number of distinct training task instances relative to the base training set used by GRPO. In the tool-use domain, we count the number of unique task prompts, corresponding to the contents of the last user message in the message history at the start of each episode. For MLE-bench, we count the number of unique starting code solutions. We find that the curriculum-only baseline heavily shrinks the number of distinet training instances encountered, indicating that prioritized sampling leads to high repetition of training instances. This reduction of task diversity may explain the relative underperformance of this bascling compared to ExIT variants, whose self-iteration steps have the effect of recovering a substantial amount of this lost. diversily. In the case of the full ExIr, we see this improved diversity under a curriculum corresponds to improved performance on held-out instances. In the basc training distribution used by GRPO, MLE-bbnnh tasks all begin with the same empty Python script template, so we see drastic increase in number distinct starting code solutions under ExIT strategies.
Method
Cosine dist. (t)
L2 dist. (t)
Uniform
N/A
N/A
Curriculum
N/A
N/A
Improve Diverge
0.10
0.05
0.11
0.06
ExIr
0.13
0.07
Table 2: Divergence metrics for training instances encountered for each method in MLE-bench tasks.
The UMAP (MeInnes et al. 2018) in Figure 7 further highlights the differences between the task sets induced
9

---

Published in Transactions on Machine Learning Research (02/2026)
Tool-use
MLE-bench
Exit
Exit
Diverge-
Diverge
Improve
Improve
Curriculum
Curriculum
GRPO
GRPO
0.0
0.2
0.4
0.6
0.8
1.0
50
100
150
Relative size of training set
Relative size of training set
Figure 7: Relative number of distinct training task instances under each method for the multi-turn tool-use domain (left) and MLEE-bench (middle). The UMAP projection of the distinct starting programs observed during MLE-bench training shows ExIT strategies increase the diversity of the training distribution (right).
by ExIT variants and the base task set a single point in the CodeXEmbed embedding space. As reported in Table 2 the directly novelty-seeking ExIT variants lead to greater mcan pairwise cosine and L2 distances among discovered task instances in the same embedding space, with the full ExIT attaining the greatest mcan pairwise distances.
6 Related works
Inference-time self-improvement. Prior works consider variants of the inference-time self-improvement setting. [Shinn et al. (2023) introduces prompting-basee based approach for adapting based previous trials of the same task with explicit feedback. Kumar et al. (2024) study directly training for single step of self- improvement with optional feedback, which is a special case of the more general k-step setting considered in this work. They train a model to self-correct an initial previous solution in two training stages, with the first focusing on generating diverse initial solutions, and the second on self-correction. In contrast, ExIT does not require staged training and trains only on single-step iterations rather than both iterations together, while also generalizing beyond two self-imponn self-improvement steps at test-time. Bauer et al. (2023). directly train RL agent to reflect multiple trials in procechurally-generate game. Importantly, ExIT performs self-iteration per timestep (i.e. turn), more granular contextual unit of iteration than the full episodic trial considered in these previous works. Per-timestep self-iteration is often more suitable gramularity for LLM tasks, in which each timestep typically maps to a full turn of meaningful interaction. EXIT strategies can thus be complementary to the trial-based self-improvement setting. Another complementary rescarch direction focuses on directly training improved verifiers for providing leedback for sell-improvement (Gou
et al. 2023 Dou al. 2024 Yuan & Xie 2025).
Task-space exploration. EXIT can be viewed as form of Prioritized Level Replay Jiang et al. (2021b PLR) that performs task-exploration at the turn level, by treating self-iteration from the response at each previous turn as task instance in itself. Our task expansions can be seen as a kind of "mutation" operator on the task buffer, likening our approach to a version of PLR that explores the task space via evolutionary search (Parker-Holder et al. 2022p. By adapting these ideas to self-improvement decision processes, ExIT demonstrates how task and trajectory-level exploration naturally coincide this setting. Outside of the self-improvement setting, previous methods consider reset-based exploration a trajectory level (Ecoffet et al. 2019 Kazemnejad et al.) 2024 Foster & Foerster 2025 Zheng et al. 2025). Unlike these approaches, ExIT modifies the specific task from the reset state (e.g. transforming it into self-improvement
self- divergence task), while remaining compute-cquivallnn. In contrast to prior methods for dynamic training- task generation (Sukhbbatar et. al.] 2017 Wang el al., 2019 2020 Dennis et al.) 2020 Zhang et. al.) 2023 Faldor et al.) 2024] Zhao et al. 2025a). EXIT forgoes a generator module and pursues open-ended task-space exploration (Jiang et al. 2023) by upcycling the learning agent's own trajectorics into new task instances. The novelty bonus used by the full EXIT strategy adapts the approach in Chung et al. (2025) to GRPO. Our results show such bonuses improve not just output diversity, but also task performance.
10

---

Published in Transactions on Machine Learning Research (02/2026)
7 Limitations and future work
Our experiments feature maximum training turn-history length of ten (in the tool-use domain). In do- mains like math and MLE-bench, where the responses tend to feature longer outputs, multi-turn history can quickly exhaust the available context length. Extending ExIT to longer multi-turn training contexts serves as important future work, c.g. by compacting the context or using tool-based retricval over the turn history. Though training FLOPS are still concentrated on base LLM computations, the use of an embedding model to compute response diversity adds extra overhead, making full ExIT only approximately compute- equivalent though with minimal impact on training time, as this extra computation parallelized with the backward pass. Evaluating full ExIT under cheaper metrics, e.g. based on edit-distance, would be valuable follow-up work. Moreover, our experiments use ExIr to drive parameter-space optimization, when the same strategies can also be applied to optimize the system prompt or task prompt template to induce greater seffiimprvemment capabilitics. Lastly, our experiments cach focus on a single task domain and make use of an older generation of open models. The improved base reasoning capabilities of more recent models may enable them to benefit more from ExIT -style fine-tuning and exhibit greater transfer to unseen tasks. We leave application of EXIT the latest base models for future work.
8 Conclusion
We introduced ExIT, an approach for RL fine-tuning an LLM for multi-step inference-times self- ieff-improveeent via purely single-step self- infrrroreeeentt tasks. The core mechanisms behind ExIT adapt ideas from autocur- riculum methods and trajectory-leva exploration for RL the K-step self iepr-immrveeeett problem sctting, Importantly, seen in our MLE-bench experiments, this problem setting maps closely to the common paradigm of performing LLM inference within search scaffold. Our results show that ExIT indeed result in improved downstream performance when the fine-tuned model is used to drive search scaffold. Improvements to the self-iteration prompts (i.e. improve and diverge) as well as extensions of this set of operations to additional search-specific actions may further improve the impact of our approach in down- stream scaffold- baafold-based applications. We believe self-supervised turn-level exploration, as embodied by the ExIT design, is promising direction for greatly enhancing the efficacy of RL fnn-tuning for LLMs.
References
Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley- Schmieg, Michael Chang, Natalie Clay, Adrian Collister, et al. Human-timescale adaptation in an open- ended task space. arXiv preprint arXiv:2301.0P 07600, 2023.
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, al. MLF-bench: Evaluating machine learning agents machine learning engineering. arXiv preprint arXic:24 10.00095, 2024.
John Joon Young Chung, Vishakh Padmakumar, Melissa Roemmele, Yuqian Sun, and Max Kreminski. Mod- ifying large language model post-training for diverse creative writing. arXiv preprint arXiv:2503.1112 2025.
Michael Dennis, Natasha Jaques, Fugene Vinitsky, Alexandre Bayen, Stuart. Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems, 04:10099-13066 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short pupers), pp. 4171 4186, 2019.
Zi- Yi Dou, Cheng-Fu Yang, Xueqing Wu, Kai- Wei Chang, and Nanyun Peng. Re-ReST: Reffection-reinforeedd self-training for language agents. ar Xio preprint or arXin:2106. 01660.1445 2024.
11

---

Published in Transactions on Machine Learning Research (02/2026)
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-Explore: new approach for hard-exploration problems. arXiv preprint or Xrxin:1900 10995, 2019.
Maxence Faldor, Jenny Zhang, Antoine Cully, and Jeff Clune. OMNI-EPIC: Open-endedness via mod- els of human notions of interestingness with environments programmed in code. arXiv preprint arXiv:2405.1588 15568, 2024.
Thomas Foster and Jakob Foerster. Learning reason at the frontier of learnability. arXiv preprint arXiv:2508.12272 2025.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. orXiv preprinl arXiv:2508.01307, 2025.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tooe-interactive critiquing. arXie preprint 1rXv:23055.1178, 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Poiyi Wang, Xiao Bi, et al. Deepsock-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. Xixi preprint arXiv:2501. 12244, 2025.
Alexander Gurung and Mirella Lapata. Learning to reason for long-form story generation. arXiv preprint arXiv:2503.2288 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. OlympiadBench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint
14008 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. ar preprint arXiv:2103.0385 2021.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed Ei-Kishky, Aiden Low, Alec Helyar, Alek sander Madry, Alex Beutel, Alex Carney, et al. OpenAl ol system card. arXie preprint 2rXiv:2412.1679) 2024.
Minqi Jiang, Michael Dennis, Jack Parker-Holdes Jakob Foerster, Edward Grefenstette, and Tim Rock- täschel. Replay-guided adversarial environment design. Advances in Neural Information Processing Sys- tems, 34:1884-1897, 2021a.
Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In International Conference on Machine Learning, pp. 4940 4950. PMLR, 2021b.
Minqi Jiang, Tim Rocktäschel, and Edward Grefenstette. General intelligence requires rethinking explo- ration. Royal Society Open Science, 2330539,, 2023.
Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Jacenko, and Yuxiang Wu. AIDE: Al-driven exploration in the space of code. arXiv preprint arXio:2502.13138 2025.
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. VinePPO: Unlocking RL potential for LLM reasoning through refined credit assignment. 2024.
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roclofs, et al. Training language models to self-correct via reinforcement learning. ar preprind arXiv:2409.1220 12917, 2024.
Hynek Kydliček. Math-verify: Math verification library. 2025. Version 0.6.1, Apache-2.0 license.
12

---

Published in Transactions on Machine Learning Research (02/2026)
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Facze Brahman, Lester James V Miranda, Alisa Liu, Nouha Duiri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint 0rXie:2411.15124 2024.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843 3857, 2022.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaune Lample, and Stanislas Polu. NuminaMath, 2024.
Ye Liu, Rui Meng, Shafiq Joty, Silvio Savarese, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. CodeXEm- bed: A generalist embedding model family for multiligual and multi-task code retrieval. or arXie preprint arXiv:2411.126.4 2024.
Zexi Liu, Yuzhu Cai, Xinyu Zhu, Yujie Zheng, Runkun Chen, Ying Wen, Yanfeng Wang, Siheng Chen, et al. ML-Master: Towards AI-for-AI via Integration of Exploration and Reasoning. arXiv preprint arXiv:2506.1644 16499, 2025a.
Zichen Liu, Changyu Chen, Wenjum Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding R1-Zero-like training: A critical perspective. arXiv preprinl ar Vii:2503.20788 2025b.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol-preview with a 1.5b model by scaling rl, 2025. Notion Blog.
Leland Mclnnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint ar Xiv: 03426, 2018
Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J Pal, and Liam Paull. Active domain random- ization. In Conference Robot Learning, pp. 1162-1176. PMLR, 2020.
Meta AT. Llama 3.2: Revolutionizing edge ai and vision with open, customiz- able models. Meta Al Blog, September 2024.
URL https://ai meta. com/blog/ 1lama-3-2-aa 33-2-connect-2222-viee w--- ede-mbbiee--deii...e. Blog post from Connect 2024: Vision,
Edge & Mobile Devices.
Jack Parker- Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefen- stette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. In International Conference Machine Learning, pp. 17473 17498. PMLR, 2022.
Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Jie Ji, Fanjia Yan, Vishnu Suresh, lon Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Forty-second International Conference Machine Learning, 2025.
Alexander Ruthorford, Michael Beukman, Timon Willi, Bruno Lacerda, Nick Hawes, and Jakob Focrster. No regrets: Investigating and improving regret approximations for curriculum discovery. Advances in Newral Information Processing Systems, 37:16071-1611 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimiza- tion algorithms. XrXiv preprint Xiv: wtx:::1000000000000 707.06947, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint SKi:22402.03300 2024.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Lan- guage agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36: 8634-880 8652, 2023.
13

---

Published in Transactions on Machine Learning Research (02/2026)
Christian Steinfeldt and Helena Mihaljevié. Evaluation and domain adaptation of similarity models for short mathematical texts. In International Conference on Intelligent Compuler Mathemalics, pp. 241- 260. Springer, 2024.
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. In- trinsic motivation and automatic curricula via asymmetric sclf-play. arXiv preprint arXiu: 1 703.05507 2017.
Qwen Team. Qwen2 technical report. ar preprint arXie:2/07.1067) 2024.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23-30 20- IEEE, 2017.
Edan Toledo, Karen Hambardzumyan, Martin Josifoski, Rishi Hazra, Nicolas Baldwin, Alexis Audran-Reiss, Michael Kuchník, Despoina Magka, Minqi Jiang, Alisia Maria Lupidi, et al. AT Research Agents for Ma- chine Learning: Search, Exploration, and Generalization in MLE-bench. arXiv preprint arXiv:2507.0255) 02554,, 2025.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaöl Mathieu, Andrew Dudzik, Junyoung Chung, David Н Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. nalure, 575(7782)3350 -544, 2019.
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. POET: Open-ended coevolution of environ- ments and their optimized solutions. In Proceedings of the genetic and evolutionary computation conference, pp. 142-151, 2019.
Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeffrey Clune, and Kenneth Stanley. Enhanced POET: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In International conference on machine learning, pp. 9940-9951. PMLR, 2020.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Processing Systems, 36:74764- 74786, 2023.
Fang Wu, Weihao Xuan, Ximing Lu, Zaid Harchaoui, and Yejin Choi. The Invisible Leash: Why RLVR May Not Escape Its Origin. arXiv preprint arXiv:2507.148. 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. DAPO: An open-source LLM reinforcement learning system at scale. arXiv preprint Xix::25003. 14476, 2025.
Yurun Yuan and Tengyang Xic. Reinforce LLM Reasoning through Multi-Agent Reflection. arXiv preprint arXiv:2506. S05.:9900600879,,, 2025.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476 114881, 2022.
Jenny Zhang, Joel Lehman, Kenneth Stanley, and Jeff Clune. OMNI: Open-endedness via models of human notions of interestingness. arXiv preprint Xiv:22306.01 01711, 2023.
Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang. Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. arXic preprint Xix::22505. 03335, 2025a.
Bingchen Zhao, Despoina Magka, Minqi Jiang, Xian Li, Roberta Raileamu, Tatiana Shavrina, Jean- Christophe Gagnon-Audet, Kelvin Niu, Shagun Sodhani, Michael Shvartsman, al. The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements, arXiv preprint or Xrxvv:5006. 22419, 2025b.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Tianyu Zheng, Tianshun Xing, Qingshui Gu, Taoran Liang, Xingwei Qu, Xin Zhou, Yizhi Li, Zhoufutu Wen, Chenghua Lin, Wenhao Huang, et al. First return, entropy-eliciting explore. or preprinl arXiv:2507.00011 2025.
Xiangxin Zhou, Zichen Liu, Anya Sims, Haonan Wang, Tianyu Pang, Chongxuan Li, Liang Wang. Min Lin, and Chao Du. Reinforcing general reasoning without verifiers. arXiv preprint arXie:2565.2149, 2025.
Richard Zhuang, Trung Vu, Alex Dimakis, and Maheswaran Sathiamoorthy, Improving multi-turn tool use with reinforcement learning. https:// htpp::/www..n http://wwwbbesppooe www.bespokccaas..t http///og////pp///////// multiig-mmuiiiii turn tool use-with- reinforcement-leamiinn,, 2025. Accessed: 2025-04-17.
15

---

Published in Transactions on Machine Learning Research (02/2026)
A Additional experiment details
A.1 Reinforcement learning system
Our GRPO trainer is based on a heavily-modiffed fork of GRPO updates, with separate training and inference ranks. In our modified traincr, training and evaluation rollouts are conducted via lask-specific multi-turn RL environments. LLM inference performed via asyn- chronous requests a vLLM instance (A)y of the model, whose weights are updated to the latest training weights after every GRPO training iteration.
which makes use of asynchronous
A.2 Competition math
Our RL training system represents each math task instance as a single-turn RL environment. instance. The reward function for cach instance is based on the outcome of comparing the extracted LLM solution with the ground-truth answer using math-verify (Kydliček] 2025). The reward is for correct answer, and otherwise. We use the same verification-based reward for self-improvement and self-divergence steps, effectively rewarding self-iteration responses within GRPO group for improving an incorrect solution (or maintaining correct solution) and penalizing responses that lead to incorrect solution.
A.3 Multi-turn tool-use
We follow the set up in Zhuang et al. 2025 adapting the BFCLv3 multi-turn- base task format into a multi- turn RL environment. Each task requires the LLM to fulfill a sequence of possibly multiple user requests by executing sequence of tool calls, with the valid function signatures provided as JSON schemas in the system prompt. At each turn the LLM policy either output tool-call in JSON format, surrounded by <tool> lags, <TASK_ BUSS_FINISSS FINISHED> to indicate the current user request has been fulfilled, or <TASK ERROR> to indicate that there is unrecoverable error. We configure vLLM to stop inference after generating any of these kinds of responses. The LLM policy receives a reward of 1.0 for correctly outputting a sequence of tool calls that both includes the ground-truth sequence and results in the underlying program state equivalent to the expected program state after executing the ground-truth solution sequence. Like in the math domain, we also use this verification-basse reward for episodes containing self-iteration steps.
Episodes in this environment not guaranteed to share the same length, leading to the possibility of some trajectories being truncated when packing input-completion pairs into training batch. To avoid this truncation, we trained on the full rendered message history with non-assistant messages masked out. To further stabilize training against overly long responses, we perform overlong filtering (Yu et al. 2025) set tint the gradient to zero for all responses lacking any of the valid response types described above. We further (2025b) to improve learning stability.
found using the normalized objective from Liu
To limit training trajectory lengths, we only the first user request and provide the agent budget. of 10 turns. During testing, provide the agent with up to 60 turns to fulfill all user requests.
A.4 MLE-bench
Competition
Score type
Worst score
Best score
random--ctts-ofpizza
AUC
0.0
1.0
spooly-arthoo--dontffcciioo
Cross-entropy loss
1.5
1.0
AUC
0.0
1.0
nomad2018-predlieee
RMS error
1.0
0.0
AUC
0.0
1.0
acriallcactus--deenfffettto
AUC
0.0
1.0
Table 3: Score ranges used for computing normalized scores (rewards) each MLE-bench competition.
16

---

Published in Transactions on Machine Learning Research (02/2026)
We adapt the Kaggle-based ML engineering tasks from Chan et al. [2024] into single-turn RL environment, parameterized by the competition ID. The task description in the prompt corresponds to the compact "obfuscated" markdown competition description. The user prompt for each task instance (i.e. competition) is shown Appendix B with a goal component varying based on if the starting code is buggy (in which case, the error trace also provided below the code block). The prompl instructs the LLM to improve this code block, which defaults to a Python script with an empty main function, with comments indicating the environment variable storing the data directory path (These paths are also described in the prompt and provided to the environment during initialization). Under ExIT strategies, the starting code block is replaced with the code solution from the previous response sampled for self-iteration.
The reward function is computed by first extracting and executing the code solution from the LLM response in a sandbox environment, which produces a submission.co csv file. We limit the maximum runtime to minutes, as solutions for the competitions we use for this study can typically be trained in just a few minutes, and this limit prevents training from stalling due the presence of inefficient scripts in a rollout batch. Then, mlebench grade executed provide score based a metric specific each competition, e.g. AUC. To compute normalized scores that reside in [0.0, 1.0], we define ranges corresponding to the worst and best score per competition and compute the reward as the normalized score as
Tworst
Tnorm
Thest
Tworst
As some competitions make use of error metrics like cross-entropy loss that are in principle unbounded, clip the worst values to reasonable finite value based on the historical submission results from human contestants for these competitions. See Table 3 for a listing of each competition score function and corresponding ranges used for reward normalization. This normalized reward is also used for self-divergence steps, while the sell-improvement reward for the k-th iterale, т computed as the normalized difference between the new solution and previous solution iterate, where F} and rk-1 are assumed be normalized as described above:
7Kk Tk-1
Timp,k = max 0,
1 - Th-1
A.5 Choice of hyperparameters
Training hyperparametess for experiments in Section aare shown in Tables 4 6 All experiments set clipping coefficient = 0.2. Except for learning rate, we use the best GRPO hyperparameters when sweeping ExIT hyperparameters. We evaluate using the default sampling configs fo each model on Hugging Face Hub.
Competition math. We validate against a held-out set of 100 problems sampled from the MATH training dataset and sweep over learning rate in [5e-7, 1e-6, 5e-6), in [0.0, 0.001, 0.01), buffer size [100, 256, 512, 1024], and self-iteration and divergence probabilities in [0.2, 0.5, 0.8).
GRPO
Curriculum
Improve
Diverge
ExIt
Learning rate Prompts per batch Rollouts prompt Epochs per batch KL coefficient, 8
1c-6
128
8
0.001
Ref. update interval, M
100
Ref. update Total buffer size Min. buffer size Self-iteration prob. Divergence prob.
1.0
512
512
512
512
128
128
128
0.5
0.5
0.5
0.2
0.2
Table 4: Hyperparameters for math experiments for each method and ablation.
17

---

Published in Transactions on Machine Learning Research (02/2026)
Multi-turn tool-use. We select hyperparameters for GRPO based 25-task validation subset held-out from the training split. The final checkpoints train on the full 100 training tasks. For ExTr methods, we select typerparameters based on those producing curricula plateauing with the highest starting step during training. treating this metric as a proxy for improved robustness over procoding steps. We sweep over learning rale in [1e-7, 5e-7, 1e-6, 5e-6], 8 in [0.0, 0.001, 0.01], bufler size [100, 320, 640), and self-iteration and divergence probabilities in [0.2, 0.5, 0.8). Choices of # epochs, M, and a are based on Zhuang al. (2025)
GRPO
Curriculum
Improve
Diverge
Exit
Learning Prompts per batch Rollouts per prompt Epochs per batch KT. coefficient,
1e-6
5e-7
5e-7
5e-7
1e-6
16
2
0.001
Ref. update interval, M
100
Ref. update Total buffer size Min. buffer size Self-iteration prob.
1.0
640
640
640
320
320
320
320
0.2
0.5
0.5
0.5
Divergence prob.
0.2
0.5
Table 5: Hyperparameters for multi-turn tool-use experiments for each method and ablation.
MLE-bench. We select hyperparameterss that maximize improvement on training tasks over 16 greedy search steps and sweep over learning rate in [5e-7, 1e-6, 5e-6], B in [0.0, 0.001, 0.01], GRPO in [0.0, 0.5, 1.0]. buffer size in [3, 24, 32, 64], minimum buffer size in [3, 8], and self-iteration and divergence probabilities in [0.2, 0.5, 0.8).
GRPO
Curriculum
Improve
Diverge
ExIt
Learning rate Prompts per batch Rollouts per prompt Epochs per batch KL coefficient, 3
le-6
4
8
1
0.001
Ref. update interval, M
100
Ref. update Total buffer size
0.5
24
32
24
Min. buffer size
8
8
3
Self-iteration prob
0.5
0.5
0.5
0.5
Divergence prob
0.2
0.5
Table 6: Hyperparameters for MLE-bench experiments for each method and ablation.
18

---

Published in Transactions on Machine Learning Research (02/2026)
B Prompts
We provide the prompt templates generic self-iteration prompls are shared across math and multi-turn tool use environments, while MLE- bench makes use of a separate prompt. format, which contains additional task-specific information required by this domain. For the MLE-bench prompt template, the self-iteration task is specified by the "goal preamble" component of the prompt. Note that in this domain, self-improvement takes the form of a debug step when the previous solution iterate has errors (which are then included below the code block as form of task feedback).
for self-iteration (self-improve and self-divergence) steps. The
Self-improve prompt
Improve your response this request:
Request: (instruction)
## Current response: (previous_reeppnss)
Self-diverge prompt
Consider your current response to this request, and provide another response that takes approach that SIGNIFICANTLY DIFFERS from the approach of the current response.
## Request:
(instruction)
## Current response:
(previous_responsse
## Your new response format:
First, provide brief aummary of the approach in the current response above. Next, state a meaningfully different approach that can be taken instead.
Finally, provide your response using this different approach.
MLE-bench prompt
You Kaggle grandmaster. Your solution to this Kaggle challenge should achieve the highest medal possible, where the medal rankings from best worst are gold, silver, and bronze. Your solution is not good enough if it does not medal.
Study the current version of solution.pry " "python (imitial_solution)) ... (ddeugginfo))
{goal -eea_bpreamble
Task description
Previews of files located at DOBLIC_RESOURCE_PAN PATH:
[previes_contentar
Write single file called solution.pp py that does the following:
19

---

Published in Transactions on Machine Learning Research (02/2026)
Loads any relevant training data described in the task deacription.
Loads any relevant test data described in the task description.
is important your code references the training and test data paths exactly as above, via these environment variables, which will already be set correctly for you. split.
Performs any arbitrary training evaluation logic on this For example, you might hold out some subset of this data for the purposes k-fold cross validation for model and hyperparameter selection.
Finally, evaluates the final model the test data and writes the results to a file in the current directory called submission.cav.
All files and directories described in the task description are located under the PUBLIC RESOURCE PATH, whose value is always available via environ eeraaggee("PPPe" ('PUBLIC RESOURCE_PATH") Besides solution.py py, you may not create any extra files or modify any other files in the directory.
Remember to define all neceasary functions before the 'if name = __ina_____ block. Your code will be run on machine with single H100 GPU.
Respond with your full working python program for this task inside triple-tick delimiters (!1)).
MLE-bench improve goal preamble
Your goal is to improve this python script to better achieve the following task:
MLE-bench debug goal preamble
goal to debug this python script, so that best achieves the following task:
MLE-bench diverge goal preamble
Your goal is to propose a new python script that improves over this python script for the following task.
Your improved script should take radically different approach to solving the problem than the one taken by the current python script.
20

---

Published in Transactions on Machine Learning Research (02/2020)
Llama-3.2-3B-Inttrue
GRPO
Curriculum
Improve
Diverge
Exit
MATH500
AMC12 2022
AMC12 2023
0.50
0,30
average
0.49
average@0
Average@8
0.28
0.48
0.26
0.188
0.47
0.24
0.17
0.46
0.45
16
k (# improvement steps)
k
AIME 2024
AIME 2025
OlympladBench
0.010
0.18
0.09
average@8
average@@
0.008
average@8
0.17
0.08
0.16
0.006
0.07
0.15
2
0.2
16
16
Minerva
0.175
average@@
0.170
0.165
0.155
012
16
Figure Net incorrect + correct improvements across all held-out math splits. Total corrections are averages computed over 8 samples per problem for each of 3 checkpoints per method, and an equivalent number of samples per problem for
C Additional experiment results
In Figure 8 we show the inference-time self- celf-improvement performance over 16 steps for cach test split in the competition math domain. Both DIVERGE and the full ExIT strategy produce improved performance across these test splits, with the exception of AIME 2025, where none of the methods perform well.
As we saw in Figure 7 ExIT strategies incrcase the effective number of distinct tasks encountered during training with respect to the curriculum-only baseline, which reduces task diversity by oversampling those task instances with higher return variance. In Table 7 embedding distances relative to the augmented task set discovered by ExIT, under the embedding space of the 400M parameter CodeXEmbed model. We see ExIT strategies are able to maintain a comparable degree of variation to the base task set while increasing the number of tasks considered for prioritized sampling.
that the base task set has slightly greater pairwise
Figure reports the mean greedy-search performance over 16 steps on the train and test competitions. Figure 10 shows geeedy-search performance per competition. We see that for training competitions, the LLM policy rapidly converges to specific solution, likely due to some degree of overfitting from directly training the task instance, while test competitions
greater differences between initial and final solution performance after 16 greedy-search steps. The initial model is unable to effectively self-improve, and slandard GRPO fine-tuning already induces some degree of self- infrommproe.t.. ExIT strategies directly using step-level exploration further onhance the model's ability to self-improve.
21

---

Published in Transactions on Machine Learning Research (02/2026)
Method
Cosine dist. (f)
L.2 dist. (f)
Uniform
0.56
0.33
Curriculum
0.56
0.32
Improve
0.52
0.30
Diverge
0.48
0.28
ExIT
0.50
0.29
Table 7: Mean pairwise distances computed over the training task instances used by each method.
Initial model
GRPO
Curriculum
Improve
Diverge
Exit
MLE-bench competitions (train)
MLE-bench competitions (test)
0.4
0.6
average@16
average@10
0.3
0.4
0.2
0.2
0.1
0.0
0.0
0
2
4
6
10
12
14
16
2
6
10
12
14
16
(# improvement steps)
(#
improvement steps)
Figure 9: Performance of models trained via each method when run in a greedy-search scaffold on the MLE- bench train (left) and test (right) competitions.
DeepSeck-RL-Disill--Noe landom-acts-ffppizze
GRPO -
Curriculum spooky-authorideenffiaad.. -inntfiiiiiiennniiiaiaon
Improve
Diverge
Exit
detecting-innults-ine social- socal
commenttayyy
0.6
0.8
0.6
average@11
0.4
average@16
average@116
0.6
0.4
0.3
0.4
0.2
0.2
0.0
8
10
12
14
16
0
2
6
10
12
14
16
2
6
8
10
12
14
16
(# Improvement steps)
iigaawtoxic-commee
beriallcactus-iidentfffiiati
0.30
0.3
0.25
average@10
0.6
average@16
average@1o
0.20
0.2
0.15
0.4
0.10
0.2
0.05
0.0
0.00
10
12
14
16
0
2
10
12
14
16
2
4
6
8
10
12
16
Figure 10: Improvement over 16 greedy-soarch steps using the initial model and the initial model fine-tuned using each method on train (top) and test (bottom) MLE-bench competitions.
22

---

Published in Transactions on Machine Learning Research (02/2026)
D Example self-iteration steps
We provide examples of self-iteration prompts and responses encountered over the course of training via the full ExIT method. We see that the model can respond successively both self celf-imprvvement and self- divergence prompts. Notably, we see in example from the tool-use domain that self-divergence can lead not only to new solution approaches, but also improvements to previonsly faulty response.
Improving math solution (self-correction)
Improve your current response to this request:
## Request:
In round-robin tournament with 6 teams, each team plays game against each other team, and each game results in one team winning and one team losing. At the end of the tournament, the teams ranked by the number of games What is the maximum number of teams that could be tied for the most wins at the end of the tournament?
## Current response:
## Step Determine the total number games played the tournament.
In round-robin tournament with 6 teams, each team plays one game against each of the other teams. So, the total number of games played the number of teams choose 2, calculated as C(6, 2) (2!(6-2)!) games.
## Step Determine the maximum number wins team can have.
Since each game results one win and one loss, the total number wins across teams is equal the total number of games played. Therefore, the maximun number of wins a team can have is the total number of games divided by the number of teams, which is 15 / = 2.5. However, since a team cannot have fraction of a win, the maximum number of wins team can have is 2.
## Step 3: Determine the maximum number of tearms that could be tied for the most wins. For maximum number of teams to be tied for the most wins, we need to maximize the number of teams with the maximum possible number of wins. Since cach team be tied for the most wins is 5. This is because if all teams had the same number of wins, and team had 2 wins, the remaining 5 teams could also have 2 wins if they each lost the team with 2 wins, but this would mean the team with achieve tie with 5 teams consider that
have at most 2 wins, the maximun number of teams that can
wins would have played 5 games, which is not possible. However, has wins and the remaining teams have I win each.
The final answer
## Step Determine the total number garees played tournament. round-robin tournament teams, each team plays one game against each of the other teams. So, the total number games played is number teams choose 2, calculated C(6, 2) = 6! (2!(6-2)!) = 15 games.
## Step 2: Determine the maximum number of wins a team can have. Since each game results in one win and one loss, the total number of wins across all teams is equal to the total number of games played. Therefore, the maximum number of wins a team can have is the total number of games divided by the number of teams, which is 15 / = 2.5. However, since a team cannot have fraction of a win, the maximum number of wins a team can have is 2.
## Step 3: Determine the maximum number of teams that could be tied for the most wins. For maximum number of teams to be tied for the most wins, we need to maximize the number of teams with the maximum possible number of wins. Since each team
have most 2 wins, the maximum number of teams that can be tied for the most wins is 5. This is because if all teams had the
number of wins, and team had 2 wins, the remaining teams could also have 2 wins if they each lost to the team with 2 wins, but this would mean the team with wins would have played games, which not possible. However, can achieve with teams if we consider that team has wins and the remaining teams have win each.
The final answer
Diverging from math solution
Consider your current response to this request, and provide another response that takes an approach that SIGNIFI- CANTLY DIFFERS from the approach of the current response.
## Request:
A 50-card deck consists of 4 cards labeled for i = 1,2, 12 and 2 cards labeled 13 ". If Bob randomly chooses
23

---

Published in Transactions on Machine Learning Research (02/2026)
cards from the dock without replacement, what is the probability that his 2 cards have the same label?
## Current response:
## Step I: Calculate the total number of ways to choose 2 cards from 50-card deck without replacement. The total number of ways to choose 2 cards from 50 cards without replacement can be calculated using combinations. This can be 1225.
expressed as (50) 21000
50x49
## Step Calculate number of ways to choose cards with the same label. There are cards labeled "i" cards labeled '13', We need to calculate the number ways choose 2 cards with the same label for each type of card.
## Step 3: Calculate the mimber of ways to chnose cards with the same lahel for the laheled 'I" cards. For each of the 12 labeled "I" cards, we can choose 2 cards out of the cards of the same label. This can be calculated using combinations as (2) 274-27 2
4! 4-3 6. Since there are 12 different labels, the total number of ways to choose 2 cards
with the same label for the cards is 12 + 72.
## Step 4: Calculate the number of ways to choose 2 cards with the same label for the labeled "13" cards. For the labeled "13" cards, we can choose 2 cards out of the 2 cards of the same label. This can be calculated using combinations 23
## Step Calculate the total number of favorable outcomes. The total number of favorable outcomes is the sum of the number of ways choose 2 cards with the same label the "i" cards and the number ways choose cards with the same label for the '13" cards. This
## Step 6: Calculate the probability. The probability choosing 2 cards with the same label can calculated by dividing the total number favorable outcomes by the total number ways to choose 2 cards. This ra
1225
73 # # Step 7: Simplify the fraction. The fraction 1225 can be simplified by dividing both the numerator and the denominator
by their greatest common divisor, which This results in
The final
1225
## Your response format:
First, provide brief summary approach in the current respouse above.
Next, state meaningfully different approach that can be taken instead.
Finally, provide your new respouse using this different approach.
**Summary of the current approach:**
The current approach involves calculating the total number of ways to choose cards from the deck and then calculating the number of ways choose cards with the same label. This is done by calculating combinations for each type of card (labeled and "13") and summing up the results.
**A mcaningfully different approach:**
different approach can be taken by using the concept of complementary probability. This involves calculating the probability that the two cards do not have the same label and subtracting this from
**New response:**
# # Step 1: Calculate the total number of ways to choose 2 cards from the deck without replacement. The total number of ways to choose 2 cards from 50 cards without replacement can be calculated using combinations. This can be expressed
as (30) 20 2J
1225.
## Step Calculate the number ways choose 2 cards that do have same label. calculate the number of ways choose cards do not have the same label, we can calculate the total number ways choose 2 cards and subtract the number of ways to choose cards with the same label.
## Step 3: Calculate the number ways to choose 2 cards with the same label. This was calculated the current response as 72 + 1 73.
## Step 4: Calculate the number of ways to choose 2 cards that do not have the same label. The number of ways to choose 2 cards that do not have the same label is 1225 73 = 1152.
## Step 5: Calculate the probability that the two cards do not have the same label. The probability that the two cards do not have the same label can be calculated by dividing the number of ways to choose 2 cards that do not have the same
24

---

Published in Transactions on Machine Learning Research (02/2026)
label by the total number of ways choose 2 cards. This is 1223
## Step Calculate the probability that the cards have the label. The probability that the cards have the label be calculated by subtracting the probability that cards do not have the label from 1. 1152
This 1225
## Step 7: Simplify the fraction. 'he fraction 1225 can simplified by dividing both the numerator denominator by their greatest common divisor, which is 1. This results in 1223
73
The final answer is:
1226
Improving tool-use decision
Message history in which the start engine call results in an error indicating that the doors must first. be locked, with the assistant mistakenly asking the user to lock the doors first }
## Request:
I've completed the maintenance on my car and ensured the doors are unlocked. Everything, especially the tires, seems in good condition. Would you kindly assist in securing the remaining doors and initiate the engine in START mode? I want everything primed before I set off.
# Current response:
TASK _RROR>>RR>>>>
<think> Based on the response, the doors need to be locked again. Let's go through the steps more carefully. </think> <tool> 'name": 'lockDoors', "parameters": "unlock": false, "door": "driver", "passenger', 'rear left', "rear right"]
</tool>
Diverging from tool-use decision
Message history preceding this turn, in which the assistant incorrectly uses city names arguments for the distance function )
## Request:
Could you provide with rough distance from Rivermist (xip 83214) to Greenway (zip 74532)? I'm eager plan visit and need know the travel distance.
## Current response: TASK ERROR
## Your new response format:
First, provide brief summary the approach in the current response above.
Next, state meaningfully different approach that can be taken instead.
Finally, provide your new response using this different approach.
Summary of the previous approach: The previous approach attempted city names to get valid zip codes but unsucccssful. Since direct zip code retrieval did not work, the previous attempt ended in
Meaningfully different approach: Instead of using city names, can directly use the provided zip codes distance estimation.
We will use the provided zip codes (83214 for Rivermist and 74532 for Greenway) to estimate the distance between the two cities. </think>
<think>
<tool> 'name": 'estimate distance', "parameters": "cityA': '83214', 'cityB': '74532' </tool>
Improving MLE-bench solution
Study the current version of solution.py:
import 08
25

---

Published in Transactions on Machine Learning Research (02/2026)
import pandas pd
from sklearn bayes inport Multinonis1MB
from sklearn
extraction. text import CountVectorizer
def main(). ():
Load the training
train_data pd.read_csv (os. path join environ (BBBLIC,RESDURGE_PATT]]] 'train.
1))
test_data read pat.phhoo.ah...................p join (os environ ("PUBLIC_RESOURCEEERR 'test.cav')
Vectorize the teat data vectorizer CountVectorizer ()
s_train vectorizer.c fit transform aerrain_datai (eext '])
x X_test vectorizer. transform (test__ttet_etee ['text tttett '])
Train the model model = (ultinomialN8()
model tiatin crain_data ('author
Get probabilities for test data
preds model preddctt.rrobb(xttee))
submission pd DataFrane {
'id': test_data ('i')),
' preds ,0,
'HLL": preds 11,
" preds [:
})
submission to_ca cav subaisssion.crr. index=False)
if namamm.... mamaain__.. main ()
Your goal is to improve this python script to better achieve the following task:
task description
thinking not shown to conserve space
To improve the Kaggle solution, I'll modify the code to:
1. Use Laplace smoothing (alpha=1) to handle sparse features
2. Ensure proper probability normalization Add logging for better debugging
3.
Here's the improved code:
python
import
import pandas pd
from sklearn naive_bayes inport MultinomialMB
from sklearn. feature_ extrrretitttiioon text import CountVectorizer from metrica import log_loss
def main
Load training and test data
crain_data read (os path join (os enviroa WPUBLICCREEOURCE_PP. '], 'train. cav
1))
test_data pd. read read__sv cov path. join (os environ [ BBASLICC__RSOUCC_PAATTTT J 'test.csv"
Vectorize the text data
vectorizer CountVectorizer ()
X_train vectorizer. fit_ transform (train_date ('text )
26

---

Published in Transactions on Machine Learning Research (02/2026)
tetest vectorizer (tesst data ['text "])
Train the model
model #altinommaiNB(alpaa11.0000 model fitaataa train_data ['author
probabilities for test
preds model. predict.proba(X_tees
Save
submission pd. DataFrame (f
rid': pot__ata[[1a)
"EAP"; preds [0]
HPLL: preda [1]
NNS": preds [: ,2]
})
subnission.tad to_cav ('subnission.cor index=False)
1f name_. as mminn_nn
import logging Logging.basicCone basicConfig (level=logging INFO)
logger
logging getLogger oogging.gotLoggerccc (__mame_.)
logger info (tt.srrriiggghhuusssssuuuuuuaaaa
logger info (f dat.ooiinnddaaaaarrrooaaaao enviroo
environ PUBLIC_ RESOURCE_ ")
main() ()
code:
1. Adda Laplace smoothing (alpha=1) to handle low-count features Uses logging for better debugging
Ensures proper probability output
4. Handles the submission format correctly
Diverging from an MLE-bench solution
Study the current version of solution.py:
import
import pandas
pd
from sklearn feature cickuur_exttrattinn text import TridfVectorizer from sklearn. bayee inport MultinomislMB
def main(): Load training testing data
prain_data_pate path. environ get BEWOOICREEUUUUEE_EEE 'train. cav
test_data_path os os. path. (oin enviroe environ get ('PUBLIC_ PUUSLIC_RESOUCC_PATE 'test.
train_data read (taahaaaaac test_date data pd read csv (tost_ttaaaaaaappp........
# Prepare the data X_train train_data. [' Comment ']
y_train train_data ( Insult
Vectorsze the text data vectorizer T ()
X_train_vec vectorizer fit (xansform(X_erring
Trasn the model model MultinomialMB() model. model.fit fodell.et(X,ursinn.ee.. y-train)
Make predsctions
X_teat test_data ('Cmment')
X_test_vec vectorizer transform (eateat)
27

---

Published in Transactions on Machine Learning Research (02/2026)
y-pred model. predict (X_test_vec)
Save the results
submission pd. DataFrane ({ Insult'; y-pred)) submission to_cav ('submisaion, cav', index=False)
___nme__ "__main__" main
Your goal propose new python script that improves over python script the following task. Your improved script should radically different approach to solving problem than the one taken by the current python script.
task description
thinking shown conserve space
solve this problem, I'll create simple machine learning model using logistic regression TF-IDF vectorization. Here's the complete solution:
import 08
import pandas as pd from sklearn. teature_extractiie text import TfidfVectorizer from sklearn. linear _noel import LogisticRegression from sklearn. metrica import roc_auc_acore
def main();
# Load data train_path = path join (os. environ
jonveorrnn. get PPPULIC_BESOURRE_PPPP 'ttrain cav
test_path os path join ewwiron.nvinnooo.e get (e PUBLLC REEEEEEEEEEEEET111 'test.ccv')
train_data pd. read (train_path)
data re.dead (teat_path)
Vectorise
vectorizer IfidfVectorizer( ()
X_train vectorizer fit transform (wwaan (trainndataa Coomment '])
Trasn model
model = LogisticRegressionn)
m (X__trrin,,, train_data Insult
# Get predictions X_test test_data "
X_test_vec vectorizer (apnaffornX_tttttt_
preds = model. bodel.predict_peee pradicttprobaxx_testttto [:,1]
Save resuits
submission pd. DataFrame (f
IInuul'' preds.
DDate' fest_data['Den) (atte')ate'),
'Comment'; test_data Comment
submission cay ('submission, cav', index=False)
1f ___ame__ main__?
main ()
This code:
1. Loads the training and test data
2. Vectorizes the comments using TF-IDF
3. Trains a logistic regression model
4. Generates predictions and saves them to submission.csv
The model uses TF-IDF for feature extraction and logistic regression for prediction, which should provide reasonable AUC scores given the problem constraints.
28


Reviews:
[{"review_id": "ogMFRsd8sq", "reviewer": "Reviewer_UHiV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Exploratory Iteration (ExIt) that desighed to train LLMs to perform multi-step self-improvement at inference-time while only training on the most informative single-step iterations. The authors evaluate ExIt across three tasks: math, function call and ML engineering, which convincing the ablation results against several baselines like GRPO.\n\nStrength:\n\n1. This paper is solid. The presentation and writing is easy to follow with detailed proof process.\n\n2. The investigated question that how to efficiently train models to perform multi-step self improvement is important.\n\n3. The ablation experiments are conducting on three tasks: Math, Function call, ML engineering, which include a wide range of practical tasks. The authors include three kinds of models from LLaMA to Qwen and Deepseek.\n\n4. The authors compare their methods with several baselines and outperform them on the three tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The experimental results and claims have sufficient evidences in the main text.", "audience_yn": "Yes", "audience_explanation": "The investigated problem is very novel and important. Because scaling test-time computation could obviously improve the performance and self-improvement is a key method of scaling test-time computation.", "requested_changes": "Weakness:\n\n1. Why are the buffer size of three tasks different? Is there any ablations to show the selected size is the best size?\n\n2. I found three tasks use three different models. Why use this kind of settings? It would be better if there are more experimental results on the same models across different domains or tasks to show the generalization performance. Reporting different models on different tasks may be suspicion of selecting a suitable model for the task.\n\n3. For math task, I couldn't find the accurate sub-task performance. In figure 4 and figure 8, it seems the improvement gains largely from the MATH500. For sub-tasks like AIME 2025, the improvements remain fair.\n\n4. An average of 20 of Math related score falls behind existing methods (such as [1]). I recommend the authors to update the models to see if EXIT could further improve the performance.\n\n[1] RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problems", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q1wb3jTouI", "reviewer": "Reviewer_qHPq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces *Exploratory Iteration (ExIt)*, a reinforcement learning strategy for training LLMs to perform multi-step self-improvement at inference time using only single-step training iterations. ExIt maintains a buffer of high-learnability partial histories and selectively samples them for further improvement or divergence to expand the task space dynamically. The authors tested math reasoning, multi-turn tool-use, and ML engineering tasks. ExIt improves both initial solution quality and iterative refinement ability, with diversity mechanisms further enhancing performance and task coverage.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, the authors have provided extensive empirical results.", "audience_yn": "Yes", "audience_explanation": "The authors address self-improvement at inference time in LLMs, with Exit being a welcome addition in this LLM era.", "requested_changes": "- Reduce repetition in the text, particularly in sections 4.0 and 4.1 by consolidating overlapping explanations into a more concise presentation.\n- Provide a clear analysis of the computational overhead involved in computing the diversity score (If I understood correctly it involves working in the embeding space) including its impact on training time and scalability.\n- Elaborate how ExIt would perform with extremely large task spaces or very long multi-turn histories\n- If possible provide an analysis of negative cases, such as scenarios where ExIt fails to improve performance or where diversity-seeking mechanisms hinder convergence", "broader_impact_concerns": "None", "additional_comments": "None", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZLvWtNoiO5", "reviewer": "Reviewer_FpLj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a way to train LLMs to exhibit multi-step self-improvement at inference-time, while avoiding the complexity to train on full reasoning chains. This is achieved by iteratively constructing a prioritized buffer of partial histories, ranked by a learnability score (variance associated with GRPO). Subsequently, one SELECTS from this buffer, and EXPANDS the buffer which each rollout.\n\nThis process is combined with self-diverging prompts and a multiplicative bonus to encourage diversity. Training is performed only on single-step self-iteration transitions, just at test time one generalizes to K-step self-improvement.\n\nThe strategy is named \"Exploratory Iteration (EXIT)\", and its efficiency is demonstrated over pure GRPO and other baselines on three tasks: competition math, multi-turn tool use, and ML engineering. This improvement is in parts explained by increases task-space diversity due to using the EXIT strategy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I think the main claims (i.e. that EXIT improves self-improvement at inference and increases task diversity) are sufficiently supported by the empirical results on three different tasks. These advantages are apparent both for the initial accuracy, as well in the further gains due to the improvement steps.\nI think also the experiments are reporducible due to the provided hyperparameter sets, and the ablation studies support sufficient generalization.\n\nSome points that from my point of view still can be improved to enhance the significance of the results are listed below.", "audience_yn": "Yes", "audience_explanation": "I think the paper has a potentially broad audience, as RL post-training is certainly an quite active research area. What I think is especially nice is that the method is relatively simple, and should be compatible with most standard GRPO pipelines (and maybe even beyond). Furthermore, the analysis of task space diversity -- although only briefly discussed in the paper -- should be ob interest to a broad number of researchers.", "requested_changes": "In general I think this is a quite nice paper presenting an interesting concept. As explained above, I think the main claims are decently supported, but I see following points for imporvement:\n\n**My major points:**\n* I think the statistical significance of the results is not entirely safe from debate, more concretely:\n   * While some parts are noted with confidence intervals (or rather std in Table 1), most other plots do not show this information.\n      * Especially for the Tool-Use environment the improvement for the K-steps seems to be rather small (in particular Fig. 3 middle + right), so hard to judge significance there\n   * I am aware that the training runs are quite expensive on a individual basis, but only 3 runs per configuration seems a bit too small to guarantee significant results\n* EXIT used $\\Delta_{max}$ for normalization, but I couldn't find a concrete definition in the paper (apart from \"is the largest possible improvement\")\n   * How can this term be evalauted / estimated for an arbitrary environment?\n   * I think it should be explicitly provided for the three considered tasks.\n* The paper claims that single-step training suffices and is more efficient, so I think this should be validated against another baseline with multi-step RL training (i.e. fixed K)\n\n**Some smaller things:**\n   * I think the \"compute-equivalent\" claim when using variance-based prioritization should be supported (e.g. some wall-clock times, overhead of sampling from and maintaining buffer, ...)\n   * The difference of the diversity metrics considered in Table 2 vs. Figure 7 might need some more explanation and contextualization. For me it was a bit difficult to understand what is actually evaluated there, and how these two analysis do or no not relate.", "broader_impact_concerns": "N/A", "additional_comments": "What is a bit unlucky: There is already a method called \"ExIt\" (Expert Iteration, see \nhttps://doi.org/10.48550/arXiv.1705.08439) in an related domain, so maybe changing the abbreviation for clear distinction makes sense.\n\nThe limitation to non-stochastic environments is a clear limitation of this work. This is acknowledged and stated to be left for future work in 4.1, but I think this limitation should already be acknowledged earlier in the paper.\n\nI think releasing the code and full config files would certainly strengthen reporducibility and enhance the relevance for other researchers.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)