Bootstrapping Task Spaces for Self-Improvement
Authors: name: Minqi Jiang, name: Andrei Lupu, name: Yoram Bachrach

Progress in many task domains emerges from repeated revisions to previous
solution attempts. Training agents that can reliably self-improve over such
sequences at inference-time is a natural target for reinforcement learning
(RL), yet the naive approach assumes a fixed maximum iteration depth, which can
be both costly and arbitrary. We present Exploratory Iteration (ExIt), a family
of autocurriculum RL methods that directly exploits the recurrent structure of
self-improvement tasks to train LLMs to perform multi-step self-improvement at
inference-time while only training on the most informative single-step
iterations. ExIt grows a task space by selectively sampling the most
informative intermediate, partial histories encountered during an episode for
continued iteration, treating these starting points as new self-iteration task
instances to train a self-improvement policy. ExIt can further pair with
explicit exploration mechanisms to sustain greater task diversity. Across
several domains, encompassing competition math, multi-turn tool-use, and
machine learning engineering, we demonstrate that ExIt strategies, starting
from either a single or many task instances, can produce policies exhibiting
strong inference-time self-improvement on held-out task instances, and the
ability to iterate towards higher performance over a step budget extending
beyond the average iteration depth encountered during training.