Title: Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks
Venue: NeurIPS 2025
Accepted: yes
Keywords: language models

agents

self-improvement

in-context learning

sequential decision-making
Abstract: Improving Large Language Model (LLM) agents for sequential decision-making tasks typically requires extensive task-specific knowledge engineering—custom prompts, curated examples, and specialized observation/action spaces. We investigate a different approach where agents automatically improve by learning from their own successful experiences without human intervention. Our method constructs and refines a database of self-generated trajectories that serve as in-context examples for future tasks. Even naive accumulation of successful trajectories yields substantial performance gains across three diverse benchmarks: ALFWorld (73\% to 89\%), Wordcraft (55\% to 64\%), and InterCode-SQL (75\% to 79\%). These improvements exceed those achieved by upgrading from gpt-4o-mini to gpt-4o and match the performance of allowing multiple attempts per task. We further enhance this approach with two innovations: database-level curation using population-based training to propagate high-performing example collections, and exemplar-level curation that selectively retains trajectories based on their empirical utility as in-context examples. With these enhancements, our method achieves 93\% success on ALFWorld—surpassing approaches that use more powerful LLMs and hand-crafted components. Our trajectory bootstrapping technique demonstrates that agents can autonomously improve through experience, offering a scalable alternative to labor-intensive knowledge engineering.

Self-Generated In-Context Examples Improve LLMAgents for Sequential Decision-Making Tasks

Abstract

Improving Large Language Model (LLM) agents for sequential decision-making tasks typically requires extensive task-specific knowledge engineering-custom prompts, curated examples, and specialized observation/action spaces. We investigate a different approach where agents automatically improve by learning from their own successful experiences without human intervention. Our method constructs and refines a database of self-generated trajectories that serve as in-context examples for future tasks. Even naive accumulation of successful trajectories yields substantial performance gains across three diverse benchmarks: ALFWorld (73% to 89%), Wordcraft (55% to 64%), and InterCode-SQL (75% to 79%). These improvements exceed those achieved by upgrading from gpt-4o-mini to gpt-4o and match the performance of allowing multiple attempts per task. We further enhance this approach with two innovations: database-level curation using population-based training to propagate high-performing example collections, and exemplar-level curation that selectively retains trajectories based on their empirical utility as in-context examples. With these enhancements, our method achieves 93% success on ALFWorld-surpassing approaches that use more powerful LLMs and hand-crafted components. Our trajectory bootstrapping technique demonstrates that agents can autonomously improve through experience, offering a scalable alternative to labor-intensive knowledge engineering.

1 Introduction

When creating LLM agents for sequential decision-making tasks, practitioners often improve agent performance by investing in task-specific knowledge engineering (through tedious prompt tuning [1], human-crafted in-context examples [2, 3] or custom observation and action spaces [4, 5]). Using these techniques, scaling agent performance comes from scaling human effort.

In this paper, we investigate an alternative path: enabling LLM agents to autonomously bootstrap their own performance by leveraging their own successful experiences via in-context learning. The efficacy of in-context learning depends critically on both the quality of the examples [2, 3] and their relevance to the current decision point [6-8]. This insight provides a natural direction for automated agent self-improvement: accumulating successful self-generated trajectories and estimating the most relevant and effective prior experiences to use as in-context examples for each action.

Our work assumes a ReAct-style agent [9] that retrieves different examples for each decision point based on their relevance to the current situation [10, 11]. We build on this foundation by focusing specifically on how to construct and refine the underlying database of self-generated examples. How can we identify which trajectories enhance performance on new tasks versus those that hinder performance? This database construction problem requires addressing both the collection of highquality trajectories and the strategic curation of the most valuable ones for future retrieval at each decision point in the agent's reasoning and acting loop.

We demonstrate that even naive database accumulation improves test-set performance from 73% to 89% on ALFWorld, 55% to 64% on Wordcraft, and 75% to 79% on InterCode-SQL. (Equivalent to what a baseline agent would achieve if it were allowed two to three attempts per task.) We further propose two database construction enhancements: (1) database-level curation that identifies and propagates high-performing example databases, and (2) exemplar-level curation that identifies helpful trajectories based on their empirical utility as in-context examples. These approaches do not require task-specific prompt engineering [12, 4] or custom observation/action space design [11, 4], but improve success rates on ALFWorld to 93%-surpassing approaches like AutoManual [4] that use more powerful LLMs and hand-crafted observation and action spaces, as well as hierarchical approaches like Autoguide [12]. The success rate improvement on ALFWorld exceeds the boost obtained from upgrading the agent's underlying LLM from gpt-4o-mini to gpt-4o. Our results highlight the practical value of trajectory bootstrapping as a dimension for scaling test-time compute.

2 Preliminaries

Sequential Decision-Making Tasks We focus on multi-step sequential decision-making tasks where agents must produce a series of actions based on observations of the environment. The sequential nature of these tasks introduces unique challenges for LLM agents, as they must interpret intermediate environmental feedback, maintain coherent reasoning across steps, and adapt their strategy based on the evolving task state.

We assume a standard POMDP setup (App. B), where an agent, given a task goal g , interacts with the environment E for up to T timesteps. At each timestep t , the agent receives an observation o t , takes an action a t , and E transitions to the next state. We consider sparse-reward environments where success is only determined at the end of an episode. This is a standard setting in prior agentic work [9, 13, 12, 4]. Please see App. B for details.

ReAct-style Agent Loop Our work assumes a ReAct-style [9] agent architecture that employs recent best practices for in-context retrieval [10, 11]. The agent operates through a three-phase approach (planning, reasoning, and acting) as formalized in Alg. 1. Two key components differentiate our implementation from basic ReAct: (1) an initial planning step where the agent generates a high-level plan before execution begins ( 1, line 3), which has been shown to boost performance in prior work [10, 14, 13, 4], and (2) dynamic retrieval of different trajectory segments for each decision point [10, 11], rather than using the same examples throughout an episode [9, 13, 12].

The agent operates through three key LLM-based functions: LLM plan generates a high-level plan p for the goal, LLM reason processes observations o t to produce reasoning r t , and LLM act determines actions a t . The Retrieve () function selects the k most relevant examples from database D based on similarity between lookup keys and examples. Our contribution focuses specifically on constructing and refining the trajectory database that powers this retrieval mechanism, without relying on task-specific

prompting, observation spaces [11] or action spaces [5, 4]. Note that in Alg. 1 all task-specific knowledge is encapsulated in D . For simplicity, we eschew other techniques, like hierarchical learning [13, 12, 4], that are also task-agnostic, but add additional agent complexity.

3 Problem Statement

Given the ReAct-style agent described in Sec. 2, our goal is to construct a trajectory database that maximizes agent performance on sequential decision-making tasks. We focus on building and refining the example database accessed by the agent's retrieval mechanism. Formally, we aim to construct a trajectory database D where each trajectory τ ∈ D captures a complete task attempt: τ = ( g, p, { ( o t , r t , a t ) } T t =1 , s ) . We aim to maximize agent performance across tasks T : D ∗ = arg max D E g ∼T [ Success ( Agent ( g, D , E , T ))] , where Success () returns the binary outcome s . We assume that we are given: (1) D initialized with a small number of human-generated trajectories, (2) a descriptor of the action space, and (3) access to a set of training tasks drawn from T that the agent can attempt. All three assumptions are typical for ReAct-based agents [9, 10, 13, 12, 4].

4 Related Work

In-context learning for agent improvement Despite the current popularity of reinforcement learning-based approaches for improving agent capabilities [15-18], in-context learning offers distinct scientific and practical advantages. These benefits include model-agnostic portability across different LLMs, efficiency in low-sample regimes [3, 19], and accessibility when implementation barriers exist for weight modification methods. Both empirical and theoretical work has established that in-context performance can scale effectively with additional examples [6, 7, 19, 8], suggesting that strategic example accumulation should lead to significant performance improvements. We focus on maximizing the value of limited examples through in-context methods, while hypothesizing that database quality, not just quantity, critically influences performance scaling. For completeness, we offer a preliminary investigation in App. F of how our collected trajectories could potentially serve as training data for fine-tuning approaches.

Automatic in-context examples Recent work has demonstrated the effectiveness of optimizing both instructional content and example curation in prompts. DSPy [20] introduced a framework for optimizing multi-step pipelines through instruction tuning and strategic example curation. Selfgenerated examples containing reasoning traces can eliminate the need for human-written examples, and these self-generated examples often contribute more to performance than optimized instructions alone [21]. These approaches typically select fixed exemplars for all task instances, whereas our method enables the dynamic selection of different in-context examples for each decision.

In-context self-improvement of LLM Agents Self-improvement methods for LLM agents either aim to solve one task (performing search/optimization) or transfer knowledge from prior tasks to novel ones (generalization) (see App. J.2 for further discussion). Approaches to solve a single task scale the number of sampled solutions at inference time [22-24] or incorporate feedback from failed attempts [25]. Knowledge transfer approaches include abstraction-based methods like ExpeL [13] and AutoGuide [12], while others employ task-specific information in their design-RAP [10] uses task-specific prompts and AutoManual [4] constructs task-specific state and action spaces (see App. J.1). Other dimensions of self-improvement include hierarchical execution [26] and optimization techniques for multi-stage systems[27-29]-techniques complementary to our approach. Rather than developing complex architectures or leveraging task-specific information, we focus on identifying which trajectories most contribute to successful outcomes as in-context examples.

In-context reinforcement learning Our work connects to the emerging area of in-context reinforcement learning, where language models perform sequential decision-making through contextual examples rather than parameter updates. Recent work has explored how transformers can implement RL algorithms in-context, both via algorithm distillation for in-context RL [30], and via transformers' ability to learn from reward trajectory contexts via supervised pretraining [31]. LLMs can balance exploration-exploitation tradeoffs through intelligent prompt design [32], and other work suggests LLMs being able to perform in-context policy iteration [33]. While these approaches focus on learning RL algorithms or policies in-context, our work addresses a complementary problem: how to

effectively curate and retrieve trajectory examples to maximize in-context learning performance for sequential decision-making tasks.

5 Methods

We now discuss three algorithms for constructing database D using a continual collection approach.

5.1 Traj-Bootstrap : Constructing a Database of Previously-Solved Tasks

Our trajectory-bootstrapping algorithm Traj-Bootstrap constructs a trajectory database D by collecting successful agent experiences. As outlined in Sec. 3, we start with a minimal set of human-provided exemplars (which could be empty), then grow the database as the agent successfully completes training tasks. This process creates a positive feedback loop where successful examples help the agent solve new tasks, generating more successful examples.

Traj-Bootstrap operates on principles similar to reward-weighted regression in reinforcement learning [34], where only successful trajectories ( s = 1 ) are stored in the database. This filtering mechanism ensures the agent learns from positive examples while avoiding potentially misleading failed attempts. Successful trajectories can be leveraged by asking the agent to imitate the successful patterns in these trajectories. Failed trajectories are more challenging to operationalize due to the credit attribution problem: it is necessary to identify the 'good' vs 'bad' parts of the trajectory so the the agent can imitate the good parts and avoid the mistakes made in the bad parts. Failed trajectories do offer the opportunity to guide exploration [25]; we leave this direction to future work.

5.2 +DB-Curation : Database-Level Data Curation

Traj-Bootstrap exhibits unpredictable performance variation across training (database construction) trials, even when following identical collection procedures. Fig. 1 illustrates this variation across five trials on the InterCode-SQL benchmark (a benchmark we use for evaluation in Sec. 6). The variance arises from two factors: (1) the stochasticity of LLM outputs creating different initial trajectories, and (2) an amplification effect where early differences in collected examples lead to wide performance variation.

This observation motivates a data curation strategy inspired by population-based training in reinforcement learning [35]. Fig. 1 shows that some databases lead to better task performance than others-so we identify the underperforming databases periodically during training and remove them, continuing growth from top-performing ones. We introduce +DBCuration , a population-based training algorithm (Alg. 2) to identify and propagate the most effective databases during the bootstrapping process.

0

200

400

600

800

Num. Training Tasks

0.72

0.74

0.76

0.78

Success Rate

InterCode-SQL

+DB-Curation maintains N database instances initialized with identical human-provided exemplars. Each instance is used by a separate agent that accumulates successful trajectories independently. Curation events occur when the number of tasks attempted reaches size thresholds (starting at size 10 and doubling thereafter). At each threshold, +DB-Curation evaluates database performance based on the agent's success rate on all training tasks since the last threshold, and replaces the worst-performing database with a copy of the top-performing database.

The key insight of this approach is that database quality emerges from collective properties-like coverage, diversity, and complementarity across examples-not just individual trajectory quality. Moreover, a single trajectory collected early in training can influence many future trajectories by guiding the agent toward particular solution strategies, creating cascading database-level effects. By selecting and propagating entire databases, we preserve these beneficial emergent properties while using a simple, computationally efficient evaluation metric based on recent performance.

5.3 +Exemplar-Curation : Exemplar-Level Data Curation

While database-level curation via +DB-Curation identifies entire sets of complementary trajectories, discarding whole databases can eliminate valuable trajectories. We find that even poor-performing databases contain individual high-quality trajectories that yield better outcomes when used as examples. Conversely, some successful trajectories may contain bad individual decisions that would be unhelpful to repeat. This observation motivates +Exemplar-Curation : identifying and selecting individual high-quality exemplars across multiple database instances based on their empirical utility as in-context examples. This approach parallels value-function learning in reinforcement learning [36], where we estimate the 'value' of each trajectory based on its contribution to successful outcomes.

We introduce a retrieval-weighted quality metric analogous to a value function to quantify each trajectory's contribution to successful outcomes:

where R ( τ ) is the set of tasks for which trajectory τ was retrieved, o i is the binary outcome of task i , and f i ( τ ) is the retrieval frequency during task i .

This value metric measures how often a trajectory is associated with successful outcomes when retrieved as an in-context example. It weights outcomes by retrieval frequency, prioritizing trajectories frequently retrieved during successful completions while penalizing those associated with failures.

Alg. 3 outlines +Exemplar-Curation. For each task in the training set, it identifies all successful trajectories across all N database instances and selects the exemplar with the highest value according to the metric. This approach constructs a composite database containing only the most effective exemplars as measured by their empirical contribution to successful outcomes on subsequent tasks.

5.4 Train-Time vs Test-Time LLM Costs

The curation methods (+DB-Curation and +Exemplar-Curation) maintain N parallel database instances during training, and therefore require N × more LLM inference during training compared

to Traj-Bootstrap. However, all methods use the same quantity and distribution of training tasks. At test time, all three methods have identical computational costs-Alg. 1 is simply provided with a different database D for each method. This contrasts with approaches that scale the number of LLM calls per test-time task to improve performance [22-25]. Our methods shift computational burden to training while maintaining efficient inference, a property our in-context methods share with fine-tuning methods.

6 Experiments

We evaluate our database construction methods through experiments addressing three key questions:

Database scaling: How does task success rate scale with increasing database size?

Improving database construction: How much do population-based training and exemplarlevel curation improve task success rate?

Overall effectiveness: How do our approaches compare to alternative approaches leveraging task-specific domain knowledge or hierarchical algorithms?

6.1 Experimental Setup

6.1.1 Benchmark Tasks

We evaluate our methods on three benchmarks: ALFWorld [37], a text-based environment for navigation and object manipulation; InterCode-SQL [38], an interactive coding environment for SQL query generation; and Wordcraft [39], a simplified adaptation of Little Alchemy requiring compositional reasoning to combine elements. These benchmarks were selected because they: (1) provide large enough task pools to support meaningful train/test splits, (2) represent diverse reasoning challenges relevant to sequential decision-making, and (3) have been used in prior work, enabling direct comparisons with existing methods.

6.1.2 Methods Compared

Our methods include:

Fixed-DB : The baseline agent as described in Sec. 2, with a fixed database of humanprovided initial examples and no database growth.

Traj-Bootstrap : The simple progressive accumulation approach from Sec. 5.1.

Traj-Bootstrap+DB-Curation : Our database-level trajectory curation from Alg. 2.

Traj-Bootstrap+Exemplar-Curation : Our exemplar-level trajectory curation from Alg. 3.

Traj-Bootstrap+DB+Exemplar-Curation : Applying both our database-level trajectory curation and propagation and our exemplar-level trajectory curation.

We compare these methods to two hierarchical designs. Autoguide [12] converts successful trajectories into explicit rules and retrieves the most contextually relevant rules, alongside low-level trajectories, at inference time. AutoManual [4] leverages hand-crafted task-specific observation and action spaces-see App. J.1 for details. Unless otherwise specified, we use GPT-4o-mini as our base LLM. We report success rates averaged over five random seeds. See App. E for additional details.

6.2 Traj-Bootstrap Results

Traj-Bootstrap performance improves with more training tasks Tab. 1 presents the final success rate metrics for our database construction methods. The performance of Traj-Bootstrap generally improves with increases in the number of training tasks attempted (Fig. 2) Performance continues to improve with more training tasks across all benchmarks, but exhibits diminishing returns-most gains occur within the first 25% of added training tasks. This efficiency decline occurs because each new example is retrieved less frequently as the database grows, influencing fewer generations, a pattern consistent with findings from Bertsch et al. [19] and Agarwal et al. [8]. As mentioned in Sec. 5, we observe performance variability across trials and within individual trials. Cross-trial variance indicates that some trials produce higher-performing databases when solving identical tasks. Within-trial fluctuations show that certain added trajectories can degrade performance.

+DB-Curation boosts performance on ALFWorld Fig. 3 illustrates how +DB-Curation can improve upon Traj-Bootstrap's performance, despite exhibiting occasional performance dips at smaller database sizes. These dips result from inaccurate estimates (due to low sample count) of database quality early in the process-introducing noise into the curation process.

+Exemplar-Curation boosts performance on InterCode-SQL and Wordcraft As seen in Tab. 1, +Exemplar-Curation yields improvements in final task success rates on InterCode-SQL and Wordcraft, and also boosts success rate at intermediate database sizes for both InterCode-SQL and Wordcraft (Fig.3). To further highlight the impact of our exemplar-level curation metric, Fig. 4 compares databases built from the 'best' trajectories that are the most empirically effective in-context examples versus the least effective trajectories, as determined by Equation 1 in Sec. 5.3. The 'best' curve is identical to +Exemplar-Curation, while the 'worst' curve selects the bottom-1 trajectory instead of top-1 in Alg. 3, line 7. Using the database of high-quality examples yields a higher success rate across all database sizes for ALFWorld and Wordcraft, and for smaller database sizes for InterCode-SQL.

+DB+Exemplar-Curation achieves best performance on ALFWorld and InterCode-SQL Fig. 3 and Tab. 1 highlight that +DB-Curation and +Exemplar-Curation can be complementary, as the combined +DB+Exemplar-Curation achieves the best final task success rates on both ALFWorld and InterCode-SQL (0.93 and 0.82 respectively). Note that on Wordcraft, +DB-Curation fails to

0

1000

2000

3000

Num. Training Tasks

0.70

0.80

0.90

Success Rate

ALFWorld

0

200

400

600

800

Num. Training Tasks

0.72

0.74

0.76

0.78

Success Rate

InterCode-SQL

0

1000 2000 3000

Num. Training Tasks

0.50

0.55

0.60

0.65

0.70

Success Rate

Wordcraft

Avg of 5 trials

0

1000

2000

3000

Num. Training Tasks

0.75

0.80

0.85

0.90

Success Rate

ALFWorld

0

200

400

600

800

Num. Training Tasks

0.74

0.76

0.78

0.80

0.82

Success Rate

InterCode-SQL

0

1000200030004000

Num. Training Tasks

0.55

0.60

0.65

0.70

Success Rate

Wordcraft

Traj-Bootstrap

+DB-Curation

+Exemplar-Curation

+DB+Exemplar-Curation

provide boosts whether or not +Exemplar-Curation is used-Traj-Bootstrap and +DB-Curation perform identically (0.64), and +Exemplar-Curation(0.71) outperforms +DB+Exemplar-Curation (0.67).

6.3 Contextualizing performance boosts from Traj-Bootstrap

To contextualize the improvements achieved by Traj-Bootstrap, we compare with several alternative strategies: test-time sampling, using a better LLM, task-specific strategies, and hierarchical strategies.

Comparison with test-time scaling Our trajectory bootstrapping approaches achieve success rate improvements equivalent to scaling test-time compute by making multiple task attempts-an advantage in scenarios where multiple attempts are impractical or when success verification is unavailable at test time. Furthermore, our approaches provide these benefits without requiring any modifications to the test-time inference process. To demonstrate the magnitude of this benefit, we compare our methods to the alternative strategy of making multiple attempts at each test task with the Fixed-DB baseline and selecting the best outcome [22-24]. Tab. 3 reports the pass@k metrics for Fixed-DB across all three benchmarks, representing the probability of at least one successful attempt when making k independent attempts at each task. Using only a single attempt per task, TrajBootstrap approach achieves success rate comparable to Fixed-DB pass@2 or pass@3 on all three benchmarks. +DB-Curation and/or +Exemplar-Curation perform nearly on the level of Fixed-DB pass@4 on ALFWorld, pass@5 on InterCode-SQL, and pass@5 for Wordcraft.

Comparsion with model upgrades On ALFWorld, after 3500 training tasks Traj-Bootstrap yields a 20-point success rate boost over Fixed-DB, significantly outperforming the 15-point improvement gained by upgrading Fixed-DB to a more powerful LLM.

Comparison with task-specific strategies Tab. 2 shows that Traj-Bootstrap+DB+ExemplarCuration using GPT-4o-mini achieves a success rate exceeds (0.93) that exceeds that of Automanual [4] configured to use a combination of GPT-4-turbo and GPT-4o-mini (0.91). Thus, our methods

0

1000

2000

3000

Num. Training Tasks

0.75

0.80

0.85

0.90

Success Rate

ALFWorld

200

400

600

800

Num. Training Tasks

0.74

0.76

0.78

0.80

Success Rate

InterCode-SQL

0

1000 2000 3000 4000

Num. Training Tasks

0.60

0.65

0.70

Success Rate

Wordcraft

Best

Worst

outperform an approach that uses a more powerful LLM and customized observation and action spaces. See App. K for a comparison to hand-crafted approaches on InterCode-SQL.

Comparison with hierarchical algorithms Given 100 training tasks, Autoguide, a hierarchical rule-learning approach, achieves a 0.79 success rate (using a combination of gpt-3.5-turbo + gpt-4turbo). Given the same number of training tasks our best approach achieves significantly greater success rate (0.86) with gpt-4o-mini (Tab. 2). While this comparison employs different algorithms and LLMs, the performance of Traj-Bootstrap suggests that self-constructed databases of low-level trajectories can be competitive with hierarchical approaches.

6.4 Extending Traj-Bootstrap

Can we predict agent success? Beyond improving agent performance, we can also utilize our self-collected examples to implement useful agent diagnostics, such as predicting an agent's success on novel tasks. On InterCode-SQL and Wordcraft, we train a calibrated Random Forest classifier of agent success based on task goal and initial observation embeddings. Classifier quality (measured via AUROC) improves with database size, reaching 0.77 for InterCode-SQL and 0.71 for Wordcraft with our largest databases. The predicted probabilities also closely match empirical success rates, indicating well-calibrated predictions. See App. H for details.

Can we use our self-collected databases for fine-tuning? We fine-tune GPT-4o-mini using trajectories from our best-performing Traj-Bootstrap+DB+Exemplar-Curation database for each benchmark. The resulting fine-tuned agents (ReAct-Finetune) outperform our in-context approach on ALFWorld (23-point vs. 20-point boost) and Wordcraft (19-point vs. 14-point), but perform worse on InterCode-SQL (4-point vs. 7-point). This suggests our self-collected examples are effective not only for in-context learning but also for creating fine-tuned agents. See App. F for details.

Does our approach generalize across models? To test whether our method captures fundamental task structure rather than model-specific artifacts, we evaluate Mixtral 8x7B Instruct v0.1 using

databases collected with GPT-4o-mini. Despite this challenging cross-model transfer scenario, our full method achieves substantial improvements: +28 points on ALFWorld (0.55 vs. 0.27), +18 points on IC-SQL (0.70 vs. 0.52), and +12 points on Wordcraft (0.52 vs. 0.40). These gains match or exceed those observed with GPT-4o-mini itself, demonstrating that curated databases can effectively transfer across different model architectures and families. See App. G for details.

7 Discussion

Contextualizing the costs of scaling performance along various axes Our self-improvement algorithm offers substantial cost savings compared to two common approaches for scaling model performance: (i) switching to a larger LLM, and (ii) test-time scaling via multiple attempts per task (assuming access to a perfect verifier).

Our full configuration with 5 parallel databases requires 3,500 × 5 training trajectories on GPT-4omini, with a worst-case database construction cost of $600. After this one-time investment, the pertask inference cost is only $0.034 (see Appendix D for detailed token usage and cost calculations). As shown in Section 6.3, this approach achieves a 0.93 task success rate on ALFWorld-outperforming GPT-4o with Fixed-DB (0.88) and matching GPT-4o-mini with Fixed-DB and 3 attempts per task (0.92).

In comparison, using GPT-4o at inference time costs $0.57 per task, breaking even with our approach after only 1,100 tasks. Test-time scaling with 3 attempts costs $0.10 per task, breaking even after 8,750 tasks. Beyond these thresholds, our method remains consistently more cost-effective while delivering superior performance.

At production scale, these savings become substantial. Consider a service processing one million ALFWorld tasks daily-a modest industry-scale throughput. Our method would save $530,000 daily versus GPT-4o, or $60,000 daily versus GPT-4o-mini with test-time compute, even accounting for all offline construction costs.

Failed trajectories are used to improve the system During the initial database construction step, our method achieves train-set success rates of 81.3% (ALFWorld), 76.9% (InterCode), and 58.6% (Wordcraft) on average across 5 training runs. This means 18.7%, 23.1%, and 41.4% of trajectories are discarded respectively. While failed trajectories are not retained for in-context use at test time, failed trajectories are used to improve agent performance via our Exemplar-Curation algorithm (Section 5.3). We do not provide the failed trajectories directly in-context due to the challenges of credit attribution-given a long trajectory with a single incorrect action, it is challenging for the LLM to identify the actions within that trajectory to emulate and the ones to avoid repeating.

Future directions The success of our approach reveals performance gains that stem primarily from accumulating successful examples, establishing a foundation for agent self-improvement where the quantity and quality of accessible data rivals the importance of architectural complexity. This parallels trends in traditional deep learning, where data curation often yields substantial improvements. Our findings point to promising research directions that approach LLM agent enhancement from a data-centric perspective-advancing both strategic data collection methods (balancing exploration versus exploitation across diverse tasks) and refined filtering techniques to maximize performance.

Acknowledgments Thank you to Brennan Shacklett, Purvi Goel, Zander Majercik, William Wang, Bradley Brown, Jon Saad-Falcon, and William Mark for valuable discussions and feedback. This work was supported by the Stanford HAI SEAMS program, Roblox, and Meta, and API credits were provided by OpenAI and together.ai.

References

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824-24837, 2022.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846 , 2023.

Minghao Chen, Yihang Li, Yanting Yang, Shiyu Yu, Binbin Lin, and Xiaofei He. Automanual: Generating instruction manuals by llm agents via interactive environmental learning. arXiv preprint arXiv:2405.16247 , 2024.

Ke Yang, Yao Liu, Sapana Chaudhary, Rasool Fakoor, Pratik Chaudhari, George Karypis, and Huzefa Rangwala. Agentoccam: A simple yet strong baseline for llm-based web agents. arXiv preprint arXiv:2410.13825 , 2024.

Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.15661 , 2022.

Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning , pages 35151-35174. PMLR, 2023.

Rishabh Agarwal, Avi Singh, Lei Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, et al. Many-shot in-context learning. Advances in Neural Information Processing Systems , 37:76930-76966, 2024.

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR) , 2023.

Tomoyuki Kagaya, Thong Jing Yuan, Yuxuan Lou, Jayashree Karlekar, Sugiri Pranata, Akira Kinose, Koki Oguri, Felix Wick, and Yang You. Rap: Retrieval-augmented planning with contextual memory for multimodal llm agents. arXiv preprint arXiv:2402.03610 , 2024.

Ruiwen Zhou, Yingxuan Yang, Muning Wen, Ying Wen, Wenhao Wang, Chunling Xi, Guoqiang Xu, Yong Yu, and Weinan Zhang. Trad: Enhancing llm agents with step-wise thought retrieval and aligned decision. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 3-13, 2024.

Yao Fu, Dong-Ki Kim, Jaekyeom Kim, Sungryull Sohn, Lajanugen Logeswaran, Kyunghoon Bae, and Honglak Lee. Autoguide: Automated generation and selection of context-aware guidelines for large language model agents. arXiv preprint arXiv:2403.08978 , 2024.

Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 19632-19642, 2024.

Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Llm-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF international conference on computer vision , pages 2998-3009, 2023.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36:53728-53741, 2023.

Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720 , 2024.

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 , 2025.

Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R Gormley, and Graham Neubig. In-context learning with long-context models: An in-depth exploration. arXiv preprint arXiv:2405.00200 , 2024.

Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03714 , 2023.

Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. Optimizing instructions and demonstrations for multi-stage language model programs. arXiv preprint arXiv:2406.11695 , 2024.

Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787 , 2024.

Evan Wang, Federico Cassano, Catherine Wu, Yunfeng Bai, Will Song, Vaskar Nath, Ziwen Han, Sean Hendryx, Summer Yue, and Hugh Zhang. Planning in natural language improves llm search for code generation. arXiv preprint arXiv:2409.03733 , 2024.

Xiyao Wang, Zhengyuan Yang, Linjie Li, Hongjin Lu, Yuancheng Xu, Chung-Ching Lin, Kevin Lin, Furong Huang, and Lijuan Wang. Scaling inference-time search with vision value model for improved visual comprehension. arXiv preprint arXiv:2412.03704 , 2024.

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems , 36:8634-8652, 2023.

Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291 , 2023.

Jon Saad-Falcon, Adrian Gamarra Lafuente, Shlok Natarajan, Nahum Maru, Hristo Todorov, Etash Guha, E Kelly Buchanan, Mayee Chen, Neel Guha, Christopher Ré, et al. Archon: An architecture search framework for inference-time techniques. arXiv preprint arXiv:2409.15254 , 2024.

Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. arXiv preprint arXiv:2408.08435 , 2024.

Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, et al. Aflow: Automating agentic workflow generation. arXiv preprint arXiv:2410.10762 , 2024.

Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, DJ Strouse, Steven Hansen, Angelos Filos, Ethan Brooks, et al. In-context reinforcement learning with algorithm distillation. arXiv preprint arXiv:2210.14215 , 2022.

Jonathan Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, and Emma Brunskill. Supervised pretraining can learn in-context reinforcement learning. Advances in Neural Information Processing Systems , 36:43057-43083, 2023.

Zhenwen Dai, Federico Tomasi, and Sina Ghiassian. In-context exploration-exploitation for reinforcement learning. arXiv preprint arXiv:2403.06826 , 2024.

Ethan Brooks, Logan Walls, Richard L Lewis, and Satinder Singh. Large language models can implement policy iteration. arXiv preprint arXiv:2210.03821 , 2022.

Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning , pages 745-750, 2007.

Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint arXiv:1711.09846 , 2017.

Andrew G Barto. Reinforcement learning: An introduction. by richard's sutton. SIAM Rev , 6 (2):423, 2021.

Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768 , 2020.

John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback. Advances in Neural Information Processing Systems , 36:23826-23854, 2023.

Minqi Jiang, Jelena Luketina, Nantas Nardelli, Pasquale Minervini, Philip HS Torr, Shimon Whiteson, and Tim Rocktäschel. Wordcraft: An environment for benchmarking commonsense agents. arXiv preprint arXiv:2007.09185 , 2020.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 , 2021.

Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 , 2023.

Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919 , 2024.

Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bertnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, 11 2019. URL http://arxiv.org/ abs/1908.10084 .

Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, PierreEmmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. arXiv preprint arXiv:2401.08281 , 2024.

Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems , 35:20744-20757, 2022.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600 , 2018.

A Limitations and Broader Impacts

We make the assumption that we are given a few human-provided examples at the start of the database construction process-an assumption standard in the Agentic literature [9, 13, 12, 4, 10]. In App. I, we explore the alternate setting of starting from an empty database, and look forward to further research on bootstrapping from zero human examples in the future. Our algorithm is dependent on the in-context capabilities of LLMs-so our algorithms may be less effective if using an LLM with weaker in-context capabilities than gpt-4o-mini, but may be more effective if paired with more advanced LLMs. In addition, our algorithm generally produces improvements in task success rates, but task success rates are not monotonically increasing, and we hope future work will help improve both the monotonicity and sample efficiency of our algorithm.

This work has the potential to allow LLM Agents applied to a variety of task domains to self-improve. This provides a variety of benefits in terms of task performance, but could lead to applications in the future where the agents perform reward hacking-type behavior-performing undesirable behaviors that are not captured in task success/failure. Since the behavior of our agents is controlled via in-context examples, a potential mitigation technique could be to manually inspect the databases of self-generated examples, or even use an LLM Judge to inspect the examples.

B Note on Sequential Decision-Making Tasks

Wefocus on multi-step sequential decision-making tasks where agents must produce a series of actions over time based on observations of the environment. The sequential nature of these tasks introduces unique challenges for LLM agents, as they must interpret intermediate environmental feedback, maintain coherent reasoning across multiple steps, and adapt their strategy based on the evolving task state. This contrasts with one-shot generation tasks (e.g., solving math problems [40], one-shot code generation [41]) where feedback is only available after the complete solution is provided. Our example-driven learning strategy is potentially also suitable to single-step decision-making tasks, but we focus on the multi-step setting due to its applicability to a number of agentic tasks in real-world settings (embodied agents [14], browser-based tasks [42], etc.).

Formally, these tasks can be modeled as Partially Observable Markov Decision Processes (POMDPs), represented by the tuple ( S , O , A , T , R , γ ) , where S denotes the underlying state space, O the observation space, A the action space, T : S × A → S defines the deterministic transition function, R : S × A → R is the reward function, and γ ∈ [0 , 1] is the discount factor. The partial observability reflects that agents don't have direct access to the full environment state but rather receive observations that provide limited information.

Given a task goal g , an episode consists of the agent interacting with the environment for a maximum of T timesteps. At each timestep t , the agent receives an observation o t ∈ O of the current state, takes an action a t ∈ A , and the environment transitions to the next state according to the transition function T . In our setting, we specifically consider sparse-reward environments where success is only determined at the end of an episode-the agent receives R = 1 for successful task completion and R = 0 otherwise. This is a standard setting in prior agentic work [9, 13, 12, 4].

C Key Agent Details

In Sec. 2, we establish an agent design that enables it to learn in-context from its own self-collected experiences. Here, we elaborate on a few key design decisions in our agent design:

Standardized prompts : we use the same simple, task-agnostic prompt templates for all tasks, rather than writing new prompts per task. These prompts are in App. E. Alternate appraoches incorporate domain-specific information into their prompts-we discuss these approaches in Appendices J.1 and K.

Two-level retrieval : We retrieve trajectories at both trajectory level (for planning) and state level (for reasoning and action selection), enabling the agent to leverage both strategic patterns and situation-specific techniques. Database D contains self-collected trajectories, and retrieval is performed at the trajectory level for the initial plan p , and in the state-level observation-reasoning-action loop for both r t and a t .

Algorithm 4 Multi-key Retrieval

Multi-key retrieval : All retrieval is performed by KNN, with similarity metric defined as the average of cosine similarities across the specified 'key' variables. For instance, in Line 3 of Alg. 1, we retrieve from D using two keys: goal g and plan p . We return similar trajectories based off the average of the cosine similarities of goals and plans when comparing each trajectory to the current trajectory. When doing state-level retrieval (Lines 7 and 10), we additionally find the most similar states within the selected trajectories via state-level key o t or r t , then return a window of states around the most similar state. This is similar to the retrieval scheme in [10]. See detailed pseudocode for retrieval in Alg. 4.

Thought-based retrieval : For the first step of a trajectory, we retrieve using the trajectorylevel keys ( g , p ) as well as the current observation o 1 (Alg. 1, line 6)-but for all subsequent steps we use reasoning r t as a key instead of observation o t (Alg. 1, line 9). This approach, inspired by Zhou et al. [11], enables generalization across trajectories with similar reasoning, and similarity across natural-language r t can be handled by generic embedding functions more easily than potentially bespoke observations o t . By retrieving at every step, we aim to retrieve the most relevant trajectories for each decision.

Generic embedding mechanism : Since g , p , and r t are all natural-language strings, we employ standard embeddings (all-MiniLM-L6-v2 [43]) that generalize across domains without task-specific engineering.

D Token Usage and Cost Analysis

We provide a detailed breakdown of token usage and costs for our self-improvement approach across all three benchmarks. This analysis demonstrates the economic viability of our method compared to alternative approaches for scaling model performance.

D.1 Per-Episode Token Usage

Table 4 summarizes the average token consumption per episode across our three evaluation domains. Token counts reflect the full agent execution trace, including both reasoning and action generation steps.

The token usage pattern reflects our agent architecture design. For any task, we require 2 requests per step (one for reasoning, one for acting), plus 1 additional request when using planning at the episode start. The variance in input token counts primarily reflects differences in accumulated trajectory history within an episode.

D.2 Per-Episode Cost Breakdown

Using GPT-4o-mini pricing (as of writing: $0.15 per 1M input tokens, $0.60 per 1M output tokens), we calculate the cost per episode for each benchmark:

Wordcraft: Cost = (5 , 047 × 0 . 15 + 68 × 0 . 60) / 1 , 000 , 000 × 8 ≈ $0 . 006

per episode

InterCode: Cost = (5 , 385 × 0 . 15 + 37 × 0 . 60) / 1 , 000 , 000 × 20 ≈ $0 . 016

per episode

ALFWorld: Cost = (3 , 706 × 0 . 15 + 30 × 0 . 60) / 1 , 000 , 000 × 61 ≈ $0 . 034 per episode

D.3 Training Database Construction Costs

Our full self-improvement configuration uses 5 parallel databases with trajectory bootstrapping and exemplar curation. For ALFWorld, we collect 3,500 training trajectories per database; for Wordcraft, we collect 4,000 trajectories per database; and for InterCode, we collect 800 trajectories per database. The worst-case cost for database construction is:

Note that these costs represent one-time upfront investments . Once the databases are constructed, the per-task inference cost remains constant regardless of how many test-time tasks are processed.

D.4 Cost Comparison with Baseline Approaches

We compare our approach against two common alternatives:

Alternative 1: Using a larger model (GPT-4o). At current pricing ($5.00/$15.00 per 1M input/output tokens), GPT-4o costs approximately:

Wordcraft: $0.20 per task

InterCode: $0.54 per task

ALFWorld: $0.57 per task

Alternative 2: Test-time scaling with multiple attempts. Using GPT-4o-mini with 3 attempts per task (assuming a perfect verifier):

Wordcraft: $0.018 per task

InterCode: $0.048 per task

ALFWorld: $0.102 per task

D.5 Breakeven Analysis

For ALFWorld (our most expensive benchmark), our method breaks even with:

GPT-4o after $595 / ($0 . 57 -$0 . 034) ≈ 1 , 110 test tasks

GPT-4o-mini with 3 attempts after $595 / ($0 . 102 -$0 . 034) ≈ 8 , 750 test tasks

D.6 Production-Scale Cost Savings

To contextualize these savings at production scale, consider a hypothetical agentic service processing 1 million ALFWorld tasks daily (a modest industry-scale throughput):

Daily cost comparison:

Our approach: 1 M × $0 . 034 + $595 (amortized) ≈ $34 , 595

GPT-4o baseline: 1 M × $0 . 57 = $570 , 000

GPT-4o-mini with 3 attempts: 1 M × $0 . 102 = $102 , 000

Daily savings:

vs. GPT-4o: $535,405 ($195M annually)

vs. 3-attempt baseline: $67,405 ($24.6M annually)

These calculations demonstrate that even modest upfront investments in self-improvement can yield substantial returns at production scale, while simultaneously improving task performance.

E Additional Implementation Details

E.1 Hyperparameters

Unless otherwise specified, we use GPT-4o-mini as our base LLM (temperature 0.1). For Fixed-DB and all Traj-Bootstrap agents, we retrieve the topk most similar trajectories at each decision step ( k = 6 for ALFWorld and InterCode-SQL, 10 for Wordcraft). We initialize each database with a small human-provided example set ( 18 for ALFWorld, 10 for InterCode-SQL, 4 for Wordcraft). With +DB-Curation, we maintain N = 5 database instances with curation every time the database size is doubled, starting with a minimum size of ten trajectories. We report success rates averaged over five random seeds. The standard deviation of the success rates is also reported. By default, we report success rate given the database at the end of the training process.

E.2 Prompt Templates

Across all benchmarks, we use standardized prompt templates for the core components of our retrieval-based ReAct agent. The same templates were used across all benchmarks with no taskspecific modifications. These templates are intentionally minimalist, focusing on providing the necessary context and retrieved examples while avoiding task-specific prompt engineering.

The templates are included below. Across all templates, the in-context examples follow the format specified in the prompt itself (for plan, the in-context examples are of form 'goal,plan', etc):

Plan:

Reason:

Act:

E.3 Retrieval Implementation

For all retrieval steps, we implement hybrid search across all the desired retrieval keys-ex. goal, plan, observation, reasoning. We return the topk examples by averaged distance across each of the keys. We implement a sliding window approach for state-level retrieval to enhance contextual relevance-we include the surrounding context (preceding and following states) up to a window of 5 steps to provide coherent episode fragments.

The retrieval mechanism is implemented using FAISS [44] for efficient similarity search as the database grows. We use exact nearest neighbor search.

E.4 Population-Based Training Details

Our database-level curation approach maintains a population of 5 database instances. Each instance is initialized with the same set of human-provided exemplars. The population undergoes curation every time the database size doubles, and performance is evaluated on the tasks attempted since the previous doubling.

The replacement strategy follows standard population-based training practices: the bottom 20% of databases (based on validation performance) are replaced with copies of the top 20%.

E.5 Quality Metric Computation

For exemplar-level curation, we track the retrieval patterns of each trajectory throughout the training process. For each task, we record: 1. Which trajectories were retrieved 2. How many times each trajectory was retrieved during the solution process 3. Whether the task was successfully completed

After completing all training tasks, we compute the quality metric Q ( τ ) for each trajectory τ as:

where R ( τ ) is the set of tasks for which trajectory τ was retrieved, o i ∈ { 0 , 1 } is the outcome of task i , and f i ( τ ) is the retrieval frequency of trajectory τ during task i .

To ensure statistical significance, we only compute the quality metric for trajectories that were retrieved for at least 3 different tasks. For trajectories with insufficient retrieval data, we assign a neutral quality score equal to the average success rate across all tasks.

E.6 Note on planning step

Following the convention from RAP [10], we omit the planning step on benchmarks with short trajectory length (Intercode-SQL, Wordcraft). This planning step is valuable for maintaining longhorizon coherence on the ALFWorld benchmarks (30 steps), and is standard in prior ReAct-based agentic work [10, 13, 12], whether the planning step is explicitly separate from reasoning, or incorporated into the first reasoning step.

F Can Self-Collected Examples Improve a Fine-Tuned LLM Agent?

We have shown that self-collected databases improve the performance of in-context LLM agents. Here, we test whether the same data can also benefit fine-tuning.

Using the OpenAI fine-tuning API, we fine-tune GPT-4o-mini on each benchmark using data from our best-performing database construction method: Traj-Bootstrap+DB+Exemplar-Curation, collected over the full training set. We fine-tune using a simple ReAct-format prompt:

We refer to the resulting fine-tuned model as ReAct-Finetune. To run the agent, we prompt it with a goal and initial observation, then alternate assistant messages (for reasoning and action) with user messages (for new observations).

Tab. 6 shows that ReAct-Finetune slightly outperforms the in-context agent on ALFWorld (0.96 vs. 0.93) and Wordcraft (0.74 vs. 0.69), while performing slightly worse on InterCode-SQL (0.79 vs. 0.82). These results suggest that self-collected examples are effective not only for in-context prompting but also for creating competitive fine-tuned agents.

G Can Databases Transfer Across LLMs?

To evaluate whether our self-improvement approach captures fundamental task structure or merely exploits model-specific artifacts, we conducted additional experiments using Mixtral 8x7B Instruct, a popular open-source model with different architecture and capabilities than GPT-4o-mini. This experiment tests a challenging transfer scenario: databases collected with GPT-4o-mini are used to improve a different model at test time, without any additional data collection or fine-tuning.

G.1 Experimental Setup

We evaluate Mixtral 8x7B on all three benchmarks, comparing the Fixed-DB baseline to TrajBootstrap+DB+Exemplar-Curation. Critically, all databases were previously collected using GPT-4omini trajectories. Each configuration is evaluated over 5 random seeds, with results averaged across trials. As seen in Table 7, despite the distribution gap between the collection model (GPT-4o-mini) and deployment model (Mixtral 8x7B), our self-improvement methods yield substantial performance gains across all benchmarks.

G.2 Comparison with In-Distribution Performance

Table 8 compares the improvements observed with Mixtral 8x7B against those achieved with GPT4o-mini (the collection model) using the same methods. The improvements with Mixtral 8x7B match or exceed those observed with GPT-4o-mini on all three benchmarks. This suggests that our approach captures fundamental task structure-such as effective exploration strategies, common failure modes, and successful action sequences-rather than exploiting model-specific quirks or artifacts.

H Can We Predict Agent Success Rates?

We have shown that increasing the number of self-collected examples improves agent performance. Here, we test whether the same examples can also predict performance on new tasks.

On InterCode-SQL and Wordcraft, task difficulty is partly observable from the goal g and initial observation o 1 . For InterCode-SQL, g is a natural-language query. For Wordcraft, g is the desired element and o 1 specifies available crafting elements. In contrast, ALFWorld task difficulty depends heavily on scene layout, which g and o 1 do not reveal. We therefore exclude ALFWorld from this analysis.

Weuse the same embedding model as in retrieval (all-MiniLM-L6-v2 [43]) to encode the concatenated string [ g ; o 1 ] . We train a calibrated Random Forest classifier to predict task success/failure, calibrating its outputs via 5-fold cross-validation with a learned sigmoid function. For each of 5 independent Traj-Bootstrap trials, we evaluate (1) the classifier's AUROC on held-out tasks, and (2) its calibration.

As shown in Fig. 5, prediction performance improves as the database grows. For InterCode-SQL, AUROC rises from 0.60 (100 tasks) to 0.77 (800 tasks). Wordcraft shows a similar trend, improving from 0.50 (100 tasks) to 0.71 (4000 tasks). In both cases, predictive accuracy increases alongside task performance.

Fig. 6 shows the calibration of the final classifiers (trained on all available training tasks). Predicted success probabilities closely match observed success rates, indicating well-calibrated models.

200

400

600

800

Num training tasks

0.00

0.25

0.50

0.75

1.00

ROC AUC

(a) InterCode-SQL

0

1000

2000

3000

Num training tasks

0.00

0.25

0.50

0.75

1.00

ROC AUC

(b) Wordcraft

0.0

0.5

1.0

Predicted Prob of T ask Success

0.0

0.5

1.0

Observed Task Success Rate

(a) InterCode-SQL

0.0

0.5

1.0

Predicted Prob of T ask Success

0.0

0.5

1.0

Observed Task Success Rate

(b) Wordcraft

I Is it Possible to Bootstrap an Agent Without Initial Hand-Crafted Examples?

Providing a small number of hand-crafted in-context examples is standard practice in the LLM agent literature [9, 13, 12, 4, 10]. However, what if we initialized Traj-Bootstrap with an empty database? In order to understand the value of the initial human-provided examples, we test Traj-Boostrap with and without the initial human-provided examples on Wordcraft. We refer to the variant initialized with an empty database as -Human-Examples. Traj-Bootstrap, initialized by default with a database of 5 human-provided trajectories for Wordcraft, achieves better performance with these starting examples than when initialized from an empty database (-Human-Examples). Performance still scales with database size for -Human-Examples-but in this case fails to reach the performance achieved via 5 human-provided examples, even after self-collecting trajectories on 4000 training tasks. On at least this one task, the initial human-provided trajectories shaped the reasoning and action patterns of the agent in a way that boosted the continual database construction process. We leave exploration of hand-crafting these in-context examples to future work.

J Key Details of Prior Agentic Approaches

J.1 How does Automanual Leverage Hand-Crafted Information

Rather than learning from self-collected examples, an alternate approach to agent construction is to leverage practitioner domain knowledge. Beyond implementing both a hierarchical learning system and code-based action spaces, Automanual [4] incorporates domain knowledge about the ALFWorld task into multiple components of the algorithm. In this section we include some code from the official Automanual GitHub to illustrate.

Observation spaces : Automanual uses a modified observation space that enhances the ALFWorld string by adding two critical pieces of information: 1) The current location of the agent, 2) What the

0

1000200030004000

Num. Training Tasks

0.45

0.50

0.55

0.60

Success Rate

Wordcraft

Traj-Bootstrap

-Human_Examples

agent is currently holding. Both of these pieces of information typically have to be deduced from the trajectory of previous observations and actions, but Automanual tracks them explicitly:

Action spaces : In ALFWorld, any task typically involves three main components: 1) Searching for an object, 2) Performing an action with the object (heating, cooling, cleaning, etc.), 3) Placing the object somewhere. Automanual significantly simplifies both the search and placement operations by providing multi-action helper functions within its code-based action space:

J.2 A note on training and test sets

The distinction between how different techniques leverage data is crucial in understanding the generalization capabilities of LLM agents. We can categorize existing approaches based on how they treat training and test data:

Single-Task Optimization : Some approaches focus exclusively on improving performance on a single task instance without concern for generalization. For example, Shinn et al. [25] leverages feedback from failed attempts to incrementally improve performance on the same task. Similarly, search methods [22, 24] expand the solution space for a specific problem instance. While these approaches can solve individual tasks, they don't transfer knowledge across different problems, essentially 'overfitting' to a single instance.

Mixed Train-Test Evaluation : Some recent work blurs training and test boundaries. For instance, RAP [10] makes multiple passes over the same dataset, allowing the system to 'learn' from some test examples before evaluating on others within the same set. This approach does not assess true generalization capability, as the model has indirect exposure to the test distribution during its learning phase.

Full Train-Test Separation : Several papers maintain a clear separation between training and test data: 1) ExpeL [13] extracts general rules from a training set of trajectories and applies them to entirely separate test tasks, 2) AutoGuide [12] generates contextual guidelines from training experiences that are evaluated on distinct test scenarios. 3) AutoManual [4] constructs hierarchical 'manuals' from training interactions that are then applied to novel test tasks.

Our approach similarly ensures that trajectories used for database construction come exclusively from designated training tasks, with evaluation conducted on a separate set of test tasks never seen during the database construction phase. This separation is essential for validating that the knowledge captured by the agent generalizes to new problems rather than memorizing specific solutions.

K Comparison to Hand-Crafted InterCode-SQL Agent

We further contextualize the performance of Traj-Bootstrap by comparing to two hand-crafted agents on InterCode-SQL. The Intercode-SQL paper [38] provides a hand-crafted agent, GameSQL to solve the task, and optionally provides the agent with a 'handicap'-giving the agent information on all relevant parts of the database schema. We denote the assisted version as GameSQL+Cheat. Neither agent provides in-context examples, and both share a bespoke, hand-crafted prompt (see App. N).

As seen in Tab. 9, Fixed-DB performs similarly to GameSQL (0.74 vs 0.73), and the performance of our best method, +DB+Exemplar-Curation, approaches the performance of GameSQL+Cheat (0.82 vs 0.84). Therefore, our database-construction techniques lift the performance of a generic ReAct-style agent nearly as much as the lift provided to the hand-crafted agent via 'handicap' access to the database schema.

L Benchmark Details

L.1 ALFWorld

ALFWorld [37] is a text-based environment that aligns with embodied tasks, allowing agents to navigate and manipulate objects through textual commands. We use the standard ALFWorld benchmark consisting of 3500 training tasks and 134 out-of-distribution test tasks across 6 task categories:

Pick & Place: Find and move objects to specified locations

Clean & Place: Find, clean, and place objects

Heat & Place: Find, heat, and place objects

Cool & Place: Find, cool, and place objects

Pick Two & Place: Find and move two objects to a specified location

Look at Object: Find an object and examine it under light

Following [10], for the ALFWorld benchmark we perform similarity search over task categories in addition to the other retrieval keys (goal, plan, observation, action). We do this to follow the convention in this prior work.

For our initial human-provided exemplars, we used the 18 successful trajectories (3 per task category) provided by Zhao et al. [13]. These trajectories were used to initialize all database instances.

The success criteria for ALFWorld tasks are defined by the environment and require the agent to satisfy all conditions specified in the goal. For example, in a 'Heat & Place' task, the agent must find the target object, place it in the microwave, turn on the microwave, and finally place the heated object at the specified destination. Both Autoguide and Automanual allow 50 actions for task completionbut choosing to employ "reasoning" counts as an action. Since we force our agent to reason at every step, we allow our agents (Fixed-DB, Traj-Bootstrap and variants) only 30 steps for task completion (on Autoguide and Automanual, the agent does not reason in practice at most steps ex. in a search procedure).

For this benchmark, we do not provide an action space string to the LLM, relying purely on the in-context examples to communicate the action space.

L.2 InterCode-SQL

InterCode-SQL [38] is an interactive coding environment for evaluating language agents' SQL programming abilities. We use a subset of the InterCode benchmark focusing on SQL query generation, built upon the Spider SQL dataset. Of the 1034 tasks in the dataset, we randomly assign 800 tasks to train and the remaining 234 tasks to test.

Each task provides a natural language query request. The agent must generate a syntactically correct SQL query that retrieves the requested information. The agent must first execute queries to understand the database schema. The environment provides feedback on syntax errors and execution results, but the agent is only allowed to submit a solution once.

The success criteria for InterCode-SQL tasks require the agent to submit a solution query within 10 steps. The environment executes the query and compares the results against a ground-truth reference.

For our initial human-provided exemplars, we collected 10 human-created trajectories for 10 randomly-selected training tasks. These trajectories were used to initialize all database instances. For all solved trajectories, we append the solution query to the goal string-since the goal of the task is to 'discover' this query through interacting with the SQL database.

We used the following action space string for InterCode-SQL:

Your action space is outputting valid mysql commands to solve the sql task. You will be evaluated on the Latest Standard Output. If you believe the latest observation is the final answer, you can complete the task by running 'submit' by itself. You have 10 iterations to solve the task. Follow the syntax and logical flow from the provided examples exactly.

L.3 Wordcraft

Wordcraft [39] is a simplified adaptation of the game Little Alchemy, where agents must combine elements to create new elements through multi-step processes. We randomly select 4000 training tasks and 500 test tasks from the subset of tasks requiring up to 2 steps to solve, with the train-test split separating the tasks into disjoint sets of goal elements.

The agent starts with a set of elements and must discover combinations that creates a particular target element specified in the goal. The environment provides feedback on successful combinations and updates the available elements accordingly.

The success criteria for Wordcraft tasks require the agent to create the target element within 4 steps, while the minimum solution length is up to 2 steps.

For our initial human-provided exemplars, we collected 4 human-annotated trajectories from randomly-selecting training tasks. These trajectories were used to initialize all database instances. We collected fewer initial trajectories for Wordcraft than for InterCode-SQL (4 vs 10) since Wordcraft is a slightly simpler task, requiring up to 4 steps for task completion while InterCode-SQL requires up to 10.

We used the following action space string for Wordcraft:

Output strings with the names of the two entities we would like to combine in this step.

L.4 Note on Benchmark Selection

We selected three sequential decision-making benchmarks that cover different reasoning challengesALFWorld [37] tests text-based navigation and object manipulation, InterCode-SQL [38] tests interactive code generation, and Wordcraft [39] tests compositional reasoning.

While prior works [13, 12] test on WebShop [45], we encountered bugs in generating achievable goals on the full benchmark (confirmed by https://github.com/princeton-nlp/WebShop/issues/43) and identified tasks with incorrect rewards.

We excluded QA benchmarks (HotPotQA [46], etc.) because performance depends on information retriever quality and LLM self-evaluation efficacy, two factors that would confound our study of LLM Self-Improvement. We plan to test our algorithms on QA benchmarks in future work.

M Computational Resources

All experiments were conducted using the following computational resources:

1 NVIDIA A5000 GPU (24GB memory) for embedding computation

64GB RAM

The majority of computation was spent on OpenAI API calls for the LLM-based decision-making. Database operations including embedding computation, storage, and retrieval accounted for less than 5% of the total computation time.

For embedding computations, we used all-MiniLM-L6-v2 [43].

For LLM inference, we used the OpenAI API for GPT-4o-mini, which required approximately:

2,000,000 API calls for ALFWorld

200,000 API calls for InterCode-SQL

500,000 API calls for Wordcraft

The total cost of API usage was approximately $3,000 USD.

N GameSQL Prompt

Yang et al. [38] write this hand-crafted prompt for the GameSQL agent:

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Abstract and intro clearly state the contributions, which match the experimental results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See App. A

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Information provided both in Sec 6 and in several sections of the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code is attached in Supplemental, and will be made publicly available. Benchmarks are publicly available.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Details across Sec. 6 and Appendix E

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All tables have standard deviation of success rate across 5 trials.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Information in Appendix M.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See App. A

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Paper offers an algorithmic framework for self-improving LLM Agent, existing safeguards for LLMs themselves apply when an LLM is used within the agentic system.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All used models and benchmarks are cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets. The paper does include code to replicate experiments in the supplemental, but this code is not a core asset introduced by the paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: Paper describes usage of LLMs as a core component of an LLM Agent.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an exemplar curation approach for LLM agents in sequential decision making tasks. The proposed approach views the training of LLM agents as exemplar curation, where a growing list of exemplars are curated using two complementary methods, DB-curation,  and exemplar-curation, to maintain a database of retrievable successful examples that the agent has attempted. The proposed approach is evaluated on three sequential decisionmaking tasks, showing improvements competitive with the state of the arts. Ablation studies validate the effectiveness of proposed curation approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The proposed approach shows strong performance across all evaluated tasks.\n- The idea of training as growing a database of retrievable knowledge is interesting, and salient for LLM agents, which are particularly effective in retrieval-augmented usage, and cannot easily be finetuned. This is also related to the idea of episodic memory in cognitive science, which is an interesting connection.  \n- The components of the proposed approach, including the curation approaches and multi key retrieval method, are well designed.\n- The method appears to make only minor task-specific modifications to ReAct, making it not overly complicated, and generally applicable.\n\nWeaknesses\n- The method appears computationally expensive, especially when maintaining N(x2, when using both curation approaches together) databases, and training over hundreds or thousands of examples. Computational costs (including the training as well as retrieval costs at every step), are not analyzed thoroughly, so we cannot be certain that the costs are justified.  \n- From the performance results, it appears that significant gains come from accruing a potentially critical mass of training exemplars, enough to cover the test distribution. For example, while ALFWorld is complex, its test set consists of 134 examples across 6 task types. I would not be surprised if the retrieved exemplars are extremely similar to the actual test task. That said, I think it would still be an interesting finding if this were the case. \n- The results are limited to GPT4o-mini, limiting generalizability.\n\nQUESTIONS:\n- Does the agent see few-shot exemplars aside from the ones it retrieves?\n- Can you provide examples of retrieved exemplars? It would be useful to show how the retrieved exemplars collective shape the agent's policy.\n- Is the high level plan fixed once generated? How important is it for the initial plan to be correct?\n- Is any tradeoff for efficiency (e.g. not retrieving at every step, retrieving only once every k steps) possible?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an approach to improve LLM Agent by retrieving and curating existing trajectories. The proposed method is based on first collecting successful agent trajectories, then improve the data quality by database-level curation or exemplar-level curation. Experiment shows that each proposed component is effective on ALFWorld, InterCode-SQL and Wordcraft in terms of improving the success rate of LLM after seeing more training trajectories.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The experiments include full ablation study of each single component in algorithm, making it easy to interpret the method and results. \n- The paper writing is clear and the result presentation is good.\n\n## Weaknesses\n- The novelty of the proposed method seems to be limited. The main idea of this paper is similar to a COLM paper [Xiao 2024], which also improves LLM agents by distilling knowledge from past trajectories, and also does automatic task decomposition + learning from failed trajectories. From the results on ALFWorld, these two methods are on par, but [Xiao 2024] uses fewer training trajectories in total. It would be nice if the authors can justify the novelty, uniqueness and advantages of the proposed method, and do some experiments to compare with them.\n- Regarding test-time costs, it would be nice if the authors can also provide details of total number of input tokens and additional computations required for retrieval, in comparison to baseline methods. \n\n\n\nXiao et al. O3D: Offline Data-driven Discovery and Distillation for Sequential Decision-Making with Large Language Models. COLM 2024.\n\nQUESTIONS:\n- Is it necessary to distinguish database-level and exemplar-level curation? Is it okay to simplify the organization and just do atomic-level data curation with a unified framework?\n- Table 2 uses different LLM models for baselines and proposed method, it would be nice to make them consistent. \n- Table 2 show the bootstrap method's performance with gpt-4o-mini. Can the authors include some results from other models (e.g. open-sourced models) to justify the generalizibility of the method?\n- Is the retrieval on trajectory-level or sub-trajectory-level?\n\nLIMITATIONS:\nLimitations are discussed in Appendix A, about the method may be less effective with weaker models.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method for improving large language model (LLM) agents on sequential decision-making tasks by leveraging self-generated, successful trajectories as in-context examples. Instead of relying on human-engineered prompts or domain-specific action/observation design, the authors introduce a trajectory bootstrapping approach that enables agents to iteratively build and refine a database of successful trajectories through autonomous experience. Key innovations include:\n\t•\tTraj-Bootstrap: a simple accumulation of successful task completions.\n\t•\t+DB-Curation: database-level curation using population-based training.\n\t•\t+Exemplar-Curation: exemplar-level selection of trajectories using a value metric based on empirical utility.\nExtensive experiments on ALFWorld, Wordcraft, and InterCode-SQL demonstrate notable performance gains, even surpassing methods that use more powerful LLMs or handcrafted components.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The paper offers a practical and scalable alternative to manual prompt engineering, validated through solid empirical results. The bootstrapping and curation strategies are clearly described and effectively evaluated across diverse tasks. The proposed method offers a general, task-agnostic mechanism that could be widely useful in agentic settings. It shows that high performance can be achieved even with lightweight models through data-centric optimization.\n\n\nWeaknesses: The methodology is well-implemented but somewhat straightforward, relying heavily on repeated LLM calls during training. While empirical gains are strong, theoretical insights are limited. The structure of the paper is confusing. Notably, placing the Problem Statement before Related Work breaks conventional scientific writing flow. This likely reflects an incomplete or unpolished draft and disrupts the narrative logic for readers. The novelty lies in the reuse and filtering of examples, which is more incremental than transformative compared to prior ReAct-style agents and retrieval-based pipelines. The overall conceptual framework builds heavily on existing ideas (ReAct, retrieval-based LLMs, in-context learning), and the innovation is mainly in implementation rather than theory or modeling.\n\nQUESTIONS:\n1\tPaper Structure Revision: The placement of Related Work after Problem Statement is unconventional and makes the reading flow less logical. Consider reorganizing the sections to follow the standard order: Introduction → Related Work → Method → Experiments.\n2\tGeneralization to New Tasks: How does the system handle domain shifts or novel goals that were not part of the training task distribution? The reliance on accumulated in-context trajectories might hurt generalization.\n3\tTraining/Test Gap Concerns: While the paper aims to reduce hand-crafted engineering, the method still depends on hand-curated starting examples and extensive self-training—how is the potential gap between training and deployment (e.g., in online settings) mitigated?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides extensive experiments consistently showing that using only successful self-generated trajectories as in-context examples can cause an increase in performance. Furthermore, they provide a dataset curation method (DB+) to store the most relevant successful trajectories that can be used with multiple reasoning algorithms.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe authors use a trajectory bootstrapping method, DB+, that samples successful self-generated trajectories to be used for self-improvement for sequential decision making\n\nLots of ablations to show consistent increased performance with successful self-generated examples across different algorithms, tasks, and models.\n\nThe idea of predicting agent success in Section 6.4 is novel\n\nLine 155 about collective properties is I think a significant benefit of this approach as it relates to the coverage of situations that the agent has examples to rely on\n \n**Weakness**\n\nMy main problem is that the paper discards the failed trajectories and leaves how to use them as future work. I understand their message is that performance can improve with just successful trajectories. However, I am worried about the inefficiency of LLM calls when discarding the failed ones. Basically my question is if the improvement is worth the amount of LLM calls? Table 2 shows a 5% increase from 100->3500 training tasks. I understand Figure 2 shows that in some cases only 1000 training tasks are needed to reach the performance of 3500 training tasks. Furthermore, in Figures 2 and 3, of these training tasks, how many of them have failed trajectories that are discarded?\n\nConsider also citing some In-Context RL papers such as [1], which also looks at multi-step decision making.\n\n[1] Dai, Zhenwen, Federico Tomasi, and Sina Ghiassian. \"In-context Exploration-Exploitation for Reinforcement Learning.\" arXiv preprint arXiv:2403.06826 (2024).\n\nQUESTIONS:\nSee Weakness 1\n\nLIMITATIONS:\nSee Weakness 1", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}