Self-Challenging Language Model Agents
Authors: Yifei Zhou
Sergey Levine
Jason E Weston
Xian Li
Sainbayar Sukhbaatar
Venue: NeurIPS 2025 poster ()
Keywords: LLM Agent
self-improvement
multi-turn tool-use
Primary area: deep_learning

Abstract: Large language models  are quickly becoming the foundation for intelligent agents that are capable of using tools. However, training such agents is challenging because it requires human creation and annotation of a diverse set of tasks, tools, and evaluation criteria. In this paper, we propose  the Self-Challenging Agent framework for training an agent on high-quality tasks that are generated by itself. The agent first plays the role of challenger and generates a task after interacting with the given tools. The tasks take the form of a novel general class of problems termed Code-as-Task, which are defined by an instruction, a verification function and solution and failure cases which serve as tests, allowing to filter only for high-quality tasks.  
The agent then takes an executor role and trains on those tasks with reinforcement learning using the evaluation feedback as a reward.  We show our method improves the performance of Llama-3.1-8B-Instruct on two existing multi-turn tool-use agent benchmarks, M$^3$ToolEval and TauBench, with a two-fold average success rate increase, despite using only self-generated training data.

Self-Challenging Language Model Agents
Yifei Zhou"
Sergey Levine UC Berkeley
Jason Weston FAIR, Meta
Xian Li FAIR, Meta
Sainbayar Sukhbaatar? FAIR, Meta
UC Berkeley
Abstract
Large language models are quickly becoming the foundation for intelligent agents that are capable of using tools. However, training such agents is challenging diverse set of tasks, tools,
because it requires human creation and annotation and evaluation criteria. In this paper, we propose the Self- Shellehhllngggngg framework for training an agent on high-quality tasks that are generated by itself. The agent first plays the role of challenger and generates a task after interacting with the given tools. The tasks take the form of a novel general class of problems termed Code-as-Tass Task, which are defined by an instruction, verification function and solution and failure cases which serve as tests, allowing to filter only for high- quality tasks. The agent then takes an executor role and trains on those tasks with reinforcement learning using the evaluation feedback as a reward. Evaluation on two existing multi-turn tool-use agent benchmarks, T and TauBench, shows the Self-Challenging framework achieves over a two-fold improvement Llama-3. -8B--.rrrrttItt despite using only self-generated training data.
Introduction
Large language models (LLMs) have demonstrated remarkable capabilities across wide array of complex tasks 20
positioning them as promising agents for autonomous decision-makee
in open-ended environments, such as multi-turn tool use 46 35| and GUI navigation [[] 41, [50]. Reinforcement learning (RL), which is already an integral part of tratiiraining of LLMs, has emerged as powerful tool for enhancing multi-turn agentic capabilities for LLMs by directly optimizing the final objective through interaction-base feedback [52
51
An essential ingredient for applying RL to train a generally capable LLM agent a large pool of high- quality tasks for agents to learn by trial and error [5]] [22]. However, relying on human annotators for task creation is inherently costly, labor-intensive, and ultimately not scalable [51] 38 47 thus underscoring the need for an automatic and reliable pipeline for synthesizing tasks. While prior works have built environment--peciffcc pipelines for synthesizing tasks to perform RL, such as collaborative designs (53), web navigation [51]. and fixing Github issues 22 14, general task synthesis pipeline for multi-turn LLM agents [37 in open- open-ended and tool-rich environments is still missing.
To come up with synthetic task generation pipeline for general multi-turn tool-use LLM agents, can start by reflecting on how human annotators design tasks given access to a set tools and the environment. They will first try the tools and interact with the environment to see what kinds of goals are possible to achieve and what would happen if these goals were achieved. The tasks would then be sent the LLM agent to see if it can achieve the goals by the human annotators.
Motivated by this intuition, we propose a Self-Challenging Agent framework to construct synthetic tasks for training general multi-tum LLM agents scale. In this framework, shown in Figure [] the agent performs two distinct roles. In the challenger role, the agent first interacts an unknown environment with tools to gather information and generate possible tasks. Subsequently, these
Work done at FAIR, Meta. Correspondance to yifiii/hou@brrklle...d 1 Equal advising.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Verifier
Challenger
siinn Function
Action
Access enfronerent state
play around enstronmen: construst task.
Environment
VSildbre finol znower
crase
access following toobe
Observation
General tools
saaass eucess
gecurnser censtut musnr
Iuroma svenes une
python
Accos to APIs chroughh fanction
return sacunss
Instruction
Bronser
Navigate
information
Exchange the kastle
lasoit order to plaso restiral
Qvery the damause through APIS
Modily the deeabows through APIs
Action *dl gelLewer, mitall
Executor
Human
User
Juk confermatine before actions
You need complete the goall (inttruction]
Observation G ('ordeed" ("er, "xy'll
danficatuum
Reward 0/1
You have access to the following tools...
Figure I: Overview Ceff-Challenging Agent. The agent takes on two roles: task challenger, and task executor. The task challenger proposes task along with verification method to verify the solution to the task. The task executor generates a solution and obtains reward from the environment based on the verification method.
synthetic tasks are tackled by the agent in the executor role, who attempts to accomplish them by interacting with the same environment. However, ensuring generated tasks are (1) feasible, (2) verifiable, and (3) difficult becomes increasingly complex in such scenarios. If the challenger agent inadvertently generates flawed or impractical tasks, risks introducing substantial noise into the training data, thereby contaminating and destabilizing the learning process of the executing agent.
To enable the ereation of high quality generated tasks we introduce the "Code-as-Tace " (CaT) formalism, wherein each generated task comprises four explicitly defined components: an instruction, verification function, an example solution, and explicitly enumerated failure cases, where all components except the instruction are expressed in code due to its generality and expressivity 44 [5] 377. This structured representation ensures task feasibility, difficulty. as exemplified by typical failure cases, and task verifiability through rigorous, executable verification criteria. By leveraging external code executors, our framework automatically filters out erroneous or impractical tasks, significantly mitigating risks associated with low-quality tasks.
exemplified by provided solutions, task
Our experiments are conducted in four different tool-use environments from V'ToolEval [37] and Tau-Bench [46] spanning tool-based calculations, web browsing, retail services, and flight booking. We apply our method generate synthetic tasks and rely purely on these synthetic tasks to fine-tune the LLM agent before evaluating on the existing out-of-distribution test tasks. Empirically, we establish the advantages our Self-Chaid Self-Challnnging Agent (SCA) framework in two important settings: distillation, where the goal is to distill the expertise of a stronger model to a weaker model without any existing tasks, and celf-improvement, where the absence of a stronger model the weaker model needs to supervise itself to make progress. For distillation, we are able to improve the student model 1 -88--tnrre (34 by 20.2% in terms of absolute average success rate across all 4 environments without any existing tasks. Even in the absence of a stronger model, through RI. training on its own synthetic tasks and rollout trajectories, SCA training able to double the success rate from 12.0% to 23.5%, outperforming the prior state-of-the-ar self-improvement method for LLM agents by a wide margin.
2 Related Work
Seff-Challenging agents. The idea of an agent self-synthesizing tasks has been explored before the emergence of LLMs. Notably, Asymmetric Self-Play (ASP) 33 21 lets one agent act in the environment first, followed by the second agent who aims to reach the same end state. The adversarial nature of their reward encourages the first agent to act in ways that are difficult to imitate. While

---

this play idea has been historically successful in relatively constrained domains like robotics simulations B 21, math reasoning [6], or even to single-turn instruction following tasks 4 47. it has not been studied in terms of how to generalize ASP to open-ended and partially observable tool-use agent environments while preserving task quality. A concurrent work [49] adopted ASP to generate coding tasks with execution verification. In this work, we propose the Tode-as-Taske formalism and thus utilize external code exccutors to ensure the synthetic tool-using tasks are feasible, verifiable, and difficult.
Autonomous improvement LLM agents. Recent studies have found that LLMs can be prompted to evaluate the outcomes of LLM agents by examining their trajectories [23] 9.. Using autonomous
LLM evaluations as reward feedback, RL [6 2 or hindsight relabeling [19 18 5 can be applied to improve the performance of LLM agents without human supervision. This self- welf-improvement
loop can be further enhanced through self- self-generated instructions [88 to enhance the diversity of the skills acquired [51 32 TO 24 48, 5I.. However, while autonomous evaluations and task proposers can work well for relatively basic tasks, they do not have explicit mechanisms to improve task quality and ensure its verifiability. In contrast, our work introduces novel framework for task challenger agent to explore the environment before generating tasks and grounds the tasks in the Code-as- Bask format to significantly improve the task quality by leveraging an external code executor. In our experiments, we show that these design decisions lead to significant improvements in both the settings.
distillation and self-improvemt
Code-as-action for tool-use LLM agents. Equipping LLMs with external tools can not only augment ext-gennraation and information information-gathering capabilities (27 25 17 8 42
30 but also enable them to take actions in the real world [46] 335 i5 33). While early explorations have designed
heterogeneous interfaces for LLM agents to use tools such as JSON or text [16 45
25 recent
studies have found that code can be used as a unified general interface for LLM agents to interact
with API functions 33 44 337, navigate through the web (31, make travel plans [4), and even
control robots in the physical world 15 [3]. However, using code as a universal interface for LLMs to autonomously generate synthetic tasks has not been studied. Our work proposes novel Code-as- Task format that formalizes tasks consisting of instructions, verification functions, example solutions, and failure cases, to ensure the quality of the tasks in open-ended, tool-rich LLM agent environments.
3 Problem Setup
In the standard L.M agent setting, the agent has the goal of executing a given task. In this executor role, the agent interacts with the environment that contains multiple tools, such as a code interpreter, web browser, and even human users, as shown in Figure [ Let us represent the agent by the executor policy = At step i, the policy will output an action a;. which can be a piece of code calling a tool API, question the user. Then the response from the tool or the user 0 will be observed and appended to the context of the LLM when sampling the next action
0 ~ π - (-100.++1, (1:), 8141 + T(5,, az).
(1)
The action will also have an effect on the enviromment and change its state st (i.e. database entries),
represented by a transition function T here. Here the initial observation Og can contain useful information like API documentation. can be further clarified by conversing with the user. Once the agent completes the task after T steps, a verification method, e.g. a function that checks the state of the environment and the solution from the agent, will be executed evaluate the success of the solution and issue reward to the agent:
can also contain instructions about the task c E C, which
E R. As in principle the interactions between a language model agent and any external environment can be considered as a form of tool use, this formulation of multi-tun turn tool-use language agents encompasses most LLM agent tasks (37], such as agentic coding 14 22). web browsing [3 [0), and even interactions with users 53 46 In this paper, however, we assume an unsupervised training setup where we do not have access to the ground-truth task set C and its verification functions R. Instead, Self- Self-Challnnging allows the LLM agent to construct its own tasks and their evaluation functions for RL training. We will detail our method in the following section.
Self-Challenging Agents
We now describe our Self-Challenging framework for training L.MM agents. Different from the standard LLM agent setting, our method enables agents with two roles: (1) As task challenger, it
3

---

Instruction user)
Example Solution
Your name Olivia Nyayan and your emall civae77A@@wwamp..coom For #W112 return the Skatebuard seypal ??.
searr_ddlliveed_orrde lles (orderrrrer.dd 140112',
itex_ida ("6843647869]]
paynent rothod "ereoddddd""ppp"""""""""" paypp pappyyall777
Verification function
Fallure (one three)
#valuste()
True
f Fasture Gade 1, seong paysent Retked
order
get_crder_ detasia
boaarn_dlliveed.deo oreerrrrteeaaaaee (order _1d "#2112",
success ouccesa and \
itea_ids ('6843847669
ordey ("retors_ 1 [0]] -- '6843647669'
payzest, www.ooteetthdd.... credit 77*)
succиa = auccesa und /
ordor ('roorr... payaaon payaont, 3ethhod,_11
"paypal_77-
raturn ROCCORD
Ensures the task feasible and verification fundtion
Ensures non- and the vertlication can detect fallures
REEECT
True
Palse
Figure 2: An example of synthetic Code-as-Tass Task (CaT) generated by the task challenger, in TauBench-based environment [46]. 'he task challenger interacts with the environment taking a series of actions calling different tools gather information, before generating the synthetic task, consisting of an instruction, verification function, example solution and failure cases (shortened in the figure for brevity). Automatic filtering is applied CaTs to keep only valid tasks where the example solution can pass the verification function and the failure cases cannot.
interacts with the environment and self-synthesizes tasks in our novel Code-as-Task format in order make high quality and challenging tasks for training. (2) As task executor, is trained on these self-synthesized tasks in order to learn to solve real tasks at test time, for which there is less available annotated data. We will describe each in turn.
4.1 Task Challenger
In the task challenger role, the agent interacts with the environment with the end goal of constructing a viable task. Accessing the tools in the environment allows the challenger to gather information about available tools and probe what type of tasks can be performed. Let 1a be the policy of the task challenger. Similar to =xee, the challenger takes a series of actions calling different tools: a++1 //tn (- a1 At the end, however, it must output task specification that includes verification function: è, Ra (- ao:T-1). In this manner, we collect set of synthetic tasks č { ) on which we can train the executor agent. While the distribution of the generated tasks might not exactly match the real-world tasks C, training on Ć could lead to improved performance on C as long there common similarities and shared skill-sets. In the next section, will explain how the class of tasks are structured, and how evaluation done to obtain rewards.
4.2 Code-as- (CaT) Class of Tasks
As described in the previous section, in order for the task challenger policy glask to generate high- quality tasks, I usk itself also needs to be a reliable agent to take a series of actions to gather more information about the environment and the unobserved states. As this not trivial especially if the tasks need to be challenging, we expect - to sometimes generate tasks that are either infeasible or ambiguous. Incorporating these tasks for training the exccutor policy geses will reduce the signal-to- noise ratio and contaminate the training process.
We thus propose the Prode-as-Tasse (CaT) class of tasks which help in filtering out such low quality tasks. As depicted in Figure 2, CaT's leverage code as an interface to automatically evaluate the quality of synthetic tasks. The basic components of CaT synthetic task include instruction and verification function. The verification function R will provide a sparse 0/1 outcome reward based on whether the final state and answer aT passes the verification function. However, false negatives (i.e. the agent gets a reward of O without doing anything wrong) and false positives (i.e. the agent gets reward of without actually fulfilling the request) can easily occur when task hallucinates during task proposals. To address these issues, we introduce additional components: an example solution and three failure cases. These are automatically checked against the verification function to ensure that tasks are feasible yet non-trivial and the verification function works properly. We observe in experiments that this makes our proposed tasks significantly higher quality. We include a specific prompt (see Appendix Figure 9 for an example) as an input lask so that will adhere to CaT and output all four components. A full output example of CaT is provided in Appendix Figure 15

---

4.3 Task Executor
Once high quality tasks for the open-ended environment are generated via the CaT formalism, they can then be directly used for training the task executor. We explore two possibilities in how these tasks can be helpful for training the executor agent. We first discuss the self-improvement setting where π _h improves itself with its own rollout trajectories through RL. We also explain an additional possibility of distillation in the presence of a stronger teacher model via the synthetic tasks.
Self-improvement. We consider the self-improvement setting where both - and π n are the same LLM. In this setting, after deriving a set of high-quality tasks é through CaT generation, we use the same LLM x to collect trajectories and annotate them with rewards calculated from verification functions R included in the tasks and apply RL for optimizing the policy. In particular, we applied one-step REINFORCE [39], the most basic RL optimization method, that minimizes:
T log Th (a.), (000000::11111 a0:1-1
L
Σ E Eaggrr éEC
RA(ST, or)
Σ
(2)
Because of our reward structure with 0/1 outcome reward, this RI. optimization objective is essentially equivalent to performing supervised finetuning (SFT) on successful trajectories (i.e. Ra(ST, (e(ST,ar) = 1)
only, i.e. Rejection Fine-Tuning
Distillation. We then consider an additional possible use of the synthetic tasks distill the domain- specific knowledge of a stronger LLM into a weaker LLM in a specific open-ended environment without any additional human data. This setting could be particularly useful when we have a larger generalist LLM that would be too costly or too slow to deploy, so we would like an automatic process to distill the capability of the larger model in this environment
smaller LLM that would be of high-quality tasks C through CaT generation, we use the stronger LLM to sample dataset 2 containing trajectories {oo.T, ao:r) on those tasks. We then apply SET with the cross entropy loss for training the weaker student LLM:
cheaper and faster to deploy. In this setting, after creating
= E (on: JED
log n 200e (at (0ort, (0:t-1)
While one alternative is train only on successful trajectories, our preliminary experiment results show that even failed trajectories from stronger model can be beneficial for weaker model to learn from, such as reflections on mistakes.
5 Experiments
We evaluate the Self-Challenging Agent (SCA) framework in actual tool-rich and open-ended LLM agent environments. Specifically, we use it to train an agent in four different environments and in both self-improvement and distillation settings using synthesized data only. The performance is then measured on actual real test tasks, and compared against baselines. Finally, we provide ablation and analysis experiments including data scaling results.
5.1 Environments
Our experiments are conducted in two multi-turn tool-use LLM agent benchmarks, featuring tasks from four different environments that come equipped with functional verifiers for reliable evaluations. In both environments, we instruct the agent to first output its thoughts and then follow them with a block of executable Python function calls or a response to a simulated human user within the environment. The execution results of the executable code or the simulated response from the user serve as the observations for the next step. We limit the maximum number of steps in each environment to be 15. The prompts used to apply our method to these environments are included in Appendix D and more details about the environments can be found in Appendix A
Va 37 is multi-turn function-callin benchmark where the success of each task is the agent's final answer with the reference solution. We report the performance in both the Calculation and Web Browsing environments with existing test tasks from ToToolEval. The Calculation environment involves using tools from a Travel Planner, DNA Sequencer, Message Decoder, and Trade Calculator (approximately six tools in each domain) to perform calculations. To use SCA in this environment, the task challenger applies the given tools to create new tasks, generates the verification function to be pattern-matching with reference solution,
determined by
5

---

Table 1: Main results. We validate the effectiveness of Self- Seff-Challenging Agents (SCA) in both the distillation and self-improvement settings compared to zero-shot LLMs and PAE [5T] baselines. For both settings, we generate 800 synthetic tasks and 12k offline rollout trajectories. Pass@ @ 1 results averaged over four independent trials, and pass @4 is calculated from the same four trials. The best results in each setting are in bold.
ToolEvll
Tau-Bench
Average
Caleulation
Web Browsing
Retail
Airline
Pass@1
Pass@1 Pass@4
Pass@4
@
Pass@4 Pass@1 Pass@4 Pass@1
Pass 4
Zero-Shof
GPT-4o
72.4
85.4
52.9 70.6
18.7
37.4
15.8 36.7
40.0 57.5
Llama-3.1-708
57.3
79.2
64.0
79.4
34.6
56.5
9.2
20.0
41.3
58.8
Llama-3.1-8B
20.3
43.8
16.2
42.1
8.9
15.7
2.5
10.0
12.0
27.9
Distillation
1lama-3.1-8B-PA..
44.3 77.1
45.6 76.5
23.7
40.9
6.7
13.3
30.1 52.0
L-B---3.I-BB-SCA
43.2 72.9
50.0 82.4
28.9
48.7
6.7 23,3
32,2 56.8
Self Self-Improve
Llama-3.1-8B-PAN
27.6
54.2
14.0
38.2
9.8
18.3
0.0
12,9
27.7
1lmma-31-8B--CA.
31.8 62.5
44.9 67.6
13.0
25.2
10.0
23.5 41.3
provides an example solution series of operations that can arrive at the reference solution, and defines failure cases as incorrect answers.
Tau-Bench [46] is multi-turn customer service environment where the LLM agent needs to interact with user (simulated by GPT-4o (20). query the database, and make corresponding modifications to fulfill the user requests. It is composed of Retail and Airline environments corresponding to customer support for E-commerce service and flight booking services. Around 15 tools such as "book fookfffhhhh and "exchange_ eereee__rrr____eee order_items" are available for each environment. When user response is needed, we use the same model as the backbone of the agent ---aaa...------tttee to simulate the user instead of GPT-4o as used at final evaluation time.
5.2 Main Comparisons
Experiment Setup. We conduct our experiments by RL fine-tuning Llama-3.1 1.mma-3.11880 1-t.a....--B---tttttte ("Llama- 3.1-8B" for short) [4] the base model xx00 In both distillation and self- seff-improvement settings, use the same fixed 1.ma--3.1-BB model as the task challenger task In the self-improvement setting, all training tasks and trajectories generated by Alama-3.1-8B itself. In the distillation setting, we use larger Llama-3. 1-70B-Instruce ("Llama-3.117000 demonstration trajectories. We compare SCA with several baselines in different settings. Firstly, we include the zero-shot performance of Llama-3.1-80 and Llama-3. 1-BB to study how much performance gain SCA can produce from the pre-trained checkpoint, and the zero-shot performance of GPTT-0 as a reference for advanced proprietary models. We also compare to the Proposer- Agent-Evaluator (PAE) [51] baseline, prior state-of-the- for task synthesis and autonomous improvement, which has been shown to be effective for open-ended GUI navigation agents. In PAE, instructions are proposed autonomously by prompting 3. --BB with the initial observation, such as the API documentation of available tools or the initial web page for web browsing. Success is also judged by prompting the same model, Llama-3. 3.1-BB.. with the instruction and the rollout trajectory. The prompts for both the PAE task proposer and evaluator are included in Appendix D The crucial differences between PAE and SCA are: (1) the task proposer in PAE generates tasks directly from API documentation and the initial observation instead of serving as an agent that actively interacts with the environment to gather information before creating the task; (2) PAE only generates instructions while our CaT tasks contain instructions, verification functions, example solutions, and failure cases; and (3) PAE prompts the same model to serve as the judge instead of relying on verification functions as in SCA. Note that the third point only makes difference in the self-improvement setting because all demonstration trajectories, regardless of success or failure, are used for the distillation setting.
for short) as the stronger model to sample
Distillation Results. The main results in the distillation setting are presented in Table I] We first note the surprising generalization to -ui-o--distribution test tasks entirely through training on synthetic tasks. Both PAE and SCA are able to achieve more than 18% and 20% absolute improvement in the average success rate, respectively. This especially noteworthy given the striking difference between synthetic tasks and real tasks. However, also observe major limitations of PAE. For example,

---

PAE can only propose very ambiguous tasks based on the limited initial information it gathers from the environment. While PAE works with distillation, particularly more straightforward and fully observable environments such Calculation with an average improvement of +23% Pass@) success rate, its tasks are mostly limited to the information from the initial observation and do not incentivize the teacher model to provide demonstrations with wide coverage over the environments when it is partially observable. In these environments, we find the task challenger agent in SCA to be superior where it can actively explore the environment to generate more precise tasks that incentivize the teacher model to sufficiently explore the environment. We find that this leads to better performance compared to PAE in all 3 partially observable environments, with the performance gain as large as 5.2% in terms of Pass@ 1 success rate in Retail. As we will see in the next section, these weaknesses in PAE's proposed tasks a larger problem in the self- reff-improvement setting, when stronger teacher model is not available.
Self-improvement Results. The main results for the self- ref--imprveeeen setting are also presented in Table We find that the quality of synthetic tasks matters much more in this setting compared to the distillation setting. While PAE is still effective in the fully observable Calculation environment, achieving +7.3% improvement in Pass@ success rate, it can only achieve marginal improvements or even decreased performance compared to the baseline in the other three partially observable environments In particular, the autonomously proposed tasks from PAE in Airline actually result in a -10% drop to 0% Pass @4. In contrast, thanks to the diversity and preciseness of the tasks generated by the task challenger agent and the accurate reward feedback from the Code-as asle--s-iio -asskkk formulation, SCA is able to achieve better performance both in the fully observable Calculation environment and the other three partially observable environments, with an average of +1 1 1.5% gain compared to the base model and + +10.6% advantage compared PAE in terms of Pass @ 1. This shows the advantage of SCA in general tool-rich and open-ended environments, over the prior state-of-the. art PAE, whose utility crucially depends on the fully observable assumption of the environment.
5.3 Analysis and Ablations
M3ToolEval Calculations
How do different RL algorithms work with SCA? While the main seff-improvement results in Table are achieved through simple Rejection Fine-Tuie Tuning on successful offline rollout trajectories described in [Seection 4.3] in principle the design of SCA should be compatible with any RL algorithm, even the state- of-the-art online RL algorithms for LLM such as PPO 28 and GRPO 29). We investigate the ef-
40
30
Success Rate
20
Zeroshat
fectiveness of SCA when training with different RL algorithms and report the results in Figure 3 includ- ing offline methods such as Rejection Funinn and DPO (26) and online methods such as GRPO and PPO. While offline methods such as Rejection Fine-Tuning and DPO can already achieve significant self-improvements from 20.3% to 31.8% Pass @ suc- cess rate, online RL on the same tasks can push the performance to another level of 43.2% Pass@1 I. How- ever, the additional performance gain comes the cost of increased infrastructure complexity to accom- modate efficient on-policy sampling and instability to hyperparameters, as we observe that the performance of GRPO suddenly drops to 0% at the end of training when it is not tuned carefully.
Rejection Fine-Tuning
DPO
PPO
GRPO
Sk
10k 15k
Number of Trajectories
Figure 3: Ablation studies of different RL algorithms with synthetic tasks generated from SCA, in the Calculation environment from ToooolEval. Pass@1 success rates are reported. We find that online RL algorithms in general attain even better performance outto--distribution test sets, but they are more unstable and require more careful tun- ing.
What is the impact of the different components in CaT? We perform human annotation experi- ments to understand the importance of different components in CaT. Specifically, we compare four different variants for generating tasks, including the PAE baseline, CaT w/ Instruction + Verifica- tion Function Only, CaT w/ Instruction + Verification Function Solution, and CaT (Instruction + Verification Function + Solution Failure Cases), all using 3lama-3.1-8B3 For CaT w/ Instruction + Verification Function Only, we perform an automatic filtering to only keep tasks where the verification function is runnable Python code. After filtering, 47.7% of tasks remain with this variant. For CaT w/ Instruction + Verification Function + Solution, we perform automatic filtering keep only tasks
7

---

Verification Only (47.7%%
Solution
(100%)
(5.25)
60
Percerage
40
18 1N FP FN 1P
TN FP FN TP TN FP
FN
Figure 4: Human annotations of synthetic task qualities. 50 rollout trajectories from Llama-3. 1-BBB in the Retail environment from attempted synthetic tasks from each variant are manually labeled to fall into one of the four categories including False Negative (FN), False Positive (FP), True Negative (TN), and True Positive (TP). The pass rates of the task challenger generating task passing all filters for each category are shown in parentheses. We observe that CaT can significantly reduce both FN and FP, which are invalid tasks or wrongly labeled trajectories.
where the solution can pass the verification function. The percentage of passing tasks for this variant 9.5%. For CaT (Instruction + Verification Function Solution + Failure Cases), we perform all automatic filtering steps to keep only tasks where the solution can pass the verification functions while none of the failure cases can pass. The percentage of passing tasks for full CaT is 5.2%.
For each variant, we obtain 50 passing synthetic tasks and collect 50 rollout trajectories from Llama- 3.1-8B in the Retail environment. For each rollout trajectory, one of the authors manually inspects the trajectory and the task to classify it into one of the four categories: (1) False Negative (FN), where the agent gets reward of O because the task is impossible to be complete. For example, the agent fails the task of "return order #W000001" but is impossible because this order does not exist. (2) False Positive (FP), where the agent gets a reward of but does not actually fulfill the instruction. For example, the agent might return an order instead of exchanging the order. (3) True Negative (TN), where the agent gets a reward of 0 because indeed makes mistake in attempting the task. (4) True Positive (TP), where the agent gets reward of 1 and it indeed fulfills the instruction. The statistics from human annotations are presented in Figure 4
Firstly, we notice that tasks from PAE result in a significant fraction of FN, with the main failure mode of generating ambiguous and impossible tasks for the agent complete. Explicitly asking the task challenger to interact with the environment to gather more information and generate a verification function along with the instruction in Verification Only amounts to marginal improvements. We find that asking the task challenger to generate an example solution and enforce the requirement of the example solution passing the verification function can significantly reduce FN by eliminating impossible tasks, but it increases FP. The main failure mode is that the verification function is too lenient and sometimes even no actions can pass the verification function. While these overly lenient verification functions do not account for large percentage initially, the filtering mechanism based on the example solution passing verification functions tends amplify this distribution. Fortunately, we can apply simple fix by adding another filtering mechanism based on failure cases to completely remove FP in CAT. However, there is still large percentage of FN even in CaT. The main failure mode of FN in CaT is incomplete instructions where the instruction does not contain all necessary information to complete the task. For example, the instruction may be "Help me return one of my latest orders" without specifying which order needs to be returned. As this group of FN involves nuanced semantic dependencies, we leave it as an open question for future research.
How does the automatic filtering mechanism in CaT affect task diversity and difficulty? One possible concern is that the heavy automatic filtering mechanism might collapse the distribution of tasks and make them less diverse. To investigate this concern, we analyze the distribution of task difficulty before and after the filtering step of CaT in the Retail environment, where we measure task difficulty using the length of the example solution. As presented in Figure 5) (a), we have found that for Llama-3.1-8B the distribution of task difficulty before the filtering step more spread out, while after the filtering step it tends to be more homogeneous. However, the distribution after filtering remains similar with the stronger model Llama-3.1 1-BB shown in Figure (b), where CaT merely refines the task distribution instead of collapsing it. This suggests that CaT filtering only removes poor generations typically found in weaker models while preserving the diversity of the distribution of valid tasks.
8

---

(a) Llama3.1--8 CaT (5.3%)
(b) Llama 3.1-70B CaT (34.4%)
8%
Percentage
5%
0%
50
100
150
200
250
300-
50
100
150
200
300
Difficulty
Difficulty
Before Filtering
After Filtering
Figure 5: Analysis of the distribution of task diffiiculty before and after the filtering step of CaT in the Retail environment. The task difficulty represented by the length of the example solution. The percentages of passing tasks after filtering are included in parentheses. We observe that CaT filtering can result in less diverse task distribution for the less capable Llama-3. 3.1....... 1-8B preserves the original task distribution for the stronger Llama-3.1-70B model.
Df-Distribution
$0.0
25.0
Pass@1 (800 tasks]
Passi44 (800
60.0-
Passg] (400
175
Passg44 (400 tasks)
15.0
(200
Pass@4 (200 tasks)
20.0
19.
530
11
D6. 12k 500
12k
Number Trajectories
Number of Irajectories
Figure 6: Scaling analysis for SCA with different number of tasks in M ToolEval Calculation. The results show that scaling the number of synthetic tasks is more effective compared to scaling the number of trajectories per task on the test data. In contrast, on the train data smaller number of tasks but more trajectories gives higher success rate, but this does not generalize
Should we scale the number of tasks or the number of trajectories per task? In order to further improve performance with more synthetic data, there are two axes along which we can generate more rollout trajectories: 1) generating more synthetic tasks to collect rollout trajectories, or 2) collecting more rollout trajectories for each synthetic task. FFguree presentss our findings in investigating these two scaling axes in Evll Calculation. Firstly, notice that scaling the number of interactions always results in improvements in Pass and Pass 4 success rates in the train set for all three different train set sizes. However, such improvements in the train set do not necessarily translate to improvements in the -fitt--diitiibuttin test set. In particular, training on 200 tasks even slightly degrades the test Pass@1 and Pass #4 performance, and training on 400 tasks only leads to marginal improvements. Only when the training set has sufficient diversity and coverage, as in the case of 800 tasks, can we steady improvement in terms of test performance we collect more rollout trajectories, This result underscores the requirement of large set of training tasks for general LLM agents and the promise of SCA automate the process large-scale task creation.
Broader Impact
The objective of this project is empower LLM agents with the ability autonomously enhance their capabilities through interactions with the environment, thereby reducing reliance on costly human supervision. Although the models trained in this paper exhibit inadequate performance and struggle with relatively simple tasks, such as booking flight tickets and processing returns, future seff-improvement methods leveraging more advanced pre-trained LLMs may potentially yield models with superhuman capabilities. To mitigate potential risks, it is essential to conduct thorough research on aligning strong models with human values, ensuring that these values are preserved even the models improve themselves to achieve superhuman performance levels.
7 Limitations
While our work shows the promise of an LLM agent generating high-quality tasks to improve itself, there are still limitations, which are also future research opportunities. First, despite the CaT formalism, there is still non-trivial percentage of False Negative examples as shown in Figure 4

---

The main failure mode is semantic nuances such ambiguity missing information that can be difficult to distinguish. This results in the suboptimality gap between RL training with SCA tasks and oracle tasks as shown in Appendix B Additionally as detailed in Appendix BJ because of the distinctions between different environments studied in this paper, the L.MM agent tends to mostly improve its environment-speciin skills instead of overall agentic capabilities. It remains open research problem on how to enhance the environment-genee agentic capabilities of LLM agents.
8 Conclusion
In this paper, present the Self-Challenging Agent (SCA) method for self self-improvement of general multi-turn tool-use LLM agents. SCA can create its own tasks to challenge itself and learn from them. To do this, it utilizes the Tode-as-Task (CaT) formulation which ensures high quality synthetic tasks. Through RL on these self-ggeeeated synthetic tasks, SCA can be used to train a Llama-3. 3.18BB model achieve an average relative success rate improvement of 95.8% on existing test tasks across four different multi-turn tool-use environments. Our results show the promise of self-improvement for general multi-turn tool-use L.M agents, without reliance the manual process of creating diverse set of tasks, tools, and evaluation criteria. While SCA serves as a preliminary step, there remains many research questions for building an effective oof--imprveement flywheel for general LLM agents.
References
[1] Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning, 2024. URL httppps://arrra..oo///a//22222 11896.
[2] Hao Bai, Yifei Zhou, Li Erran Li, Sergey Levine, and Aviral Kumar. Digi-q: Learning q-value functions for training device-control agents, 2025. URL https://arxiv.orrt wwwg/b//////////oo/. 15760
[3] Adrien Baranes and Pierre Yves Oudeyer. Active learning of inverse models with intrinsically motivated goal exploration robots. Robotics and Autonomous Systems, 61(1)779-73, January 2013. ISSN 0921 19911-88 -8890. doi: 10. r0.0016..oobot.2222....0000 URL http://dx doi. org/10 i.16/j. robot. 2012.022220 008
[4] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.0133 2024.
[5] Cédric Colas, Laetitia Teodorescu, Pierre Yves Oudeyer, Xingdi Yuan, and Mare-Alexandre Côté. Augmenting autotelic agents with large language models, 2023. URL https://a/xxi org/abs/23055 12487
[6] Kefan Dong and Tengyu Ma. Stp: pelf-play IIm theorem provers with iterative conjecturing and proving, 2025. URL
wwt.wws.....//.../ aewgg//b///0...... 2502 00212
[7] Maxence Faldor, Jenny Zhang, Antoine Cully, and Jeff Clune. Omni-epic: Open-endedness via models of human notions of interestingness with environments programmed in code, 2025. URL httpthttt//:aiiiiii www.oogg.//...ooooooooo....o.../ /40/ 2205 15568
[8] Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. Stabletoolbench: Towards stable large-scale benchmarking on tool learning of large language models, 2025. URL hwtp:httt//// pettps://arxiv org/ abse///3a/443.... /20.//443.000 07714
[9] Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models, 2024. URL htttttsss///xiii org abs/ 2401 13919
[10] Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Hongming Zhang, Tianqing Fang, Zhen- zhong Lan, and Dong Yu. Openwebvoyager: Building multimodal web agents via iterative real-world exploration, feedback and optimization, 2024. URL //arxiv.... org/ abs/
2410. 19609)
10

---

[11] Mengkang Hu, Pu Zhao, Can Xu, Qingfeng Sun, Jianguang Lou, Qingwei Lin, Ping Luo, Saravan Rajmohan, and Dongmei Zhang. Agentgen: Enhancing planning abilities for large language model based agent via environment and task generation, 2024. URL https://arxa org/abs .00644004444444
[12] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents, 2022. URL https
/aarxiv.org hww.:oggg//bbaaaae www.ab///...../ 07207
[13] Wenlong Huang, Chen Wang. Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models, 2023. URL
https://arxs org/
05973
[14] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024.
URL httpt:h/taaaaati org abs/2310 231000677000000
[15] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control, 2023. URL attps://arxiv. www.o/////ba/ooo ablaaabb//22// 07753
[16] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du Chenhui Zhang, Sheng Shen, Tianjun Zhang. Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench:
Evaluating ilms as agents, 2023. URL httph:tt//saet //axxi: org/abs/ abs/2308. 2308. 036888
[17] Dheeraj Mekala, Jason Weston, Jack Lanchantin, Roberta Raileanu, Maria Lomeli, Jingbo Shang, and Jane Dwivedi-Yit Yu. Toolverifier: Generalization to new tools via self-verification, aataa/aa/22000 14158.
2024. URL
[18] Shikhar Murty, Christopher Manning, Peter Shaw, Mandar Joshi, and Kenton Lee. Bagel: Bootstrapping agents by guiding exploration with language, 2024. //aptts:://rxiv...tt/
abs/2403.08m 08140
[19] Shikhar Murty, Hao Zhu, Dzmitry Bahdanau, and Christopher D. Manning. Nnetnav: Unsu- pervised learning of browser agents through environment interaction in the wild, 2025. URL attps://arxi:
org/ abs/ 2410. 02907
[20] OpenAl. Gpt-4 technical report, 2024. URL
[21] OpenAl OpenAI, Matthias Plappert, Raul Sampedro, Tao Xu, llge Akkaya, Vineet Kosaraju, Peter Welinder, Ruben D' Sa, Arthur Petron, Henrique P. d. O. Pinto, Alex Paino, Hyeonwoo Noh, Lilian Weng, Qiming Yuan, Casey Chu, and Wojeiech Zaremba. Asymmetric self-play for automatic goal discovery in robotic manipulation, 2021. URL URLhttpps: //arxi/...ivv org/ abs/ 2101. 04882]
[22] Jayy Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym, 2024. URL https //arxiv.oo org/ hts//ab//200 /2tp./212..00 21139
[23] Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Au- tonomous evaluation and refinement of digital agents, 2024.
2604.06474
[24] Zehan Qi, Xiao Liu, Iat Long long, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, Tianjie Zhang, Wei Xu, Jie Tang, and Yuxiao Dong. Webrl: Training Ilm web agents via self-evolving online curriculum reinforcement learning, 2025. URL hntts:://ariis org/
02337
[25] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023, URL // aarxivv org/ abs/2307 16789

---

[26] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2024. URt.tttss: /nttps://arxiv.... org/ borg/abs//on /205. 18290
[27] Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom, Toolformer: Language models can teach themselves to use tools, 2023. URL
04761
[28] John Schulman, Filipp Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal 06347
policy optimization algorithms, 2017.
[29] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiio Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https: /tap::://axxiix org/ abs/
2402.03300
[30] Zhengliang Shi, Shen Gao, Lingyong Yan, Yue Feng, Xiuyi Chen, Zhumin Chen, Dawei Yin, Suzan Verberne, and Zhaochun Ren. Tool learning in the wild: Empowering language models as automatic tool agents, 2025. httpst:::://// htttps://arzvvoog///a/// httpg//bb/24400 16533
[31] Yueqi Song, Frank Xu, Shuyan Zhou, and Graham Neubig. Beyond browsing: Api-based web agents, 2025. URL httpps://axxiii. http://axxvvv.r/aaaaa// http://op//ab//aaa/eeeee 2414..164.1114444 16464
[32] Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan Ö. Ank. Learn- by-interact: A data-centric framework for self-adaptive agents in realistic environments, 2025.
URL httttttts///xxiii/ www.aawwwwwoo.ooaaaaooaaaaoooort abs 2at..//50....00
[33]
Sainbayar Sukhbaatar, Zeming Lin, llya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob
Fergus. Intrinsic motivation and automatic currieula via asymmetric self-play, 2018. URL
org/ 05407
[34]
Llma3 Team. The Ilama 3 herd of models, 2024. URL. attps://arxiv. http:///////b///0/0/
21783
[35]
Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. Appworld: A controllable world of apps and people for benchmarking interactive coding agents, 2024. URL https
7/arxiv. org/ waagg/////// a407 18901
[36] Taiyi Wang, Zhihao Wu, Jianheng Liu, Jianye Hao, Jun Wang, and Kun Shao. Distrl: An asynchronous distributed reinforcement learning framework for on-device control agents, 2025.
URL htttttttt///axiiii wwt/ 2410. 14803
[37] Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better I1m agents, 2024. URL btsttp://aaxxi..rrg/attt
01030
[38] Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruet: Aligning language models with self-generated instruc- tions, 2023. URL httph: wwg.aawwwwww..ooaaooooooaaeooeee abtp://ss./22.... 2212. 10560
[39] Ronald J. Williams. Simple statistical -oludiint-folloinng algorithms for connectionist reinforce- ment learning. Machine Learning, 8:229-256, 2004. URL httpps:/appp http:://app.smmmaiccccchht semanticscholar org Otg/CorpusIDD. 19115634
[40] Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: A benchmark for real-world planning with language agents, 2024.
URI. http///xxvvvv///// aabs/2002.ce 01622
[41] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024. URL httpps //arxiv. org/abs/////oeooooogo...ooeeeett abs/ abs/2404 07972
12

---

[42] Wei Xiong, Chengshuai Shi, Jiaming Shen, Aviv Rosenberg, Zhen Qin, Daniele Calandriello, Misha Khalman, Rishabh Joshi, Bilal Piot, Mohammad Saleh, Chi Jin, Tong Zhang, and Tianqi Liu. Building math agents with multi-tur iterative preference learning, 2025. URL
https: paaps:///axiv.. http//vvv.or/////////.//../ abs 02392
[43] Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang. Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang. Yiqing Xie, Shuyan Zhou, and Graham Neubig. Theagentcompany: Benchmarking IIm agents on consequential real world tasks, 2024. URL pestps://arxihe org/ / abs/ 2412. 14161
[44] Ke Yang, Jiateng Liu, John Wu, Chaoqi Yang, Yi R. Fung, Sha Li, Zixuan Huang, Xu Cao, Xingyao Wang. Yiquan Wang, Heng Ji, and Chengxiang Zhai. If Ilm is the wizard, then code is the wand: A survey on how code empowers large language models to serve as intelligent agents, 2024. URL httpt: //tp:////axiv./// org/ abs/ 221./4411..0000 00812
[45] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023. URL. https
//arxiv.org/ 2aab/2211.... 03629
[46] Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. b-bench: A benchmark for tool-agent interaction in real-world domains, 2024. URL bRtps://////ar wrg. abs/ 12045)
[47] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhaaatar, Jing Xu, and Jason Weston. Self-rewarding language models, 2024. URL httpss///a//
[48] Jenny Zhang. Joel Lehman, Kenneth Stanley, and Jeff Clune. Omni: Open-endedness via models of human notions of interestingness, 2024. URL
01711
[49] Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data, 2025.
[50] Shuyan Zhou, Frank Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Weharena: A realistic web environment for building autonomous agents, 2024. URL htttps://arxiv.orr.. /bs/ 2307 13854]
[51] Yifei Zhou, Qianlan Yang, Kaixiang Lin, Min Bai, Xiong Zhou, Yu-Xiong Wang, Sergey Levine, and Erran Li. Proposer-agent -egeetttevaluuaorrpaee) Autonomous skill discovery for foundation model internet agents, 2024. /RRLaattps//aarii..orrgg www.rr//abb//aaa abwp://bb//22..... 13194
[52] Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-tum rl, 2024. URL httpt:t/ss//aaiv /bs/ 2402.19446)
[53] Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, and Xian Li. Sweet-rl: Training multi-turn 11m agents collaborative reasoning tasks, 2025.
htaps:////rxivv org/aba/ absp///3..0.....//// 2503 15478
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper addresses the challenge of generating synthetiic tasks for LLm
agents.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: There is dedicated section in the main text discussing limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
No theoretical results provided in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The appendix includes all the prompts that we have used.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [No]
Justification: N/A
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
httpa//ccccpl/oggiiiio//e/eeeeea /
sei.aiisssoooioiyyyoo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have included the choices of hyperparameters in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: Because of constraints of computational resources, we are unable obtain statistical evidence for our experiments. However, we do have scaling experiments in Figure 6 that contain repeated experiments with different amount of data.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This will be included in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conforms with the NeurlPS code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: There is dedicated appendix section for discussing broader impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Our experiments are conducted under constrained environments and thus do not have high risks for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: They creators have been properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cow/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: No new asset is released
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
We have included a screenshot of the annotation page in the appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Does not involve crowdsourcing, labeling is done by the authors.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: Core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
20

---

A Additional Environment Details
Due to space constraints of the main text, we provide additional details of the environment this sections.
ToolEEval: an example task in this environment is: "Decode message that went through three steps: first, a Caesar cipher with a shift of 3; then reversed; and finally, encoded to hexadecimal. The final hex-encoded message is '726177746661537 is fully observable, and almost all information about this environment is encoded in the initial observation containing the API documentation of each tool. The Web Browsing environment involves text-based navigation through set of synthetic web pages to find specific information. Initial observable information in only contains the landing web page and the tools that the agent can use to navigate the website; more information must be gathered through explicit interaction with the environment.
:" Compared to other environments, Calculation
Tau-Bench: example task in this environment is "You are Fatima Johnson in 78712 You want to cancel all pending orders (since they are no longer needed) and return the watch you have received (but nothing else), and you want to know the total amount you can get back. You a private person that does not want to reveal much about yourself," In that case, the ground-trum truth verifier checks if the desired changes have been actually made in the database. When we apply generate synthetic tasks for the tools and setting of this benchmark, we set the verification function python function that performs database queries to verify desired effects, the example solution to be a series of database modification calls, and the failure cases to be some close attempts. The initial observations for both environments only contain the API documentations for each tool, and any information regarding the database is only available to the agent through explicit function calling such "get reservation details". We noticed 40% of the tasks in Airline are considered successful when no actions are taken so we removed them from our experiments to reduce noise.
method to
Environment modifications for task challenger: each time when the task challenger tries to synthesize a task in CaT format, we will reset the hidden states of the environment (c.g. the flight and retail database) and run an automatic checker. The automatic checker verifies 1) all code is runnable, 2) the example solution can pass the verification function, and 3) the failure cases cannot pass the verification. If the automatic checker fails, it will return its error traceback to the task challenger to revise the task.
B Additional Discussions
In this section, we present additional experiment results and analysis to understand the limitations of Sefl-Challngging Agent (SCA) and possibility for future research.
How does the improvements from SCA transfer between multiple environments? To under- stand whether the improvements from SCA can generalize across different environments, we compare the performance of training separate models for each individual environment and that of training a joint model with aggregate data from all environments. As shown in Table 2 found marginal im- provements, or even worse performances, of training a joint model on the data from all environments. By inspecting the trajectories from pretrained Llama-3.1-81 -8B--Insrttttttt and 3Jama-3.1-8B-SCC. we found that the main challenges faced by the agent in those environments are environment-speciine e.g. "book hotel" tool in Calculation does not take position arguments or the agent should first ask for the user emails and zip code before checking the account details, and Self- Challenging Agent mainly improved the LLM agents' capabilities get around these wwtrronmnttsspeciii challenges. It remains an open research problem if we can design self-improvement method that enhances the pirronment--enerral agentic capabities.
How does the task quality from SCA compare with oracle tasks? We conduct additional experiments to investigate how tasks generated by SCA compare with high-quality oracle tasks in Eval Calculation. To construct oracle tasks, we manually create 5 seed tasks for each domain in Calculation that resemble existing test tasks and prompt Llama-3.1 to generate similar tasks with detailed documentation describing the implementation of cach tool. We then prompt
21

---

Table 2: Cross-environment transfer analysis where we compare performance training separate model for each environment with that of training the same model with aggregate data from environments. We found marginal, not worse, benefits from aggregate training for all 4 environments.
Evll
Tau-Bench
Average
Calculation
Web Browsing
Retail
Airline
Pass
Pass@4
P
Pass
Pass
Pass
Pass@1 1
Pass@8
Pass 1
Pass@-
Self-loprove
3.maa-.1-8B-SC..
31.8
62.5
44.9
67.6
13,0
25.2
4.2
10.0
23.5
41.3
Llama 3.1 11-81---50 SCA-SCA- CarA--Crs----i Task
30.7
$8.3
45.6
67.6
11.3
23.5
4.2
10.0
23.0
39.8
M3ToolEval Calculation PPO
Zeroshot
SCA
50
Oracle Tasks
Success Rate
40
30
M
20
5k
10k
15k
Number of Trajectories
Figure 7: Comparison with oracle tasks M ToolEvvll Calculation with PPO policy optimization, Both training set contains 800 tasks. While training on synthetic tasks generated from SCA can result significant improvements on out-of-distribation test tasks, there is still a gap compared to training on oracle tasks.
Llama-3.1-88 16 times generate a Python expression to solve each task and retain only those with a consensus rate between 25% to 50%. The consensus results serve reference solutions. Throughout this process, we manually inspect the generated tasks to ensure quality. This approach yields training set containing 800 tasks for Calculation. We run PPO with identical hyperparameters on both the 800 tasks generated by SCA and the 800 oracle tasks, and report the comparison results in Figure Our findings indicate that while SCA generate high-quality tasks that facilitate self-improvementt, there remains a sub-optimality gap compared to using oracle tasks, suggesting opportunities for future research to further refine the quality of synthetic tasks.
Table Ablation on distillation where we considered the role of unsuccessful trajectories in the distillation. We found that when the performance of the teacher model is significantly better than the student model, distillation using both successful and unsuccessful trajectories tend to work better than using only successful trajectories.
M ToolEval
Tuu-Bench
Average
Calculation
Web Browsing
Retail
Airline
Pass@
Pass@4
Pass
Pass@4
Pass@
Pass@4
Pass
Pass@4
Pass@
Pass
Distiilation Llama-3. 1--B-B-C--BBCCAA
82.4
23.3 13.3
56.8
Llama-3. 1-88---- 38--SCA-Only-Suceeeee Suleess
33.3
50.0
41.2
79.4
28.7
49.6
7.5
27.8
48.1
22

---

Should we use unsuccessful trajectories for distillation? To ablate on the best strategy in the distillation setting, we con- duct additional ablation study and report the results in Table [3] We have observe that distillation on both successful and un- successful trajectories tend to yield better results compared to successful trajectories only across the board. This is probably be- cause when the capability of the teacher is significantly better than the student model, even unsuccessful trajectories can contain useful experience such reasoning about mistakes for the student model to learn from.
Table Ablation study on the effectiveness of the filtering step in SCA. Pass @ is averaged across four independent runs. We generate 800 tasks and rollout 12K agent trajectories for each method.
Model
pass @
pass @ 4
Llama-3.1-8P ---St-tntttuct
8.9
15.7
PAE
9.8
18.3
SCA (no filtering)
8.9
20.8
SCA
13.0
25.2
How effective is the automatic filtering step in SCA? While Figure 4 has shown that automatic filtering can significantly improve the quality of synthetic tasks, we carry out additional ablation experiments to verify that the improvements in the task quality indeed result in an improved policy. Specifically, considered the ablation SCA (no filtering) that prompts the task challenger to interact with the environment and directly generate the instruction and evaluation function. The only difference of SCA (no filtering) from SCA is the lack of the CaT filtering mechanism. As presented in Table 4 while SCA no filtering might have got some positive learning signals from the synthetic tasks to improve pass @4 performance, we found that the generated instructions and evaluation functions are quite noisy without filtering (as shown in the human analysis in Figure 4) and resulted in sub-optimal performance compared to SCA, e.g. pass drops from 13.0 to 8.9. This ablation provides direct evidence of the effectiveness of the automatic filtering step in SCA.
C Details for Human Annotations
For completeness, we have included in Figure an illustration of the annotation interface that we have built for the authors to annotate the results presented in Figure For each trajectory, the author will see the initial instruction, the entire interaction history, the reward predicted by the verification function, and the verification function to classify each trajectory into the following four categories with descriptions provided as:
1. True Positive (Agent actually completes the task and the reward is 1)
2. True Negative (Agent actually failed the task with a reward of 0 and the task is feasible)
3. False Positive (Agent did not complete the task but reward is 1)
4. False Negative (Agent got reward of 0 but it is because the task ambiguous or verification function is wrong).
In total 200 trajectories are annotated for each variant presented in Figure 4 and it took in total 8 human-hours.
D Prompts
For completeness and reproducibility, we include all major prompts that have used in our experiments in this section. In particular, the prompt for the autonomous evaluator in the PAE baseline provided in Figure [3 and the prompt for the user simulator for Retail and Airline environments is provided in Figure 14 We have also provided the prompts for task challengers in Figure
and 12 We observe that stronger LLMs such as Llama-3. 1-70B are relatively robust to the prompts while weaker LLMs such as Llama-3.1 1-BB are sometimes sensitive to the prompts, mainly because of the length of the instruction required to specify the environment and action space and the suboptimal mnstrcction-fllowiii capabilities of weaker LLMs for long prompts.
23

---

Conversation Vaualizer
Mac ind yourr
Mi Henuue
Lo sundiiwa dier mas Mnvore furn
lealle lpen peurm dint
centre mesoreiaee
car
1dra
fomeooieee S01T7 Prene
Du Hooo
unolee adureer ontiona snd aheld lev wootaed
1u008-1hte rigo
Figure 8: Annotation interface we have build for human annotation results presented Figure
E
Compute Usage
For reproducibility, we include the compute usage in Table 5 be challenges generation and rollouts generations where the agent needs to engage in multi-turn interactions with the environment, whereas Rejection Fine-Tuning and evaluation take much less time.
We find the main bottlenecks to
Table 5: Compute Usage for our main experiments in the self-improvement setting. All experiments conducted 8xA100 80G. The unit is the number of hours 100 80G.
Environment Calculation Web Browsing
Challenges Generation
Rollouts Generation
Rejection Tiee-Tunig
Evaluation
8
24
0.5
8
0.5
Retail
32
24
2
Airline
32
24
1
Hyperparameters
For reproducibility, we have included the hyperparameters for different RL algorithms used in We found that Rejection Fine-Tuning and DPO are relatively stable with
Table T and Figure 3 respect to hyperparameter choices while online RL methods PPO and GRPO require more careful hyperparameter tuning.
G Qualitative Examples for Synthetic Tasks
and 19 qualitative examples of the main failure modes of different variants discussed in Figure In particular, the main failure mode for PAE task generation is that most the tasks are ambiguous and non-verifiable. The main failure mode for only introducing verification function is that the
To give more details for CaT, we provide full example Figure 15 Figure I6 17
18

---

Figure 9: Example prompt for task challenger in Retail environment, After the tasks generated, we redact the order number from the instruction to prevent the agent from knowing the order number before logging into the user's account.
You are helpfal agen, Your tast in to gerorate eynthenic instrestion given she saee id und the solecions and evoluation Functiona for this wase 1d. tour insrrectious avlutions und evaluss cona cea be beaed oa the following icoka.
too can also isteract uith the anviromment to try to complet gatter nore internation for uniting aralsaatio fenetione astattone.
yourself
apens should able the tollowing bools (orpenazed in pysasa fxnctions) slong vish the deftsed fileds
follow
ctaateursion>
Your Sopaia Jackson You are pessiaiatic, ontginng, ral. emall is sophis. (acknadd77Seerupll
For order #43971450, rives say that you vont to cancel the order but then the agent axtr for confirration, zyy that you sart to aniveal exchange Vuter Buctle capucity" "o00e", 'auteriel"s "sicisleso studl', "cole"": 'greva")
to d'auterio""s "glenn' "i via ccodit,caed.dde 8766680
crder 132054390, cancel it becsuze "onder by cistaze".
</zeatraction>
serelsation, fuaction?
(order
*7653802001") lentorfer (ercbarge..ttat)]).
(ordaz ("ecchangs iteaat "S78573n029") and les funder" eaachangg atv_atema"))
oodaz exccaagge payeant, asthod sd1) "credit, chesse""
recears. encrar Mi get _rddrdde, Satalla( (2801/0000 ["atatue*) "cancalled
vecura RECTAFE
c/evaluactin, Fencticn
collutinn
excharge_del indee [tteere "#81677458",
stem, dde [17583196111 "), nay 15ea, (*6710777005)) pepent_rechad, inw.haidieerrrdtts.oooo.
</aolssion>
czaeple
stea_ide [175239000011
[*6760777161) wethdd de""credtt_anns57776600
diuilere.come
exmple elgo-ccaze colution that should not pens the ovaluation fusotios
axcharge_ www.veree..ceeeeeooooeeeeeeeeeee coder steea (crder, 14 "962977635",
itea, ids [*7352002001") "1, bay, idd (*575875718"), payoest sethod, wattoo__dd""ceeeeeeeee id""credit 2a0r__574822222 6748622") </fal caree
slaps your postrssive
unuctly following (ocat:
code calling the soale
(xb(c))
80 ACTIOI
de 1f you have slevady obsaiaed the information you aved, you cun directly sasvez.
(80002T:
2a00 ahtstred wssther information through rall.
can proceed raking evaisation tunctios, solutier, fatlore cases
Eusction?
evuluate():
</evaSussion, Function
rackution/2ze wtwps//////////
cfasTere_ wwt.e//_eee//e/eeeeeeTa
ctattere_ cfNtieeee.CMMETT www.e//r.../Ailom......
BID ARSVER
action and response should Mzapped ACTIBS:... ASTIDN, JASMEE: ANEVER respertiondy. Onlyy cutpat sne action or ansver, SNCC YoU SUBNUTTED
CASSOC
the evaluntion function "avaluass("", Sake TERS-snne..
and provads soletion ofter vaecating which the vrelustion function abould retura Texs. You abould waite loua & fullure
"TE nINTInG THE SOECTIEN in AND EPRCCATTON SIRCTION FIRST TU mi IF IT Is CINJECT,
ARSSER AFTII YOU HAVE MADE SIRE THEY ARE RUSHATCE AND CONJECT.
You should come up with your ONL InstrRotion wIth the usor 16) (user_i0);
APTRA CRECKTRO TSER DETATLS AND THEIR OADER DETATLS
ORTIS 1D,
atc.).
"HAT PROVIDED
USEA RETURS,
VRAT
GET INFORMATION.
25

---

Figure 10: Example prompt for task challenger in Airline environment.
Toi are helpfal aget, Your tast 1n to pererate synthentc inntrection grown the cour 14 and the aslurions and aratcation fincctonn for thie user id. tour inscrections, soletione and evalusctoon can be taeed on the dollowtin scole. You can alwo ixberacs with the coris ensost to tay to copplete the tesk youroelf se gellor exce information
for criting evslustion Tenesioon and oelutions.
agent shoold be able the tollowing books (uspessued in pyshsa fxnctions) along with the dedibed fileds these toola:
Scoold deac?
cisstrucsion>
Your
paraaignndin www.ie......17.. 1177.
For
Irom ATL PEL,
vant to thane for choupeal veruaay Flighi und Loe the altuz the uraginal seservution. tex are happy with original paynant for raftnd.
</instractios
cersisation, Fusctions
avaluata
recrwes TImA
aus Jounds
assecrelios. gel.ceourvatiinn htw..iii...ttieeeeeeerrrie.... "MOSKNL-)
f'rllight
if f1-flight, [ *HAT179* fnddff"ne [["dara"] 19024-06-009;
founde s=
rectats:
(ensevvalitnl
asocoss socceso and (essersationl cabun"] on "econssy"" ?
recuca seccess
dlavaluacioo, Finctico
Selatime?
spdate, www.rratttennttiii.. pigasraattttttttaarattttt "MEGINU" cabin "aconmmy"..
fiigats [(*tiight "HATLIO", "dasa": 12034-26-24"),
("Giihttanmeee"" "EAT172" "date"; 12904-05-244 05-20-0-3333 poymest..id "5iI1_carri_8871779
ssupply
abould paur
cancale renervattin cancel www.erettttooneeeeeeetttttooeeee "Nosis"") </fellure_case>
the
tus sumple sige-cure solution that abould sut paus the vvaluation fucction, vat is updates the flighs with lhe vrong piysast suilidd
"MESRM"" cabto "econmmy". tilgate [(ttiigts mubbart: "MaTLI0", "data" 12026-28-20),
spdate, rasareation diigttat
("ilight aumber": "2AT172" "datet: 22024-05-241)) payaest "credit, cazd, card 8587175")
esoctly following
TH00GRT:
<Besacning
contezt inform the decision making.)
BD THUUQE7
python coda for calling the tools (eent(oe((hc))
BD ACT10N
you alvendy obisined the intvrnution you aved, you cmn directly move..
#N5UCK:
Seveltation, function?
emuloate(::
c/avaluacion, Puncticon
csolution/ze aor solction)
<halleze, www.aa////aaaaaccooo caaaa////alllerer coss?
respectively.
s YOUR oo
Noln cure to name She evaliation funcsion "*
and scoride a solstion ofter sxecuting vosch the eroloation fusction should return Teus.
Tou abould weite et lesat 3 fuliure cases,
PINCTION CONNECT.
CONJECT,
RADE instractios with user 16: (uzer_id)
coms up with your
COMSAND
INPORMATION.
CREATIIV
REASOSABLY DEFFERENT
IN8TNICT101S.
26

---

Figure 11: Example prompt for task challenger Calculation environment.
You use helptul ussistent that Leiee coms up will a took to be siven to un ugeat,
abece the goal is to cleck if the agent cen cuereotlyy use the toolo so Look for informes.vo.
"Tee teok contains besk on instruction and at exgected ontget, and an esangle solutson. both you sgent accens to the aase aus toola.
up
accppl by agent
yeu
abould firas the
code
thought
shat kind of base you imagining thet think about shat information needed to validute the tazk feuzible You sbould bagan und invorution 'kclion: und mol with Ead Action'.
Brasple: lctton lettel, mase Cargenent, 1alit actsonta
when you ace done, cutpuc the resuit using 'Aaaver sax Rnd Acsver', sbare axe is your construction of a tass inclodiog the insteuction, the excepte solution,
the espected outges.
2esp your instroction ciestrrctiono, </rsttmrsion,, ecappla astution in casopla,mntutton expected cutpas Seapected, oxtguit, s/espected, output?. enmple woutius abould after esocution abould usme exproted oxiges.
Pythsu
exeeted save the to voriable 'reaull".
ralse
azsepie:
cinstrucsion
BHA ("AOCTAG', 'BY2ARC' (STOSST), check chiic wo'ld find the ualld 361 </instraction)
Suapecied..uutge
0T3607
</expected, octpet>
cezeeple, s0lstion>
volid, sequences [aoq for peq in das, sequezess if is va,velid dos, (equacciosue)
recolt ear(oslid, exx(onlld_ raqquenne,,, keyelan) valid_reguneee alra valld IEA raguecas*
c/azappla, solutiors
exmplee:
with Find acmericcal hotal burtnace 2023-08-15
Sive total budget for sttp:///tatractiins conpecsed.ootpur>
112.71
</expected, output>
Suzeeple, aolesple_ools..com aokution>
swollt san!
budpet, www.uisterieeee (ittuiitttrriiiggeee hetpattt_ppiaatightteee ["petca"], hotel, pricee nightttetee ("pcice_pre xiigh")) zue zon_nightare)
flight (nod_fiighhe((ron.ooootttt (eooeeaorrattoone to.looation000
c/esumply
Anaver
tio your tore.
First coma with a candtdatt inseraction that se of stuslar diretcuityy the azsple solesion, it should une 3-4 zoole from sbone.
"hen interoct with the toolo to by to oolive the tank to see if it Jecoable,
and not, sodify the lesteuction on to make 15 fasoible.
you
feamible taak, extpet the renuit saing
Answer',
xas your constraction
tuok incluting instruction, the voample aolstion, the expected cusput. expected autpot silber simzle
Suas sure
AgrDrsEk:
VaUUE Baci PYINH CONMAD Ap<priEk: SEED oree NECOMATION SRCESSSRY ros TasK TESTRUCTION
RBSUL.T..
cINCESTE TEMIT sucH ('ANCTAO". vezAAe VoTOUST) TNSTRAS PLACTPOT BEAS ETRR LIST SEOIRIEES.
27

---

Figure 12: Example prompt for task challenger Web Browsing environment.
are helpral anetatann that triee coee up with taat to be grown an agent, the goal tn chack the agent can cormectly use the toole Toot Information. tect contatne hesa an instruction and at axgected ontpet.
Bull you the spent avesss to the sone ueu Loola.
deoc)
up
you
the
the
oode
it.
stould bagin taol trcoration Actton: and with "Ind action!.
Beangler Josion Anteol mase fargsment, 1'laad Autionia
When you are dane, corpur the resils using "Aaezer var Rid Neaver's ',
<berv sax is your conatruction of a task inckading the instruction und the uxpected putpat.
Yeap your instructicn in Canttrcction, s/azstewsios"
and your aspacted neeger corpertad s/arpactad,
Zere an azaeplas
Answer:
departmers Kisberly ey.
Suapec
</expacted,nn
Azavez
Tarn 16. Moster azxeple:
fnreer:
Fiad the curcvat poi of Lugeniary Vand. </beatraction>
respecced, output>
</argactad__nnggrr>
Piret Then creativity conatruct sools for
information you information have obtaiood from the toola, Your constructed tark should he atat tar to the axseptes abone bat wore divente.
fortho the aoire
still pozuible be solved.
Mace zura that the esgacted antpor in atther stngle nember stngle startng anerer
to it can be ctecked by string natching,
38tDYBE YOU SEED PRINe THE VMDUE IN RACH PYTNOи OONMAD TO 882 THE RB8UL.99.
Figure 13: Example prompt for autonomous evaluator in PAE baseline.
You are an LIM jusge that dacides shacter the reguest of an neer bas been cornersly hand wi by agant. You viTl be prseticed the goal of the newr, and the tstaracel on hivtory batzwen the user and the agent.
cocone-SDV YOUR TiRS-e
gosl snse:
(iastcuction)
history
agent:
kttryy
corpucting your gudgeeant, cotpot thanght
etnought> c'etenghts tage,
c/verties>>
HL3ASE GESEX008 YoUR RATISO LOwl TI AGET REA60SABLE ATTENPTS, EVES THE 8807897 18 PARTIALLY COMPLETED. Tous verdiet abould "SU00aS0"
following:
*M02 20C0258*
Figure 14: Example prompt for user simulator in Retail and Airline.
agest. (teppaay)))
(chin will (this will be sent the agent). Thet, gesarate Line Uour Beaposus ziralate the
what
Zo not gire all instruction
Only provide the incormation that peosided the insteuciiva.
for the
Do not ballusincte informatios lus that
For azsppie, 18 the agent sece for the order id but io. s8 not neattoond in the tnscraction,
do not nake an order id, stat any you do not remember or zave it.
If the instiaction gool is satisifiel, gerorete "ste0710Pate" os the User Reegense vithout anything else to and the converonsion. Do not repent the evsct Instruction in the convereation Tratead, une your words to conney the rose informasion.
Try so rale comerraction. an natural at porstbll, and attik to the personalttsen in the instrrctionn
Teer Sarpanen:
<tta
(chie utll. be and the agent)>
28

---

Table Hyperparameters for different RL algorithms.
All Environments
learning rate
le-5
Rejection Fine-Ture Fine-Tuning
size
epochs
context length
16192
learning rate
2c-7
batch
DPO
epochs
6
context length
16192
beta
0.1
learning
lc-6
critic learning rate
le-5
batch size
256
ppo epochs
1
PPO
context length
16192
clip ratio
0.1
number of tasks each iteration
32
number of rollouts each task
8
kl coefficient
0.001
learning rate batch size
1c-6
256
ppo epochs context length
1
16192
GRPO
clip ratio
number tasks each iteration
32
number rollouts each task kl coefficient
0.001
verification function is often infeasible. The main failure mode for only including verification function and an example solution that the verification function can often be fulfilled by even unsuccessful attempts. Finally, the main failure mode for CaT is that sometimes the instruction does not contain enough information to solve the task. Eliminating this type of task failures require examining semantic nuances and we leave it as an important direction for future research.
H Example Interaction of Task Challenger
To better understand the role task challenger, we include a visualization an example trajectory of the task challenger in Retail. As shown in Figure 20 thanks to the instruction following reasoning capabilities of LLMs, the task challenger has very good prior for taking reasonable actions adequately explore relevant task information in environment and generate high-quality task.
instruction-following and
29

---

Code-as-Task (CaT)
Instruction: (To User) Your name is Olivia Nguyen and your email olivia4794 http://xxxxmmcccccoommm For #W112, return the Skateboard via paypal 77.
Verification Function:
evaluate
success Truc
get_order_detaala ("EW11222
and
ordar ["return_ "J [][0] == '6843647669"
and
order ("return__ waw.:a_payaeattetto___eee -- "paypsl_77*
success
Example Solution:
offer interscting with the environment to gather informatson
return_ aeceerrtteeestreeeeeeeeeee (order_id "AW112",
item_ids ('6843647669"),
Failure Cases:
Paclure urong payment
aeturn_delivered
aawarrdddrdeertteeeeee arder_itenss(e (ardar_id (ordrrrt *AW112",
item_ ids ('6843647669"),
Bayment_nethod_1e idd "ddt""crriittcarddd.77
Failure Case 2, wrong stem debarn_ddliverra_oordeer.... (order_id *#XOO"",
iten_ids ['6123456789"],
Failure Case 8, cancelsng instead neturning caucel_orders (order, "#V112",
item_ids ('6843647669"))
Figure 15: An example of high-quality Code-as-Tass Task (CaT) in TauBench. The instruction is first sent to an LLM-simullted user generate the initial message. Automatic filtering will he applied CaT's only keep valid tasks where the example solution can pass the verification function and failure cases cannot.
Main Failure Mode PAE
Instruction: (To User) noah brown 6181, like find my user first and see have any pending orders. If do, please check the status of each order and if any of them are delivered, I'd like to request a return for some items. Then. I'm interested in exchanging the remaining items in those orders new items of the same product type. After that, if have any pending orders left, can you please cancel them all due mistake? Make sure to follow up with me email confirm the status of each step and keep me updated.
Figure 16: Main failure mode PAE task generations in TauBench Retail as presented Figure 4 The task is ambiguous and non-verifiable.
30

---

Main Failure Mode of Verification Function Only
(To User) Your name Noah Brown and your email is noah. boah.brown8382G6 brown8882@cxamplcc com. You are modImn22eyd87011 For one of your latest order with Sabotta Degree Tea Series For Tea Drinkers and Chamber Stick, first say that you want cancel the order but when the agent asks for confirmation, say that you want instead return product; via (arl__ard_2211111 For the other order, cancel because of "order by mistake".
Instruction:
Verification Function:
success True
user_ 12345") result get_order_ detaild (ord0r__100-1113988322
last_namo= "Brow"", zip-"
and result ( (*status "*e* "returned*
result goi_order_ detaill (ortair srnrder_dd"#1i71122 success
(*status "cancelled"
return success
Main Failure Mode of Verification-Onle task generations in TauBench Retail as presented in Figure The task is not feasible because the order in the verification function does not exist.
Figure 17:
Main Failure Mode for Verification Only Solution
Instruction: (To User) Your name is Noah Brown and your email is wath.boown8822exxmmpp For your latest order, I want to exchange Water Bottle 'capacity': '500ml', 'material': 'stainless steel', 'color': 'green' to 'material': "glass"; via paypal paypal_S5969955 $796936.
Verification Function:
def avaluata(): the order detasls order get_order_detade detaile ('#X1787190*)
if success order ["status"] - "peading (iten modified)"
modified exccessfolly
Example Solution:
modify Modify-pendinge wwnding_orddr......oeoooooooeeee
order itema (order_ 1at ""an_ids==""63380000 new new_ -de
"], pernent__ethhd.... ipp'pppplllpp 5796936*)
Figure 18: Main failure mode Verification Solution task generations TauBench Retail as presented Figure The verification can be easily fulfilled even exchanging the wrong item.
31

---

Main Failure Mode of CaT
Instruction: (To User) Return the Smart Thermostat with item id 4983901480 from the order via paypal_ 3820631.
Verification Function:
def
evaluate():
success True
jet_orderrdetatte ("87889771)))
success success and order ["statua"] == "return requested" succesa success and len (order ('r (tretorn
and order ("return_itene"" [0] -- "4983901480"
success success and order
"psypal_3820631""
Example Solution:
return_ htt.r/ddwllardddrddereeeeee (order_1d *AU7896761", item_ids ["4803681337"],
3820331)
Failure Cases:
Failure Case 1, wrong idem
seaaaandddleeeee_oorrr...e..e (order_id "AV7895761", iten_ids ("4983901480*),
38206611)
Faclure Case wrong <tee
re deluur,ddliered
(order "AU7895761", item_ide ("4983901481"),
pttpratt_eetoodddddpspppp..... Patlune Case 1, wrong payment method
return_ orr.eddddiereed..drr..iiii. (order_iee *AU7895761", item_ide ("4983901480").
Figure 19: Main failure mode of CaT task generations with verification functions, example solutions, and failure in TauBench Retail presented Figure The instruction does contain enough inforation for the task completed because it does mention any relevant information user account.
32

---

Example Trajectory for Task Challenger
Enviromment:
instructions
You ebould cone up with your oun instruction with the user 1d: bromm AFTER CHECKING THE USER DETAILE AND THIIR ORDER DS7AILS.
Taask Challenger:
Devez arrivva as information vaa never through Loola provided
Environment:
Banindar; Only outgus actiion not: hoth, Invalid format. Plaase include wither ACTION:...POD ACTTON < AKSVER:.. .DD AUSMER.
Task Challenger:
ACTICR:
(waes
END ACTION
Environment:
( U'name' ('firat_neme' 'Boah', "laxt__nema' 'Brown' "addreas's ('addrrosdts Sanaet Drive', 'addrwas2'; 'state': 'zip': "80279"),
Buite "eity": "Denewz' "countyy"
"payaent nethddal ('paypal 57227333' ('source' "paypal'
payyal__r27SSe "eredit.cardd card card_7885802" ('aource "credit_card!" 'brand's "mastarcard", 'Jast_four' 18212",
ccrestt,
seaddit_card, sar_d_tt__rd__7158000000 }), "ordera's ('66n878072-1>>
Task Challenger:
ACTION:
ges_ www.oorer__dettee orr END ACTION
Environment:
(' ('order_1ee add:: '#7767807'', 'user_id'; 'noah, brown, 6101", 'addrece': ('addreesl's 1986 Suncet Drive' 'addrese2'; "Buite 250 "city': 'Denvez', "country" " "BAA "state"; 100 'zip': '80279"), "icexa': [('naze': 'Electzic
Kettle', producc__dd id'; "1078968781" idten_id'; '232397200"". 'price"; 145. 98, 'optious': ('capacity', 'It', "matorial" glase", 'colar'; black')), ('naze': "Gaaing Kouse" 'product id': '5713190933', 'iten_dd'; 12193828750", 'price'; 182.15, "options" ('color': "black" 'smmar type : 'laser' 'consectivtyy "sired ("naze" "Backpack", 'product 1252478922"
"iter, id's '866775316"', 'grice's 205 35, 'aptiona's ('coloe'; 'green', "asse": 'mail", 'raterial"; polyyererr' "copartmentt 'laptop11 1)], "talfillmett'' trackiakig ['5171770007881 I. "iter_id'': [2223992008' 19195628750" '866711110 131. 'statux': 'deliverdd
someen_bisstrry (C"tramaaction.type "payzent', "smount: 514.48, "payzent, pyypont_pethod_tee bdth 0727300077
Challenger:
ACTION:
pwi.ppoddes_deiee www.eelllppppuucttt..ccc Bariiictt_ddd------ (1-510799)7781")
END ACTION
Environmcnt:
('nare':
'product__dd; idd;: "1075065781" 'verienta's ('1940311997'; ('ites_id':
"capecity"; 'material's 'glam'. "caler's "ailowr"), 'available's "price': 137.17),
9132233862" ('isen.dd" 'price';
"options": E"cxpacity': ('isenidd"
'plactic",
('capacity'; "1.62", 'satertal"; 'glass', 'color's "white") 'svailable": Tree, 'price" 'opriens":
('capacity's "matertal"; 'stainless 'color": ), 'swatlable": True, "price": 153.26)33
Task Challenger:
ANSKER:
your axe pessinistic outgoing.
say you 22200299200 exchange
agent confinmation, say ('capacity's "1.5L', 'saterial': 'glass', ('capacity"; 'saterial'; "glass", 'color": "shite paypal.6727833
you
For exder zodification cancellation, say you are not interested any further changes. </insttractinn>
wwwllatioonffueet. def evaluate(): success Truee
order get_order_detaill (arrerrddettllla((a/77000)
success succese and (order (erder["statue ["status"] "exchange requested")
success success and (order "exchango, _ieeeann___teeee"" "] "2523972008") and (ontorder (eeederrexxcange (tcon="
success success and (ordue ('ezchange new. mes_itens itena" "9472639378") and Jan (order: ( a'axchange_eee _it_tuaae_______aaa success success and order ('exchango payssutttppppott..oom.m idt) as "12ppal_7227300"
return auссаzа
c/avaluat.ion_ ctt.iionffuuct.iin function?
caclution>
exchange. delligeedeliiee wetpeeddeeeeeeeeeerooo order. steneerrderrrrr.. '#V7578072", stea_.dds ddd ["292977200*1, no9_1ter__dd ["9472639370"),
-a payment_ www.od_ ppppppp 572255557
</solution>
<failure caae>
[onitted because of space]
</fmilure_com
END AKSUER
Figure 20: Example challenger trajectory using Llama-3.1- 3.1-70B Instruct in Retail.
33


Reviews:
[{"review_id": "nHfboX5gPI", "reviewer": "Reviewer_Noo7", "summary": "This paper proposes the framework of Self-Challenging Agents (SCA), enabling agents to generate high-quality tasks by themselves and conduct training. By introducing the \"Code-as-Task\" (CaT) form, ensure the feasibility, verifiability and difficulty of generating tasks. In multiple rounds of tool usage scenarios, the SCA framework has achieved significant performance improvements in both distillation and self-improvement Settings, demonstrating its potential in reducing human supervision and enhancing the autonomous learning ability of agents.", "questions": "Pleade see weaknesses for details.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. This paper presents a novel SCA framework and CaT form, addressing the issue of difficult task quality assurance in existing task generation methods and providing an effective approach for the self-improvement of LLM agents.\n\n2. Extensive experiments were conducted in multiple multi-round tool usage environments, and significant performance improvements were achieved compared with the baseline method, verifying the effectiveness and practicality of the proposed method.\n\n**Weaknesses**\n\n1. The implementation details of the CaT architecture need further clarification. For instance, does the task challenger go through multiple rounds of feedback processes when generating the validation function? And how does the cat framework handle the situation where the task challenger is unable to provide the correct instance solution?\n\n2. It needs to be compared with other similar self-improvement frameworks, such as Self-Adapting Language Models", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and positive feedback on the paper. We address your concerns below:\n\n\n## The implementation details of the CaT architecture need further clarification. \n\nYes, if the task challenger fails to construct a valid task, it will receive feedback from the environment. If the task challenger fails to construct a valid task within 10 turns, this trajectory where the proposer interacts with the environment will be filtered out with no valid task. Indeed we noticed in our experiments that sometimes this can be as low as a 5% valid task construction rate for the task challenger as shown in Figure 4, but CaT still leads to significant self-improvement results.\n\nBecause of the space constraints, a lot of additional experiment details were deferred to the appendix. For instance, Appendix A includes additional environment-specific implementations of CaT and Appendix I provides an example of the task challenger interaction trajectory. We will move more explanations into the main text in the camera ready version using the extra page of space.\n\n## It needs to be compared with other self-improvement framework, such as Self-Adapting Language Model\n\nIn Table 1, we do have comparisons with the state-of-the-art self-improvement framework for agents PAE at the time of submission, and we have shown the advantage of SCA. While Self-Adapting Language Model is also related to self-improvement, it is released on June 12th – after the submission date – and it has not been tested in the agent domain."}, {"review_id": "u3YTVXEeFz", "reviewer": "Reviewer_BRDP", "summary": "The authors proposed the Self-Challenging framework that trains an agent on high-quality tasks that are generated by itself. There are two roles for the agent: task challenger that gather info and possible tasks,  and task executor that are trained with these tasks. The evaluation on two agent benchmarks \"M3ToolEval\" and \"TauBench\", shows the improvements with self-generated training data.", "questions": "1. There are four different categories for these generated trajectories: false negative, false positive, true positive, true negative. In the appendix, the false negative is defined as \"Agent got a reward of 0 but it is because the task is ambiguous or verification function is wrong\". And False positive is \"Agent did not complete the task but reward is 1\". How about the case that \"the reward is 1 and verification function is wrong\"?\n\n2. In Table 3, the ablation study shows on successful trajectories only lead worse performance. What is the ration of successful trajectories? the lower performance can be a consequence of un-sufficient samples in the training. It seems hard to conclude that unsuccessful are useful.\n\n3. The plot in Figure 6 is not smooth. Are multiple runs in the experiments?", "limitations": "No. Some discussions on how to prevent high risks of generated data can be done.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This work proposes a framework that can interact with environment can generate training data by itself. In addition, large improvements are obtained with self-generated training data after RL training. The framework is completed and shows the consist results on two agent benchmark.\n\n2. There are several good interesting analysis in the experiments: comparison between self-improvement and distillation; the effect of different RL algorithms; the impact of different components of framework CaT; scale effect of the number of tasks or the number of trajectories per task etc. All of these analysis enrich the contribution of this work.  \n\nWeaknesses:\n1. When checking deeper into two different settings (self-improvement and distillation), two different training loss are used. According to Table 1, the distillation setting achieves higher results. In this setting, a stronger LLM are used to sample a dataset among tasks in framework CaT, then, SFT are used in the sampled dataset.  Is it unclear whether SFT loss can be better in Self-improvement loss (RL algorithm).\n\n2. In this proposed framework, as claimed by the authors, a non-trivial percentage of  false negative example can be generated. intuitively , false positive samples can hurt the model seriously. Although as shown in Figure 4, the false positive ratio is very low in some environment (Retail environment). However, it is hard to guarantee whether the low false positive ratio in other environments. What are the possible ways to prevent these risks?", "was_revised": true, "final_justification": "The rebuttal solves most of my previous concerned parts (for example, training details). However, I am still concerns about the correctness of the syntactic \"verifier\". Maybe it is not a big issue for complex tasks, however, it is highly risky parts. Although the results (one Retail environment of one model ) in Figure 4 already show the reduced False Positive and False Negative rates, only two multi-turn tool-use LLM agent benchmarks are tested. It is still lack the strong convincing evidence. More related results would be more convincing.", "rebuttal": "Thank you for your review and positive feedback on the paper. We address your concerns below:\n\n\n## It is unclear whether SFT loss can be better in Self-improvement loss (RL algorithm).\nWe believe that there is a misunderstanding here. While indeed we adopted REINFORCE as our primary RL algorithm for the self-improvement setting, because of the reward structure of CaT in our experiments it is actually equivalent to SFT. As explained in Line 166-168, “Because of our reward structure with 0/1 outcome reward, this RL optimization objective is essentially equivalent to performing supervised finetuning (SFT) on successful trajectories (i.e. Rˆ 167 cˆ(sT , aT ) = 1) only, i.e. Rejection Fine-Tuning”. Importantly, however, comparisons of more RL losses – DPO, PPO and GRPO – can be found in Figure 3.\n\n## What are the possible ways to prevent these risks of false positives?\nAn important component of SCA is generating failure cases when creating the task – which we have found to significantly reduce false positives, see Figure 4. When the model generates failure cases, it can try to come up with as many corner test cases and failure modes as possible. CaT validation procedure ensures that the final tasks pass the tests of these corner test cases, so that during agent rollouts those corner cases will not be accidentally classified as success. Intuitively, the more corner test cases there are, the more robust the task should be against these false positives.\n\n## How about the reward is 1 and verification is wrong?\nIn our understanding, for the case of the reward is 1 and verification is wrong seems to mean that the agent got a reward 1 but it should get a reward of 0, meaning that the agent did not finish the task. This case seems to be included in \"Agent did not complete the task but reward is 1\", studied in Fig 4 and Sec 5.3 (see also Appendix C for annotation details).\n\n\n## It seems hard to conclude that unsuccessful are useful from experiments in Table 3.\nIndeed, there is a confounding factor in this appendix experiment that there are only about 30% successful trajectories compared to all trajectories, if we keep the number of interactions the same for fair comparison. As a result, we refrained from making general conclusions in the appendix and only report this experiment result as an observation that we have found when the performance of the teacher model is significantly better than the student model. Further study of these aspects in future work would be good!\n\n## The plot in Figure 6 is not smooth. Are multiple runs in the experiments?\nBecause of the compute constraint and the fact that we need to train a separate model for each individual data point, we were unable to perform multiple runs to produce a smoother curve. However, from the 16 data points that we have gathered in Figure 6 with different numbers of training trajectories and training tasks, it already underscores the effectiveness of scaling the number of tasks through SCA compared to solely scaling the number of trajectories with the same tasks."}, {"review_id": "e7EbsGWWNa", "reviewer": "Reviewer_mkt7", "summary": "This work tailors a framework to generate high-quality tasks for training LLM-based agents, aiming at alleviating the human efforts required to create such tasks. Agents in this framework perform two distinct roles, i.e., challenger to interacts in an unknown environment with tools to gather information and generate possible tasks, and executor to accomplish these generated tasks by interacting with the same environment. Code-as-Task and several mechanisms are incorporated to ensure the quality of generated tasks.", "questions": "The paradigm in which the challenger generates questions for the executor to solve reminds me of Generative Adversarial Networks (GANs), where a major challenge lies in the need for both the generator and the discriminator to improve simultaneously, Otherwise, a large gap in capability between the two can lead to suboptimal training results. In this work, I have only observed improvements to the executor, raising the question of whether this work might encounter similar issues.", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "1. The proposed method is novel and targets a practically valuable issue in training LLM-based agents.\n2. This manuscript is well-organized with clear illustrations, making it easy to follow.\n\n### Weaknesses\n1. The experiments were conducted with only one model scale and one model architecture, which raises concerns about the generalizability of the proposed approach. Models of different sizes and architectures possess varying initial capabilities, so it remains unclear whether this method can be applied to a broader range of scenarios.\n2. The discussion of failure cases is insufficient. More analysis is needed on the scenarios in which the proposed method fails to perform effectively.\n3. More comparisons with existing methods in terms of computational cost are needed to demonstrate the practicality of the proposed approach.", "was_revised": true, "final_justification": "My previous score is accurate.", "rebuttal": "Thank you for your review and positive feedback on the paper. We address your concerns below:\n\n\n## The experiments were conducted with only one model scale and one model architecture, which raises concerns about the generalizability of the proposed approach.\nBecause of compute and GPU memory constraints, we were only able to perform experiments at 7B scale. However, in our empirical results, SCA has been shown to improve agent performances across different environments (Table 1), across distillation and self-improvement settings (Table 1), across data scales (Figure 6), and across RL loss choices (Figure 3). We believe that these experiments already provide evidence for the generalizability of SCA.\n\n## The discussion of failure cases is insufficient.\nBecause of space constraints, we have deferred additional analysis of failure cases to the appendix (Appendix B and H, including examples in Figs 16,17, 18 and 19) and also referred to them in a dedicated Limitations Section (Section 6). Figure 4 shows that there is still a non-trivial percentage of False Negative samples and Appendix B demonstrates that there is still a sub-optimality gap between synthetic tasks from SCA and manually crafted tasks. We hope the discussion of these limitations can provide directions for future research. In the camera ready version with additional one page space we can move more of this discussion to the main paper.\n\n## More comparisons with existing methods in terms of computational cost are needed to demonstrate the practicality of the proposed approach.\nThe computational cost of SCA is comparable to other self-improvement methods like PAE because the main computation overhead lies in collecting agent rollouts to perform RL instead of the task proposals. We will provide more details on this in the camera ready version.\n\n\n## In this work, I have only observed improvements to the executor, raising the question of whether this work might encounter similar issues.\nWe also believe that such self-play would be a very promising future direction so that the executor and challenger can improve together. In our experiments, we did observe that the executor started to saturate on the tasks the challengers can offer. We believe this is an exciting research direction for future work."}, {"review_id": "1UtP9vLYIT", "reviewer": "Reviewer_ndna", "summary": "This paper proposes Self-Challenging Agents (SCA), a framework for training multi-turn, tool-using language agents using self-generated tasks. The agent plays both a challenger role, where it generates novel tasks through environment interaction, and an executor role, where it learns from those tasks via reinforcement learning. To ensure task quality, the paper introduces a Code-as-Task (CaT) format, which combines an instruction, a verification function, an example solution, and failure cases. The method is evaluated on M3ToolEval and TauBench benchmarks in both self-improvement and distillation settings, showing promising performance gains without relying on existing tasks.", "questions": "Can you include an explicit ablation study comparing model performance with vs. without CaT filtering (e.g., training on unfiltered tasks)?\n\nPlease clarify how the same model can generate and solve tasks that are both “challenging” and beneficial, if it already knows the answer.\n\nConsider standardizing terminology and formatting (e.g., TauBench vs. taubench), and clearly explain the numeric labels in Figure 4.\n\nIf possible, report reward or success rate distributions of CaT-filtered vs. unfiltered tasks, to provide stronger support for the value of the filtering pipeline.\n\nIf the rebuttal includes direct evidence that filtering leads to performance improvements (e.g., via ablations), and clearly explains how the challenger/executor dynamic leads to new capabilities rather than repetition, I would be open to raising the score.", "limitations": "The paper contains a dedicated Limitations section that acknowledges key challenges such as false negatives and environment-specific overfitting.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\nThe paper tackles an important problem: scalable task synthesis for multi-turn agent training, without manual annotations.\n\nThe proposed CaT format is a creative attempt to ensure verifiability and difficulty of synthesized tasks.\n\nEmpirical results suggest notable gains in both distillation and self-improvement settings, particularly in partially observable environments.\n\n**Weaknesses:**\n\nThe central claim is that the filtering process (i.e., verifying the CaT components) leads to higher-quality tasks and better agent performance. However, no direct ablation on task filtering vs. not filtering is presented. The closest evidence (Table 4 / Figure 4) involves human-labeled task quality and difficulty distributions, which only indirectly support the claim.\n\nThe filtering relies on the challenger model generating a correct solution. But in the self-improvement setting, the challenger and executor are the same model. If the model can already generate valid solutions, it’s unclear why these tasks would lead to further improvement. There might need a clear explanation of how the model benefits from solving tasks it can already solve.\n\nThere are some writing inconsistencies, such as the inconsistent use of “TauBench” vs. “taubench”, “Verification Only” vs. “verification-function only”. Some visual elements (e.g., Figure 4's “100% / 47.7% / 9.5%” labels) are not adequately explained in the text.", "was_revised": true, "final_justification": "Concerns about the paper including evidence of filtering effectiveness and challenger & resolver dynamics have been addressed. If the final version clearly explains the underlying intuition and includes the experimental results to demonstrate the effectiveness of the proposed  methods, I will raise my score to 4.", "rebuttal": "Thank you for your review and feedback on the paper. To address your concerns, we have collected additional experiment results with direct evidence of performance improvements and explain the dynamics between challenger and executor.\n\n\n## Direct evidence that filtering leads to performance improvements \nTo address your concern, we have included an explicit study comparing the model performance with and without CaT filtering. Specifically, we considered the ablation “SCA no filtering” that prompts the task challenger to interact with the environment and directly generate the instruction and evaluation function. The only difference of “SCA no filtering” from “SCA” is the lack of the CaT filtering mechanism. We generate 800 tasks and rollout 12K agent trajectories for each method. As shown in the following table, while SCA no filtering might have got some positive learning signals from the synthetic tasks to improve pass@4 performance, we found that the generated instructions and evaluation functions are quite noisy without filtering (as shown in the human analysis in Figure 4) and resulted in sub-optimal performance compared to SCA, e.g. pass@1 drops from 13.0 to 8.9. This shows that the filtering proposed in our paper is important.\n\n\nTau-Bench Retail\n| Model               | pass@1 (average over four runs) | pass@4 |\n|---------------------|--------|--------|\n| Llama-3.1-8B-Instruct        | 8.9    | 15.7   |\n| PAE                 | 9.8    | 18.3   |\n| SCA no filtering    | 8.9    | 20.8   |\n| SCA                 | 13.0   | 25.2   |\n\n## Dynamics between challenger and executor\nWe respectively disagree that the inclusion of an example solution in CaT limits the potential of self-improvement. In SCA, self-improvement can happen in at least two different ways:\n**Many tasks are easier to verify than proposing a solution** (https://www.jasonwei.net/blog/asymmetry-of-verification-and-verifiers-law) so it might be easier for the agent to come up with an evaluation function than proposing a solution. Once the task challenger has an evaluation function, it can try to “guess” the solution even if it does not know how to solve the problem. Thanks to the filtering mechanism in CaT, the wrong guesses are filtered and the valid tasks that remain can be used to reinforce the correct guesses when the same model serves as the executor. In this way, the executor keeps practicing by being rewarded for correct guesses until eventually it knows how to solve the problem and does not need to guess any more. Hence, the example solutions help it find what it can improve.\n**Hindsight Experience Replay** (https://arxiv.org/abs/1707.01495) has historically been a successful self-improvement method over many applications. The core idea is that the agent might accidentally hit some goals through explorations and we can use the goals that the agent accidentally hits as “hindsight experience” to improve the goal-reaching capability of the agent. SCA can operate in the same way as Hindsight Experience Replay. In the retail environment, the challenger constructing the task might randomly explore some user information in the database and try to run some commands such as exchanging a water bottle to a different size. If any one of the commands succeed, a confirmation message will be returned saying that the water bottle has been exchanged successfully. Based on the confirmation, the challenger knows that some changes have been made and can generate a user instruction in hindsight such as exchange the water bottle for John. But completing the task in the forward way might be much harder where the agent needs to first intentionally figure out the user id for John, and use the user id to find the transaction that contains a water bottle, and checks what sizes are available.\n\n\n## There are some writing inconsistencies, such as “Taubench” and “taubench”.\nThanks for spotting those inconsistencies, we will fix them in the next version."}]

Author rebuttal:

Meta-review: This paper introduces self-challenging agents, a framework for enabling language model agents to generate their own training tasks. The framework incorporates a code-as-task format, which ensures verifiability and difficulty by combining instructions, verification functions, solutions, and failure cases. The agent plays dual roles as challenger and executor, and is evaluated in both self-improvement and distillation settings on two multi-turn tool-use benchmarks (M3ToolEval and TauBench), achieving consistent performance gains.

**Strengths:** 
1. Address an important and practical challenge of scalable task synthesis for multi-turn agents without manual labels; 
2. A novel CaT formulation that improves task quality and reduces supervision;
3. Comprehensive experiments and analyses that highlight improvements across different training strategies.

**Weaknesses:**
1. Only one model scale and two benchmarks are tested; 
2. Insufficient comparisons with related self-improvement frameworks and computational efficiency;
3. Limited discussion of failure cases and risks (e.g., false positives in task verification); and
4. Certain writing and presentation inconsistencies. 

**Conclusion:**

Overall, despite some weaknesses, I agree with the reviewers that the paper makes a meaningful contribution by proposing a novel and practical framework for self-improving agents, with promising empirical results. Therefore, I recommend acceptance. The final revision should include broader experimental validation, stronger ablations, and clearer explanation of the dynamics between challenger and executor roles.
Decision: Accept (poster)