AgentRefine: Enhancing Agent Generalization through Refinement Tuning
Authors: Dayuan Fu
Keqing He
Yejie Wang
Wentao Hong
Zhuoma GongQue
Weihao Zeng
Wei Wang
Jingang Wang
Xunliang Cai
Weiran Xu
Venue: ICLR 2025 Poster (2025)
Keywords: agent
self-refine
diversity
generalization
data synthesis
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: The self-refine data can expand the search space of LLM agent and improve the reason quality, leading a generalized performance in agent tasks.

Abstract: Large Language Model (LLM) based agents have proved their ability to perform complex tasks like humans. However, there is still a large gap between open-sourced LLMs and commercial models like the GPT series. In this paper, we focus on improving the agent generalization capabilities of LLMs via instruction tuning. We first observe that the existing agent training corpus exhibits satisfactory results on held-in evaluation sets but fails to generalize to held-out sets. These agent-tuning works face severe formatting errors and are frequently stuck in the same mistake for a long while. We analyze that the poor generalization ability comes from overfitting to several manual agent environments and a lack of adaptation to new situations. They struggle with the wrong action steps and can not learn from the experience but just memorize existing observation-action relations. Inspired by the insight, we propose a novel AgentRefine framework for agent-tuning. The core idea is to enable the model to learn to correct its mistakes via observation in the trajectory. Specifically, we propose an agent synthesis framework to encompass a diverse array of environments and tasks and prompt a strong LLM to refine its error action according to the environment feedback. AgentRefine significantly outperforms state-of-the-art agent-tuning work in terms of generalization ability on diverse agent tasks. It also has better robustness facing perturbation and can generate diversified thought in inference. Our findings establish the correlation between agent generalization and self-refinement and provide a new paradigm for future research.

Published as a conference paper at ICLR 2025
AGENTREFINE: ENHANCING AGENT GENERALIZA- TION THROUGH REFINEMENT TUNING
Dayuan ; Keqing Yejie Wang', Wentao Hong's Zhuoma Gongque', Weihao Zeng', Wei Wang". Jingang Wang?. Xunliang Cai², Weiran Xu
BBejjin University of Posts and Telccommunications, Beijing, China Meituan, Beijing, China
ABSTRACT
Large Language Model (LLM) based agents have proved their ability to perform complex tasks like humans. However, there is still large gap between open- sourced L.MMs and commercial models like the GPT series. this paper, we focus improving the agent generalization capabilities of LLMs via instruction tuning. We first observe that the existing agent training corpus exhibits satisfactory results held-in evaluation sets but fails to generalize to held-out sets. These agent- tuning works face severe formatting errors and are frequently stuck in the same mistake for long while. We analyze that the poor generalization ability comes from overfitting to several manual agent environments and a lack of adaptation to situations. They struggle with the wrong action steps and the experience but just memorize existing observation-acee action relations. Inspired by the insight, we propose novel AgentRefine framework for agent-tuning. The core idea is to enable the model to learn to correct its mistakes via observation in the trajectory. Specifically, we propose an agent synthesis framework to encom- pass a diverse array of environments and tasks and prompt strong LLM to refine its error action according to environment feedback. AgentRefine significantly outperforms late-offthhe-rt agent-tuning work in terms of generalization ability on diverse agent tasks. also has better robustness facing perturbation and can generate diversified thought in inference. Our findings establish the correlation between agent generalization and self-refinement and provide a new paradigm for future research.
learn from
INTRODUCTION
Language agents Mialon et al. 2023 Sumers et al. 2023),
Held-in
Held-out
which harness the powerful capabilities of large language models (LLMs) to perceive environments, make decisions, and take actions, have emerged as an effective solution to com- plex real-world problems. Plenty of agent projects such as Au- toGPT (Sig). GPT-Engineer (gpt), and BabyAGI (yoh) have employed LLMs as the core controllers, showing potential for practical applications. Both prompt engineering (Yao et al.) Zhao et al. 2024) and framework prac- 2024 Shinn et al. 2024) have been proposed enhance the agent capability of top-tier commercial LLMs
Progress %%)
Agent-FL
AgentRvelnne
2022 Fu et al.
2024
BabyAl SciWorld
Jericho
tice Yao et al.
Figure 1: Overall progress score among 5 tasks. Agent-FLAN has been trained on Held-in task.
like GPT-4. Recently, open-sourced LLMs (Dubey et al. 2024 Jiang et 2023) are emerging as effective alternatives GPT
models and show promising results.
Many efforts have been made enhance the agent capability of open-sourced LLMs via finetuning. Deng et al. 2024); Qin et al. 2023) carefully define single task schema and collect agent data for
"Equal contribution. Emails: Edy@bupt.edd edu. en, Code: http://lwwbb..oooouuuuuDuuu////a "ooresponniig authors.

---

Published as a conference paper at ICLR 2025
specific vertical fields. Further, Zeng al. (2023): Chen al. (2024); Hu al. (2024) extend to diverse agent tasks and cover hual-quality unseen tasks. Although these works achieve admirable perfor-
(CoT)) rationale (ao et al. 2022) to
enhance the agent performance mance on held-in agent tasks where the collected training data share the same environment, their generalizability to more held-out sets is poor (shown in Figure [ To solve the generalization is- sue of agent- -unnng, (Zeng et al. 2023] Chen et al., (204)) mix general alignment data, ShareGPT (Chiang et al., 2023) with their agent data. They conclude that the general capabilities of LLMs are necessary for the generalization of agent tasks and training solely on agent data always leads to a decline in held-out agent performance.
In this work, we revisit the hypothesis that training solely on agent data can't generalize to new environments and delve into the reasons behind agent capability generalization. We first investigate the errors of the existing agent-tuning work in the new agent environments and most of them are formatting errors, illogical reasoning, and duplicated generation. While the integration of general data ratios can partially mitigate these errors, we find current agent models struggle with the same mistake and repeat erroneous actions, even when the environment provides explicit negative feed- back. Inspired by (Shinn et al. 2024 Madaan al. 2024), connect the generalization of agent capability with self-refinement (Madaan et al.) 2024) according to the feedback signals from the agent environment. We argue good agent should recognize its mistakes and refine the previous actions by interacting with the environment. The self-refinement ability enables the agent to learn from its mistakes, avoiding getting trapped in specific predicament, and allows discover the correct sequence of actions through reasonable exploration.
Expanding on the aforementioned insight, our objective is develop generalized agent-tuning data and establish the correlation between agent generalization and self-refinement. To this end, we first propose an agent synthesis framework to encompass diverse array of environments and tasks drawing upon extensive human persona data (Chan et al. 2024) that reflects various professional roles and personal interests. The diversity of agent environments prevents the model from overfitting to single scenario. 'hen for each generated agent environment and corresponding task, we ask strong LLM to simulate a multi-turn interaction. After generating each turn, we use a verifier to detect whether it contains format or logical errors. We keep the error turn and prompt LLM to refine its action according to the observation. The final agent data will undergo self-refinement processes and ultimately lead to correct result. We find that we call Re eemeentnt Tuning, enhances the agent to explore more viable actions while meeting bad situations, thereby resulting in better generalization to new agent environments.
on the self-refinement data, which
In this paper, we present AgentRefine, which investigates the self-refinement in agent-tuning to enhance agent generalization. We perform refinement tuning using our synthesis data the LLaMA3 (Dubey et al., 2024) and Mistral-v0.3 (Jiang et al.) 2023). Our experiments in terms of five agent evaluation tasks demonstrate that AgentRefine significantly outperforms state-of-the-art agent-tuning work. The key findings are summarized as follows:
While existing agent-tuning work improve held-in agent performance, they hardly general- ize the ability to new agent tasks. In contrast, our AgentRefine does not depend on mem- orizing training trajectories but learns to self-refine its mistakes and explore more actions and reasonable paths.
Our experiments demonstrate that agent-tuning on normal trajectories performs poorly to the small perturbation of agent environments, like the action description. Refinement tun- ing exhibits greater robustness to environmental changes.
Further analysis indicates the diversity of agent environments and thoughts contributes to refinement tuning.
2 RETHINK THE GENERALIZATION OF AGENT-TUNING
Current agent-tuning works lack generalization to new agent tasks. Figure compares the perfor- mance between held-in and held-out agent tasks, where Agent-FLAN utilizes the Alfworld environ- ment to gather training data and subsequently makes direct predictions for the held-out tasks. We observe a clear performance drop between the two settings.
2

---

Published as a conference paper at ICLR 2025
n) Alfworld
by Alfworld (action changed)
c) SciWorld
Gval: cool bxnato is irieosatt....
Gool: per azd torveo micmowve
Goat: metal lirk, whiciie loeated around the bashoom,
Acallablee octione..... tike (ob)) frrm (naup)
Acsiloblle actiotc from (rocpp) okk (ebj)
Anallable actions
Acton Io coumanap I.
dation Boon counterie
Thouger & the fotk wiigh be the Bianee
Obsonwdinn
happets
Action 1g
Action Biecninn aoaeootoo
Aalen 72
Obrernation You pick the tonato from the oostremop
coumtetto
dalen Z3: resoo
Action No:
Figure 2: Example of parameter memorization in Agent-FLAN.
Memorizing trajectories leads to overfitling. To further figure the reason behind the poor generalization, we employ a study on the robustness of Agent-FLAN, EegureD]dssplay the different output results in three evaluation settings where (a) denotes the original output the held-in Alf- world task, (b) represents the modified Alfworld task with only reordering the action description, and (c) means the held-out Sci task. Agent-FLAN fits well into the held-in agent environment but fails to recognize subtle perturbations or handle new tasks ($4.)) Moreover, we analyze the bad cases of existing agent-tuning work in the held-out tasks and observe that once the model outputs an
error action, the entire process will be stuck in the same error mode for while, regardless the observation (:7). These ex- perimental results indicate that traditional approaches merely
we's Peratttte
memorize the correct trajectory information, fundamentally leading to lack of generalization capability.
Suase Bate
Not memorize but self-refine. Inspired by recent work (Shinn eet al. 2024: Madaan al.) (024)), connect the generaliza- tion of agent capability with self-refinement based on envi- ronment feedback. We hypothesize that self-refinement abil- ity enables the agent to learn from its mistakes and discover the correct sequence of actions through reasonable exploration
Agmi--ym
Agmn-HLAN
Aganikeanee
The success rate varia- tion via perturbation
Figure
3 METHODOLOGY
3.1 DATA CONSTRUCTION
Inspired by the Tabletop Role-playing game (TRPG), AgentRefine data's construction process can be divided into three parts: script generation, trajectory generation, and verification, as shown in Figure 4 The script generation requires the LLM to generate script with the environment, tasks, and available actions based on the persona. In the trajectory generation phase, the LLM is required simultaneously play the roles of both Dungeon Master (DM) and player to generate multi-turn agent data containing errors and refine steps based on the script. The verification will verify the script and trajectory, giving LLM the mistake it has made within given persona and the LLM will regenerate the script/trajectory based on the verifier's s response.
Script Generation We first sample persona P; from diverse personas (Chan prompt the LLM to generate a script with the environment, tasks, and available actions based on P:- The environment will include locations, items, and player information that may appear in the interaction. To assist the LLM in understanding the environment, we prompt the LLM to display the hierarchical relationships between locations/items in JSON format. We also require the LLM to generate some interfering locations/items, to ensure that some erroncous steps are likely to occur during trajectory generation. After generating the environment, the LLM will generate a clear and specific task. Finally, the LLM will generate series of available actions. For each action, require the LLM to generate an action name, validation code (a regular expression), and valid parameters. The structure of the script can be seen in Appendix L
2024). and
Trajectory Generation Given a script, the LLM can simulate multi-ture turti interactions between the DM and the player within one call. Specifically, the DM's turn is divided into three stages: thinking.
3

---

Published as a conference paper at ICLR 2025
Data Construction
Refinement Tuning
Script
Trajectory
Trajectery
Sompp
acfice short
Rafremerr Tunnno
Erroncous Misking
Persona Hubb
LLM
Sornec
Reponerncc
ccrificction
actions
Reperertts
X
-> J(0)
Compute
Inttomments
Trajectory
Rale based
Action Check
nemterr emprneed to
Quariity Shock
AgentRefire
(uevect)
Figure 4: The pipeline of AgentRefine data generation and refinement tuning.
observing, and evaluating. In the thinking stage, we require the LLM to evaluate the player's state and known information so far and analyze the observations the player can obtain based on the last action. The observing stage will provide the observations the player uating stage, the DM will assess whether the player's ' last action contains parameter errors, logical errors, and location errors (act in the wrong place). The player's 's turn similar ReAct, requiring the LLM to analyze the current state through thought and then propose an action. The structure of the trajectory can be found in Appendix M
obtain, while in the eval-
Verification The verifier will check both the script and the trajectory. In script part, to ensure the validity of the action names, we apply the validation code the action names and only save the
script if all actions pass the validation |_] In the trajectory part, if the generated trajectory has: (1) ISON format error certain turn (2) The task is not completed in the final turn 1 (3) In the player's / turn its action can not match any validation code with corresponding parameters and the DM does not provide a parameters error turn + 1, we will save all previous turns up to / and prompt the I.I to continue generating. If the DM evaluates that the task is completed but the number of crror-refine turns in the trajectory is less than two, we will provide turns to the LLM and require it to regenerate the trajectory from the beginning. Detailed verification steps can be seen in Appendix
3.2 GENERATION SETUP
We use gpt-4o-200 opt--4-2024-05 2024-05-1-5 13 to generate the script and trajectory. We will save all trajectories that can pass verification in 4 LLM calls (including script generation and trajectory generation). We primarily adopt the I-shot trajectory example approach in trajectory generation and the 3-shot script examples in script generation to help LLM follow the format and give diversified result. In Appendix [ we use -22.5 (Liu et al. 2024) as the open-source LLM to generate the script and trajectory.
3.3 REFINEMENT TUNING
After generating the complete trajectory. we convert the trajectory into Refinement Tuning dataset D RT, specifically, the user turn is the DM's observation, while the assistant turn is the Player's thought and action, in ReAct (Yao et al., 2022 format. To prevent interference from error turns generated by the LLM, we changed the loss function J(O), as shown in Equation | where N is the total turn number of a given data x, Ty, A,, O, is the thought, action, and observation in turn j. A, is correct 1(A)) 1 else 1(A,) 0.
((Ø) = DET
Σ log (No (T;, A/T, (T, (1
(1)
DDu to the near-infinite parameter space of actions in virtual environments such code editing, answering, and searching, these actions will not be verified in both script generation and trajectory generation
4

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
4.1 EXPERIMENT SETUP
Training We use the LLaMA3-baa series models (Dubey et al., 2024) for most of our experiments. For mistral (Jiang et al., 2023), we use mistral-v0.3. We applied the original llama3 (or mistral)'s multi-turn chat template. We use LLaMA-Factory (Zheng et al. 2024) to train our models. The training hyperparameter details can be seen in Appendix
Tasks We select 5 tasks: SciWorld (Wang et al. 2022), Alfworld (Shridhar et al.) 2020). BabyAI Shevalier-Boisvedr et al. 2018). PDDL (Vallati et al. 2015), and Jericho (Hausknecht et al. 2020).
all of them testing models' decision-making ability. We the AgentBoard (Ma et al. 2024)
framework for experiments, this framework can determine whether the agent has completed all tasks (success rate) and whether the agent has reached key nodes (progress rate). The Held-in task refers to Alfworld, while the Held-out tasks are the results obtained by the weighted average of other tasks based on AgentBoard (Ma et al. and the historical thought, action, and observation will be transformed into the chat format instead of plaintext. We adjusted the example prompts on Llama-3-8B 3-8B-Inntruect and never changed them during this work. (except $4.3). The max turn is 30 for all these tasks in inference. To further prove AgentRefine's generalization, followed by the choice in ReAct ( (Yao al., (022), We choose reasoning task QA (Yang et al. (2018) in the ablation experiment. We use Wikipedia search in LATS (Zhou et al.) 2023) as the environment, randomly sample 300 questions from HotpotQA, and test the exact match (EM) and Fl score of those methods. The max turn is 8 for HotpotQA task in inference. It should be emphasized that we will only use environment feedback in the inference and we will not use GPT4's 's judgement the feedback.
2024) We change AgentBoard's prompts from Act-only to ReAct
Baseline For the close-source model, we choose GPT-4o (0p1-40-2024-05-133 and GPT40-mini 111-440-ini-202400--111 For the open source model, we choose 1at-Lllmaa-----nnttuue Meta- Llama-3-70B Instrrcc, and Mistral- 1-l-----t--nntttOtO---
For fine-tuned mode, we choose Agent-FLAN (Chen al. 2024). AgentGym (Xi et al. 2024), and AgentGen (Hu al.. 2024) the baseline. They are all trying to solve the agent generalization problem. Agent-FLAN is an improvement of AgentTunning (Zeng et al., 2023). focusing on training "thought" in ReAct. AgentGym uses lots of environments to ensure generalization and AgentGen uses LIMA (Zhou et al. (204) to synthe- size diversified agent-tuning data. FLAN includes Alfworld in its training set. AgentGym includes Alfworld, BabyAl, and SciWorld in its training set. These datasets will be seen as Held- in test tasks for the corresponding method. Since Agent-FLAN and AgentGym's 's original model is LLaMA2-Chat, for fair comparison, we reproduce them under LLaMA3 and Mistral. Since AgentGym has not open sourced, we only report the result in (Hu et al. 2024
4.2 MAIN RESULTS
Table the performance comparison of AgentRefine and other methods across different fam- ilies and sizes. It is important to emphasize that some methods sample training data in the same environment as the task; in such cases, consider this task for these methods be held-in. We identify the held-in metrics for each method with an underscore. It can be observed that compared to other agent works, our method shows significant advantages in held-out tasks, For example, it leads Agent-FLAN by 13.3% in Sciworld Success Rate. Notably, in some tasks, AgentRefine can even match the performance of the GPT-4o series. This demonstrates the strong generalization capability of AgentRefine. We also observe that AgentRefine can not outperform held-in training methods. However, in $ 4.3, we will demonstrate that these held-in methods simply memorize the mapping between observation and action, and very small perturbation can render these methods ineffec- tive. Furthermore, we also notice that I.LMAA3-8B- Instruct exhibits very strong performance in many tasks. We attribute this to its extensive use of Alignment data and additional RL training. In subsequent experiments, we also mix alignment data and AgentRefine and achieve further gains.
Effect of Refinement Tuning To further investigate the effectiveness of Refinement Tuning, we mask the loss of refinement trajectory tokens. Table 2 shows that after masking the refinement, the model's 's performance over 5 tasks drops dramatically. For instance, there is approximately 43% performance drop in Sciworld which, to some extent, reflects the necessity of Refinement Tuning for Agent tasks. we also re-generated training set without error and refinement trajectories, which
5

---

Published as a conference paper at ICLR 2025
Method
Alfworld
BahyAl
SeiWorld
PDDL
Jericho
Success
Progress
Success
Progress
Success
Progress
Success
Progress
Success
Progress
GPT Series
GPT-4o
66.4
79.9
76.9
61.7
69.8
GPT-44omini
37.3
65.0
36.6
$1.9
23.3
49.8
25.0
49.1
10.0
28.5
LLaMA-3-88 Series
LLaMA-3 3------881 ----nnsiuet
22.4
46.1
45.5
56.5
7.8
41.1
10.0
38.4
0.0
24.3
AgentGen
29.1
47.6
20.5
35.0
11.7
23.0
AgentGym
61.9
76.9
47.3
61.4
18.9
47.5
1.7
16.6
0.0
12.9
Agent-Fi..n FINAN
67.2
79.7
25.0
35.3
1.1
10.9
8.3
25.5
0.0
10.1
AgeniReline
44.8
63.8
37.5
50.4
14.4
42.6
16.6
37.8
10.0
32.3
Mistral Series
Mistral-7B 1--mntrructv0033
12.4
13.3
AgentGiym
76.9
86.7
40.2
$6.3
15.6
48.3
1.7
7.3
0.0
13.0
Agent-FLAN
77.6
AgentRetine
31.4
68.3
25.9
42.4
4,4
22.4
11.7
32.8
5.0
28.8
LLaMA-3-70B
LLaMA-3-7000 Instruct 67.2
75.2
48.2
61.8
42.2
75.4
55.0
79.8
25.0
46.4
Agent-FLAN
80.5
86.8
32.1
41.2
16.4
25.0
53.7
0.0
13.6
AgentRefine
67.2
72.1
44.6
59.7
17.7
46.4
38.3
58.6
15.0
37.2
Table 1: Main Results. The underlined text indicates that the training data is sampled in the same environment as the task and is considered as held-in evaluation. We use the original result in Agent- Gen and reproduce AgentGymm and Agent-FLAN's results.
Method
Alfiworld
BabyAl
SciWorld
PDDI.
Jericho
Success
Progress
Success
Progress
Success
Progress
Success
Progress
Success
Progress
AgentRefine
48.5
61.5
37.1
$1.7
7.7
33.1
21.7
37.4
5.0
26.2
refinement loss
40.3
58.8
34.8
45.6
4.4
22.7
20.0
37.4
0.0
16.1
w/o refinement data
49.3
65.2
30.4
43.1
5.5
21.3
11.7
32,5
0.0
13.8
29.9
43.9
23.2
31.6
3.3
19.0
8.3
28.3
5.0
18.4
Table 2: Ablation study of Refinement Tuning. This experiment is in the data size of 8000.
completely eliminates the impact of Refinement Tuning. From Table 2 model trained on data without refinement trajectories experiences a similar magnitude of perfor- mance drop across all tasks.
we can observe that the
In our proposed Refinement Tuning, we mask the loss of erroneous turn tokens to prevent the model from learning incorrect thought processes. To verify whether this process necessary, we train model learning assistant turn to- kens the same data. 2able that the model learned erroncous tokens results in very adverse consequences, with nearly 75% drop in Sciworld. This conclusion is contrary to (Ye et al., 2024). In fact, we find that the model's per- formance on these tasks can continue drop low level with the continued learning of data with erroneous trajecto- ries. We believe that least for agent Refinement Tuning, eliminating the loss of erroneous turns is crucial. Other- wise, models will learn incorrect reasoning processes, lead- ing to poor performance on held-out tasks.
Aueeage Suctest Pngreer
Figure 5: The model's performance as the AgentRefine train data scales up.
Scaling ReentRefine We experiment and analyze the relationship between the data size of the Agen- tRefnne training set and model performance, with the results shown in Figure From the results, we can observe that the model demonstrates significant gains in performance as the data size increases from 4k to 64k, which illustrates the effectiveness of the AgentRefine data.
4.3 ROBUSTNESS ANALYSIS
Previous work has extensively trained on held-in tasks but shows poor performance on held-out tasks. One possible reason is that models simply memorize the key-value pairs between observation

---

Published as a conference paper at ICLR 2025
Medel
Alfwoeld
Peturbutun
Potubutus
Poturbutes 3 Pregrco
Pettrbbtton é Fresrco
Poturbiian 5 Pregze
Acerag
Sial
Seccos
Progecs
Scccos
Pregres
Seccos
Pregres
Seccess
Saocen
Saccem
Saaccn
Suacco
Pource
ELaMA33-8--B0 -i---oaad
AscaKSuue
61.5
163
19.1
59.2
49.2
65.3
11.8
53.0
38,8
48.3
58
25.7
36.3
35.4
1937
16:55
AgrnsIEAN
Table Performance for different models across various perturbations.
and actions from training data, rather than learning to infer correct actions based on the task and observation. To test the hypothesis above, we conduct data perturbation experiments on held-in task. Specifically, we select the Alfworld, which belongs to the held-in category for both AgentGym and Agent-FLAN. We perturb the candidate actions in Alfworld ensuring that the perturbed ones consist of different tokens (or token order) but express the same semantic information. The detail perturbation rules are shown in Appendix K
Table shows the experimental results. It can be observed that simple data perturbation leads to significant performance drop on the original held-in task. For example, under the average score, AgentGym's s Success Rate drops by 25.6%, while Agent-FLAN experiences an even more severe performance decline of 30.4%. Their standard deviation is close to 20%. In comparison, Our Agen- (Refine has a 3.7% increase in the average and low standard deviation, 3.73%, indicating that learns decision-making capabilities rather than just simple memorization.
4.4 DIVERSITY ANALYSIS
Aperiiam Agentkation
Thought Diversity Figure illustrates the distribu- tion of thought diversity across three agent datasets. We extracted the thought content from all ReAct rounds and vectorized them. We randomly sampled 8100 data from all thoughts and visual- ized them via dimensionality reduction using t-SNE (Van der Maaten & Hinton 2008). Compared to Agent-FLAN and AgentGymm the data of AgentRe- fine are more widely distributed and numerous in Figure [6] indicating a higher diversity of thoughts in AgentRefine. This suggests that the AgentRefine data can better teach the model to think diversely, achieving broader exploration space.
Environment Diversity Figure 7 shows the simi- larity relationship between the AgentRefine environ- ment and the test datasets. We randomly selected the instructions from 100 data (50 from AgentRefine
Figure 6: The t-SNE figure among Agent- FLAN, AgentGym, and AgentRefine's Thought.
and 10 from each test set) and removed the one-shot examples from the test sets. As shown in Figure the similarity between the AgentRefine environment and the environments less than 0.5 (bottom left and top right sections), indicating a certain degree of difference between our environment and the environments.
03
igertheree
46
Best-of-N Table presents the performance of the three agents on Best-of-N (BoN). We set the decod- ing temperature
Rabya
1, executed each target task ten times, and took the highest score as the progress rate. there was at least one successful result among the ten executions, the success rate would be 1; other- wise, would be O. The results in Table show that the BoN performance using any training data always better than greedy, with the improvement of AgentRefine being particularly notable, averaging
Jndd
Miverti
Avtyel
Figure 7: The similarity heatmap between different environments in 6 sources.

---

Published as a conference paper at ICLR 2025
Model
Alfworld
BabyAI
SciWorld
PDDL
Jericho
Prugress
Success
Progress
Success
Prugress
Success
Progress
Success
Progress
AgentGym--reedl
61.9
76.9
47.3
61.4
18.9
47.5
1.7
16.6
0.0
12.9
Aye BoN
99.3
99.3
73.2
87.2
58.9
85.6
16.6
42.1
5.0
22.2
37.4
22.4
25.9
25.8
40.0
38.1
14.9
25.5
5.0
67.2
79.7
25.0
33.3
1.1
10.9
8.3
25.3
0.0
10.1
Agent-FLAN--E
98.1
56.7
10.0
11.7
5.0
28.3
18.4
18.8
21.4
8.9
22.6
3.4
14.3
5.0
12.1
AgemtRelaa--reecy
44.8
63.8
37.5
50.4
14.4
42.6
16.6
37.8
10.0
32.3
AgentRefine BoN
93.3
96.6
67.0
81.5
40.0
71.0
30.0
57.3
25
52.5
48.5
32.8
29.5
31.1
25.6
28.4
13.4
19.5
15.0
20.2
Table 4: Best-of-N results among three methods.
over 25%. The marked improvement AgentRefine compared to the other two datasets is likely due to its higher diversity and quality of chain-of-thougle oftuuhh It also demonstrates that existing agent-tuning models have great potential. To gradually improve the model's performance, this result suggests that should construct better reinforcement learning agent data towards generalization in future work.
SYNTHESIS FROM OPEN SOURCE MODEL.
In the main experiment, we use GPT-4o to synthesize the AgentRefine data. In this chapter, we attempt to replace it with open-source models to complete data synthesis process. S our results under 4000 training data. It can be observed that, compared to Agent-FLAN, which used GPT-4 for data synthesis, the AgentRefine data synthesized with the open-source model DeepSeek- v2.5 exhibits significant advantages the held-out tasks. For example, it leads Agent-FLAN by 11.6% in the BI Success Rate metric, further proving the advantages of AgentRefine. Ad- ditionally, observe noticeable gap between the data synthesized with DeepSeek and the data synthesized with GPT-4o. This indicates that using more capable models for data synthesis does indeed yield higher-quality training data and results in greater performance gains.
Alfworld
BabyAI
SciWorld
PDDL
Jericho
Success
Progress
Success
Progress
Success
Progress
Success
Progress
Success
Progress
Ageat-FLAN
67.2
79.7
25.0
35,3
1.1
25.5
0.0
10.1
AgentReline DeepSeek
32.0
44.2
36.6
48.1
22
21.6
16.6
36.7
5.0
29.0
AgeatRelne-GPP-40
36.6
55.9
33.9
44.1
11.1
31.4
18.3
37.9
10.0
28.8
Table 5: Performance on Different Synthesis Models, we synthesize 4000 data via deepseek-v2. -2.5. The underlined text indicates that the training data is sampled in the same environment as the task and is considered as held-in evaluation
GENERLIZATION IN REASONING TASK
Method
EM
F1
Figure g presents the results the reasoning task, HotpotQA (Yang et al. 2018). The result shows that AgentRefine outperforms other methods Hot- potQA's EM and Fl metrics. It proves that AgentRe- fine's generalization still works on reasoning prob- lems.
29.3
36.6
AgentGym
28.0
37.4
Agent-FLAN AgentRefine
24.6
32.4
37.0
44.6
Figure 8: Model Performance reasoning task, Hotpot QA.
7 CASE STUDY
Figure presents examples Agent-FLAN and AgentRefine in Jericho and Sciworld. The cases show that Refinement Tuning enhance the diversity and quality of the model's thinking, which helps improve the model's 's exploration breadth and efficiency and avoid always getting stuck loops in environment.
In Jericho, Agent-FLAN mistakenly believes it is not in the cell and attempts to go to cell. After failing, it chooses check valid actions. Although check valid actions is correct choice,
8

---

Published as a conference paper at ICLR 2025
a) Jericho
b) SeiWorld
Goal:: grienter ship locked your well. get oun your cell out isto the comrtdo without getling caught. You could List distrattoon Ageut-
metal which located around the bokkam themmeneter in the kitchen... If the metal fork is above 0.0 degnees celeins .bbec in the pramog bix
the
(Both 2 agents have dound a key after some tums)
(Both ageas locaied küichen ofter Agrat-Plan
AgontRefine
AgentRefine
Acaion 2: go bedroom
Thought
Thought Looks pet cought. need to uy again, Pollugs a dillis aut apgroacia might wock Action H.: chack velidaation Obvevatioon 23: Thought should techock
Thoughe I nned ledtnal the In keschen It might he the Brssse
Obvervation 2: No known setion...
avsilaabic to escape from ship.
should go back to the sell sastl
Acrion counterto
Thought see 'go bedroeet not valid action should by nasigneing
fist the key again.
Acsiove
cell
sonnnoeerr 3ocusion retured to the uSl
Observation You such thing
Artion Pa coumictuu
Action 5: go hallway
anyi
Thought Staae navigating to the
Thoughh a+1: the cell anymoree lt Ieseems thee losl
key
Elstractions might help escape.
Action 14: Look Obveratioon 14
Action &: inventory.
Sedroom directly iss't possibe.......
Thongle 12: The fork not
Obvervttion 6: ....A (that is opent...
bodcomm
Action I+ check valid actions
Thoughr 26: I'm being cought
inventary cill:
places check. have give up Bniteseun
Thought 2: Gioen the acsilable doves,
(epesi above action
BesS
Should go 50 the bedncons to find the mecal fook
interact with the pressure gange.
Aesion 72: Bank reseen
Action 26: push gunge
[1]
Figure 9: Comparison case study on Jericho and SciWorld between Agent- -LAN and AgentRefine.
Agent-FLAN does not correct its erroneous decision based on the returned results and repeats the go to cell and check valid actions error loop. In contrast, AgentRefine, upon realizing its actions are not achieving the goal, tries various new methods instead of endlessly repeating previously tried incorrect actions.
In Sciworld, Agent-FLAN ignores the hint in the Goal that the fork is in the bedroom and chooses to search in the kitchen. Additionally, Agent-FLAN, having memorized the Alfworld dataset, attempts to output locations can only be found in Alfworld (drawer, countertop, and the action format go to {place)), )), which do not exist in SciWorld. Conversely, AgentRefine can clearly find the thermometer and decide to go bedroom search for the fork. After go bedroom fails, decides to go hallway based on several rounds of observation. In Thought 6, although AgentRefine mistakenly believes it cannot reach the bedroom. its judgement shows it can revise its decisions using short-term memory (from turn 2). When Observation 6 provides clear information about the bedroom, AgentRefine can correct its wrong decision Thought 6 and reach the bedroom. This indicates that AgentRefine's ' improvement in results is not due to memorizing prior knowledge from training data but rather its ability to efficiently utilize and integrate multiple key pieces of information from short-term memory to correct errors in historical decisions.
8 GPT-4 JUDGEMENT'S RELIABILITY
Figure 10 shows the comparison of GPT-4 and hu- man judgement on whether a turn needs to be re- fined. We randomly sampled 50 trajectories from the generated trajectory. each trajectory, we ran- domly sampled right turn and I wrong turn. We asked the human annotator to label the correctness of the turn. The human annotator receives the his- torical thought, action, and observation before the right/wrong turn well as the fight/wrong turn's 's thought, and action in ReAct format. also receives the script corresponding to the trajectories. The sults show that in the turns that GPT-4 labeled right, 94% are aligned with human judgment, and in the turns that GPT-4 labeled wrong, 82% are aligned with human judgment. This indicates that GPT-4's judgement is reasonable.
GPT-4
Right
Wrong
Human
Right
47
9
Wrong
41
Figure 10: judgement and human's judgement. The right column/line means human/GPT-4 con- siders this turn doesn't 't need to be refined. The wrong column/line means human/GPT- considers this turn needs to be refined.
The comparison of GPT-4's
GENERALIZATON BETWEEN GENERAL DATA AND AGENT DATA

---

Published as a conference paper at ICLR 2025
we ShareCPPT
Both Agent-FLAN and AgentTuning have found that incor- porating general data can enhance the model's generalization abillity. This improvement arises from the improvement of instruction-follooing capability. Figure shows the changes in model performance after incorporating ShareGPT. Aligned with them, we also found that general data like ShareGPT can continually improve the model's Held-out task performance.
Success Rate
w. StareGPPT
Agent-FLAN AgentRefine
Figure 11: The success rate by in- corporating ShareGPT
10 RELATED WORK
Agent Finetuning To enhance the decision-making, capabilities of open-source models, series of works currently focus on training Agent trajectories. A small number of models choose the decompose- watppooss--tee-eeeettteo paradigm (Yin al. 2024), while the majority opt for using ReAct (Yao)
iet al. 2022). Most works sample from the dataset and train the model using methods such as SFT or DPO (Rafalov et al.] 2024) to improve their ability to handle Held-in problems (Zeng et al. 2023 Hu et al. 2024) Xi et al.) 2024, Chen et al.) 2024). AgentTuning. Agent- FLAN, and AgentGien attempt train generalizable agent models. AgentTuning and Agent-FLAN have found that using general data like ShareGPT can improve generalization. AgentGym aims to enhance generalization by enabling the model to continuously learn new tasks and treating all tasks as Held-in. AgentGen is the first to attempt direct environment synthesis, improving generalization by enhancing the diver- sity of training data. this work, we demonstrate that the above approaches still have limitations in terms of generalization, specifically in terms of easily overfitting on single data sets, getting stuck in reasoning, and learning incorrect reasoning patterns. To address this issue, we increased the di- versity of training agent data through synthetic data, significantly alleviating the model's overfitting problem. Additionally, we add refinement steps in the trajectory. We show that whether the train- ing data includes the refinement process affects the model's reasoning pattern, and adding synthetic refinement processes greatly enhances the generalization performance of L.Mss
Data Synthesis Due the impending depletion of web data, the of synthetic data has become a research hotspot. The synthesis can be divided into query synthesis and response synthesis. Most agent-tuning approaches synthesize the response in different ways like the plan (Yin et al. 2024).
ReAct format (Zeng et al.] (023). JSON format (Zhang et al.[2202)) chat format (Chen et al. 2024
pair format (Xiong et al. 2024), or evaluation of the state knowledge (Qiao et al. 2024), etc. The other way is to synthesize queries, like evolving a given query (Xu et al. (202) or using pre-train data as seed generate new data (Chan et al. 2024). Among agent research, only AgentGen explores query synthesis. AgentRefine tries to synthesize queries and responses at the same time
and uses a verifier to supervise the quality of the responses.
Self- Relf-Refine Self-refine refers to the process where model iteratively generates better results through feedback. SELF-REFINE (Madaan et al. 2024 Huang et al. 2023) finds GPT-4 can find and correct mistakes itself in compulsory pipeline generate answer, asking refinement advise and use the question and the advise to generate answer again. AgentRefine trains models to develop step-level refinement abilities. This means the model can spontancously adjust its decision pro- cesses based on feedback from the environment, rather than relying on compulsory guidance from a pipeline at instance-level. AgentRefine is also the first approach to identify the connection between step-level refinement and agent generalization.
11 CONCLUSION
In this work, we study the generalized agent abilities for open-source L.MSs via agent tuning- Cur- rent work performs well on held-in evaluation sets but fails to generalize to held-out sets because of overfitting to several manual agent environments. We present the AgentRefine approach to enable the model to correct its mistakes based the environment feedback. Experiments demonstrate that AgentRefine significantly outperforms state-of state-of-thhear art agent-tuning work in terms of generalization ability diverse agent benchmarks. Our analysis shows that self-refinement enables the robust- ness of agent capability and the diversity of agent environments and thoughts further enhances the performance. We hope to provide new insight for future agent research.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
agnificant-gravitas/aaann Autogpt is the vision of accessible for everyone, to use and to build on. our mission is provide the tools, so that you can focus on what matters. https:// (Accessed on 09/29/2024).
github. com wtt../oo//ngnnngnnnnnnoom
Platform to experiment with the ai software engineer. termi- nal based, note: Very different from hutps//gptengineer.gg htttss://itthiiii com/ ggt- neer (Accessed on 09/29/2024).
yoheinakajimmlbabyagi httppp///sssttu....o///aaaaaaaaa /bbbyagi (Accessed on 09/29/2024).
Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data creation with 1,000,000,000 personas. arXiv preprint arXiv:2406.200940 2024.
Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. Agent-flan: Designing data and methods of effective agent tuning for large language models. arXiv preprint arXiv:2403. 12881, 2024.
Maxime Shevalier-Boisvert Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. arXiv preprint 1rX1v:1810.08272, 2018
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
03 30 30-vicuna/
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang. Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
Dayuan Fu, Jianzhao Huang, Siyuan Lu, Guanting Dong, Yejie Wang, Keqing He, and Weiran Xu. Preact: Predicting future in react enhances agent's planning ability. arXiv preprint arXiv:2402.11534, 2024.
Matthew Hausknecht, Prithviraj Ammanabrolu, Mlec-Alexandre Côté, and Xingdi Yuan. Interac- tive fiction games: A colossal adventure. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7903-7910, 2020.
Mengkang Hu, Pu Zhao, Can Xu, Qingfeng Sun, Jianguang Lou, Qingwei Lin, Ping Luo, Saravan Rajmohan, and Dongmei Zhang. Agentgen: Enhancing planning abilities for large language model based agent via environment and task generation. arXiv preprint Mxxix:2408.007666 2024.
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06877 2023.
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseck-v2: A strong, economical, and efficient mixture- of-experts language model. arXiv preprint arXiv:2405.04A4 04444, 2024.
Chang Ma. Junlei Zhang, Zhihao Zhu, Cheng Yang. Yujiu Yang. Yaohui Jin, Zhenzhong Lan, Ling- peng Kong, and Junxian He. Agentboard: An analytical evaluation board of mulli-turr Ilm agents. arXiv preprint arXiv:2401.133 13178, 2024.
11

---

Published as a conference paper at ICLR 2025
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self- feedbakk. Advances in Neural Information Processing Systems. 36, 2024.
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. Augmented language models: survey. Trans. Mach. URL https://api: semant
Learn. Res., 2023, 2023.
256868474]
Shuofei Qiao, Runnan Fang, Ningyu Zhang, Yuqi Zhu, Xiang Chen, Shumin Deng, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Agent planning with world knowledge model, 2024.
URL httph:///axxiiii org/ abs/2405. 14205
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.1678) 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1500--506, 2020.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Mohit Shridhar, Xingdi Yuan, Mare-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint ar arXiv:2010.03766. 2020.
Theodore R. Sumers, Shanyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. Cognitive architectures for language agents. Trans. Mach. Learn. Res., 2024, 2023. URL https:
//api.aemantee www.ssmmttnnccoolaae.eeo.c
Mauro Vallati, Lukas Chrpa, Marek Grzeš, Thomas Leo McCluskey, Mark Roberts, Scott Sanner, et al. The 2014 international planning competition: Progress and trends. Ai Magazine, 36(3):
90-98, 2015.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Ruoyao Wang, Peter Jansen, Alxx-Alex Côté, and Prithviraj Ammanabrolu. Scienceworld: Is your agent smarter than 5th grader? arXiv preprint m:Xiv:2203.07544 2022.
Zhiheng Xi, Yiwen Ding. Wenxiang Chen, Boyang Hong. Honglin Guo, Junzhe Wang, Dingwen Yang, Chenyang Liao, Xin Guo, Wei He, et al. Agentgym: Evolving large language model-based agents across diverse environments. arXiv preprint arXiv:2406.04155 2024.
Weimin Xiong, Yifan Song. Xiutian Zhao, Wenhao Wu, Xun Wang. Ke Wang, Cheng Li, Wei Peng. and Sujian Li. Watch every step! I1m agent learning via iterative step-level process refinement. arXiv preprint arXiv:2406.1 1176, 2024.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.1 12244, 2023.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv: (888009600) 2018.
12

---

Published as a conference paper at ICLR 2025
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv.2210003629, 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physies of language models: Part 2.2, how to learn from mistakes on grade-school math problems, 2024. URL https://arxix: are/abs// /2008240.... 16293
Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin. Agent lumos: Unified and modular training for open-source language agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 23380-12403, 2024.
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttun- ing: Enabling generalized agent abilities for 1lms. arXiv preprint arXiv:2310.1288 128222 2023.
Jianguo Zhang. Tian Lan, Rithesh Murthy, Zhiwei Liu, Weiran Yao, Juntao Tan, Thai Hoang, Liang- wei Yang. Yihao Feng, Zuxin Liu, al. Agentohana: Design unified data and training pipeline for effective agent learning. arXiv preprint 1xXx:22422115506. 2024.
Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin. Yong-Jin Liu, and Gao Huang, Expel: Llm agents are experiential Iearners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19642, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances Neural Information Processing Systems, 36:46595-46623, 2023.
Yaowei Zheng, Richong Zhang, Junhao Zhang. Yanhan Ye, and Zheyan Luo. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint Arxiv:2403. 13372, 2024.
Andy Zhou, Kai Yan, Michal guage agent tree search unifies reasoning acting and planning in language models. arXiv preprint arXiv:2310.0440 2023.
Haohan Wang, and Yu-Xiong Wang. Lan-
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
ACKNOWLEDGMENT
This work was partially supported by the State Key Laboratory of Massive Personalized Customiza- tion System and Technology (No. H&C-MPCC2003-00-0770) "Research on active semantic discovery technology based on SG-CIM and its application in power grid equipment supply chain optimization". China Unicom Software Research Institute "Framework Agreement for Seven Model Technology Research and Application Demonstration Projects (Software Development for Government Enterprise Content Generation) of China Unicom Software Research Institute from 2024 to 2025" (No.550033 1818), and the National Natural Science Foundation China (NSFC No.62076031 and No.62076036).
State Grid Technology Project
000-20241622AA-11111
ETHICS STATEMENT
When using a large amount of open-source resources for data synthesis, an important issue is the generation of harmful and malicious data. In work, use Bersona-Hub. a synthesized dataset that has undergone security processing. We use it to synthesize tasks and environmental information, which pass secondary review and are safe to use. However, our method may have potential risks
13

---

Published as a conference paper at ICLR 2025
of misuse, such enhancing LLM's capabilities in malicious agent tasks, like generating attack codes. Therefore, adhering to ethical guidelines is crucial to ensuring the responsible use of this technology.
A
TASKS STATISTIC
Table
6 presents the number test data and domains in the 5 tasks. These number calcu- lates the Held-out Task score. Specifically, Held out (BabyAIscon 112 +
SciW orldscore 90 + PDDLscore 60 + Jerichoscore * 20)/282
task
Alfworld
BabyAI
SciWorld
PDDL
Jericho
#num
134
I12
20
Domain
Science Experiment
Houschold Tasks
Robot Explocation
Strategy Games
Long Text Games
Table 6: tasks statistic in AgentBoard. #num refers to the number of data for testing.
B THE HISTORY OF AGENT-TUNING
In recent years, LLM-Based Agents have become a popular paradigm. However, improving LLM performance on agent tasks during the post-training phase remains a challenging issue. Previous work typically sampled and trained in fixed environments (with Held-in data that is distributionally similar to the test (ata)(Xi et al., (024), which significantly improved performance on specific tasks (test sets that are distributionally similar the training data). However, performance drops sharply once the task changes.
AgentTuning (Zeng et al. 2023) was the first to recognize this issue by adding portion of general alignment data to the single ainle--aennt data, alleviating the problem and demonstrating initial generaliza- tion capabilities. Agent-FLA FLAN (Chen al. 2024) further improved the single-agent data, enhancing the model's generalization in agent tasks.
In work, demonstrate that the above approaches still have significant limitations in terms of generalization, specifically in terms of easily overfitting on single data sets, getting stuck in reason- ing, and learning incorrect reasoning patterns (as discussed in Figure [ Figure 9 and Section 4.3 etc.). To address this issue, we increased the diversity of training agent data through synthetic data, significantly alleviating the model's overfitting problem. Additionally, we add refinement steps in the trajectory. We show that whether the training data includes the refinement process affects the model's 'ss reasoning pattern, and adding synthetic refinement processes greatly enhances the general- ization performance of L.Mss.
C SYNTHESIS DATA WITH PERSONA
Persona represents diverse and rich information content. Persona hub (Chan et al. 2024) contains 1,000,000,000 personas after filtering via diverse. If the filter cosine similarity is 0.5, it can still generate I million diverse personas. The persona hub also demonstrated that the data generated the persona hub has similar diversity to the persona data and its scaling experience shows that data generated via the persona hub is not yet saturated at the size of IM under math problem.
D TRAINING HYPER PARAMETER
For all models, the learning rate is 5e-6 with a cosine learning rate scheduler and no warm-up steps. The batch size is 64. The max length is 8192 for 7/8b models and 4096 for 70b models due to limited storage for DeepSpeed (Rasley et al. 2020) usage. Aligned with Agent-FLAN, we choose AgentRefine with 32000 data for the default training setting. Aligned with AgentGen (Hu et al., 2024), train model for 10 epochs and select the checkpoint with the best average results to report. We also modified the LLaMA- LLaMA-Factory's SET loss to Equation [ with LLLMA-Factory' s default settings.
Other settings are aligned
14

---

Published as a conference paper at ICLR 2025
E COMPARISON AMONG AGENT DATASETS
Table compares the number of trajectories, the methods to obtain environments and trajectories, the held-in tasks in the AgentBoard benchmark, and the availability of refinement steps among Agent- FLAN, AgentGym, AgentGen, and AgentRefine. AgentRefine can easily scale its data and includes refinement steps in the training set. AgentGen and our work contemporary. Our commonality lies in synthesizing diverse environments, but we place more emphasis on enhancing refinement
abilities.
Method
Trajectory
Environment
Erajectory
Heldinn
Reinemenn
Agent.
manuill
sampled
AgentGym
14485
manual
sampled
Alfwodd, BabyAl. SeiWodd
No No
AgentGien
7246
synthetic
sampled syntactic
NIA
AgenReline
(mas) 61000
synthetic
NIA
Yes
Table 7: Comparison of AgentRefine with other method covers several aspects: the number of trajectories, the way to get environment, the way to get trajectory, the held-in task in AgentBoard, availability of refinement step
F IND FILTERING EXPERIMENTS
To remove the interference from IND data, we perform experiment where train model using data that excludes all IND training data. Agent-FLAN removes 672 samples out of 34440 samples, and AgentGym removes 5350 samples out of 14485 samples. The result in Table g shows that AgentRefine outperforms the other two methods in all tasks. This demonstrates that our method significantly improves over previous methods.
RabyAT
Jericho
Success
Progress
Success
Progress
Success
Progress
Success
Progress
Success
Progress
LLaMA 3B Instruct
22.4
46.1
45.5
56.5
7.8.
41.1
10.0
38.4
0.0
24.3
Agentien
29.1
47.6
20.5
35.0
11.7
23.0
AgentGym ind data
5.9
28.7
27.7
40.0
2.2
14.3
13.7
Agent-FLAN FLAN w/o ind data
1.5
19.7
32.1
45.0
2.2
12.1
6.6
23.6
0.0
14.5
AgenRefine
Table 8: IND Filtering Experiments
G REFLEXION EXPERIMENT
Table 9 presents the results with Reflexion (Shin et al. 2024). It shows that AgentRefine outper- forms other methods when adding Reflexion, especially Alfworld, since AgentRefine isn't 't trained on any Alfworld data, yet it outperforms AgentGym, and Agent-FLAN, whose models are trained on Alfworld data. This indicates that AgentRefine can utilize Reflexion more effectively than other methods.
BabyAl
Success
Progress
Sucoess
Progress
Success
Progress
Success
Progress
Success
Progress
LLLMA-3-8B-- M-MMA----B-Insttuet.. Reflexion
41.2
56.2
45.5
56.5
39.4
10.0
38.4
5.0
20.9
Agentiym Reflexion
91.8
60.9
23.3
30.6
16.6
12.1
Agcat-FLAN FLAN Reflexion
X
89.4
32.1
12.3
3.5
13.1
10.0
24.8
0.0
9.7
AgemRReine Reflexion
90.3
95.6
37.5
50.4
16.6
44.5
16.6
37.8
10.0
32.7
Table Reflexion Experiment. The underlined text indicates that the training data is sampled in the same environment as the task and is considered as held-in evaluation
H STANDARD DEVIATIONS
Table 10 shows the average and standard deviation for each task. We use the results from Table (decoding temperature 1.0 with 10 sample times). AgentRefine's average performance exceeds
15

---

Published as a conference paper at ICLR 2025
that of other methods by least 2 standard deviations in most OOD tasks. This demonstrates that our method represents a significant improvement over previous methods.
Model
Alfworld
BabyAI
SeiWorld
PDDL
Jericho
Success
Progress
Success
Progress
Success
Progress
Success
Progress
Success
Progress
AgentGymm
61.3as
78.0
18.22a.3
61.2423
25.5.47
55.4sa2
4.5.1.
16.9
0.0
15.3
Ageut-FLAN
51.7439
71.6428
314:30
41.443.
1.2
11.1g12
3.8
16.4227
0.0g0.
10.5
AgentRefine
60.1
72.9424
37.6s1a
52.2a19
10.-aa2
35.0a3.
13.2=2.
87.4a22
11.0240
30.9a32
Table 10: Model's average performance and standard deviations on different data. We used a high temperature and randomly sampled 10 times. The underlined text indicates that the training data sampled in the same environment as the task and is considered as the held-in evaluation.
I ROBUSTNESS ANALYSIS WITH DIFFERENT COMPONENTS
Misil
Alfecehl
Pertucfediss
Partudration 2
Petteistius 3 Pmngress
Parantation 4 Progress
Ancrage
STD
Success
Progress
Success
Progress
Scccess
Phogress
Success
Stccess
Success
Prigress
Seccess
Progress
ApentRetuue
65.1
inining
26.1
134
65.6
3.30
verifccrsonn
36.1
39.5
49.2
3400
15.6
36.30
7.08
Table 11: Ablation study across various perturbations. We experimented with small data size (i.e.8000) and in "w half training data" setting, we use 4000 data. The w/o verification setting contains data in 3 styles: 1. The data that does not contain a refinement step. 2. The data with wrong parameter/action name but is not identified by the GPT-4. 3. The data is correct and has the refinement step (i.e. a subset of the AgentRefine data). We remove incomplete data or the data that not be parsed into the training data
Table 11 presents the contribution robustness among different components. When training on 4000 data, the standard deviation of the success score is almost double that of the baseline which means the number of the training data is the most important factor for the model's robustness.
J MODEL'S INSTRUCTION-FOLLOWWNG ABILITY
We use MTT-bench (Zheng et al., 2023) to test mod- els' cnstruction-followiii ability and use gpt-4o- 2024-05-13 judge the score.
Method
MT-bench
ALAN 3.73
+ShareGPT
5.71
AgentRetine
3.96
The score AgentRefine is approximately 0.2 points higher than that of Agent-FLAN regardless of whether ShareGPT is incorporated. After incor- porating ShareGPT, both show an improvement of about 2 points.
+ShareGPT 5.91
Figure 12: Model Performance on Different
Tasks
K PERTURBATION DETAILS
We have made 5 perturbation in Alfworld:
Perturbation 1: change clean {obj} with {recep), cool {obj} with {recep), heat {obj)with {recep} to clean {obj} using {recep), cool {obj} using {recep). heat {obj} using {recep) in the instruction
Perturbation 2: change go to {recep) to move to {recep} in the instruction
Perturbation 3: change take {abj} from {recep) to from {recep} take {obj} in the instruction
Perturbation 4: delete all space between item name and item number the instruction.
Perturbation 5: remove all IND data in the training set and retrain the model.
We also revise the environment adjust these changes.
16

---

Published as a conference paper at ICLR 2025
L
SCRIPT GENERATION
Script Generation Format
"Thought" (string, compulsory) "The design of the environment, goal and available actions of the player to achieve.",
"Environment"
"initial state", (string, compul sory) "The initial state of the environment.",
"places and objects"
"<The name of the place "information" (string, optional) "The information of the place or object, which will only be shown to player when the object is examined/opened/ looked or the player have just step in its receptacle etc.",
object>" {
"<The information of the place or object
>" (string, optional) "The information of the place or object, which will only be provide to DM", "<The name of the place
object>": "information" : (string, optional) The information of the place object, which will only be shown player when the object is beaminnd/opened///eee
or the player have just step in its receptacle etc. It must be concrete (for example, if you add information in document, you need to give the important part of the document context instead of brief introduction.)."
"location" (string, optional) "The relative location between the object/place and its json upper level object/place (i.e. receptacle)..
"relative location" (list of string optional) ["The relative location of the places or objects in the same json level."
b
"relative location" (list of string, optional) ["The relative location of the places objects in the same ison level ."]
"pl "player":
"information":
(string, compulsory) "The player's restrictions."
17

---

Published as a conference paper at ICLR 2025
"Goal" (string, compulsory) "The goal of the player to achieve. It need to be clear (has unique and concrete completion conditions), achievable and be finished by one person.", "Completion Conditions" (list of string, compulsory
"The specific conditions that the player must meet complete the task."
1, "Available Actions"
"<The name of the action>"
"description" (string, optional) "The description of the action.",
"special format" (string, optional) "The special format of the action. Only when the parameter is not in the place/object this and their information above key. (This key is compulsory when answering the question and editing the code.)",
"verification code" (string, compulsory) The reqular expression of the action.". ", "parameters"
"<The name of the parameter>" (list of string, optional) ["The value of the parameter if action has placeholder. Remember all possible parameter (the possible place, possible object the possible item/text in the \" information of place/object or the imformation in the completion conditions) should be in the list. will strictly check the player's actions according the given parameters. So you should give all possible parameters with correct name
M TRAJECTORY GENERATION
Trajectory Generation Format
"turn": (int, compulsory) "The turn number, the first turn number should be 0, DM s turn number should be even." " "role": (compulsory) "DM",
18

---

Published as a conference paper at ICLR 2025
"Thought": (string, compulsozy) "The thought of the DM, contains the analyze of the knowledge the player have known and the chain-of-thought to decide the observation.", "Observation': (string, compulsory) "The observation of the DM, contains the information the player should know.", "parameter_erroe" error": (bool, compulsory) "The error log of the DM, if the player's last action did not match the format of the available actions
splace_error", (bool, compulsory) "The error log of the DM, if the player's last action act at a wrong place",
"logic "logic_error"; error": (bool, compulsory) "The error log of the DM, if the player's s last action matches the available action but the observation on is not changed under the action or went back to the sitiuation that history has been. (for example, go north then go south) "progress_raten (float, compulsory) "The progress rate of the task, the max value should be 1.0 which means task finsihed. " "finished"; (bool, compulsory) "The flag of the task, if the task is finished, the value should be true."
"turn": (int, compulsory) "The turn number, the first turn number should be 1, Player's number should be odd.",
" "role": (compulsory) "Player",
"Thought": (string, compulsory) "The thought of the Player, contains the chain-of-thought to decide the action. You should remove the \" Thought:\" at the beginning the json output, although DM should ask for this format in the first turn.",
this string
"Action": (string, compulscry) "The action of the Player, its format and the parameter MUST follow the script. You should remove the \" Action:\" at the beginning of this string in the json output, although DM should ask for this format in the first turn."
N ERROR TURN STATISTICS
Figure presents the error turn statistics in AgentRefine (32000). Most of the error-refine pairs consist of one turn, which accounts for about 16% among all turns. How- ever, AgentRefine also includes error-refine pairs whose lengths exceed three turns.
Figure 13: The statistics of Continuous Error Turns in AgentRefine
19

---

Published as a conference paper at ICLR 2025
O TRAJECTORY VERIFICATION
Algorithm presents the Trajectory Verification pipeline.
Algorithm Trajectory Verification
1: Input: Available Actions, Trajectory, Verified Trajectory
2: The Verified Trajectory will be set to an empty list this the first verification of the persona or the last generation's fault is error <
3: Initialize: eπror nrror_num=0
4: if JSON format verification does not pass then
5: JSON format verification does not pass
6: end if
7: for turn in Trajectory do
8: if' JSON keys in do not match the requirement then
9:
return Verified Trajectory and the signal end if
11: if Player's 's turn then
We only check the action when DM considers it correct.
13:
if not next DM turn shows error signal then
Player's action doesn't match any action, (and its parameter) Available Actions
then
15:
return Verified Trajectory and the signal
16:
end if end
17:
18:
end if
19:
if DM's turn then
if Error signal then
21:
error_num +=
end if'
23:
if This is the last turn then
# The last turn should have any error if Error signal then
25:
26:
return Verified Trajectory and the signal end if
27:
28:
# The last turn should finish the task if No 'Task Succeed' in Observation then return Verified Trajectory and the signal end if
29:
30:
31:
32:
# We need at least 2 error-refine turns. if error _om < then
33:
34:
return Verified Trajectory and the signal
end if end if
36:
37:
end
38:
Verified Trajectory <- Verified Trajectory + turn
39:
end for
20


Reviews:
[{"review_id": "joabRj7fVk", "reviewer": "Reviewer_i2Mf", "summary": "The paper proposes a novel framework to improve the generalization capabilities of LLMs based agents. The authors identify that existing agent-tuning methods often overfit to specific environments and fail to generalize to new tasks. To address this, the paper introduces AgentRefine, which leverages a agent synthesis framework to encompass a diverse array of environments and tasks drawing upon extensive human persona data, enabling the model to learn from its mistakes through a process of refinement tuning. The experiments demonstrate that AgentRefine method outperforms state-of-the-art methods in terms of generalization, robustness to perturbations, and the ability to generate diverse thoughts during inference.", "questions": "Refer to weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed method's idea seems like meta learning, which trains the policy on diverse tasks for quickly adapting to novel tasks. This idea makes sense to me and seems new in agent domain. \n\nI appreciate authors' rethinking on the generalization of agent-tuning. The issue of memorizing trajectory leading to overfitting seems valid to me.\n\nThe experiment evaluates the performance of AgentRefine from wide range of perspectives.\nThe findings establish a correlation between agent generalization and multi-task agent training mechanism / self-refinement, providing a new paradigm for future research in agent-tuning.", "weaknesses": "Overall AgentRefine is a simple and effective method. However, the main idea is not new, as discussed in related work, Agent-FLAN and AgentGen have proposed to train generalist agents using general data. The idea of refinement is also widely studied as discussed in introduction. I encourage authors to clearly differentiate AgentRefine from these prior works. Highlight unique aspects or improvements over existing methods. Consider incorporating a comparative analysis to demonstrate the advantages of AgentRefine.\n\nI feel the procedure suffers from a high risk of generating low-diversity tasks, as the script generation is based on human persona data, which is limited in a certain domain. In contrast, a generalist agent is expected to complete any tasks. \n\nThe goal of the proposed method is to build a LLM-based agent to generalize to novel tasks. However, this way to generate agent tasks does not bring new knowledge to LLMs, but enabling the LLMs to follow the output format more strictly, as it trains LLMs on the data generated by LLMs themselves. \n\nBesides, the source of performance improvement is not clear. For instance, why the LLM-generated trajectories can improve performance on novel tasks? Authors can provide some examples of the evaluation tasks, and examples of the generated tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "480kKKA4Hf", "reviewer": "Reviewer_HUkR", "summary": "The paper presents a framework aimed at improving the generalization capabilities of Large Language Model (LLM) based agents through instruction tuning. The authors observe that existing agent training methods overfit to specific environments and struggle with new situations, leading to poor generalization. To address this, they propose AgentRefine, which incorporates self-refinement processes to enable the model to learn from its mistakes and adapt to diverse environments and tasks.", "questions": "see weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-organized and easy to follow, with a clear progression from motivation to methodology.\n2. The identification of the generalization gap in existing LLM-based agents and the proposal of a self-refinement approach to address it is a rational step forward in the field.", "weaknesses": "1. The problem of generalization in LLM-based agents has been extensively discussed in previous literature, making the contribution of this work less novel. For example,  [1]  investigates the robustness of accuracy measurements in large language models (LLMs) when the order of answer labels is shuffled, using the MMLU dataset as a testbed.\n2. The methodology, while intuitive, lacks significant innovation, as the approach of enhancing generalization through data synthesis is not new [2].\n3. The experimental results do not demonstrate a strong improvement over existing methods, which questions the practical impact of the proposed approach. An apple-to-apples comparison of your main results to show the advantage of the algorithm would make your results more straightforward and strong, instead of using a lot of underlined text to filter out the results where training data is sampled in the same environment as the task.\n\n[1] Changing Answer Order Can Decrease MMLU Accuracy.\n\n[2] Knowledgeable Agents by Offline Reinforcement Learning from Large Language Model Rollouts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IWb3GrhLcF", "reviewer": "Reviewer_NknQ", "summary": "This paper discusses using synthetic data to improve the generalization ability of agents on held-out sets. Previous agent-tuning work often chose to construct agent-tuning data on held-in sets. The authors demonstrate that although these methods can greatly improve the performance of agents on held-in sets, they usually lead to overfitting, which in turn affects the performance of agents on held-out sets. Based on this observation, the authors propose AgentRefine. This method does not use task-related information at all. Instead, it uses LLM to complete the entire data generation process, including task generation, trajectory generation, and verification to construct the agent-tuning dataset, thus avoiding the possibility of overfitting to held-in sets from the very start. In the constructed dataset, the authors emphasize the ability of the agent to correct errors based on the feedback, which further improves the agent's generalization ability. They validate AgentRefine in multiple scenarios, and the experimental results show that finetuned agents outperform other baselines on held-out sets.", "questions": "1. Can you also provide more detailed statistics of your experiments, for example, the std of each task?\n\n2. How does the agent get an error signal during the evaluation?\n\n3. For the LLaMA-3-70B Series, the performance of AgentRefine is worse than the base model? Am I misunderstanding something?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper discusses the generalization ability of agents, which is a very important topic for the community.\n\n2. The authors provide quantitative analysis to explain their insight, which is very convincing.\n\n3. Synthesizing data with almost no task-specific information is a very practical setting, and the improvement of generalization ability in this paper is impressive.", "weaknesses": "1. The presentation of this paper should be improved and some grammar mistakes should be fixed.\n\n2. Some important baselines, for example, Reflexion[1], are missing and should be included.\n\n3. They only consider decision-making tasks in their experiments. However, as they claimed on the generalization ability, tasks of different types should also be included, for example, reasoning tasks. \n\n[1] Shinn, Noah, et al. \"Reflexion: Language agents with verbal reinforcement learning.\" NeurIPS, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2t8ciSJyZX", "reviewer": "Reviewer_opnM", "summary": "The paper proposes AgentRefine, a framework designed to enhance the generalization capabilities of large language model (LLM)-based agents through a self-refinement process. The core idea is to enable agents to learn from their mistakes by refining their actions based on feedback from the environment. The authors introduce a data generation pipeline that simulates diverse environments and tasks, followed by a refinement tuning process to improve agent robustness and generalization. Experimental results show that AgentRefine outperforms state-of-the-art methods in held-out tasks, demonstrating improved generalization and robustness.", "questions": "See above weakness section", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The introduction of a self-refinement process for agent tuning is a novel contribution to the field. By allowing agents to correct their mistakes based on environmental feedback, the authors propose an interesting alternative to traditional fine-tuning methods.\n2. The use of diverse environments and tasks in data generation helps mitigate overfitting to specific scenarios, which is a common issue in LLM-based agents.\n3. The experiments show that AgentRefine outperforms baselines in held-out tasks, suggesting that the approach has potential for improving generalization.", "weaknesses": "1.  The paper relies heavily on GPT-4 for generating both scripts and trajectories. This raises several concerns:\n   - The quality of the generated data depends entirely on GPT-4's ability to detect and correct errors\n   - The method is not truly \"self-refinement\" since it requires external stronger models for error detection and correction\n   - The authors should analyze what happens when using weaker LLMs for data generation and verification\n\n2. The verification process has potential flaws:\n  - It uses LLMs to verify the correctness of scripts and trajectories without human validation\n  - The paper lacks analysis of verification failure cases or error rates\n  - The authors should include human evaluation of the verification process accuracy\n\n3. While the paper shows improved performance, it lacks analysis of whether this is simply distillation from GPT-4 rather than true generalization and how much of the improvement comes from the refinement process versus having access to GPT-4's knowledge\n\n4. The experiments only scale up to 64k examples. Would the computational cost of generating refinement data with GPT-4 makes large-scale training difficult? Also, the authors should analyze the cost-benefit tradeoff of generating more refinement data\n\n5. While the paper shows some robustness analysis, the perturbation experiments are limited to only action descriptions. More diverse types of perturbations should be tested. The analysis should include how different components (script generation, verification, refinement) contribute to robustness", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)