AgentRefine: Enhancing Agent Generalization through Refinement Tuning

Authors: Dayuan Fu, Keqing He, Yejie Wang, Wentao Hong, Zhuoma GongQue, Weihao Zeng, Wei Wang, Jingang Wang, Xunliang Cai, Weiran Xu
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: agent, self-refine, diversity, generalization, data synthesis

Abstract:
Large Language Model (LLM) based agents have proved their ability to perform complex tasks like humans. However, there is still a large gap between open-sourced LLMs and commercial models like the GPT series. In this paper, we focus on improving the agent generalization capabilities of LLMs via instruction tuning. We first observe that the existing agent training corpus exhibits satisfactory results on held-in evaluation sets but fails to generalize to held-out sets. These agent-tuning works face severe formatting errors and are frequently stuck in the same mistake for a long while. We analyze that the poor generalization ability comes from overfitting to several manual agent environments and a lack of adaptation to new situations. They struggle with the wrong action steps and can not learn from the experience but just memorize existing observation-action relations. Inspired by the insight, we propose a novel AgentRefine framework for agent-tuning. The core idea is to enable the model to learn to correct its mistakes via observation in the trajectory. Specifically, we propose an agent synthesis framework to encompass a diverse array of environments and tasks and prompt a strong LLM to refine its error action according to the environment feedback. AgentRefine significantly outperforms state-of-the-art agent-tuning work in terms of generalization ability on diverse agent tasks. It also has better robustness facing perturbation and can generate diversified thought in inference. Our findings establish the correlation between agent generalization and self-refinement and provide a new paradigm for future research.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730677045468

thread_type:
official_review

<thread_object>
creator: reviewer_opnm

<content>
summary:
The paper proposes AgentRefine, a framework designed to enhance the generalization capabilities of large language model (LLM)-based agents through a self-refinement process. The core idea is to enable agents to learn from their mistakes by refining their actions based on feedback from the environment. The authors introduce a data generation pipeline that simulates diverse environments and tasks, followed by a refinement tuning process to improve agent robustness and generalization. Experimental results show that AgentRefine outperforms state-of-the-art methods in held-out tasks, demonstrating improved generalization and robustness.

strengths:
1. The introduction of a self-refinement process for agent tuning is a novel contribution to the field. By allowing agents to correct their mistakes based on environmental feedback, the authors propose an interesting alternative to traditional fine-tuning methods.
2. The use of diverse environments and tasks in data generation helps mitigate overfitting to specific scenarios, which is a common issue in LLM-based agents.
3. The experiments show that AgentRefine outperforms baselines in held-out tasks, suggesting that the approach has potential for improving generalization.

weaknesses:
1.  The paper relies heavily on GPT-4 for generating both scripts and trajectories. This raises several concerns:
   - The quality of the generated data depends entirely on GPT-4's ability to detect and correct errors
   - The method is not truly "self-refinement" since it requires external stronger models for error detection and correction
   - The authors should analyze what happens when using weaker LLMs for data generation and verification

2. The verification process has potential flaws:
  - It uses LLMs to verify the correctness of scripts and trajectories without human validation
  - The paper lacks analysis of verification failure cases or error rates
  - The authors should include human evaluation of the verification process accuracy

3. While the paper shows improved performance, it lacks analysis of whether this is simply distillation from GPT-4 rather than true generalization and how much of the improvement comes from the refinement process versus having access to GPT-4's knowledge

4. The experiments only scale up to 64k examples. Would the computational cost of generating refinement data with GPT-4 makes large-scale training difficult? Also, the authors should analyze the cost-benefit tradeoff of generating more refinement data

5. While the paper shows some robustness analysis, the perturbation experiments are limited to only action descriptions. More diverse types of perturbations should be tested. The analysis should include how different components (script generation, verification, refinement) contribute to robustness

questions:
See above weakness section
</content>
</thread_object>

</thread>
<thread>
creation date:
1730909585084

thread_type:
official_review

<thread_object>
creator: reviewer_nknq

<content>
summary:
This paper discusses using synthetic data to improve the generalization ability of agents on held-out sets. Previous agent-tuning work often chose to construct agent-tuning data on held-in sets. The authors demonstrate that although these methods can greatly improve the performance of agents on held-in sets, they usually lead to overfitting, which in turn affects the performance of agents on held-out sets. Based on this observation, the authors propose AgentRefine. This method does not use task-related information at all. Instead, it uses LLM to complete the entire data generation process, including task generation, trajectory generation, and verification to construct the agent-tuning dataset, thus avoiding the possibility of overfitting to held-in sets from the very start. In the constructed dataset, the authors emphasize the ability of the agent to correct errors based on the feedback, which further improves the agent's generalization ability. They validate AgentRefine in multiple scenarios, and the experimental results show that finetuned agents outperform other baselines on held-out sets.

strengths:
1. This paper discusses the generalization ability of agents, which is a very important topic for the community.

2. The authors provide quantitative analysis to explain their insight, which is very convincing.

3. Synthesizing data with almost no task-specific information is a very practical setting, and the improvement of generalization ability in this paper is impressive.

weaknesses:
1. The presentation of this paper should be improved and some grammar mistakes should be fixed.

2. Some important baselines, for example, Reflexion[1], are missing and should be included.

3. They only consider decision-making tasks in their experiments. However, as they claimed on the generalization ability, tasks of different types should also be included, for example, reasoning tasks. 

[1] Shinn, Noah, et al. "Reflexion: Language agents with verbal reinforcement learning." NeurIPS, 2023.

questions:
1. Can you also provide more detailed statistics of your experiments, for example, the std of each task?

2. How does the agent get an error signal during the evaluation?

3. For the LLaMA-3-70B Series, the performance of AgentRefine is worse than the base model? Am I misunderstanding something?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730966271322

thread_type:
official_review

<thread_object>
creator: reviewer_hukr

<content>
summary:
The paper presents a framework aimed at improving the generalization capabilities of Large Language Model (LLM) based agents through instruction tuning. The authors observe that existing agent training methods overfit to specific environments and struggle with new situations, leading to poor generalization. To address this, they propose AgentRefine, which incorporates self-refinement processes to enable the model to learn from its mistakes and adapt to diverse environments and tasks.

strengths:
1. The paper is well-organized and easy to follow, with a clear progression from motivation to methodology.
2. The identification of the generalization gap in existing LLM-based agents and the proposal of a self-refinement approach to address it is a rational step forward in the field.

weaknesses:
1. The problem of generalization in LLM-based agents has been extensively discussed in previous literature, making the contribution of this work less novel. For example,  [1]  investigates the robustness of accuracy measurements in large language models (LLMs) when the order of answer labels is shuffled, using the MMLU dataset as a testbed.
2. The methodology, while intuitive, lacks significant innovation, as the approach of enhancing generalization through data synthesis is not new [2].
3. The experimental results do not demonstrate a strong improvement over existing methods, which questions the practical impact of the proposed approach. An apple-to-apples comparison of your main results to show the advantage of the algorithm would make your results more straightforward and strong, instead of using a lot of underlined text to filter out the results where training data is sampled in the same environment as the task.

[1] Changing Answer Order Can Decrease MMLU Accuracy.

[2] Knowledgeable Agents by Offline Reinforcement Learning from Large Language Model Rollouts.

questions:
see weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1731301157920

thread_type:
official_review

<thread_object>
creator: reviewer_i2mf

<content>
summary:
The paper proposes a novel framework to improve the generalization capabilities of LLMs based agents. The authors identify that existing agent-tuning methods often overfit to specific environments and fail to generalize to new tasks. To address this, the paper introduces AgentRefine, which leverages a agent synthesis framework to encompass a diverse array of environments and tasks drawing upon extensive human persona data, enabling the model to learn from its mistakes through a process of refinement tuning. The experiments demonstrate that AgentRefine method outperforms state-of-the-art methods in terms of generalization, robustness to perturbations, and the ability to generate diverse thoughts during inference.

strengths:
The proposed method's idea seems like meta learning, which trains the policy on diverse tasks for quickly adapting to novel tasks. This idea makes sense to me and seems new in agent domain. 

I appreciate authors' rethinking on the generalization of agent-tuning. The issue of memorizing trajectory leading to overfitting seems valid to me.

The experiment evaluates the performance of AgentRefine from wide range of perspectives.
The findings establish a correlation between agent generalization and multi-task agent training mechanism / self-refinement, providing a new paradigm for future research in agent-tuning.

weaknesses:
Overall AgentRefine is a simple and effective method. However, the main idea is not new, as discussed in related work, Agent-FLAN and AgentGen have proposed to train generalist agents using general data. The idea of refinement is also widely studied as discussed in introduction. I encourage authors to clearly differentiate AgentRefine from these prior works. Highlight unique aspects or improvements over existing methods. Consider incorporating a comparative analysis to demonstrate the advantages of AgentRefine.

I feel the procedure suffers from a high risk of generating low-diversity tasks, as the script generation is based on human persona data, which is limited in a certain domain. In contrast, a generalist agent is expected to complete any tasks. 

The goal of the proposed method is to build a LLM-based agent to generalize to novel tasks. However, this way to generate agent tasks does not bring new knowledge to LLMs, but enabling the LLMs to follow the output format more strictly, as it trains LLMs on the data generated by LLMs themselves. 

Besides, the source of performance improvement is not clear. For instance, why the LLM-generated trajectories can improve performance on novel tasks? Authors can provide some examples of the evaluation tasks, and examples of the generated tasks.

questions:
Refer to weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732178909042

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to Reviewers and Revision Submitted

comment:
We thank all the reviewers for their insightful comments and suggestions. We have revised the paper to address the reviewers’ concerns. Below we summarize the major revisions (the main revisions are marked with blue text in the pdf, we also made some minor layout changes to fit the page limit), while we reply to the comments of each reviewer separately.

The major revisions are:

1. Add an explanation to clarify the differences between our work and previous agent-tuning studies in Appendix B and the related work. (Reviewer i2Mf, HUkR)
2. Revise the introduction of self-refine in the related work. (Reviewer HUkR)
3. Provide an introduction to the persona hub in Appendix C. (Reviewer i2Mf,opnM)
4. Add IND filtering experiment to eliminate the influence of IND dataset in Appendix F. (Reviewer HUkR)
5. Add a new method - Reflexion in Appendix G. (Reviewer NknQ)
6. Add a new reasoning benchmark - HotpotQA in Section 6. (Reviewer NknQ)
7. Provide the standard deviation of the results in Appendix H. (Reviewer NknQ)
8. Provide an explanation to clarify the inference pipeline in the main text and updated Figure 4. (Reviewer NknQ, opnM)
9. Add a GPT-4 judgement verification experiment in Section 8. (Reviewer opnM)
10. Add 2 more perturbations to the robustness experiment, analyze the contribution among different components and update perturbation details in the Section4.3, Appendix I and Appendix K. (Reviewer opnM)
11. Correct the typos and grammar mistakes. (Reviewer NknQ)
12. Move Section "Synthesis from Open Source Model" from Appendix F to Section 5.  (Reviewer opnM)

We appreciate the reviewers for their valuable comments and suggestions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732536867278

thread_type:
official_comment

<thread_object>
creator: area_chair_jswu

<content>
comment:
Dear Reviewers,


This is a friendly reminder that the discussion will end on Nov. 26th (anywhere on Earth). If you have not already, please take a close look at all reviews and author responses, and comment on whether your original rating stands. 


Thanks, 

AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces AgentRefine, a novel framework aimed at enhancing the generalization capabilities of large language model (LLM)-based agents. The approach tackles the overfitting problem prevalent in existing agent-tuning methods by using a data generation pipeline that simulates diverse environments and tasks. 
- The framework avoids task-specific overfitting by synthesizing data with minimal reliance on task-specific information.
- The method addresses the generalization gap by leveraging diverse environments and tasks, ensuring agents adapt well to held-out scenarios.
- Experimental results demonstrate that AgentRefine outperforms existing baselines, highlighting its effectiveness.

The weaknesses are (1)the main idea, while practical, is not significantly novel; (2) the method heavily depends on GPT-4 for script and trajectory generation as well as error verification.

Currently, I think the strengths outweigh the weaknesses.