Rational Decision-Making Agent with Learning Internal Utility Judgment

Authors: Yining Ye, Xin Cong, Shizuo Tian, Yujia Qin, Chong Liu, Yankai Lin, Zhiyuan Liu, Maosong Sun
Primary area: generative models
Keywords: Decision Making, Autonomous Agent, Large Lanugage Model, Elo Rating

Abstract:
With remarkable advancements, large language models (LLMs) have attracted significant efforts to develop LLM-based agents capable of executing intricate multi-step decision-making tasks. Existing approaches predominantly build upon the external performance measure to guide the decision-making process but the reliance on the external performance measure as prior is problematic in real-world scenarios, where such prior may be unavailable, flawed, or even erroneous. For genuine autonomous decision-making for LLM-based agents, it is imperative to develop rationality from their posterior experiences to judge the utility of each decision independently. In this work, we propose RaDAgent (Rational Decision-Making Agent), which fosters the development of its rationality through an iterative framework involving Experience Exploration and Utility Learning. Within this framework, Elo-based Utility Learning is devised to assign Elo scores to individual decision steps to judge their utilities via pairwise comparisons. Consequently, these Elo scores guide the decision-making process to derive optimal outcomes. Experimental results on the Game of 24, WebShop, ToolBench and RestBench datasets demonstrate RaDAgent’s superiority over baselines, achieving about 7.8% improvement on average. Besides, RaDAgent also can reduce costs (ChatGPT API calls), highlighting its effectiveness and efficiency.

Decision: accept (poster)
Reviewer scores: 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730149954128

thread_type:
official_review

<thread_object>
creator: reviewer_v6tk

<content>
summary:
This work proposes RaDAgent (Rational Decision-Making Agent) that utilizes experience exploration and utility learning. Notably, Elo scores are assigned to each step to evaluate its utilities, which helps the agent to derive the optimal decisions. The authors provide empirical evidence that validates the efficacy of the proposed method on several tasks such as Game of 24, WebShop, and ToolBench. They also show that LLMs and the decision-making approaches are complementary, mutually enhancing each other to achieve superior performance outcomes.

strengths:
Overall, the idea of this work is novel and intuitive.

Overall, this paper is well-written and well-presented.

The proposed RaDAgent method significantly outperforms the baseline methods across most of tasks.

weaknesses:
I would recommend putting the Related Work section after the Introduction so that readers can gain more background information of this work. Indeed, explaining how the literature or baselines work can help the readers better understand the motivation of authors' idea.

The authors need to enhance the language and fix typos/grammar issues, e.g., in line 96-97, "take action", line 139 "Elo score" should be plural, etc.

Line 97-98, in general, one would use different functions to represent the policy and transition functions in MDP. I would expect a better introduction of MDP on page 2. For example, one can use a tuple (S, A, R, P, \mu) to represent an MDP. Please see more details in RL paper or Sutton's RL book. In addition, \pi is often used to denote the policy (action generator).

Even though mentioning in the Limitation section by authors, I am still interested in if RaDAgent can be implemented in other (at least one state-of-the-art) LLMs (e.g., Claude, Llama), especially not the GPT-series. Also, why the authors choose GPT-series as their backbone LLM? If RaDAgent does not work for other LLMs like Claude, what could be the possible reasons?

questions:
Please refer to the "weakness" section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730392790378

thread_type:
official_review

<thread_object>
creator: reviewer_edtj

<content>
summary:
This paper introduces a novel approach named RaDAgent that addresses the limitation of LLM-based agents' dependence on external performance measures for decision-making. Unlike existing methods that rely on potentially unreliable or unavailable external metrics, RaDAgent develops internal rationality through an iterative framework combining Experience Exploration and Utility Learning. Besides, the framework incorporates the Elo Rating system to learn and assign utilities to decision steps through pairwise comparisons, enabling agents to make autonomous decisions based on learned internal judgment.

strengths:
The main strength of this paper are:
1. The paper proposes a novel idea by shifting away from the conventional reliance on external performance measures to an internal utility judgment approach for LLM-based agents. 
2. The framework provides a practical solution for developing internal rationality in LLM-based agents.
3. The experimental results demonstrate promising performance across multiple datasets,

weaknesses:
The  main weakness of this paper are:
1. The Elo rating system might oversimplify the complexity of decision utilities. Besides, the pairwise comparison approach could be time-consuming for complex tasks.
2. The convergence for the Elo-based utility learning lacks theoretical analysis. While the paper demonstrates empirical effectiveness of the Elo rating system in learning decision utilities, it fails to provide rigorous theoretical guarantees about the convergence properties of this approach. An analysis of whether and how quickly the Elo scores converge to stable values would strengthen the theoretical foundation of the approach and provide better understanding of its reliability in different scenarios.

questions:
Since the paper lacks convergence analysis of this approach, I wonder how much additional computational overhead this method requires compared to existing approaches. The time cost of conducting multiple comparisons and updating Elo scores, especially for complex decision scenarios, could be substantial and needs to be evaluated against baseline methods.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730678188615

thread_type:
official_review

<thread_object>
creator: reviewer_ctwe

<content>
summary:
The paper studies how to improve the performance of LLM-agents on sequential decision making tasks. It discussed the drawbacks of relying on pre-defined external metrics to guide the decision making of LLM-agents. Furthermore, they propose to use Elo score together with Experience Exploration techniques.

strengths:
The proposal of using Elo score is reasonable and points out an interesting direction of rethinking how evaluation metrics can guide the decision making of LLM agents.

Meanwhile, the experiments are good and extensive.

weaknesses:
1) the paper is rather empirical and heuristic, with combination of two heuristic techniques.
2) Elo itself also has significant issues [1]. For examples, it may provide no useful signals sometimes. In other words, such a black-box use of Elo without significant extensions/more detailed discussions on its potential limitations is a sign of limited novelty.

[1] Re-evaluating evaluations. NeurIPS 2018

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1733293554880

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We would like to express our sincere gratitude to all the reviewers for their valuable time and insightful feedback. In response to their suggestions, we have made several revisions, including additional experiments and clarifications. Below is a summary of the key changes made:
1. We have conducted further experiments with other large language models, including `claude-3-5-haiku` and `llama-3.1-8b`. These results validate the effectiveness of our proposed method across various LLMs (Reviewer v6Tk).
2. Our paper has been revised for improved clarity and precision. This includes adjustments to the positioning of the "Related Works" section, correction of typo errors, and the introduction of formal MDP notations (Reviewer v6Tk).
3. We have analyzed the API call usage of RaDAgent, demonstrating that incorporating Elo score calculations significantly reduces the number of API calls required for decision-making. This highlights the efficiency of our RaDAgent's decision process (Reviewer edTJ).
4. We have clarified the motivation and rationale for selecting the Elo rating system in our agent's decision-making process. Specifically, we emphasize how the inherent characteristics of the Elo algorithm align with the decision-making requirements of the agent (Reviewers edTJ and CTwe).
5. We have highlighted the technical novelty and innovation of adapting the traditional Elo rating system to agent decision-making scenarios (Reviewer CTwe).

Once again, we extend our thanks to all reviewers and area chairs for their invaluable contributions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a new approach named RaDAgent that addresses the limitation of LLM agents' dependence on external performance measures for decision-making. Instead, the authors propose to use the internal rationality of the agents through an iterative framework combining Experience Exploration and Utility Learning. The framework also incorporates the Elo Rating system to learn and assign utilities, enabling agents to make autonomous decisions based on the learned internal judgment. Overall, the introduction of internal rationality and Elo rating in building LLM agents for decision-making is novel, the experimental results are promising, and the manuscript is overall well-written and easy to follow. The main concerns were regarding the justification of the use of Elo rating system and the lack of theoretical analysis, and the re-organization of certain parts/sections. Overall the paper contains new ideas and results that can benefit the advancement of LLM agents for decision-making. I suggest the authors to incorporate the feedback and new results from the rebuttal in the next version of the paper.