EcoAct: Economic Agent Determines When to Register What Action

## Abstract

Abstract Recent advancements have enabled Large Language Models (LLMs) to function as agents that can perform actions 1 1 1 Unless otherwise stated, the term ’action’ is defined as using a specific tool across the paper. using external tools.
This requires registering, i.e. integrating tool information into the LLM context prior to taking actions.
Current methods indiscriminately incorporate all candidate tools into the agent’s context and retain them across multiple reasoning steps. This process remains opaque to LLM agents and is not integrated into their reasoning procedures, leading to inefficiencies due to increased context length from irrelevant tools.
To address this, we introduce EcoAct , a tool-using algorithm that allows LLMs to selectively register tools as needed, optimizing context use. By integrating the tool registration process into the reasoning procedure, EcoAct reduces computational costs by over 50% in multi-step reasoning tasks while maintaining performance, as demonstrated through extensive experiments. Moreover, it can be plugged into any reasoning pipeline with only minor modifications to the prompt, making it applicable to LLM agents now and future.

## 1 Introduction

Large language models (LLMs) have been conceptualized as agents and have demonstrated their capability to perform a broad range of complex tasks. When augmented with external tools (Yuan et al., ; Qu et al., ; ), LLM agents can extend their functionality beyond conventional natural language processing (Qin et al., ). For example, LLM agents equipped with scientific tools can conduct scientific research (Bran et al., ; Ghafarollahi & Buehler, ), while those integrated with physical robotic systems are capable of performing robotic manipulations (Ahn et al., ; Huang et al., ). External tools essentially expand the action space of LLM agents, enabling them to leverage existing functionalities to accomplish a variety of complex tasks (Xi et al., ; Wu et al., ; Peng et al., ; Wu et al., ; Shridhar et al., ; Hua et al., , , ).

To equip LLM agents with external tools, they must undergo a *tool registration* procedure. Specifically, information about the candidate tools needs to be added to the context of the LLMs that support the agents. This information represents essential details for tool usage, including tool names, descriptions in natural language, and instructions for input parameters.
The current practice in tool registration indiscriminately incorporates all candidate tools into the agent’s context, where these candidate tools are preemptively selected by users or retrieved automatically through external algorithms (Ocker et al., ; Qin et al., ; Gao et al., ). LLM-based agents will then process contextual information from all registered tools and select the appropriate tool for each reasoning step.
However, this paradigm, which involves preparing all tools in advance and keeping the full information of the registered tools within the LLM’s operational context, introduces one key issue: the tool registration process is opaque to the agents and not fully integrated into their autonomous reasoning pipelines.
Each time the LLM is invoked, information from all passively registered tools is processed, even though not all tools are necessary and only one single tool can be utilized in each step, which drives inefficiencies in both cost and inference time (see Figure 1a). The problem becomes more pronounced as the number of pre-registered tools grows, imposing an even greater burden on the agent’s decision-making process.
The agent possesses the capacity to *reason to act* with their intrinsic reasoning mechanism but lacks the ability to *reason to register*.

Figure: Figure 1: Overview of EcoAct, illustrating its effects after being integrated into the single-trace reasoning algorithm ReAct, which can serve as a fundamental component of complex reasoning methods. (a) In ReAct, all tools are registered in advance, retaining full information of these tools within the LLM’s operational context at each reasoning step. This leads to unnecessarily long contexts, as tools irrelevant to the current problem remain included. (b) In contrast, EcoAct leverages ReAct’s intrinsic reasoning capabilities to register only the tools deemed necessary, based on their concise and distinct identifiers - tool names, thus addressing the mentioned efficiency issues.
Refer to caption: x1.png

In this work, we present EcoAct, a general tool-using paradigm that integrates the tool registration procedure into the LLM agents’ reasoning procedure, granting them discretionary authority, which is the freedom to register any tools they wish to use at any time through their intrinsic reasoning mechanisms (see Figure 1b).
For any potentially useful tool, EcoAct prompts the agent to reason about registering the function before utilizing it, rather than passively accepting pre-prepared tools at each reasoning step.
EcoAct gives agents the flexibility to register tools according to actual needs, thereby retaining only the necessary tools in the context and reducing costs.
While the effectiveness of this tool registration process can be further enhanced through other agent reasoning methods (Yao et al., , ; Qin et al., ; Wei et al., ), ensuring that tools are registered appropriately is essential for maximizing the agent’s task-solving capabilities.
Specifically, before the agent begins taking actions to solve the user’s query using its intrinsic reasoning algorithms, we only provide the agent with one single meta-tool named *tool_register*, which enables the agent to register any tools deemed useful based on lightweight but easily-identifiable contextual information - tool names at any time step. The agent here will rely on this meta-tool to extend its skill library and solve the problem with its own self-registered tools.
Additionally, the agent’s intrinsic reasoning algorithms are seamlessly integrated into EcoAct. This integration enables the agent to employ its own reasoning logic to determine when and which actions to register.

We conduct extensive experiments on the ToolBench benchmark (Qin et al., ), which involves a diverse array of large-scale tools. We utilize EcoAct to enhance both the classic single-trace reasoning method ReAct (Yao et al., ) and the complex tree-structured reasoning method DFSDT (Qin et al., ), applying it across multiple models, The results show that the enhanced reasoning algorithms even can achieve monetary cost savings of over 50% on queries involving large tools from ToolBench, without compromising performance.
Additionally, we conduct further analysis to demonstrate the effectiveness of key design choices in the proposed algorithm, regarding aspects such as the granularity of tool registration and the concise context used during tool registration.

Our contributions are summarized below:
(1) We highlight a key limitation in the current tool-utilization paradigm of LLM agent systems: tool registration is essentially opaque to the LLM agents. Indiscriminately maintaining information about all registered tools within the LLM’s operational context imposes a significant burden on the agent’s decision-making process.
(2) We introduce EcoAct, a plug-and-play algorithm that could seamlessly integrate tool registration into the agent’s intrinsic reasoning procedures. The agent could *reason to determine when to register what tools* based on its needs, thereby mitigating the burden of processing all accessible tools in the backed LLM calling by only maintaining necessary tools.
(3) We conduct comprehensive experiments using the ToolBench benchmark, which encompasses a wide range of large-scale tools. Our results demonstrate that the enhancement of EcoAct enables significant cost savings through various reasoning methods. Notably, for queries involving large tools from ToolBench, we observe cost reductions exceeding 50% across multiple models.

## 2 Method

In this section, we present EcoAct, a general tool-using algorithm designed to mitigate efficiency issues in agent tool-using scenarios. We begin by formulating the research problem and then provide the details of each component designed in EcoAct.

### 2.1 Problem setup

We start by defining the relevant notations and outlining the research problem. Consider a language agent and a set of tools $\mathcal{Z}=\{z_{i}\}_{i=i}^{I}$ that the agent could access. The agent’s objective is to address user queries according to a specific policy $\pi$. At any given decision time step $t$, the agent receives two types of information: (1) the historical context $c_{t}$ which includes all previous action-observation pairs, and (2) a set of accessible tools $\mathcal{Z}$ that can be used in this time step. The agent then must determine the next action to take. Formally, this decision process can be expressed as:

$$ $\pi(c_{t},\tilde{\mathcal{Z}})\rightarrow a_{t},\ \ \text{s.t.}\ \ a_{t}\in \mathcal{A},$ (1) $$

where $a_{t}$ denotes the action that been taken at time step $t$. It represents one specific tool-calling from accessible tool set $\tilde{\mathcal{Z}}$. $\mathcal{A}$ denotes the action space of this language agent. Consequently, the size of the tool space is equivalent to the size of the action space, i.e., $|\mathcal{A}|=|\mathcal{Z}|$.

In evaluating a specific agent algorithm, the total token consumption required to complete user queries, which encompass both input and output tokens—serves as a general metric for assessing the algorithm’s cost (Chen et al., ; Wang et al., ; Hidvégi et al., ; Cheng et al., ; Ding et al., ). This is because token consumption is positively correlated with budget expenditure and latency, particularly in the context of large language models as a service (Gan et al., ; Sun et al., ).
At time step $t$, we use the cost function $j(c_{t},\tilde{\mathcal{Z}},a_{t})$ to represent the cost associated with making a decision at that step $t$. The one-step cost is given by:

$$ $j(c_{t},\tilde{\mathcal{Z}},a_{t})=\alpha\cdot(l(c_{t})+l(\tilde{\mathcal{Z}}) )+\beta\cdot a_{t},$ (2) $$

where $l$ measures the token length. $\alpha$ and $\beta$ denote the cost per input token and output token, respectively, which are determined by the LLMs inference service provider. Under this formulation, the total cost $\mathcal{J}$ for completing users query with $n$ reasoning steps is:

$$ $\mathcal{J}^{\text{total}}=\sum_{t=1}^{n}j(c_{t},\tilde{\mathcal{Z}},a_{t}),\ \text{where}\ a_{t}=\pi(c_{t},\tilde{\mathcal{Z}}).$ (3) $$

The focus of our research is to minimize the total cost $\mathcal{J}^{\text{total}}$ while maintaining a good performance in response to user queries.

### 2.2 EcoAct

Motivation. According to Equation 3, the polynomial $\mathcal{J}^{\text{total}}$ depends on $\tilde{\mathcal{Z}}$, $c_{t}$, and $a_{t}$. We primarily examines the token consumption associated with the input tool set $\tilde{\mathcal{Z}}$ at each time step, considering $c_{t}$ and $a_{t}$ as less controllable factors in practice.
Most approaches for identifying $\tilde{\mathcal{Z}}$ at each time step rely on a retrieval-based methods. A subset of tools is retrieved for each query and registered with the agent, which then makes sequential decisions until the problem is resolved Qin et al. (); Patil et al. (). However, a key limitation of this once-for-all paradigam is that each decision step processes contextual information from all retrieved tools, despite only one tool being utilized per step, which drives cost and latency.
In Figure 2(a), we present the ratio of tokens consumed by the necessary tools used at each decision step compared to the total number of tokens from all input tools, using the candidate tools retrieved by state-of-the-art tool using method AnyTool (Du et al., ) in ToolBench (Qin et al., ).
It is observed that most portion of tokens is allocated to redundant tools rather than those that are actually executed.
Except for cost issues, choosing from a large set of tools with extensive contextual information poses a challenge for large language models, as it results in a needle-in-a-haystack problem (Li et al., ).
(^2^22 [https://github.com/gkamradt/LLMTest_NeedleInAHaystack](https://github.com/gkamradt/LLMTest_NeedleInAHaystack)).

Figure: (a) Token cost: necessary vs. actual for tool-using
Refer to caption: x2.png

Overview.
The core concept behind EcoAct is to empower agents to autonomously register tools they find useful, rather than passively relying on pre-assigned tools. Two key questions arise in the design of this algorithm:
(1) Is there a short yet distinct context that can assist agents in filtering and selecting the necessary tools from the available options without adding extra computational cost?
(2) Given the answer to the first question, how can tool registration be seamlessly integrated into the agent’s intrinsic reasoning process, enabling agents to determine, at each reasoning step, which tool to register and when, based on this distinctive information?

To address the first question, we propose utilizing tool names as easily recognizable tags to assist agents in determining which tools should be registered.
To operationalize this, we introduce a meta-tool, *tool_register*, and define an action enabling agents to register any tools they consider relevant based on these tool names.
The workflow is as follows:
(1) Initialization: Prior to task execution, agents are equipped solely with the *tool_register* meta-tool. Simultaneously, users provide queries that include all available tool names along with instructions on how to use the tool register. Once the agent registers a tool by name using *tool_register*, detailed information about the registered tool becomes available.
(2) Reasoning: At each reasoning step, the agent can either register a new tool or invoke a previously registered one, depending on the task requirements. We then detail the design of this process and the intuition behind each step.

##### Tool name as informer.

The initial phase of EcoAct involves identifying a concise yet distinct context from the candidate tools, allowing agents to determine which tools are essential and which are not, utilizing their intrinsic reasoning capabilities without increasing computational load.
As illustrated in Figure 2(b), we present the average token consumption associated with each component of an external tool.
We could observe that the majority of token consumption is attributable to tool descriptions and input parameter instructions, whereas tool names account for only 6.6% of the total token usage.
Inspired by the efficiency of human tool using, wherein the utility of a tool can often be inferred from its name without recalling every specific detail, we propose leveraging tool names as the most easily identifiable markers to identify which tools require in-depth learning and which can be bypassed.
Since tool names are often highly recognizable, this approach intuitively imposes minimal burden on the language model’s context processing.

##### tool_register as meta-tool.

To achieve the objective of retaining only the necessary tools based on their names, we propose enabling agents to actively register tools using their intrinsic reasoning capabilities. Specifically, before the agent engages in a task, we (1) provide it with a list of tool names, which incurs minimal token usage, and (2) introduce a single tool, *tool_register*, which facilitates an action allowing the agent to register tools deemed useful based on their names at each time step. When the agent invokes *tool_register* with a selected tool name, it receives the complete information about the tool and adds it to its skill library.
Essentially, we initialize the action space $\mathcal{A}$ with a single ”meta-action,” i.e., $\mathcal{A}_{t=0}={\tilde{a}}$. Here, $\tilde{a}$ represents the action of using *tool_register* to register one tool, thereby expanding the action space over time. This strategy prevents the indiscriminate incorporation of all candidate tools into the context, fostering more efficient tool utilization.

Since tool registration has been integrated as a specialized action within the agent’s action space, our algorithm offers several distinct advantages:
(1) Orthogonal to Agent Reasoning Algorithms: Our method essentially forges a meta-tool capable of registering any tool deemed useful across various agent reasoning algorithms. As demonstrated in Section 3.3.1, it is agnostic to the specific reasoning algorithms employed and performs effectively with diverse reasoning techniques.
(2) Efficiency with Large-Scale Toolsets: When dealing with queries with a vast number of tools, our method significantly reduces operational costs, achieving notable cost savings as detailed in Section 3.2. This efficiency arises because directly integrating a large number of tools into the agent is more costly. EcoAct minimizes the number of tools registered. Thus, our method is particularly beneficial in scenarios involving extensive toolsets, as it ensures that only the essential tools are being utilized.
(3) Simplicity and Intuitiveness: Our method mirrors human problem-solving strategies involving multiple tools: it first filters tools based on simple identifiers (tool names) and then examines the details of the selected tools before use. This approach not only simplifies the process but also provides a general framework that could inspire the design of other agent algorithms.

## 3 Experiments

We conduct experiments to prove the superiority of the proposed method. We begin by providing the experimental settings in Section 3.1. We then evaluate the EcoAct on ToolBench benchmark to verify its effectiveness in Section 3.2. Finally, we perform in-depth investigations in the last two sections to provide a better understanding of EcoAct.

### 3.1 Experimental setup

Data preparation. We mainly conduct experiments on the ToolBench (Qin et al., ), which is large-scale dataset for tool use. It involves 16,464 tools in total which has been widely used as the benchmark to make evaluations of tool use algorithm (Du et al., ; ).
ToolBench comprises six subsets G1-Instruction (G1-I), G1-Tool (G1-T), G1-Category (G1-C), G2-Instruction (G2-I), G2-Category (G2-C), and G3-Instruction (G3-I). These subsets are classified according to varying levels of complexity in tool use, with differences in ’Instruction’, ’Category’, and ’Tool’ reflecting the relationships between tool categories in these test subsets and those in the training sets. Following the same setting with AnyTool (Du et al., ), we adopted the filtered benchmark which excludes all non-solvable queries in ToolBench. The remaining queries in these six subsets are 115, 132, 142, 107, 98, and 38, respectively. Unless specified otherwise, for each query, we use the state-of-the-art method AnyTool (Du et al., ) to retrieve tools for each query in all experiments across the paper. More details of this benchmark could be found in Appendix A.

Evaluation metrics. We primarily use two metrics to make evaluations: the pass rate and the cost, with the latter measured in monetary terms. Pass rate essentially measures LLM’s ability to successfully execute an instruction within limited budgets. We utilize the evaluation script from (Du et al., ) to get the pass rate results in all experiments of our paper, which addresses issues related to artificially inflated pass rates (Du et al., ). We utilize GPT-4-turbo to make the pass rate evaluations, applying the same prompts as those used in ToolBench.
Unless specified otherwise, we report the cost in US cents. More details about the evaluation prototype can be found in Appendix A.2.

### 3.2 Main results

**Table 1: Comparison of the basic agent reasoning algorithm ReAct with its variant augmented with EcoAct. We show the pass rate performance and cost per query in US cents (¢) with different models in ToolBench (Qin et al., ) benchmark. We could observe that EcoAct significantly reduces costs associated with ReAct while maintaining comparable performance.**
| Model | Method | G1 | G2 | G3 | Average |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PR (%) | Cost (¢) | PR (%) | Cost (¢) | PR (%) | Cost (¢) | PR (%) | Cost (¢) |  |  |  |
| GPT-4-turbo | ReaAct | 16.2 | 6.3 | 18.5 | 8.1 | 13.2 | 11.5 | 16.0 | 8.6 |  |
| GPT-4-turbo | ReaAct w/ EcoAct | 16.7 | 5.9 ($\bm{\downarrow 6.4\%}$) | 18.0 | 6.1 ($\bm{\downarrow 24.7\%}$) | 13.2 | 7.7 ($\bm{\downarrow 33.1\%}$) | 16.0 | 6.6 |  |
| GPT-4o | ReaAct | 19.8 | 4.9 | 20.5 | 6.7 | 18.4 | 10.7 | 19.6 | 7.4 |  |
| GPT-4o | ReaAct w/ EcoAct | 20.1 | 3.7 ($\bm{\downarrow 24.5\%}$) | 20.8 | 4.8 ($\bm{\downarrow 28.4\%}$) | 21.1 | 5.8 ($\bm{\downarrow 45.8\%}$) | 20.7 | 4.8 |  |

Figure: (a) GPT-4-turbo: cost
Refer to caption: x4.png

EcoAct essentially serves as a plug-and-play component for different agent reasoning algorithms. Here, we mainly evaluate the impact of EcoAct on the classical reasoning method ReAct (Yao et al., ). Our aim is to assess both its effect on the overall performance and its influence on the token cost associated with solving user queries. We mainly investigate our algorithm on ReAct because ReAct as the classic single trace agent reasoning algorithm could be regarded as the basic component of different agent reasoning algorithms (Yao et al., ; Qin et al., ) with multiple reasoning traces. We expect the conclusions drawn from ReAct experiments could potentially be generalized to more complex algorithms in the future.

Overall results. In this study, we compare ReAct with its variant augmented with EcoAct and present the aggregated results of different subsets in G1, G2, and G3 from ToolBench benchmark in Table 1, focusing on both performance and monetary costs, with different models. Our findings indicate that EcoAct significantly reduces the costs associated with ReAct while maintaining, and in some cases exceeding, its performance. This suggests that EcoAct serves as a ”free lunch” component, enhancing ReAct without diminishing its reasoning capabilities. Given that ReAct functions as a foundational component for more complex reasoning algorithms, the experimental results highlight the potential for applying this cost-saving advantage to more sophisticated algorithms built upon ReAct.
Interestingly, we observe minor performance improvements in some subsets, such as G1 on the GPT-4-turbo model and G3 on the GPT-4o model. These improvements may stem from our method’s ability to address the ”needle-in-a-haystack” problem (Li et al., ). By progressively expanding its tool library, the agent reduces the difficulty of selecting the most appropriate tool from a large set, thereby enhancing overall performance.

Performance on multiple tool scales. To better show the advantages of EcoAct, we also present the performance metrics and cost-saving percentages across various tool scales as assessed in the benchmark in Figure 3.
We want to investigate the effect of our method for the queries with different tool scales.
We could observe that EcoAct provides greater cost savings for queries involving large-scale tools, achieving token savings of 54.35% and 53.82%, respectively, with large-scale tools (Level 3). This is due to the fact that traditional tools encounter higher costs when processing large-scale inputs, as they require the entire set of tools to be fed into the language model, resulting in increased expenses. In contrast, EcoAct addresses this issue by inputting only the complete information for registered tools, thereby avoiding redundant costs and optimizing overall efficiency.
Additionally, we note a slight cost increase in Level 1 in some cases. This occurs because EcoAct introduces an additional LLM calling procedure for tool registration step. When the number of tools for specific queries is small, the cost of incorporating all tool information may be less than the cost of this extra LLM call, causing the advantages of our method to diminish.

### 3.3 More analysis

#### 3.3.1 Extension to complex reasoning strategy

**Table 2: Comparison of multiple-traces reasoning strategy DFSDT (Qin et al., ) with its variant augmented with EcoAct. We could observe that EcoAct still could significantly reduces costs associated with DFSDT while maintaining comparable performance.**
| Method | G2-instruction | G3-instruction |  |  |
| --- | --- | --- | --- | --- |
| PR (%) | Cost (¢) | PR (%) | Cost (¢) |  |
| DFSDT | 31.8 | 30.8 | 28.9 | 44.3 |
| DFSDT w/ EcoAct | 31.8 | 22.9 ($\downarrow 25.7\%$) | 26.3 | 21.8 ($\downarrow 49.2\%$) |

In this section, we examine the impact of EcoAct on the performance of the multiple-traces reasoning strategy DFSDT (Qin et al., ; Du et al., ).
DFSDT allows agents to assess multiple reasoning paths and make informed decisions about whether to retract steps or continue along a promising path. The results, as shown in Table 2, indicate that integrating EcoAct with DFSDT results in notable cost savings while maintaining comparable performance on the most advanced model GPT-4o.
Additionally, we observe that the cost per query in DFSDT is considerably higher than in the single-trace reasoning algorithm ReAct, for both our method and the baseline. This is due to the increased token usage and reasoning steps required by the multiple-traces approach. Consequently, the absolute cost savings achieved through our method are even more pronounced.
These findings suggest that EcoAct is both versatile and beneficial across different reasoning methods. Whether applied to the single-trace reasoning of ReAct or the more complex DFSDT approach, EcoAct consistently enhances performance, affirming its effectiveness as a plug-and-play solution.

#### 3.3.2 Skill-library evolution

Figure: (a) G1-instruction
Refer to caption: extracted/5974793/figure/evo_G1_instruction.png

We then investigate how the number of registered tools evolves over time in the ReAct method when enhanced with EcoAct. Our primary objective is to investigate whether, in scenarios where a large number of tools are available for users’ queries, EcoAct could cause ReAct to register an excessive number of tools greedily. Such behavior could lead the algorithm to revert to its original state by registering all available tools at the begining time, thereby undermining the benefits of EcoAct.
To answer this question, we selected queries where the number of tools exceeded 20 from each group one of G1-I, G2-I, G3-I and conducted experiments to track the ratio of registered tools to the total number of available tools over time. We also compared this ratio across different models. The results, averaged within each subset of data, are displayed in Figure  4.

From the results we could observe that
(1) EcoAct is flexible. Tool registrations occur throughout the entire problem-solving process, suggesting that the agent is capable of registering any tool it deems useful at any point. Moreover, EcoAct may encompass a self-correction mechanism —if the agent realizes that a registered tool is unsuitable after obtaining more detailed information, it can leverage its intrinsic reasoning ability to register a more appropriate one in any time step.
(2) We also could observe that the ultimately registered tools constitute only a small fraction of the total available tools, approximately 30% in all three subsets with all LLM models. The resukts demonstrates that, in cases where a large number of tools are available, most of them are redundant (about 70%), and EcoAct could effectively prevents the registration of these redundant tools.

### 3.4 Ablations

#### 3.4.1 Single-Tool vs. multiple-tool registration

**Table 3: We compared two tool registration mechanisms in EcoAct: (1) single-tool registration per step, and (2) multiple-tool registration per step. Experiments on G2/G3-I. subsets from ToolBench benchmark, using EcoAct to augment the ReAct algorithm on the GPT-4o model, revealed that multiple-tool registration led to a significant performance decline, even worse than standard ReAct.**
| Method | G2-instruction | G3-instruction | Average |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| PR (%) | Cost (¢) | PR (%) | Cost (¢) | PR (%) | Cost (¢) |  |
| ReAct | 20.5 | 6.7 | 18.4 | 10.7 | 19.5 | 8.7 |
| ReAct w/ EcoAct (Single Tool Reg.) | 20.8 | 4.8 | 21.1 | 5.8 | 21.0 | 5.3 |
| ReAct w/ EcoAct (Multiple Tools Reg.) | 14.9 ($\pagecolor{red!25}\downarrow 5.9\%$) | 4.5 | 13.2 ($\pagecolor{red!25}\downarrow 7.9\%$) | 5.4 | 14.1 | 5.0 |

In our approach, the proposed meta-tool *tool_register* is designed to register only one tool per tool registration action in each time step. This naturally raises one critical question: could registering multiple tools simultaneously reduce costs while maintaining comparable performance? The intuition behind this is that each tool registration essentially require one LLM calling, which incurs token costs. If the agent could leverage its internal reasoning mechanisms to register several potentially useful tools in one interaction, it might lead to cost savings due to the decrease of LLM call number.
To explore this hypothesis, we modified *tool_register* to allow for the registration of multiple tools at once, allowing agents to select as many tools as deemed necessary based on their reasoning.
We conducted experiments using the G2-I and G3-I datasets in state-of-the-art GPT-4o model according to Table 1, where we use Ecoct to augment ReAct and present the results in Table 3.

From the results, we could observe that enabling *tool_register* to handle multiple tool registrations per action results in minor cost savings. However, the performance of EcoAct decreases significantly, with a 5.9% and 7.9% drop in the G2-instruction and G3-instruction tasks, respectively. The cost savings arise from reducing repeated LLM calls, which otherwise require inputting the entire conversation history each time. However, the performance drop may be attributed to the agent’s tendency to greedily register multiple tools at once, which introduces complexity for each action taking. This increased complexity makes it easier for the agent to incorrectly select a tool from the larger pool, compared to having only a single registered tool, potentially leading to error propagation.

#### 3.4.2 Tool names vs. descriptions information in tool registration

**Table 4: We compared two variants of EcoAct for tool registration: (1) using tool names only, and (2) using both names and descriptions. Experiments on G2/G3-I subsets from ToolBench benchmark, using EcoAct to augment the ReAct algorithm on the GPT-4o model, showed that adding tool descriptions did not significantly improve performance but increased costs.**
| Method | G2-instruction | G3-instruction | Average |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| PR (%) | Cost (¢) | PR (%) | Cost (¢) | PR (%) | Cost (¢) |  |
| ReAct | 20.5 | 6.7 | 18.4 | 10.7 | 19.5 | 8.7 |
| ReAct w/ EcoAct (Tool reg. by tool names) | 20.8 | 4.8 | 21.1 | 5.8 | 21.0 | 5.3 |
| ReAct w/ EcoAct (Tool reg. by tool names and des.) | 21.4 | 6.3 ($\uparrow 31.3\%$) | 18.4 | 9.6 ($\uparrow 65.5\%$) | 19.9 | 8.0 |

We then investigate the feasibility of incorporating both tool names and descriptions in the tool registration process. We aim to address the following questions: Is the tool name sufficient for accurate tool registration? Does the inclusion of tool descriptions enhance registration performance? We also examine whether this modification affects the associated costs. To evaluate these questions, we conduct experiments using the G2-instruction and G3-instruction subsets, incorporating all available tool descriptions for registration within the ReAct framework, leveraging the arguments from EcoAct in the GPT-4o model. The results of our experiments are presented in Table 4.

From the results, we could observe that the including tool descriptions for tool registration does not necessarily lead to a noticeable improvement in performance. However, this approach incurs a significant increase in cost, comparable to that of standard ReAct. Specifically, the cost increases 31.3% and 65.5% in G2-instruction and G3-instruction respectively.
This finding suggests that tool names alone provide sufficient information for the agent to perform correct tool registration. This is because the context of tool descriptions is obviously larger than tool names. Consequently, the inclusion of tool descriptions may be unnecessary and could result in substantial cost increases.

## 4 Related Works

Large language models (LLMs) represent a major breakthrough in artificial intelligence, prompting an increasing body of research dedicated to employing LLMs in the construction of autonomous agents capable of performing complex tasks (Xi et al., ; Wu et al., ; Peng et al., ; Shridhar et al., ; Song et al., ; Wu et al., ; Zhang et al., ; Ma et al., ). In these LLM-based agents, the ability to leverage external functions, tools, or actions to interact with the environment or solve sub-tasks is crucial. These external tools empower agents to go beyond natural language processing. For instance, LLM agents equipped with scientific tools can conduct scientific research (Bran et al., ; Ghafarollahi & Buehler, ), while those integrated with robotic systems can perform robotic manipulation tasks (Ahn et al., ; Huang et al., ).

To enable agents to use external tools, they must undergo a process called *tool registration*, where relevant tool information is integrated into the LLM’s context prior to the agent taking action. This process becomes challenging when the number of available tools exceeds the context limits. One approach to mitigate this limitation is through retrieval-augmented generation (RAG) (Lewis et al., ; Gao et al., ). For example, Patil et al. (); Li et al. () use a pre-trained text embedding model to retrieve relevant tools from a large tool pool. Similarly, Qin et al. () trained an additional API retriever to identify essential tools using curated tool retrieval data.

To handle user queries with registered tools, various reasoning algorithms for LLM agents have been explored recently  (Yao et al., ; Qin et al., ). Specifically, Yao et al. () propose an approach that interleaves the generation of reasoning traces with tool-using actions, leading to more reliable and factual responses. Qin et al. () introduce DFSDT, a decision tree-based method that expands the search space, increasing the likelihood of identifying a valid tool-using path. However, these reasoning algorithms do not integrate with the tool registration process, which can result in unnecessary costs due to the registration of irrelevant tools. In contrast, our approach seamlessly incorporates tool registration into these reasoning algorithms, allowing agents to autonomously reason about and register only the necessary tools, thereby avoiding such inefficiencies.

## 5 Conclusion

In this work, we propose EcoAct, a simple yet effective approach that seamlessly integrates tool registration into the intrinsic reasoning processes of LLM agents. The core concept involves initializing the agent with a meta-tool named *tool_register*, which enables the agent to selectively register tools deemed useful based on their names at each time step. This action allows the agent to avoid indiscriminately incorporating all candidate tools into its context, instead retaining only relevant information across reasoning steps, thereby achieving significant cost savings.
We evaluate EcoAct on the ToolBench dataset, augmenting various reasoning methods, and demonstrate that EcoAct significantly reduces computational costs while maintaining comparable performance.

## Appendix A More Details about ToolBench

ToolBench is a large-scale tool-usage dataset comprising 16,464 real-world RESTful APIs across 49 categories from the RapidAPI Hub.
All queries in this benchmark were generated by prompting ChatGPT to create diverse tasks involving these APIs, covering both single-tool and multi-tool usage scenarios.
Through careful human evaluation, the authors determined that the generated instructions exhibit high diversity, reflecting a wide range of practical applications.
This benchmark has been widely adopted as a standard evaluation tool in several studies (Du et al., ; ).

### A.1 Subsets information

The dataset is categorized into three levels: G1, G2, and G3, which correspond to single-tool instructions, intra-category multi-tool instructions, and intra-collection multi-tool instructions, respectively. Each level is further subdivided into three subcategories:

- •
Instruction: unseen instructions using the same set of tools as in the training data.
- •
Tool: unseen tools from previously encountered category as those in the training data.
- •
Category: unseen tools from an entirely different, previously unobserved category

However, since our EcoAct algorithm does not rely on training data, the distinctions between these three subsets are minimal.

### A.2 Evaluation protocol

We adopt the same pass rate evaluation protocol as outlined in AnyTool (Du et al., ). In the original ToolBench benchmark, the authors employ a two-stage evaluation process. In the first stage, ToolBench uses an LLM (GPT-4 in our paper) to assess whether the selected API candidates can address the query, classifying them as either ’solvable’ or ’non-solvable’. For queries deemed ’solvable’, the LLM then evaluates the effectiveness of the solution, labeling it as either ’solved’ or ’unsolved’. The pass rate is calculated using the following equation:

$$ $\text{Pass Rate}=\frac{\text{Non-solvable}+\text{Solved}}{\text{Non-solvable}+ \text{Solved}+\text{Unsolved}}$ (4) $$

A key issue with this evaluation protocol arises when there is a large number of ’non-solvable’ queries identified by GPT-4. This can result in an artificially high pass rate, despite many queries remaining unsolved. To mitigate this, Du et al. () conducted a manual review of all queries, retaining only those that can be resolved. Consequently, the pass rate is calculated using the following equation:

$$ $\text{Pass Rate}=\frac{\text{Solved}}{\text{Solved}+\text{Unsolved}}$ (5) $$

More information of this evaluation protocol could be found in the original paper (Du et al., ).
In terms of cost calculation, the monetary cost is computed based on the corresponding pricing from Microsoft Azure.

## Appendix B More Implementation Details

When using AnyTool to retrieve tools for each query, we set the maximum size of the API-Candidate Pool to 64, drawing on the findings of the AnyTool paper, which suggest that a pool size of 64 nearly saturates performance.
Additionally, we increased the maximum reasoning steps to 24, up from the default of 12, to explore the behavior of EcoAct under conditions without budget constraints.
If not otherwise specific in the paper, we use the same hyperparameters as it is used in ToolBench (Qin et al., ), considering hyperparameters play a critical role in determining how well an algorithm performs (Zhang et al., , ).

## Appendix C Prompts

### C.1 Prompt design for EcoAct

**Table 5: Prompt for EcoAct.**
| You are AutoGPT, you can use many tools (functions) to do the following task.<br>First I will give you the task description, and your task start.<br>At each step, you need to give your thought to analyze the status now and what to do next, with a function call to actually excute your step.<br>After the call, you will get the call result, and you are now in a new state. Then you will analyze your status now, then decide what to do next.. After many (Thought-call) pairs, you finally perform the task, then you can give your finial answer.<br>Remember: 1.the state change is irreversible, you can’t go back to one of the former state, if you want to restart the task, say "I give up and restart".<br>2.All the thought is short, at most in 5 sentence.<br>3.You can do more then one trys, so if your plan is to continusly try some conditions, you can do one of the conditions per try.<br>Let’s Begin!<br>Task description: You should use functions to help handle the real time user querys. But every function needs to be selected using "function_selection" function before use it. Remember:<br>1.ALWAYS call F̈inishf̈unction at the end of the task. And the final answer should contain enough information to show to the user,If you can’t handle the task, or you find that function calls always fail(the function is not valid now), use function Finish->give_up_and_restart. 2. do not call the function you have not successfully selected. |
| --- |

### C.2 tool_register

### C.3 Prompt for ReAct/DSFDT

Table: Table 6: Prompt for Creating ReAct/DSFDT.

### C.4 Prompt for pass rate evaluations

#### C.4.1 Prompt template for verifying whether the query has been resolved

#### C.4.2 Prompt template for verifying whether the query is solvable

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. What is the maximum number of steps the LLM can perform before failure is declared?
soundness: 1
strengths: The authors conduct experiments using ReAct and DFSDT on the ToolBench benchmark, demonstrating cost reductions.
summary: This paper proposes a cost-reduction method in which tools are registered only when required by the LLM, thereby optimizing context usage. The method is claimed to maintain performance while reducing costs with models like GPT-4 and GPT-4-turbo.
weaknesses: The proposed method reduces context length at each reasoning step by listing all tools at the beginning of the instruction, then registering only a single tool as needed. While this approach yields cost savings, the benefit is somewhat straightforward, and the performance impact is minimal. The method primarily benefits prompting techniques that provide all tool information at each step. Additionally, the cost reductions are specific to API expenses, making it unclear how this approach would apply to open-source models that run locally with comparable performance. This substantially limits the method's applicability and overall impact.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Have you considered other approaches for minimizing tool registration overhead besides selective registration, e.g., RAG, fine-tuning, etc., for tool selection?

2. How would EcoAct recover from errors when initial tool selections fail, especially under ambiguities present in the tool names?

3. How would EcoAct perform if there are multiple tools that are feasible for the user query?

4. Why didn’t you consider employing open source LLMs for experiments?

5. Are there any comparisons with related works like ToolLlama, Gorilla, etc.?
soundness: 2
strengths: **Explicit Tool Registration Step:** EcoAct introduces an explicit tool registration step early in the reasoning process to select the most appropriate tool, effectively reducing token costs. This proactive approach helps optimize computational efficiency by only registering tools when they are needed, rather than pre-registering all possible tools.

**Comprehensive Experimental Results:** The paper presents comprehensive experiments and results that clearly demonstrate the effectiveness of the proposed method. The evaluations are easy to follow and provide strong empirical evidence for EcoAct's ability to reduce computational costs while maintaining performance, making it easier for readers to understand the benefits of the approach.

**Clear Presentation and Writing:** The paper is well-written, with a clear and logical flow that makes it easy to understand the motivations, methods, and results. The clarity in presenting complex concepts and experimental findings reflects careful attention to detail and greatly enhances the accessibility of the work, making it approachable for a broad audience.
summary: This paper introduces EcoAct, a novel approach designed to enhance tool registration efficiency for Large Language Model (LLM) agents by selectively registering tools during reasoning tasks. Unlike current methods, which pre-register all candidate tools regardless of their necessity, EcoAct integrates tool registration into the reasoning process, optimizing context length and computational costs. The authors demonstrate through extensive experiments that EcoAct reduces computational costs by over 50% while maintaining performance on complex, multi-step reasoning tasks.
weaknesses: **Need for Comparative Experiments Against RAG-Based Methods:** The paper does not clearly demonstrate how EcoAct performs in comparison to Retrieval-Augmented Generation (RAG)-based tool selection methods. In RAG-based approaches, relevant tool candidates are retrieved by computing similarity measures—such as cosine similarity between semantic embeddings—between the user's query and tool representations stored in a vectorized retrieval index. While the paper provides a solid experimental evaluation, it is crucial for the authors to present comparative experiments against RAG-based methods. Such methods could potentially reduce token usage, thereby achieving benefits similar to EcoAct in terms of efficiency and cost savings. A direct comparison would help to highlight the unique advantages of EcoAct, as well as any areas where it may fall short compared to RAG-based approaches.

**Dependence on Tool Naming for Effective Selection:** EcoAct's performance is heavily influenced by how tools are named. The approach assumes that tool names are descriptive enough for effective selection, but this assumption may not always hold true. The authors should specify any requirements or conditions for tool names to ensure proper functioning of EcoAct. Additionally, experiments examining different levels of descriptiveness or ambiguity in tool names would help clarify the robustness of EcoAct under varying conditions.

**Lack of Novelty and Details:** Even though the effectiveness of the proposed method in reducing token costs is evident, the approach of simply adding a tool registration step is too simplistic. The authors have not considered various conditions for ensuring the robustness of EcoAct.

**Lack of Qualitative Analysis of Tool Registrations:** The paper lacks a qualitative analysis of the tool registration process, including notable examples of when and why specific tools are registered. Providing examples would help illustrate the types of tools that are beneficial for registration and shed light on EcoAct's decision-making process, making it easier for readers to understand the practical implications and real-world applicability of the method.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: Q1. Why were no open-source models included in the evaluation, especially given that files named "llama*" and "davinci*" appear in the provided code? Could these models be tested or discussed?

Q2. Is there an analysis of the strengths of the models used and the tools provided? For example, how precisely do tool descriptions and the tool register need to be defined for optimal cost and accuracy?

Q3. Why do models with descriptions perform worse on average in G2 and G3 - Instruction (as shown in Table 4)? Could the authors provide an explanation for this behavior?
soundness: 3
strengths: + The paper is well written, with ideas presented clearly and in an organized manner.
+ The paper introduces a simple yet effective idea for saving tokens when using tools or solving problems, which could have practical implications for reducing costs.
+ The authors provide a thorough ablation study, offering valuable insights into the effects of different components (like Single-Tool vs. Multiple-Tool Registration) on model performance.
summary: The paper introduces EcoAct, an innovative tool-using algorithm designed to enhance the efficiency of large language model agents in multi-step reasoning tasks. Traditional approaches integrate all candidate tools into the LLM’s context, making the process inefficient, as the agent must process unnecessary information. EcoAct addresses this by enabling the agent to selectively register tools as needed during each reasoning step. This method optimizes context length and reduces token usage, significantly lowering computational costs—achieving over 50% cost reduction in some tests—without compromising performance.
weaknesses: Model Selection and Analysis

- The evaluation focuses on two relatively strong models, but no open-source models are included, despite files named “llama*” and “davinci*” appearing in the provided code. This raises the question of why these models were not evaluated or discussed in the paper. Additionally, there is no analysis of model strengths or the tools used. For instance, an examination of how precisely tool descriptions and the tool register need to be defined for optimal cost and accuracy would be valuable. Specifically, exploring the threshold between model size/abilities and the level of detail in tool descriptions could offer insights into balancing model performance with cost. 

Performance Issues and Explanation

- It’s concerning that models with descriptions tend to perform worse on average in G2 and G3 - Instruction (as shown in Table 4), yet no explanation is provided for this behavior.

- The effect of function names on model performance is not addressed. For example, what happens if function names are unintuitive (e.g., def do_or_not(*)) or if there are several similarly named functions (like “calculator” vs. “scientific calculator”)? This ambiguity could impact the model’s accuracy and effectiveness.

Code Quality and Documentation

- The provided code is poorly written, which could make understanding and reproducing the results challenging.

Visual and Presentation Inconsistencies

- The scale in Figure 3 (a) and (b) is inconsistent. Using a common scale across both graphs would allow for a clearer comparison of results.

Missing Evaluation Protocols

- The paper lacks details on the evaluation protocol, such as model temperature settings, maximum tokens, and other relevant parameters. 

Including these analyses would add significant value to the paper, and I would be glad to raise my score if these areas are addressed.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. How much would you agree with my claim that the context of the experiments (containing many possible tools) is not very representative? What do you expect the results to look like in different settings, and why?

2. The URL provided in appendix A is not working for me (it times out with error 522). Would it be possible to confirm that it is working, or fix it if not? I'm especially interested in better understanding the tasks that were used for the evaluation. My current understanding from looking online is that they are mostly pretty simple, with the main challenge being deciding which tool is appropriate, and then making the correct call. If the difficulty of the tasks essentially reduces to choosing the right tool, then I'm less impressed by the results. On the other hand, if the tasks often require multiple steps that build upon each other and the agent must react to the outputs of the tools in nuanced ways, then I'm more impressed by the finding that the agent is fairly good at deciding which tools to use over the course of the task.

3. Did you test alternative methods for reducing the token usage in the agentic workflows in question? How confident are you that simple alternatives cannot outperform your method?

4. This isn't of much consequence, but the claim in section 2.1 that "the size of the tool space is equivalent to the size of the action space" seems false to me. Isn't it the case that many tools can be called in potentially infinite ways (via differences in the parameters), such that the action space is infinite while the tool space is finite?
soundness: 3
strengths: **Originality**

The method introduced by the paper seems original. There exists some work on dynamically managing an agent's tool library, but I'm not aware of this particular approach having been investigated before (this is a bit surprising to me, as the approach is relatively simple).


**Quality**

The paper's experiments are well-motivated and follow naturally from the research question. The ablation experiments are also well-chosen and add flavor to the results. There do not seem to be any methodological issues, and the results clearly support the claims. I don't find any of the empirical claims to be dubious.


**Clarity**
The paper is clearly-written and all its aspects are well-motivated. The metrics and figures are well-chosen and clearly support the claims. The question, method, experiments, and results all follow clearly from each other. When appropriate, fruitful discussions about the mechanisms at play are discussed and add clarity to the text (e.g. section 3.3.2 is interesting and substantially enhanced my understanding of the results).


**Significance**

Agentic workflows with tools are being increasingly adopted, and especially as inference-heavy methods gain more prominence, reducing token usage with no detriment to performance may become increasingly important. That said, I don't think the exact problem this paper focuses is particularly significant (see Weaknesses).
summary: The paper proposes a general method for reducing token usage in LLM agent workflows and assesses its effects on performance and cost. The method involves withholding from the LLM by default the details of the available tools. Instead, apart from the historical context of the task at hand, the LLM receives, at each step, only the names of the available tools, the details of a "tool registering" tool, and the details of previously registered tools (initially, none). Use of a tool must be preceded by a call to the "tool registering" tool in order to register the desired tool, after which the tool becomes available for use. This is a general method, and can be applied to most LLM agent workflows. The experiments are done in the context of ToolBench, and show a substantial decrease in cost with a small effect on performance. The paper also discusses two alternative setups: one where the LLM agent is able to register multiple tools at once (which performs substantially worse than either the vanilla or the single-tool registering setups), and another where the LLM agent also has access to all tools' descriptions, rather than just their names (which does not offer as much reduction in cost).
weaknesses: 1. The paper's experiments are done in the context of ToolBench, where very many tools may be available. This does not seem like a representative or particularly important setting to me. A majority of agentic workflows either take place on more constrained contexts, or use more general tools. In either case, the number of tools available to the agent is much smaller than in the context of the experiments. The authors acknowledge this and helpfully make the distinction in Figure 3, but it remains a significant limitation of the paper.

2. The paper's method is not compared with alternative simpler solutions. An example of a possible alternative would be to begin by asking the LLM to filter the list of available tools given the initial task, and then have it attempt to complete the task with only the selected tools. More complex combinations are possible (e.g. start with a list of filtered tools, but still allow the agent to register new tools). It would be very informative to see how the proposed method compares to such baselines.

3. The previous concern also exists at a more fundamental level: it can be argued that the proper solution for the problem the paper tackles is to make use of prompt caching. The context including the details of all available tools is constant across completion requests, and could be precomputed and reused across calls for very cheap. At the moment of writing this, both the OpenAI and Anthropic APIs support some form of prompt caching. It seems plausible that support for this will become even more widespread and more efficient, so the economic formulation of this work is not very compelling. There is still a surprising result here along the lines of the models being able to exercise good judgement about which tools they need at which times. I was surprised to see that the effect on performance was small. I think a framing of this problem and investigation in terms of the abilities of the language models is not too far away and is possibly very compelling.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: * The paper clearly presents and demonstrates that there is a reduction in computational cost by letting the LLM choose to register a tool rather than being provided with the whole toolset. However, could you clarify whether the tool registration process can be a bottleneck if the LLM needs to iteratively select/register a tool (or a sequential set of tools) in order to achieve the task?
* Also, is there any extra time/steps incurred for registering/ querying the tool?
* Could you also clarify your results in Section 3.4.1 please? Why do using multiple tool registration significantly reduces the performance  vs the React basecase and the EcoAct single tool case? Why your explanation refers to the tendency to registering multiple tools leading to increased complexity, how is it differ to the React basecase when there are multiple tools also being registered?
soundness: 3
strengths: The paper is well-written and has a clear structure. The rationale of the framework and the experiments have been presented in a logical and detailed way. The paper provided both a conceptual description of the problem and practical examples to illustrate how EcoAct could help improve the sequence of thought-action while reducing computational cost. The step proposed is simple and generalisable for most single and multi LLM frameworks. The experimental results show a good potential and support the hypothesis that this modified framework does not affect the performance of ReAct and can improve on computational cost. Overall, the paper demonstrates good quality and clarity.
summary: The paper presents a variant of the ReAct framework by introducing a tool registration process. The rationale is that this would help the agent not having to carry the whole tool registering through the steps and could reduce the computational cost. The paper shows some good experimental results which demonstrate comparable performance and reduced computational cost compared to the ReAct baseline. The step introduced is simple and generalisable for other multi-tool LLM algorithm/framework.
weaknesses: The paper is generally well written and presented but might further improve if it demonstrates more of the following:
* Presentation: Minor typo: "results" - line 386
* Experimental design and data analysis:
    * While the experiments were conducted on ToolBench and captured a variety of tools and subsets (G1, G2 and G3), apart from the general stats of the pass rate, the analysis did not offer a lot of insights on how these subsets differ and why/how the tool selection step of EcoAct helps in each of these subsets. There is some hypothesis proposed in the Overall result section but the paper did not delve further into any analysis to support or provide more evidence on the " method’s ability to address the ”needle-in-a-haystack” problem" (line 317).
    * The results have been largely based on GPT-4o and GPT-4-turbo. The results might be more convincing and comprehensive if there are more baseline for comparison, such as using other LLM models. 
* While the experimental results show good cost improvement compared to the baseline of ReAct, it might need to distinguish the novelty and its computational advantage to other related works that also improve on the efficiency of API calling/token cost of tool information, such as:
    * Wu, Mengsong, Tong Zhu, Han Han, Chuanyuan Tan, Xiang Zhang, and Wenliang Chen. "Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmark." arXiv preprint arXiv:2405.08355 (2024).
    * Wang, Pei, Yanan Wu, Zekun Wang, Jiaheng Liu, Xiaoshuai Song, Zhongyuan Peng, Ken Deng et al. "MTU-Bench: A Multi-granularity Tool-Use Benchmark for Large Language Models." arXiv preprint arXiv:2410.11710 (2024).
    * Wang, Renxi, Xudong Han, Lei Ji, Shu Wang, Timothy Baldwin, and Haonan Li. "ToolGen: Unified Tool Retrieval and Calling via Generation." arXiv preprint arXiv:2410.03439 (2024).

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Please see weakness
soundness: 3
strengths: * Clear writing and good presentation.
* EcoAct is effective and reduce token cost without performance drop.
summary: The paper introduces EcoAct, an algorithm that optimizes the tool registration process for Large Language Models (LLMs) acting as agents. Traditionally, LLMs incorporate all candidate tools into their context, leading to inefficiencies. EcoAct allows LLMs to selectively register tools as needed, reducing computational costs by over 50% in multi-step reasoning tasks without sacrificing performance. It integrates tool registration into the reasoning process, making it universally applicable to LLM agents. Extensive experiments on the ToolBench benchmark demonstrate significant cost savings and maintained performance across various models and reasoning methods.
weaknesses: * In Figure 3a, why gpt4-turbo cost using EcoAct is higher (level 1)? Please explain this.
* Lack of technical novelty. It seems like that EcoAct only a few changes compared to React (from Tabel 5 and 6).

Meta-review: This paper studies an emerging setting in using large language models (LLMs) by allowing LLMs to function as agents that can make use of external tools in solving some tasks. It proposes a method that allows it to focus on only some of the candidate tools that are more likely to be useful and hence makes it more efficient than considering all candidates.

Major strengths:
- Having an efficient and effective way to select the right external tools to use is essential to the agentic approach to using LLMs. The paper studies an important problem in this emerging direction.
- Comprehensive experiments and ablation studies are presented.

Major weaknesses:
- The improvement brought by the proposed method may not be significant when the number of tools is small.
- The incorporation of RAG-based methods is not considered in the paper to demonstrate better how the proposed method can help.
- The effectiveness of the proposed method depends on such things as naming and detailed descriptions of the tools.
- The LLMs included are limited.

Despite its merits, some important issues need to be addressed to make the paper more ready for publication. The authors are encouraged to improve their paper for future submission by considering the comments and suggestions of the reviewers.

Decision: Reject