Venue: NAACL 2025

# Smurfs: Multi-Agent System using Context-Efficient DFSDT for Tool Planning

Junzhi Chen\({}^{*}\), Juhao Liang\({}^{*}\), Benyou Wang\({}^{@}\)

The Chinese University of Hong Kong, Shenzhen

Shenzhen Research Institute of Big Data

wangbenyou@cuhk.edu.cn

These authors contributed equally to this work. �Corresponding author.

###### Abstract

Teaching large language models (LLMs) to use tools for solving complex problems can grant them human-like reasoning abilities. ReAct and its variants are popular frameworks for tool use in both single-agent and multi-agent systems. To address issues like error propagation and limited exploration in ReAct, the Deep First Search Decision Tree (DFSDT) was proposed, but it faces challenges such as rollback instability, redundant context, and premature termination in single-agent settings. We introduce "Smurfs," a novel multi-agent system (MAS) that enhances DFSDT with a modular, context-efficient, and training-free design. Smurfs surpasses baseline methods in both the open-ended StableTooolBench and the closed-ended HotpotQA tasks, reducing token usage by 60.9% compared to DFSDT and enabling Mistral-7b to perform on par with GPT-4-DFSDT. Extensive ablation studies confirm the effectiveness of Smurfs' core components, offering valuable insights for the construction and interpretation of MAS, and paving the way for future exploration. We release the code at [https://github.com/FreedomIntelligence/Smurfs](https://github.com/FreedomIntelligence/Smurfs).

+
Footnote †: _Volume 1: Long Papers)_, pages 3281–3298

+
Footnote †: _Volume 1: Long Papers)_, pages 3281–3298

## 1 Introduction

The ability to manipulate tools for complex tasks has long been considered a distinctive characteristic of humans Oakley and Museum (1972); Ambrose (2001). Can we extend this ability to today's large language models (LLMs), enabling them to utilize multiple tools to perform complex tasks beyond their inherent capabilities? If LLMs can use external tools to access knowledge or execute tasks beyond their fixed language modeling capabilities, we can shift the focus of LLM training towards enhancing their reasoning and tool-use skills. This shift would allow tools to supplement what LLMs should know or execute, thereby improving the parameter efficiency of the LLMs.

In previous multi-agent systems for tool planning, methods like Chain-of-Thought (CoT) Wei et al. (2023), ReAct Yao et al. (2022), and the more advanced DFSDT Qin et al. (2024) have been proposed to enhance LLMs' ability to handle complex multi-step tasks. However, these approaches face notable limitations. ReAct has trouble in handling _error propagation_ and _limited exploration_. DFSDT incorporates a rollback mechanism and depth-first search approach to address limitations of ReAct, but it suffers from _instability_ when the base model struggles with long-context reasoning. It also introduces inefficiencies due to _redundant context_ handling and risks _premature termination_ when solving multi-step problems. These challenges highlight the need for further innovation to better manage context and reasoning complexity in multi-tool planning systems.

In this paper, we introduce _'Smurfs'_, an innovative multi-agent system (MAS) framework inspired by the collaborative and versatile nature of its namesake cartoon characters. The proposed framework leverages enhanced DFSDT to perform complex tool planning tasks without the need for additional training. The effectiveness of Smurfs is demonstrated through both open-ended and closed-ended tool planning benchmark experiments Guo et al. (2024); Yang et al. (2018), where it consis

\begin{table}
\begin{tabular}{l|c c c l} \hline \hline  & Pass Rate \(\uparrow\) & Win Rate \(\downarrow\) & \# of Tokens & \# of Tokens \\  & (g) & (\%) & per request \(\downarrow\) & per query \(\downarrow\) \\ \hline ReAct & \(44.4_{i+1}\) & June & 1,424 & 6,479 \\ DFSDT & \(55.4_{i+2}\) & 60.4 & 1,743 & 20,714 \\
**Smurfs’ (ours)** & \(57.4_{i+1}\) & 62.4 & 459 & 8,096 \\ \hline \end{tabular}
\end{table}
Table 1: Comparison of token cost and performance between tool planning methods over StableToolBench. Existing methods, _ReAct_ and _DFSDT_, have limitations due to high token costs or poor performance. The results are averaged over the subtasks within StableToolBench.



study, followed by a case study, further investigates the reasons behind this effectiveness. These results establish a new state-of-the-art in the field and provide concrete evidence of the advantages of a multi-agent approach in enhancing LLM capabilities.

The contributions of this paper can be summarized as follows:

1. We introduce a highly modular, context-efficient, and training-free MAS framework that utilizes an enhanced DFSDT to improve the tool planning capabilities of LLMs. Experiments demonstrate the effectiveness of this approach, which also proves to be more cost-efficient compared to existing methods.
2. Through ablation studies, we uncover the underlying factors contributing to the effectiveness of the MAS framework, offering valuable insights for future research.

## 2 Related Work

To augment LLMs to do multi-tool planning for solving complex problems, previous work has seen numerous attempts. Chain-of-Thought Wei et al. (2023) is the first to propose the method of thought and answer chain reasoning. ReAct Yao et al. (2022) further introduce the thought-action-observation format for tool chain reasoning, leading to the development of various multi-tool planning methods Chen et al. (2023a); Xu et al. (2023); Shinn et al. (2023). The latest work, Deep First Search Decision Tree (DFSDT) Qin et al. (2024), is proposed to address the inherent limitations of CoT and ReACT: **(1) Error Propagation:** Error occurs at early stage of planning will result in wrong answer in the end, but it can only be identified until reaching the end of the planning chain **(2) Limited Exploration**: Single solution chain can't explore the planning space completely.

DFSDT is powerful in addressing multi-tool planning problems. Its core concept involves employing a depth-first search (DFS) approach for multi-tool planning and backtracks whenever an LLM think the solving process has entered a wrong state (for more details, see Appendix B). When a tool fails or is deemed inadequate for solving the current problem, DFSDT backtracks to the previous solution state and attempts to resolve the issue using another solution plan. However, three limitations are identified with the mechanism of DFSDT:

**Limit I: Instability of the Rollback Mechanism** The rollback mechanism in DFSDT is determined by the model. The number of steps to roll back and the selection of new tools after rollback are guided using prompt containing the errors encountered in the previous failed trajectory. When the model is sufficiently robust, this rollback mechanism serves as a highly flexible and efficient planning strategy. However, when the model's capability is insufficient, it will fail to execute the correct rollback mechanism, i.e. retry the same error tools or roll

Figure 1: Demonstration of the whole process of the Smurfs framework.

 

back too far.

Limit II: Redundant ContextIn the process of planning with DFSDT, each tool plan is generated using the entire conversation history (including all the thoughts, actions, action inputs and tool responses) as context. However, in reality, each step of tool planning only requires a very small portion of the relevant history for effective planning.

The context redundancy not only increases computational overhead but also reduces the accuracy of model inference due to the inclusion of irrelevant historical data. As highlighted by Liu et al. (2024), redundant context become particularly noticeable in tasks requiring assimilation and processing of large inputs, like verbose tool documents and API responses. The situation worsens when LLMs are supplemented with external information, such as document retrieval or online searching Petroni et al. (2020); Ram et al. (2023); Mallen et al. (2022). Although numerous language models capable of handling larger contexts are emerging Dai et al. (2019); Dao et al. (2022), they often face significant performance degradation when the important information is located at some positions Liu et al. (2024); Shi et al. (2023), which is known as the '_lost-in-the-middle_' problem.

Limit III: Premature TerminationThe termination mechanism set by DFSDT involves adding a termination tool to the model's selectable toolkit. When the model selects this termination tool, DFSDT stops and provides an answer. However, in practical applications, this mechanism often prematurely terminates when dealing with complex problems requiring multi-step reasoning. We hypothesize that this issue arises due to the redundant interference of other tool information and history information, which disrupts the model's ability to judge whether the original problem should be terminated. Instead, the model focuses on whether the current sub-problem requires termination, leading the mechanism to terminate after resolving the sub-problem.

In conclusion, DFSDT relies highly on the base model's reasoning ability, especially long context reasoning ability to make roll back decision, termination decision and tool choice decision at the same time, which is a very difficult task even with the most powerful LLM like GPT4 Yuan et al. (2024).

Multi-Agent SystemTo address the limitations inherent in DFSDT and to further enhance LLM's multi-tool planning capabilities, MAS has emerged as a natural solution. Inspired by human social division of labor and cooperation, MAS aim to enable AI agents to accomplish more complex tasks through the division of labor and collaboration. By decomposing the task of DFSDT to multiple agents and giving them only the information they need, we can enable LLMs to use DFSDT more effectively and more efficiently.

Previous work on multi-agent system mainly focus on coding and society simulation area Hong et al. (2023); Qian et al. (2024); Park et al. (2023); Li et al. (2023); Wu et al. (2023). For tool-use scenario, most multi-agent systems still use the ReAct style reasoning Qiao et al. (2024); Chen et al. (2023); Yuan et al. (2024); Liu et al. (2023); Song et al. (2023); Yin et al. (2024); Xu et al. (2023), only using multi-agent discussion and revision to increase reasoning quality, which still inherits the limitation of ReAct. Therefore, this paper aims to construct a novel MAS framework to address the aforementioned limitations. Table 2 shows the difference between different tool-use systems. More detailed discussion can be seen in Appendix A.

Token CompressionToken compression refers to compressing tokens fed into LLMs while preserving inference performance, thus reducing computational overhead and mitigating the constraints imposed by long context limits. Previous works Mu et al. (2024); Fu et al. (2024) have explored token compression techniques, with a focus on compressing token embeddings. In contrast, Smurfs achieves context compression by filtering the input context for each tool planning process. Table 6 shows the detailed difference between different compression techniques.

In the future, these compression methods can be used together in multi-tool planning scenarios to achieve more efficient token compression. Each agent's system prompt could be compressed using gist tokens Mu et al. (2024), tool descriptions

\begin{table}
\begin{tabular}{|p{56.9pt} p{87.1pt} p{87.1pt} p{87.1pt} p{87.1pt} p{87.1pt}|} \hline
**Model Model** & **Mnits-topal** & **Training** & **Compressing** & **Reactions** & **Planning** \\ \hline RoRoAttoriori (RSM) & \(\mathbf{\mu}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & baseline \\ Goadsson et al. (2023) & \(\mathbf{\tau}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & Global \\ Heegaard Citynet (Shi et al., 2013) & \(\mathbf{\tau}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & Global \\ HoSAL (Gan et al., 2013) & \(\mathbf{\tau}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & Baseline \\ Feader et al. (2016) & \(\mathbf{\tau}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & Baseline \\ PiSPDT (Ora et al., 2016) & \(\mathbf{\tau}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & Baseline \\ Flaxpert et al. (2016) & \(\mathbf{\tau}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & Baseline \\ Kim et al. (2016) & \(\mathbf{\tau}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & Baseline \\ Gansafara et al. (2016) & \(\mathbf{\tau}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & \(\mathbf{\tau}^{\prime}\) & Baseline \\ Kim et al.

 

could be managed with CAMPHOR's compression approach Fu et al. (2024), and the context input by each agent could be compressed using Smurfs.

## 3 Smurfs: MAS with Context Efficient DFSDT

_The Smurfs, the beloved cartoon characters, symbolize unity and resourcefulness, and are good at using tools to overcome any challenge they encounter._

### Framework Overview

Figure 1 illustrates the entire workflow for the Smurfs framework. Initially, the **Planning Agent** identifies the user's complex query and breaks it down into manageable sub-tasks. **Executor Agents** are then tasked with collecting task specific information, utilizing access to external tools. **Answer Agent** compiles the findings into a cohesive response, which is subsequently verified by the **Verifier Agent** to ensure accuracy and relevance.

By dividing tasks among different agents, each agent can focus on a specific part of the task, accessing only the necessary history as context during task execution, which effectively addresses the issue of **redundant context**. The redesign of the rollback mechanism to incorporate memory and tool list rollback mechanisms addresses the **instability of the rollback mechanism**. Drawing on the concept of least-to-most prompting Zhou et al. (2023), the original problem is first decomposed into sub-problems for macro-level planning. Subsequently, Smurfs is used to solve each sub-problem at the micro-level, with macro-level planning guiding the micro-level planning, thereby resolving the issue of **premature termination**.

In the rest of this section, the mechanism of the system and the functions of each agent will be detailed. More details of memory system can be seen at Appendix C.

### Agent Components

In this section, we introduce the two core components of the Smurfs system:

ToolsThe tool documents about the tools that Smurfs can utilize in the completion of a complex task are denoted as \(D={\left\{n_{i},d_{i},p_{i}\right\}}^{|d|}_{i=1}\), where n represents the tool name, d represents tool usage description, p represents parameter description and \(|d|\) represents the amount of the available tools. The available tool list is denoted as \(\tau={\left\{n_{i},d_{i}\right\}}^{|\tau|}_{i=1}\). \(\tau_{t}\) denotes the tool list Smurfs can utilize at time t.

MemoryThe memory of the agent system at time t is the history of the task-solving process before t, denoted as \(M=(m_{1},m_{2},...,m_{t-1})\) and \(m_{i}=(\gamma_{i},a_{i})\), where \(m_{i}\) represents memory element at time i and \(\gamma_{i}\), \(a_{i}\) represents thought and answer generated by the system at time i. There are two types of memory in Smurfs: local memory and global memory. the local memory is used to record the ongoing solution trajectory and to generate the next action in the current trajectory. The global memory, meanwhile, records all trajectories and is used to generate the sub-problem's answer by combining all trajectory records when the maximum number of retries is exceeded. This local-global combined memory system ensures that the planning of the current solution trajectory is not influenced by the context of erroneous trajectories. It also generates an answer that combines all trajectories when the verifier agent cannot determine task completion within the maximum number of planned steps. This memory system ensures context efficiency during the task-solving process.

### Macro Planning

Planning AgentThe primary responsibility of the Planning Agent is doing macro-level planning through task decomposition to prevent **premature termination**. The inference process of the Planning Agent is:

\[Plan~{}P:(p1,p2,...)=PA(q) \tag{1}\]

Where \(p_{i}\) represents sub-problem of the original query q, PA represents the Planning Agent. After the task decomposition, the agent system will use Executor Agent, Answer Agent an Verifier Agent to solve each sub-problem using DFSDT collaboratively in a sequential order. To utilize the answer of the previous sub-problem when solving subsequent sub-problem, the strategy known as least-to-most prompting Zhou et al. (2023) is used.

### Subtask Solving Process

After introducing the function of plan agent, this section outlines how the agents collaborate to solve sub-tasks, as shown in Figure 2.

Stable RollbackTo address the **instability of the rollback mechanism** in DFSDT, we propose a rollback mechanism based on rules. Whenever 

an error occurs while using a tool \(\tau_{t,i}\) at time t, the tool list at t \(\tau_{t}\) will pop \(\tau_{t,i}\) out and reperform tool selection and tool planning (ensuring that the faulty tool is not selected again). If, at time t, the tool list becomes empty, it signifies that after the system choosing tool \(\tau_{t-1,j}\) at time t-1, no subsequent trajectory can solve the problem. In this case, the agent system will roll back to time t-1, meaning that the local memory M will pop out the memory element \(m_{t-1}\) at time t-1, and the tool list at time t-1 \(\tau_{t-1}\) will pop out too \(\tau_{t-1,j}\). The agent system will then set the time t=t-1 and continue planning. This rule-based rollback mechanism, compared to the original model-based rollback mechanism of DFSDT, is less flexible and might reduce rollback efficiency. However, it is more stable, ensuring the correctness of deep first search and enabling models with weaker capabilities to utilize DFSDT on tool planning.

Executor AgentThe Executor Agent is responsible for choosing and executing the tools to solve the sub-tasks. At each time t, the agent can invoke one tool to tackle the given task:

\[\gamma=EA.gen\_thought(p,M,\tau,h) \tag{2}\] \[\alpha=EA.choose\_tool(p,\gamma,\tau)\] (3) \[\beta=EA.gen\_arguments(p,M,D[\alpha])\] (4) \[r=EA.call\_tool(\alpha,\beta) \tag{5}\]

Where p is the sub-problem from Planning Agent, h is the hint from the Verifier Agent, \(\tau\) is the tool list, M is local memory, \(D[\alpha]\) means the tool document of tool \(\alpha\). The agent, using the ReACT format (Yao et al., 2022) to choose the tool and arguments, then execute the tool. Noticed that each inference process only uses the relevant part from the local memory and tool list to reduce the context redundancy. More detailed information of the Executor Agent can be found in Figure 5.

Answer AgentTo mitigate the performance degradation caused by lengthy contexts, we introduce the Answer Agent role, designed to extract crucial content for each step and sub-problem:

\[Answer:a=AA(q,r,M) \tag{6}\]

Where q is sub-problem from the Planning Agent, r is response from the Executor Agent, M is the local memory (or global memory if max retry reaches). As the 'lost-in-the-middle' theory described in section 1, retaining all information may not always be beneficial, particularly in cases where the solution path is challenging to discern. Therefore, the primary role of the Answer Agent is to succinctly summarize the generated answers and tool responses to maintain the memory efficiency.

Verifier AgentThe Verifier Agent serves as an early-stopping and reflection mechanism, allowing for a balance between effectiveness and efficiency:

\[h,c=VA(q,a) \tag{7}\]

Where q denotes the sub-problems from the Planning Agent, a denotes the answer from the answer

Figure 2: Details of the subtask-solving process of the Smurfs framework. The dotted line represents that the agent can see the memory and the full line stands for operation.

 

agent, h and c denotes hint and check status respectively. If check status generated is 0, that means the Verifier Agent thinks the sub-problem isn't completed, the system will add the thought and answer of this time to the local and global memory, set t=t+1 and continue the inference procedure.If check status is 1, the sub-problems is thought to be solved and the system will start to deal with the next sub-problem.

## 4 Experiments

To evaluate both the effectiveness and efficiency of the Smurfs framework, in this section, we carried out two multi-tool planning tasks: (1) an open-ended task, _StableToolBench_Guo et al. (2024), and (2) a closed-ended task, _HotpotQA_Yang et al. (2018). In addition to these main experiments designed to assess the entire framework, we conducted an ablation studies followed by a case study to test the capabilities of each component within the multi-agent framework and investigate the underlying reasons for its effectiveness.

### Open-ended Task: StableToolBench

StableToolBench is a tool learning benchmark derived from ToolBench Qin et al. (2024), encompassing multi-step tool usage tasks across over 16,000 APIs. The benchmark employs two metrics for evaluation: **(1) Pass Rate** measures the percentage of instructions successfully executed within the allocated budget. **(2) Win Rate** represents the preference selection by a ChatGPT evaluator when presented with two solution paths.

BaselinesFollowing the original paper that introduced the benchmark, we adopt _ReACT_ (\(CoT\)) Wei et al. (2023) and _DFSDT_Youvron et al. (2023) as baseline methods for comparison. Additionally, we include the backbones used in the paper: gpt-3.5-turbo-0613 (GPT-3.5 Turbo) (OpenAI) and gpt-4-turbo-preview (GPT-4 Turbo). To explore the adaptability of the tool-planning methods, we also include Mistral-7B-Instruct-v0.2 (Mistral-7B) Jiang et al. (2023) as one of the selected backbones in our experiments.

SettingsTo minimize the influence of varying tool APIs on experimental results, we conducted all experiments using the same API cache Guo et al. (2024). For a fair comparison among the candidate methods and to reduce variability, each model was executed once and evaluated three times, with results averaged. Other settings follow those specified in the original benchmark paper.

ResultsTable 3 displays the results on StableToolBench. For the untrained LLM, Mistral-7B, existing agent frameworks did not improve its performance in tool planning tasks; Mistral-7B failed these tasks when integrated with the ReACT and DFSDT frameworks *. However, Smurfs exhibited exceptional performance: when combined with Mistral-7B, Smurfs achieved competitive scores among the baselines. Through its task decomposition mechanism, Smurfs transforms long-context tasks into simpler ones, enabling the untrained model to effectively utilize external tools for managing complex tasks. Regarding closed-source models, specifically GPT4 in these experiments, Smurfs also demonstrated outstanding performance on the benchmark compared to other agent frameworks and achieved state-of-the-art results on the benchmark. Its high success rate suggests that Smurfs is more effective at finding optimal solution paths compared to ChatGPT.

Footnote †: Experiment results show that Mistral-7B failed to correctly execute the ‘finish’ action during inference, resulting in invalid responses.

\begin{table}
\begin{tabular}{l|c c c c c c c c c c c c c c} \hline \multirow{2}{*}{**Backbone**} & \multicolumn{3}{c|}{**Method**} & \multicolumn{3}{c|}{**II-Inst.**} & \multicolumn{3}{c|}{**II-Cat.**} & \multicolumn{3}{c|}{**II-Tech.**} & \multicolumn{3}{c}{**BL-Cast.**} \\  & & Pass & Win & Pass & Win & Pass & Win & Pass & Win & Pass & Win & Pass & Win \\ \hline GPT-3.5 Turbo & RACT & 41.6\({}^{\circ}\)\({}^{\dagger}\)\({}^{\dagger}\) & \({}^{\dagger}\)\({}^{\dagger}\)\({}^{\dagger}\) & \({}^{\dagger}\)\({}^{\dagger}\)\({}^{\dagger}\)\({}^{\dagger}\) & 52.5\({}^{\circ}\)\({}^{\dagger}\)\({}^{\dagger}\)\({}^{\dagger}\) & 52.2\({}^{\circ}\)\({}^{\dagger}\)\({}^{\dagger}\) & 31.6\({}^{\circ}\)\({}^{\dagger}\)\({}^{\dagger}\) & 39.9\({}^{\circ}\)\({}^{\dagger}\)\({}^{\dagger}\) & 44.4\({}^{\circ}\)\({}^{\dagger}\)\({}^{\dagger}\) & / \\ GPF-3.5 Turbo & PDSTD & 54.1\({}^{\circ}\)\({}^{\dagger}\)\({}^{\dagger}\)\({}^{\dagger}\) & 64.1\({}^{\circ}\)\({}^{\dagger}\)\({}^{\dagger}\) & 59.5\({}^{\circ}\)\({}^{\dagger}\)\({}^{\dagger}\) & 54.8\({}^{\circ}\)\({}^{\dagger}\)\({}^{\dagger}\) & 65.9\({}^{\circ}\)\({}^{\dagger}\)\({}^{\dagger}\) & 62.9\({}^{\circ}\)\({}^{\dagger}\)\({}^{\dagger}\) & 64.0\({}^{\circ}\)\({}^{\dagger}\)\({}^{\dagger}\)\({}^{\dagger}\)\({}^{\dagger}\) & 54.5\({}^{\circ}\)\({}^{\dagger}\)\({}^{\dagger}\) & 64.4\({}^{\circ}\)\({}^{\dagger}\) & 64.0\({}^{\circ}\)\({}^{\dagger}\) \\ GFF3.5 Turbo & Samarts & 60.3\({}^{\circ}\)\({}^{\dagger}\)\({}^{\dagger}\)\({}^{\dagger}\)\({}^{\dagger}\)\({}^{\dagger}\) & 63.0\({}^{\circ}\)\({}^{\dagger}\)\ 

Further AnalysisWe conducted a detailed analysis of the token costs associated with each tool planning method for the tasks, a critical evaluation aspect for multihop reasoning tasks. As shown in Table 1 (detailed in Appendix G), the average token costs per question and API request are evaluated for ReACT, DFSDT, and Smurfs on StableToBlench. The analysis reveals that DFSDT generally requires about 20,000 tokens per question, encompassing both prompt and completion tokens. This is nearly three times the token cost compared to ReACT and twice as much as Smurfs. Despite this higher token cost, DFSDT does not demonstrate commensurate effectiveness improvements over other methods. These findings underscore the cost-efficiency of the proposed MAS framework, Smurfs, which not only reduces token expenditure in solving multihop planning tasks but also delivers outstanding performance in evaluations.

### Closed-ended Task: HotpotQA

Compared to open-ended tasks, closed-ended tasks provide a more stable and robust evaluation. To this end, we evaluate the methods on HotpotQA Yang et al. (2018) in addition to StableToBlench. HotpotQA is a multi-hop QA task that is challenging due to the requirement for rich background knowledge, with answers typically being short entities or yes/no responses.

BaselinesThe compared baselines include **CoT**Wei et al. (2023), **ReAct**Yao et al. (2022), **Chameleon**Lu et al. (2023), **Reflexion**Shinn et al. (2023), **BOLAA**Liu et al. (2023), **ReWOO**Xu et al. (2023), **FireAct**Chen et al. (2023a), **AutoAct**Qiao et al. (2024).

Settings and MetricsFollowing the settings in Qiao et al. (2024), we use open-source Llamama-2 models Touvron et al. (2023) and Mistral-7B Jiang et al. (2023) as the backbones of each agent to evaluate the performance of Smurfs. The evaluation metrics is \({\tt reward}\in[0,1]\), defined as the F1 score grading between the prediction and ground-truth answer. For more details about the experiment, see Appendix D.

ResultsSmurfs, as an untrained MAS system, not only comprehensively outperforms untreated agents but also achieves and even surpasses the accuracy of trained agents across most backbone models. This sufficiently demonstrates that the mechanism of smurfs ensures strong generalization capabilities while maintaining high effectiveness.

Observations indicate that the performance of LLama-2-13b-chat on smurfs-related tasks is suboptimal, likely due to its limited capabilities in tool arguments generation. Specifically, the primary issue identified is that, when the Executor agent successfully selects relevant tool, it tends to produce hallucination arguments that can't be used by the tools. This indicates that LLama-2-13b-chat may need further training for usage of tools. The experiment results may substantiate this viewpoint, demonstrating that the untrained methods of llama-2-13b-chat generally exhibit significantly lower accuracy compared to the trained methods. To provide further insight into the model's performance in the tool planning process, we manually categorized the types of errors made by Smurfs on the hotpotQA hard dataset in appendix E.

## 5 Ablation Study

### Importance of each component in MAS

We performed an ablation study to investigate the impact of each agent in our framework. We removed each agent individually, except for the indiis 

pensable Executor Agent, and compared the results to the complete framework.

**Settings (1) Verifier Agent Removal:** Without Verifier Agent, the framework uses a generatl DFSDT in each subtask, i.e. including a finish tool into the tool library and deciding whether to stop at tool choice step. **(2) Answer Agent Removal:** Without Answer Agent, the framework uses full tool response instead of the summary of tool response in its memory. **(3) Planning Agent Removal:** Without Planning Agent, the framework uses Verifier Agent to decide whether the task is completed. **(4) Planning & Verifier Agent Removal:** Without Planning and Verifier Agent, the only difference between Smurfs and DFSDT will be the Answer Agent, i.e. including a finish tool in the tool library, deciding whether to stop at tool choice step and using summary of tool response in its memory.

ResultsTable 5 shows the experiment result, highlighting several key insights regarding the impact of different components in the multi-agent system.

First, the removal of any component generally reduces the win rate, highlighting each component's significance. Second, performance degradation trends remain consistent: Eliminating the Answer Agent has minimal impact, whereas removing the Planning Agent causes the greatest decline. However, this does not imply the relative importance of these agents but rather suggests that the Verifier-Planning Agent workflow is more robust than the Verifier-Answer Agent workflow.

Third, the performance impact varies by model capability. For GPT-3.5, the removal of the Answer Agent reduces both the pass and the win rates, while for GPT-4, it preserves the win rate and im

Figure 3: The illustration of how GPT4-Smurfs and GPT4-DFSDT solve long context problem. The two subquestions and their corresponding answers are marked in two colors.

\begin{table}
\begin{tabular}{c|c c} \hline \hline  & \multicolumn{2}{c}{I3-Inst.} \\  & Pass (\%) & Win (\(\%\)) \\ \hline GPT-3.5 Turbo _with Smurfs_ & \(60.1_{\pm 1.0}\) & 57.4 \\ _w/o Answer Agent_ & \(57.4_{\pm 2.9}\) & \(49.2\) \\ _w/o Verifier Agent_ & \(54.1_{\pm 2.7}\) & \(42.6\) \\ _w/o Planning Agent_ & \(35.5_{\pm 3.3}\) & \(42.6\) \\ _w/o Planning \& Verifier Agent_ & \(58.5_{\pm 2.0}\) & \(57.4\) \\ \hline GPT-4 Turbo _with Smurfs_ & \(70.5_{\pm 1.0}\) & 72.1 \\ _w/o Answer Agent_ & \(82.2_{\pm 2.5}\) & \(72.1\) \\ _w/o Verifier Agent_ & \(79.2_{\pm 0.8}\) & \(63.9\) \\ _w/o Planning Agent_ & \(71.9_{\pm 2.8}\) & \(63.9\) \\ _w/o Planning \& Verifier Agent_ & \(79.8_{\pm 2.8}\) & \(67.2\) \\ \hline \hline \end{tabular}
\end{table}
Table 5: Ablation study on Stable ToolBench I3-Inst subset to investigate the importance of each component within the framework.

 

proves the pass rate, probably due to the superior long-text reasoning ability of GPT-4, which processes additional tool response more effectively. Removal of the Planning Agent significantly reduces the pass rate of GPT-3.5 but has a minimal effect on GPT-4, suggesting that the GPT-4 Verifier Agent is more robust. **These findings may indicate that more powerful models can compensate for agent removal, sometimes even benefiting from additional context, while weaker models benefit more from complex workflow and compressed context.**

We also noticed an interesting phenomenon: when only the Planning Agent is removed, the system experiences a major decline in performance, while removing the Verifier Agent in addition to the Planning Agent improves performance again. This may indicate that the Verifier Agent's ability to pause tasks or provide next-step guidance is suboptimal and requires further training.

In conclusion, our findings offer some insight into the relationship between model capabilities and multi-agent system performance. The optimal number of agents and workflow may vary depending on the capacity of the model. **We propose the hypothesis that weaker models benefit more from complex multi-agent systems and context segmentation, while stronger models perform better with comprehensive context and simpler agent systems.** To validate this hypothesis, more extensive ablation studies are needed under a wider range of models and constraints to explore the influence of different context structures and reasoning workflows. We leave this investigation for future work.

### Case Study

As shown in Figure 3, although GPT4-DFSDT and GPT4-Smurfs use the same tool calls to solve the problem, GPT4-DFSDT only answers the first sub-question correctly while GPT4-Smurfs answers both sub-questions accurately. In the process of addressing the second sub-question, it is notable that the tool response only mentions titles of film and television products related to "Star Wars", without addressing OTT platforms. GPT-4-DFSDT erroneously interprets these titles as responses to the question, while GPT-4-Smurfs adeptly identifies this discrepancy and provides a more appropriate response. This case highlights that in situations where tool responses are lengthy and questions are complex, the single agent framework like DFSDT may be susceptible to distractions from irrelevant information, leading to erroneous answers. Conversely, the context-efficient Smurfs framework demonstrates a reduced susceptibility to irrelevant information, thereby generating more accurate answers.

## 6 Conclusion

In this paper, we propose _Smurfs_, an innovative MAS framework designed to enhance the tool-planning capabilities of LLMs without requiring additional training. Through extensive experiments on both open-ended and closed-ended tool planning benchmarks, Smurfs demonstrate its effectiveness by consistently outperforming baseline methods. Ablation study further provides deeper insights into the impact of each agent in our framework. Based on the ablation study, we propose a hypothesis that can be further tested through more comprehensive experiments in future research: weaker models benefit more from complex multi-agent systems and context segmentation, while stronger models perform better with comprehensive context and simpler agent systems. The findings not only advance the state-of-the-art in multi-tool planning systems but also highlight the potential of modular, training-free frameworks for LLMs in various practical applications.

Looking forward, future research could focus on exploring Smurfs' use in new domains, such as facilitating the synthesis of high-quality multi-tool planning data and enhancing the base model's reasoning and tool-use abilities, further advancing the field of adaptive AI systems.

### Limitations

Model Size Constraints: Due to computational constraints, our experiments did not include larger and more diverse types of LLMs. We believe this would not affect the main observations of this paper.

Agent Component Scale-Up: Although we selected the most common and intuitive agent roles for the proposed MAS, there are many possibilities for researchers to explore. Investigating more well-designed agent roles may help improve the effectiveness of the agent system, and developing automated methods to identify these roles could facilitate effective scaling.

Acknowledging these limitations, future research should aim to address these gaps to provide 

a more comprehensive understanding of the Smurfs framework's capabilities and potential areas for improvement.

## Acknowledgements

This work was supported by the Shenzhen Science and Technology Program (JCYJ20220818103001002), Shenzhen Doctoral Startup Funding (RCBS20221008093330065), Tianyuan Fund for Mathematics of National Natural Science Foundation of China (NSFC) (12326608), Shenzhen Key Laboratory of Cross-Modal Cognitive Computing (grant number ZDSYS20230626091302006), and Shenzhen Stability Science Program 2023.

## References

* A. M. Barnett, J. R. C. Russell, D. E. P. K. Campbell, S. L. B. G. Williams, and T. W. H. Freeman (2007)The role of a new class of deep learning algorithms in neural networks. In Advances in Neural Information Processing Systems, pp. 421-430. Cited by: SS1.
* B. Chen, C. Shu, E. Shareghi, N. Collier, K. Narasimhan, and S. Yao (2023)FireAct: toward language agent fine-tuning. PreprintarXiv:2310.05915. External Links: Link Cited by: SS1.
* W. Chen, Y. Su, J. Zuo, C. Yang, C. Yuan, C. Chan, H. Yu, Y. Lu, Y. Hung, C. Qian, Y. Qin, X. Cong, R. Xie, Z. Liu, M. Sun, and J. Zhou (2023)Agentverse: facilitating multi-agent collaboration and exploring emergent behaviors. PreprintarXiv:2308.10848. Cited by: SS1.
* Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, and R. Salakhutdinov (2019)Transformer-xl: attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860. Cited by: SS1.
* T. Doo, D. Fu, S. Ermon, A. Rudra, and C. Re (2022)Flashattention: fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems35, pp. 16344-16359. External Links: ISSN 1053-649X, Document, Link Cited by: SS1.
* Y. Fu, R. Anantha, and J. Cheng (2024)Camphor: collaborative agents for multi-input planning and high-order reasoning on device. PreprintarXiv:2410.09407. Cited by: SS1.
* Z. Guo, S. Cheng, H. Wang, S. Liang, Y. Qin, P. Li, Z. Liu, M. Sun, and Y. Liu (2024)Stabletoolbench: towards stable large-scale benchmarking on tool learning of large language models. PreprintarXiv:2403.07714. Cited by: SS1.
* S. Hong, X. Zheng, J. Chen, Y. Cheng, J. Wang, C. Zhang, Z. Wang, S. Ka Shing Yau, Z. Lin, L. Zhou, et al. (2023)MetagPT: meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352. Cited by: SS1.
* A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. Singh Chaplot, D. de las Casas, F. Bressand, G. Lempyel, G. Lample, L. Saulnier, et al. (2023)Mistral 7b. arXiv preprint arXiv:2310.06825. Cited by: SS1.
* Guohao Li, Hasan Abed Al Kader Hammoud, H. Itani, D. Khizbulkin, and Bernard Ghanem (2023)Camel: communicative agents for "mind" exploration of large language model society. PreprintarXiv:2303.17760. Cited by: SS1.
* N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang (2024)Lost in the middle: how language models use long contexts. Transactions of the Association for Computational Linguistics12, pp. 157-173. Cited by: SS1.
* Z. Liu, W. Yao, J. Zhang, L. Xue, S. Heinecke, R. Murthy, Y. Feng, Z. Chen, J. Carlos Niebles, D. Arpit, R. Xu, P. Mui, H. Wang, C. Xiong, and S. Savarese (2023)Bolaa: Benchmarking and orchestrating llm-augmented autonomous agents. PreprintarXiv:2308.05960. Cited by: SS1.
* P. Lu, B. Peng, H. Cheng, M. Galley, K. Chang, Y. N. Wu, S. Zhu, and J. Gao (2023)Chameleon: plug-and-play compositional reasoning with large language models. PreprintarXiv:2304.09842. Cited by: SS1.
* A. Mallen, A. Asai, V. Zhong, R. Das, D. Khashabi, and H. Hajishirzi (2022)When not to trust language models: investigating effectiveness of parametric and non-parametric memories. arXiv preprint arXiv:2212.10511. Cited by: SS1.
* J. Mu, X. Lisa Li, and N. Goodman (2024)Learning to compress prompts with gist tokens. PreprintarXiv:2304.08467. Cited by: SS1.
* K. Page Oakley and L. Britsh Museum (1972)Man the tool-maker.. Technical report 538. British Museum (Natural History) London. Note: [https://openai.com/blog/chatgtpt](https://openai.com/blog/chatgtpt) Cited by: SS1.
* O. Park, J. C. O'Brien, C. J. Cai, M. Monfroy, P. Liang, and M. S. Bernstein (2023)Generative agents: interactive simulacar of human behavior. PreprintarXiv:2304.03442. Cited by: SS1.
* F. Petroni, P. Lewis, A. Picktus, T. Rocktaschel, Y. Wu, A. H. Miller, and S. Riedel (2020)How context affects language models' factual predictions. arXiv preprint arXiv:2005.04611. Cited by: SS1.
* C. Qian, W. Liu, H. Liu, N. Chen, Y. Dong, J. Li, C. Yang, W. Chen, Y. Su, X. Cong, J. Xu, D. Li, Z. Liu, and M. Sun (2024)Chatdev: communicative agents for software development. PreprintarXiv:2307.07924.

 

Shuofei Qiao, Ningyu Zhang, Runnan Fang, Yujie Luo, Wangchunshu Zhou, Yuchen Eleanor Jiang, Chengfei Lv, and Huaiyun Chen. 2024. Autoact: Automatic agent learning from scratch for qa via self-planning. _Preprint_, arXiv:2401.05268.
* Qin et al. (2024) Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxil Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gersteinta, dahai li, Zhiyuan Liu, and Maosong Sun. 2024. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. In _The Twelfth International Conference on Learning Representations_.
* Ram et al. (2023) Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. _Transactions of the Association for Computational Linguistics_, 11:1316-1331.
* Shen et al. (2023) Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugging-grpt: Solving ai tasks with chatgpt and its friends in hugging face. _Preprint_, arXiv:2303.17580.
* Shi et al. (2023) Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Scharli, and Denmy Zhou. 2023. Large language models can be easily distracted by irrelevant context. _Preprint_, arXiv:2302.00093.
* Shinn et al. (2023) Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. _Preprint_, arXiv:2303.11366.
* Song et al. (2023) Yifan Song, Weimin Xiong, Dawei Zhu, Wenhao Wu, Han Qian, Mingbo Song, Hailiang Huang, Cheng Li, Ke Wang, Rong Yao, Ye Tian, and Sujian Li. 2023. Restgpt: Connecting large language models with real-world restful apis. _Preprint_, arXiv:2306.06624.
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. _arXiv preprint arXiv:2302.13971_.
* Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. _Preprint_, arXiv:2201.11903.
* Wu et al. (2023) Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beubin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. 2023. Autogen: Enabling next-gen llm applications via multi-agent conversation. _Preprint_, arXiv:2308.08155.
* Xu et al. (2023) Binfeng Xu, Zhiyuan Peng, Bowen Lei, Subhabrata Mukherjee, Yuchen Liu, and Dongkuan Xu. 2023. Rewoo: Decoupling reasoning from observations for efficient augmented language models. _Preprint_, arXiv:2305.18323.
* Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
* Yao et al. (2022) Shunya Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. _arXiv preprint arXiv:2210.03629_.
* Yin et al. (2024) Da Yin, Faeze Brahman, Abhilsha Ravichander, Khytathi Chandu, Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin. 2024. Agent lumos: Unified and modular training for open-source language agents. _Preprint_, arXiv:2311.05657.
* Yuan et al. (2024) Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Yongliang Shen, Ren Kan, Dongsheng Li, and Deqing Yang. 2024. Easytool: Enhancing llvm-based agents with concise tool instruction. _arXiv preprint arXiv:2401.06201_.
* Zhou et al. (2023) Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. 2023. Least-to-most prompting enables complex reasoning in large language models. _Preprint_, arXiv:2205.10625.

## Appendix A Comparison between Smurfs and existing MAS

In conclusion, Smurfs stands out compared to existing MAS in followig terms:

FlexibilitySmurfs utilize enhanced version of DFSDT, which intergrades global and iterative planning, while existing MAS mainly use ReAct and iterative planning. This make Smurfs more flexible in the planning process.

AdaptabilitySmurfs realize superior performance on multiple different benchmarks, proving the adaptability of Smurfs. Existing MAS like RestGPT, Lumos are tailored for specific downstream task and need additional training to be used in other scenario.

Learning EfficiencySmurfs do not need training, thus have the highest learning efficiency among existing MAS. It only need query and tool documentation from the user 

To further illustrate these points, we conducted a detailed comparison between Smurfs and two well-known Multi-Agent Systems, highlighting their differences and the adjustments required when learning out-of-box tasks.

CamelCAMEL (Li et al., 2023) is a communicative agent framework. It uses role-play technique and inception-prompting technique to achieve autonomous cooperation between agents. CAMEL does not natively support tool use settings. CAMEL is considered to perform poorly when generalizing to new tasks (e.g., on the MATH dataset Wu et al. (2023)).

AutogenAutogen (Wu et al., 2023) is not designed as an agent framework for any specific task scenario. Instead, it provides a multi-agent conversation framework that allows users to customize agent characteristics and complete tasks through discussions among different agents. Autogen addresses various problems not by employing a uniform workflow, but by allowing users to design customized agents and workflows flexibly based on their tasks.

SmurfsSmurfs is designed as a unified workflow specifically for complex multi-tool planning scenarios. For different tasks, Smurfs only needs to adjust the few-shot examples in the agent prompts and provide documentation for the tools applicable to the task, allowing Smurfs to generalize to other task scenarios. Smurfs was initially designed for the Stable-toolbench, which itself encompasses various types of tasks and has access to over 16,000 plugins. HotpotQA was subsequently introduced to evaluate the performance of Smurfs on closed-end tasks. When migrating from Stabletoolbench to HotpotQA, only the few-shot examples in the planning agent prompts were modified, along with the provision of plugin documentation for HotpotQA. The rest of the system continued using the same unified framework as in Stabletoolbench. Experiment results demonstrate that our untrained out-of-box unified framework achieves and even surpasses performance of agent systems specifically trained on HotpotQA, such as Autoact and Fireact, showcasing Smurfs' flexibility in generalization.

## Appendix B Details of DFSDT

DFSDT Qin et al. (2024) gives control to the model to stop and rollback the solution trajectory by using Finish tool, thus addressing limitations of ReACT. Finish tool has two parameters **give answer**: model thinks the task is finished and decide to give answer and **give up and restart**: model thinks current trajectory can't lead to correct answer and decide to rollback.

### Details of the Smurfs

Executor Agent DetailsAs illustrated in Figure 5, given a sub-problem p, Executor Agent first thinks about what to do this time, generates thought \(\gamma\) according to p, local memory M, hint h from the Verifier Agent and tool list at the current step \(\tau\). Then it will choose action \(\alpha\) to complete the subproblem using p, \(\gamma\) and \(\tau\). After that, parameters of \(\alpha\) are generated using p, local memory M and tool description of the action \(D[\alpha]\). Tool is then invoked to complete the task.

Memory DetailsAs illustrated in Figure 4, there are four kinds of memory in Smurfs. Local memory stores thought-answer pairs of the current solution trajectory, while global memory stores all history solution trajectory (including those that is backtracked). Tool list only stores available tools' name

Figure 4: Demonstration of the memory of the Smurfs framework.

 

and its usage description, while tool doc stores all detailed information about the tools including parameters details. Through using different kinds of memory under different circumstances, Smurfs can use DFSDT in a context efficient way.

Restart MechanismEvery time Smurfs generate an intermediate output, a format checker is used to check whether the output is of the expected format. If not, Smurfs will retry the same step until reach retry limit or generate correct format output. This mechanism is used in addition to the rollback mechanism to handle the situation where the system can generate correct content but fail to follow the output format.

## Appendix D Experiment Settings for Hotpot QA

Following settings in [10], which is randomly select 300 dev questions divided into three levels for evaluation, with 100 questions in each level. For tool library that can be used in HotpotQA, see Table 7

## E Error Analysis on HotpotQA

To provide further insight into the model performance in the tool planning process, we manually categorized the types of error made by Smurfs in the hotpotQA hard dataset. Table 8 shows that the most frequent errors committed by mistral-7b Smurfs is tool argument fail, followed by bad planning and answer miss. Smurfs do not make tool choice errors and premature termination errors. This shows that Smurfs actually alleviates the premature termination problem, making the tool choice process more robust. Additionally, we note that a portion of the error samples are false negatives which arise when the generated answers differ in expression from the ground truth but are equivalent in meaning. This highlights potential directions for future improvements in Smurfs.

## F Prompts for multi-agent implementation

Prompts used by each agent and their example outputs are shown in Figure 6 to 12.

## G Token Cost on Stable ToolBench Evaluation

We analyzed the token cost for the StableToolsBench experiments. As shown in Table 9, the total token cost for each subtask within the StableToolBench is compared across three candidate tool-planning methods. The data demonstrates that, across all tasks from easy to hard, DFSDT consistently incurs high token costs, while the other two methods maintain relatively low token costs. This verifies the context-efficiency of the proposed method.

\begin{table}
\begin{tabular}{|p{54.1pt}|p{85.4pt}|p{85.4pt}|p{85.4pt}|} \hline
**Compression Method** & **Applicable Scenario** & **Compressed Object** & **Implementation** \\ \hline Gist Tokens (Mu et al., 2024) & General scenarios & Frequently used system prompts & Training LLM to compress system prompts, reducing token usage. \\ \hline CAMPHOR (Fu et al., 2024) & Tool use scenarios & Tool descriptions & Adopts a similar approach to gist tokens, compressing each tool description into a single token, thus reducing token cost. \\ \hline Smurfs & Complex multi-tool planning scenarios & Input context in tool planing & Operates on multi-tool planning workflows, compressing the context needed for each tool planning process. \\ \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{p{85.4pt}}{} & \multicolumn{1}{p{85.4pt}}{} \\ \end{tabular}
\end{table}
Table 6: Comparison of Token Compression Methods 



\begin{table}
\begin{tabular}{c l p{142.3pt} c} \hline \hline
**Name** & **Definition** & **Usage** \\ \hline BingSearch & BingSearch engine can search for rich knowledge on the internet based on keywords, which can compensate for knowledge fallacy and knowledge outdated. & BingSearch[query], which searches the exact detailed query on the Internet and returns the relevant information to the query. Be specific and precise with your query to increase the chances of getting relevant results. For example, Bingsearch[popular dog breeds in the United States] \\ \hline Retrieve & Retrieve additional background knowledge crucial for tackling complex problems. It is especially beneficial for specialized domains like science and mathematics, providing context for the task & Retrieve[entity], which retrieves the exact entity on Wikipedia and returns the first paragraph if it exists. If not, it will return some similar entities to retrieve. For example, Retrieve[Milhouse] \\ \hline Lookup & A Lookup Tool returns the next sentence containing the target string in the page from the search tool, simulating Ctrl+F functionality on the browser. & Lookup[keyword], which returns the next sentence containing the keyword in the last passage successfully found by Retrieve or BingSearch. For example, Lookup[river]. \\ \hline \hline  & & & \\ \end{tabular}
\end{table}
Table 7: Tool library for HotpotQA.

Figure 5: Details of the executor agent working process

\begin{table}
\begin{tabular}{l|p{86.7pt} p{90.6pt} p{90.6pt} p{90.6pt} p{90.6pt} p{90.6pt} p{90.6pt} p{90.6pt} p{90.6pt} p{90.6pt} p{90.6pt}} \hline \hline
**Backbone** & **Method** & \multicolumn{3}{c|}{**In-Task**} & \multicolumn{3}{c|}{**Toal Orgment Fall**} & \multicolumn{3}{c}{**False/ToolBench**} & \multicolumn{3}{c}{**Limit.**} & \multicolumn{3}{c}{**Limit.**} & \multicolumn{3}{c}{**Limit.**} & \multicolumn{3}{c}{**Average**} \\  & Total & Avg. & Test & Avg. & Test & Avg. & Test & Avg. & Test & Avg. & Test & Avg. \\ \hline GPT+35 Tretve & RaACT & 1,130,304 & 6,198 & 52,46,678 & 5,990 & 1,010,514 & 6,396 & 590,855 & 7,265 & 524,510 & 7,778 & 461,212 & 7,599 & 383,663 & 6,764 \\ GPT+35 Tretve & DeGSP & 5,307,062 & 20,284 & 2,745,657 & 17,948 & 3,152,532 & 9,953,260 & 2,648 & 3,909,386 & 59,220 & 1,590,785 & 25,800 & 2,768,452 & 21,847 \\ \hline OursS3 Tretve & SimCLR & 1,209,048 & 7,217 & 3,193,748 & (11,178) & 1,445,358 & **3,200** & 590,688 & 7,658 & 1,009,362 & 10,3441 & 6,032,088 & 1,932,700 & 94,717 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Error analysis for Smurfs on HotpotQA Hard.





**Prompt:**

You need to decompose a complex user's question into some simple sub-tasks and let the model execute it step by step. Please note that:

1. You should only decompose this complex user's question into some simple sub-tasks which can be executed easily by using a single tool.

2. Each simple subtask should be expressed into natural language.

3. Each subtask should contain the necessary information from the original question and should be complete, explicit and self-consistent.

4. You must ONLY output in a parsible JSON format. An example output looks like:

"Tasks": ["Task 1", "Task 2", ...] J

"" This is the user's question: I'm planning a trip to Turkey and need information about postal codes in Istanbul. Can you provide me with the postal code and district for Istanbul province with plate number 34? Additionally, I would like to know if there are any transit agencies available in Istanbul. Please fetch their names and contact numbers.

Output: "Tasks": ["Find the postal codes and districts for plate number 34 in Istanbul.", "Search for transit agencies and their contact numbers in Istanbul."]

This is the user's question: I recently moved to a new address and I need to update my information. Can you retrieve my address details using the postal code 75094080? Additionally, I would like to know the companies that offer shipping services.

Output: ["Tasks": ["retrieve the address details using the postal code 75094080", "search for companies that offer shipping services to my address"]]

This is the user's question: {question}

Output:

**Example Output:**

["Tasks": ["Determine the postal code and district for Istanbul province with plate number 34.", "Find out if there are any transit agencies in Istanbul.", "Get the names of the transit agencies in Istanbul.", "Obtain the contact numbers for the transit agencies in Istanbul." ]

Figure 6: An example prompt for task decomposition in the framework.





**Example Output:**

['Reason": "To determine the postal code and district for a specific location based on a plate number, we would typically need to access a combination of databases, including vehicle registration databases and postal code databases. Since we do not have direct access to these databases, we will need to call external APIs to retrieve this information. Therefore, the user's question requires an API call.", "Choice": "Yes"]

Figure 8: An example prompt for tool check in the framework.

Figure 7: An example prompt for tool check in the framework.





**Example Agent Action Input:**

Figure 10: An example prompt for action input generation in the framework.

Figure 9: An example prompt for action generation in the framework.





**Prompt:**

You should answer the question based on the response output by the API tool.

Please note that:

1. Answer the question in natural language based on the API response reasonably and effectively.

2. The user cannot directly get API response, so you need to make full use of the response and give the information in the response that can satisfy the user's question in as much detail as possible.

3. Do not output answer that is too long. Output in 3-6 sentences is OK.

This is the user's question:

{question}

 This is the API response:

{call_result}

Output:

**Example Output:**

The postal codes and districts for the plate number 34 in Istanbul include: Adakar district with postal code 34975 for Burgazada area, 34970 for Buyukada area, 34973 for Heyeldiada area and 34977 for Kinalada area. There is also the Arnavutkoy district with postal code 34275 for areas like Anadolu, Arnavutkoy Merkez, Imrahor, Islambey, Mustafa Kemal Pasa, Nenehatun, and Yvuz Selim. Another area in Arnavatkov district is Baklah with postal code 34277.

Figure 11: An example prompt for Answer Agent in the framework.