MetaTool: Facilitating Large Language Models to Master Tools with Meta-task Augmentation

## Abstract

Abstract Utilizing tools with Large Language Models (LLMs) is essential for grounding AI agents in real-world applications. The prevailing approach involves few-shot prompting with demonstrations or fine-tuning with expert annotations. However, mere in-context demonstrations may fail to cover sufficient knowledge for complex tools and tasks. Training on solution paths is also hindered by the high cost of expert annotations and generalizing to new tools. A core challenge of generalizable tool use lies in understanding the “meta”, or fundamental natures of tools that are transferable across tasks, such as causality and constraints. In this paper, we present MetaTool , a novel tool learning methodology designed to generalize across any reusable toolset. Our approach incorporates a self-supervised augmentation technique derived from a series of meta-tasks. This involves predicting masked elements in the tool execution process. The self-supervised procedure enables scalable generation of high-quality QA data, which is handy for supervising tool understanding. By incorporating meta-task data into task-oriented training, our method significantly enhances the performance of open-source LLMs, achieving results comparable to ChatGPT in both tool-based planning and chatting scenarios. Through large-scale instruction tuning, the MetaTool model demonstrates impressive zero-shot generalizability on new tasks.

## 1 Introduction

Distinguished from other species, a unique characteristic of human advanced intelligence is using complex tools, which expands the frontiers neural intelligence can reach. With the advent of powerful foundation models, AI has the potential to solve complex tasks with these external mechanisms. LLMs have been majorly oriented towards either tool-augmented chatbots equipped with retrievers and search engines, or tool-oriented agents (e.g. web navigation Rawles et al. (); Hong et al. (), embodied manipulation Chi et al. ()) that achieve task objectives through tool output Qin et al. (). While the former emphasizes generalizing to various tools, the latter focuses on complex tools and scenarios.

To efficiently integrate LLMs with tools, a mainstream way relies on in-context learning (ICL). The model is provided with the “cookbook” of tools in zero-shot prompting or demonstrations in few-shot prompting Xu et al. (); Paranjape et al. (); Brown et al. (). It may work well on simple tools with frameworks like LangChain Chase (). However, for complex tasks with sophisticated tools, in-context learning is limited that demonstrations can not exhaust all scenarios, and manuals are also limited in length. Ultimately, it’s impractical to expect LLMs to be intelligent enough to master any tool without the experience of using it.
On the other side, training-based methods Qin et al. (); Patil et al. (); Dubey et al. () mainly adopt supervised fine-tuning with annotated expert solutions. Despite the difficulties in scaling up the optimal annotation, supervision with task solutions has limitations. Task-agnostic knowledge of tools can be neglected, which hinders the generalization to diverse scenarios or new tools. Self-play training methods like Toolformer Schick et al. () and TALM Parisi et al. () integrate the inference process with self-supervised tool calling data. Although such manner maintains the generality of tool calling, it’s constrained in question-answering scenarios.

Figure: Figure 1: Paradigm comparison between existing tool learning methods and proposed meta-task augmentation. While the prevailing methods are limited in generalizing to complex scenarios or new tools, MetaTool enables gaining transferable tool understanding from task-agnostic knowledge.
Refer to caption: extracted/5910279/Figure_1.jpg

Empirically, human learners get familiar with tools such as hammers (e.g., for nailing and smashing) before engaging in complex construction. Generalizable tool use should be achieved based on the fundamental understanding of tools themselves that holds stable for different objectives, namely task-agnostic (illustrated in Figure 1). Naturally the formation of tool understanding can be disentangled from the learning of task solving. In this paper, we introduce MetaTool, a general methodology that enables both complex tool mastery and unseen tool generalization on top of task-agnostic tool understanding. We design a set of meta-tasks inquiring about the causality of the toolset as an autonomous system and its functionality as a function. Given a callable toolset (e.g. APIs, functions), meta-task data is constructed in a scalable self-supervised way based on unsupervised or self-play tool executions. Augmenting task-oriented training with meta-task data, LLMs learn to solve problems while deepening the mastery of tools. We conduct experiments on both complex tool-oriented tasks and tool-augmented benchmarks, demonstrating that MetaTool significantly exceeds models trained merely on annotated solutions in both worlds and is competitive with the latest LLMs (ChatGPT) with the size of 8B.
The overall contribution can be summarized in three-folds:

- •
We introduce a new tool learning method that facilitates LLMs to master tools with task-agnostic tool understanding.
- •
We propose an integral set of self-supervised meta-tasks that dissect the tool execution process. Meta-tasks enable expert-free data generation and augmentation across tool-augmented and tool-oriented scenarios.
- •
Extensive evaluation on both tool-oriented tasks and tool-augmented benchmarks demonstrates the effectiveness and generality of MetaTool, narrowing the gap between open-source models and state-of-the-art LLMs.

## 2 Approach

In this section, we first formalize the task of using a close toolset and define 6 general meta-tasks that are key to tool understanding. Then we show how to construct datasets in an integral self-supervised way covering different scenarios. In the end, we describe several schemes to augment tool learning with meta-tasks.

### 2.1 Self-supervised Meta-tasks for Tool Understanding

Problem formalization. A tool-use task can be generally defined as a Markovian tuple $\left\langle\mathcal{S},\mathcal{A},\mathcal{T},g\right\rangle$, where $\mathcal{S},\mathcal{A},\mathcal{T}$ is the state space, action space, and toolset, and $g$ is the goal state of the task. Toolset $\mathcal{T}=\{t\}_{N}$ consists of $N$ tools, each as a state transition function $s^{\prime}=t(s,\theta)$ that formalizes the outcome of state change when feeding the input parameters $\theta$ into the tool. An action $a=\left\langle t,\theta\right\rangle\in\mathcal{A}$ specifies the tool and its input. As an autonomous agent, an LLM should iteratively respond with tool calling and inputs according to the state until it reaches the goal. Broadly, when the tools can not alter any external state, tool output like retrieval results can be regarded as the state, and the desired information is the goal $g$.

We enhance the tool understanding of the model with self-supervised surrogate (pretext) tasks instead of in-context descriptions or demonstrations. Formally, we regard tools as external systems that implement state transition mappings. Tool understanding, therefore, involves comprehending the perception-action process of these systems (referred to as tool execution) and should be generalizable to various task objectives.

Figure: Figure 2: Illustration of developing self-supervised meta-tasks from unsupervised tool execution process.
Refer to caption: extracted/5910279/Figure_2.jpg

Meta-task definition. We begin with single-step tool execution $\mathcal{D}=\{s,a,s^{\prime}\}$, peeling off the task goal $g$ context, which results as unsupervised data. Six surrogate tasks (meta-tasks) are designed based on the dataset $\mathcal{D}$. Basically, the model is required to predict masked elements of the execution process. It aligns with the idea of masked language models such as Bert Devlin (), but in a broader and structural granularity. We define the meta-tasks as below (Figure 2):

- •
Effect: The model predicts the outcome state $P(s^{\prime}|a,s)$ given the initial state and the action.
- •
Decision-making: The model decides a feasible action $P(a|s,s^{\prime})$ given the initial and outcome state.
- •
Reversion: The model deduces the initial state $P(s|a,s^{\prime})$ given the action and the outcome state.
- •
Input Boundary: The model determines whether an action can be successfully executed, namely whether the action falls in the valid action space, given the current state: $P(\mathbbm{1}_{s^{\prime}\neq s}|a,s)$.
- •
Output Boundary: The model determines whether a state can be reached with any action given the current state: $P(\mathbbm{1}_{\exists(t,\theta),s^{\prime}=t(s,\theta)}|s,s^{\prime})$.
- •
Counterfact: The model predicts the new outcome state $P(s^{\prime\prime}|a,s^{\prime},a^{\prime})$ if a new action $a^{\prime}$ were executed given that the current action $a$ results in the current outcome $s^{\prime}$.

Effect, decision-making, reversion meta-tasks emphasize the causality of a tool, regarding the action as the intervention to the state Pearl (); Pearl & Mackenzie () and the outcome as the causal effect is determined by the tool mechanism. On top of that, counterfact task is the composition of reversion and effect, further imagining the outcome altered from the fact in effect task. This meta-task raises higher requirements on counterfactual reasoning Bareinboim et al. (); Zhang & Bareinboim (), an advanced form of causal reasoning that humans use to contemplate ’what if’. Moreover, tools implemented as APIs may receive non-executable inputs and result in ineffective outcomes. Thus the input and output domains are also unique features of a tool as a function. We consider input boundary meta-task that emphasizes the tool affordance that refers to what actions can be executed considering the situation and the precondition. output boundary meta-task emphasizes the functionality of tools, that is, what goals can and cannot be achieved given the current state.

Figure: Figure 3: Two-step approach to construct metaset. We illustrate two examplified processes of both tool-oriented and tool-augmented scenarios, which don’t require any expert annotation. $x_{i}^{D},y_{i}^{D}$ denotes the $i$-th question-answer pair of decision-making meta-task, et cetera.
Refer to caption: extracted/5910279/Figure_3.jpg

### 2.2 Metaset Construction

Based on the definitions, the dataset of meta-tasks (referred to as metaset) is generated as question-answering pairs to maintain the conversational skills of LLMs. To answer the meta-task questions, the trained model needs to understand the toolset mechanisms from the corresponding aspects. Given a set of reusable and callable tools $T$, the metaset $\mathcal{M}=\{x_{n}^{m},y_{n}^{m}\}_{n=1}^{N}$ can be constructed in two steps, as illustrated in Figure 3, where $x_{n}^{m},y_{n}^{m}$ is the $n$-th question-answer pair of meta-task $m$. First, we generate tool execution data $\mathcal{D}$ with the toolset. For a limited amount of tools and state space, stochastic sampling can be applied to initialize state $s\sim P(\mathcal{S})$ and action $a\sim P(\mathcal{A})$, and obtain the tool output $s^{\prime}$. For large toolsets and diverse task scenarios that are hard to enumerate, we incorporate LLMs with self-play or tree search techniques to reduce redundant trials $a,s^{\prime}\sim P(\mathrm{LLM}(s,g))$. Note that the LLM does not need to be proficient in tool tasks, as the execution data $\mathcal{D}$ is irrelevant to the task performance. Non-executable actions also contain valuable knowledge and can be transferred as invalid samples in input boundary meta-task. The tool execution data should be sufficient to cover the various scenarios that may arise during the tasks.

Second, for the $n$-th sample and meta-task $m$, we insert the variables of states and actions into $K$ sets of templates (diversified with GPT-4) to obtain diverse QA data.

$$ $(x_{n}^{m},y_{n}^{m})=\mathrm{Mask}(a_{n},s_{n},s^{\prime}_{n},\Phi_{k}^{m})$ (1) $$

, where $\Phi_{k}^{m}$ is the sampled template for meta-task $m$. Particularly, in tool-augmented scenarios, predicting the output of tools such as QA systems can be impractical. Nonetheless, the retrieval result itself is inherently meaningless (e.g., ’sunny’); however, it gains complete significance when combined with the context of tool calling (e.g., ’the weather in New York is sunny’), as showcased in Figure 3. Thus we modify the context into a more informative state in such scenarios by prompting LLMs $s^{*}_{n}=\mathrm{LLM}(s^{\prime}_{n},a_{n},t)$, which is trivial for most language models. At last, we arrange multiple metaset pairs of the same toolset into multi-turn QA data as the metaset $\mathcal{M}$, in order to maintain multi-turn dialogue capacity.

### 2.3 Tool Learning Augmented with Meta-tasks

By incorporating meta-task data, we explore several approaches to augment the tool learning for the purpose of achieving task objectives: a) In-context learning: We randomly sample several demonstrations of each meta-task and add them to the system prompt to facilitate tool understanding in a training-free manner. Such task-agnostic knowledge includes the interpretation of rules, supplementing the solution demonstrations. b) 2-stage learning: Since we aim to build the model’s tool understanding as the foundation of tool-oriented learning, an intuitive idea is to train the LLM first on the metasets as the surrogate tasks and then on the solution data. In order to maintain the general ability of the model in the first stage, only the parameters of the query and value projection layers of the Transformer are updated instead of full-parameter training. c) Data augmentation: In this approach, we utilize the metaset as the augmented data of conventional instruction tuning methods that the metasets are mixed with solution data and the model is trained uniformly. The model trained on the mixed data is referred to as MetaTool.

## 3 Experiments

In this section, we evaluate our approach in both tool-oriented agent and tool-augmented chatbot scenarios. On the one hand, we fine-tune the LLM to master a specific toolset for achieving various complex objectives. On the other hand, we conduct large-scale instruction tuning to enable the model to generalize to new tasks and understand new tools through zero-shot documentation.

### 3.1 Tool-oriented Scenarios

#### 3.1.1 Task setup

We adopt 3 tool-oriented tasks that emphasize complex tool execution and sequential planning. Among them SAW is newly designed while the other two are introduced from PlanBench Valmeekam et al. (). The key challenge of these tasks is to understand the rules (preconditions) and the environmental dynamics caused by actions. The task definition and dataset construction are elaborated below.

SpellAnyWord (SAW). In this task, the agent needs to sequentially construct a string that contains the target string as a continuous substring. The initial state of the task is a void string. Two non-degradable tools (functions) are avaliable: Add: to add two adjacent letters in the alphabet to the end of the current string. The tool input $\theta$ should be the preceding letter (e.g. passing ’a’ to Add on current string ” will result in ’ab’). Swap: to swap the position of two adjacent letters in the current string. The input should be the preceding letter (e.g. passing ’a’ to Swap on ’ab’ will result in ’ba’). An example task: The target string is ’any’. A successful action sequence can be [Add(’a’), Add(’n’), Add(’y’), Swap(’a’), Add(’o’)], which will result in a state sequence [’ab’, ’abno’, ’abnoyz’, ’banoyz’, ’banyoz’] and the final string ’banyoz’ has ’any’ as a substring. To eliminate the basis from tokenization, we format each string as a list of letters in prompts throughout the task.

BlocksWolrd (BW). In this scenario, the agent needs to stack several blocks on the table into a target state with one hand. Only one block can be moved at a time. Two tools (functions) are avaliable: Pick: to pick a block in the hand. The tool input should be the target block indicated by its color (e.g. Pick(’yellow’)). Blocks cannot be picked if there are blocks on top of them or there’s already a block in the hand. Stack: to stack the block in the hand onto the target block or table. The input should be the color of the target block or ’table’ (e.g. Stack(’white’), Stack(’table’)). Blocks cannot be stacked on a block with another block already on top of it or there’s no block in the hand.

Logistics (LOG). The agent needs to solve a logistics problem by arranging trucks and airplanes to transport the package to the target location. Locations are grouped by cities. Trucks can be used to move packages between locations in the same city and planes can be used to move packages between cities. Two tools (functions) are available: Truck: to transport the truck and the package (if there is any) from one location to another. Plane: to transport the airplane and the package (if there is any) from one location to another. The tool input should be the starting and ending location indicated by numbers. (e.g. Truck(1,2), Plane(2,4)). An action is invalid when there is no truck or airplane at the starting location.

Datasets collection. For the SAW task, we randomly sample 2k target strings (from 2 letters to 10 letters) as task goals. We modify the BlocksWorld and Logistics tasks from PlanBench into the tool-use version, thus 2k goals for each task are adopted following the original configuration. Optimal action sequences are obtained with heuristic strategy as the solution data. For each annotated action, we generate a thought with GPT-generated templates. The thoughts analyze the situation and what to do following ReACT Yao et al. () to leverage the model’s reasoning ability. Thus the solution contains a sequence of thought-tool-input tuples.

#### 3.1.2 Implementation details

Our model is fine-tuned based on LLaMA3-8b-instruct AI@Meta () with parameter-efficient fine-tuning method Qlora Dettmers et al. () on 8 A100 GPUs. We utilize the instruction tuning version of LLaMA3 since comprehending tool-oriented tasks with specific objectives is the basis of tool understanding and use. For each task, we train the model on 10k meta-task data and 10k solution data for 3 epochs with AdamW optimizer and the learning rate of 2e-4. The models are tested in a simulated environment that receives the action of using a tool and returns the outcome and current state. We evaluated the model performance on 100 unseen cases of each three tasks.

**Table 1: Results on tool-oriented tasks. ICL: in-context learning with meta-task demonstrations. ChatGPT and GPT-4 are provided with tool documentation and few-shot demonstrations.**
| Models | SAW | BW | LOG |
| --- | --- | --- | --- |
| ChatGPT | 22.6 | 23.3 | 50.4 |
| ChatGPT-ICL | 20.2 | 20.5 | 43.6 |
| GPT-4 | 28.6 | 43.0 | 46.6 |
| GPT-4-ICL | 27.4 | 40.0 | 37.0 |
| Vicuna-7b | 4.8 | 5.5 | 0.0 |
| LLaMA3-8b-instruct | 6.0 | 6.7 | 6.0 |
| LLaMA3-solution | 9.5 | 19.2 | 8.2 |
| LLaMA3-ICL | 4.8 | 17.8 | 2.0 |
| LLaMA3-2-stage | 9.5 | 21.9 | 12.3 |
| MetaTool | 32.1 | 37.5 | 30.1 |

#### 3.1.3 Results analysis

Overall comparison. We evaluate the success rate (SR%) of completing each task and show the performances of several models in Table 1. Overall, SOTA closed-source LLMs show impressive zero-shot performance on tool-oriented tasks compared with open-source LLMs including LLaMA3 and Vicuna. By training on both meta-tasks and solution data, our model MetaTool gains significant improvement (+20.9%SR on average) compared with mere training on solution data (LLaMA3-solution). MetaTool also surpasses GPT-4/ChatGPT in the SAW/BW tasks (+3.5%/14.2%SR). ChatGPT represents the model of GPT-3.5-turbo throughout our experiments. Both GPT and LLaMA3 show weaker performances when provided with meta-task demonstrations (ICL) since demonstrating limited cases can be redundant or misleading without proper design. LLaMA3-2-stage that trained on meta-tasks first gains limited improvement compared with the baseline. We conjecture that learning meta-tasks without practicing tool use (training on action sequences) cannot effectively facilitate tool-use ability with tool understanding. Also fine-tuning with specific QA data may affect the basic linguistic ability of the model. The overall results show that LLMs (including the most powerful ones like GPT-4) still have difficulties conquering complex tool using tasks, especially in planning with tools.

Ablation study. We study the ablation of different data components and report the performances in Table 2. It’s worth noticing that merely training on meta-tasks can improve the model’s zero-shot performance on tool-oriented tasks (line 7), contrary to providing demonstrations of meta-tasks in the system prompt (LLaMA3-ICL in Table 1). When removing QA data from each meta-task, the model performance shows varying degrees of degradation, which verifies the profits of meta-tasks. The meta-tasks of effect and decision-making have a relatively greater influence on the model’s tool understanding capability. Theoretically, these meta-tasks emphasize the causal mechanism of tools that is more fundamental than others.

### 3.2 Tool-augmented Scenarios

#### 3.2.1 Task setup

Among the various tool/function calling benchmarks, we study our method on two of the most influential ones: ToolBench Qin et al. () and Berkeley Function Calling Leaderboard (BFCL) Yan et al. (). ToolBench contains diverse user requests with a massive amount (over 16k) of real-world API tools, which are publicly available on the RapidAPI website. The testset is categorized into six distinct groups and contains 1200 instructions (200 each): I1-inst., I1-tool, I1-cat., I2-inst, I2-cat., and I3-inst. Groups labeled with I1, I2, I3 include single-tool tasks, intra-category multi-tool tasks, and extra-category multi-tool tasks, respectively. Groups labeled with “inst.”, ”tool”, ”cat.” include unseen user instructions, unseen tools, and unseen categories (e.g. sports, entertainment) of tools, respectively. Two evaluation metrics are designed based on ChatGPT: (1) Pass Rate, calculated by the proportion of instructions successfully completed within a limited budget; (2) Win Rate measured by asking a ChatGPT evaluator to select its preference for two solution paths. For each user instruction (e.g. “Can you recommend some popular restaurants within 5km to hold a party?”), the model calls an API and responds to the query based on the tool output.

ToolBench also provides 126k instruction-solution pairs for training, which are generated with GPT4 and depth-first tree search (DFS). On top of that, we extract unsupervised tool execution data and generate 650k self-supervised data of meta-tasks following the procedure in Figure 3. We then conduct instruction tuning based on the mixed data and LLaMA3-8B-instruct model, trained for 1 epoch to avoid overfitting. Both the solution data and the meta-task data share the same loss setting as we construct the metaset as QA pairs. The context-aware states are generated with the open-sourced LLaMA3-70B-instruct model. All evaluated LLMs are prompted in the ReACT manner to leverage their reasoning ability. Other model implementation details are in line with that described in section 4.1.2.

BFCL benchmark is established mainly for the purpose of zero-shot evaluation without holistic training data. The benchmark contains 4251 testing cases in total and is categorized into non-live (self-designed), live (user-contributed), multi-turn, and Hallucination (relevance or irrelevance determination) groups. The model performance is measured by action accuracy with Abstract Syntax Tree (AST) Patil et al. ().
We test the MetaTool model trained with ToolBench data on BFCL to evaluate the zero-shot ICL ability and generality of our methodology.

**Table 3: ToolBench results. ChaGPT doesn’t have the Win Rate score since all other Win Rates are measured by comparing with its solution paths.**
| Models | I1-Inst. | I1-Tool | I1-Cat. | I2-Inst. | I2-Cat. | I3-Inst. | Averages |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Pass | Win | Pass | Win | Pass | Win | Pass | Win | Pass | Win | Pass | Win | Pass | Win |  |
| ChatGPT | 41.5 | - | 41.0 | - | 41.0 | - | 34.5 | - | 46.5 | - | 22.0 | - | 37.8 | - |
| Claude-2 | 5.5 | 31.0 | 3.5 | 27.8 | 5.5 | 33.8 | 6.0 | 35.0 | 6.0 | 31.5 | 14.0 | 47.5 | 6.8 | 34.4 |
| GPT-4 | 53.5 | 60.0 | 50.0 | 58.8 | 53.5 | 63.5 | 67.0 | 65.8 | 72.0 | 60.3 | 47.0 | 78.0 | 57.2 | 64.4 |
| ToolLLaMA-2 | 25.0 | 45.0 | 29.0 | 42.0 | 33.0 | 47.5 | 30.5 | 50.8 | 31.5 | 41.8 | 25.0 | 55.0 | 29.0 | 47.0 |
| LLaMA3-8B-inst. | 0.0 | 0.0 | 0.0 | 0.0 | 0.1 | 0.0 | 0.0 | 0.0 | 0.1 | 0.1 | 0.0 | 0.0 | 0.0 | 0.0 |
| LLaMA3-solution | 32.1 | 45.3 | 39.0 | 43.9 | 36.4 | 43.0 | 40.1 | 52.5 | 40.1 | 43.4 | 35.6 | 61.8 | 37.2 | 48.3 |
| MetaTool (8B) | 42.5 | 52.1 | 41.8 | 51.3 | 43.3 | 46.1 | 52.0 | 54.9 | 50.0 | 54.0 | 45.5 | 74.5 | 45.9 | 55.5 |

**Table 4: BFCL results. M&P denotes the test set of multiple parallel. Hal., rel., and irrel. represent the relevance and irrelevance set of the hallucination group. Simple Ave. denotes the average accuracy of non-live simple, live simple, and multi-turn for fair comparison.**
| Models | Non-live | Live | Multi Turn | Hal. | Simple |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| simple | multiple | parallel | M&P | simple | multiple | parallel | M&P | base | rel. | irrel. | Ave. |  |
| GPT-4-turbo | 60.6 | 91.0 | 90.0 | 89.0 | 67.8 | 74.5 | 75.0 | 62.5 | 33.5 | 70.7 | 79.8 | 54.0 |
| o1-mini | 68.9 | 89.0 | 73.5 | 70.5 | 62.8 | 65.1 | 68.8 | 58.3 | 16.0 | 46.3 | 88.7 | 49.2 |
| Hermes-2 (8B) | 61.3 | 82.5 | 75.5 | 75.0 | 55.8 | 53.1 | 43.8 | 41.7 | 1.5 | 51.2 | 62.7 | 39.5 |
| LLaMA3-solution | 71.3 | 64.0 | 13.5 | 10.0 | 56.6 | 34.9 | 37.5 | 12.5 | 5.0 | 100.0 | 8.3 | 44.3 |
| MetaTool (8B) | 78.3 | 55.0 | 66.0 | 63.5 | 58.1 | 50.1 | 18.8 | 37.5 | 6.5 | 100.0 | 25.4 | 47.6 |

#### 3.2.2 Results analysis

As the ToolBench results shown in Table 3, MetaTool (8B) achieves the second-best performance across all groups merely behind GPT-4, superior to other models including ChatGPT (+8.1% pass rate) and training-based ToolLLaMA-2 (7B) Qin et al. () (+16.9% pass rate, +8.5% win rate). Especially, our method significantly improves the performance of LLaMA3 (originally incapable) compared to LLaMA3-solution which is merely trained on solution data (+8.7% pass rate, +7.2% win rate). The comprehensive advantages show the effectiveness of meta-task augmentation. Besides, the notable superiority of GPT-4 can be partly attributed to the fact that all the training and testing instructions are generated with itself. Thus GPT-4 may be more familiar with the distribution and inner motivation of these instructions.

Table 4 shows the zero-shot performance on the BFCL benchmark. It’s worth noticing that except for the sets of non-live simple, live simple, and multi-turn base, zero-shot comparison on other test sets is less fair, since MetaTool is merely trained in ToolBench scenarios with a unique task pattern (fixed system prompt) that the model calls a single tool once a time then waits for the tool output. For example, in the ”multiple” tasks LLMs are asked to call multiple tools in one response. Nonetheless, we modify the parser of MetaTool to continue generating tokens to fit the requirements of multiple and parallel scenarios. In the ”irrelevance” tasks, LLMs have access to tools irrelevant to the instruction and should give up calling any tools, which never occurs in ToolBench scenarios. Therefore in the first place, we count the average accuracy (Simple Ave. in Table 4) of three fair sets and witness that MetaTool surpasses LLaMA3-solution (+3.3%) and Hermes-2-theta (+8.1%) which is also trained based on LLaMA3-8B-instruct, and is close to the latest OpenAI o1-mini (-1.6%). It’s also impressive that MetaTool obtains the highest 78.3% accuracy on the non-live simple set, 17.7% higher than the 1st rank model GPT-4-turbo. Despite the transferring barriers, MetaTool still achieves moderate performance on the other sets (e.g. multiple, parallel, hallucination) with an average of 52.0% accuracy, significantly higher than the 35.1% of LLaMA3-solution (+16.9%). Overall, the zero-shot results on BFCL clearly demonstrate the exceptional generalizability of MetaTool. With additional data augmentation from diverse scenarios, MetaTool has the potential for significant improvement.

Figure: Figure 4: Case study of MetaTool compared with 2 baselines on BlocksWorld task. Actions in red denote invalid ones (e.g. pick up a block at the bottom). LLaMA3-solution is the LLaMA model trained on task solution data.
Refer to caption: extracted/5910279/Figure_4.png

### 3.3 Qualitative Case Study

As showcased in Figure 4, the agent is required to construct stacks containing a green block on top of a yellow block from a pile of 4 blocks. With mere descriptions of tools in the prompts, LLaMA3 fails to understand the precondition of using tools resulting in invalid actions. Training on tool-oriented solution data, LLaMA3-solution attempts to lift the yellow block successfully but fails to sequentially achieve the task goal and falls into repetitive loops. The proposed MetaTool model achieves the target state with an effective action sequence (although still not the optimal efficiency) and corresponding reasoning. These 3 models correspond to the 3 paradigms illustrated in Figure 1. The results show that LLMs can learn tool use better on the basis of robust tool understanding.

## 4 Related Works

### 4.1 Tool learning

Recent studies have shed light on the potential of utilizing tools to augment LLMs with external factual knowledge Qin et al. (); Nakano et al. (); Song et al. (); Hao et al. (); Shen et al. (); Gao et al. (); Wu et al. (); Qian et al. (); Zhuang et al. (); Schick et al. () which is targeted at tool-augmented question-answering scenarios, towards the ‘tools for AI’ purpose in general. On the other side, With the burgeoning intelligence in reasoning and perception, LLMs’ tool-use capability can be widely applied in the automation of various domains including Embodied AI Wang et al. (; ), web manipulation Rawles et al. (); Hong et al. (); Yang et al. (); Deng et al. (); He et al. (); Zhou et al. (), and image/video editing Wang et al. (); Argaw et al. (); Hang et al. (); Fu et al. ().This line of work is intended for tool-oriented planning scenarios for the ‘AI for tools’ purpose. Effectively mastering complex tools challenges the model to comprehend the precondition and potential outcome of using tools. In this paper, we aim to facilitate LLMs for both tool-oriented and tool-augmented tasks by learning robust tool understanding.

### 4.2 Tool understanding

As noted by Hernik & Csibra (), when learning to utilize a specific tool, children perceive it as an object with particular functions, engaging in a cognitive process to understand its purpose and operation. Analogously, a comprehensive understanding of the tools’ functionalities is indispensable for enabling the controller to use tools proficiently. In real-world scenarios, tools are typically accompanied by a manual (or tutorial), which provides sufficient
relevant details about their functionalities and usage. Endowed with strong few-shot learning Brown et al. () and zero-shot learning Wei et al. () capabilities, foundation models can be prompted to unravel tools’ functionalities and comprehend how to use them. To this end, we can construct suitable task-specific
prompts either through manual design Vemprala et al. () or retrieval Zhou et al. (). However, prompting is restricted by input context length, thus the situation may be more challenging with multiple complex tools with long descriptions. While most training-based tool learning methods rely on extensive expert-annotated solution data for goal-oriented tasks, the knowledge contained in the tool execution process itself remains unutilized. We propose a self-supervised data augmentation method to efficiently endow LLMs the comprehension of a set of tools.

## 5 Conclusion

In this work, we introduced MetaTool, an efficient and generalizable method that facilitates tool learning with task-agnostic comprehension. This is achieved by deriving self-supervised meta-task data from tool execution actions. Augmented the meta-tasks into either complex toolset fine-tuning or large-scale instruction tuning, our model exhibits sophisticated tool mastery as well as generality in in-context learning. Evaluated on multiple tool use benchmarks, MetaTool outperforms models trained on expert solutions and showcases comparable performance with ChatGPT in a size of 8B.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: See weaknesses
soundness: 3
strengths: 1. A novel meta-learning approach for tool learning showing improved tool-learning results.
2. The ablation study verifies the effectiveness of each meta-task.
summary: This paper proposes to achieve generalizable tool learning by additionally training models on meta-reasoning QA tasks. The meta-reasoning data are constructed by asking questions about the tool-using process in multiple directions, including action effect, decision-making, reversion, action input boundary, etc. Experiment results show improved tool learning performance on tasks including SAW, BW, LOG, Toolbench and BFCL.
weaknesses: 1. In lines 224-226, "In order to maintain the general ability of the model in the first stage, only the parameters of the query and value projection layers of the Transformer are updated instead of full-parameter training." This constraint might also affect learning ability and make comparisons unfair. Results ensuring similar settings will make results more convincing.

2. The "LLaMA3-solution" baselines are updated fewer times (10k*3) compared with other models ((10k + 10k)*3). It would be fairer to test both training with more steps and using more solution data to ensure similar update steps.

3. Lack of 2-stage results on ToolBench and BFCL benchmarks, nor LLaMA3-8B-inst results on BFCL. Would be better to explain the setup more clearly.

4. Authors used the early stop to prevent overfitting, which also creates the possibility for larger variance due to the arbitrary selection of epoch numbers. Any more comprehensive results to eliminate this hyperparameter selection?

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 1
questions: Typo

Line 146: Bert $\rightarrow$ BERT 

Line 208: ',' should follow the math formula

Line 256: BlocksWolrd $\rightarrow$ BlocksWorld

Line 334: ChaGPT $\rightarrow$ ChatGPT 

Line 363: ReACT $\rightarrow$ ReAct 

[1] Devlin, J. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

[2] OpenAI (2022). Introducing ChatGPT. https://openai.com/index/chatgpt.

[3] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., \& Cao, Y. ReAct: Synergizing Reasoning and Acting in Language Models. In The Eleventh International Conference on Learning Representations.

Clarification

Line 352: Two evaluation metrics are designed based on ChatGPT: (1) Pass Rate, calculated
by the proportion of instructions successfully completed within a limited budget; (2)Win Rate,
measured by asking a ChatGPT evaluator to select its preference for two solution paths.

Question: What does 'based on ChatGPT' mean? If you use the Pass Rate, it is an automatic evaluation that does not require ChatGPT.
soundness: 2
strengths: This is a paper with good methodology and solid experiments.

The paper tackles the crucial challenge of enabling LLMs to effectively use tools, a capability essential for expanding their real-world applicability.

The paper demonstrates a good understanding of related LLMs and tool learning work, with a comprehensive list of references.

 The experimental setup and evaluation metrics are well-defined, the ablation studies are comprehensive, and the results demonstrate a clear improvement over baselines trained solely on solution data.

The tool-oriented scenario is an interesting setup. This is a new environment to evaluate LLMs' tool-using capability.
summary: This paper proposes MetaTool, a novel approach for training Large Language Models (LLMs) to use tools effectively. The authors argue that current methods, which rely on demonstrations or expert annotations, need to be revised in generalizing to complex tools and tasks. MetaTool addresses this by introducing a set of self-supervised meta-tasks that focus on the fundamental nature of tools, such as causality and constraints. These meta-tasks are used to generate high-quality training data without human annotation. Through extensive experiments, MetaTool demonstrates superior performance to other LLMs on tool-oriented tasks, achieving results comparable to ChatGPT, and exhibits impressive generalization capabilities on new tasks.
weaknesses: This paper has several weaknesses regarding the writing and the experiment results.

The idea of using self-supervised tasks to improve tool understanding is not entirely new. The authors themselves acknowledge that previous works like Toolformer and TALM have explored similar approaches. The main difference in MetaTool seems to be the specific set of meta-tasks proposed, but their novelty and contribution need further justification.

 The problem formulation and definition are clear. However, I am also curious about the real-world scenarios of these six settings. Are there any specific rationales behind why we should apply this causal inference mechanism to tool use?

 The paper lacks a comprehensive comparison with a wider range of state-of-the-art tool-learning models, including those employing self-supervised learning. This limitation makes it difficult to claim that MetaTool significantly improves over existing approaches [1]. There are more baselines on the Berkeley Function-Calling Leaderboard to compare with. Some of them are trained from LLaMA-3.1-8B as well.

 From what I can read through this paper, the self-play and tree search lack implementation details. This is an interesting way of generating synthetic data. However, the authors did not explain that clearly. For example, the method section should describe using the ReAct to generate thought-tool-input tuples instead of the experiment section.

Some claims or writings in this paper are contradictory or confusing.

 The analogy to BERT isn't that pertinent to me. We are still training the LLMs under an autoregressive objective with LLaMA-3-8B. The model is trained on an augmented/generated dataset using BERT's 'masking' process but is not trained to predict what is missing in the context.

What is the model version of ChatGPT? What about GPT-4 and Claude-2? The authors should specify the model versions more clearly.

 It will be helpful if the authors add $\uparrow$ and $\downarrow$ to show which directions of the metrics are better.

 Line 225 ... are updated instead of full-parameter training. Which part of the training requires full-parameter training? Isn't this model trained using QLoRA?

 The tables and figures are organized confusingly. Table 3 is for tool-augmented scenarios but is placed in Section 3.1. Figure 4 is for tool-oriented scenarios, but the authors separate Section 3.3 for result analysis (only for the results in Section 3.1).

 All the tables and figures should have hyperlinks.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: + Could MetaTool work with tools in more dynamic, unpredictable domains, like finance or autonomous driving, where tool results might vary or need to adapt in real-time?

+ The paper mentions that MetaTool’s data generation is efficient, but how does this scale up if we’re working with very large toolsets? What’s the computational cost?

+ Do the authors plan to add more meta-tasks or tweak existing ones to improve model understanding? Would expanding to include tasks around probabilistic reasoning or continuous learning help MetaTool become even more generalizable?
soundness: 3
strengths: + The approach is unique because it emphasizes a general, task-independent understanding of tools. MetaTool's focus on foundational tool knowledge is different from the usual heavy reliance on labeled data, allowing the model to handle more situations with less specific training.

+ The experiments are thorough, covering both scenarios where the model has to use a tool in sequence (like planning) and where it’s just one part of a conversation or task. MetaTool consistently performs well across the board, and the ablation study is detailed, showing which parts of the model contribute the most to performance.

+ The explanation of the meta-tasks is clear and easy to follow. Each task seems thoughtfully designed to address different aspects of tool use, and the visuals in the paper, like the figures comparing MetaTool with other methods, really help make the results easy to understand.
summary: This paper introduces MetaTool, a new way to help large language models (LLMs) better understand and use tools. Instead of relying on traditional methods, like giving examples in prompts or using labeled training data, MetaTool focuses on self-supervised learning through meta-tasks. The idea is to teach models the basics of how tools work, like understanding cause and effect, what actions are allowed, and what outcomes to expect. They introduce six meta-tasks—Effect, Decision-making, Reversion, Input Boundary, Output Boundary, and Counterfact—that cover these foundational ideas. MetaTool shows impressive results across different tool-based tasks and even competes well against models like ChatGPT in both planning and chat scenarios.
weaknesses: - While the benchmarks are solid, some of them are in simulated environments. This setup might not fully reflect the unpredictability of real-world situations, so testing MetaTool in live, dynamic environments could strengthen the case for its broader applicability.

- The self-supervised meta-task setup is definitely innovative, but it might be tricky for people who want to apply it to new tools or unique domains without more support. Offering some guidance or a framework for adding new tools could make MetaTool easier to use widely.

- Although MetaTool performs well against strong baselines, it would be interesting to see it compared to other recent multi-task or hierarchical learning approaches, which also aim to improve model generalization. This would give a better sense of where MetaTool stands in terms of scalability and flexibility.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: - The authors mentioned, “we modify the context into a more informative state in such scenarios by prompting LLMs s ∗ n = LLM(s ′ n , an, t), which is trivial for most language models.” Will the complexity of this generated instruction have an impact on the model performance? It would be helpful to see a few qualitative examples of this "context generation" (e.g., prompt input, output).
soundness: 3
strengths: - The data generation process is scalable: The paper uses self-supervised techniques to generate training data for tool use via a list of meta-tasks without requiring expert annotations. The meta-task framework allows for the automatic generation of diverse training examples that cover various aspects of tool understanding.

- Reasonable Meta-Task Design: The six meta-tasks (Effect, Decision-making, Reversion, Input Boundary, Output Boundary, and Counterfact) are well-designed to capture different aspects of tool understanding and/or action execution, which seems generalizable.

- Good empirical results: The experimental results show decent performance gains, with MetaTool achieving comparable results to ChatGPT while using much smaller models (8B parameters). Also ablation study on tool-oriented tasks showcase the effectiveness of meta-tasks generation even in the absence of “solution” data. Ablation study in ToolBench and BFCL validate the effectiveness of this data generation approach.
summary: This paper introduces MetaTool, a methodology for improving how large language models (LLMs) learn to use tools. Instead of relying solely on few-shot prompting or supervised fine-tuning with expert annotations, the authors propose a self-supervised approach based on meta-tasks that capture fundamental aspects of tool usage. The method generates training data by predicting masked elements in tool execution processes, enabling LLMs to develop a deeper understanding of tool functionality, causality, and constraints. The approach demonstrates improvements in both tool-based planning and chatting scenarios, achieving performance comparable to ChatGPT while using much smaller models.
weaknesses: - Lack of clarity regarding “solution data”: It feels like the author didn’t fully introduce and define these -concepts and brought them up abruptly in line 227. And in the evaluation section, it is a little unclear what data each baselines were trained on. It would be helpful if the authors could define different types of data in the experiment setup and use a consistent notation across the entire paper.

- Lake of details about the “meta-task generation”: it would be good to replace figure 4 with a qualitative example of metatasks generated for real-world benchmarks (e.g., ToolBench, BFCL). Also I’d appreciate the authors providing the actual prompt used for metaset construction, specifically about L199-201 “For large toolsets and diverse task scenarios that are hard to enumerate, we incorporate LLMs with self-play or tree search techniques to reduce redundant trials...”

Meta-review: This paper introduces MetaTool, a novel approach to enhance large language models’ (LLMs) ability to use tools. Unlike traditional methods that rely on prompts or labeled data, MetaTool employs self-supervised learning through six meta-tasks: Effect, Decision-making, Reversion, Input Boundary, Output Boundary, and Counterfact. These tasks teach foundational concepts like cause and effect, permissible actions, and expected outcomes. MetaTool demonstrates strong performance across tool-based tasks, rivaling models like ChatGPT in planning and chat scenarios. 

While empirical results demonstrate the effectiveness of the method, most experiences have been done in simulated environments, leaving questions on the applicability of the proposed approach in more realistic scenarios. In addition, reviewers noted some ambiguity in the methodological details and the clarity on fair comparisons.

Decision: Reject