MTU-Bench: A Multi-granularity Tool-Use Benchmark for Large Language Models

## Abstract

Abstract Large Language Models (LLMs) have displayed massive improvements in reasoning and decision-making skills and can hold natural conversations with users.
Recently, many tool-use benchmark datasets have been proposed.
However, existing datasets have the following limitations:
(1). Insufficient evaluation scenarios (e.g., only cover limited tool-use scenes).
(2). Extensive evaluation costs (e.g., GPT API costs).
To address these limitations,
in this work,
we propose a multi-granularity tool-use benchmark for large language models called MTU-Bench. For the “multi-granularity” property,
our MTU-Bench covers five tool usage scenes (i.e., single-turn and single-tool, single-turn and multiple-tool, multiple-turn and single-tool, multiple-turn and multiple-tool, and out-of-distribution tasks).
Besides,
all evaluation metrics of our MTU-Bench are based on the prediction results and the ground truth without using any GPT or human evaluation metrics.
Moreover, our MTU-Bench is collected by transforming existing high-quality datasets to simulate real-world tool usage scenarios,
and we also propose an instruction dataset called MTU-Instruct data to enhance the tool-use abilities of existing LLMs.
Comprehensive experimental results demonstrate the effectiveness of our MTU-Bench.
Code and data will be released at https://github.com/MTU-Bench-Team/MTU-Bench.git .

## 1 Introduction

Figure: Figure 1: The circles from inside to outside represent the data source, scenes included in MTU-Instruct, and the automatic evaluation metrics in MTU-Eval.
Refer to caption: x1.png

Since the release of large language models (LLMs) such as GPT-4 (OpenAI, ),
Natural Language Processing (NLP) has entered a new wave of advancements,
even being considered as the spark of Artificial General Intelligence (AGI) (Bubeck et al., ).
Recently, there has been a surge of research focused on enabling LLMs to interface with external tools,
such as calculators (Cobbe et al., ), search engines (Schick et al., ), and booking service APIs (Qin et al., ).
This approach, referred to as Tool Learning (Schick et al., ; Qin et al., ; ; Wang et al., ),
allows LLMs to not only accurately perform precise calculations,
but also maintain up-to-date information.
Furthermore, it enables LLMs to function as end-to-end AI assistants that are capable of fulfilling real-world user needs such as booking hotels or ordering food.
Thus, Tool Learning is a critical step to transform LLMs into general AI agents.

Previous works have explored to stimulate the ability to call tools for LLMs (Schick et al., ; Qin et al., ; Zhuang et al., ; Tang et al., ; Paranjape et al., ; Li et al., ).
For example,  (Schick et al., ) propose to convert tool calls into text spans, such as $<$API$>$some parameter key-value pairs$<$/API$>$ to denote the tool name and parameters with an additional special token (i.e., $<$/API$>$) to show the initiation of a tool execution.
Moreover, the recent works (i.e., ToolBench (Xu et al., ; Qin et al., ), APIBench (Patil et al., ), and API-Bank (Li et al., )) have investigated instruction tuning data or evaluation for tool-use.

However, we observe that exhibit several limitations to varying degrees as shown in Table 1:
(1) some do not consider multi-turn dialogue scenarios (Patil et al., ; Xu et al., ; Zhuang et al., );
(2) some do not address multi-tool usage scenarios (Tang et al., ; Patil et al., ; Li et al., ; Xu et al., );
(3) several works use external API tools to deduce user instructions, but these synthesized instructions often do not accurately align with actual real-world user needs (Qin et al., );
(4) many of them rely on GPT for the evaluation, leading to heavy evaluation costs (Qin et al., ; Tang et al., );
and (5) many do not comprehensively assess fine-grained aspects of tool-use (Li et al., ; Qin et al., ; Patil et al., ), such as the accuracy of tool call orders, complex tool calls involving inheritance relationships, per-dialogue turn accuracy of tool and parameter selection, four-quadrant analysis of single/multi-turn and single/multi-tool settings, etc.

**Table 1: Comparison of various tool-use benchmark datasets. “Auto. Eval.” denotes “automatic evaluation without GPT”. “S-S”, “S-M”, “M-S”, and “M-M” denote single-turn single-tool, single-turn multi-tool, multi-turn single-tool, and multi-turn multi-tool, respectively. “OOD” refers to whether the benchmark accounts for an out-of-distribution setting, where the test set consists of domains different from those in the training set. “Real-World” means whether the dialogues in the benchmark are sampled from real-world scenarios. The numbers in the evaluation range mean: 1 tool selection, 2 parameter selection, 3 dialogue-level success rate, 4 turn-based success rate, 5 tool number, and 6 tool order.**
| Dataset | #Dialogues | #Tools | #Turn-#Tool | Real-World | Auto. Eval. | Eval. Range | Train | Test | OOD |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MetaTool<br>(Huang et al., ) | 21,127 | 199 | S-S, S-M | ✗ | ✓ | 123 | ✗ | ✓ | ✗ |
| API-Bank<br>(Li et al., ) | 2,202 | 2,211 | S-S, S-M,<br>M-S, M-M | ✗ | ✓ | 123 | ✓ | ✓ | ✗ |
| ToolLLM<br>(Qin et al., ) | 12,657 | 16,464 | S-S, S-M | ✗ | ✗ | 23 | ✓ | ✓ | ✓ |
| API-Bench<br>(Patil et al., ) | 17,002 | 1,645 | S-S | ✗ | ✓ | 23 | ✓ | ✓ | ✗ |
| ToolAlpaca<br>(Tang et al., ) | 3,938 | 400 | S-S, S-M,<br>M-S, M-M | ✗ | ✗ | 1234 | ✓ | ✓ | ✗ |
| ToolQA<br>(Zhuang et al., ) | 1,530 | 13 | S-S, S-M | ✗ | ✓ | 23 | ✗ | ✓ | ✗ |
| MTU-Bench<br>(Ours) | 159,061 | 136 | S-S, S-M,<br>M-S, M-M | ✓ | ✓ | 123456 | ✓ | ✓ | ✓ |

To remedy these issues, in Figure 1, we introduce MTU-Bench (Multi-Granularity Tool-Use Benchmark), which comprises both MTU-Instruct for training and MTU-Eval for evaluation.
As illustrated in Figure 2, we sample real-world user instructions from various existing open-source dialogue datasets such as MultiWOZ (Budzianowski et al., ) and SGD (Rastogi et al., ; Lee et al., ).
After instruction clustering, the detected user intents and slot filling are leveraged to synthesize API calls using GPT-4 (OpenAI, ).
The synthesized data includes the thoughts, the actions (i.e., tool names), the action parameters, and the observations (i.e., the generated API execution results).
This data forms our MTU-Bench dataset.
Following meticulous quality verification by GPT-4 and manual check,
we split the MTU-Bench data into training and testing splits, involving 54798 dialogues in total, as well as 136 tools.
In our MTU-Eval, we propose a series of fine-grained metrics such as tool selection accuracy, parameter selection accuracy, success rate, turn success rate, task process rate, tool number accuracy, tool order accuracy, etc., to evaluate the tool-use abilities in a comprehensive manner,
where the GPT API costs are not needed for evaluation.
Moreover,
we also pick out a hard subset from the test split to include more complex tool-use scenarios such as easily confusable tools, nonsensical or noisy tools, tool parameter updating, etc.

Finally, by fine-tuning LLaMA3 (Dubey et al., ) on MTU-Bench, we find that our resulting model, MTU-LLaMA, performs the best in various scenarios and metrics, demonstrating the effectiveness of our MTU-Instruct.

In summary, our contributions are as follows:
(1). MTU-Bench: We introduce a novel automated data synthesis pipeline designed to derive high-quality, fine-grained tool-use datasets from pre-existing task-oriented dialogue datasets. This pipeline facilitates the creation of MTU-Bench, comprising MTU-Instruct for training purposes and MTU-Eval for evaluation. (2). MTU-Instruct and MTU-Eval: We introduce the high-quality and diverse instruction tuning dataset, MTU-Instruct, to improve models’ tool-use capabilities in real-world scenarios. Additionally, we propose a novel automatic evaluation framework, MTU-Eval, which assesses various tool-use settings through comprehensive and fine-grained metrics, free of GPT-based evaluators. (3). MTU-LLaMA and Experimental Findings: After instruction tuning on MTU-Instruct, we obtain a strong open-source model for tool-use, MTU-LLaMA. Our comprehensive experiments reveal several findings regarding the tool-use capabilities of LLMs, particularly in terms of multi-turn dialogue scenarios, multi-tool settings, and error cases. These findings offer valuable insights for advancing tool-use in LLMs.

## 2 MTU-Bench

The MTU-Bench involves both MTU-Instruct for training and MTU-Eval for evaluation.
We first present the data construction and analysis in §2.1, and then show the evaluation procedure in §2.2.

### 2.1 MTU-Instruct

Figure: Figure 2: The workflow for MTU-Instruct construction. It involves five steps: (1) Data Collection, (2) Tool Creation, (3) Tool Clustering, (4) Tool Documentation, and (5) Tool-Use Synthesis.
Refer to caption: x2.png

#### 2.1.1 Data Construction

Many previous methods based on scraped API documentation and GPT-4 inspired synthesis faced limitations due to a lack of data diversity, resulting in weak generalization capabilities.
Inspired by the mapping relationships between intents and APIs, as well as between slots and tool parameters, we designed an automated data synthesis pipeline that transforms traditional dialogue datasets into tool-use datasets. To enhance data diversity, we collected datasets from multiple NLP datasets and standardized them into a unified tool documentation. The diversity is illustrated in Figure 4.
As shown in Figure 2, the construction of MTU-Bench involves five primary steps:
(1) collecting task-oriented dialogue datasets, particularly those containing intents and slots,
(2) creating tools through grammar-based transformations or GPT-4 synthesis,
(3) clustering the tools based on their similarities,
(4) generating tool documentation using GPT-4, and
(5) synthesizing tool-use samples consisting of thoughts, actions(tool calls), action inputs, observations, and adjusted responses based on the dialogue and tool documentation, followed by a holistic validation process.

##### Data Collection.

To improve the diversity of our dataset, we collect several open-source task-oriented dialogue datasets as our data sources.
These datasets focus on dialogues for specific tasks such as flight reservations or movie bookings, which are highly suitable for synthesizing tool-use data.
The multi-turn dialogue datasets include MultiWOZ (Budzianowski et al., ), SGD (Rastogi et al., ), TaskMaster (Byrne et al., ) and MetaLWOZ (Shalyminov et al., ).
The single-turn dialogue datasets include ATIS (Hemphill et al., ) and SNIPS (Siddhant et al., ).
They provide diverse task-oriented dialogues across various domains, real-world conversation and fine-grained annotation, encompassing both single-turn and multi-turn dialogues.

##### Tool Creation.

We employ two approaches to create tools.
(1) Grammar-based creation. For dialogue datasets that already have detected intents and filled slots, we directly convert the intents into tool names and the slots into tool parameters. For example, in the user query “find a flight from charlotte to las vegas”, the intent “Flight” will convert to the tool name, and the slot “from_location=charlotte, to_location=las vegas” will convert to the parameters, resulting in the tool call Flight(from_location=“charlotte”,to_location=“las vegas”).
(2) LLM-based creation. For dialogue datasets without predefined intents or slots, we utilize GPT-4 to make the tools.
Based on the contextual situation, we categorize it into five scenarios: information missing, information exists, information confirmed, aimless chatting and specific API call.
Provided with the historical dialogue context and the current round of conversation, especially the assistant’s response, LLM needs to determine which situation belongs to the current situation.
Information missing. If the response is asking for important information, it should be the situation of missing information, no tool call should be made, and necessary parameters should be accumulated for related tools.
Information exists. When the LLM can provide a response based solely on the information from the dialogue history, no tool call will be made and the model can directly reply.
Information confirmed. When the assistant is confirming information (e.g.,Would you like to confirm this flight reservation?), this is classified as “information to be confirmed”.
Aimless chatting. If the scenario pertains to aimless chatting or situations that do not necessitate tool invocation, no tool call should be made.
Specific API call. Only if the LLM determines that an API call is necessary to fulfill the user’s request, it is encouraged to generate an appropriate pseudo-tool for invocation, along with a corresponding description and parameters of this tool.

##### Tool Clustering.

Due to the diversity in both the intent detection and slot-filling strategies, as well as the creation of specific tools based on LLM, the synthesized tool set can be highly redundant.
For example, tools like “search_movie” and “find_movie” may have different names but essentially perform the same function.
To address this redundancy, we introduce a tool clustering phase.
Specifically, we cluster the tool names based on InsTag(Lu et al., ) with a fixed distance threshold. Then, all tool names and their parameter names are standardized to the centroid of their respective cluster, resulting in a reduction ratio of 20:1.

##### Tool Documentation.

To enable models to use specific tools effectively, we compile all tool usage into a comprehensive tool document.
This document allows the model to determine the appropriate tool names and their usage. We prompt the GPT-4 model to write a description for each tool generated in the previous step, along with information about its parameters(required and optional) and returns.
The collection of these tool entries forms the final tool document, which is included as part of the LLMs’ context. Please refer to Appendix B for the details.

##### Tool-Use Data Synthesis.

In this step, we convert all samples from task-oriented dialogue datasets into tool-use dataset with GPT-4, following the format of ReAct (Yao et al., ).
We provide the dialogue history and the tool document generated in the previous steps as context for GPT-4, and then prompt it to generate three key components:
(1) thought: the reasoning process behind the tool selection,
(2) action: the name of the tool being invoked,
and (3) action input: the parameters used in the tool call along with their values.
This chain of thought prompting technique enhances the model’s ability to reason over the most appropriate tool and accurately input the parameter values.
We also allow the model to generate any additional parameters needed that are beyond those listed in the tool document, to ensure completeness and flexibility of tool-use.

We further ask the GPT-4 to simulate tool execution, generating observation (i.e., the results of the simulated tool execution) and then produce the final response for the current dialogue turn.
The observations are aligned with the return information in the tool document and are generated in a structured format, such as a JSON dictionary.
The model then formulates a response based on the observations, either to report the status of the tool execution or to complete the dialogue turn.

To ensure data quality, we apply various quality filters and adjustments, including heuristic rules, GPT-4, and manual annotation.
For example, we split some multi-turn dialogues into single-turn dialogues to balance the data distribution.
For the training set, we use GPT-4 to verify the accuracy and necessity of tool selection, check parameter matching, adjust thoughts, rewrite response and ensure consistency in the tool definitions. Through GPT-4, we filter out about 10% of defective samples.
For the test set, we hired multiple experts to conduct manual quality checks based on similar principles. Each sample was checked by three experts, and the differences in labeling were determined by the fourth expert.

Based on the number of dialogue turns and the number of tools in each dialogue,
the synthesized data can be categorized into four types: (1) Single-turn Single-tool (S-S), (2) Single-turn Multi-tool (S-M), (3) Multi-turn Single-tool (M-S), and (4) Multi-turn Multi-tool (M-M).

For more detailed information about the construction of MTU-Bench, including prompt templates and the tool documentation, please refer to the Appendix B.

#### 2.1.2 Data Analysis

Figure: Figure 3: The domain distribution of tools.
Refer to caption: x3.png

##### Statistics.

Figure 4 delineates the statistical metrics of our MTU-Bench, underscoring its substantial scale and diversity. It is particularly designed to encompass multi-tool and multi-turn settings, as well as *real-world* domains and topics.

##### Diversity.

Figure 4 illustrates the domain and topic distribution of the tools within our MTU-Bench. Figures 9 and 10 depict the distributions of dialogue turn counts, word counts, tool numbers, and other length-related metrics across both training and testing splits. These figures underscore the diversity of our MTU-Bench in terms of length and topic distribution.

We refer the readers to Appendix C for a more detailed data analysis and statistics.

### 2.2 MTU-Eval

We propose MTU-Eval—the first evaluation framework that encompasses multiple levels of difficulty, diverse domains and cases of tool-use, varying numbers of dialogue turns and tools, as well as multifaceted evaluation by considering various granularities and aspects of LLM tool-use.

We present MTU-Eval in two parts: (1) Test Set Splitting and (2) Evaluation Metrics.

#### 2.2.1 Test Set Splitting

We construct two distinct test sets from MTU-Bench through manual sampling: (1) the normal test set and (2) the more challenging hard test set.

##### Normal/Hard Test Set.

MTU-Bench includes data from 31 different topics, such as weather-related and calendar-related tasks.
Initially, we select data from 5 topics as an Out-of-Distribution (OOD) test split.
From the remaining 26 topics, we further split the data into a training set and an in-domain test set.
For more challenging evaluations, we manually curate a hard test set,
which includes more complex tool-use cases, such as those involving long tool parameters, easily confusable tools, parameter value updating, scenarios with a large number of tools, etc., as listed in Appendix D.

#### 2.2.2 Metrics

##### For Single-Tool Scenarios (S-S & M-S).

For scenarios where there is just a single tool involved, we evaluate two metrics: Tool Selection Accuracy (TS) and Parameter Selection Accuracy (PS).

##### For Multi-Turn Scenarios (M-S & M-M).

In multi-turn dialogues, the following metrics are introduced:
(1) Success Rate (SR): A binary metric where the entire dialogue is considered successful for tool-use: (=1) only if there are no errors throughout all turns; otherwise, it is considered unsuccessful (=0).
(2) Averaged Turn Success Rate (ATS): We first evaluate each dialogue turn with the tool-use success rate (where each turn is marked as either 0 or 1), then average the binary scores with the total dialogue turns of a dialogue session. This score takes into account the finer-grained success rate of tool-use at the turn level.
(3) Soft Averaged Turn Success Rate (SATS): This metric adjusts the ATS based on the proximity of errors to the current turn. Specifically: If a turn is incorrect, the score is 0. If a turn is correct, given $j$ as the index of this turn and $i$ as the index of the last incorrect turn, the score is $1$ when $j<i$ and $1-e^{-(j-i)}$ when $j>i$. This design is based on the intuition that a closer incorrect turn can negatively impact subsequent turns. Moreover, the closer the turn becomes incorrect, the lower the overall accumulated score, even if the remaining turns are correct.
(4) Task Process Rate (TPR): This is calculated as the ratio of the first incorrect turn to the total number of turns. This metric is included to capture how early in the dialogue the first mistake occurs, as earlier errors tend to disrupt the overall task flow more significantly.

##### For Multi-Tool Scenarios (S-M & M-M).

For scenarios involving multiple tools, the following metrics are introduced:
(1) Tool Number Accuracy (TN): Denote the predicted tool list as “Pred” and the ground truth tool list as “GT”, $\text{TN}=\mid\text{Pred}\cap\text{GT}\mid/\mid\text{Pred}\cup\text{GT}\mid$,
where $\mid\mid$ denotes the number of tools.
(2) Tool Order Accuracy (TO): This metric evaluates the correctness of the tool sequence, adjusted by a decay factor: $\text{TO}=t\times\mid\text{LCR(GT, Pred)}\mid/\mid\text{GT}\mid$, where LCR is the longest common subsequence, and $t$ is a decay coefficient calculated as: $t=\cos\left((\pi/2)\times(i/|\text{Pred}|\right)$, where $i$ is the starting position of the longest common subsequence. The value of $t$ ranges from 0 to 1, with a faster decay for positions later in the sequence.

These metrics offer a comprehensive and fine-grained evaluation of LLMs’ tool-use capabilities.
Unlike conventional approaches that focus only on overall success rates (Qin et al., ), our metrics account for the dynamics along dialogue turns and the dependencies between multiple tools.
We refer the reader to Appendix D for more examples of how to compute these metrics.

## 3 Experiments

##### Experimental Setup.

We evaluate 5 closed-source LLMs such as GPT-3.5 (OpenAI, ), GPT-4 (OpenAI, ), Qwen-Max (Team, ), GLM-4-Plus(^1^11[https://bigmodel.cn/dev/api/normal-model/glm-4](https://bigmodel.cn/dev/api/normal-model/glm-4)) and DeepSeek2.5 (DeepSeek-AI, ).
We also evaluate numerous open-source LLMs such as LLaMA2 (Touvron et al., ) and LLaMA3 (AI@Meta, ) series, Qwen1.5 (Team, ) and Qwen2 (Yang et al., ) series, Mistral (Jiang et al., ), ChatGLM3 and GLM-4 (GLM et al., ; Du et al., ; Zeng et al., ) series.
Then, we also compare 2 models specifically enhanced for tool-use: ToolLLaMA (Qin et al., ) and our MTU-LLaMA, which isfine-tuned on MTU-Instruct based on LLaMA3-8B-Instruct.
Note that all baselines are instruction-tuned models.
We refer the readers to Appendix D for more details on the hard cases, the prompt templates, and the metric computation.

### 3.1 Main Results

**Table 2: Results of different models on the normal set of MTU-Eval (S-S & M-S). “S-S” and “M-S” denote “Single-Turn Single-Tool” and “Multi-Turn Single-Tool” settings, respectively. We utilize green (1st), blue (2nd), and yellow (3rd) backgrounds to distinguish the top three results within both open-source and tool-use-specific models. We employ bold and underlined text to denote the top and second-best results across all model categories (same markers for the other tables). All the baselines are instruction-tuned models.**
| Models | S-S | M-S |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TS | PS | Avg. | TS | PS | ATS | SATS | SR | TPR | Avg. |  |
| Closed-Source Large Language Models |  |  |  |  |  |  |  |  |  |  |
| GPT-4 | 95.83 | 52.08 | 73.96 | 88.10 | 74.49 | 73.67 | 67.36 | 29.63 | 45.35 | 63.10 |
| GPT-3.5 | 84.62 | 46.15 | 65.38 | 69.05 | 50.68 | 50.51 | 40.81 | 1.85 | 12.70 | 37.60 |
| Qwen-Max | 91.67 | 50.00 | 70.83 | 86.73 | 66.67 | 64.96 | 57.88 | 20.37 | 35.53 | 55.36 |
| GLM-4-Plus | 95.83 | 50.00 | 72.92 | 85.03 | 71.43 | 72.01 | 65.19 | 24.07 | 44.18 | 60.32 |
| DeepSeek V2.5 | 93.75 | 43.75 | 68.75 | 86.39 | 69.39 | 68.09 | 60.20 | 18.52 | 38.04 | 56.77 |
| Open-Source Large Language Models |  |  |  |  |  |  |  |  |  |  |
| LLaMA2-7B | 15.38 | 3.85 | 9.62 | 33.67 | 28.91 | 26.78 | 20.67 | 0.00 | 7.12 | 19.53 |
| LLaMA2-70B | 70.59 | 33.33 | 47.28 | 47.28 | 30.95 | 30.18 | 23.78 | 0.00 | 9.82 | 23.67 |
| LLaMA3-8B | 65.38 | 30.77 | 48.08 | 35.71 | 17.35 | 17.53 | 12.34 | 0.00 | 1.72 | 14.11 |
| LLaMA3-70B | 86.54 | 57.69 | 72.12 | 79.25 | 61.90 | 62.18 | 54.81 | 14.81 | 32.33 | 50.88 |
| Qwen1.5-14B | 75.00 | 34.62 | 54.81 | 62.93 | 36.73 | 35.17 | 27.37 | 1.85 | 6.34 | 28.40 |
| Qwen1.5-72B | 78.85 | 38.46 | 58.65 | 80.95 | 61.22 | 59.08 | 50.88 | 16.67 | 28.75 | 48.75 |
| Qwen2-7B | 73.08 | 38.46 | 55.77 | 71.09 | 49.66 | 49.59 | 40.14 | 5.56 | 13.30 | 38.22 |
| Qwen2-72B | 86.54 | 48.08 | 67.31 | 79.93 | 61.22 | 58.52 | 50.28 | 14.81 | 25.15 | 48.32 |
| Mistral-7B | 60.42 | 25.00 | 42.71 | 61.22 | 37.07 | 37.33 | 29.14 | 3.70 | 9.98 | 29.74 |
| ChatGLM3-6B | 10.00 | 0.00 | 5.00 | 22.90 | 5.99 | 5.79 | 3.66 | 0.00 | 0.00 | 6.39 |
| GLM-4-9B | 91.67 | 45.83 | 68.75 | 63.95 | 42.18 | 42.72 | 35.98 | 3.70 | 19.01 | 34.59 |
| Tool-Use-Specific Large Language Models |  |  |  |  |  |  |  |  |  |  |
| ToolLLaMA2-7B | 85.42 | 18.75 | 52.08 | 31.97 | 7.82 | 7.6 | 5.20 | 0.0 | 5.73 | 9.72 |
| MTU-LLaMA (Ours) | 92.31 | 50.00 | 71.15 | 81.63 | 67.69 | 66.94 | 58.74 | 9.26 | 32.47 | 52.79 |

**Table 3: Results of different models on the normal set of MTU-Eval (S-M & M-M). “S-M” and “M-M” denote “Single-Turn Multi-Tool” and “Multi-Turn Multi-Tool” settings, respectively.**
| Models | S-M | M-M |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TN | TO | Avg. | TN | TO | ATS | SATS | SR | TPR | Avg. |  |
| Closed-Source Large Language Models |  |  |  |  |  |  |  |  |  |  |
| GPT-4 | 66.85 | 70.52 | 68.68 | 72.10 | 73.38 | 68.77 | 66.07 | 30.95 | 59.52 | 61.80 |
| GPT-3.5 | 32.64 | 38.22 | 35.43 | 24.72 | 25.46 | 16.11 | 12.22 | 0.00 | 3.97 | 13.75 |
| Qwen-Max | 39.76 | 48.82 | 39.29 | 62.00 | 64.07 | 56.27 | 55.27 | 4.76 | 52.38 | 49.13 |
| GLM-4-Plus | 45.76 | 48.48 | 47.12 | 53.95 | 54.58 | 49.17 | 45.72 | 4.76 | 39.48 | 41.28 |
| DeepSeek V2.5 | 56.88 | 60.28 | 58.58 | 50.15 | 51.79 | 44.84 | 41.26 | 7.14 | 34.72 | 38.32 |
| Open-Source Large Language Models |  |  |  |  |  |  |  |  |  |  |
| LLaMA2-7B | 3.39 | 3.94 | 3.67 | 22.22 | 22.22 | 22.90 | 21.80 | 0.00 | 19.92 | 19.92 |
| LLaMA2-70B | 6.82 | 8.48 | 7.65 | 30.12 | 30.49 | 28.77 | 28.77 | 0.00 | 28.77 | 28.77 |
| LLaMA3-8B | 14.79 | 20.30 | 17.55 | 9.43 | 10.04 | 4.44 | 2.81 | 0.00 | 0.00 | 4.46 |
| LLaMA3-70B | 26.85 | 32.68 | 29.76 | 33.60 | 35.71 | 26.94 | 23.82 | 0.00 | 17.86 | 22.99 |
| Qwen1.5-14B | 22.12 | 28.22 | 25.17 | 27.78 | 28.67 | 21.07 | 19.04 | 0.00 | 14.88 | 18.57 |
| Qwen1.5-72B | 28.04 | 30.60 | 29.32 | 23.00 | 23.31 | 52.94 | 51.86 | 0.00 | 7.34 | 26.41 |
| Qwen2-7B | 24.52 | 29.59 | 27.05 | 21.04 | 22.75 | 15.24 | 11.52 | 0.00 | 4.76 | 12.55 |
| Qwen2-72B | 52.76 | 59.98 | 56.37 | 45.93 | 47.67 | 42.02 | 38.07 | 7.14 | 29.76 | 29.76 |
| Mistral-7B | 14.21 | 18.22 | 16.22 | 10.15 | 11.11 | 5.44 | 3.66 | 0.00 | 0.60 | 5.16 |
| ChatGLM3-6B | 6.53 | 8.56 | 7.55 | 10.64 | 11.11 | 9.21 | 8.01 | 2.38 | 5.95 | 7.88 |
| GLM-4-9B | 23.64 | 27.58 | 25.61 | 16.17 | 16.45 | 9.48 | 6.13 | 0.00 | 0.00 | 8.04 |
| Tool-Use-Specific Large Language Models |  |  |  |  |  |  |  |  |  |  |
| ToolLLaMA2-7B | 11.52 | 11.52 | 11.51 | 4.07 | 4.07 | 2.78 | 2.34 | 0.00 | 1.59 | 2.48 |
| MTU-LLaMA (Ours) | 55.39 | 58.55 | 56.97 | 42.47 | 43.42 | 39.64 | 32.50 | 7.14 | 19.05 | 30.70 |

##### Overall Performance.

The experimental results for the normal set are presented in Table 2 (S-S & M-S) and Table 3 (S-M & M-M).
The results on the hard set are illustrated in Table 4.
These results reveal several key findings:
(1) Open-source models typically exhibit inferior performance compared to closed-source models in nearly all metrics, with the exception of GPT-3.5. However, certain models, including LLaMA3-70B and Qwen2-72B, demonstrate results comparable to those achieved by closed-source models.
(2) GPT-4 consistently exhibits superior performance on the normal set; however, its performance decreases on the hard set compared to GLM-4-Plus. Qwen-Max demonstrates exceptional performance in the M-M setting, with its advantages becoming more pronounced in the hard setting, even surpassing GPT-4. Similarly, GLM-4-Plus exhibits outstanding performance in the S-S and M-S settings, and its superiority is further amplified in the hard setting, also exceeding that of GPT-4. DeepSeek V2.5 performs admirably in the S-M setting.
(3) Our MTU-LLaMA exhibits substantial advancements over its initialization, i.e., LLaMA3-8B-Instruct, across all settings and metrics. It is also competitive with some closed-source models, underscoring the effectiveness of our MTU-Instruct.
(4) Generally, all the models perform better on the normal set than on the hard set, indicating LLMs’ limitations in handling more challenging tool-use scenarios.
(5) Notably, despite being fine-tuned specifically for tool-use, ToolLLaMA exhibits poor performance across all settings and metrics, suggesting its limited generalizability.

##### Effect of Multi-Turn.

We compare the single-turn (S-S, S-M) and multi-turn (M-S, M-M) settings across the Tables 2, 3, and 4, and have following findings:
(1) Both closed-source and open-source models tend to perform worse in multi-turn settings (M-S and M-M) compared to single-turn settings (S-S and S-M).
(2) Our MTU-LLaMA shows relatively better adaptation and robustness to multi-turn settings.
(3) Based on our novel TPR metric, we can observe that LLMs typically experience tool-use errors within the initial 30%-50% turns for closed-source models, and within the first 0%-30% turns for open-source models.
(4) Most models such as Qwen2-72B have significantly higher ATS scores than TPR scores. This implies that while LLMs frequently encounter tool-use errors in the initial turns, they can still correctly use tools in subsequent turns in most cases. However, this correctness does not account for the cascading effect of previous errors, but solely considers the success rate of independent tool usage.
(5) Fortunately, the SATS scores can be treated as an equilibrium between ATS and TPR metrics, which simultaneously account for the positions at which tool-use errors occur and the subsequent impact on later turns.

**Table 4: Average scores on the hard set of MTU-Eval. (Detailed results are shown in Appendix E).**
| Models | S-S | M-S | S-M | M-M |
| --- | --- | --- | --- | --- |
| Closed-Source Large Language Models |  |  |  |  |
| GPT-4 | 77.88 | 44.61 | 58.07 | 41.36 |
| GPT-3.5 | 41.96 | 30.86 | 18.39 | 11.87 |
| Qwen-Max | 77.88 | 42.11 | 24.01 | 45.08 |
| GLM-4-Plus | 82.69 | 47.61 | 30.90 | 39.53 |
| DeepSeek V2.5 | 80.77 | 44.94 | 40.01 | 30.62 |
| Open-Source Large Language Models |  |  |  |  |
| LLaMA2-7B | 28.57 | 17.13 | 2.35 | 11.76 |
| LLaMA2-70B | 28.57 | 23.46 | 1.74 | 16.79 |
| LLaMA3-8B | 25.89 | 12.85 | 9.91 | 5.89 |
| LLaMA3-70B | 71.43 | 40.40 | 20.67 | 20.56 |
| Qwen1.5-14B | 44.64 | 29.39 | 12.81 | 9.37 |
| Qwen1.5-72B | 56.73 | 29.92 | 18.85 | 17.93 |
| Qwen2-7B | 58.93 | 28.73 | 17.50 | 10.17 |
| Qwen2-72B | 68.40 | 38.42 | 37.14 | 25.13 |
| Mistral-7B | 26.92 | 26.04 | 11.48 | 10.84 |
| ChatGLM3-6B | 9.09 | 5.57 | 18.89 | 9.52 |
| GLM-4-9B | 47.12 | 30.22 | 18.98 | 9.52 |
| Tool-Use-Specific Large Language Models |  |  |  |  |
| ToolLLaMA2-7B | 18.27 | 10.19 | 0.51 | 2.34 |
| MTU-LLaMA (Ours) | 37.5 | 43.10 | 39.31 | 24.70 |

##### Effect of Multi-Tool.

Based on multi-tool settings (S-M and M-M) across Tables 2, 3, and 4, we derive the following findings:
(1) Multi-tool settings (S-M and M-M) show significant complexity, leading to noticeable performance drops for most models.
(2) Despite the complexity, models like GPT-4, Qwen2-72B and our MTU-LLaMA show stronger robustness.
(3) In contrast, the good models in single-tool settings such as GLM-4-Plus (closed-source) and LLaMA3-70B (open-source), are surpassed by Qwen-Max (closed-source) and Qwen2-72B (open-source), respectively, indicating the superior performance of Qwen series in multi-tool settings.
(4) The model rankings by TN and TO are highly consistent, implying that models with better control over the number of tools also tend to manage tool sequences effectively, suggesting a strong correlation between these capabilities.

### 3.2 Analysis

##### OOD Performance.

To evaluate the generality of MTU-LLaMA, we measure its performance on the OOD test split of MTU-Bench and two other OOD tool-use benchmarks, i.e., API-Bank (Li et al., ) and ToolTalk (Farn & Shin, ). in Table 5, we compare the performance of MTU-LLaMA, LLaMA3-8B-Instruct, and GPT-4 on these benchmarks under the M-S setting,
and We observe that MTU-LLaMA outperforms LLaMA3-8B-Instruct on all three OOD benchmarks.
Notably, MTU-LLaMA achieves performance comparable to that of GPT-4 on API-Bank,
which show strong generalizability of MTU-LLaMA.

**Table 5: OOD Performance of our MTU-LLaMA.**
| Models | MTU-Bench (OOD) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| TS | PS | ATS | SATS | SR | TPR | Avg. |  |
| GPT-4 | 67.28 | 67.57 | 65.19 | 47.23 | 32.31 | 37.72 | 52.88 |
| LLaMA3-8B | 36.71 | 36.71 | 35.53 | 23.00 | 7.69 | 11.31 | 25.16 |
| MTU-LLaMA | 47.34 | 37.88 | 62.05 | 54.58 | 9.26 | 35.49 | 41.10 |
| Models | ToolTalk |  |  |  |  |  |  |
| TS | PS | ATS | SATS | SR | TPR | Avg. |  |
| GPT-4 | 51.08 | 51.60 | 44.90 | 39.74 | 6.90 | 27.13 | 36.89 |
| LLaMA3-8B | 30.00 | 30.86 | 26.72 | 22.66 | 0.00 | 15.44 | 20.95 |
| MTU-LLaMA | 30.25 | 31.23 | 30.87 | 28.51 | 3.45 | 22.82 | 24.52 |
| Models | API-Bank |  |  |  |  |  |  |
| TS | PS | ATS | SATS | SR | TPR | Avg. |  |
| GPT-4 | 48.56 | 48.56 | 45.56 | 44.59 | 38.32 | 41.10 | 44.45 |
| LLaMA3-8B | 14.03 | 14.03 | 13.86 | 11.94 | 7.74 | 8.63 | 11.71 |
| MTU-LLaMA | 51.80 | 51.80 | 48.58 | 45.57 | 38.10 | 38.10 | 45.66 |

Figure: Figure 5: (a) Effect of dialogue turns. (b) Effect of different number of tools.
Refer to caption: x4.png

##### Effect of Dialogue Turns.

We illustrate the impact of dialogue turns on tool selection accuracy in Figure 5. We observe that performance slightly declines as the number of dialogue turns increases. Our MTU-LLaMA exhibits the most gradual decrease in performance, demonstrating its robustness to higher dialogue turns.

##### Effect of Tool Numbers.

Figure 5 shows the impact of tool numbers on tool selection accuracy. As the number of tools increases, both GPT-3.5 and LLaMA3-70B experience notable declines in performance, with LLaMA3-70B showing a sharper drop. In contrast, MTU-LLaMA maintains relatively stable accuracy, demonstrating its superior handling of multiple tool calls.

Figure: Figure 6: Scaling Law of LLMs on MTU-Eval.
Refer to caption: x5.png

##### Error Analysis.

In Figure 7,
we use five LLMs to analyze the different types of errors (i.e., “Action Error”, “Parameter Error” and “Format Error”).
Specifically,
the “Action Error” and “Parameter Error” denotes to select wrong tools and wrong parameters,
respectively,
and the “Format Error” means that the model cannot follow instructions well and outputs wrong formats,
which cannot be resolved well.
Figure 7 illustrates two primary findings:
(1) “Action Error” occurs more often than other errors, specifically in challenge M-M setting,
and stronger models show fewer errors.
(2) “Format Error” usually exists in weaker LLMs (e.g., LLaMA3-8B-Instruct and ChatGLM4-9B),
and our fine-tuned version MTU-LLaMA greatly reduces format issues,
which shows the effectiveness of MTU-Instruct.
See Appendix E for detailed error cases.

Figure: Figure 7: Analysis of different error types for different LLMs under M-S and M-M settings.
Refer to caption: x6.png

##### Scaling Law

We evaluate the performance of MTU-LLaMA across different model sizes, using LLaMA2 models with 7B, 13B, and 70B parameters as initialization, which are fine-tuned with MTU-Instruct. The results in Figure 6 show that the performance of MTU-LLaMA improves as the model size increases, suggesting its scalability.

##### Consistency Between Our Proposed Metrics and Human Evaluation.

To show the validity of our proposed metrics (SATS, TN, and TO), we evaluate the consistency between these novel metrics and the human evaluation results.
We randomly sample 50 instances from the M-S subset for the SATS metric and 50 instances from the S-M subset for the TN and TO metrics to compare two models: GPT-3.5 and LLaMA3-8B.
Human annotators are also asked to compare the two models.
In Table 6, we report the Pearson correlation coefficient between our metrics and human evaluation results, as well as the Pearson correlation among human annotators, which shows excellent consistency between these metrics and human evaluations.
More details can be found
in Appendix E.

**Table 6: Consistency for our proposed metrics.**
| Metric | Annotation Count | Consistency |  |
| --- | --- | --- | --- |
| Metric-Human | Human-Human |  |  |
| SATS | 50 | 0.8280 | 0.9344 |
| TN | 50 | 0.8497 | 0.9245 |
| TO | 50 | 0.8821 | 0.9482 |

## 4 Related Works

##### Instruction Tuning for Tool Learning.

The objective of tool learning is to equip large language models (LLMs) with human-like tool usage capabilities (Yang et al., ; Shen et al., ; Qu et al., ).
To achieve this, LLMs are typically fine-tuned with tool instruction data to improve their performance in tool planning, selection, calling, and response generation (Schick et al., ; Liang et al., ; Kong et al., ).
However, existing tool instruction datasets either have limitations in multi-turn dialogue and multi-tool usage scenarios, or are based on synthetic data, resulting in misalignment with real-world user needs (Huang et al., ; Li et al., ; Patil et al., ; Zhuang et al., ). In this paper, we introduce MTU-Instruct, a large-scale instruction dataset to improve LLMs’ performance in diverse real-world tool-use scenarios.

##### Evaluation Benchmarks for Tool Learning.

Many tool-use benchmarks have been proposed, but they still have many limitations.
Firstly, in Table 1, these benchmarks have limited capabilities to assess complex scenarios (e.g., multi-turn dialogues, multiple tools, and cross-domain tool generalization) (Huang et al., ; Li et al., ; Patil et al., ; Tang et al., ).
Secondly, some benchmarks rely excessively on GPT models, potentially leading to subjective and unstable results with heavy costs (Qin et al., ; Tang et al., ).
Finally, existing assessments often overlook critical dimensions, such as the order of multi-tool invocation, the impact of erroneous calls on subsequent interactions, and the accuracy of tool parameter selection, resulting in evaluations lacking comprehensiveness and depth (Zhuang et al., ; Li et al., ; Patil et al., ).
In contrast, MTU-Bench not only includes extensive multi-turn dialogues and multiple tool scenarios but also introduces testing for OOD tool generalization. By employing automated evaluation and incorporating metrics like SATS, TN, and TO, MTU-Eval achieves a more
comprehensive assessment.

## 5 Conclusion

In this work,
we propose a multi-granularity tool-use benchmark for LLMs called MTU-Bench,
which consists of MTU-Instruct and MTU-Eval.
Specifically,
first,
the MTU-Instruct dataset is used to enhance the tool-use abilities of existing LLMs,
and the MTU-Eval with multiple tool-use scenes is applied to benchmark the tool-use abilities comprehensively.
Notably,
all evaluation metrics of our MTU-Eval are based on the prediction results and the ground truth without using any GPT or human evaluation metrics.
Moreover,
Comprehensive experimental results demonstrate the effectiveness of our MTU-Bench.
Finally, we hope MTU-Bench can guide developers and researchers in understanding the tool-use capabilities of LLMs and facilitate the growth of foundation models.

## Ethics

In developing MTU-Bench and MTU-LLaMA, we recognize several ethical considerations that arise from the broader context of integrating tool-use capabilities into large language models (LLMs). As these models become more capable of interacting with real-world systems—such as those involving financial services, healthcare, and other critical domains—we must consider the potential risks associated with misuse. For instance, there is the possibility that LLMs could be exploited to access sensitive tools or manipulate information in ways that could harm individuals or organizations.

While our work aims to improve the accuracy and efficiency of tool use, we are mindful of the importance of ensuring that these technologies are deployed responsibly. We advocate for the implementation of robust safeguards, including transparency in decision-making processes, fairness in how tools are applied, and accountability in real-world usage. Furthermore, we encourage future research and development efforts to focus on mitigating potential biases and ensuring that these systems are secure and trustworthy when handling sensitive tasks.

## Appendix A Limitations

Despite the strengths of our proposed MTU-Bench, there are still several limitations to consider.
First, although MTU-Bench incorporates a diverse range of real-world user instructions, it may not fully capture all potential edge cases or highly complex interactions that can occur in dynamic, real-world environments.
Second, while MTU-Eval provides comprehensive fine-grained metrics for evaluating tool-use abilities, these metrics are based on predefined benchmarks, which may not account for every possible tool-use challenge in evolving real-world applications.
Although MTU-LLaMA demonstrates strong generalization across various metrics and scenarios, further research is needed to explore its adaptability to increasingly complex and emerging tool functionalities in real-time settings.
Future work should focus on broadening the benchmark’s coverage and exploring more dynamic and complex real-world use cases.

## Appendix B Details of MTU-Bench Construction

##### Prompt Templates for MTU-Bench Construction.

The construction procedures of MTU-Bench involves numerous prompt templates, as listed below, for tool making (tool name synthesis and tool parameter synthesis), thought synthesis, observation simulation, and data quality check.

##### Example of Tool Document.

As shown in Figure 8, the tool document allows the model to determine the appropriate tool names and their usages. It contains all the tools we synthesized, each tool including its corresponding tool description, necessary parameters, optional parameters, parameter description and data type, as well as the returns.

Figure: Figure 8: The JSON structure of the tool “setAlarm”, which is used to create a new alarm based on user-specified parameters. The structure includes required, optional and result (i.e., return) parameters, along with their corresponding data types, descriptions, formats and default values.
Refer to caption: x7.png

## Appendix C More MTU-Bench Data Analysis.

Figure: Figure 9: Length distributions of the training data.
Refer to caption: x8.png

Figure: Figure 10: Length distributions of the evaluation data.
Refer to caption: x9.png

**Table 7: The number of dialogues under different settings.**
|  |  | Test |  |
| --- | --- | --- | --- |
| Setting | Train | normal | hard |
| S-S | 14277 | 52 | 56 |
| S-M | 13641 | 55 | 39 |
| M-S | 19007 | 54 | 31 |
| M-M | 7442 | 42 | 37 |
| OOD | - | 65 |  |

##### Length Distribution.

We illustrate the length distributions in Figure 9, and Figure 10 for the training and evaluation data, respectively.

##### More Statistics.

Table 7 provides a comprehensive summary of the dialogue statistics across diverse settings, distinct splits and subsets.

## Appendix D Details of MTU-Eval

##### Hard cases in the hard test set

encompass extensive parameters, nonsensical tool names, determination of specific parameter values, inability to call tools, interaction among multiple tools, and multi-turn parameter inheritance, as delineated in Table 8.

**Table 8: Hard cases in the hard test set.**
| Type | Description | Case |
| --- | --- | --- |
| Extensive Parameters | The quantity of parameters required to be specified exceeds six. | User: Can you book me a non-refundable, one-stop flight on Delta Airlines from New York to Chicago, leaving on the 5th and returning on the 11th of this month in Economy class?<br>Target: Action: ReserveRoundtripFlights Action Input: {“airlines”: “ Delta Airlines”, “departure_date”: “2024-01-05”, “destination_city”: “Chicago”, “origin_city”: “New York”, “return_date”: “2024-01-11”, “seating_class”: “Economy”, “number_stops”: “1”, “refundable”: False} |
| Nonsensical Tool Names | The designation of the tool lacks significance, for instance, “abc”. | System Prompt: …The following is a list of APIs and their parameters that you can use: { “name”: “eee”, “description”: “Book an appointment at a dentist for a given time and date”, “required_parameters”: [“dentist_name”, “appointment_time”, “appointment_date”]…<br>User: I would like to book an appointment with Yvonne Yang at Greenview Dental Care on the 11th at 11:45. Do they offer cosmetic services?<br>Target: Action: eee Action Input:<br>{“appointment_date”: “2024-01-11”, “appointment_time”: “11:45”, “dentist_name”: “Yvonne Yang“} |
| Determination of Specific Parameter Values | Parameter values must conform to specific criteria. | Tool Definition: “name”: “searchHotel”, “description”: “To search for hotels based on a set of criteria including rating, type, amenities, location, and price range.”, “optional_parameters”: […“name”: “priceRange”, “choices” = [“cheap”, “moderate”, “expensive”]<br>User: Can you help me find a moderately priced hotel in the centre with parking and internet availability?<br>Target: Action: SearchHotel Action Input: “area”: “centre”, “parking”: True, “internet”: True, “priceRange”: “moderate”, “return_info”: [“all”] |
| Inability to Call Tools | The list of tools does not contain any suitable options. | System Prompt: You are a …The following is a list of APIs and their parameters that you can use…<br>Prompt: Can you find me a drama movie with subtitles directed by Claire Denis?<br>Target: Thought: The user wants to find a drama movie with subtitles directed by Claire Denis. However, there is no available API in the provided list that supports the searching for movies. Therefore, it’s not possible to meet the user’s request with the current tools. |
| Interaction among Multiple Tools | In a single turn of dialogue, more than four tools are utilized. | User: I am looking for the date of a concert in New York, followed by the weather forecast for that day. And I want to find a dentist, book an appointment at 10:00 on Febrary 2nd. And please add the dentist’s phone to my contact list. Thank you!<br>Target<br>Action: FindEvents Action Input: {“event_type”: “concert”, “city”: “New York”}<br>Action : GetWeather Action Input: {“city”: “New York”, “date”: “FindEvents.date”}<br>Action: BookAppointment Action Input: {“dentist_name”: “FindProvider.dentist_name”, “appointment_time”: “10:00”, “appointment_date”: “2024-02-02”}<br>Action: AddNewContact Action Input: {“contact_name”: “FindProvider.dentist_name”, “phone_number”: “FindProvider.phone_number”} |
| Multi-Turn Parameter Inheritance | The parameters for the current turn must be extracted from the historical dialogue. | User: Find some music please.<br>Assistant: Action: LookupMusic<br>Observation: {“artist”: “Ariana Grande”, “song_name”: “Be My Baby”}<br>Assistant: Ok, how about Be My Baby by Ariana Grande?<br>User: That sounds good.<br>Assistant: Should I play this now?<br>User: Yes. Play on kitchen speaker please.<br>Target: Action: PlayMedia Action Input: { “playback_device”: “kitchen speaker”, “song_name”: “Be My Baby” } |

##### Prompt Templates for MTU-Eval.

During the evaluation, the models are provided with distinct system prompts for various settings, which encompass both the comprehensive task specifications and the tool documentation.
We list the system prompts used for evaluation in Boxes D.

##### Examples for computing metrics.

The examples are listed below:

Tool Selection Accuracy (abbr. TS)
(for S-S, M-S)

✓Positive Example:

- User: “Search for the latest news on artificial intelligence.”

- Assistant: OK! Let me search the web. search_web(query="latest news on AI")

- Ground Truth Tool: search_web

- Assistant’s Tool: search_web

- Score: TS = 1

- Explanation: The assistant correctly selected the tool search_web, matching the ground truth.

✗Negative Example:

- User: “Calculate the square root of 16.”

- Assistant: OK! Let me use the calculator. search_web(query="square root of 16")

- Ground Truth Tool: calculate

- Assistant’s Tool: search_web

- Score: TS = 0

- Explanation: The assistant incorrectly selected search_web instead of the correct tool calculate.

Parameter Selection Accuracy (abbr. PS)
(for S-S, M-S)

✓Positive Example:

- User: “Translate ‘good evening’ to German.”

- Assistant: Sure, translating now. translate(text="good evening", target_language="German")

- Ground Truth Parameters: {text=“good evening”, target_language=“German”}

- Assistant’s Parameters: {text=“good evening”, target_language=“German”}

- Score: PS = 1

- Explanation: The assistant provided the correct parameters matching the ground truth.

✗Negative Example:

- User: “Set an alarm for 6 AM tomorrow.”

- Assistant: Setting your alarm. set_alarm(time="6 PM", name="tomorrow")

- Ground Truth Parameters: {time=“6 AM”, date=“tomorrow”}

- Assistant’s Parameters: {time=“6 PM”, name=“tomorrow”}

- Score: PS = 0

- Explanation: The assistant incorrectly set the value for the time parameter. The assistant incorrectly set the key for the date parameter.

Success Rate (abbr. SR)
(for M-S, M-M)

✓Positive Example:

- User: “I need to send an email to Alex.”

- Assistant: Sure, composing an email. compose_email(recipient="Alex", subject="Hello", body="Hi Alex, how are you?")

- User: “Also, schedule a meeting with him next Monday at 2 PM.”

- Assistant: Scheduling the meeting. schedule_meeting(participant="Alex", date="next Monday", time="2 PM")

- Ground Truth: No errors throughout all turns.

- Score: SR = 1

- Explanation: The assistant correctly used the tools in all dialogue turns without any errors.

✗Negative Example:

- User: “Find the fastest route to the airport.”

- Assistant: Finding the fastest route. get_directions(destination="airport")

- User: “Book a taxi to pick me up in 30 minutes.”

- Assistant: Booking a taxi. book_taxi(pickup_time="in 3 minutes")

- Ground Truth: Assistant should have set the pickup time to “in 30 minutes”.

- Score: SR = 0

- Explanation: The assistant made an error in the second turn (incorrect pickup time), so the entire dialogue is considered unsuccessful.

Averaged Turn Success Rate (abbr. ATS)
(for M-S, M-M)

✓Positive Example:

- Turn 1:

- User: “What’s on my schedule today?”

- Assistant: Checking your schedule. get_schedule(date="today")

- Turn Success: 1

- Turn 2:

- User: “Add a dentist appointment at 4 PM.”

- Assistant: Adding the appointment. add_event(event="dentist appointment", time="4 PM", date="today")

- Turn Success: 1

- Turn 3:

- User: “Remind me an hour before.”

- Assistant: Setting a reminder. set_reminder(event="dentist appointment", time="3 PM", date="today")

- Turn Success: 1

- Score: ATS = (1 + 1 + 1)/3 = 1.0

- Explanation: All turns were executed correctly.

✗Negative Example:

- Turn 1:

- User: “Play my workout playlist.”

- Assistant: Playing your playlist. play_music(playlist="workout")

- Turn Success: 1

- Turn 2:

- User: “Skip to the next song.”

- Assistant: Restarting the song. play_music(action="restart")

- Turn Success: 0 (Incorrect action parameter)

- Turn 3:

- User: “Increase the volume.”

- Assistant: Increasing volume. adjust_volume(level="up")

- Turn Success: 1

- Score: ATS = (1 + 0 + 1)/3 = 0.6667

- Explanation: One out of three turns was unsuccessful, leading to an ATS of approximately 0.6667.

Soft Averaged Turn Success Rate (abbr. SATS)
(for M-S, M-M)

✓Positive Example:

- Turn 1:

- User: “What’s the current time in Tokyo?”

- Assistant: The current time is… get_time(location="Tokyo") $\to$ 3:00 PM

- Turn Success: 1

- Turn 2:

- User: “Schedule a call with the Japan office at that time tomorrow.”

- Assistant: Scheduling the call. schedule_call(participant="Japan office", time="3:00 PM", date="tomorrow")

- Turn Success: 1

- Score: SATS = (1 + 1)/2 = 1.0

- Explanation: Both turns were successful, so the SATS is 1.

✗Negative Example:

- Turn 1:

- User: “Check the weather in New York.”

- Assistant: Checking the weather. get_weather(location="New York")

- Turn Success: 1

- Turn 2:

- User: “Schedule a meeting tomorrow at 10 AM.”

- Assistant: Scheduling the meeting. schedule_meeting(date="tomorrow", time="10 AM")

- Turn Success: 1

- Turn 3:

- User: “Set an alarm for 9 AM.”

- Assistant: Setting an alarm. set_alarm(time="9 PM")

- Turn Success: 0 (Incorrect time parameter)

- Turn 4:

- User: “Remind me to call John at 11:00 AM.”

- Assistant: Setting a reminder. set_reminder(event="call John", time="11:00 AM")

- Turn Success: 1

- Turn 5:

- User: “Send an email to Sarah.”

- Assistant: Composing an email. compose_email(recipient="Sarah", subject="Hello", body="Hi Sarah, ...")

- Turn Success: 1

Calculations:

- Last incorrect turn index $i=3$

- For $j=1$ and $j=2$:

- Since $j<i$, Score = 1

- For $j=3$:

- Turn is incorrect, Score = 0

- For $j=4$:

- $j>i$, Score = $1-e^{-(4-3)}=1-e^{-1}\approx 0.6321$

- For $j=5$:

- $j>i$, Score = $1-e^{-(5-3)}=1-e^{-2}\approx 0.8647$

- Score: SATS = $\frac{1+1+0+0.6321+0.8647}{5}\approx\frac{3.4968}{5}\approx 0.6994$

- Explanation: The error in the third turn reduces the scores of subsequent turns due to the exponential decay, resulting in a SATS of approximately 0.6994.

Task Process Rate (abbr. TPR)
(for M-S, M-M)

✓Positive Example:

- Total Turns: $n=4$

- First Incorrect Turn Index: No incorrect turns.

- Score: TPR = $\frac{n}{n}=1$

- Explanation: Since there are no errors, the task was processed completely.

✗Negative Example:

- Total Turns: $n=5$

- First Incorrect Turn Index: $i=3$ (error occurs at turn 3)

- Score: TPR = $\frac{i-1}{n}=\frac{2}{5}=0.4$

- Explanation: The task process rate indicates the proportion of the task completed before the first error.

Tool Number Accuracy (abbr. TN)
(for S-M, M-M)

✓Positive Example:

- Ground Truth Tool List (GT): {search_web, summarize_text, translate_text}

- Assistant’s Predicted Tool List (Pred): {search_web, summarize_text, translate_text}

- Score: TN = $\frac{|\text{Pred}\cap\text{GT}|}{|\text{Pred}\cup\text{GT}|}=\frac{3}{3}=1.0$

- Explanation: All tools predicted by the assistant match the ground truth.

✗Negative Example:

- Ground Truth Tool List (GT): {search_web, translate_text}

- Assistant’s Predicted Tool List (Pred): {search_web, play_music, set_alarm}

- Score: TN = $\frac{|\text{Pred}\cap\text{GT}|}{|\text{Pred}\cup\text{GT}|}=\frac{1}{4}=0.25$

- Explanation: Only search_web is common between the predicted and ground truth lists, out of four unique tools.

Tool Order Accuracy (abbr. TO)
(for S-M, M-M)

✓Positive Example:

- Ground Truth Tool Sequence (GT): [search_web, extract_data, generate_report]

- Assistant’s Predicted Tool Sequence (Pred): [search_web, extract_data, generate_report]

- Longest Common Subsequence (LCS): [search_web, extract_data, generate_report]

- Starting Position (i): 1

- Total Tools in Pred: $|\text{Pred}|=3$

- Decay Coefficient (t):

$t=\cos\left(\frac{\pi}{2}\times\frac{i}{|\text{Pred}|}\right)=\cos\left(\frac{
\pi}{2}\times\frac{1}{3}\right)\approx 0.8660$

- Score:

$\text{TO}=t\times\frac{|\text{LCS}|}{|\text{GT}|}=0.8660\times\frac{3}{3}=0.86
60\times 1=0.8660$

- Explanation: The assistant’s sequence perfectly matches the ground truth, resulting in a high TO score.

✗Negative Example:

- Ground Truth Tool Sequence (GT): [get_weather, plan_route, book_hotel]

- Assistant’s Predicted Tool Sequence (Pred): [book_hotel, plan_route, get_weather]

- Longest Common Subsequence (LCS): [plan_route]

- Starting Position (i): 2

- Total Tools in Pred: $|\text{Pred}|=3$

- Decay Coefficient (t):

$t=\cos\left(\frac{\pi}{2}\times\frac{i}{|\text{Pred}|}\right)=\cos\left(\frac{
\pi}{2}\times\frac{2}{3}\right)\approx 0.5000$

- Score:

$\text{TO}=t\times\frac{|\text{LCS}|}{|\text{GT}|}=0.5000\times\frac{1}{3}
\approx 0.1667$

- Explanation: Only one tool matches in sequence, and it starts at the second position, leading to a low TO score.

##### Tool Selection Accuracy (abbr. TS)

##### Parameter Selection Accuracy (abbr. PS)

##### Success Rate (abbr. SR)

##### Averaged Turn Success Rate (abbr. ATS)

##### Soft Averaged Turn Success Rate (abbr. SATS)

##### Task Process Rate (abbr. TPR)

##### Tool Number Accuracy (abbr. TN)

##### Tool Order Accuracy (abbr. TO)

## Appendix E More Experimental Analysis

##### Detailed Evaluation Results on Hard Set.

In Table 9 and Table 10, we present the performance of various LLMs evaluated on the hard subset of MTU-Eval.
Our analysis indicates that among closed-source models, GLM-4-Plus excels in handling single-tool scenarios, surpassing even GPT-4. Conversely, Qwen-Max demonstrates superior performance in multi-turn and multi-tool scenarios, also outperforming GPT-4.
In single-turn multi-tool scenarios, GPT-4 and DeepSeek V2.5 exhibit relatively better performance.
Furthermore, there remains a significant gap between open-source models and their closed-source counterparts.
Among the open-source models, our MTU-LLaMA, Qwen2-72B, and LLaMA3-70B lead the performance metrics.
Specifically, Qwen2-72B shows a slight advantage in multi-tool scenarios, while LLaMA3-70B excels in single-tool scenarios. MTU-LLaMA, however, demonstrates a balanced ability across almost all evaluated settings.

**Table 9: Results of different models on the hard set of MTU-Eval (S-S & M-S). “S-S” and “M-S” denote “Single-Turn Single-Tool” and “Multi-Turn Single-Tool” settings, respectively. All the baselines are instruction-tuned models.**
| Models | S-S | M-S |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TS | PS | Avg. | TS | PS | ATS | SATS | SR | TPR | Avg. |  |
| Closed-Source Large Language Models |  |  |  |  |  |  |  |  |  |  |
| GPT-4 | 88.46 | 67.31 | 77.88 | 85.47 | 62.01 | 55.77 | 46.57 | 0.00 | 17.80 | 44.61 |
| GPT-3.5 | 50.00 | 33.93 | 41.96 | 71.51 | 43.02 | 39.45 | 29.33 | 0.00 | 1.83 | 30.86 |
| Qwen-Max | 88.46 | 67.31 | 77.88 | 80.45 | 56.98 | 50.54 | 41.77 | 3.23 | 19.67 | 42.11 |
| GLM-4-Plus | 92.31 | 73.08 | 82.69 | 84.92 | 63.69 | 58.40 | 48.79 | 6.45 | 23.44 | 47.61 |
| DeepSeek V2.5 | 90.38 | 71.15 | 80.77 | 83.24 | 58.10 | 56.10 | 47.28 | 3.23 | 21.66 | 44.94 |
| Open-Source Large Language Models |  |  |  |  |  |  |  |  |  |  |
| LLaMA2-7B | 32.14 | 25.00 | 28.57 | 34.64 | 26.82 | 22.32 | 16.32 | 0.00 | 2.70 | 17.13 |
| LLaMA2-70B | 32.14 | 25.00 | 28.57 | 52.51 | 30.17 | 28.45 | 21.62 | 0.00 | 8.03 | 23.46 |
| LLaMA3-8B | 33.93 | 17.86 | 25.89 | 37.99 | 13.97 | 11.64 | 8.15 | 0.00 | 5.38 | 12.85 |
| LLaMA3-70B | 82.14 | 60.71 | 71.43 | 75.42 | 52.51 | 51.68 | 42.03 | 3.23 | 17.53 | 40.40 |
| Qwen1.5-14B | 44.64 | 44.64 | 44.64 | 45.81 | 45.81 | 40.28 | 32.05 | 0.00 | 12.40 | 29.39 |
| Qwen1.5-72B | 57.69 | 55.77 | 56.73 | 48.11 | 48.11 | 40.10 | 31.62 | 0.00 | 11.57 | 29.92 |
| Qwen2-7B | 67.86 | 50.00 | 58.93 | 64.80 | 36.31 | 32.90 | 24.91 | 0.00 | 13.48 | 28.73 |
| Qwen2-72B | 78.93 | 57.86 | 68.40 | 78.77 | 50.84 | 47.19 | 37.20 | 0.00 | 16.50 | 38.42 |
| Mistral-7B | 36.54 | 17.31 | 26.92 | 63.69 | 34.64 | 31.29 | 23.90 | 0.00 | 2.74 | 26.04 |
| ChatGLM3-6B | 13.64 | 4.55 | 9.09 | 21.44 | 5.21 | 4.13 | 2.61 | 0.00 | 0.00 | 5.57 |
| GLM-4-9B | 59.62 | 34.62 | 47.12 | 64.80 | 37.99 | 36.56 | 29.42 | 0.00 | 12.52 | 30.22 |
| Tool-Use-Specific Large Language Models |  |  |  |  |  |  |  |  |  |  |
| ToolLLaMA2-7B | 32.69 | 3.85 | 18.27 | 25.70 | 3.35 | 3.50 | 2.60 | 0.00 | 0.65 | 10.19 |
| MTU-LLaMA(ours) | 51.79 | 28.57 | 40.18 | 78.21 | 59.22 | 57.35 | 46.90 | 0.00 | 16.93 | 43.10 |

**Table 10: Results of different models on the hard set of MTU-Eval (S-M & M-M). “S-M” and “M-M” denote “Single-Turn Multi-Tool” and “Multi-Turn Multi-Tool” settings, respectively. All the baselines are instruction-tuned models.**
| Models | S-M | M-M |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TN | TO | Avg. | TN | TO | ATS | SATS | SR | TPR | Avg. |  |
| Closed-Source Large Language Models |  |  |  |  |  |  |  |  |  |  |
| GPT-4 | 57.77 | 58.37 | 58.07 | 57.08 | 58.31 | 49.28 | 45.44 | 0.00 | 38.06 | 41.36 |
| GPT-3.5 | 17.75 | 19.04 | 18.39 | 21.31 | 22.39 | 15.09 | 10.41 | 0.00 | 2.03 | 11.87 |
| Qwen-Max | 23.75 | 24.28 | 24.01 | 57.04 | 58.56 | 49.96 | 49.13 | 8.11 | 47.70 | 45.08 |
| GLM-4-Plus | 32.43 | 29.36 | 30.90 | 52.74 | 53.74 | 47.03 | 43.11 | 5.41 | 35.18 | 39.53 |
| DeepSeek V2.5 | 40.06 | 39.96 | 40.01 | 43.51 | 45.21 | 35.45 | 32.49 | 0.00 | 27.07 | 30.62 |
| Open-Source Large Language Models |  |  |  |  |  |  |  |  |  |  |
| LLaMA2-7B | 2.14 | 2.56 | 2.35 | 15.14 | 15.27 | 15.00 | 13.69 | 0.00 | 11.44 | 11.76 |
| LLaMA2-70B | 1.39 | 2.08 | 1.74 | 20.23 | 20.74 | 20.54 | 20.04 | 0.00 | 19.19 | 16.79 |
| LLaMA3-8B | 9.25 | 10.57 | 9.91 | 11.45 | 12.60 | 6.26 | 4.33 | 0.00 | 0.68 | 5.89 |
| LLaMA3-70B | 19.70 | 21.64 | 20.67 | 31.74 | 33.13 | 24.86 | 20.63 | 0.00 | 13.02 | 20.56 |
| Qwen1.5-14B | 12.23 | 13.39 | 12.81 | 15.14 | 16.92 | 11.98 | 8.82 | 0.00 | 3.38 | 9.37 |
| Qwen1.5-72B | 19.27 | 18.42 | 18.85 | 26.92 | 27.07 | 21.71 | 18.61 | 0.00 | 13.29 | 17.93 |
| Qwen2-7B | 15.24 | 16.37 | 17.50 | 16.03 | 16.41 | 13.92 | 10.37 | 0.00 | 4.28 | 10.17 |
| Qwen2-72B | 35.12 | 36.13 | 37.14 | 35.05 | 36.83 | 30.72 | 27.52 | 0.00 | 20.68 | 25.13 |
| Mistral-7B | 10.18 | 12.79 | 11.48 | 17.05 | 17.81 | 13.38 | 10.69 | 0.00 | 6.08 | 10.84 |
| ChatGLM3-6B | 4.58 | 6.57 | 5.58 | 16.54 | 16.67 | 13.92 | 12.32 | 0.00 | 9.23 | 11.45 |
| GLM-4-9B | 20.00 | 17.96 | 18.98 | 17.43 | 17.94 | 12.75 | 8.31 | 0.00 | 0.68 | 9.52 |
| Tool-Use-Specific Large Language Models |  |  |  |  |  |  |  |  |  |  |
| ToolLLaMA2-7B | 0.51 | 0.51 | 0.51 | 3.82 | 3.82 | 3.15 | 2.32 | 0.00 | 0.90 | 2.34 |
| MTU-LLaMA(ours) | 39.77 | 38.85 | 39.31 | 35.90 | 36.67 | 34.14 | 27.37 | 0.00 | 14.14 | 24.70 |

##### Error Analysis.

The error cases of GPT-4 results on S-S, M-S, and S-M are enumerated in Tables 11, 12, and 13, respectively.
Additionally, a comprehensive breakdown of error frequencies across different models and error categories is illustrated in Table 14.

**Table 11: Error cases of GPT-4 results on S-S.**
| Type | Description | Case |
| --- | --- | --- |
| Tool<br>Selection | The model incorrectly<br>selects an unintended<br>tool. | User: Please book 5 economy bus tickets from Sacramento to San Diego forthe 10th<br>of this month.<br>Predict: Action: FindBus Action Input: {“departure_date” : “2024-01-10”, …<br>Target: Action: BuyBusTicket Action Input: {“departure_date” : “2024-01-10”, … |
| Paramete<br>Omission | The model omits nece<br>ssary parameters in<br>the results. | User: I need to find an affordable round trip flight from Washington to Mexico for 4<br>people, departing on the 10th and returning on the 13th of this month.<br>Predict: Action: SearchRoundtripFlights Action Input: {“origin_city” : “washington”,<br>“destination_city”:“mexico”, “departure_date”: “2024-01-10”, “return_date”:<br>“2024-01-13”, “passengers” : “4”}<br>Target: Action: SearchRoundtripFlights Action Input: {“origin_city” : “washington”,<br>“destination_city”:“mexico”, “departure_date”: “2024-01-10”, “return_date”:<br>“2024-01-13”, “passengers” : “4”, “seating_class” : “economy” } |
| Parameter<br>Misalignment | The model did not<br>populate the parameters<br>according to the<br>specified criteria. | User: I need to find a round trip flight to NYC from London, leaving day after<br>tomorrow and returning on the 9th, in Economy class.<br>Predict: Action: SearchRoundtripFlights Action Input: {“departure_date” : “2024-01-06”,<br>“return_date” : “2024-01-09”, “seating_class” : “Economy”, “origin_airport” : “London”,<br>“destination_airport” : “nyc”}<br>Target: Action: SearchRoundtripFlights Action Input: {“departure_date” : “2024-01-06”,<br>“return_date” : “2024-01-09”, “seating_class” : “Economy”, “origin_airport” : “London”,<br>“destination_airport” : “New York”} |

**Table 12: Error cases of GPT-4 results on M-S.**
| Type | Description | Case |
| --- | --- | --- |
| Repeated<br>Call | When the information<br>already exists in the<br>previous conversation<br>history, the tool shou-<br>ld not be called<br>repeatedly. | User: Milton Country Park<br>Assistant: …Action: SearchAttraction…<br>Observation: {“name”: “Milton Country Park”, “address”: “milton county park, milton”,<br>“type”: “park”, “entrance_fee”: “free”, “phone”: “765-987554”}<br>Assistant: It’s a free park on the north side of town.<br>User: Could I get the phone number for that?<br>Predict: Thought: The user wants the phone about Milton Country Park,<br>so I need to use the searchAttraction API to get the details.<br>Action: SearchAttraction Action Input: …<br>Target: Thought: The phone number has be received in the history,<br>so there is no need to call any tool.<br>The phone number is 765987554. |
| Parameter<br>Hallucination | The model incorrectly<br>assumes a parameter<br>without sufficient context. | …(Dialogue history)…<br>User: … I need a bus for two people that would be leaving Portland on the 5th.<br>…(Dialogue history)…<br>Assistant: …Action: FindBus…<br>Observation: {“fare”: “39”, “from_location”: “Portland”, “from_station”: “Portland Bus<br>Station”, “leaving_date”: “2024-01-05”, “leaving_time”: “10:20”, “to_location”: “Vancouver”,<br>“to_station”:“Pacific Central Station”, “transfers”: “0”, “travelers”: “2”}<br>User: I want to reserve the hotel room.<br>Assistant: …You would like to check in on 5th?<br>User: I want to check in on the 8th.<br>Predict: Thought: The user has provided a check-in date, which, combined with information<br>from past queries (hotel name and destination), enables the ReserveHotel API to reserve a<br>hotel for the user.<br>Action: ReserveHotel Action Input: {“hotel_name”: “Atrium Inn Vancouver”,<br>“check_in_date”:“2024-01-08”, “number_of_days”: “1”, “destination”: “Vancouver”}<br>Target: Thought: A specific check-in date has been provided, but the duration of the stay is still<br>unknown.For how many days? |
| Parameter<br>Inheritance | The model should inherit<br>the information paramet-<br>ers from the conversation<br>history and update the<br>current state in a timely<br>manner. | User: Is Dumbo playing today in Santa Rosa?<br>Assistant: …Action: GetTimesForMovie…<br>User: Could you get me a taxi there? How long is that?<br>Observation: {“genre”: “Fantasy”, “location”: “Santa Rosa”, “movie_name”: “Dumbo”, “price”:<br>“9”,“show_date”: “2024-01-04”, “show_time”: “22:00”, “show_type”: “3d”, “street_address”:<br>“409 Aviation Boulevard”, “theater_name”: “Airport Stadium 12”}<br>Predict: Thought: The user want to get a taxi to go to the movie theater.<br>Action: GetRide Action Input: {“destination”: “theater”, “number_of_riders”: “1”}<br>Target: Thought: The user want to get a taxi to go to the movie which is at 409 Aviation<br>Boulevard. I can call GetRide.<br>Action: GetRide Action Input: {“destination”: “409 Aviation Boulevard”, “number_of_riders”:<br>“1”} |

**Table 13: Error cases of GPT-4 results on S-M.**
| Type | Description | Case |
| --- | --- | --- |
| Call<br>Fewer Tools | The model fails to call all<br>the necessary tools. | User:<br>I need to set up an appointment with a dentist in Los Angeles who<br>offers cosmetic services on June 15th at 2:00 pm. Afterwards, set me<br>a reminder two hours before the appointment named “Dentist Appointment”.<br>Predict:<br>Action: FindProvider…<br>Action: SetAppointmentReminder…<br>Target:<br>Action: FindProvider…<br>Action: BookAppointment…<br>Action: SetAppointmentReminder… |
| Call<br>More Tools | The model calls<br>additional unnecessary tools. | User: Hi, can you help me find a song of 2019. And modify it as my 3pm<br>alarm’s sound?<br>Predict:<br>Action: LookupSong…<br>Action: GetAlarm…<br>Action: ChangeAlarmSound…<br>Target:<br>Action: LookupSong…<br>Action: ChangeAlarmSound… |
| Call<br>Wrong Tool | The model selects an inapp-<br>ropriate tool for the task, po-<br>tentially leading to incorrect<br>results. | User:<br>I’m looking for a pop music to play and set as my alarm sound for my<br>3pm alarm today. Can you help me with that?<br>Predict:<br>Action: LookupSong…<br>Action: SetAlarm…<br>Target:<br>Action: LookupSong…<br>Action: ChangeAlarmSound |

**Table 14: Error Frequencies across Various Models and Error Types. Bold number signifies the most critical error type for each model. Underlined number represents the most critical model for each error type.**
| Models | Action Error | Parameter Error | Format Error | SUM |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Missed | Excessive | Incorrect | SUM |  |  |  |  |
| S-S |  |  |  |  |  |  |  |
| GPT4 | 0 | 10 | 3 | 13 | 26 | 4 | 43 |
| LLaMA3-70B-Instruct | 2 | 9 | 5 | 16 | 27 | 1 | 44 |
| ChatGLM4-9B | 16 | 4 | 8 | 28 | 35 | 1 | 64 |
| LLaMA3-8B-Instruct | 9 | 3 | 10 | 22 | 38 | 12 | 72 |
| MTU-LLaMA(Ours) | 8 | 6 | 10 | 24 | 26 | 0 | 50 |
| S-M |  |  |  |  |  |  |  |
| GPT4 | 8 | 6 | 4 | 18 | 36 | 0 | 54 |
| LLaMA3-70B-Instruct | 9 | 9 | 13 | 31 | 56 | 4 | 91 |
| ChatGLM4-9B | 28 | 2 | 10 | 40 | 52 | 1 | 93 |
| LLaMA3-8B-Instruct | 8 | 25 | 11 | 44 | 22 | 28 | 94 |
| MTU-LLaMA(Ours) | 8 | 4 | 5 | 17 | 49 | 0 | 66 |
| M-S |  |  |  |  |  |  |  |
| GPT4 | 25 | 18 | 4 | 47 | 96 | 14 | 157 |
| LLaMA3-70B-Instruct | 69 | 22 | 3 | 94 | 92 | 11 | 197 |
| ChatGLM4-9B | 111 | 9 | 9 | 129 | 112 | 40 | 281 |
| LLaMA3-8B-Instruct | 91 | 24 | 17 | 132 | 113 | 101 | 346 |
| MTU-LLaMA(Ours) | 56 | 24 | 3 | 83 | 73 | 30 | 186 |
| M-M |  |  |  |  |  |  |  |
| GPT4 | 11 | 38 | 4 | 53 | 44 | 0 | 97 |
| LLaMA3-70B-Instruct | 8 | 114 | 7 | 129 | 62 | 5 | 196 |
| ChatGLM4-9B | 28 | 145 | 10 | 183 | 38 | 15 | 236 |
| LLaMA3-8B-Instruct | 2 | 207 | 4 | 213 | 6 | 46 | 265 |
| MTU-LLaMA(Ours) | 14 | 103 | 3 | 120 | 28 | 1 | 149 |
| SUM | 511 | 782 | 143 | 1436 | 1031 | 314 | 2781 |

##### Details on the Human Evaluation.

Five annotators with extensive expertise in large language models (LLM) are engaged to assess the prediction outcomes. The annotators assign win/lose scores where a score of 1 represents a victory for GPT-3.5, a score of 0 indicates a tie, and a score of -1 signifies a win for LLaMA3-8B.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: I have included the main questions in weaknesses box and the authors can response according to the comments there.
soundness: 2
strengths: The strengths of this paper are summarized as follows,

- The proposed MTU-BENCH cover more tool-use scenes than previous datasets.

- The evaluation cost of MTU-BENCH is cheaper.

- The authors will open-source all relevant code and data, supporting further research in this field by the community.
summary: This paper proposes a multi-granularity tool-use benchmark for large language models, called MTU-Bench.

The main contribution of this paper can be summarized as,

-  a novel automated data synthesis pipeline is designed to generate high-quality, fine-grained tool-use datasets from pre-existing task-oriented dialogue datasets.

- introduce MTU-Instruct and MTU-Eval.

Comprehensive experimental results demonstrate the effectiveness of the proposed MTU-Bench.
weaknesses: The questions, concerns and weaknesses of this paper are summarized as follows,

- In Section 2.1.1, during the Data Collection phase, the authors should provide a more detailed and comprehensive list of the specific criteria and standards used for dataset selection.

- There appear to be situations where multiple classification criteria, such as 'Information missing,' 'Information exists,' 'Information confirmed,' 'Aimless chatting,' and 'Specific API call,' could apply simultaneously. How should these cases be handled?

- Any visualization results with specific examples can be shown for Tool Clustering in Section 2.1.1?

- Is data quality truly assured? Since the data is synthesized by GPT-4 and also validated by GPT-4, can the reliability of the synthetic data be guaranteed?

- The overall presentation of Section 2.1.1 is not very strong, and many details are not clearly explained (such as quality filters and adjustments). The authors should refine this section thoroughly.

- The content in Section 2.1.2 does not fully align with what is presented in the introduction. The authors should add a reasonable comparison with previous datasets at an appropriate place in the paper.

- Could the authors provide some experimental results that train other open-sourced LLM on MTU-BENCH?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.You classified the types of model errors (such as operation errors, parameter errors, format errors), but did not delve into the specific reasons why the errors occurred or possible resolution strategies. Can the model's specific error patterns under different task complexity or tool combinations be further analyzed to help improve the model?

2.As a binary indicator, the success rate requires that the entire conversation is completely error-free to be successful, which may be too stringent for multi-round conversations. In practical applications, some small errors may not significantly affect the completion of the final task, especially in scenarios where users can tolerate partial mistakes. Such strict standards will cause the model to be marked as a failure due to small errors even at high performance, failing to reflect the overall effect of the model. Therefore, is it necessary to introduce a fault-tolerant mechanism for SR, such as allowing a small number of insignificant errors to exist?

3.SATS uses an exponential decay method to reduce the impact of early errors on subsequent rounds. While this decay mechanism captures the temporal location of errors, it may not be effective enough to cope with the impact of different types of errors. For example, some errors (such as parameter errors) may invalidate the entire conversation, while others (such as minor tool selection errors) have less impact on subsequent conversations. Is it possible to incorporate error type and severity into the decay function to get a more precise round success rate?

4.Current multi-round evaluation metrics do not differentiate between the type of error (e.g. tool selection error, parameter selection error, etc.) and severity. However, different error types have significantly different effects on dialogue. For example, parameter errors often have a greater impact than tool selection errors because parameter errors can lead to complete failure of the task. Therefore, should error type and severity be included in the assessment and given different weights, thereby improving the accuracy of the assessment?
5.In order to ensure the validity of the results in real applications, are there any plans to introduce a part of real human-labeled data and compare the performance difference of the model on real data and synthetic data?
soundness: 3
strengths: MTU-Bench introduces a unique multi-granularity benchmark for tool-use in LLMs, covering single/multi-turn and single/multi-tool tasks, addressing limitations of prior benchmarks with a cost-effective, automated evaluation approach. It includes detailed metrics (e.g., tool and parameter selection accuracy, task process rate) that provide deep insights into model performance across diverse scenarios, adding rigor to the evaluation process.
summary: This paper presents MTU-Bench, a benchmark designed to evaluate large language models (LLMs) in tool-use across diverse and complex dialogue settings, including single-turn, multi-turn, single-tool, and multi-tool tasks. MTU-Bench addresses limitations in existing benchmarks by incorporating automated, cost-effective metrics that do not require GPT-based evaluations. Key contributions include a large dataset of tool-use dialogues synthesized and validated with GPT-4, a detailed evaluation framework (MTU-Eval) with fine-grained metrics, and the introduction of MTU-LLaMA, a model fine-tuned for tool-use tasks that shows strong performance. This work provides a comprehensive benchmark that captures real-world complexities, supporting future advancements in tool-using LLMs.
weaknesses: 1.The paper’s result analysis could be more comprehensive. While it presents performance comparisons across different models and scenarios, it lacks deeper exploration into the causes behind inconsistent model results, particularly in challenging multi-tool and multi-turn settings. A more granular investigation into factors such as specific error patterns, model architecture differences, or the influence of training data could provide actionable insights. This would help identify underlying reasons for performance variability and guide targeted improvements in model design and training strategies.

2.The paper does not indicate whether the experiments were conducted multiple times or if statistical confidence measures were applied to the results. Without multiple runs or confidence intervals, the stability and reliability of the reported results are uncertain, particularly in complex, multi-turn, multi-tool scenarios where model performance can vary significantly. This omission limits the ability to assess whether observed differences between models (e.g., GPT-4 vs. MTU-LLaMA) are statistically significant or simply due to random variation. Conducting multiple experimental runs and reporting average results with confidence intervals would strengthen the reliability of findings and clarify performance comparisons across different settings.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: The paper notes that existing benchmarks lack sufficient evaluation scenarios. Your proposed benchmark seems to merely add a few extra scenarios—how does this impact tool usage? Are there fundamental differences between the new scenarios and the previous ones? Without strong experimental evidence, it may appear that you are simply expanding the dataset. If your goal is to build a comprehensive evaluation suite, it seems to lack a thorough analysis and discussion of its completeness.

Your paper claims that previous datasets were not based on real-world data, yet the dataset you present is constructed using GPT-4 and existing datasets, rather than data collected from actual application users. How do these data fundamentally differ from previous datasets in accurately reflecting real-world scenarios?

Although your dataset is more detailed and extensive than previous ones, it remains unclear which specific challenges it addresses. Could combining existing evaluation datasets achieve similar results? What unique value does your benchmark provide that makes it indispensable for evaluating tool usage?
soundness: 3
strengths: Comprehensive Evaluation: The paper provides a thorough evaluation of both open-source and proprietary models using the newly proposed benchmark, covering a wide range of scenarios.

Detailed Experimental Setup: The experiments are well-designed and extensive, allowing for a clear comparison of LLMs' tool-use capabilities.

Improved Scenario Coverage: By incorporating multiple scenarios, including multi-turn and multi-tool settings, the benchmark offers a more nuanced evaluation of LLMs, which is a step forward from existing benchmarks.

Clarity and Structure: The paper is well-structured, making it easy to follow the methodology and understand the results.
summary: This paper introduces MTU-Bench, a benchmark designed to evaluate large language models (LLMs) in terms of their ability to use tools across various scenarios. MTU-Bench addresses limitations in existing benchmarks by providing more comprehensive scenario coverage and reducing evaluation costs. It includes five distinct tool usage scenarios and relies on prediction results and ground truth for evaluation. The paper's key contributions include the MTU-Bench benchmark, the MTU-Instruct and MTU-Eval datasets, and the MTU-LLaMA model, which is fine-tuned to demonstrate strong tool-use capabilities. The experimental results highlight the benchmark's effectiveness in enhancing LLMs' tool-use skills.
weaknesses: Limited Novelty: While the benchmark offers more scenarios and finer-grained evaluations, it lacks a significant innovation or breakthrough that fundamentally advances the field. The paper needs to clearly articulate how these additions lead to new insights or directions in tool-use capabilities for LLMs.

Lack of Real-World Impact: The paper does not provide concrete examples or case studies demonstrating how the new benchmark can lead to improvements in real-world applications. For example, the introduction of the COCO dataset in the object detection field highlighted specific challenges that state-of-the-art methods at the time struggled with, such as detecting small objects, handling occlusions, and recognizing a wider variety of categories. This enabled researchers to evaluate and improve their models effectively. In contrast, MTU-Bench does not clearly show how it highlights current limitations of LLM tool-use or how it can similarly drive innovation in practical applications.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: 1.How to calculate the Parameter Selection Accuracy in single-turn scenarios? Specifically, is it determined solely by whether the LLM's output matches the ground truth exactly? There are instances where the parameters provided by the LLM might convey the same meaning as the ground truth but differ in specific characters or formatting. Would a more nuanced approach to assessing parameter selection accuracy be beneficial? Do you use the same way of defining correctness in multi-turn scenarios?

2. Since the tools called in MTU-Bench are all synthetic and do not use the real-world API, do you have any experiments that show that the metrics used in the paper are consistent across models when applying datasets built from real-world API calls?

3. In tool creation, how do you make sure that the tools you create make sense? Besides, since the tools are merged in the latter stage, how can you guarantee that the tool provided can fit the request by the current turn? 

4. Most API doc are generated from GPT-4, which indicates that the distribution of API docs possibly falls into the distribution into the GPT4 capability distribution, how to overcome such bias?

5. Limited discussion of current tool-calling benchmarks
[1] T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step, ACL 2024
[2] GTA: A Benchmark for General Tool Agents, NeurIPS 2024

6. Considering that this is a benchmark paper, the authors are highly suggested to provide the dataset data in the supple files, which allows the reviewer to have a glance at the quantity of the dataset in person, which is different from the dedicated demos selected in the supplementary. 

7. The author shows the performance gain on the MTU-Eval, however, it is actually an in-domain SFT. What is the performance gain on another tool benchmark?
soundness: 2
strengths: 1. Originality: MTU-Bench introduces a multi-granularity evaluation framework, considering the number of tools invoked and the number of turns required for interaction. This multi-granularity framework captures a variety of real-world interactions that are often overlooked in existing benchmarks. The construction of this benchmark employs a unique synthesis pipeline by transforming task-oriented dialogue datasets into tool-usage datasets.

2. Quality: The paper evaluates multiple LLMs (both open-source and closed-source) across a range of scenarios, including multi-tool, multi-turn, and out-of-distribution (OOD) tasks, ensuring robustness and completeness in the experimental validation.

3. Clarity: The paper follows a clear narrative, starting with motivation and problem formulation, followed by methodology, experiments, and detailed discussions on results, which makes it easy for readers to follow the flow of ideas.

4. Significance: The paradigm of transforming existing dialogue datasets into tool-use datasets opens new possibilities for exploring the tool-use ability of LLMs. The automated evaluation framework MTU-Eval significantly reduces the cost and complexity of benchmarking LLMs
summary: This paper proposes MTU-Bench, a benchmark dataset for evaluating the ability of a large language model(LLM) to invoke tools in multiple scenarios. MTU-Bench provides a more granular and detailed approach compared to previous studies in this domain by considering two key dimensions: (1) the number of tools that can be invoked within a conversation and (2) the number of rounds of tool call involved in multi-turn dialogues. Moreover, the construction pipeline for the MTU-Bench dataset is novel and carefully designed. It begins by collecting tasks from traditional conversation datasets, and these datasets are then transformed through a synthesis process into tool-oriented conversations, simulating realistic tool usage. This innovative approach serves as a scalable paradigm for expanding both the variety of tools and the diversity of conversations available for research. In addition, the paper proposes MTU-Eval, an automated evaluation framework that does not require an external LLM, which reduces the cost of evaluation to a large extent. Finally, the MTU-Instruct dataset is introduced for fine-tuning the tool-usage capabilities of the model, demonstrating the excellent performance of the fine-tuned model in a variety of complex tasks.
weaknesses: 1. Unable to cover complex tool-calling scenarios: The idea of the paper is to convert the dialogue into a function call dataset, which is an interesting and extendable idea. However, this also introduces a bottleneck that the dataset may not cover too much complex queries that request the agent to perform more than 5 steps to finish (the paper also states that only an average of 2 calls per turn in the dataset). 

2. Limitations of Virtual APIs: MTU-Bench generates all tools based on traditional conversation data, resulting in all APIs being fictitious. This approach may make LLM behave severely differently from when it calls real-world API, for example, if the model generates a wrong call in an executable env, it has the possibility to recover from its mistakes. (However, the reviewer does not expect this problem to be solved in this setting, but it would be good if the author could propose some nice practical thoughts to that)

3. Insufficiently Designed Evaluation Metrics: SATS and TPR overly emphasize the position of the first error, failing to adequately consider the LLM's ability to handle early mistakes. For example, if an LLM can disregard an early error and correctly execute subsequent tasks, it demonstrates robustness that the current metric does not capture.

Meta-review: The paper introduced a new benchmark MTU-Bench for evaluating the LLM's abilities to use tools in multiple scenarios. MTU-Bench considers more granular settings in aspects of the number of tools that can use and the number of rounds of tools can use in multi-turn conversation. The authors also provide a pipeline to construct such usage scenarios from existing high-quality datasets. 
They also present MTU-Eval for automated evaluation and MTU-Instruct for tool use fine-tuning.

The proposed multi-granularity tool-use benchmark is new and useful. It provides a practical, i.e., low cost, and comprehensive evaluation of existing tool usage capabilities. The paper provides a wide evaluation and analysis of multiple LLms across scenarios.

The rebuttal has addressed most of the concerns and was acknowledged by several reviewers. The only reviewer who gave a rating of 5 didn't respond and I believe most of the concerns can be addressed by the rebuttal and other reviews.

I recommend accepting this paper. This work can help push the development and exploration of tool-use capabilities of LLMs, which is important for future LLM applications.

Decision: Accept (Poster)