MTU-Bench: A Multi-granularity Tool-Use Benchmark for Large Language Models
Authors: Pei Wang
Yanan Wu
Noah Wang
Jiaheng Liu
Xiaoshuai Song
Z.Y. Peng
Ken Deng
Chenchen Zhang
JiakaiWang
Junran Peng
Ge Zhang
Hangyu Guo
Zhaoxiang Zhang
Wenbo Su
Bo Zheng
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Tool-usage
Benchmark
Primary area: datasets and benchmarks

Abstract: Large Language Models (LLMs) have displayed massive improvements in reason- ing and decision-making skills and can hold natural conversations with users. Recently, many tool-use benchmark datasets have been proposed. However, existing datasets have the following limitations: (1). Insufficient evaluation scenarios (e.g., only cover limited tool-use scenes). (2). Extensive evaluation costs (e.g., GPT API costs). To address these limitations, in this work, we propose a multi-granularity tool-use benchmark for large language models called MTU-Bench. For the "multi-granularity" property, our MTU-Bench covers five tool usage scenes (i.e., single-turn and single-tool, single-turn and multiple-tool, multiple-turn and single-tool, multiple-turn and multiple-tool, and out-of-distribution tasks). Besides, all evaluation metrics of our MTU-Bench are based on the prediction results and the ground truth without using any GPT or human evaluation metrics. Moreover, our MTU-Bench is collected by transforming existing high-quality datasets to simulate real-world tool usage scenarios, and we also propose an instruction dataset called MTU-Instruct data to enhance the tool-use abilities of existing LLMs. Comprehensive experimental results demonstrate the effectiveness of our MTU-Bench.

The circles from inside outside represent the data source, scenes included in MTU-Bench. and the automatic evaluation met- rics in MTU-Eval. Published as a conference paper at ICLR 2025
MTU-BENCH: A MULTI-GRANULARITY TOOL-USE BENCHMARK FOR LARGE LANGUAGE MODELS
Pei Wang'a ' Yanan Wu Zekun Wang', Jiihenng Liu
Xiaoshuai Song', Zhongyuan Penglis, Ken Deng', Chenchen Zhang', Jiakai Wang', Junran Peng", Ge Zhang', Hangyu Guo', Zhaoxiang Zhang', Wenbo Su Bo Zheng' Allibaaa Group, "Nanjing University, University of Chinese Academy of Sciences, University of Waterloo (yupei. wp, lixing. wyn, 1jh411989 @alibaba-ine com
ABSTRACT
Large Language Models (LLMs) have displayed massive improvements in reason- ing and decision-making skills and can hold natural conversations with users. Recently, many tool-use benchmark datasets have been proposed. However, existing datasets have the following limitations: (1). Insufficient evaluation scenarios (e.g., only cover limited tool-use scenes). (2). Extensive evalua- tion costs (e.g., GPT API costs). To address these limitations, in this work, we propose a nulti-granuulartyy tool-use benchmark for large language models called MTU-Bench. For the "multi-gran "multi-granularity" property, our MTU-Bench covers five tool usage scenes (i.e., single-turn and single-tool, single-turn and multiple- tool, multiple-turn and single-tool, multiple-turn and multiple-tool, and out-of- distribution tasks). Besides, all evaluation metrics our ATU-Bench are based the prediction results and the ground truth without using any GPT hu- man evaluation metrics. Moreover, our MTU-Bench collected by transforming existing high-quality datasets simulate real-world tool usage scenarios, and we also propose an instruction dataset called MTU-Instruct data to enhance the tool-use abilities of existing LLMs. Comprehensive experimental results demon- strate the effectiveness of our MTU-Bench. Code and data will be released at Tew--MUU-/eee/eeMeeeeeee MM//-BMCUUh...//- git.
bttps://githuu: Bwwoo//TU-BBBnc---ee
INTRODUCTION
Since the release large language models (LLMs) such as GPT-4 (OpenAl 2023), Natural Language Processing (NLP) has entered a new wave of advancements, even being considered the spark of Artificial General Intelligence (AGI) (Bubeck) let al.) 2023). Recently, there has been a surge of research focused enabling LLMs interface with external tools, such as calculators (Cobbe et al., 22021) search engines (Schick al. (023), and booking service APIs QQn al.) 2023b). This approach, referred to as Tool learning (Schick et al.) 2023 Qin al.) 2023a b) Waan et al.) 2023), allows L.L.Ms to not only accurately perform precise ca calculations, but also maintain up-to- date information. Furthermore, it enables LLMs to function as end-to-end AI assistants that are capable of fulfilling real-world user needs such as booking hotels or ordering food. Thus, Tool Learning is critical step to transform LLMs into general AI agents.
TPR
M-$
M-M
10
Mlunwoz
TS
SNIPS
Nis
MTU
S.
5.5
NW
385
4OD
SAT
SIV
Figure I:
Previous works have explored stimulate the ability call tools for LLMs (Schick et al. 2023 Qin et al.) 2023b Zhuang ei al.
al. 2023
Tang et al. 2023 Paranjape et al. 2023
First three authors contributed equally.
Corresponding Author: Jiaheng Liu.

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of various tool-use benchmark datasets. "Auto. Eval." denotes "automatic evaluation without GPT". "S-S", "S-M", "M-S", and "M-M" denote single-turn single-tool, single- turn multi-tool, multi-turn single-tool, and multi-turn multi-tool, respectively. "OOD" refers to whether the benchmark accounts for out-of-distribution setting, where the test set consists of domains different from those the training set. "Real-World" means whether the dialogues in the benchmark are sampled from weal-world scenarios. The numbers the evaluation range mean: tool selection,
2 parameter selection, dialogue-level success rate, turn-based success rate, tool number, and 6 tool order.
Real- World
Auto. Eval.
iDinlogues
#Turn-VTool
Range
Train
Text
OOD
MotaTool
21.127
199
S-S. S-M
V
Huang etal. 2024
API-Bank
S-S, S-M.
al
2,202
2,211
M.S,
X
v
!
ToolLLM
Qin et al
20236
12,657
16,464
S.S.
X
X
e
4
API-Bench
17,002
1,645
S-S
X
V
23
x
Patil al.
2023
ToolAppaca
Tang et al.
2023
3,938
400
S-M. M.S. M.M
X
133
x
TooiQA
1,530
S.S.
23
/
Zhuang ot al
2023
T-Eval
23,305
15
S-S.
X
4
13
Chen et al.
GTA
WWan
2024
229
S-S.
1
X
x
(Ours)
MTU-Bench
54,798
136
M-S, M-M
S-S, S.M.
/
2023). For example, (Schick et al., (203)) propose convert
tool calls into text spans, such as < <API> PARAMETER
KEY-VALUE PAIRS</API> to denote the tool name and parameters with an additional special
token (i.e., </API>) to show the initiation of a tool execution. Moreover, the recent works (i.e..
ToolBench (Xu et al. 2023 Qin et al.) 20236). APIBench (Patil et al. 2023), and API-Bank (Li et al.
2023 have investigated instruction tuning data or evaluation for tool-use.
However, we observe that exhibit several limitations to varying degrees as shown in Table (1) some do not consider multi-turn dialogue scenarios (Patil et al.] [023 Xu et al. 2023) Zhuang et al. 2023) (2) some do not address multi-tool usage scenarios (Tang al.] 2023 Patil et al. [202]] Li al.)
2023 Xu et al.) 2023); (3) several works use external API tools to deduce user instructions, but these
synthesized instructions often do not accurately align with actual real-world user needs (Qin et al.
2023b;; (4) many of them rely GPT for the evaluation, leading to heavy evaluation costs (Qin ct al. 2023b; Tang et al. 2023); and (5) many do not comprehensively assess fine-grained aspects of tool-use (Li et al.) 2023; Qin et al. 2023b Patil al. 2023), such as the accuracy of tool call orders, complex tool calls involving inheritance relationships, per-dialogue turn accuracy of tool and parameter selection, etc.
To remedy these issues, in Figure [ we introduce MTU-Bench (Multi-Granularin Tool-Use Benchmark), which comprises both MTUUInstruct for training and MTU-Eval for evaluation. As illustrated in Figure we sample real-world user instructions from various existing open-source dialogue datasets such as MultiWOZ (Budzianowski al. (0188) and SGD (Rastogi et al.) 2020al] Lee et al. 2022). After instruction clustering, the detected user intents and slot filling are leveraged to synthesize API calls using GPT-4 (OpenAl 2023). The synthesized data includes the thoughts, the actions (i.e., tool names), the action parameters, and the observations (i.e., the generated API execu- tion results). This data forms our MTU-Bench dataset. Following meticulous quality verification by GPT-4 and manual check, we split the MTU-Bench data into training and testing splits, involving 54798 dialogues in total, as well as 136 tools. In our MTU-Eval, we propose a series of fine ane-grained metrics such tool selection accuracy, parameter selection accuracy, success rate, turn success rate, task process rate, tool number accuracy. tool order accuracy, etc., to evaluate the tool-use abilities in comprehensive manner, where the GPT API costs not needed for evaluation. Moreover, also
2

---

Published as a conference paper at ICLR 2025
pick out hard subset from the test split to include more complex tool-use scenarios such easily confusable tools, nonsensical or noisy tools, tool parameter updating, etc.
Finally, by fine-tuning LLaMA-3 (Dubey et al., 2024) on MTU-Bench, we find that our result- ing model, MTU-LLaMA, performs the best in various scenarios and metrics, demonstrating the effectiveness of our ITU--nstruc.
In summary, our contributions are as follows: (1), MTU-Bench: We introduce a novel automated data synthesis pipeline designed to derive high-quality, fine-grained tool-use datasets from pre-existing task-oriented dialogue datasets. This pipeline facilitates the creation of MTU-Bench, comprising MTU-Instruct for training purposes and MTU-Eval for evaluation. (2). ITU--nntructt and MTU-
Eval: We introduce the high-quality and diverse instruction tuning dataset, MTU-Instruct, to improve models' tool-use capabilities in real-world scenarios. Additionally, we propose a novel automatic evaluation framework, MTU-Eval, which assesses various tool-use settings through comprehensive and fine- -ne-grained metrics, free of GPT-based evaluators. (3). MTU-LLaMA and Experimental Findings: After instruction tuning on MTU-Instruct, we obtain a strong open-source model for tool- use, MTU-LLaMA, Our comprehensive experiments reveal several findings regarding the tool-use capabilities of LLMs, particularly in terms of multi-turn dialogue scenarios, multi-tool settings, and error cases. These findings offer valuable insights for advancing tool-use in LLMs.
2
MTU-BENCH
The MTU-Bench involves both MTU-Instruct for training and MTU-Eval for evaluation. We first present the data construction and analysis in 1 and then show the evaluation procedure in 2.2
2.1
MTU-INSTRUCT
Dialogue Datasets
2
Intent Detection and Slot Filling
Toal Documentation
Tool-Use Date Synthesis
Toil miaama Startiedvee
MuleWOz
SNIPS
Fallngg
Deecroptinn Cearch revva intrradoo
Bez madchrrnni vich and Morshingg dant..
hhsplion Kange"o
TaskMaster
SimDial
Aloe:
into Fienc'
Jury 1 soul)
Rletrre: leve (ncoa)
S
are sokins avalable for Fiiday right?
Alic ward witch Irosption
Tool Making
X
Tool Clustering
Tool hrme Mooclica
taee deele tade Fickey
parr want chapptcc
That Descripter: for specc/cc a11 senche 1911.
Ea on pamaat "hcappann
1907
Ti
Mock
endil happora be nearnsl
T2
bocied
T3
Friday
ba the basvel
Figure 2: The workflow for MTU- construction. It involves five steps: (1) Data Collection, (2) Tool Creation, (3) Tool Clustering, (4) Tool Documentation, and (5) Tool-Use Synthesis.
2.1.1 DATA CONSTRUCTION
Many previous methods based scraped API documentation and GPT-4 inspired synthesis faced limitations due to a lack of data diversity, resulting in weak generalization capabilities. Inspired by the mapping relationships between intents and APIs, well between slots and tool parameters, we designed an automated data synthesis pipeline that transforms traditional dialogue datasets into tool-use datasets. To enhance data diversity, we collected datasets from multiple NLP datasets and standardized them into a unified tool documentation. The diversity is illustrated in Figure 3 As shown in Figure the construction of MTU-Bench involves five primary steps: (1) collecting task-oriented dialogue datasets, particularly those containing intents and slots, (2) creating tools through grammar- based transformations or GPT-4 synthesis, (3) clustering the tools based on their similarities, (4) generating tool documentation using GPT-4 and (5) synthesizing tool-use samples consisting of thoughts, actions(tool calls), action inputs, observations, and adjusted responses based on the dialogue and tool documentation, followed by a holistic validation process.
Data Collection. To improve the diversity of our dataset, we collect several open-source task- oriented dialogue datasets our data sources. These datasets focus on dialogues for specific tasks
3

---

Published as a conference paper at ICLR 2025
such flight reservations or movie bookings, which are highly suitable for synthesizing tool-use data. The multi-turn dialogue datasets include MultiWOZ (Budzianowski et 201122018. SGD Raatogii
et al,, 2020b), TaskMaster (Byrne et al. 2019) and MetaLWOZ (Shalyminov et al. 2020). The single-turn dialogue datasets include AT'S (Hemphill et al. 1990) and SNIPS (Siddhant et al. 2018).
They provide diverse task-oriented dialogues across various domains, real-world conversation and fine-grained annotation, encompassing both single-turn and multi-turn dialogues.
Tool Creation. We employ two approaches to create tools. (1) Grammar-based creation. For dialogue datasets that already have detected intents and filled slots, we directly convert the in- tents into tool names and the slots into tool parameters. For example, in the user query "find a flight from charlotte to las vegas". the intent "Flight" will convert to the tool name, and the slot "from_location location=chariotte to lo_location=ls vegas" " will convert to the parameters, resulting in the tool call FLIGHT(FROM "OCATTIN="CHARLOTTT
,TO_ NOCATION="LAS VEGAS"), )) (2) LLM-based creation. For dialogue datasets without predefined intents or slots, we utilize GPT-4 to make the tools. Based on the contextual situation, we categorize it into five scenarios: information missing, information exists, information confirmed, aimless chatting and specific API call. Provided with the historical dialogue context and the current round of conversation, especially the assistant's 's response. LLM needs to determine which situation belongs to the current situation. Information missing. If the response is asking for important information, should he the situation of missing information, no tool call should be made, and necessary parameters should be accumulated for related tools. Information exists. When the LLM can provide a response based solely on the information from the dialogue history, tool call will be made and the model can directly reply. Information confirmed. When the assistant is confirming information (e.g., Would you like to confirm this flight reservation?), ?)) this is classified as "information to be confirmed". Aimless chatting. If the scenario pertains to aimless chatting or situations that do not necessitate tool invocation, no tool call should be made. Specific API call. Only if the LLM determines that API call is necessary fulfill the user's request, it is encouraged to generate an appropriate pseudo-tool for invocation, along with a corresponding description and parameters of this tool.
Tool Clustering. Due to the diversity in both the intent detection and slot-filling strategies, as well as the creation of specific tools based on LLM, the synthesized tool set can be highly redundant. For example, tools like "search movic" and "find movic" may have different names but essentially perform the same function. To address this redundancy, we introduce a tool clustering phase. Specifically, we cluster the tool names based on Ins Tag(Lu al. 2023) with a fixed distance threshold. Then, all tool names and their parameter names are standardized to the centroid of their respective cluster, resulting in reduction ratio of 20:1.
Tool Documentation. To enable models to use specific tools effectively, we compile all tool usage into a comprehensive tool document. This document allows the model to determine the appropriate tool names and their usage. We prompt the GPT-4 model to write description for each tool generated in the previous step, along with information about its parameters(required and optional) and returns. The collection of these tool entries forms the final tool document, which is included as part of the LLMs' context. Please refer to Appendix B for the details.
Tool-Use Data Synthesis. In this step, we convert all samples from task-orienter dialogue datasets into tool-use dataset with GPT-4, following the format of ReAct (Yao et al. 2023). We provide the dialogue history and the tool document generated in the previous steps as context for GPT-4, and then prompt it to generate three key components: (1) thought: the reasoning process behind the tool selection, (2) action: the name of the tool being invoked, and (3) action input: the parameters used in the tool call along with their values. This chain of thought prompting technique enhances the model's s ability to reason over the most appropriate tool and accurately input the parameter values. We also allow the model to generate any additional parameters needed that are beyond those listed in the tool document, to ensure completeness and flexibility of tool-use.
We further ask the GPT-4 to simulate tool execution, generating observation (i.e., the results of the simulated tool execution) and then produce the final response for the current dialogue turn. The observations are aligned with the return information in the tool document and are generated in structured format, such as a JSON dictionary. The model then formulates a response based on the observations, either report the status of the tool execution to complete the dialogue tur.
4

---

Published as a conference paper at ICLR 2025
To ensure data quality, we apply various quality filters and adjustments, including heuristic rules, GPT-4, and manual annotation. For the training set, we use GPT-4 to verify the accuracy and necessity of tool selection, check parameter matching, adjust thoughts, rewrite response and ensure consistency in the tool definitions. Through GPT-4, we filter out about 10% of defective samples. For the test set, we hired multiple experts conduct manual quality checks based on similar principles. Fach sample was checked by three experts, and the differences in labeling were determined by the fourth expert.
Based on the number of dialogue turns and the number of tools in each dialogue, the synthesized data can be categorized into four types: (1) Single-turn Single-tool (S-S), (2) Single-turn Multi-tool (S-M), (3) Multi-tum Single-tool (M-S), and (4) Multi-tum Multi-tool (M-M).
For more detailed information about the construction of MTU-Bench, including prompt templates and the tool documentation, please refer to the Appendix
2.1.2 DATA ANALYSIS
Domain
#Topics #Toels
#Dialogues (Train/Test)
Place
27
29,667 / 150
Media
2
18
13,730 07200/9
getpput thesutss
Traffic
17,720 /89
abondt consttrmtio
per Tolees
Daily Life
23.706
Bus
ausiiior
aetion aetiorenieerpeeee eelerrantee
Money
10
1,740/32
Treih
Donese
Antoraation
Hombe
aqjutt lighting
Information
12
9,125 11225/75
Rerter
Prantucation
Disease
2
3,371 / 125
Acdi teet
Others
-/
Contec
Additional Statisties
Alam gettane
ged
#Dialogues
54,367 /431
(Train/Test)
Cotterdder
#Tools
122 / 14
(Train/Tst))
Avg. Turns
Dialogue
Avg.
Dialogue
Tools
Tur
Figure 3: The domain distribution of tools.
Figure 4: Statistics of MTU-Bench.
Statistics. Figure shows the statistical metrics. Our MTU-Bench is particularly designed to include multi-too and multi-turn settings, as well as real-world domains and topics.
Diversity. Figure illustrates the domain and topic distribution of the tools within our MTU-Bende Figures and [T depict the distributions of dialogue turn counts, word counts, tool numbers, and other length-relane metrics across both training and testing splits. These figures underscore the diversity of our MTU-Bench in terms of length and topic distribution.
We refer the readers Appendix for a more detailed data analysis.
2.2 MTU-EVAL
We propose MTU-Eval the first evaluation framework that encompasses multiple levels of difficulty, diverse domains and cases of tool-use, varying numbers of dialogue turns and tools, well as multifaceted evaluation by considering various granularities and aspects of LLM tool-use.
We present MTU-Eval in two parts: (1) Test Set Splitting and (2) Evaluation Metrics.
5

---

Published as a conference paper at ICLR 2025
2.2.1 TEST SET SPLITTING
We construct two distinct test sets from MTU-Bench through manual sampling: (1) the normal test set and (2) the more challenging hard test set.
Normal/Hard Test MTU-Bench includes data from 31 different topics, such as weather-related and calendar-related tasks. Initially, we select data from 5 topics as an Out-of-Distie citto--Distribution (OOD) test split. From the remaining 26 topics, we further split the data into training set and an in-domain test set. For more challenging evaluations, we manually curate a hard test set, which includes more complex tool-use cases, such those involving long tool parameters, easily confusable tools, parameter value updating, scenarios with a large number of tools, etc., as listed in Appendix D
2.2.2 METRICS
For Single-Tool Scenarios (S-S & M-S). For scenarios where there is just a single tool involved, we evaluate two metrics: Tool Selection Accuracy (TS) and Parameter Selection Accuracy (PS).
For Multi- Muuli-TTurn Scenarios (M-S & M-M). In multi-turn dialogues, the following metrics are intro- duced: (1) Success Rate (SR): A binary metric where the entire dialogue is considered successful for
tool-use: (=1) only if there are no errors throughout all turns; otherwise, it is considered unsuccessful (=0). (2) Averaged Turn Success Rate (ATS): We first evaluate each dialogue turn with the tool-use success rate (where each turn is marked as either 0 or 1), then average the binary scores with the total dialogue turns of a dialogue session. This score takes into account the finer-grained success rate of tool-use at the turn level. (3) Soft Averaged Turn Success Rate (SATS): This metric adjusts the ATS based on the proximity of errors to the current turn. Specifically: If a turn is incorrect, the score is O. If turn is correct, given j as the index of this turn and as the index of the last incorrect turn, the score is 1 when j < and 1 e (--i) when j > i. This design is based on the intuition that an incorrect earlier turn can negatively impact subsequent turns. Moreover, the earlier the turn becomes incorrect, the lower the overall accumulated score, even if the remaining turns are correct. (4) Task Process Rate (TPR): This is calculated as the ratio of the first incorrect turn to the total number of turns. This metric is included to capture how early in the dialogue the first mistake occurs, as carlier errors tend to disrupt the overall task flow more significantly.
For Multi- Tool Scenarios (S-M & M-M). For scenarios with multiple tools, the following metrics used: (1) Tool Number Accuracy (TN): Denote the predicted tool list "Pred" and the ground truth tool list as "GT", TN- =| Pred
PredUGT J. (2) Tool Order Accuracy (TO): Evaluate the correctness of the tool sequence, adjusted by a decay factor: TO tx LCR(GT, Pred) / GT J. where LCR is the longest common subsequence, and / is a decay coefficient calculated as: t = cos ((T /2) x (i/ (red)) where is the starting position of the longest common subsequence. The value of t ranges from to 1, with a faster decay for positions later in the sequence.
Unlike conventional approaches that focus only on overall success rates QQin al.) (023b), our metrics account for the dynamics along dialogue turns and the dependencies between multiple tools. We refer the reader to Appendix for more examples of how to compute these metrics.
3 EXPERIMENTS
Experimental Setup. We evaluate 5 closed-source LLMs such GPT-3.5 (?). GPT-4 OpenAI
2023). Qwen-Max (Team 2024), GLM-4-Plus and DeepScek2.5 (DeepSeek-Al 2024). We also eval- uate numerous open-source LLMs such as LLaMA2 (Touvron et al. 2023) and LLaMA3 (Al の Metal
2024) series, Qwen 1.5 Team 2024) and Qwen2 ( YYang et al. 2024) series, Mistral (Jing et al. 2023
ChatGLM3 and GLM-4 (GLM et al. 2024 Du et al. 2022 Zeng al. 2022) series, well as Then, we also compare 2 models specifically enhanced for tool-use: ToolL LaMA (Qin et al.) 2023b) and our MTU-LLaMA, which is fine-tuned on MTU-Instruct based on LLaMA3-8B-Instee
Note that all baselines are instruction-tunee models. We refer the reader to Appendix D for more details about the evaluation including the hard cases, the prompt templates, and the metric computation.
https: gmodel. www.d///ddv.l...eeeee /nc hwtp///mmaalmlmmm -modee //11m-

---

Published as a conference paper at ICLR 2025
Table 2: Results of different models on the normal set of MTU-Eval (S-S & M-S). "S-S" and "M-S" denote "Single- Turn Single-Tool"" and "Multi-Turn Single-Tool" settings, respectively. We utilize green (Ist), blue (2nd), and yellow (3rd) backgrounds to distinguish the top three results within both open-source and col-use-speciife models. We employ bold and underlined text to denote the top and second-best results across all model categories (same markers for the other tables). All the baselines are instruction-tuned models.
S-S
M-S
Models
TS
PS
Avg.
TS
PS
ATS
SATS
SR
TPR
Avg.
Closed-Souree Large Language Models
GPT-4
95,83
52.08
73.96
88.10
74.49
73.67
67.36
29.63
45,35
63,10
GPT-3.5
84.62
46.15
65.38
69.05
50.68
50.51
40.81
1.85
12.70
37.60
Qwen-Max
91.67
50.00
70.83
86.73
66.67
64.96
57.88
20.37
35.53
55.36
GI-M-4-Plus
95.83
50.00
72.92
85.03
71.43
72.01
65.19
24.07
44.18
60.32
DeepSeck V2.5
93.75
43.75
68.75
86.39
69.39
68.09
60.20
18.52
38.04
56.77
Open-Source Large Language Models
LLaMA2-7B
15.38
3.85
9.62
33.67
28.91
26.78
20.67
0.00
7.12
19.53
I.LMMA-70BB
70.59
33.33
47.28
47.28
30.95
30.18
23.78
0.00
9.82
23.67
LLaMA3-8B
65.38
30.77
48.08
35.71
17.35
17.53
12.34
0.00
1.72
14.11
LLaMA3-70B
86.54
$7.69
72.12
79.25
61.90
62.18
54.81
14.81
32.33
50.88
Qwen 1. 1.5-14B
75.00
34.62
54.81
62.93
36.73
35.17
27.37
1.85
6.34
28.
Qwen 1. 1,5-22
78.85
38.46
58.65
80.95
61.22
59.08
50.88
16.67
28.75
48.75
Qwen2-7B
73.08
38.46
55.77
71.09
49.66
49.59
40.14
5.56
13.30
38.22
Qwen2-72B
86.54
48.08
67.31
79.93
61.22
58.52
50.28
14.81
25.15
48.32
Mistral-7B
60.42
25.00
42.71
61.22
37.07
37.33
29.14
3.70
9.98
29.74
ChatGLM3-6B
10.00
0.00
5.00
22.90
5.99
5.79
3.66
0.00
0.00
6.39
GL.M-499B
91.67
45.83
68.75
63.95
42.18
42.72
35.98
3.70
19.01
34.59
Tool Use Specifici Large Language Models
Too1LLaMA2-7B
85.42
18.75
52.08
31.97
7.82
7.6
5.20
0.0
5.73
9.72
MTU-LLaMA (Ours)
92.31
50.00
71.15
81.63
67.69
66.94
58.74
9.26
32.47
52.79
3.1
MAIN RESULTS
The experimental results for the normal set are presented in Table 2 (S-S M-S) and Table 3 (S-M M-M). The results on the hard set are illustrated in Table 4 These results reveal several key findings: (1) Open-source models typically exhibit inferior performance compared closed-source models in nearly all metries, with the exception of GPT-3.5. However, certain models, including LLaMA3-70B and Qwen2-72B, demonstrate results comparable to those achieved by closed-source models. (2) GPT-4 consistently exhibits superior performance on the normal set; however, its performance decreases the hard set compared to GLM---Pllu.. Qwen-Max demonstrates exceptional performance in the M-M setting, with its advantages becoming more pronounced in the hard setting, even surpassing GPT-4. Similarly, GLM-4-Plus exhibits outstanding performance in the S-S and M-S settings, and its superiority is further amplified in the hard setting, also exceeding that of GPT-4. DeepSeek V2.5 performs admirably in the S-M setting. (3) Our MTU-LLaMA exhibits substantial advancements over its initialization. i.e., LLaMA3-8B-Instrre In-Ittruut,, across all settings and metrics. also competitive with some closed-source models, underscoring the effectiveness of our ITU-Inttruct. (4) Generally, all the models perform better on the normal set than on the hard set, indicating LLMs' limitations in handling more challenging tool-use scenarios. (5) Notably, despite being fine-tuned specifically for tool-use, TooILLaMA exhibits poor performance across all settings and metrics, suggesting its limited generalizability.
Overall Performance.
Effect of Multi-Ture across the Tables 2 and and have following findings: (1) Both closed-source and open-source models tend to perform worse in multi-turn settings (M-S and M-M) compared to single-turn settings (S-S and S-M). (2) Our MTU-LLaMA shows relatively better adaptation and robustness multi-turn settings. (3) Based on our novel TPR metric, we can observe that LLMs typically experience tool-use errors within the initial 30%-50% turns for closed-source models, and within the first 0%-30% turns for open-source models. (4) Most models such as Qwen2-72B have significantly higher ATS scores than TPR scores. This implies that while LLMs frequently encounter tool-use emrrr in the initial
We compare the single-turn (S-S, S-M) and multi-turn (M-S, M-M) settings
7

---

Published as a conference paper at ICLR 2025
Table 3: Results of different models on the normal set of MTU-Eval (S-M & M-M). "S-M" and "M-M" denote SSingleTTur Multi-Tool" and "Multi-Tie Turn Multi-Tool" settings, respectively.
S-M
M-M
Models
TN
TO
Avg.
TN
TO
ATS
SATS
SR
TPR
Avg.
Closed- Source Large Language Models
GPT-4
66.85
70.52
68.68
72.10
73.38
68.77
66.07
30.95
59.52
61.80
GPT-3.5
32.64
38.22
35.43
24.72
25.46
16.11
12.22
0.00
3.97
13.75
Qwen-Max
39.76
48.82
39.29
62,00
64.07
56.27
55.27
4.76
52.38
49.13
GLM-4-Plus
45.76
48.48
47.12
53,95
54,58
49.17
45.72
4.76
39.48
41.28
DecpSeck V2.5
56.88
60.28
58.58
50.15
51.79
44.84
41.26
7.14
34.72
38.32
Open-Source Large Language Models
LLMMA-7B
3.39
3.94
3.67
22.22
22.22
22.90
21.80
0.00
19.92
19.92
LLaMA--0BB
6.82
8.48
7.65
30.12
30.49
28.77
28.77
0.00
28.77
28.77
LLaMA3-SB
14.79
20.30
17.55
9.43
10.04
4.44
2.81
0.00
0.00
4.46
LLaMA3-70B
26.85
32.68
29.76
33.60
35.71
26.94
23.82
0.00
17.86
22.99
Qwen1.5-2
22.12
28.22
25.17
27.78
28.67
21.07
19.04
0.00
14.88
18.57
Qwen 15-72B
28.04
30.60
29.32
23.00
23.31
52.94
51.86
0.00
7.34
26.41
Qwen2-7B
24.52
29.59
27.05
21,04
22.75
15.24
11.52
0.00
4.76
12.55
Qwen2-72B
52.76
59.98
56.37
45.93
47.67
42.02
38.07
7.14
29.76
29.76
Mistral-7B
14,21
18.22
16.22
10.15
11.11
5.44
3.66
0.00
0.60
5.16
ChatGLM3-6 GLM-4-9B
8.56
7.55
10.64
11.11
9.21
2.38
5.95
7,88
27.58
25.61
16.17
16.45
9.48
0.00
0.00
8.04
Tool- Use Speciffiiiii Large Language Models
ToolLLaMA2-7B
11.52
11,52
11.51
4.07
4.07
2.78
2.34
0.00
1.59
2.48
MTU-LLaMA (Ours)
55.39
58.55
56.97
42.47
43.42
39.64
32.50
7.14
19.05
30.70
turns, they can still correctly use tools in subsequent turns in most cases. However, this correctness does not account for the cascading effect of previous errors, but solely considers the success rate of equilibrium between
independent tool usage. (5) Fortunately, the SATS scores be treated ATS and TPR metrics, which simultaneously account for the positions at which tool-use errors occur and the subsequent impact later turns.
Table 4: Results on the hard set of MTU- Eval. We report the average scores for each setting (Detailed results are shown in Ap- pendix
Effect of Multi-Tool. Based on multi-tool settings (S-M and M-M) across Tables 2 and derive the following findings: (1) Multi tool settings (S-M and M- M) show significant complexity, leading to noticeable performance drops for most models.
(2) Despite the complexity, models like GPT-4, Qwen2-72B and our MTU-LLaMA show stronger robustness. (3) In con- trast, the good models in single-tool settings such as GLM-4-Plus (onsed-source) and LLaMA3-70B (open- source), are surpassed by Qwen-Max (closed-souree and Qwen2-72B (open-source), respectively, indicating the superior performance of Qwen series in multi-tool settings. (4) The model rankings by TN and TO are highly consistent, implying that models with better con- trol over the number of tools also tend to manage tool sequences effectively, suggesting strong correlation between these capabilities.
Modek
Cloved Cloved-Sowre Source Large Language Models
GPT-4
77.88
44.61
58.07
41.36
GPT-3.5
Qwen-Max
77.88
42.11
24.01
45.08
GLM-4-Pho
82.69
47.61
30.90
39.53
DeepSeck
Open Source Large Lunguage Models
LLaMA2-7B
28.57
17.13
2.35
11.76
LLaMA2-70B
LLLMA3-8B
25.89
12.85
9.91
5.89
LLaMA3-70B
Qwen1 5-148
44.64
29.39
12.81
9.37
Oweal.5-72B
56.73
29.92
18.85
17.93
Qwan2-78
58.93
28.73
17.50
10.17
Qwen2 728
68.40
38.42
37.14
25.13
Mistral-7B
26.92
26.04
11.48
10.84
ChaGGLM3-B
18.89
9.52
3.2 ANALYSIS
GLM-4-9B
47.12
30.22
18.98
9.53
Thol Use Speciffc Large Language Models
OOD Performance. To evaluate the generality of MTU-LLaMA, we measure its performance on the OOD test split of MTU-Bench and two other OOD
ToolLLaMA2-78
18.27
2.34
MTU-LLaMA (Ours)
37.5
43.10
39.31
21.70
tool-use benchmarks, i.e., API-Bank (Li al. 2023)
and Tool (Fam & Shin 2023 in Table
8

---

Published as a conference paper at ICLR 2025
(6) IS vs Dialogue Turns
(b) TS vs Number of Tools
(56)
-O- GPT3.5
LLAMAA-NBB-boo
MTU-LLaMAA
MTL-LL
Dialogue Turns
Numher of Tools
Figure 5: (a) Effect of dialogue turns. (b) Effect of different number of tools.
pare the performance of MTU-LLaMA, LLaMA3-8B-Mee IB-Inrttuc,,,, and GPT-4 on these benchmarks under the M-S setting, and We observe that MTU-LLaMA outperforms LLaMA3-8B Instrruttt all three OOD benchmarks. Notably, MTU-LLaMA achieves performance comparable to that of GPT-4 on API-Bank, which show strong generalizability of MTU-LLaMA.
Effect of Dialogue Turns. We illustrate the im- pact of dialogue turns on tool selection accuracy Figure 5 We observe that performance slightly declines as the number of dialogue turns increases. Our MTU-LLaMA exhibits the most gradual de- crease in performance, demonstrating its robustness to higher dialogue turns.
Table 5: OOD Performance of our MTU- LLaMA.
(OOD)
SR
Avg.
67.57
32:31
LLaMA3-8D 4B
36.71
36.71
35.53
23.00
7.69
11:31
25.16
MTU-LLLMA
47.34
57.88
62.05
54.58
9.26
35.49
41.10
Models
TolTalk
TS
PS
AT8
SATS
SR
TPR
Axg.
Effect of Tool Numbers. Figure shows the impact of tool numbers on tool selection accu- racy. As the number of tools increases, the per- formance of GPT-3.5 and LLaMA3-70B degrades, with ILLMMA-70BB showing sharper drop. In contrast, MTU-LLaMA maintains relatively stable accuracy, demonstrating its superior handling of multiple tool calls.
LLaMA3-8B
GPT-4
51.88
30.00
$1.60
30.86
44.50
26.72
22.66
39.74
6:90
0.00
27.13
15:44
20.95
36.89
MTU-LLMM.
31.23
28.51
3
24.52
Models
API-Bank
SR
Avg.
38.32
LLMMAS-BR
14.03
13,56
7.74
11.71
MTU-LL-MA
51.80
51.80
48.58
45.57
38.10
38.00
45.66
Error Analysis. In Figure 7 we use five LLMs to an- alyze the different types of errors (i.e., "Action Error". "Parameter Error" and "Format Error"). Specifically, the "Action Error" and "Parameter Error" denotes select wrong tools and wrong parameters, respectively, and the "Format Error" means that the model cannot follow in- structions well and outputs wrong formats, which cannot be resolved well. Figure 7 illlustrates two primary find- ings: (1) "Action Error" occurs more often than other errors, specifically in challenge M-M setting, and stronger models show fewer errors. (2) "Format Error" usually exists in weaker LLMs (e.g., LLLMA3-8B-Inste Cha9GLM4-99), and our fine-tuned version MTU-LLaMA greatly reduces format issues, which shows the effectiveness of MTU-Instrun See Appendix E for detailed error cases.
HIULLANU--1
24
15
d5
Figure 6: Scaling Law of LLMs on MTU-Eval.
and
Scaling Law We evaluate the performance of MTU-LLaMA across different model sizes, using LLaMA2 models with 7B, 13B, and 70B parameters as initialization, which are fine-tuned with MTU-Instruet. The results Figure show that the performance of MTU-LLaMA improves as the model size increases, suggesting its scalability.
Consistency Between Our Proposed Metrics and Human Evaluation. To show the validity of our proposed metrics (SATS, TN, and TO), we evaluate the consistency between these novel metrics and the human evaluation results. We randomly sample 50 instances from the M-S subset for the

---

Published as a conference paper at ICLR 2025
(4) mumber ences (M-S)
(b) emors (M-M)
30:55
12
BaMMAS B8-tatract
2
ChiiLL
43.00%
CaaGGMM
72.54%
47.72%
ILAMAS
63.26
MTU-ULMMA
MAMA
Acticn Erer
Actie
Pasametuz Emee
Parunatis
100
250
252
350
150
Emor Count
Emor Count
Figure Analysis of different error types for different LLMs under M-S and M-M settings.
SATS metric and 50 instances from the S-M subset for the TN and TO metrics to compare two models: GPT-3.5 and LLaMA3-8B. Human annotators are also asked to compare the two models. In Table 6 we report the Pearson correlation coefficient between our metries and human evaluation results, as well as the Pearson correlation among human annotators, which shows excellent consistency between these metrics and human evaluations. More details can be found in Appendix
4 RELATED WORKS
Instruction Tuning for Tool Learning. The ob- jective of tool learning to equip large language models (LLMs) with human-like tool usage capa-
Table 6: Consistency for our proposed met- rics.
bilities Yang et al., 2023; Shen et al. 2023 Qu et al. 2024). To achieve this, LLMs are typically fine-tuned with tool instruction data to improve their performance in tool planning, selection, calling, and
Annotation Count
Metric-Human
Consistency
Human: Human
SATS
50
0,8280
0.9344
0.8497
0,9245
TO
50
0.8821
0.9482
response generation (Schick et al. 2023 Liang et al.
2023: Kong al.) 2023). However, existing tool instruction datasets either have limitations in multi- tum dialogue and multi-tool usage scenarios, or are based on synthetic data, resulting in misalignment
with real-world user needs (Huang et al., 2024 Li et al. [023 Patil al., 2023 Zhuang al.
2023). In this paper, we introduce MTU-Instruct, a large-scale instruction dataset to improve L.MSs
performance in diverse real-world tool-use scenarios.
Evaluation Benchmarks for Tool Learning. Many tool-use benchmarks have been proposed, but they still have many limitations. Firstly, Table these benchmarks have limited capabili- ties to assess complex scenarios (e.g., multi-turn dialogues, multiple tools, and cross-domain tool generalization) (Huang et al., 20224 Li ct al., 20233 Patil et al. 2023 Tang et al. 2023). Secondly, some benchmarks rely excessively results with heavy costs (Qin et al., 2023b Tang et al. 2023). Finally, existing assessments often overlook critical dimensions, such as the order of multi-tool invocation, the impact of erroneous calls on subsequent interactions, and the accuracy of tool parameter selection, resulting in evaluations lacking comprehensiveness and depth (Zhuang al. 2023 Li et al. 20233] Patil al.! (023).. In contrast, MTU-Bench not only includes extensive multi-turn dialogues and multiple tool scenarios but also introduces testing for OOD tool generalization. By employing automated evaluation and incorporating metrics like SATS, TN, and TO, MTU-Eval achieves a more comprehensive assessment.
GPT models, potentially leading to subjective and unstable
5 CONCLUSION
In this work, we propose multi- granularity tool-use benchmark for LLMs called MTU- which consists of MTU-Instruct and MTU-Eval. Specifically, first, the MTU-Instruct dataset is used to enhance the tool-use abilities of existing LLMs,, and the MTU-Eval with multiple tool- use scenes is applied to benchmark the tool-use abilities comprehensively. Notably, all evaluation metrics of our MTU-Eval are based the prediction results and the ground truth without using any GPT or human evaluation metrics. Moreover, Comprehensive experimental results demonstrate the effectiveness of our MTU-Bench Finally, hope MTU-Bende can guide developers and researchers in understanding the tool-use capabilities LLMs and facilitate the growth of foundation models.
10

---

Published as a conference paper at ICLR 2025
ETHICS
In developing MTU-Bench and MTU-LLaMA, we recognize several ethical considerations that arise from the broader context of integrating tool-use capabilities into large language models (LLMs). As these models become more capable of interacting with real-world systems -scch as those involving financial services, healthcare, and other critical domains we must consider the potential risks associated with misuse. For instance, there is the possibility that LLMs could be exploited to access sensitive tools or manipulate information in ways that could harm individuals or organizations.
While our work aims to improve the accuracy and efficiency of tool use, we are mindful of the importance of ensuring that these technologies are deployed responsibly. We advocate for the implementation of robust safeguards, including transparency in decision-making processes, fairness in how tools are applied, and accountability in real-world usage. Furthermore, we encourage futture research and development efforts to focus mitigating potential biases and ensuring that these systems are secure and trustworthy when handling sensitive tasks.
REFERENCES
Meta. Llama model card. 2024. m.ob/maii//M01 CARD mdl
-aaeeett-llmaa/aaae 1ataaammammmaaaaa.a 1lama3/
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Bee Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg,
Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4. March 2023. URL
W1 with-gpt-47)
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašié. Multiwoz - large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. arXiv preprint arXiv: 1810.00278, 2018.
Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Daniel Duckworth, Semih Yavuz, Ben Goodrich, Amit Dubey, Andy Cedilnik, and Young Kim. Taskmaster- Toward a realistic and diverse dialog dataset. arXiv preprint arXiv: 1909.05358, 2019.
Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo Songyang Zhang, Dahua Lin, Kai Chen, and Feng Zhao. T-eval: Evaluating the tool utilization capability of large language models step by step, 2024. URL https: http:p:://aaxiiooooe org /bs/
2312. 14033
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv: 10.1-81/0.14168, 2021.
DeLpSeek-AL Deepseck-v2: A strong, economical, and efficient mixture-of-experrs language model, 2024.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiczhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 320-335, 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson. Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton
11

---

Published as a conference paper at ICLR 2025
Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Farcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, liyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong. Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika lyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala- I Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang. Pengwei Li, Petar Vasic, Peter Weng. Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang. Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang. Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov. Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisemman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery. Eleonora Presani, Emily Hahn. Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanaa, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean- -aanistt Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen. Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong. Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia. Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A. Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumoy, Maya Lathi,
12

---

Published as a conference paper at ICLR 2025
Meghan Keneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong. Ning Zhang. Norman Cheng. Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj. Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang. Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang. Shugiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe. Soumith Chintala, Stephanie Max. Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho. Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang. Wes Bouaziz, Will Constable, Xiaocheng Tang. Xiaofang Wang, Xiaojian Wu, Xiaolan Wang. Xide Xia, Xilun Wu, Xinbo Gao. Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang. Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang. Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary Vito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The llama 3 herd of models. arXiv preprint arXiv: 2407.2 2024.
Nicholas Farn and Richard Shin. Tooltalk: Evaluating tool-usage in a conversation setting. arXiv preprint arXiv:2311.10773 2023.
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang. Jiadai Sun, Jiajie Zhang. Jiale Cheng, Jiayi Gui, Jie Tang. Jing Zhang. Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia. Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du Zhenyu Hou, and Zihan Wang. Chatglm: A family of large language models from glm- 100 to glm-4 all tools, 2024.
C. T. Hemphill, J. J. Godfrey, and G. Doddington. The atis spoken language systems pilot corpus, In IILT, 1990.
Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun. Metatool benchmark for large language models: Deciding whether use tools and which to use. In The Twelfth International Conference on Learning Representations, ICIR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 20244 net wttp://rwwwwweeeeeouuuummmeeme forum? RdoRoc22t 1g
URL
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b, arXiv preprint arXiv:2310.06825, 2023.
Yilun Kong, Jingqing Ruan, Yihong Chen, Bin Zhang, Tianpeng Bao, Shiwei Shi, Guoqing Du, Xiaoru Hu, Hangyu Mao, Ziyue Li, et al. Tpt-v2: Boosting task planning and tool usage of large language badel-based agents in real-world systems. arXiv preprint arXiv:2311.11311 2023.
Harrison Lee, Raghav Gupta, Abhinav Rastogi, Yuan Cao, Bin Zhang, and Yonghui Wu. Sgd-x: A benchmark for robust generalization in schema-guider dialogue systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 10938-10946, 2022.
Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A benchmark for tool-augmented llms, 2023.
13

---

Published as a conference paper at ICLR 2025
Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, Yun Wang, Linjun Shou, Ming Gong, and Nan Duan. Taskmatrix.ai ai: Completing tasks by connecting foundation models with millions of apis. arXiv preprint arXiv: Arxiv- 2303. 16434,4422224 2023.
Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. # mstag: Instruction tagging for analyzing supervised fine-tuning of large language models. In The Twelfth International Conference on Learning Representations, 2023.
OpenAl. Gpt-4 technical report. PREPRINT, 2023.
Bhargavi Paranjape, Scott Lundberg. Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. Art: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arXiv: Arxiv-2303. 090/4, 2023.
Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv: 2305.15334, 2023.
Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Y. Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shi Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bo Li, Ziwei Tang, Jing Yi, Yu Zhu, Zhenning Dai, Lan Yan, Xin Cong, Ya-Ting Lu, Weilin Zhao, Yuxiang Huang. Jun-Han Yan, Xu Han, Xian Sun, Dahai Li, Jason Phang, Cheng Yang, Tongshuang Wu, Heng Ji, Zhiyuan Liu, and Maosong Sun. Tool learning with foundation models. ARXIVORG, 2023a. doi: D0.48550/rXxi.2334..555
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023b.
Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai,, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji- Rong Wen. Tool learning with large language models: A survey. arXiv preprint 2rXiv:2405.1793: 2024.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8689-866 8696,8666660 2020a.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. Towards scalable multi-domain conversational agents: The schema-guie guided dialogue dataset. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 8689-8696, 2020b.
Timo Schick, Jane Dwivedi-Yuu Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. CoRR, 04462 2023. doi: 10.48500/aXiv.2202207711.. URL https
//doi. wtt.lo///......5//aara..... 2tt.xki.v.3....77.....
Igor Shalyminov, Alessandro Sordoni, Adam Atkinson, and Hannes Schulz. Fast domain adaptation for goal-oriented dialogue using a hybrid generative-retrieval transformer. In ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8039 8043. IEEE, 2020.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hug- ginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv: Arxiv-2303.17580, 2023.
Aditya Siddhant, Anuj Goyal, and A. Metallinou. Unsupervised transfer learning for spoken language understanding in intelligent agents. AAAI Conference on Artificial Intelligence, 2018. doi: A33 131003939999 (ttps:s:: www.rriwwooggg/e. abs 1811.1111111 05370v1
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. Toolalpaca: General- ized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05301
2023.
14

---

Published as a conference paper at ICLR 2025
Qwen Team. Introducing qwenl.5, February 2024. URL https://awenine http://wwwwwn.....hu....b.b gwthub.tthhoo. blog/qweni.5/
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.09288, 2023.
Jize Wang, Zerun Ma, Yining Li, Songyang Zhang, Cailian Chen, Kai Chen, and Xinyi Le. Gta: A benchmark for general tool agents, 2024. URL httpt:://aaxiiiis http://wwww.r.////0/0/////// 08713
Zekun Wang, Ge Zhang, Kexin Yang, Ning Shi, Wangchunshu Zhou, Shaochun Hao, Guangzheng Xiong, Yizhi Li, Mong Yuan Sim, Xiuying Chen, Qingqing Zhu, Zhenzhu Yang. Adam Nik, Qi Liu, Chenghua Lin, Shi Wang, Ruibo Liu, Wenhu Chen, Ke Xu, Dayiheng Liu, Yike Guo, and Jie Fu. Interactive natural language processing. arXiv preprint arXiv: 2300.13245 2023. URL
org/ abs 2305. 13246v1
Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. On the tool manipulation capability of open-source large language models. arXiv preprint arXiv: 2305.16504 2023.
An Yang. Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint 10Xi::2407.10611, 2024.
Sherry Yang, Ofrr Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans. Foun- dation models for decision making: Problems, methods, and opportunities. arXiv preprint 0:Xi::2303.0419, 2023.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. Re- act: synergizing reasoning and acting in language models (2022). arXiv preprint arXiv:2210.03629, 2023.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, al. Glm- 130b: An open bilingual pre-trained model. arXiv preprini arXiv:2210.02444 2022.
Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang, Toolga: A dataset for llm question answering with external tools. arXiv preprint arXiv:2306. 13304, 2023.
15

---

Published as a conference paper at ICLR 2025
PCA of Embeddings with Augmented Samples
nat (2. shc, inrooressss
Charge, 11% 2oliing
adjust Bonting
hew Sontact
ad new contact
Own (01) contect
mante musl valamee
yolume
volume
share Jocation
dave cumnet Jscotion
sham postion
shung
acoess sport deda cut Lapal, rosglt
Figure 8: We extract embeddings for the names of tools and apply a fixed distance threshold for clustering. In this Figure, we present the clustering visualization results after PCA for partial samples from five categories.
A LIMITATIONS
Despite the strengths of our proposed MTU-Bench, there are still several limitations to consider. First, although MTU-Bench incorporates diverse range of real-world user instructions, it may not fully capture all potential edge cases or highly complex interactions that can occur dynamic, wolldwrldd environments. Second, while MTU-Eval provides comprehensive fine-grained metrics for evaluating tool-use abilities, these metrics are based on predefined benchmarks, which may not account for every possible tool-use challenge in evolving real-world applications. Although MTU-LLaMA demonstrates strong generalization across various metrics and scenarios, further research is needed to explore its adaptability increasingly complex and emerging tool functionalities in real-time settings. Future work should focus on broadening the benchmark's 's coverage and exploring more dynamic and complex real-world use cases.
B DETAILS OF MTU-BENCH CONSTRUCTION
Details for Data Filtering. We perform a rigorous data filtering process to ensure that the data selected for MUT-Bench is reliable and of high quality. The filtering process is as follows:
Exclusion of Unsuitable Intents: We filter out intents that are not suitable for tool calls, particularly those that are difficult to tackle with external tools. For example, conversations seeking naming suggestions excluded, as they inherently challenging to define for tool usage. Manual verification is employed to achieve this, resulting in 26% data exclusion rate.
Redundancy Elimination through Clustering: The synthesis of tools can lead to redundancies, such as share location, share share_my my__otciioon and info. To mitigate this, we adopt clustering approach to group similar tools, retaining only one representative tool from each cluster. We establish a lower threshold for initial clustering to ensure each cluster contains unique tool, followed by man- ual elimination any duplicates. This process achieves a reduction ratio of approximately 20:1, as illustrated in Figure
Filtering of Undefined Tools: About 6% of the synthesized data includes tools that are not defined the tool library. This data is filtered out using rule-based matching methods.
16

---

Published as a conference paper at ICLR 2025
Parameter Correctness Check: Approximately 16.9% of the synthesized data fails our correctness checks, comprising 3.2% of cases with fabricated non-existent parameters and 13.7% where generated parameters do not meet format validation requirements.
LLM Verification: We utilize an LLM, specifically GPT-4, recheck the correctness of all answers. This process results an additional 10% of data being filtered out.
Manual Quality Annotation: We conduct manual quality checks utilizing multiple experts. For the training set, 500 samples are randomly selected and each is evaluated by three experts, with any discrepancies resolved by fourth. This approach yields a 96% accuracy rate. For the test set, all samples are meticulously calibrated by hired experts, achieving final accuracy of 100%.
Prompt Templates for MTU-Bench Construction. The construction procedure of MTU-Bench involves numerous prompt templates, as listed below, for tool making (tool name synthesis and tool parameter synthesis), thought synthesis, observation simulation, and data quality check.
17

---

Published as a conference paper at ICLR 2025
Prompt Template (Tool Name Synthesis)
I will give you a dialogue, including historical dialogue information and current round dialogue information. Please help determine whether Assistant needs to call an \"API"" to obtain specific information or perform certain operations in order to solve the user issue in the current round of conversation.
1. If yes, and Assistant's s response shows that the user's s query has been solved, then return me with aapi namel" in \ttags"". The name of the API should be concise and easy understand, such as search restaurant, book_ restaurant, etc. The API name should start with verb and include the specific domain name. Related domains includes [domain info
1. etc. Note that if multiple APIs need to be called, return me with "'api_namp api name,
...\" in \"tags\..
2. If Assistant can provide the current response without calling the API, return me V"no
need to iall\"in \ttags\..
3. If Assistant needs to call API but is unable do so due to lack of necessary information, inform me of lllack of necessary information!" in \"tags\.. For example, due the lack of "restaurant name", the API for "oooo_restauratt" cannot be called.
If Assistant is only confirming existing information with the User, inform me of the \" confirmation information!" in \ttags\.
6. If some information in the historical conversation can help Assistant respond to the current issue without calling APIs, inform me that Vinformation already exists\" in \"tags\.. Please reply to me in JSON format: (V" Annallsii""" str, \ttgg\\: str).
Prompt Template (Tool Parameter Synthesis)
I will provide you with conversation segment that includes both historical dialogue information and current round dialogue information. Based on this information, please help determine whether, to solve the problem presented by the User in the current round of dialogue, the Assistant calls a specific API to obtain the necessary information or to perform related actions. Please respond according the following guidelines:
1. If an API call is required and the Assistant's s response solves the User's s problem, please specify which API called and reply me in the following JSON format: \"Action!"; V" api_ ppi_namel", Also, provide the parameters required for calling that API in the format: \" Action Inputl": (V"parameter name!": "vaalue!"....""
2. If it is impossible to call an API due to missing necessary parameter information, please explain in the ("Thought\" section due the absence of which parameters, which API cannot be called.
If answering the User's question does not require calling an API, please explain the \" Thought\" section why there is need to API.
Please strictly use the API names and parameter names I provide, and refrain from fabricating any. If the required parameter is not defined within list, you allowed to introduce new parameter names. Beyond this allowance, do not utilize any API names and parameter names that are beyond the specified range, to ensure consistency and accuracy. 5. Please include a section called \"Thought\" in your answer where you clearly and unambiguously demonstrate your thought process when solving or answering the question. Below are the APIs and their parameters information you can use: [apis_ information]
Please reply in the following JSON format: (\"Thought\": \'strl, \"Actiont:: V'apinnamel", \"Action Inputi": (Vparameter "aarameter_namel".. \"valuel", .... )). If there no information for \" Action\", \'Action Input)", or \"Thought",, please return empty string.
18

---

Published as a conference paper at ICLR 2025
Prompt Template (Thought Synthesis)
I will provide you with a multi-round conversation and API call information for each round. Your task is to add or verify the Assistants' thinking process of API call to the dialog.
Your need return the modified conversation with API call results added me, strictly
following the following format:
'nModified Dialogue:\n[
OJunUser: lxxougTh xxx. AnAction: xxx. AnAction Input
xxx. InObservation: xxx. AnAssistan: xxxx. An[Round []nnUser: xxx. inThought: xxx.1 nAssistant: xxx. In(Round 2]...
That is to say, after each round of the user's question, the Assistant must think about
whether to call the API and how to call it, and present this in '"Thoughtt".
For example:
Modified Dialogue:
[Round 0]
User: Find me some fantasy movies I can watch online, something directed by Neil
Marshall.
Thought: The user is looking for fantasy movies directed by Neil Marshall to watch online. I need query database or service for movies that fit these criteria.
Action:
FindMovies
Action Input:
('directed by': 'Neil Marshall', 'genre': 'Fantasy')
Observation: (" ('directed by': 'Neil Marshall', genre": 'Fantasy', 'subtitles:: 'False', 'title':
'Hellboy')
Assistant: Are you interested in Hellboy?
[Round 1]
User: I would enjoy watching Hellboy.
Thought: The user has not expressed any request, so the tool is not invoked here. Assistant: Shall I play the movie?
[Round 2]
User: Yes, begin playing the movie. Yes, play it without subtitles.
Thought: The user has confirmed they want to start watching the movie "Hellboy" and specifically requested play without subtitles. I'll proceed execute the play command with subtitles disabled.
Action: PlayMovie
Action Input: (' subtittes' "False", 'title": "Hellboy")
Observation: ( ('directed_by" by': 'Neil Marshall', "genre': 'Fantasy"; 'subtitles': 'False', 'title":
'Hellboy'}
Assistant: The movie now playing.
Requirements:
1. After \" l"Action\", it shows which API call. If not, it indicates that API calls required for this round. The parameters passed by calling the API are displayed after Action Input\". You just need supplement the API call return results for the rounds that have called the API, and place them after \"Observationn" \"Observation"" shows the return result of the API.
2. Please return the thinking process in \"Thhught'". will provide you with reference APIs and parameter names.
6. Except for adding \"Thought\" to the round of API calls, all other information remains in
its original content.
7. Please return the complete conversation.
The following a list of APIs and their parameters for your reference: [apis information]
19

---

Published as a conference paper at ICLR 2025
Prompt Template (Observation Simulation)
I will provide you with a multi-round conversation and API call information for each round. Your task is to add the return result of API call the dialog based on Assistant's response.
You need return the modified conversation with API call results added me, strictly following the following format:
'nModified Dialogue:/n|
OJunUser: lxx.uhThought; xxx. AnAction: xxx. AnAction Input XXX. InObservation: xxx. AnAsiistan: xxxx. An(Round []nnUser: xxx. inThuugh: xxx.1
nAssistant: xxx. /n in[Round 2]...
For example:
Modified Dialogue:
[Round 0]
User: Find me some fantasy movies I can watch online, something directed by Neil
Marshall.
Thought: The user is looking for fantasy movies directed by Neil Marshall to watch online. I need query database or service for movies that fit these criteria.
Action: FindMovies
Action Input:
('directed, by": 'Neil Marshall', 'genre': 'Fantasy']
Observation: {" d'directed_by 'Neil Marshall', "genre:: "Fantasy,, "subtitles:: 'False', 'title":
'Hellboy'}
Assistant: Are you interested in Hellboy?
[Round
User: I would enjoy watching Hellboy.
Assistant: Shall I play the movic?
[Round 2]
User: Yes, begin playing the movie. Yes, play it without subtitles.
Thought: The user has confirmed they want start watching the movie "Hellboy" and specifically requested to play without subtitles. I'll proceed to execute the play command with subtitles disabled.
Action: Plaie
Action Input: subbiites'' 'False", 'title': 'Hellboy')
Observation: ( didreetedd by': 'Neil Marshall', genre': 'Fantasy', 'subtitles:: 'False', 'title":
'Hellboy'}
Assistant: The movie is now playing.
Requirements:
1. After \"Action\", it shows which API call. If not, it indicates that API calls not required for this round. The parameters passed by calling the API are displayed after Action Inputk". You just need to supplement the API call return results for the rounds that have called the API, and place them after ("Observation!".
2. Please return the information in \"Observation!" in JSON format, for example: (V" : \"valuel", i"parameternamel". \"value\"...) Specifically, every round with \"Action\" and \"Action Inputk", you should add an \"Observation"" after the A'Action Inputl", which should fill in the information returned by the API. For example: InAction: FindMovies \nAction Input: d'directed_ by": 'Neil Marshall', 'genre": 'Fantasy') } nObservation: ('directed_be by": 'Neil Marshall', 'genre:: "Fantasy,, 'subtitles:: 'False', 'title ''Helloy')..
3. If Assistant's response shows that it has not yet received the specific information returned by the API tool call, that "InObservation"": (Verrror!": \"Time out.\")
4. If the result returned by Assistant shows that relevant information is found, the API call returns an empty dict, such as: "InObservation":: {}-
5. I will provide you with reference APIs and parameter names. Ensure that all parameter
names used are defined and should not fabricated.
6. Except for adding API return results the round of API calls, all other information
remains in its original content.
Please return the complete conversation. The following is a list of APIs and their parameters for your reference: [apis information]
20

---

Published as a conference paper at ICLR 2025
Prompt Template (Data Quality Check)
Please review the provided conversation snippet, which includes historical dialogue, current round dialogue, and the API call made in this round. Your task is to verify the accuracy of the API and parameters used in this round of dialogue. Assume the Assistant does have knowledge of real-world information such as cinemas or restaurants; it relies on API calls to access information or carry out actions such as making reservation. Use the guidelines below to correct any inaccuracies.
1. Check and correct the API selection in the V "'Action"" field. Common errors include: Assistant's s response indicates that an API call was made, but the V" I"Action\" field is empty. b. Assistant don't need to call any API to reply to the user's s current round of conversation. In this case, calling is not necessary. but there is an API name in the V" A'Action'" field. c. The assistant's s response shows that the necessary information required for API calls is missing, Therefore, the assistant is asking the user for additional information, but there is an API name in the \" l"Action\". d. The API listed in the \" A"ctionn"" is incorrect.
2. Verify and correct the parameters listed \" I"Action Inpuri" and ensure they correctly match the API call. The \" Action Inputt" should be formatted (V" [parameter_namel"" valuel", V "parameter_namal
("value\",...)
Revise the content in '"Thought\" to include the correct rationale for selecting tools and parameters. The thought should only consider historical conversations and current user issues, assuming that the assistant's response is unknown.
4. Ensure all API and parameter names used are defined and should not be invented. Here a list of APIs and their parameters for your reference:
popisinffrmation
Please respond in the following JSON format: {) [V"Thoughtl": str, \"Action!"; V'api namel", \"Action Inputl": (\" ('parameter namel": \ v"valuel",... If there is no \"Action!" or \" ("Action Inputl" information, please return an empty string.
Example of Tool Document. As shown Figure 9 the tool document allows the model to determine the appropriate tool names and their usages. It contains all the tools we synthesized, each tool including its corresponding tool description, necessary parameters, optional parameters, parameter description and data type, well as the returns.
C MORE MTU-BENCH DATA ANALYSIS.
Length Distribution. We illustrate the length distribu- tions in Figure 10 and Figure II for the training and evaluation data, respectively.
Table 7: The number of dialogues under different settings.
More Statistics. Table 7 provides a comprehensive sum- mary of the dialogue statistics across diverse settings, dis- tinct splits and subsets.
Test
Setting
Train
normal
hard
S-S
14277
52
56
D DETAILS OF MTU-EVAL
S-M
13641
55
39
M-S
19007
54
31
Hard cases in the hard test set. encompass extensive parameters, nonsensical tool names, determination of spe- cific parameter values, inability to call tools, interaction among multiple tools, and multi-turn parameter inheri- tance, as delineated in Table g
M-M
7442
42
37
OOD
65
Prompt Templates for MTU-Eval. During the evaluation, the models are provided with distinct
system prompts for various settings, which encompass both the comprehensive task specifications and the tool documentation. We list the system prompts used for evaluation Boxes
21

---

Published as a conference paper at ICLR 2025
"name": "sctAlarm",
"description": "This tool is used for setting alarm based the user's specified time, label, recurrence pattern, sound preference, and specific day(s) for the alarm to activate.", "required parameters".
"name": "time",
"type": "string".
"description": "The specified time for the alarm to go off.",
"format": "HH:MM"
"name": "label",
"type": "string".
"description": "A custom name description for the alarm.".
"optional |
"name": "recurrence".
"type": "array".
"description": "It specifies how often alarm should repeat. Each element can be a keyword (eeverdday', 'weekdays', 'weekends'), name of the day ('Monday'. 'Tuesday', 'Wednesday, Thursday', 'Friday', 'Saturday', 'Sunday'), ) or specific date in 'YYYY-MM-DD' format.".
"items":
"strin"."
"anyOf":
("enum": "everyday", "weekdays", "weekends", "Monday", "Tuesday",
Wednesday" " "Thursday", "Friday", "Saturday". "Sunday" }},
p"patter"" (MNdd(44-Nd(22--)d22255 1.
"default": ["everyday"]]
"name": "sound",
"type": "string".
"description: "The chosen sound for the alarm when it goes off."
"name": "vibrate".
"type": "int",
"description": "Specifies the intensity of the vibration for the alarm. A value of 0 means no vibration. the intensity of the vibration is not specified, the default intensity is 5.". "default":
"result : [
Figure 9: The JSON structure of the tool "setAlarm", which is used to create new alarm based on user-specified parameters. The structure includes required, optional and result (i.e., return) parameters, along with their corresponding data types, descriptions, formats and default values.
22

---

Published as a conference paper at ICLR 2025
(a) Number of Dialogue Tums
(b) Word Count of the Dialogue
(c) Word Count the User Instruction
243.22
1750
Mean 48.39
Mediacei
Media--BBi.
Mediza-120
1500
12200
1989
1900
7300
2200
(d) Number of Tooo Calls the Outputs
(c) Number of Tools in the System Prompt
(1) Number of Tool Paramelers
Mezzel
14000
Max
1900
Mediandd
12000
Mediae-4
1000
6000
4000
Figure 10: Length distributions of the training data.
Number Dialogue Turn
Count the Dialogue
Count the User Instruction
Moue-2.5 2.56
200
Moar- 195.22
128
--- Mear 45.48
Modape1
175
Midio--550
106
Medac-500
150
150
125
68-
100
(4) Number of Tool Calls the Outputs
(6) Number of Tools in the System Prompt
( Number of Tool Parameters
150
M 31
M
Ma
100
Matsne10
140
Medisz-d.
Medarel
20
//
Figure 11: Length distributions of the evaluation data.
23

---

Published as a conference paper at ICLR 2025
Table 8:
Hard cases in the hard test set.
Type Extensive Pa- rameters
Description
Case
The quantity parameters required to be specified exceeds six.
User: Can you book non-refundable, one-stop flight Delta Airlines from New York to Chicago, leaving on the 5th returning on the 11th of this month in Economy class?
Target: Action: ReserveRoundtripPlgge Airlines", "departure date": 3024-01-05". "destination city": "Chicago". "ori- gin city": "New York". "return date": "2024-01-11", "seating class": "Econ- omy", "number_stops, : "I", "refundable"; False)
Action Input: ("airlines": "Delta
Nonsensical Tool Names
The designa- tion of the tool lacks signifi- cance, for in- stance, "abe".
System Prompt: ... The following a list APIs and their parameters that you can use: { "name": "eee". "description". "Book an appointment at dentist for a given time and date", "required_parameterss parameters": ["dentist name", "appoint- ment ment_time", "appointment coonnt__ate"__..
User: would like to book an appointment with Yvonne Yang Greenview Dental Care the 1 1th at 11:45. Do they offer cosmetic services?
Target: Action: Action Input:
("appointment date" "2024-01-11", "appointment time": "11:45", "den-
tist_name": "Yvonne Yang")
Determination of Specific Parameter Values
Parameter val- ues must con- form to spe- cific criteria.
Tool Definition: "name": "searchHotell", "description": "To search for hotels based on a set of criteria including rating, type, amenities, loca- tion. and price range.", "optional parameters": [.."namee: "priceRange", "choices" = ["cheap"" "moderate", "expensive"]
User: Can you help me find moderately priced hotel in the centre with parking and internet availability?
Target: Action: SearchHotel Action Input: "area": "centre", "parking": True,
"internet": True. "priceRange": "moderate" "return info": ["all"]
Inability to Call Tools
The of tools does not contain any suitable options.
System Prompt: You are a The following is list APIs and their parameters that you can use...
Prompt: Can you find drama movie with subtitles directed by Claire Denis? Target: Thought: The user wants to find drama movie with subti- tles directed by Claire Denis. However, there is no available API in the provided list that supports the searching for movies. Therefore, it's not possible meet user's 's request with the current tools.
Interaction among Multi- ple Tools
In single of dia- logue, more than four tools are utilized.
User: I am looking for the date of concert in New York, followed by the weather forecast for that day. And I want find dentist, book appointment 10:00 on Febrary 2nd. And please add the dentist's phone to my contact list.
Thank you!
Target Action: FindEvennss Action Input: ('event type": "concert". "city": "New York") Action GetWeather Action Input: ("city": "New York", "date": "Find- Events. date") Action: BookAppointment Action Input: ("dentist_name" "Find- Provider.dentist_nm mnamsst,___"",. "appointment_ _tme" "10:00", "appointment date": "2024-02-02") Action: AddNew Contact Action Input: ("contact name": "Find- Provider.dentist name", "phone number": "FindProvider.phone "
The param- Parameter eters for the Inheritance current turn must ex- tracted from the historical dialogue.
Multi-Turn
User: Find some music please.
Assistant: Action: LookupMusic
Ohservation: ["artist": "Ariana Grande". "sson_name"" "Be My Baby") Assistant: Ok, how about Be My Baby by Ariana Grande?
User: That sounds good.
Assistant: Should play this now?
User: Yes. Play on kitchen speaker please.
Target: Action: PlayMedia Action Input: { "playback device": "kitchen
speaker". "song name": "Be My Baby"
24

---

Published as a conference paper at ICLR 2025
System Prompt (S-S)
Please reply to the user based on their input and historical conversation information. You choose to call external tools to implement it. Here the call requirements and information about available APIs.
1. Please provide your thought process in \"Thought"", including user intent analysis, whether to call APIs, and how to call APIs.
When a user's i request can be satisfied by calling API, please provide the required calling information in the following format: 'nAction: The name of the API to be called./ nAction Input: The parameter information required call the API, in Json format.
3. If the user's needs can be met without calling the API, then API call action will be made.
4. If there is lack of mandatory information that makes it impossible call specific API then no API call action will be made.
5. The name and parameter name of the API must be consistent with the provided API information. The value of the parameter should be extracted from the context, and the information should not be fictional. If none of the apis provided are available to meet the user's s requirements, no Action is taken.
7. If you need to call the API, your output format should be: Thought: xxxxinAction: xxxx'nAction Input: xxxx For example:
Thought: The user is looking for a one-way flight three people from Las Vegas to Atlanta specific date, March 5th. Let's 's call the SearchOnewayFlighh API to find the available flights. AnAction: Acction SeurchOnewayFlighhtaAccc Vegasl", l'destination_cii city!": A"Allantaa,, V deepartureedatee 1"2024-03-05", seating class\": \'Ecooomy\"
Input: I'orrigi_cctyl": V"Las
If you don't 't need to call the API, your output format should be:
Thought: xxxx The following list of APIs and their parameters that you can use:
[apis information]
25

---

Published as a conference paper at ICLR 2025
System Prompt (S-M)
Please reply to the user based on their input. You can choose to call external APIs to implement it. Here the call requirements and information about available APIs.
1. Please provide your thought process in V"Thought\", including user intent analysis, whether call APIs, and how call APIs.
2. When a user's request can be satisfied by calling APIs, please provide the required calling information the following format: 'nActtin: The name of the APIs. InAciinn Input: the parameter information required to call APIs, in Json format. For example, \ nAction: \appi_ name
-. V"paameter value .I",. .."..../IAAAction: \" a"api_ hapl__aae____ B\cInnttion Input: (V Berrameter_nam_Beeee :.1\": parameter_ value_ ...2".....)
\aapi_name Altlonction Input: ("paraaeter_nnaee paattter_ammm__A_e
3. There may be interaction relationship between APIs, where the parameter value returned by the previous API call needs to be used as the parameter value for the next API call. Please use ("previous API name.return parameterrreerrrreeeem name!" the parameter value for the new API call.
4. Multiple APIs may need be called to meet the user's needs. Please pay attention to the order of APIs' call.
There may be interaction relationship between APIs, where the parameter value returned by the previous API call needs to be used as the parameter value for the next API as the parameter value for call. Please use '"previous API returr wwtpppptaaaaapappee the new API call.
The name and parameter name the API must be consistent with provided API information. The value of the parameter should be extracted from the context, and the information should not be fictional.
7. If you need to call the API, your output format should be:
ThThughh: xxxx'nAction xxxx'nAction Input: xxxx'nAction: xxxxInAction Input:xxxxl nAction: xxxx'nActin Input: xxxX...
you don't need to call the API, your output format should be: 'n 'hTohuth xxxx
The following a list of APIs and their parameters that you can use:
[apis information]
26

---

Published as a conference paper at ICLR 2025
System Prompt (M-S)
Please reply to the user based on their input and historical conversation information. You choose to call external tools to implement it. Here the call requirements and information about available APIs.
1. Please provide your thought process in \"Thought"", including user intent analysis, whether to call APIs, and how to call APIs.
When a user's request can be satisfied by calling API, please provide the required calling information in the following format: 'nAction: The name of the API to be called.) nAction Input: The parameter information required call the API, in Json format.
3. N"Observation"" is the information returned by API calls.
If the user's needs can be met without calling the API, then no API call action will be made.
If there is lack of mandatory information that makes it impossible to call a specific API then no API call action will be made. Note that the dialogue history already contains the required information, there is no need call the tool again.
The name and parameter name the API must be consistent with the provided API information. The value of the parameter should be extracted from the context, and the information should not be fictional.
If you need call the API, your output format should be: inThought: xxxx'nAction xxxxInAction Input: ххxx
Therefore, if you don't need to call the API, your output format should be:
InThought: XXXX
The following a list of APIs and their parameters that you can use:
[apis information]
27

---

Published as a conference paper at ICLR 2025
System Prompt (M-M)
Please reply to the user based on their input and history conversation. You can choose to call external APIs implement it. Here the call requirements and information about available APIs.
1. Please provide your thought process in \"Thought"", including user intent analysis, whether to call APIs, and how to call APIs.
When a user's request can be satisfied by calling APIs, please provide the required calling information in the following format: 'nAction: The name of the APIs InAcionn Input: the parameter information required to call APIs, in Json format. For example, nAction: aapi name hame_A\"'nA Alctonction Input: (\" (\"parameter name _.I1e peparameter_vall value_A .......JnAAttion "'api_name, paa__aammmemettiiiiii B\"AnAction Input: (V" "parameter_mame B.I"":
parameter_ value_Ba ____BB.2."..
There may be an interaction relationship between APIs, where the parameter value returned the previous API call needs to be used the parameter value for the next API call. Please use 1"previous_Aie Bactoous_Pl__aammae saaneereurr__prameete.ree
as the parameter value for
the new API call.
4. Multiple APIs may need to be called to meet the user's needs. Please pay attention to the order of APIs' call.
5. The name and parameter name of the API must be consistent with the provided API information. The value of the parameter should be extracted from the context, and the information should not be fictional.
If you need call the API, your output format should be:
Thought: xxxiinAction: xxxx'nAction Input: xxxx'nAction: xxxxInAction Input:xxxx) nAction: xxxxlnAction Input: xxXX...
If you don't It need to call the API, your output format should be: Thought: xxxx
The following is list of APIs and their parameters that you [apis information]
28

---

Published as a conference paper at ICLR 2025
Examples for computing metrics.
The examples are listed below:
Tool Selection Accuracy (abbr. TS)
(for S-S, M-S)
Positive Example:
User: "Search for the latest news on artificial intelligence."
Assistant: OK! Let me search the web. search_wer web (query="latest news on AI")
Ground Truth Tool: search_waa web
Assistant's Tool:
Score: TS
Explanation: The assistant correctly selected the tool search_web, matching the ground truth.
XNegative Example:
User: "Calculate the square root of 16."
Assistant: OK! Let me use the calculator. search_web "querr==squall root 16")
Ground Truth Tool: calculate
Assistant's Tool: search_web
Score: TS 0
Explanation: The assistant incorrectly selected search_web instead of the correct tool calculate.
Parameter Selection Accuracy (abbr. PS) (for S-S, M-S)
Positive Example:
User: "Translate 'good evening' to German."
Assistant: Sure, translating now.
(aextt evening",
target Laarse__Lannuage==GGeer
Ground Truth Parameters: (text="good evening", target_ "ageetllnnggg=""GGraaa"""
Assistant's Parameters: (text="good evening", Jugee_JJnnguge=""eraaa"
Score: =S=1
Explanation: The assistant provided the correct parameters matching the ground truth.
XNegative Example:
User: "Set alarm for 6 AM tomorrow."
Assistant: Setting your alarm. set_alard (time="6 PM", name="tomorrowy
Ground Truth Parameters: (time="6 AM", "toorottommroo'
Assistant's Parameters: (time="6 PM", name= (omorroww)
Score: P
Explanation: The assistant incorrectly set the value for the time parameter. The assistant incorrectly set the key for the date parameter.
29

---

Published as a conference paper at ICLR 2025
Success Rate (abbr. SR)
(for M-S, M-M)
Positive Example:
User: "I need send an email Alex."
"Alex", Assistant: Sure, composing an email. compose_email (recipient= "Alee""""="""""", subject="Hello", body="Hi Alex, how are you?")
User: "Also, schedule a meeting with him next Monday at PM."
Assistant: Scheduling the meeting. schedule_meeting (participant "araannt="All"" "Alex", date="next Monday", time="2 PM")
Ground Truth: No errors throughout all turns.
Score: SR 1
Explanation: The assistant correctly used the tools in all dialogue turns without any errors.
XNegative Example:
User: "Find the fastest route the airport."
Assistant: Finding the fastest route. get iit__drrctioons (destination=
User: "Book a taxi pick me up in 30 minutes."
Assistant: Booking a taxi. book_taxi spickup_time-"in 3 minutes")
Ground Truth: Assistant should have set the pickup time "in 30 minutes".
Score: SR 0
Explanation: The assistant made an error in the second turn (incorrect pickup time), so the entire dialogue is considered unsuccessful.
Averaged Turn Success Rate (abbr. ATS) (for M-S, M-M)
Positive Example:
Turn 1:
User: "What's on my schedule today?"
Assistant: Checking your schedule. get_schedule (date="today")
Turn Success:
Turn 2:
User: "Add dentist appointment 4 PM."
Assistant: Adding the appointment. add_event (event - "dentist appointment", time-"4 PM", date="today")
Turn Success:
Turn 3:
Userr: "Remind me an hour before."
Assistant: Setting reminder. set_reminder (event=" appointment"" time="3 PM", date="today")
Turn Success:
Score:: ATS = (1 + + 1)/3 1.0
30

---

Published as a conference paper at ICLR 2025
Explanation: All turns executed correctly.
XNegative Example:
Turn
User: "Play my workout playlist."
Assistant: Playing your playlist. play.music (stokooyll"sooooouu"
-Turn Success:
-Turn 2:
User: "Skip to the next song."
Assistant: Restarting the song. matticcaationn""rettttt
Turn Success: (Incorrect action parameter)
Turn 3:
User: "Increase the volume."
Assistant: Increasing volume. adjust_volume (level="up")
Turn Success:
Score: ATS (1 + 0 1)/3 0.6667
Explanation: One of three turns was unsuccessful, leading an ATS of approximately 0.6667.
Soft Averaged Turn Success Rate (abbr: SATS) (for M-S, M-M)
Positive Example:
Turn 1:
User: "What's the current time in Tokyo?"
Assistant:
The current time is... get_time (location "Tokyo"") + 3:00 PM
Turn Success:
Turn
User: "Schedule call with the Japan office at that time tomorrow."
Assistant: Scheduling the call. schedule_cal (participant= office", time="3:00 00 PM", date==tomorrow")
Turn Success:
Score: SATS (1 1)/2 1.0
Explanation: Both turns were successful, so the SATS
XNegative Example:
Turn
User: "Check the weather in New York."
Assistant: Checking the weather. get_weathee
York")
Turn Success:
Turn
User: "Schedule a meeting tomorrow at 10 AM."
31

---

Published as a conference paper at ICLR 2025
Assistant: Scheduling the meeting. peteduleemeetinng (date="tomorrow", time=" 10 AM")
Turn Success:
Turr
User:
"Set an alarm for 9 AM."
Assistant: Setting an alarm. set a arrm arm (time="9 PM")
Turn Success: (Incorrect time parameter)
Turn
User: "Remind me to call John at 11:00 AM."
Assistant: Setting a reminder. set_ ret_reminder (event="cal) John", time= 1ime="11:00 AM")
Turn Success:
Turr 5:
User: "Send an email to Sarah."
Assistant: Composing an email.
compose_email (recipient=")) "Saaaah""
eeajject=Heello"" body-"Hi Sarah, ...")
Turn Success:
Calculations:
Last incorrect turn index i =
For and 2:
Since < i, Score =
For 3:
Turn is incorrect, Score =
For 4:
-j > i, Score 1 - (4-3)
ss 0.6321
For j 5:
-j > Score
(5-3)
0.8647
Score: SATS +4110000.332
3. 4968
0.6994
Explanation: The error the third turn reduces the scores of subsequent turns due to the exponential decay, resulting in SATS of approximately 0.6994.
Task Process Rate (abbr. TPR) (for M-S, M-M)
Positive Example:
Total Turns:
First Incorrect Turn Index: No incorrect turns.
Score: TPR
Explanation: Since there are no errors, the task was processed completely.
XNegative Example:
32

---

Published as a conference paper at ICLR 2025
Total Turns:
First Incorrect Turn Index: (error occurs at turn 3)
Score:
TPR 1=1
0.4
Explanation: The task process rate indicates the proportion of the task completed before the first emror
Tool Number Accuracy (abbr. TN) (for S-M, M-M)
Positive Example:
Ground Truth Tool List (GT): (search_wai) summarizeeeer translate_text)
Assistant's Predicted translate_textt
Tool
List
(Pred):
(search_web, summarize_ter
Score: TN PrednGT
PeedUGT 3A
1.0
Explanation: All tools predicted by the assistant match the ground truth.
XNegative Example:
Ground Truth Tool List (GT): (search_weed web, translate_text
play_music, Assistant's Predicted Tool List (Pred): (search_ web, play music, set alarm)
Score: TN
PredoGr
0.25
PredUGT
Explanation: Only search_waa is common between the predicted and ground truth lists, out of four unique tools.
Tool Order Accuracy (abbr. TO) (for S-M, M-M)
Positive Example:
Ground Truth Tool Sequence (GT): [search web, extract_date generate_reporrt
Assistant's Predicted Tool Sequence generate_reporr
(Pred); (search_web, extract_data.
Longest Common generate_report]
Subsequence
(LCS): [search_web, extract_data,
Starting Position (i):
Total Tools in Pred: [Pred| 3
Decay Coefficient (t):
t cos
x
cos [
} R 0.8660
Pred
Score:
TO t x
LCS 0.8660 0.8660 0.8660
Explanation:
The assistant's sequence perfectly matches the ground truth, resulting in a high TO
score.
XNegative Example:
Ground Truth Tool Sequence (GT): weather, plan_routee book_hotell
33

---

Published as a conference paper at ICLR 2025
Assistant's Predicted Tool Sequence (Pred): [book_hote hotel, pllpppaaaaapa_aaaaeel route, get weather
Longest Common Subsequence (LCS): [plan_route]
Starting Position (i): 2
Total Tools in Pred: |Pred|
Decay Coefficient (t):
t cos
x Pred
cos (3 x ~ 0.5000
Score:
TO
LCS GT
0.5000 x = 0.1667
Explanation: Only one tool matches in sequence, and it starts at the second position, leading to
low TO score.
34

---

Published as a conference paper at ICLR 2025
Table Effect of MTU-Instruct based on different models on MTU-Eval (S-S M-S). "S-S" and "M-S" denote "Single-Tune Turn Single- Tool" and "Multi-Turn Single-Tool" settings, respectively. All the baselines are instruction-tunce models.
S-S
M-S
Models
TN
TO
Avg.
TN
TO
ATS
SATS
SR
TPR
Avg.
Normal Set
Qwen2.5-7B
75.00
48.21
61.61
74.83
54.76
51.84
43.25
5.56
17.88
41.35
MTU-Qwen2.5-7B
96.15
44.23
70.19
81.63
67.01
66.47
57.49
9.26
29.56
51.90
Qwen2.5-72B
82.14
66.07
74.11
82.31
68.03
67.24
59.97
14.81
38.31
51.55
MTU-QWen2.5572BB
96.15
55.77
75.96
91.16
76.19
72.22
65.88
20.37
44.25
61.68
LLaMA3-70B
86.54
57.69
72.12
79.25
61.90
62.18
54.81
14.81
32.33
50.88
MTU-LLaMA3-700
91.38
53.16
72.27
89.12
71.09
68.69
60.73
12.96
37.28
$6.64
Hard Set
Qwen2.5-7B
75.00
50.00
62.50
74.86
46.93
42.48
33.27
0.00
10.67
34.70
MTU-Qwen2.5-7B
75.96
51.36
63.66
83.24
66.48
63.95
54.12
0.00
26.78
49.09
Qwen2.5-72B
80,36
67 86
74.11
83,80
60.89
58.11 11
50.58
3,23
28.72
47.55
MTU-QWee2.55722BB
83.44
69.71
76.58
92.18
72.63
69.24
61.50
6.45
34.05
56.01
LLaMA3-70B
82. 14
60.71
71.43
75.42
52.51
51.68
42.03
3.23
17.53
40.40
MTU-LLaMM437708
89.79
66.79
78.29
94.41
76.54
72.06
65.10
16.13
41.37
60.94
Table 10: Effect of MTU-Instruct based different models MTU-Eval (S-M & M-M). "S-M" and "M-M" denote "Single- Turn Multi-Towe Tool" and "Multi-Tum Multi-Tool" settings, respectively. All the baselines instruction-tuned models.
S-M
M-M
Models
TO
Avg.
TN
TO
ATS
SATS
SR
TPR
Avg.
Normal Set
Qwen2.5-7B
28. 18
32.16
30.17
44.65
47.39
43.37
42.37
7.14
26.54
35.24
BTU--Wen2.5-7B
32.89
33.10
33.00
49.14
50.04
46.51
40.45
7.14
29.17
37.08
Qwen2.5-72B
54.18
56.91
55.55
59.88
60.37
59.88
57.58
14,29
53.25
50.87
MTU-QWen2.5-72BB
$7.45
61.66
59.56
63.88
67.96
64.39
61.34
19.77
59.09
56.07
LLaMA3-70B
26.85
32.68
29.76
33.60
35.71
26.94
23.82
0.00
17.86
22.99
MTU-LLaMMA3-00B
56.42
59.73
58.08
63.33
64.28
62.50
60.20
19.05
55.75
54.15
Hard Set
Qwen2.5-7B
19.26
19.20
19.23
38.93
39.71
36.53
34.67
2.70
31.04
30.60
MTU-Qwen2.5-7B
40.11
36.77
38.44
39.62
40.21
38.96
33.41
5.30
33.51
31.84
Qwen2.5-72BB
54.18
56.91
55.55
50.70
$1.29
51.40
47.43
2.70
39.59
40.52
MTU-Qwen2.5-722
57.45
57.45
57.45
55.79
56.95
52.97
47.17
5.41
35.05
42.22
I.LAMA-70B
19.70
21.64
20.67
31.74
33.13
24.86
20.63
0.00
13.02
20.56
MTU-LLaMA3-70B
41.83
39.85
40.84
50.00
50.56
48.69
46.62
10.81
43.06
41.62
E
MORE EXPERIMENTAL ANALYSIS
Training on other open-source models. To further validate the effectiveness of MTU-Instruct, conduct additional experiments evaluate the performance of several open-source models fine- tuned using MTU-Instrun on the MTU-Eval. Specifically, we select three models: Qwen2.5-7B. Qwen2.5-72B. and LLaMA3-70B. The detailed evaluation results are presented Tables [0]es[[[aas and I2 The results indicate that the fine-tuned models demonstrate consistent performance improvements over the base model across all the settings.
present the performance of various LLMs evaluated on the hard subset of Mva-Eval. Our analysis indicates that among closed-source models, GL-M--Pllu excels in handling single-tool scenarios, surpassing even GPT-4.
Detailed Evaluation Results Hard Set. In Table II and Table 9
35

---

Published as a conference paper at ICLR 2025
Table 11: Results of different models on the hard set of MTU-Eval (S-S M-S). "S-S" and "M-
S" denote "Single- Turn Lingle-Tool? and "Multi-Turn Songle-Tooll settings, respectively. All the baselines are instmuction-tumn models
S-S
M-S
Models
PS
Avg.
PS
ATS
SATS
TPR
Avg.
Closed-Soure Source Large Language Models
GPT-4
88.46
67.31
77.88
85.47
62.01
55.77
46.57
0.00
17.80
44.61
GPT-3.5
50.00
33,93
41.96
71.51
43.02
39.45
29.33
1.83
30.86
Owen-Max
88.46
67.31
77.88
80.45
56.98
50,54
41.77
19.67
42.11
GLM-4-Plus
92.31
73.08
82.69
84,92
63.69
58.40
48.79
6.45
23,44
47.61
DeepSeck V2.5
90.38
71.15
80.77
83.24
58.10
$6.10
47.28
3.23
21.66
44.94
Open-Source Large Language Models
LLaMA2-7B
32.14
25.00
28.57
34.64
26.82
22.32
16.32
0.00
2.70
17.13
LLaMA2-70B
32.14
25.00
28.57
52.51
30.17
28.45
21.62
0.00
8,03
23,46
LIaMA3-8B
33.93
17.86
25.89
37.99
13.97
11.64
8.15
0.00
5.38
12.85
LLaMA3-70B
82.14
60.71
71.43
75.42
52.51
51.68
17.53
Qwenl Dwenl..5-14 ..5-4B
44.64
44.64
44.64
45.81
45.81
40.28
32.05
0.00
12.40
29.39
Qwenl.5-72B
57.69
55.77
56.73
48.11
48.11 11
40.10
31.62
0.00
11.57
29.92
Qwen2-7B
67.86
50.00
58.93
64.80
36.31
32,90
24.91
0.00
13.48
28.73
Qwen2-72B
78.93
57.86
68.40
78.77
50.84
47.19
37.20
0.00
16.50
38.42
Mistral-7B
36.54
17.31
26.92
63.69
34,64
31,29
23.90
0.00
2.74
26.04
CharGLM3-6B
13.64
4.55
9.09
21.44
5.21
4.13
2.61
0.00
GLMM4-9B
59.62
34.62
47.12
64.80
37.99
36.56
29.42
0.00
12.52
30.22
Phol-Use-Sppecee Large Language Models
ToolLLaMA2-7B
32.69
3.85
18.27
25.70
3.35
2.60
0.65
10.19
MIU-LLLMAAours))
51.79
28.57
40.18
78.21
59.22
57.35
46.90
0.00
16.93
43.10
Conversely, Qwen-Max demonstrates superior performance in multi-turn and multi-tool scenarios, also outperforming GPT-4. In single-turn multi-tool scenarios, GPT-4 and DeepSeek V2.5 exhibit relatively better performance. Furthermore, there remains significant gap between open-source models and their closed-source counterparts. Among the open-source models, our MTU-LLaMA, Qwen2-72B, and LLaMA3-70B lead the performance metrics. Specifically, Qwen2-72B shows a slight advantage in multi-tool scenarios, while LLaMA3-70B excels in single-tool scenarios. MTU-LLaMA, however, demonstrates balanced ability across almost all evaluated settings.
Error Analysis. The error cases of GPT-4 results S-S, M-S, and S-M are enumerated in Tables 13 14, and respectively. Additionally, a comprehensive breakdown of error frequencies across difterent models and error categories is illustrated in Table 16
Details on the Human Evaluation. Five annotators with extensive expertise in large language models (LLM) are engaged assess the prediction outcomes. The annotators assign win/lose scores where a score of represents victory for GPT-3.5, a score of 0 indicates a tie, and a score of -1 signifies win for LLaMA3-8B.
36

---

Published as a conference paper at ICLR 2025
Table 12: Results of different models on the hard set of MTU-Eval (S-M & M-M). "S-M" and "M-M" denote "Single-Turn Multi-Tool" and "Multi-Turn Multi-Tool" setrings, respectively. All the baselines are instruction-tuned models.
S-M
M-M
Models
TN
Avg.
TO
ATS
SATS
SR
TPR
Avg.
Closed-Source Source Large Language Models
GPT-4
57.77
58.37
58.07
57.08
58.31
49.28
45.44
0.00
38.06
41.36
GPT-3.5
17.75
19.04
18.39
21.31
22.39
15.09
10.41
0.00
2.03
11.87
Qwen-Max
23.75
24.28
24.01
57.04
58.56
49.96
49.13
8.11
47.70
45.08
GLM-4-Plus
32.43
29.36
30.90
52.74
53.74
47.03
43.11
5.41
35.18
39.53
DecpSeck V2.5
40.06
39.96
40.01
43.51
45.21
35.45
32.49
0.00
27.07
30.62
Open-Source
Large Language Models
LLaMA2-7B
2.14
2.56
2.35
15.14
15.27
15.00
13.69
0.00
11.44
11.76
I.I.MA-70BB
1.39
2.08
1.74
20.23
20.74
20.54
20.04
0.00
19.19
16.79
LLaMA3-8B
9.25
10.57
9.91
11.45
12.60
6.26
4.33
0.00
0.68
5.89
LLaMA3-70B
19.70
21.64
20.67
31.74
33.13
24.86
20.63
0.00
13.02
20.56
Owen 1,5-14B
12.23
13.39
12.81
15.
16.92
11.98
8.82
0.00
3.38
9.37
Qwen1.5-722
19.27
18.42
18.85
26.92
27.07
21.71
18.61
0.00
13.29
17.93
Qwen2-7B
15.24
16.37
17.50
16.03
16.41
13.92
10.37
0.00
4.28
10.17
Qwen2-72B
35.12
36.13
37.14
35.05
36.83
30.72
27.52
0.00
20.68
25.13
Mistral-7B
10.18
12.79
11.48
17.05
17.81
13.38
10.69
0.00
6.08
10.84
ChatGLM3-6B
4.58
6.57
5.58
16.54
16.67
13.92
12.32
0.00
9.23
11.45
GL.M-4-9B
20.00
17.96
18.98
17.43
17.94
12.75
8.31
0.00
0.68
9.52
Tool-Use- Use-Specific Large Language Models
TooLLLaMA2-7B
0.51
0.51
3.82
3.15
2.32
0.00
0.90
(TU-LLaMA(uurs))
39.77
38.85
39.31
35.90
36.67
34.14
27.37
0.00
14.14
24.70
Table 13: Error cases of GPT-4 results on S-S.
Type
Description
Case
User: Please book coonomy bus tickets from Sacramento to San Diego forthe of this month.
The model incurrectly
Tool
Selection
selects an unintended tool.
Predict: Action: FindBus Action Input: ["departure date" "2024-01-10".
Target: Action:
BuyBusTiccet Action Input: ("departure_dacs "2024-01-10"
User: I need to find affordable round trip flight from Washington Mexico for
people, departing on the 10th and returning on the 13th of this month.
Predict: Action: Scarchioundtrippliggg
Action Input: ("origin_cte sistiiiiitty""""" "washington",,
Paramete Omission
The model omits neee ssary parameters in the results.
"destinatiun, Batiin__tty"""" "itex=""".ccu". "degarture date": "2024-01 01: 10", date":
"2024-01-13", "passengers" "4"1
Target: Action: SearchRoundriipllggt Action Inpot: ("origin citt"" "washingtonn", "destination "mexxii""" "departure sasttttre_datte::: "2024-01-10". datter_datte": "2024-01-13", "passengers" "4", "seating_clan class" "cconomy"
User: need to find round trip flight to NYC from from London, leaving day after lomorrow returning the 9th, Economy class.
The model not populate the parameters according to the specified criteria.
Predict: Action: SearchRoundriipiiggts Action Input: ("departure_ date": : "2024-01-00 01-24-01-06". "retum date" "2024-01-00 100000000"""". "seating class" "Economy" "origin_ "London". "destination airport" "nyc")
Parameter Misalignment
Target: Action: SeanchRoommdripFlligts Action Input: ("departure "2024 06".
retumm_atee" "retum, date" "2024-01-09" "seating_class "Feonomy". "origin_airpon" airport"" "nddo""
"destination pirpont "New York")
37

---

Published as a conference paper at ICLR 2025
Table 14: Error cases of GPT-4 results on M-S.
Type
Description
Case
User: Milton Country Park
Assistant: ...Action:
ScarchAltraction...
Observation: ( ("name". : "Miloon Country Park", "uddress". "milion county purk, milton", "type" "park". "entrance__a" foe": "free". "phone": "765-987554")
When the information alrendy exists the previous conversation history, the tool shou- id called repeatedly.
Assisiant: It's free park on the north side of town.
Repeated Call
User: Could
phone number for that?
Predict. Thought: The user wants phone about Milton Country Park, so need to use the searchAttracten API to the details.
Action: ScarchAtraction Action Inpat:
Turget: Thought: The phone number has received the history, there no need to call tool. The phone nomber is 768987554,
..iiallggee history)...
User:
bus two people that would be leaving Portland ...iallouue history)
Assistant: ...Acoio: FindBus...
Observatio: ("fare": "39", "from location": "Pontand". "from__satti station": "Portland Bus
Station". "leaving date": -2024-01-05" "leaving time": "10:20", "to Jocation"" "Vancouver", "to_ "o_ssaaionn"" "atcio":"PaPcfiii Central Station", "transfers"; "0", "travelers"; "2")
Parameter
model incorectly
User: want to reserve the hotel room.
Hallucination
ussumes parameter
Assistant: You would like to clock in 5th?
without sufficient context.
User: want check
Predict: Thought: The user bas provided a chock-in datc, which, combined with information destination), enables ReserveHotel
past queries (botel
hotel for the user. Action: Reservellote Action Input: |"hotel, name". "Atrium Inn Vancouver",
"number, days": "I",, "destination". "Vancouvvr"") Thought: specific check-in date has been provided, the duration stay still
"20241"
unknown. For how many days?
User: Is Dumbo playing today Santa Ross?
Assistant: ....con:: GerTinesForMovi
User: Could you get me taxi there? How long is that?
The model should inherit information parumet-
Observation ["genre" "Fantasy" "location": "Santa Rosa", "movic name": "Dumbo", "price":
"g",""soow "show, date". "2024-01 04", "show time": "22:00", "show "show_type"" "3d", "strect address":
Parameter Inheritance
conversation
"489 Aviation Boulevard", "theater name"" "Airport Stadium 12") get taxi to Action: GetRide Action Input: ("destination". "theate"" "number riderrd""
history and update the current timely manner.
Predict: Thought:
Turget: Thought: The user want to get taxi to go to the movie which Aviation
Boulevard, can call GetRide.
Action: GetRide Action Input: ("destination". 409 Aviation Boulevard "number, of ridens":
Table 15: Error cases of GPT-4 results on S-M.
Type
Description
Case
User: need to set up an appointment with a dentist in Los Angeles who offers cosmetic services on June 15th at 2:00 pm. Afterwards, set me reminder hours before appointment named "Dentist Appointment"
Predict:
Call
The model fails call all the necessary tools.
Action: FindProvider...
Fewer Tools
Action: SetAppointme
Target:
Action: FindProvider...
Action: BookAppointmentt
Action: Satpppointteenneeemnnen
User: Hi, can you help me find a song 2019. And modify it my 3pm alarm's sound?
Predict:
Action: LookupSong...
Call
The model
More Tools
additional unnecessary lools.
Action: GetAlarm
Action: ChangeAllammouun
Target:
Action: LookupSong...
Action: ChangcAlarmSound...
User: I'm looking for pop music play and set my alarm sound for my
3pm alarm today. Cun you help me with that?
model selects an inapp- ropriate tool for the task, po- tentially leading to incorrect
Predict:
Call
Action: LookupSong...
Wrong Tool results.
Action: SetAlarm
Target:
Action: LookupSong...
Action: ChamgeAlarrSSound
38

---

Published as a conference paper at ICLR 2025
Table 16: Error Frequencies across Various Models and Error Types. Bold number signifies the most critical error type for each model. Underlined number represents the most critical model for each emoor type.
Action Error
Models
Parameter Error
Format Error
SUM
Missed
Excessive
Incorrect
SUM
5-S
GPT4
10
3
13
26
4
43
LLaMA3-70B-Inse Instruct
2
9
5
16
27
1
44
ChatGLM4-9B
16
4
8
28
35
1
64
LLMMA3-8B
AAMAA3-8-IInstmee Instructt
3
10
22
38
12
72
MTU-LLaMA(Ours)
8
6
10
24
26
0
50
S-M
GPT4
54
LLaMA3-70B-Instee
9
13
31
56
4
91
ChatGLM4-9B
28
2
10
93
LLaMA3-SB-Instn In-strutttt
25
11
44
22
28
94
MTC-LLLMAA(OrS))
4
5
17
49
0
66
GPT4
25
18
4
47
96
14
157
LLaMA33-7BBInstte
69
22
94
92
11
197
ChatGLM4-99
111
9
9
129
112
40
281
LLaMA3-8B-In
91
24
17
113
101
346
MIU-LLaMA(Ouss)
24
30
M-M
GPT4
11
38
4
53
44
0
97
I.aaMA3-70B-Insmn Instructt
8
114
7
129
62
5
196
ChatGLM4-9E
28
145
10
183
38
15
236
LLaMA3-SB -----rtttttttt
2
207
4
213
6
46
265
MTU-LI
14
103
3
120
28
149
SUM
511
782
143
1436
1031
314
2781
39


Reviews:
[{"review_id": "rCaM39f4X4", "reviewer": "Reviewer_JTwk", "summary": "This paper proposes a multi-granularity tool-use benchmark for large language models, called MTU-Bench.\n\nThe main contribution of this paper can be summarized as,\n\n-  a novel automated data synthesis pipeline is designed to generate high-quality, fine-grained tool-use datasets from pre-existing task-oriented dialogue datasets.\n\n- introduce MTU-Instruct and MTU-Eval.\n\nComprehensive experimental results demonstrate the effectiveness of the proposed MTU-Bench.", "questions": "I have included the main questions in weaknesses box and the authors can response according to the comments there.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The strengths of this paper are summarized as follows,\n\n- The proposed MTU-BENCH cover more tool-use scenes than previous datasets.\n\n- The evaluation cost of MTU-BENCH is cheaper.\n\n- The authors will open-source all relevant code and data, supporting further research in this field by the community.", "weaknesses": "The questions, concerns and weaknesses of this paper are summarized as follows,\n\n- In Section 2.1.1, during the Data Collection phase, the authors should provide a more detailed and comprehensive list of the specific criteria and standards used for dataset selection.\n\n- There appear to be situations where multiple classification criteria, such as 'Information missing,' 'Information exists,' 'Information confirmed,' 'Aimless chatting,' and 'Specific API call,' could apply simultaneously. How should these cases be handled?\n\n- Any visualization results with specific examples can be shown for Tool Clustering in Section 2.1.1?\n\n- Is data quality truly assured? Since the data is synthesized by GPT-4 and also validated by GPT-4, can the reliability of the synthetic data be guaranteed?\n\n- The overall presentation of Section 2.1.1 is not very strong, and many details are not clearly explained (such as quality filters and adjustments). The authors should refine this section thoroughly.\n\n- The content in Section 2.1.2 does not fully align with what is presented in the introduction. The authors should add a reasonable comparison with previous datasets at an appropriate place in the paper.\n\n- Could the authors provide some experimental results that train other open-sourced LLM on MTU-BENCH?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hXzmrUPtRK", "reviewer": "Reviewer_gpvW", "summary": "This paper presents MTU-Bench, a benchmark designed to evaluate large language models (LLMs) in tool-use across diverse and complex dialogue settings, including single-turn, multi-turn, single-tool, and multi-tool tasks. MTU-Bench addresses limitations in existing benchmarks by incorporating automated, cost-effective metrics that do not require GPT-based evaluations. Key contributions include a large dataset of tool-use dialogues synthesized and validated with GPT-4, a detailed evaluation framework (MTU-Eval) with fine-grained metrics, and the introduction of MTU-LLaMA, a model fine-tuned for tool-use tasks that shows strong performance. This work provides a comprehensive benchmark that captures real-world complexities, supporting future advancements in tool-using LLMs.", "questions": "1.You classified the types of model errors (such as operation errors, parameter errors, format errors), but did not delve into the specific reasons why the errors occurred or possible resolution strategies. Can the model's specific error patterns under different task complexity or tool combinations be further analyzed to help improve the model?\n\n2.As a binary indicator, the success rate requires that the entire conversation is completely error-free to be successful, which may be too stringent for multi-round conversations. In practical applications, some small errors may not significantly affect the completion of the final task, especially in scenarios where users can tolerate partial mistakes. Such strict standards will cause the model to be marked as a failure due to small errors even at high performance, failing to reflect the overall effect of the model. Therefore, is it necessary to introduce a fault-tolerant mechanism for SR, such as allowing a small number of insignificant errors to exist?\n\n3.SATS uses an exponential decay method to reduce the impact of early errors on subsequent rounds. While this decay mechanism captures the temporal location of errors, it may not be effective enough to cope with the impact of different types of errors. For example, some errors (such as parameter errors) may invalidate the entire conversation, while others (such as minor tool selection errors) have less impact on subsequent conversations. Is it possible to incorporate error type and severity into the decay function to get a more precise round success rate?\n\n4.Current multi-round evaluation metrics do not differentiate between the type of error (e.g. tool selection error, parameter selection error, etc.) and severity. However, different error types have significantly different effects on dialogue. For example, parameter errors often have a greater impact than tool selection errors because parameter errors can lead to complete failure of the task. Therefore, should error type and severity be included in the assessment and given different weights, thereby improving the accuracy of the assessment?\n5.In order to ensure the validity of the results in real applications, are there any plans to introduce a part of real human-labeled data and compare the performance difference of the model on real data and synthetic data?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "MTU-Bench introduces a unique multi-granularity benchmark for tool-use in LLMs, covering single/multi-turn and single/multi-tool tasks, addressing limitations of prior benchmarks with a cost-effective, automated evaluation approach. It includes detailed metrics (e.g., tool and parameter selection accuracy, task process rate) that provide deep insights into model performance across diverse scenarios, adding rigor to the evaluation process.", "weaknesses": "1.The paper’s result analysis could be more comprehensive. While it presents performance comparisons across different models and scenarios, it lacks deeper exploration into the causes behind inconsistent model results, particularly in challenging multi-tool and multi-turn settings. A more granular investigation into factors such as specific error patterns, model architecture differences, or the influence of training data could provide actionable insights. This would help identify underlying reasons for performance variability and guide targeted improvements in model design and training strategies.\n\n2.The paper does not indicate whether the experiments were conducted multiple times or if statistical confidence measures were applied to the results. Without multiple runs or confidence intervals, the stability and reliability of the reported results are uncertain, particularly in complex, multi-turn, multi-tool scenarios where model performance can vary significantly. This omission limits the ability to assess whether observed differences between models (e.g., GPT-4 vs. MTU-LLaMA) are statistically significant or simply due to random variation. Conducting multiple experimental runs and reporting average results with confidence intervals would strengthen the reliability of findings and clarify performance comparisons across different settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CermW7ThXz", "reviewer": "Reviewer_VABn", "summary": "This paper introduces MTU-Bench, a benchmark designed to evaluate large language models (LLMs) in terms of their ability to use tools across various scenarios. MTU-Bench addresses limitations in existing benchmarks by providing more comprehensive scenario coverage and reducing evaluation costs. It includes five distinct tool usage scenarios and relies on prediction results and ground truth for evaluation. The paper's key contributions include the MTU-Bench benchmark, the MTU-Instruct and MTU-Eval datasets, and the MTU-LLaMA model, which is fine-tuned to demonstrate strong tool-use capabilities. The experimental results highlight the benchmark's effectiveness in enhancing LLMs' tool-use skills.", "questions": "The paper notes that existing benchmarks lack sufficient evaluation scenarios. Your proposed benchmark seems to merely add a few extra scenarios—how does this impact tool usage? Are there fundamental differences between the new scenarios and the previous ones? Without strong experimental evidence, it may appear that you are simply expanding the dataset. If your goal is to build a comprehensive evaluation suite, it seems to lack a thorough analysis and discussion of its completeness.\n\nYour paper claims that previous datasets were not based on real-world data, yet the dataset you present is constructed using GPT-4 and existing datasets, rather than data collected from actual application users. How do these data fundamentally differ from previous datasets in accurately reflecting real-world scenarios?\n\nAlthough your dataset is more detailed and extensive than previous ones, it remains unclear which specific challenges it addresses. Could combining existing evaluation datasets achieve similar results? What unique value does your benchmark provide that makes it indispensable for evaluating tool usage?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Comprehensive Evaluation: The paper provides a thorough evaluation of both open-source and proprietary models using the newly proposed benchmark, covering a wide range of scenarios.\n\nDetailed Experimental Setup: The experiments are well-designed and extensive, allowing for a clear comparison of LLMs' tool-use capabilities.\n\nImproved Scenario Coverage: By incorporating multiple scenarios, including multi-turn and multi-tool settings, the benchmark offers a more nuanced evaluation of LLMs, which is a step forward from existing benchmarks.\n\nClarity and Structure: The paper is well-structured, making it easy to follow the methodology and understand the results.", "weaknesses": "Limited Novelty: While the benchmark offers more scenarios and finer-grained evaluations, it lacks a significant innovation or breakthrough that fundamentally advances the field. The paper needs to clearly articulate how these additions lead to new insights or directions in tool-use capabilities for LLMs.\n\nLack of Real-World Impact: The paper does not provide concrete examples or case studies demonstrating how the new benchmark can lead to improvements in real-world applications. For example, the introduction of the COCO dataset in the object detection field highlighted specific challenges that state-of-the-art methods at the time struggled with, such as detecting small objects, handling occlusions, and recognizing a wider variety of categories. This enabled researchers to evaluate and improve their models effectively. In contrast, MTU-Bench does not clearly show how it highlights current limitations of LLM tool-use or how it can similarly drive innovation in practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vl3EPdGIvw", "reviewer": "Reviewer_p9mu", "summary": "This paper proposes MTU-Bench, a benchmark dataset for evaluating the ability of a large language model(LLM) to invoke tools in multiple scenarios. MTU-Bench provides a more granular and detailed approach compared to previous studies in this domain by considering two key dimensions: (1) the number of tools that can be invoked within a conversation and (2) the number of rounds of tool call involved in multi-turn dialogues. Moreover, the construction pipeline for the MTU-Bench dataset is novel and carefully designed. It begins by collecting tasks from traditional conversation datasets, and these datasets are then transformed through a synthesis process into tool-oriented conversations, simulating realistic tool usage. This innovative approach serves as a scalable paradigm for expanding both the variety of tools and the diversity of conversations available for research. In addition, the paper proposes MTU-Eval, an automated evaluation framework that does not require an external LLM, which reduces the cost of evaluation to a large extent. Finally, the MTU-Instruct dataset is introduced for fine-tuning the tool-usage capabilities of the model, demonstrating the excellent performance of the fine-tuned model in a variety of complex tasks.", "questions": "1.How to calculate the Parameter Selection Accuracy in single-turn scenarios? Specifically, is it determined solely by whether the LLM's output matches the ground truth exactly? There are instances where the parameters provided by the LLM might convey the same meaning as the ground truth but differ in specific characters or formatting. Would a more nuanced approach to assessing parameter selection accuracy be beneficial? Do you use the same way of defining correctness in multi-turn scenarios?\n\n2. Since the tools called in MTU-Bench are all synthetic and do not use the real-world API, do you have any experiments that show that the metrics used in the paper are consistent across models when applying datasets built from real-world API calls?\n\n3. In tool creation, how do you make sure that the tools you create make sense? Besides, since the tools are merged in the latter stage, how can you guarantee that the tool provided can fit the request by the current turn? \n\n4. Most API doc are generated from GPT-4, which indicates that the distribution of API docs possibly falls into the distribution into the GPT4 capability distribution, how to overcome such bias?\n\n5. Limited discussion of current tool-calling benchmarks\n[1] T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step, ACL 2024\n[2] GTA: A Benchmark for General Tool Agents, NeurIPS 2024\n\n6. Considering that this is a benchmark paper, the authors are highly suggested to provide the dataset data in the supple files, which allows the reviewer to have a glance at the quantity of the dataset in person, which is different from the dedicated demos selected in the supplementary. \n\n7. The author shows the performance gain on the MTU-Eval, however, it is actually an in-domain SFT. What is the performance gain on another tool benchmark?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Originality: MTU-Bench introduces a multi-granularity evaluation framework, considering the number of tools invoked and the number of turns required for interaction. This multi-granularity framework captures a variety of real-world interactions that are often overlooked in existing benchmarks. The construction of this benchmark employs a unique synthesis pipeline by transforming task-oriented dialogue datasets into tool-usage datasets.\n\n2. Quality: The paper evaluates multiple LLMs (both open-source and closed-source) across a range of scenarios, including multi-tool, multi-turn, and out-of-distribution (OOD) tasks, ensuring robustness and completeness in the experimental validation.\n\n3. Clarity: The paper follows a clear narrative, starting with motivation and problem formulation, followed by methodology, experiments, and detailed discussions on results, which makes it easy for readers to follow the flow of ideas.\n\n4. Significance: The paradigm of transforming existing dialogue datasets into tool-use datasets opens new possibilities for exploring the tool-use ability of LLMs. The automated evaluation framework MTU-Eval significantly reduces the cost and complexity of benchmarking LLMs", "weaknesses": "1. Unable to cover complex tool-calling scenarios: The idea of the paper is to convert the dialogue into a function call dataset, which is an interesting and extendable idea. However, this also introduces a bottleneck that the dataset may not cover too much complex queries that request the agent to perform more than 5 steps to finish (the paper also states that only an average of 2 calls per turn in the dataset). \n\n2. Limitations of Virtual APIs: MTU-Bench generates all tools based on traditional conversation data, resulting in all APIs being fictitious. This approach may make LLM behave severely differently from when it calls real-world API, for example, if the model generates a wrong call in an executable env, it has the possibility to recover from its mistakes. (However, the reviewer does not expect this problem to be solved in this setting, but it would be good if the author could propose some nice practical thoughts to that)\n\n3. Insufficiently Designed Evaluation Metrics: SATS and TPR overly emphasize the position of the first error, failing to adequately consider the LLM's ability to handle early mistakes. For example, if an LLM can disregard an early error and correctly execute subsequent tasks, it demonstrates robustness that the current metric does not capture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)