ToolACE: Winning the Points of LLM Function Calling
Authors: Weiwen Liu
Xu Huang
Xingshan Zeng
xinlong hao
Shuai Yu
Dexun Li
Shuai Wang
Weinan Gan
Zhengying Liu
Yuanqing Yu
Zezhong WANG
Yuxian Wang
Wu Ning
Yutai Hou
Bin Wang
Chuhan Wu
Wang Xinzhi
Yong Liu
Yasheng Wang
Duyu Tang
Dandan Tu
Lifeng Shang
Xin Jiang
Ruiming Tang
Defu Lian
Qun Liu
Enhong Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Tool leaning
Function calling
Large language models
Primary area: foundation or frontier models, including LLMs
TL;DR: Improving the function calling capability of large language models with data accuracy, diversity, and complexity.

Abstract: Function calling significantly extends the application boundary of large language models (LLMs), where high-quality and diverse training data is critical for unlocking this capability. However, collecting and annotating real function-calling data is challenging, while synthetic data from existing pipelines often lack coverage and accuracy. In this paper, we present ToolACE, an automatic agentic pipeline designed to generate accurate, complex, and diverse tool-learning data, specifically tailored to the capabilities of LLMs. ToolACE leverages a novel self-evolution synthesis process to curate a comprehensive API pool of 26,507 diverse APIs. Dialogs are further generated through the interplay among multiple agents, under the guidance of a complexity evaluator. To ensure data accuracy, we implement a dual-layer verification system combining rule-based and model-based checks. We demonstrate that models trained on our synthesized data---even with only 8B parameters---achieve state-of-the-art performance, comparable to the latest GPT-4 models. Our model and a subset of the data are publicly available at https://huggingface.co/Team-ACE.

Published as a conference paper at ICLR 2025
TOOLACE: WINNING THE POINTS OF LLM FUNCTION CALLING
Weiwen Liu Xu Huang" 33 Xingshan Zeng?? Xinlong Hao", Shuai Yu², Dexun Li2, Shuai Wang". Weinan Gan?, Zhengying Liu?, Yuanqing Yu Zezhong Wang", Yuxian Wang', Wu Ning Yutai Hou', Bin Wang", Chuhan Wu Xinzhi Wang". Yong Liu", Yasheng Wang 22 Duyu Tang', Dandan Tu Lifeng Shang?, Xin Jiang?, Ruiming Tang 22 Defu Lian 2 Qun Liu and Enhong Chen?
Shanghai Jiao Tong University
Huuwwei Noah's Ark Lab University of Science and Technology of China
Huaweii Technologies Co., Ltd
Tsinnhua University
'hhe Chinese University of Hong Kong
wwliu@sjtu.edu.cm zeng. wingg.innnsannneee huawwii.com wacpiaaangs@mmil.st..s......
ABSTRACT
Function calling significantly extends the application boundary of large language models (LLMs), where high-quality and diverse training data is critical for unlock- ing this capability. However, collecting and annotating real canction-calling data is challenging. while synthetic data from existing pipelines often lack coverage and accuracy. In this paper, we present ToolACE, an automatic agentic pipeline designed to generate accurate, complex, and diverse tool-learning data, specifically tailored to the capabilities of LLMs. ToolACE leverages a novel self-evolution synthesis process to curate comprehensive API pool of 26,507 diverse APIs. Dialogs are further generated through the interplay among multiple agents, under the guidance of a complexity evaluator. To ensure data accuracy, we implement a dual-layer verification system combining base- and model-based checks. We demonstrate that models trained 8B parameters-achivee state-of-the-art performance, comparable to the latest GPT-4 models. Our model and a subset of the data are publicly available https: htttps:/hugqiin http:///hggginngccghee ca/ Too/Team-AC
synthesized data-even with only
INTRODUCTION
Equipping Large Language Models (LLMs) with external tools has significantly enhanced the capability of AI Agents to solve complex real-world tasks Huang et al. (2024): Qin et al. (2023); Qu et al. 2024). The integration of function calling enables LLMs to access up-to-date information, perform delicate computations, and utilize third-party services, thereby unlocking a wide range of potential applications across various fields, e.g., workflow automation Zhong et al. (2023). financial reporting Theuma & Shareghi (2024). and travel planning Hao et al. (2024).
Function calls in real-world applications are often diverse and complex, driven by the varied function- alitics of API and the broad range of tasks Qin et al. (2023). APIs often undergo rapid updates to meet diverse user needs, necessitating models capable of robust zero-shot generalization. Addition- ally, users' requirements can be complex or ambiguous, leading to scenarios where multiple tools are employed in parallel dependent manner, require multi-turn interactions. This highlights the importance of managing intricate instructions and accommodating various function-calling scenarios.
Despite these challenges, current tool-augmented LLMs primarily focus on simple funetion-calling tasks with limited diversity and complexity Qu et al. (2024). They mainly rely on existing public APIs for task construction, which restricts their zero-shot capabilities and applicability single-turn queries, neglecting more complex scenarios such as dependent or multi-turn interactions Qin et al.
'qqual Contributions. *Corresponding authors. In this paper, APIs, tools, functions, and plugins used interchangeably.

---

Published as a conference paper at ICLR 2025
Table I: Comparison of ToolACE with other representative tool-augmented LLMs (n/a represents not available.). ToolACE comprehensively incorporates the broadest range of APIs and domains, supports complex nested parameters (Nested), accommodates both parallel (Parallel) and depen- dent (Dependent) function calls, and addresses various types of tool-related data (Multi-type).
Model
#API
#Domain
Nested
Parallel
Dependent
Multi-type
Gorilla Patil al.
202.3
1645
3
x
ToolAlpaca
Tang al.
2023
3938
50
x
TooILLM Om al.
2023
16464
49
X
Functionary Meetkai
2024
n/a
n/a
X
XLAM Liu ct al.
20244
3673
21
X
V
X
X
Granite Abdclaziz et al.
(2024
1/a
n/a
V
X
TooIACE
26507
390
(2023): : Tang et al. (2033) Liu et al. (022).. [ an overview of the data statistics used these representative -ool-augmetted LLMs. Moreover, executions of function calls demand precise API selection and parameter configuration, which are highly dependent on the quality and accuracy of underlying data. As data becomes increasingly diverse and complex, generating accurate samples with simple pipelines introduced by the existing work becomes significantly more challenging.
In this paper, we present ToolACE, systematic tool-learnig pipeline that automatically synthesizes accurate, diverse, and complex function calling data, with the awareness of the model's 's capability.
cenction-callingg scenarios enhances their proficiency and zero-shot tool usage Zhang et al. (2024). Instead of relying on public APIs, ToolACE introduces a Tool Self-Evolution Synthesis (TSS) method. TSS uses a process to generate tools across multiple domains with diverse data types and constraints. Starting with pretraining data to ensure comprehensive coverage, this iterative process of self-evolution and continual updates expands the diversity of the API pool, enabling more sophisticated data generation.
Evolutionary Diversity. Exposing LLMs broad range
Self-Guided Complexity. Instruction-followii function-calling skills. LLMs learn more effectively when the complexity of the data slightly exceeds their current capability Du et al. (2023). To address this, we propose a self-guided dialog generation process (SDG), where the LLM serves an evaluator to regulate complexity. Four types of function- calling data are generated with multi-agent interactions, following a self-guided complication strategy.
data should possess sufficient complexity to foster
Refined Accuracy. Data accuracy is fundamental the effectiveness of tool-augmented L.Mss. ToolACE employs a dual-layer verification (DLV) system, integrating both bale-based and model- based checkers, guarantee the executability and consistency of the synthesized data.
Equipped with data accuracy, complexity, and diversity. ToolACE aims to enhance the function-calling capability of LLMs with strong generalization. Our contributions are outlined follows:
We propose novel automated data pipeline for function calls, ToolACE, which comprises tool self-evolution synthesis module, a self-guided dialog generation module, and dual-layer verification module. To our knowledge, this is the first work to highlight the benefits of synthesizing diverse APIs to improve the generalization of function calls.
We develop a self-guided complication strategy to generate various types of function-calling dialogs with appropriate complexity. The given LLM is utilized as the complexity evaluator to guide the complexity level the generated data. The quality of the generated data ensured through a dual-layer verification process, which combines both rule checkers and model checkers.
We conduct experiments
two widely adopted benchmarks: BFCL Yan al. (2024) and APIBank Li et al. (2023). With only 8B parameters, ToolACE significantly outperforms existing open- source LLMs and is competitive with the latest GPT-4 models.
2 DATA GENERATION PIPELINE
Effective of synthetic data significantly enhances the capabilities of large language models (LLMs) Mitra et al. (2024). Hence, in ToolACE, we propose an automated agentic framework for tool learning generate high-quality, diverse, and complex data, guided by the capability of
2

---

Published as a conference paper at ICLR 2025
Tool Solf-Evolution Symiheaii
Self-Guried Dialog Generalio
Duaf-Loyer Validation
Sath-vvoive Update
Type Checking
Cenerated
Complexity Evaluator
Value Constrainds
APIs
Format
Vertficttion
Sampleed Tool List
Rule Checker
Synthesize
Mult-Agent Generator
API Example
Model Checker
User Agent
Assistant Agent
Tool Agent
Complictioo
db db
Single
Dependent
Pretraining Daca
X
X
Parallel
Non-tool
Figure I: The overall framework of ToolACE, which mainly consists of Tool Self-evolution Synthesis (TSS), Self-Guided Dialog Generation (SDG), and Dual-Layer Validation Process (DLV).
the given LLM to be tuned, illustrated in Figure 1 The proposed framework deploys various agents to recursively synthesize diverse APIs, collaboratively construct dialogs with appropriate complexity, and rigorously reflect on data quality. The following sections present our Tool Self- evolution Synthesis (TSS) module, Self-Guided Dialog Generation (SDG) module, and Dual-Layer Validation Process (DLV).
2.1 TooL SELF-EVOLUTION SYNTHESIS
The variety of APIs significantly underpins the diversity of the function-calling data. As shown in Table [ ToolACE has established a comprehensive API pool that surpasses other representative tool-augmented LLMs in both quantity and domain coverage, incorporating both real and synthesized APIs. Beyond collecting real API data, we developed a Tool Self-Evolution Synthesis (TSS) module that synthesizes API definitions with various data types and constraints, which encompasses three major steps: 1) Speciation, 2) Adaptation, and 3) Evolution.
Speciation. APIs with extensive domain coverage enable tool-augmented LLMs to learn wider array of use cases from various applications and industries, thereby significantly enhancing their generalization ability. In the speciation step, we propose to create hierarchical API context tree to guide the synthesis process with possible API domains and functionalities.
We observe that the pretraining data for LLMs encompasses one of the most diverse sources of human corpus, providing solid foundation for extracting various API domains and use cases. Starting with API-related raw documents from the pretraining data (e.g.. technical manuals, API documentation, product specifications,
guides, and tutorials). we prompt agent powered by frontier LLM to extract an API domain along with all possible API functionalities or use cases from each document. Children nodes of the context are recursively generated at each step, with each node denoting possible API functionality (e.g., get the weather forecast, get the stock price, send an email). Figure g the Appendix A showcases the subtree under the entertainment domain an example.
Adaptation. In the adaption step, we specify the domain and diversity level of each API. We sample subtree and obtain unique functionalities from the API context tree for each individual API, so that different APIs possess distinct functionalities. For example, some APIs may cover more nodes, thereby acquiring more nomain-specing and detailed capabilities. Whereas some APIs may only include single node from the context tree, focusing
easy, straightforward purpose.
The evolution step involves the continuous improvement and adaptation the APT based on outcomes and new requirements. An LLM is instructed to synthesize new APIs according to sampled subtree of the API context tree and an API example. The generated definitions of new APIs required to be clear and thorough. We then apply a set of diversity indicators, e.g., adding new functionalities or parameters, including additional constraints, mutating parameter type, and updating returned results, to diversify the generated APIs. We maintain an API example buffer containing various API examples. Iteratively, we sample an example from the buffer, adapt it to the current subtree of functionalities, and generate the next generation of the APIs.
Evolution.
3

---

Published as a conference paper at ICLR 2025
The proposed TSS module facilitates the efficient generation of diverse set of API documentation, with nested types including lists of lists or lists of dictionaries.
2.2 SELF-GUIDED DIALOG GENERATION
The effectiveness of function-calling data is closely tied to the capabilities of the LLM. For different LLMs, the knowledge and abilities they have learned during the pretraining phase are different, thereby the function-calling calling data they require should also differ Du al. (2023). For instance. an LLM with 0.5B parameters may struggle to comprehend extremely complex data with long dependencies between APIs. contrast, a well-trained 70B L.LM can easily handle straightforward queries with clear intentions and simple APIs. In both cases, the data is unproductive for the given LLM, highlighting the importance of tailoring data generation to align with the model's capabilities.
Hence, to ensure the generated dialogs indeed fill the ability gap for the given LLM, we propose self-guided dialog generation (SDG) module to synthesize the function-calling dialogs, as shown in the middle part of Figure []] SDG consists of a complexity evaluator and multi-agent generator. Various types of cunction-callingg dialogs are generated via the interaction of multiple agents. The LLM to be tuned serves as the evaluator, assessing the complexity of the generated data. Data that is deemed too simple or too complex is dynamically adjusted under the guidance of the evaluator.
2.2.1 MULTI-AGEN DIALOG GENERATION
We propose a multi-agent framework to generate the four types of dunction-calling dialogs: single function calls, parallel function calls, dependent function calls, and non-tool-use dialogs.
The data generator includes three agents user, assistant, and tool-eache each simulated by an LLM. One API candidates sampled from our curated API pool and present the sampled APIs to the agents. Dialogs are then generated through role-playing among the three agents, each agent is provided with necessary role assignment and detailed task description continue the conversation. The user agent mainly makes requests or provides additional information to the assistant, with self-guided complication process to adjust the dialog complexity. The assistant agent addresses the user's queries equipped with the given APIs. The action space of the assistant agent includes: calling the APIs, requesting further information, summarizing the tool feedback, and providing non-tool-use answers. To ensure data quality, each assistant action is generated multiple times, and only responses with consistent decisions across multiple instances are adopted. specialized and structured thinking process specifically designed for function calls is also applied to enhance the assistant's 's tool-calling decisions. The tool agent acts as the API executor, processing tool descriptions and input parameters provided by the assistant, and outputs the potential exccution results.
For each function-calling dialog, the user agent initiates request related to the given sampled APIs. The assistant agent reviews the request and decides whether to call API ask for additional information. tool calls are required, the tool agent will provide simulated results, and the assistant agent will summarize the results and present the user. The generation process continues with the user agent querying again or responding to the assistant's question until the target turn length is reached.
2.2.2 DATA COMPLEXITY EVALUATION
Different LLMs exhibit varying knowledge and capabilities, which necessitates the use of different data to optimize tool usage performance. However, much of the existing research overlooks the correlation between the model capability and the training data, leading to suboptimal data efficiency.
In this work, employ the LLM be tuned, denoted M, as the evaluator, and use the loss on a data sample of (a. y) pairs for M to assess data complexity, denoted as HM(x,y) y). The data complexity is measured
HM(I, v)
log (t..,t))
Thy
where is the input query, and [t,,...................... tn is the response with ny tokens. Here, t, denotes the i-th token for i 1,..., ny. and represents the probability of predicting the next token. A higher loss implies that the data sample (Ir, y) has been found harder to learn for the model M.
4

---

Published as a conference paper at ICLR 2025
Loss
Loss
Loss
2
8
2
5
0.1 0.2 0.3 0.4 0.5 0.6 0.9 Dissimilarity
Availabel taols
Used tools
(a) Number of candidate APIs
(b) Number of utilized APIs
(c) Dissimilarity
Relationships between loss and (1) the number of candidate APIs available for selection, (2) the number of APIs utilized, and (3) the dissimilarity between the user query and the API descriptions.
Figure 2:
Our findings suggest that the loss of data sample is generally positively correlated with (1) the number of candidate APIs available for selection, (2) the number of APIs utilized, and (3) the dissimilarity between the user query and the API descriptions, as demonstrated in Figure Intuitively, as the number of candidate APIs increases, selecting the correct one becomes more difficult. Similarly, the of higher number of APIS reflects greater query complexity, while larger discrepancies between the user query and the API descriptions demand more sophisticated reasoning to identify the correct function. These validate the use of loss measure of data complexity in function calling.
To establish an appropriate complexity range for the given LLM M, we create a small, prior data set that spans various levels of complexity. A data sample that is correctly generated by M indicates that the model has already mastered the corresponding tool usage case, and thus this sample is unnecessary for further fine-tune conittning.. The associated loss serves as a reference lower bound for data complexity. Conversely, if the loss of data sample remains high after fine-tuning, it may indicate that the sample too complex for the model learn and this loss serves reference upper bound.
Our evaluator provides the suitable complexity range, along with the loss of the given data sample, as the guidance information the multi aggnt generator in generating the training data.
2.2.3 SELF-GUIDED COMPLICATION
After obtaining the complexity the current data from the evaluator, the user agent's instructions dynamically adjusted. If the data sample is too simple for the LLM, the user agent is instructed to generate a more complex query-ona that either requires additional APIs diverges further from the API description to increase complexity. Conversely, if the data sample exceeds the LLM's 's capacity, the user agent is prompted produce a simpler query. In this way, the data generation process continually adapted to better match the model's performance level.
2.3 DUAL-LAYER DATA VERIFICATION
critical factor inftuencing the function callinn capability of LLMs the accuracy and reliability of the training data. Data that inconsistent inaccurate can hinder the model's ability interpret and execute functions Liu et al. (2024). Unlike general question-answerine data, where verifying correctness can be challenging, function-calling data is function call must strictly match the format specified in the API definition. Building on this insight, we propose an automatic dual-layer verification system (DLV) verify our synthesized data, as shown in the right part of Figure [] which consists of a rule verification layer, and a model verification layer, where these results are all overseen by human experts.
verifiable. This is because successful
Rule Verification Layer. The rule verification layer deploys rule checker to ensure that the data strictly adheres to the predefined syntactic and structural requirements the API, covering four key aspects: API definition clarity, function calling executability, dialog correctness, and data sample consistency, guided by a meticulously curated set of rules, as listed in Appendix B
For instance, to verify function calling executability, implement the following procedures: First, we confirm that the API name matches one from the given tool list. Next, we verify that all required parameters accurately provided. Finally, regular expressions ensure that the parameter
5

---

Published as a conference paper at ICLR 2025
formats and patterns adhere to those specified in the API documentation. These procedures allow us to validate the correctness and executability of function calls without the need for actual execution, which enhances efficiency and reduces deployment overhead.
Model Verification Layer. The model verification layer further incorporates LLMs to filter out erroneous data that cannot be detected by the rule checker, with primary focus on content quality. However, we find that presenting data sample directly to the LLM for correctness evaluation is too complex, often resulting in unsatisfactory outcomes. To address this, we decompose the model verification task into several sub-queries that mainly cover three key aspects:
Hallucination Detection: Identifies whether the values of input parameters in function calls are labricated- not mentioned either the user query or the system prompt.
Consistency Validation: Verifies that the responses can effectively complete the user's task and ensures the dialogue content adheres to the constraints and instructions in the user query and system prompt.
Tool Response Check: Ensures that the simulated tool responses align with the API definition.
Each aspect is evaluated by an individual expert agent, powered by an LLM. We also incorporate other verification prompts to eliminate repetitive responses and meaningless tokens in the data.
EXPERIMENT
3.1 EXPERIMENT SETUP
To validate the efféctiieneess of our approach, have conducted extensive experiments by training LLMs with the generated data. We train the open-source LLM, 1-8-AA...-B--Isteee AI@ Meta (2024), in the supervised fine-tuning (SFT) manner, for most of the experiments. We refer to the model as ToolACE-8B. We also validate our data with other backbone LLMs like Qwen-series Bai (023). Due the limited resources, we adopt the parameter-officient training strategy LoRA Hu et al. 2022 to fine-tune the model. As for the hyper-parameters setting, we adopt one of the most common settings, which sets the rank as 16 and alpha as 32 for all modules. We compare the overall performance with the state-of-the-att API-based and open-souree models, like GPT-series fine-tuned function calling models including Corilla-OpenFunctionns v2 Patil et al. (2023) and xI.AM- series Liu et al. (2024). Experiments are conducted on two representative benchmarks, including BFCL Yan et al. (2024) and API-Bank Li al. (2023). The two benehmarks are comprehensive and executable function call evaluations specifically designed to assess the ability of LLMs to invoke functions. We then conduct in-depth ablation study to reveal the effectiveness of accuracy, diversity, and complexity. More experimental settings including benchmark details, evaluation metrics, and training settings are shown in Appendix
2 as well as
3.2 OVERALL PERFORMANCE ANALYSIS
To assess the effectiveness of our ToolACE-88 -B model regarding its functional calling capabilities, we compare ACE-A model with various representative models. The results are summarized in Table andd Table 3 respectively.
The findings in BFCL indicate that API-based models demonstrate significant advantages over open- source models, such as the Claude series and the GPT-4 series. Open-source models fine-tuned for function calling, such as Funetionary and xLAM, exhibit competitive performance, but still fall short of the leading models. Our TooIACE-SB model outperforms most API-based and open-source models in both the AST and Exec categories of BFCL, and continues to exhibit substantial advantages over all the open-soune source models in the context of API-Bank, demonstrating the effectiveness of our training data for functional calling. This is mainly attributed our accurate, diverse, and complex synthesized data, which enhances the zero-shot function calling capability of the L.M. Additionally, ToolACE
fttt:s: pretts::/chhttpst ccatttpt...tptppp/ com
Thhe overall performance evaluated on the latest BFCI.v3 and subsequent studies evaluated on only non-live categories since there testing samples in these categories, showing more robust results.

---

Published as a conference paper at ICLR 2025
Table Accuracy performance comparison on BFCL-v3 leaderboard (updated on 09/20/2024). The top 20 models are listed for comparison. FC denotes the model is tailored for functional calling. (A) and (E) present AST and executable category, respectively. Rel and Irrel are abbreviations for relevance and irrelevance.
Single
Multi
Hallucination
Rank
Overall
Model
Non-live (A)
Non-live (E)
Live (A)
Multi turn
Rel
Irrel
59.49
GFT-4-uuto-2224-0000000
82.65
83.80
73.39
21.62
70.73
79.79
59.29
GPT-40-2024-88-06
85.52
82.96
71.79
21.25
63.41
82.91
3
59.22
T6olACE-8B (FC)
89.27
90.07
73.21
14.37
85.37
83.81
59.13
ALAM-Bx22br(FF))
89.75
89.32
72.81
15.62
97.56
75.23
58.45
GPT-44o-aim-202440--888 (FC)
82.83
81.80
67.53
25.75
82.93
71.83
57.94
ALAM-8x7b+rFC)
88.44
85.89
71.97
15.75
92.68
72.35
7
57.21
GPTT-4o-minl-22044001188 (Prompt)
86.54
87.95
72.77
11.62
80.49
79.20
55.82
mistral-large-4407 (FC)
84.12
83.09
67.17
20.50
78.05
48.93
55.67
GPT 2024 (Prompt)
10
54.83
Claude-3.5 1------oot--222-ae (FC)
70.35
66.34
71.39
23.50
63.41
75.91
11
53.66
GPT 4o 2024
201T-40-002--000 08 06 (Prompi))
80.90
77.89
73.88
6.12
$3.66
89.56
12
53.43
GPT--41 (71-441-mmni-200 mini-2022--9-991---1 19-12 (Prompt)
75.48
76.86
71.17
11.00
46.34
88.07
53.01
Gemini-
(FC)
14
52.53
Gemini- Pro- Preview. 0514 (PC)
75.54
77.46
69.26
10.87
60.98
80.56
15
51.93
-.5.- T...--uu---02555111 (FCC
84,52
81.66
59.00
19.12
97.56
35.83
16
51.78
FireFunction (FC)
85.71
84.23
61.71
11.62
87.80
$2.94
Open Mistral-Nemoo-2222 Nemo-2407 (FC)
14.25
18
51.45
xLAM 7b-fc-r fc-r (FC)
86.83
85.02
68.81
0.00
80.49
79.76
19
51.01
Gorilla- Porila-OpenFPnnciions--22 (FC)
87.29
84.96
68.59
0.00
85.37
73.13
20
49.63
Claude 3 Opus 20240229 (FC)
58.40
63.16
70.50
15.62
73.17
76.40
Meta-Llama.3-705 (Prompt)
87.41
50.63
Table Accuracy performance comparison on API-Bank evaluation system. Bold values represent the highest performance for API-based and open-source models, respectively.
Model
Call
Retrieval+Call
gpt-3.5-turoo-0125
70.43
52.59
gpt-4-0613
75.94
48.89
API-based
9m-44-urhha22244-44-99
72.43
39.26
got-4o-mini-2024-07-BB
74.69
45,93
gpt-4o-2024-05-13
76.19
42.96
Alpaca-7B
24.06
5.19
ChatGLM-6B
23.62
13.33
Open-source
Lynx-7B
49.87
30.37
XLAM-7b-fe-r
32.83
21.48
LLaMA-3.1-8B-Insiie
71.18
37.04
ToolACE-8B
75.94
47.41
excels mitigating hallucination, achieving impressive relevance and irrelevance scores of 85.37% and 83.81%, respectively. These results highlight its ability in maintaining an excellent balance between the two categories, unlike other models that either suffer from significant imbalance or underperform in both categories. ToolACE-8B also consistently and significantly outperforms xLAM- 7b-fc-r, which is also fine-tuned for function calling with similar size, all categories, providing compelling evidence of its superiority. Furthermore, our ToolACE-81 model shows consistent advantageous performance on API-Bank compared with all open-source models, demonstrating comparable performance with GPT--4-series models.
7

---

Published as a conference paper at ICLR 2025
W.O. dual
w.o. model
Final
Score
92.5
90.82
90.56
91.90
91.18
91.81
90.00
89.17
89.59
91.41
87.5
85.0
82.92
82.5
AST Figure 3: Ablation study of the dual-layer verification(DLL).
Exec
Irrelevance
Overall
85.0
92.5
90.0
91.13
1ic 90.00
Easy
30.42
58.75
Mod km
Hard 80.47
89.65
87.5
58.34 86. 20.8
Low
Medium
89.18 88. as 18.41
High
87.5
06.25
85.0
85.0
62.5
02.5
77.5
75,0
75.0
use
Irrelevance
Overall
Tool
Ireeevance
Overall
Figure 4: Ablation study complexity.
Figure 5: Ablation study on diversity.
3.3
ABLATION STUDY
3.3.1 ABLATION ON ACCURACY
Effects of the verification system. As detailed previous sections, our verification system comprises Iwo layers: rule checker and model checker. To evaluate the efficacy of each layer, train LLLMA33118B-IInSn.
with LoRA using three distinct datasets: (1) data without any verification (denoted as w.o. dual), (2) data without model checking (denoted as w.o. model), and (3) data subjected to dual-layer verification (denoted as Final). It is important to note that datasets with more verification layers contain smaller amounts of data, as some data is filtered out during the verification process. The resulting fine-tuned models are assessed using the BFCL benchmark, with outcomes summarized Figure Comparative analysis reveals that the model trained on data without model checking surpasses that trained on unverified data in terms of both executable and overall accuracy, thereby validating the rule checker's effectiveness. Moreover, the model trained on dually verified data significantly outperforms both ablation models in terms of AST and overall accuracy, underscoring the indispensable role of the model checker.
3.3.2 ABLATION ON COMPLEXITY
Data Sampling for Various Complexity. To effectively assess the impact of dataset complexity on the model's performance, have conducted sampling of the entire dataset based on the aforementioned complexity assessment metrics. We compute and sort the complexity for each data sample using Eq. (T), and select the bottom, middle, and top 60,000 instancess as ToolACE ToolACEmadione ToolACEnard: respectively, yielding three distinct subsets of varying complexity levels
The rationale behind this stratified sampling approach is create controlled environment where the influence of complexity can be systematically analyzed. By maintaining equal sample sizes across subsets, we ensure fair comparison while varying the complexity, which allows for a more nuanced understanding of how complexity affects model performance.
Effects of Complexity. We conduct experiments by training subsets with varying complexity and evaluate the fine-tuned models on the BFCL benchmark. The results are illustrated in Figure The model trained on ToolACEmeduum shows slight superiority compared with another two subsets, for both overall and tool-use accuracy. This finding aligns with our hypothesis that optimal data complexity essential for LLM training, as data that is either too simple overly complex prevent the model from reaching its full performance potential.
with those three
8

---

Published as a conference paper at ICLR 2025
100
100
100
85.89
84,58
52.94
85,24
80
70.80
Aosumicy (8)
60
80
73,00
Aosurayy
60.98.6
40
Raw
Fino-lnnod
Accunccy
60
41.58 35
48,12
26,24
20
20
20
1314,12
17,10
0
0.50
0.00
1.88
4.9
4B
Fine
7B
0.50
0.00
1,88
40
76
0.5B
1.88
4B
78
Model Size
Model Size
Model Size
(a) AST Accuracy
(h) Exec Accuracy
(c) Overall Accuracy
Figure 6: Scaling performance of model size. The backbone I.L.s are Qwen-1.5-xl -B-Chat series because this series offers models ranging from 0.5B to several billion parameters, enabling compre- hensive analysis the relationship between model scale and performance.
3.3.3 ABLATION ON DIVERSITY
Data Sampling for Various Diversity. To assess the impacts of the diversity, we generate three subsets with varying diversity, namely ToolACEaows ToolACEmedium and ToolACEnigh- Initially, all APIs are clustered into 30 groups based on the API context tree. Subsequently, three API subsets are constructed by selecting APIs from 6, 14, and 30 clusters, respectively. Instances are then categorized into three subsets according to their associated APIs. Approximately 30,000 instances are randomly selected from each subset, resulting three training sets with distinct levels of diversity.
Effects of Diversity. Experiments are conducted to train 1..aMA33.1-880 1-8---.-----s-tuce on three subsets described above. The results on BFCL are reported Figure [5] A positive correlation between training data diversity and overall model accuracy is observed, emphasizing the critical role of API diversity model performance. Notably, improvements in relevance detection are particularly pronounced, suggesting that exposure to wider range of APIs enhances the model's ability to discriminate between subtle API differences, thereby enhancing the ability of irrelevance detection.
3.4 SCALING PERFORMANCE OF MODEL SIZE
Scaling laws posit correlation between model size and performance. To investigate the scalability of functional calling capabilities, conduct experiments using the Qwen-1.5-xB Bhhttttt series, which includes a range of model sizes (0.5B, 1,8B, 4B, 7B, etc.). Both raw and fine-tuned (using dataset) models are evaluated the BFCL benchmark, with results presented in Figure 6 As expected, larger models exhibit superior performance in functional calling, as evidenced by improvements in both AST and Executable accuracy. While smaller raw models (0.5B and 1.8B) showed minimal function-calling ability, struggling to generate structured outputs, fine-tuning on the ToolACE dataset significantly enhanced their capabilities. The fine-tuned models exhibit consistent scaling performance, highlighting the potential of ToolACE to boost the performance of larger LLMs.
3.5 STUDY ON VARIOUS BACKBONE LLMs
To investigate the influence of the LLM backbone, we experiment with several (approximately) 8B-scale models: -wwenl.5-B-CCL Bai et al.) (2023). LLaMA-3-8B-Instrice and LLaMA-3.1-88 Instruct. Fine-tuned models are evaluated on the BFCL. benchmark, with results presented in Figure 7 Across all models, fine-tuning yields substantial performance gains, highlighting the effectiveness of our ToolACE. Due to differences in pre-training corpora, such as Qwen trained with more Chinese conversational samples, raw models exhibit varying functional calling capabilities, with LLaMA-3.1-8BB ---.--8---nnttuue demonstrating superior performance. While this hierarchy persisted after fine-tuning, the performance gaps narrowed, suggesting that our dataset can potentially enhance the functional -aattionalcaaling abilities of those LLMs tailored for other skills, such as conversational skills.
3.6 STUDY ON GENERAL CAPABILITIES
To assess the impact of ToolACE training on broader capabilities of LLMs, conduct experiments across multiple benchmarks evaluating general ability, including MMLU Hendrycks et al. (2021a]b). HumanEyal Chen al. (2021) (coding), GSM8K Cobbe al. (2021) (mathematics), Common-

---

Published as a conference paper at ICLR 2025
BFC
100
Raw
ToolACE
86.45 07.47
Accuracy (%%
80
75.06
60
CSQA
MMLU
48.12
02
40
Qwen1.5-77
LLaMA-3-88
LLaMA-3.1-8
HumanBval
Figure 7: Performance various LLMs.
Figure 8: General capabilities.
SenseQA Talmor et al. (2019) (reasoning), and BFCL al. (2024) (functional calling). Raw LLaMA-3- 1----3----nnstuue LLaMA-3.. -.--A----BB-Insttuee functionally specialized XLAM-7B-fo-r, and GPT-4 serve as baselines. Results are presented Figure X ToolACE-8B substantially improves over XLAM-7B--c-- across most benchmarks, with particularly pronounced gains in MMLU, GSM8K, and CommonSenseQA.. Compared to GPT-4, ToolACE-SB shows clear limitations in reasoning and understanding, This is primarily due to the scale of the model and its training corpus. Compared to the raw LLaMA-3.1 1-----3..------sse 88-Instruc, ToolACE-8B demonstrates negligible performance degradation on some benchmarks while achieving significant enhancements in functional calling. These findings suggest that the ToolACE dataset effectively enhances functional calling capabilities without compro- mising the underlying LLM's general abilities. This success highlights the potential of specialized models in one specific domain, the challenge of simultaneously enhancing multiple capabilities, alongside functional-callnng performance, remains an open question. The detailed analysis of the limitations can be referred to in Appendix
4 RELATED WORK
Tool Learning. Integrating external tools allows LLMs to expand the boundaries of their capabilities, enabling more specialized, precise, and dependable problem-solving (Qin et al., 2023). Methods for equipping LLLMs with tool-use capabilities generally fall into two types: tuning-free approaches and tool-augmented tuning. Tuning-free methods let LLMs tools by providing in-context tool descriptions and examples, requiring no additional training Mialon et al. (2023): : Hsich et al. (2023); Ruan al. (2023). A well-known technique ReAct Yao al. (2023). which enables LLMs to alternate between reasoning and actions to solve complex tasks. However, as these approaches depend heavily on the model's initial abilities, tugl-nngeedtedd tuning has gained more attention for directly improving tool use Qin et al. (2023); SShick et al. (2023); Patil et al. (223); Tang et al. (2023): Liu al. (2024); Abdelaziz et (2024). Many these methods rely on existing APIs but lack robust systems for generating and validating data. Our ToolACE overcomes this limitation by implementing a well-designed pipeline that ensures greater diversity, complexity, and accuracy.
Data Synthesis. As LLMs grow more advanced, relying solely existing human-generated data becomes insufficient for further progress Bauer et al. (2024). key strategy involves modifying or augmenting datasets using specialized prompting techniques Wang al. (2023): Xu et al. (2023); Yu et al. (2023). Given the scarcity of tool-use datasets, Basu et al. (2024) repurpose data from other domains for tool-use applications, while others QQn al. (2023); Tang et al. (2023): Liu et al. (2024) depend on publicly available APIs, often producing single-turn instructions with basic tool interactions. ToolACE offers more comprehensive approach, incorporating both tool synthesis and dialogue generation, along with a verification module to ensure data quality.
5 CONCLUSION
This paper presents ToolACE, an automated data generation pipeline developed to enhance the function-calling capabilities of large language models. ToolACE employs novel self-evolution synthesis process and self-guided data generation method to curate accurate, complex. and diverse synthetic APIs and dialogs. Our results demonstrate that even smaller models trained with ToolACE can achieve state-of of-the-art performance, thereby advancing the field and setting new benchmarks for tool-augmented AI agents.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Ibrahim Abdelaziz, Kinjal Basu, Mayank Agarwal, Sadhana Kumaravel, Matthew Stallone, Rameswar Panda, Yara Rizk, GP Bhargav, Maxwell Crouse, Chulaka Gunasekara, et al. Granite-functionn calling model: Introducing function calling abilities via multi-task learning of granular tasks. arXiv preprint arXiv:2407.00122, 2024.
AI@ Meta. Llama model card. 2024, URLhttps:///gthbbbccom wwwtenttaalllaaaa 1lama/aaa////aa/ Llama3/
CARD md
Jinze Bai, Shuai Bai, Yunfci Chu, Zeyu Cui, Kai Dang. Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309.1660 2023.
Kinjal Basu, Ibrahim Abdelaziz. Subhajit Chaudhury, Soham Dan, Maxwell Crouse, Asim Munawar, Sadhana Kumaravel, Vinod Muthusamy, Pavan Kapanipathi, and Luis A Lastras. Api-blend: A comprehensive corpora for training and benchmarking api llms. arXiv preprint arXiv:2402.15491, 2024.
André Bauer, Simon Trapp. Michael Stenger, Robert Leppich, Samuel Kounev, Mark Leznik, Kyle Chard, and Ian Foster. Comprehensive exploration of synthetic data generation: A survey, arXiv preprint arXiv:2401.0255 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkín, Brooke Chan, Scott Gray. Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. 2021.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
Qianlong Du, Chengqing Zong, and Jiajun Zhang. Mods: Model-oriented data selection for instruction tuning. arXiv preprint arXiv:2311.15655 2023.
Yilun Hao, Yongchao Chen, Yang Zhang, and Chuchu Fan. Large language models can plan your travels rigorously with formal verification tools. arXiv preprint arXiv:2404. 11991, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021b.
Cheng- Yu Hsich, Si-An Chen, Chun-Liang Li, Yasuhisa Fujii, Alexander Ratner, Chen- Yu Lee, Ranjay Krishna, and Tomas Pfister. Tool documentation enables zero-shot tool-usage with large language models, 2023.
11

---

Published as a conference paper at ICLR 2025
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. International Conference Learning Representations, 2022. URL
iddnZeVKeeFYff
Shijue Huang, Wanjun Zhong, Jianqiao Lu, Qi Zhu, Jiahui Gao, Weiwen Liu, Yutai Hou, Xingshan Zeng, Yasheng Wang, Lifeng Shang, et al. Planning, creation, usage: Benchmarking Ilms for comprehensive tool utilization in real-world complex scenarios. arXiv preprint urXiv:2401.17167, 2024.
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for cool-augmented llms. arXiv preprint 3xXiv:2304.0844: 2023.
Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, et al. Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. arXiv preprint arXiv:2406./885/8 2024.
Meetkai. Aunctionary.meetkal 2024.
www.yaeee........... com
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yi Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. Augmented language models: survey, 2023.
Arindam Mitra, Luciano Del Corro, Guoqing Zheng, Shweti Mahajan, Dany Rouhana, Andres Codas, Yadong Lu, Wei-ge Chen, Olga Vrousgos, Corby Rosset, et al. Agentinstruct: Toward generative teaching with agentic flows. arXiv preprint arXiv:2407.03350. 2024.
Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint M.Xiv:2305.15332 2023.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, al. Toolllm: Facilitating large language models master 16000+ real-world apis. arXiv preprint arXiv:2307. 16789, 2023.
Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji- Rong Wen. Tool learning with large language models: A survey, arXiv preprint arXiv:2405.179:: 2024.
Jingqing Ruan, Yihong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, Guoqing Du, Shiwei Shi, Hangyu Mao, Ziyue Li, Xingyu Zeng, and Rui Zhao. Tptu: Large language model-based ai agents for task planning and tool usage, 2023.
Timo Schick, Jane Dwivedi-Yis Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA:: A question answering challenge targeting commonsense knowledge. Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp. 4149 4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/N19. 1421. URL wwt.aaaaannnnoooooooooooo www.aatwwwoooo.oooooooooe1.111-. wwwp/ooggggioo
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. Toolal- paca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXivv2306.05301, 2023.
Adrian Theuma and Ehsan Shareghi. Equipping language models with tool use capability for tabular data analysis in finance. arXiv preprint arXiv:2401. / 2024.
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannanch Hajishirzi. belf-instruutt Aligning language models with self generreee instructions. In The sl Annual Meeting The Association For Computational Linguistics, 2023.
12

---

Published as a conference paper at ICLR 2025
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.1222 2023.
Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Berkeley function calling leaderboard. httpps://orilla. cs berkeley.edu/blooe. 2024.
berkeley ciaction_callian leaderboard.htm.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting language models. The Eleventh International Conference Learning Representations, 2023.
Longhui Yu, Weisen Jiang, Han Shi, YU Jincheng, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference Learning Representations, 2023.
Dylan Zhang, Justin Wang, and Francois Charton. Instruction diversity drives generalization to unseen tasks. arXiv preprint arXiv:2402. 10891, 2024.
Ruizhe Zhong. Xingbo Du, Shixiong Kai, Zhentao Tang, Siyuan Xu, Hui-Ling Zhen, Jianye Hao, Qiang Xu, Mingxuan Yuan, and Junchi Yan. Llm4eda: Emerging progress in large language models for electronic design automation, arXiv preprint arXiv:2401.1222 2023.
13

---

Published as a conference paper at ICLR 2025
A AN EXAMPLE SUBTREE OF THE API CONTEXT TREE FOR THE Entertainment DOMAIN.
O
API Domain
Entertainment
Education
Music
Anime
Books
Music Streaming
Live Music (enhance the experience live music events)
(user-specific
music streaming
service)
Figure A subtree the constructed API context tree for the Entertainment domain.
B RULE EXAMPLES IN RULE VERIFICATION LAYER
Table/ooutiins the check rules we use, which consists of four aspects: API definition clarity, function calling executability, dialog correctness, and data sample consistency.
Table 4: Example rules for the ToolACE rule checker.
Aspect
Rules
Check if the API definition complies with JSON Schema specifications.
API Definition Clarity
Check if the API dcfinition contains all necessary fields.
Check if the API name is in the tool list.
Function Calling Exccutability
Check if all required parameters are provided.
Check if all the parameter formats and patterns match the API definition.
Check if the dialog contain all necessary fields.
Check if the assistant's 's response is too long.
Dialog Correctness
Check for invalid characters in the responses.
Check for mixed-language responses.
Check if response complete.
Check if the API names the function call and tool response are consistent. Check for format conflicts with the requirements defined the system prompt. Check if the order of the dialogue roles is correct.
Data Sample Consistency
Check if the tool response follows the function call.
C EXPERIMENTAL DETAILS
C. BENCHMARKS
BFCL. Berkeley Function-Callin Benchmark (BFCL) Yan al. (2024) is comprehensive evaluation framework for assessing the function-calling capabilities of LLMs across various languages, application domains, and complex cases. BFCL covers tasks including multiple function calls, parallel function calls, multi-turn function calls, and multi-step function calls. BFCL contains 4,951 test cases: 3,951 single-turn cases and 1,000 multi-turn cases, focusing dynamic, real-world scenarios.
BFCL evaluates multiple function calling tasks using the following metrics:
14

---

Published as a conference paper at ICLR 2025
Abstract Syntax Tree (AST) Evaluation: AST evaluation compares the abstract syntax tree of the function output to the ground truth and the function definition. captures the correctness of matching the functions, required parameters, parameter types, and values.
Executable Function Evaluation: Executable function evaluation assesses the accuracy of the generated API call by executing it and comparing the output with the ground-truth output.
Irrelevance: Irrelevance measures the model's ability to refrain from making function calls given irrelevant user queries. The irrelevance score calculated as the number of correct non-function-call predictions divided by the total number of test samples.
Relevance: Relevance evaluates the model's ability to output function calls relevant to the user query. In this category, the correctness of the parameter values is not considered. The relevance score is calculated as the number of correct function-call predictions divided by the total number of test samples.
Overall Accuracy: Overall accuracy the unweighted average of the accuracies across all sub-categories.
2023) consists of 314 tool-use dialogues with 753 API calls to assess LLMs' capabilities planning, retrieving, and calling APIs, with 363 single calls and 122 multiple calls. API-Bank assesses LLM performance across three capabilities:
API-Bank. API-Bank Li et al.
Call: The ability to call an API based on a given query when the APIs are known.
Retrieval+Cor Call: The ability to retrieve and call a single API when the APIs are unknown.
Plan+ Retrnn--erieellCaall The ability to continuously plan, retrieve, and call multiple APIs when the APIs unknown.
The evaluation metric for API-Bank is accuracy, calculated as the number of correct predictions divided by the total number of predictions.
C.2 HYPER-PARAMETERS
The hyper-parameters of the training process are illustrated Table
Table 5:
in experiments for training.
Learning Rate
WarmUp Ratio
LR
Batch Size
LoRA
LoRA alpha
Epochs
Scheduler
rank
10-1
cosine
48
16
D CASE STUDY
Here we present a selection of cases from our generated data, showcasing various examples of tool utilization and function calls.
Figure 10 presents a data sample of parallel function calling. This type of data focuses on building the model 's ability understand and extract key parameters from the user query. which makes models learn to call the correct function repeatedly for accomplishing task. In this example, the query indicates that the user needs to get the event information of Theatre, Dance, and Musie during 2021-04-01 and 2021-05-01 respectively. The assistant correctly recognizes that it needs to call performanceArt..gee wet.rrrrnneettuumiiinggggg wwt_uupcoongeevveeee eeennts for three times with different assignments of the parameter "category".
Figure II shows a data sample of multiple function calling. This kind of data focuses on giving the model the ability to distinguish between different functions. which makes models learn to choose the correct function(s) for finishing a task. There are five functions in the tool list in the case. The user needs the information of the live event count for soccer, NBA stats for the 2025 season, and the latest
15

---

Published as a conference paper at ICLR 2025
Case 1: parallel function calling:
ToolList
pnrformanceArttgeetuuommmi eeentt required: ["start_date date", end__ate"] optional: ["cat- e]ory"]
User
I would like to know about upcoming Theatre, Dance, and Music performance art events
happening between 2021-04-01 and 2021-05-01.
Assistant
ser_tuuunnneeeetessstrrrt (aattssstar_dtt=222222 end dat_ddde=e- modd_dtt==2021105-- "2021-05-01" category="Theatre")
perfornanceArtleee_ppommmm crnnominn_ceetttsstt__dddde
end "2021-05-01"
category="Danees
performanceArt..et uppoming._events(ttar events(start date="20211044-01 date=""02] "2021-0-00---"""
Calegory="Music""
Figure 10: A case on parallel function calling.
Case multiple function calling:
ToolList
Bet365 Inplay Filter required: optional: ["sport_id") [" ["skip_espoore esports"] Get Live Events Count by Sport required: ["sport", "locale", "timezone"]
National Team Statistics required: ["player_id"
Get All NBA Stats required: ["scasons"] optional: ["page", "per_page", "player_ids", "game ids"]
Get Latest Team Media required: ["team id"]
User
I plan to watch some sports events and want to catch up the latest NBA stats. Could you find the live event count for soccer, get NBA stats for the 2025 season, and the latest media for the Lakers?
Assistant
Get Live Events Count by Sport(sport= "saccer,, locale="en_LR timezone=-4.0),
Get All NBA Stats(seasons=("2233 (0025" page=1, per_page=10), Get Latest Team Media(team id=11.ddd
Figure 11: A case on multiple function calling.
media for the Lakers. The assistant chooses the corresponding Get Live Events Count by Sport, Get All NBA Stats, and Get Latest Team Media from the tool list and calls them to finish the task.
Figure 12 presents cases that focus enhancing the model's ability determine the relevance of a function to a user's query. In the first scenario, the user requests movie showtimes at specific cinemas. However, the available functions are designed only for retrieving event details or specific book characters, rendering them irrelevant to the user's 's request. The assistant appropriately declines to provide an answer, citing the lack of relevant functionality. to start an Android emulator, but the essential parameters- device name, system image, and API level-are missing. The assistant correctly identifies this gap and requests the necessary information rather than attempting an incomplete operation. This type of data is vital for training the model to recognize when a function not relevant or when it lacks the required inputs, ensuring that the system responds accurately and avoids executing unsupported tasks.
the second scenario, the user asks
E ADDITIONAL EXPERIMENTS
E.1 COMPARISON WITH OTHER TRAINING DATA
To further highlight the value of our ToolACE dataset, conducted additional experiments on two representative datasets (TooILLM and xLAM), as summarized in Table Specifically, we trained models using the amount of data (25,000 samples) and the same base model 1------..---B--A --nn-------s--nnttructt)) to ensure a fair comparison. The trained models were then evaluated on the BFCL benchmark. The results show that the model trained with our dataset consistently outperforms the others across all
16

---

Published as a conference paper at ICLR 2025
Case 3: relevance
ToolList
Event optional: I"name", "club", "club_ "string filter"]
"club_name", name", "start_timme"
"time_ dime_modifiect
Get Character by ID required: ["id"]
User
Can you check the showtimes for the film 45?
"Avengers: Endgame" Cinema 21 and Cinema
Assistant
The provided functions do not support searching for showtimes at specific cinemas. The functions available are limited to retrieving events based on filters retrieving a spe- cific character from the Throne of Glass series. Therefore, the question lacks the parameters required by the given functions.
ToolList
runAndroodEEmulato required: [device name, system image, level]
User
Can you start the Android emulator for me?
Assistant
start the Android emulator, lacks required parameters: device name, system image, and api level. Please provide these details to proceed.
The function runAndroodEmmlltor
Figure 12: A case on irrelevance detection.
categories, further validating the effectiveness of our approach. Notably, the model trained on the xLAM dataset exhibits relatively poor performance in irrelevance detection, likely due lack of diverse sample types, such as cases where provided tools cannot solve the task. Moreover, the TooILLM dataset, which primarily focuses on multi-step and dependent cases, demonstrates weak generalization on the BFCL benchmark.
Table 6: Performances of training with different training datasets. The models are evaluated on the BFCL benchmark.
Training data
Overall
Non-live(A)
Non-live(E)
Live(A)
Multi turn
Rel
Irrel
TooILLM(2.5w)
24.90
42.46
36.36
39.45
0.00
100.00
4.41
XLAM(2.5w)
40.51
81.94
81.77
43.18
4.38
73.17
11.87
ToolACE(2.5w) (Ours)
58.19
86.96
84.73
71.35
16.50
75.61
86.42
E.2 ABLATION ON VARIOUS TYPES OF DATA
To underscore the importance of incorporating diverse data types-such Nested, Parallel, Depen- dent, and Multi-type, as described in Table maintain the same overall dataset size (25,000) and selectively replace samples from the Nested, Parallel, Dependent, and Multi-type categories with samples from other data types. Then we train the LLaMA-3. 100MAA33.1-00 1-8B-Innstruct model and evaluate its performance on the BFCL benchmark. The results are summarized in Table 7
The findings show that removing parallel execution data significantly impairs the model's ability to invoke multiple tools concurrently. This leads notable decrease performance on Non-live AST and execution tasks, which rely heavily parallel tool usage. Furthermore, excluding multi-type samples hampers the model's ability to detect when the candidate tools are irrelevant the question, resulting in only 6.99% accuracy in irrelevance detection. The model's ability to handle multi-turn function calls is also impaired. multi-turn testing, the models sometimes are required not to call functions, but to ask clarifying questions instead.
In contrast, removing nested and dependent samples has a relatively minor effect on the model's tool-using ability in the BFCL task. Few test samples require nested arguments, and almost none involve dependent tool usage. However, including Dependent and Nested data types contributes to greater data diversity, leading slight improvements in overall performance.
17

---

Published as a conference paper at ICLR 2025
Table Ablation study on various types of data in ToolACE datasets. The models are evaluated on
BFCL benchmark.
Subset
Overall
Non-live(A)
Non-live(E)
Live(A)
Multi turn
Rel
Irrel
w.o. Parallel
50.60
74.75
77.30
72.19
1.75
78.05
85.05
W.O. Dependent
57.97
87.63
85.55
71.17
15.50
80.49
85.62
Nested
57.19
85,46
84.48
70.19
15.38
78.05
86.45
W.o. Multi-type 42.71
89.46
85.50
47.89
1.75
95.12
6.99
ToolACE 58.19
86.96
84.73
71.35
16.50
75.61
86.42
Table Ablation study on complexity evaluator. The evaluator represents the model used to evaluate the complexity. The learner denotes the model to be trained. Qwen-7B, Qwen- and I.LaMA88B are abbreviations Qwenl.5-7B-Chat.
Qwenl.5-115 1--B14-Chhh,, and LLaMA-3, 3.1-8B, respectively.
Evaluator
Learner
Overall
Non-live(A)
Non-live(E)
Live(A)
Multi turn
Rel
Irrel
Qwen-7B
LLaMA-8B
57.61
90.42
85.88
71.30
13.12
87.80
78.12
Qwen- 14B
LLaMA-8B
57.67
87.98
87.02
73.30
11.75
87.80
84.00
LLaMA-8B
LLaMA-8B
59.22
89.27
90.07
73.21
14.37
85.37
83.81
E.3 ABLATION ON COMPLEXITY EVALUATOR
To assess the complexity of the training data, propose self- evaluation method, where the model being trained serves as its own evaluator. To verify the suitability of this approach, we conduct additional experiment using
independent model (Qwen1.5-1) 1-5-1----hhhh 7htt,,, selected for its comparable size to ensure fairness) as the evaluator. The results, shown in Table g indicate that using the model being trained as the complexity evaluator offers more accurate guidance, leading to improved performance on the BFCL benchmark. Notably, when the complexity score is assessed using more advanced model (Qwen-14B), some simpler training samples- those deemed easy by the evaluator but not necessarily by the learner may be excluded. This leads to slight performance gains more challenging tasks (e.g., Live AST) but results in degradations on Non-live AST tasks [ Conversely, when the evaluator is less capable than the learner, the retained samples tend to be relatively easier for learner, resulting in improved performance on Non-live AST tasks but a decline in performance on Live AST tasks.
Table 9: Comparison between in-context learning and finetuning.
Method
Non-live(A)
Non-live(E)
Live(A)
Rel
Irrel
LLaMA-SB (3-shot)
58.81
53.32 90.07
36.83
82.93
23.66
ToolACE (finetuning)
89.27
73.21
85.37
83.81
F PROMPTING TEMPLATES
To provide better comprehension of the two benchmarks used in experiments, we have illustrated two examples for BFCL and API-Bank in Figure 13 and Figure respectively.
G FINETUNING VS IN-CONTEXT LEARNING
Given 3 shots for LLaMA- 1-8- the model still fails to generate correct arguments for such a simple example, such as Figure 16 demonstrating the limited ability in tool using under the in-context learning setting. Besides, due the addition of few-shot examples, the input the
"ivve AST tasks involve rarer and more complex functions compared to Non-live AST tasks, detailed in BFCL's documentation.
18

---

Published as a conference paper at ICLR 2025
model consumes lot more tokens than the fine-tuned model, which successfully addresses the aforementioned example in a zero-shot setting, as presented Figure [3]
Furthermore, conducted experiments BFCL under the RAG-based few-shot in-context learning setting. Specifically, we use the training samples as few-shot examples and retrieve the top most relevant ones according the user's question and the provided tools with the BGE model to guide in-context learning. The results illustrated in Table / show that few-shot in-context learning not only underperforms fine-tuning in BFCL but also falls short of the zero-shot setting. In many cases, illustrated in Figure T6 the model is misled by the tools in the few-shot examples due to its limited reasoning ability and generalization, selecting those instead of the tools in the test sample, which further exacerbates the model's 's ballucination phenomenon.
H LIMITATIONS
While we have conducted extensive experiments to demonstrate the effectiveness of our synthesis dataset enhancing the functional-calling performance, several challenges remain in our research.
Data Complexity Evaluation. The computational complexity of data complexity evaluation is influenced by the size the model being trained, which limits scalability as both the model size and the number of training samples increase. Additionally, the non-uniform sampling may introduce bias, such as causing the model to struggle with learning difficult examples after one round of training, effectively remaining in its comfort zone. In future work, will further explore the proposed complexity-base -ased sampling strategy to perform iterative training and sampling over multiple rounds, thereby progressively enhancing the model's generalization capability on more challenging samples.
General Capabilities. Although ToolACE demonstrates comparable performance in functional calling. still lags behind GPT-4 in other capabilities. While this success highlights the potential of specialized models in one specific domain, the challenge of simultancously enhancing multiple capabilities, alongside -unctional-calling performance, remains an open question. Exploring the collaboration of multiple small, domain-speciff agents may provide a promising direction for addressing this limitation.
19

---

Published as a conference paper at ICLR 2025
System:
You expert composing fanctione. of posaible functions.
given question
Based the question, you will need make one or more function/tool calls to schieve the purpese. If of the functiens given question lacks the parameters required by the function, alse point it out. You should only
be used, point out. If the
the function call in the tools cail sections.
It you decide to invoke any of the functionns) you MCST put it in the format [Sunc_namel
nemermpammmmmmmmmm.....mm pepearams_valull, parans_ness sata__nnee-aaaas__eleeeee junc_named [params)] You SHOUTO NOT include lleze is 1ist of Lunctions in JSON Zormat that you can invoke:
other text in the reaponse.
[["name:: "get_weather_dazz "description": "retches weather data from the latitude longitude.", "parsmeters"; ["type": open-Meteo APT the ("coordinates"; ("Lype"; "array", *ilems"; ("Lype": "dict", "propezties"; "Eloat"), "description": "The boalo_binmiil_proballe
location. "H), "sequired": and longitude "description": "Calculaces
('coordinates"]]] ("namet; the prebability of getting k successes in trials.", "parameters"; ("type": *dict", "properties"; ("=": (*type*: "integer", "descriptien"; "The number of trials. "k": |"type": "float", "description": "The number of sucnesses. "), "p": (*type": probability success. "I], "requirea"; [*n",
"Float", "desoription"; "p"]][]
User: I'm planning small cutdoer event in OLLawa, and need to make sure the seather is going to cocperate. Could you fetch the current weather for me at letitude 45.4215 and longitude -75.6972 using the open-Neteo
Also, i'm rurning amall game at the event, and T'm curious about the chances of x' nning. Tf have 10 sttempts at winning
this game the times?
time 50%, how likely
Assistant: sget_keather data (coordinates= (45.4,44 -75.6972]), per.biiromill_prcbslllig (m=10, k=5.0, pr0.5)1
Figure 13: Example of BFCL. benchmark. The blue text represents the part to be evaluated.
20

---

Published as a conference paper at ICLR 2025
System: saged on the given API description and the existing conversation history 1..t, please generate the APT request that the should call in step and outpuc it in the Cormal of [ApiNane Ikeyl=' valuel', key2=" the actual API name, and replace the key and value with the actual parameters. Your output should start with square bracket "I" and end with square bracket "I" Do not output any other explanation or prompt or the resu t of the APT call in your output.
...). replace the ApiName with
This year is 2023.
Input:
User: [User's plain text reepense of API call]
AL: lai's plain text]
User: [User'' a plain text or reapense of APT call]
Expected output:
Nape (keyl=rvalue'', key2ar ...]]
API descriptions [i"nmme: "GetUserTooenn" "description": "Get the token
and password.", "smput_parametess"" ("username": ("type": "str", "description": "the of the user."), "pessword"; ("sype": "str*, "deseription"" "The password of the ",,, "oatput_ppramelers
(*Loken"; "Lype"; *str", "description" the user. I"name": "AddAlarm", "description": "The API setting alarn includes parameter for the alarm time.", inpot_parrmeters"" ("token": ("type": "str", "description"; "Uzer"s token. "time": ; ("type": "str", "descristion": "the time for slarm. "cutpot_ parrppttparamttars": (*status"; (*typs"; "str", "deseription": "success or failed"]]]
fr-tm-td in:ix:es"),,
you help en
tomorrow?
Assistant: Sure, to set an slarm, I need to authorize your scccunt. Can you please and password?
provide
my
password
[Setusericken (username= foo", passward="baz""
Tool:
[jetuserTcken Response: ("token": "2oxdeTv@b5mddmags11"
Assistant: Okay, I got your token. What time de you want the alarm to be aet for?
User:
8 an tomorrow. Today is 2021-10-13.
Assistant:
[AddA larm (to.kn=* toxeeoooeee 1021111021 08:00:00")
Tool:
[Addk] Respense: "success")
Assistant:
For Lomozrow.
Figure 14: Example of API-Bank benchmark. The blue text represents the part to be evaluated.
21

---

Published as a conference paper at ICLR 2025
System:
You are expert composing fanctions. are given a question more
set of possible fonstions. Based the question, you wi.l need to make functien/tocl calls to achieve the parpose. If noge of the function can be used, point if the given questien lacks the parameters required by the function, also point it out. You should only return the function call in tools call sacticns.
If you decide to invake any of the (sl you BEST put it in the format [*une_name (parama taaaa-aaaaaa_aaaalllllaaaaaaaaaa Tunc_name2 [parana) You SHOULD NOT include any other text in the reeponse. Here is list of functions in JSON format that you can invoke. [f'name': gall_saboluute_rresseee
saram__nameeepprane warrrmmsswalleeeeee lus2,
ddsscription': 'Caleulates the absolute pressure from gauge and atmospheria preasurea.', 'parametera': 'atm_pressuue': I 'type' 'inzeger', 'description': "the stmospheric pressure stmospheres (atm). Default is atm if not provided." I, "integer', 'descriation': "The gauge pressure atmoapheres (atm). Must be provided." 'required"; "gauge_preseuze'
'type' 'diat', 'properties':
'type'
User: Calculate the absolute pressure in paseals given atmospheria pressure atm and a gauge presaure of 2 atm.
Assistant:
httpsabolute__eeeeeeeee (ate pressureel,, 2ouge_pressure=11
Figure 15: Case: model fine-tuned with our ToolACE data can solve the question in the zero-shot setting.
22

---

Published as a conference paper at ICLR 2025
You are an expert in composing fanctions. You are given a question and set of possible functions. Based on the question, you will need to make one or more funetien/tool calls to achieve the purpose. If none of the function it out, If the given questien lacks the paremeters required by the Sunction, also point out. You should only return the function call in tools call sactiens.
System:
be used, point
you decide to invake any the function(sl, you SUST put it in format of ["unc_namel (parama Searama_namal-ppp palsaaamm_aaalue Tunc_namez [parana) You SHOUL Not include any other Lext in the reeponse. Heze is list of functiens JSoN format that you invoke.
va va02,
[["naee; "solve__uadratie", 'description"; "Solve a quadratic equation given coefficienta a, b, and return real roota, If not specified, function may return conglex roots,", "pazamezers"; (*type*; "dict", "prepertiea": coefficient of the squared tern in the quadratic equation."), "b"; ("type"; "integer", ddescription:: "The coefficient of the linear term in the quadratic equatien."; "e"; ("type": "integes", "descriptien"; "The constent term the quadratic equstion, "), sroot_type"; ("type": "string", "description": "The type of rocts to return: 'real' for real roota, 'all' for both "required"; ["s", "b", "e"]]]]
If optional 'root_rype'
'real', the function will only
("a": ("type"; "integer", "description"; *The
complex rocts. Dafault value is "real"."),,
Bere are some examplea you refer:
Available tools: [['name': 'Funetioninterse:: calenlateRoots', ctt.://ceeeeeeeecccoccaaaeeee///
'dascriptien':
ideetifies the roots of the aquation formed by setting two furctions equal to each other.', , 'parameters';
('description': "The equatien obtained by setting two fuactions equai, e.g., '3x2 i 2x 1 x3 2x + 4'.", ", "type": "string'), J, 'preciaion': l'description': "The numerical precision for calculating rocts, specifled "The numerical method to use for finding reots, such as "Nesten-kaphhon' or 'Bisaction'. lher: Could you please help ma find the roota of the equation 3x2 ix +
l'type'; 'dict', pproserties': l'equation'
decinal.', 'Lype': "float"),
'methed'; : 'description':
'type': 'string'
'roquired'; ['equatson']]]
using the Disection sethed [FunctionIntersee) arrreioonnneerett...alllaaeeoer precision=0. (1),, runctionintersect.o ealculateRoots tog.laateootttoootttaaaoeoa method= Raphson", precision-O.01)
Raphson nethod? Assistant: again using the 7x 2 - 6", "Biseti"iissction", "quatii=="xx2
7x = 0",
camplaxxrros- coo=_finder', 'description' 'Finds roats of complex polynomial within a specified region.', Usez: aeed find the rocts
Availabla tools: (I'namo':
within four different reçions specified by the following centers and radii: (1+21, 5), (-3-41, 3), (0-11, 4), 12+21, 61. Each analysis should be done with high resolution during the optimel times of day.
polynomial "2l 23 222
Ansistant:
campplxx_roo____e_ finder (pllmmmapplm + zi 2x2 + 6", regions ("centar":
(*real*: 1.3, "imaginary": 2.0, "radius": 5.0), search_ ____ppaaaaaaaaaa___a ("rssottr 1000, "Lime": "sorning")), complex_root_finngg (inder (polycciala)))
23 222
6", region=
("real": -3.C, "imaginery": 4.01, "radius":
("resolutttrr 3.0), roattoootttteet yn complex_rost
1000, "time"; "morning")), 2x + 6", vegion="senterr" *real*: 0.0, "imaginary"; -1.0, "redius": 4.0, paarcc resoolttio::: 1000, "time": "morning"), complex_root root_ finder (pclyootalo))) - 23 - 2x2 + x 6*, ("centar""eter"" (*real*; 2.3, "imaginary": 2.0), "radius"; 6.01, parch_parameterr="""eoooo 1000, "time": "soraing"));
'description': 'solvas definite and inderinite integrals for a given mathematica fonction.', 'parameerr'"" l'type': 'dict', 'preperties'; l'function'; ('type': 'string', 'dascription': "Mathematical furatior expressed in standard mathematies notation.' 'limits': l'type'; 'diat', 'prcperties'; ('lowes': l'Lype'; 'string', 'paltera';
available taols: [l'name's paaleslun.incogralsslte.
rs3yide(V) (W..M--135 1s', "description': "Lower limit of the integral. Must be numeric value, e.g.r 1-3,5', '0', '2',",, 'upper': : l'type': 'string', , 'pattern':
r=7V (VA. Ad-1 'deacription': *Upper linit of the integral. Must be a numeric value, e.g., 13", 14.5', '20' "i,,, 'required'; ['Lower', 'upper"]]; 'required'; ('funetion']])
Daer: T'm currently preparing for my calculua exam and need to solve aome incegrals. Can you solve the integral of fuaction f(x) 3x2 from Assistant: Aaleuuus.integrallllle "0", "upper": *4"))]
4 for me? x: 3*x4*2", Simitsm("loveri)
User: Find sll the roots of quadratis equation given coefficients a -11,
and o -4.
Assistant: [ealeulus. inwegnnnnnleeenneoo. (functions "lambbd x: 3xx**2", 1imita="lcwer"; "0", "uppor": *2"]]
Figure 16:
Case: Few-shot in-context learning cannot solve the quite easy question, hallucinating on calling tools in few-shot examples instead of the test sample.
23


Reviews:
[{"review_id": "yT2gFaaChE", "reviewer": "Reviewer_Pz3q", "summary": "This paper tries to improve the function calling capability of LLM by finetuning on a newly collected function calling dataset. Specifically, the authors propose a pipeline to collect new API usage data. This dataset is then used to finetune llama 8b model and shows comparative performances on two API using benchmarks.", "questions": "- As mentioned in weakness, additional experiment with other baselines should be included e.g. ToolLLM. Even the authors provide some results of xLAM in Table 2, I noticed that they are tuned on different base models other than 8b. So it is hard to draw conclusions and give credit to the dataset itself or to the base models. According to Table 3, 8b seems already very good at API calling evaluations.\n- The evaluation and experiment process is not quite clear. e.g. what are the benchmark APIBank, BFCL evaluating? What are their input, output, ground truth, etc? What is the metric used in Table 2,  Table 3?\n- Table 2 has many categories in the performances: Single Turn, Multi Turn, Live, Hallucination. Compared to the base model used by author (llama 8b), some categories have only limited performance gain while some could be much higher due to finetuning (Multi turn), thus leading to a higher average score. I don't understand these comparison categories and I don't see authors' analysis in that.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Collecting new data in a scalable way is important\n- The performance looks interesting as well by finetuning a small model", "weaknesses": "The experiment section is not quite convincing yet. Since the authors want to show the effectiveness of using their newly collected API (which according to table 1) is much more comprehensive, the authors should compare the performance obtained by finetuning on Table 1 datasets e.g. ToolLLM and that obtained by finetuning on their newly collected API data", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "unmuIWonar", "reviewer": "Reviewer_AcuD", "summary": "Great idea and well written. The authors address the challenges of collecting accurate, diverse, and complex tool-usage data for LLMs by introducing a novel self-evolution synthesis process. ToolACE synthesizes a comprehensive API pool and generates data through agent-based dialogues, guided by a complexity evaluator to ensure the difficulty level is suited to the model's capabilities. The paper presents dual-layer verification (DLV) to maintain data quality, combining rule-based checks and model-based validation. Experiments demonstrate that models trained on ToolACE data achieve state-of-the-art performance in function calling, outperforming models such as GPT-4 in specific benchmarks like BFCL and APIBank.", "questions": "In Figure 3, the \"without model\" approach occasionally outperforms the \"with model\" approach. Please provide an analysis to explain the reasons for this phenomenon.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "ToolACE introduces a unique self-evolution synthesis method, which is a systematic approach to generating diverse and complex data for function calling, addressing a key limitation in existing tool-augmented LLMs. The paper provides extensive experiments and ablation studies, comparing ToolACE-trained models with existing benchmarks on widely used datasets like BFCL and APIBank, and demonstrating superior performance.", "weaknesses": "1. Please include a complete example of a prompt and LLM response in the appendix so that readers can intuitively understand how the process works in practice.\n\n2. The paper lacks clarity and involves overly complex technical concepts. Although constructing a simulated dataset and fine-tuning the model are effective approaches to enhancing the LLM's function call capabilities, the additional concepts introduced, such as Self-Evolution, Self-Guided, Dual-Layer, and Multi-Agent, make the main idea harder to discern, leading to confusion for the reader. While the authors may believe these terms add richness to the paper, they detract from its central focus.\n\n3. In the ablation study, it would be valuable to compare the Retrieval-Augmented Generation (RAG) approach for retrieving task-relevant tools with In-Context Learning to optimize tool usage. Given the same level of engineering effort, explore whether these methods could achieve results comparable to fine-tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g8cgECC9aZ", "reviewer": "Reviewer_ViF9", "summary": "The paper \"ToolACE: Enhancing Function Calling with Accuracy, Complexity, and Diversity\" presents a novel data generation pipeline for function-calling tasks LLMs. The approach leverages a tool self-evolution synthesis module, a self-guided dialog generation module, and a dual-layer verification module to create accurate, complex, and diverse tool-calling scenarios. ToolACE aims to improve LLMs' zero-shot function-calling capabilities by generating comprehensive training data that is validated through rule-based and model-based checks. The experiments show promising results, particularly with the ToolACE-8B model, which outperforms several existing LLMs.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The introduction of ToolACE's multi-step data generation, including evolutionary diversity and self-guided complexity, provides an innovative solution for generating complex and diverse function-calling data.\n\n- The DLV system, combining rule-based and model-based checks, enhances the reliability of the generated data. This is a strong point, as it helps maintain data quality, which is critical for training LLMs effectively.\n\n- The paper provides an extensive set of experiments, including comparisons with state-of-the-art models and an ablation study to assess the contribution of different components like accuracy, complexity, and diversity in the dataset. These experiments illustrate the potential benefits of the proposed pipeline.", "weaknesses": "- The evaluation scenarios are limited to synthetic function-calling tasks and benchmarks like BFCL and APIBank. The paper would benefit from more realistic evaluations or applications in real-world tool usage scenarios. This would better demonstrate ToolACE’s utility beyond controlled benchmark settings.\n\n- The self-guided dialog generation process heavily relies on the LLM being trained to evaluate the complexity of generated data. This creates a circular dependency where the model is used both as a learner and an evaluator, which may introduce bias in the complexity estimation. More external validation or use of independent evaluators would make the results more robust.\n\n- The use of complexity-based sampling to dynamically adjust dialog difficulty has merit but may lead to unintended biases, as data that is either too simple or too complex is filtered out. The approach may fail to fully explore the impact of diverse and extreme cases, leading to gaps in the model’s capabilities in certain contexts.\n\n- While the paper compares ToolACE to several other function-calling models, the comparison is often superficial. The benefits of using ToolACE versus simpler data augmentation techniques are not well articulated, and it is unclear how much of the improvement can be attributed to the synthesis method versus the increased volume of data.\n\n- The paper claims that ToolACE-8B is competitive with GPT-4 series models. However, it does not fully address the limitations of ToolACE-8B in terms of generalization and applicability to a broader range of tasks beyond function calling. A more detailed discussion of these limitations would provide a more balanced perspective.\n\n- The font size in Figures is too small, which is unclear for readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "puHUu71IdW", "reviewer": "Reviewer_ygcU", "summary": "ToolACE sampled API-related data from LLM pre-training corpora, obtaining 26,507 APIs, and used a User Agent - Assistant Agent - Tool Agent structure to synthesize appropriate conversational data to obtain API call training data. Rules + LLM were used during the process to ensure the effectiveness of the synthetic data. Finally, it used an 8B model with LoRA to validate the effects.", "questions": "1. Since Nested, Parallel, Dependent, and Multi-type are essentially subsets of programming language capabilities, if they are indeed effective, does this suggest that direct training with programming languages (like Python) would be better? Furthermore, is the Data Interpreter[1] approach of directly exposing tool interfaces through Python a better solution? This needs further analysis.\n\n[1] Hong, Sirui, et al. \"Data interpreter: An LLM agent for data science.\" arXiv preprint arXiv:2402.18679 (2024).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1. Ranks third on the BFCL-v3 leaderboard (updated on 09/20/2024), and first among open-source models on API-Bank.\n2. The relatively large volume of synthetic data demonstrates the benefits of model fine-tuning at a larger scale.\n3. Covers various categories of tool calls including Nested, Parallel, Dependent, and Multi-type.", "weaknesses": "1. The paper shows high complexity with many unclear details. For example, how are the varying complexity levels (easy, medium, hard) actually defined?\n2. While the paper repeatedly mentions Nested, Parallel, Dependent, and Multi-type, it doesn't analyze their connection to actual performance or conduct ablation studies.\n3. The relationship between data scale and performance is not demonstrated, making it difficult to determine which aspects actually contributed to the effectiveness.\n4. There are concerns about potential data leakage between BFCL and TSS - can the authors prove there isn't significant data leakage?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)