Robust Function-Calling for On-Device Language Model via Function Masking
Authors: Qiqiang Lin
Muning Wen
Qiuying Peng
Guanyu Nie
Junwei Liao
Jun Wang
Xiaoyun Mo
Jiamu Zhou
Cheng Cheng
Yin Zhao
Jun Wang
Weinan Zhang
Venue: ICLR 2025 Spotlight (2025)
Keywords: language models
function-calling
mobile assistant
tool-using
Primary area: foundation or frontier models, including LLMs

Abstract: Large language models have demonstrated impressive value in performing as autonomous agents when equipped with external tools and API calls. Nonetheless, effectively harnessing their potential for executing complex tasks crucially relies on enhancements in their function-calling capabilities. This paper identifies a critical gap in existing function-calling models, where performance varies significantly across benchmarks, often due to over-fitting to specific naming conventions. To address such an issue, we introduce Hammer, a novel family of foundation models specifically engineered for on-device function calling. Hammer employs an augmented dataset that enhances models’ sensitivity to irrelevant functions and incorporates function masking techniques to minimize over-fitting. Our empirical evaluations reveal that Hammer not only outperforms larger models but also demonstrates robust generalization across diverse benchmarks, achieving state-of-the-art results. Our open-source contributions include a specialized dataset for irrelevance detection, a tuning framework for enhanced generalization, and the Hammer models, establishing a new standard for function-calling performance.

Published as a conference paper at ICLR 2025
ROBUST FUNCTION-CALLING FOR ON-DEVICE LANGUAGE MODELS VIA FUNCTION MASKING
Qiqiang Lin
Muning 2en
Qiuying Pengl- 1-f
Guanyu Nie
Junwei Liao?
Jun Wang
Xiaoyun Mo!
Jiamu Zhou'
Cheng Cheng'
Yin Zhao
Jun Wang
Weinan Zhang
OPPO Research Institute
SSanghai Jiao Tong University
'lowa State University
(penggiuying wangjun7 weopnn..po..ooomom com wnzhanges jtu. edu.
ABSTRACT
Large language models have demonstrated impressive value in performing au- tonomous agents when equipped with external tools and API calls. Nonethe- less, effectively harnessing their potential for executing complex tasks crucially relies on enhancements in their function-calling capabilities. This paper identi- fies critical gap in existing function-calling models, where performance varies significantly across benchmarks, often due to being misled by specific naming conventions. To address such an issue, introduce Hammer, a novel family of foundation models specifically engineered for on-device function calling. Ham- mer employs an augmented dataset that enhances models' sensitivity irrelevant misleading. functions and incorporates function masking techniques to minimize Our empirical evaluations reveal that Hammer not only outperforms larger models but also demonstrates robust generalization across diverse benchmarks, achiev- ing state-of-the-ae results. Our open-source contributions include a specialized dataset for irreevvacee detection, a tuning framework for enhanced generalization, and the Hammer models, establishing new standard for function-calling perfor- mance.
INTRODUCTION
Large language models (LLMs) have demonstrated remarkable proficiency in addressing a wide range of natural language processing tasks (Chowdhary & Chowdhary (0ood,.ry22000)... as well as in handling long-context reasoning and complex planning (Wen et al. 2024). The use of LLMs as autonomous agents to assist humans in completing intricate tasks is increasingly in demand and is now more feasible from technical standpoint than ever before (Gunter et al. 2024). To fully capitalize on the potential of LLMs as autonomous agents, it is crucial for these models to accurately identify and utilize external tools or application programming interfaces (APIs), thereby enabling them to effectively execute complex tasks (Abdelaziz et al. 2024 Patil al. 2023). Central this capability the model's 's ability to select appropriate functions from a given set of options, provide accurate input arguments, and ultimately fulfill the user's intent. Furthermore, in scenarios where no suitable function exists within the available options, the model must have the ability to decline the task, rather than making incorrect attempts (Patil al. 2023). Recent advancements have introduced a variety of relevant datasets and benchmarks (Li et al. 2223) Wu et al.] (2024). along with the
release of powerful models specifically designed for function-calling tasks (Zhang et al. [2024] Patil
et al. 2023 Abdelaziz et al. 2024). Some models even simulate real-world scenarios, such as
'Equal Contribution.
sonnesonn Author.
Work conducted during an internship OPPO Research Institute.

---

Published as a conference paper at ICLR 2025
Table 1: Inconsistent performance of existing function-calling models across different benchmarks. For example, although XL-AM-BB-fc achieved the best performance on most of the benchmarks, its performance significantly declined on the other two, resulting in the lowest average score overall.
Models
BFCI.
API Bank
SealTool
Tool Alpl-AApcca
Nexus Raven
Avg.
Gorilla-Opeat Saniilaa-Oppauunctioas---22 (FC)
79.1
62.5
91.1
51.3
68.4 75.1
70.48
Granite 2013 FunctionCalling (FC)
76.63
92.7
74.186
xi A----BB-fc (FC)
79.41
72.45
76.9
59.0
57.5
69.052
ticketing systems, to mimic more realistic use cases Yao al. 2024 Chen al. 2024a Despite
these significant strides in the development of function function-calling models, our investigation reveals a critical gap: many existing models demonstrate considerable performance variations across different benchmarks. As illustrated in Table I] this inconsistency underscores the need for further research into the robustness and generalization of function-calling models across diverse and practical task environments.
Achieving such stability across diverse benchmarks is crucial, as it indicates the model's 's capability to generalize effectively to real-world applications (Yao et al.) 2022 Zhang et al.) 2025). Driven by this objective, we conduct a thorough analysis the instability observed in existing models when executing function-calle calllnn tasks. Our findings highlight that one of the primary factors influ- encing generalization performance across benchmarks is the misleading nature of specific naming conventions for functions and parameters. Consequently, existing models tend to perform well on benchmarks that closely align with the naming conventions in the training data but suffer notable performance declines when encountering benchmarks with differing naming styles. This problem is examined in detail in Section 3, which motivates us to propose the function-masking technique.
In this paper, we present the Hammer, family of lightweight models specifically fine-tuned for on-device funetion-calling tasks. This work is underpinned by a carefully designed irrelevance- augmented dataset and the use of function masking techniques, both aimed enhancing the gener- alization capabilities of the models. To improve the models' ability to determine whether the user's s intent aligns with the available function calls, we augment the CAAA-ffunttioncaat caaoonn-clllnnn--i dataset (Liu et al.) 2024b) with an additional 7,500 instances specifically tailored for irrelevance detection. Furthermore, we introduce a function masking technique, which shifts the models' focus from func- tion and parameter names to their descriptions, effectively reducing potential misinterpretations.
Following these advancements, Hammer demonstrates robust function-calling performance and strong generalization across variety of benchmarks. Despite containing only billion param- eters, Hammer outperforms many larger open-source models and competes with top-tier closed- source models, such GPT-4 (Achiam et al. (023) and GPT-4o (Islam & Moushi 2024), the Berkeley Function Calling Leaderboard (BFCL) v2 (Yan et al. (022)). We benchmark Hammer and other models, including Salesforce's 's xLAM series (Zhang et al. (0224) and IBM's Granite-20B: FunctionCalling (Abdelaziz et al. 2024). across a range of representative datasets, such as API-Bank (Li al. 2023 Tool-Alpaca (Tang et 2023), Seal-Tools (Wu et 2024), and Nexus Raven API Evaluation (Srinivasan et al. 2023). The results consistently highlight Hammer's 's exceptional generalization capabilities. The key contributions of our work could be summarized follows:
Tuning Framework: A straightforward yet effective framework evolving function mask- ing to tune function-calling models toward robust generalization capabilities.
Augmented Dataset: A specialized dataset with 7,500 instances designed to enhance lan- guage models' awareness of irrelevance between candidate functions and user intent.
Consistent SOTA Models: Hammer, a family of well-trained function-calling models that demonstrate of-te-of-the-ar performance across multiple benchmarks TT
2 RELATED WORKS
LLMs as Agents for Function-Calling Recent research has shown significant interest in leveraging
LLMs as autonomous agents to perform complex tasks through function calling and tool usage Erdogan al. 2024a Chen al. 2024b). IBM Granite-205 1aieee20O--FFnnnnnnaaa
model (Abdelaziz ct al. 2024) proposes a multi-task learning framework trained on seven core function-calling tasks, demonstrating superior performance over other open models on the BFCL v2 benchmark. APIGen
2

---

Published as a conference paper at ICLR 2025
Table 2: Performance of different models on Berkeley Funetion-Calling Leaderboard (as of date 09/20/2024). The rank is based on the overall accuracy, which is a weighted average of different evaluation categories. "FC" stands for function-calle calling mode in contrast using a customized "Prompt" to extract the function calls. For the complete list, refer to Table 10 in Appendix E
Rank
Model
Overall Ace
AST Summary
Exec. Summary
Irrelevance
Relevance
2-- (Prompt)
85.79
85.50
89.25
61.35
97.56
GPT-4-1006 1066-reviees (Prompt)
85.00
86.31
87.38
64.98
90.24
GPT 4-61-4-0613 (Promp))
84.74
84,66
87.57
75.57
82.93
Hammer-7B (FC)
72.87
92.68
GPT-4-turbu: 04-4-04--999--- (Prompu)
83.89
85.41
88.13
61.82
82.93
GPT--o-miii-20224007-- (Prompt)
83.35
80.52
87.95
79.20
80.49
Functionary-Mrciiu--.o
81.06
73.23
70.73
Punctionary-Smaall--3--88 (FC)
80.21
78.64
83,45
68.36
85.37
ALAM-7B-fc (FC)
79.41
72.77
85.68
79.76
80.49
Gorilla OpenFuncciooss v2-7B (FC)
85.37
Sumcciionry-Sumall333----8- (FC)
78.96
76.16
83.04
72.32
80.49
F (FC)
77.45
74.20
84.23
52.94
$7.80
Gramio--00B-FucctionCClllig
76.63
66.73
72.43
95.12
Hammer-4B (FC)
76.05
69.59
80.82
68.66
90.24
xLAM- (FC)
74.90
67.37
80.80
61.21
95.12
Hermes-2-Pro-llam--333 (FC)
74.78
72.09
81.29
53,80
80.49
Hammer- 1,5B (FC)
73.04
65.53
75.86
72.18
92.68
Command-R-Plus (FC)
72.04
66.32
77.41
52.75
92.68
Hermes-2-Pro-JJanna-- (FC)
66,18
64.18
74.05
55.16
53.66
46
Hemmss2-Pto--isttll (PC)
65.44
60.82
74.25
38.55
75.61
Hermes-2-Thett-LJamms (FC)
62.66
51,22
FineFunction-11 1-1-66 (FC)
48.11
38.16
41.20
68.55
95.12
Table 3: Performance comparison of different models several academic benchmarks. The rank
based on the average F1 score on "Func. + Args", which indicates both function selection and parameter filling are accurate. For intermediate details see Table 11 in Appendix
Academic Benchmarks (FI Fune. + Args)
API-Bank
API-Bank
Scal-Tools (Single-Tool)
Nexus Raven
Fl
Model
Tool-Alpaca
Average
GPT-4-0613 (Prompt)
84.78
93.95
91.60
GPT-4o-mini (Prompt)
89.28
67.52
54,69
86.00
84.59
76.42
Hammer-7B (FC)
85.79
66.40
59.86
91.66
77.35
76.21
Crannte-30B-PunciionCllll (FC)
92.70
75.14
72.56
Hammer-4B 4B (FC)
81.46
61.01
$6.96
92.45
64.89
71.35
xLAM-7B (FC)
Gorilla-OpenFunntioon-222 (FC)
70.34
$4.69
51.26
91.11
68.41
67.
KLAMM-1.3B-fc (FC)
83.70
64.32
50,58
80.43
54,80
66.77
Hammer- 1.5B (FC)
72.30
59.71
53.48
88.65
56.88
66.20
Qwen2-7B-Instraet (Prompt)
60.62
49.50
48.11
77.51
63.47
59.84
Qwee2-1.5 Bruu2-1.B-Lnstruus (Prompt)
Qwenl.5 5-B Chat (Prompt)
59.78
38.48
16.98
62.32
33.70
42.25
adopts an automated pipeline for generating high-quality, diverse function-calling datasets, building 7B-parameter model surpass GPT-4's performance (Liu et al.) (024b). Similarly, ToolACE (Liu et al. 2024a) generates diverse tool-learning datasets, allowing its 8B-parameter ToolACE- 8B model to achieve state-of-the-ee results on the BFCL v2, rivaling the latest GPT-4 models. Further studies explore various dimensions of function calling, such as improving efficiency through parallel function calls (Zhang et al. 2016). identifying vulnerabilities function calling processes (Siiivasaan et al.) 2023). and developing benchmarks to evaluate L.MMs ability to handle diverse function calls (Kim et al. 2024). Collectively, this body of work emphasizes the role of function calling in enabling LLMs to act autonomously and integrate external tools and resources effectively.
Datasets and Benchmarks for Function-Calling Evaluation. Substantial advancements have been made in developing datasets and benchmarks to assess the function-calling capabilities of LLMs. API-BLEND (Basu et al.) 2024) introduces large corpus for training and systematically testing tool-augmented LLMs. It includes real-world scenarios involving API-related tasks such API/tool detection, slot filling, and sequencing of detected APIs. API-Bank (Li et al. 2023) provides a com- prehensive dataset featuring 2,8 distinct APIs and 1,888 dialogues with 4,149 API calls. This
'The source code is available at hhtpps: /tttls://gtthta www.///aee//eeetteeeccccooo H/ammmeeetttt/ere.i.aaaa Hammer] while the aug- mented dataset and models can he accessed at https: httsppa:///ugg/// //uggi ng face co co/Madexgent..... The latest release, the Hammer 2.1 models, significantly improves multi-tum and multi-step function calling capabilities.
3

---

Published as a conference paper at ICLR 2025
Candidate Functions:
Query eig the London
Function to Call:
Candidate Functions
Toree' eane!": "Srranaaa
Function
Function to
Function 2
parmmsenrS parm
Function
Figure I: Demonstration of simple function-calling process.
dataset is designed to evaluate LLMs' tool-utilization capabilities, including planning, retrieval, and API-calling proficiency. APIGen (Liu et al. 2024b) employs automated and rigorous data gener- ation process to create a diverse dataset that includes various query styles, such as parallel function calling, and undergoes multi-stage verification process to ensure data accuracy and relevance. Seal-Tools (Wu et al., 2024) introduces a large-scale, self-instruct API-like tool-learning dataset that incorporates practical application scenarios and nested tool calls.
Tuning Techniques for Function-Calling Models. IBM's Granite-20B-- 20B--unctionnalling model is trained using a multi-task learning approach, which enables language models develop function- calling capabilities by mastering range of granular tasks (Abdelaziz et al. 2024). TinyAgent focuses on equipping small language models (SLMs) with complex reasoning and function-calling abilities, allowing for secure and private deployment at the edge. It employs LoRA fine-tuning, incorporates negative samples, and uses in-context examples selected via retrieval- -uggeeetd gen- eration (RAG) (Gao et al. 2023) to enhance function selection and orchestration accuracy through directed acyelic graph (DAG) comparison (Erdogan et al. (024b). The xL.AM series utilizes a super- vised fine-tuning (SFT) approach with direct preference optimization (DPO) (Rafailov et al.) (024) alignment, integrating data parallelism, LoRA, and cosine learning rate scheduler to optimize per- formance across various categories of function-calling agents (Zhang al. 2024).
3 PROBLEM STATEMENT AND ANALYSIS
This section aims to introduce and analyze the common challenges that function-calling models encounter in practical applications. Through this analysis, we seek identify methods enhance models' stability and generalization capabilities in real-world scenarios.
Before delving into the specific issues, we present typical function-callinn process, illustrated in Figure T In this process, each candidate encompasses several components, including the function name, parameter names, default values, and descriptions. The objective of the model to output complete and accurate function-calling code that accomplish users' intent or, alternatively, out- put an empty list to indicate that none of the given candidates can satisfy the user's requirements Yan et al. 2024). Achieving this goal hinges on the model's ability to accurately align the user's intent with the functionality of the candidate functions, i.e., selecting the appropriate function, and its capacity to comprehend the usage of each parameter, i.e., populating the function with the correct arguments. However, certain recurring issues have been observed in practice.
3.1 MISLEADINGNESS BY FUNCTION NAME AND PARAMETER NAME
As illustrated in Figure the definition of function typically comprises the function name, param- eter names, and descriptions. The format of function and parameter names is often quite compact, e.g. cal sum or max value, and influenced by the designer's 's personal style and preferences. When model attempts to infer the function's purpose solely from the function name, this compactness can lead ambiguities, misguiding the model's selection, particularly the presence of complex functionalities (Gunter 1992). For instance, function named parse_data might be intended for parsing JSON data, but the same name could refer to parsing CSV files different context, lead- ing potential misinterpretations. Similarly, when deducing the usage of parameters based on their names, models may be misled by the historical usage similarly named parameters the training dataset. More specifically, these misleading scenarios be categorized into several cases.
Misled by Function Names. When a user intent aligns closely with a function name present in the training labels, the model may incorrectly prioritize that function from the candidate list during
4

---

Published as a conference paper at ICLR 2025
testing, even if its functionality diverges significantly from the intended operation. For example, a function named fet ch_data is included in the training pairs for retrieving user data from a database, but in the testing set, a function with the same name retrieves data from external API, the model may erroneously select based solely on the name.
Misled by Parameter Names. When the functionality and descriptions of parameters change within the testing environment, the model frequently clings to its original patterns of parameter usage, resulting incorrect function calls. For instance, function's parameter timeout is expected to be an integer representing seconds in one context, but in another context, it is defined as a string in the format " "10s", the model's reliance on the original integer format may lead erroneous calls.
Disturbed by Naming Preferences. The model's robustness can diminish when the naming con- ventions of functions parameters the testing environment diverge from those in the training dataset. Variations, such as discrepancies between Came SamelCar Case and snak kelcase may adversely lower the model's confidence, as an on-device lightweight model may struggle to generalize across different naming styles.
3.2 THE IMPACT OF EXCESSIVE FOCUS ON THE NAMING
To investigate the extent which existing models rely on function and parameter names and cor- responding impact, conducted a case study using the XLAM-IB-fc model the Scal-Tools benchmark. Specifically, we masked the function and parameter names the test set, i.e., replaced them with random strings, and observed how the model's performance changed. As shown in (Fig-) ure 2 after masking the function and parameter names, even though the descriptions contained all necessary information about the function's 's purpose and usage, the performance of XLAM-1B- fo dropped significantly. This result confirms the model's 's overreliance on function and parameter names, highlighting the potential risks this behavior may pose in real-world applications.
Func. Name
Func. Name Args
095
689
085
Sccare
8.0
0.75
no mask
no mask
mask
func
arg name all mask
arg name all mask
665
1.8--
Hammer-1155 1.8
1AM4-1.38-K
Hammer-1 1.
Case studies examining the performance degradation when function names and parameter names are obfuscated during test time. For a detailed analysis of the error cases, see Appendix B In contrast, Figure also presents the performance of our Hammer model under the same setting. Hammer exhibited a much smaller performance drop, demonstrating its robustness when faced with arbitrary function and parameter naming patterns. This resilience suggests that Hammer relies more heavily on the function descriptions rather than compact, potentially ambiguous names. Section will provide detailed explanation of Hammer's training methodology.
Figure 2:
4 METHODOLOGY
In this section, we describe our detailed approach and augmented dataset to fine-tune the Hammers, a series of robust language models designed for function-calling.
4.1 FUNCTION MASKING
In light of the analysis in Section 3 direct approach to mitigate these issues involves minimizing the interference from function names and parameter names, while enforcing the model to compre- hend the functionality and usage of candidates based on their descriptions. In contrast, the descrip- tions provide a more flexible natural language explanation, often encapsulating the information that function and parameter names aim to convey. While descriptions personal style to some extent, they tend to be more accurate and detailed, thus reducing the likeli- hood of ambiguity misguidance. Consequently, when training funetion-calling models, face
also reflect the designer's
5

---

DATASET Published as a conference paper at ICLR 2025
Supervised Fine-Tuning
www.tien
Augeested
Data Batch
Random Masking
Label Update
Masked Batch
Function Masking
Cansidate Faxetions
Monked Candibiee Fanstions
Figure 3: Step-by-step building workflow of Hammer series with function masking. Considering the complexities of real-world scenarios, the masking operation does not apply to all samples. Instead, Hammer enjoys a masking ratio of 0.33 before each training epoch, which yields the best overall performance across all benchmarks.
the challenge of not knowing the preferences naming styles of function designers in real-world applications. Thus, it is reasonable expect that the trained model should understand the func- tion's 's purpose and usage through its description rather than attempting infer functionality based on potentially ambiguous, compact components such as function and parameter names.
To this end, propose tuning framework for function-callie anism, with a full pipeline shown in Figure 3] As for the randomization process, we first randomly determine the length I. E (5, 15) of the random string, and then randomly select T. characters from set comprising 52 uppercase and lowercase letters and 10 digits to replace the corresponding names. This framework aims guide the model's attention toward the description, and thus enhance the model's generalization capabilities in practice. Specifically, in our proposed framework:
models based masking mech-
Function names in candidates are masked by replacing them with randomly generated strings during training. This technique minimizes the model's reliance on memorizing function names, prompting it to understand the function's 's purpose solely through its de- scription. By doing so. the model becomes more adaptable across various coding practices, as it is less influenced by common naming conventions.
Parameter names in candidates substituted with random strings as well, ensuring the model focuses on the parameter descriptions rather than the specific names, which often vary between implementations.
Default parameter values in candidates are randomized and appended to the parameter descriptions. This also guides models to pay more attention to the parameter descriptions.
Labels in batch updated according the masked candidate list, i.e. replacing the function name and parameter name with corresponding masked strings in candidates.
By focusing on the description, the model can more accurately grasp the function's intent and ex- pected behavior, ensuring robust performance across diverse naming conventions and avoiding pit- falls introduced by overfitting to specific naming patterns in the training data. It also promotes better generalization, the description typically offers comprehensive view of the function's 's role, beyond what can be conveyed by concise names.
4.2 IRRELEVANCE-AUGMENN
During the fine-tuning process using the Llam-ffuctioon-callin-6600 dataset, identified con- cerning inverse relationship between the model's 's ability to accurately execute function calls and its capacity for irrelevance detection specifically, the ability to assess whether there exists no func- tion call in the candidate set aligns with the user's s intent. This observation analyzed further in Section 5.6 and Appendix G It indicates that, while fine-tuning lightweight language models

---

Published as a conference paper at ICLR 2025
datasets specialized in function selection improve their accuracy in choosing appropriate fune- tions from a predefined set, it may unintentionally impair their ability to detect irrelevance. As a result, models might generate inappropriate function calls, even in the absence of valid options.
To address this issue, we propose an areelvancce-ugmeenee datasct. This augmentation, applied dataset, incorporates 7,500 examples sampled from the original training set. In constructing this dataset, we removed the correct function from the candidate list for each sampled example and replaced the labels with empty lists, indicating that all candidates are irrelevant. This approach ensures that any potential data contamination is limited to what already exists within the beyond the original dataset. More details about the data contamination analysis refer to to/Appendix C
to the original
cotnnctioo-calling---e
cunction-callingg 60k dataset, as the process does not introduce additional data
5 EVALUATION
5.1
EXPERIMENTAL SETUP
In this section, we show the superiority of our Hammers in performance and robustness across various benchmarks as well as in-depth analysis to verify the effectiveness our augmented dataset and approach. Besides, potential limitations of function-masking could be found in Appendix D
Benchmarks. To assess the generalizability of Hammers, we conducted evaluations using a vari- cty of function-calling benchmarks, all of which represent out-of-domain challenges for model. The Berkeley Function-Caliie
Leaderboard (BFCL) Yann et al. 2024) provides a comprehensive dataset comprising over 1,700 instances. It covers tasks such Simple Function, Multiple Function, Parallel Function, and Parallel Multiple Function for Python, as well as function relevance detec- tion. REST API, JavaScript, and Java for non-Python environments. API-Bank (Li et al.) (023), consisting of 314 tool-use dialogues and 753 API calls, evaluates models' ability to correctly in- voke known API (I-1)) based query, and to retrieve and call APIs from candidate list (1.-2). Similarly, Nexus Raven API Evaluation (Srinivasan et al. 2023) offers 318 test examples across 65 distinct APIs. contributing further to the evaluation of function-calling capabilities. Tool-Alpaca Tang et al., 2023) employs synthetic data generation method, featuring 271 tool-use instances in 50 categories. For evaluation, we utilized 100 simulated test examples from this dataset, similar to Nexus Raven. Lastly, Seal-Tools (Wu et al. (024) represents one of the most extensive and recent benchmarks, with 4,076 automatically generated APIs across various life domains. As one of the newest benchmarks, Seal-Tools presents a relatively lower risk of data leakage. As for the Tool- Bench (Qin et al.] 2023). we chose not to utilize because of potential data contamination risks. Specifically, the xlam-fure cinnction-calling
dataset which serves as training set, generated using ToolBench (Liu et al., (024b). This poses a significant risk of data contamination if ToolBench is used benchmark, potentially leading to unfair advantages compared with certain baselines.
Evaluation Metrics. BFCL assesses dunction-calling models through two primary evaluation meth- ods: Abstract Syntax Tree (AST) Evaluation and Executable Function Evaluation Yan al. 2024). The AST evaluation emphasizes the syntactic precision of the generated function calls, ensuring that the model's output adheres a predefined function documentation in terms of structure and parame- ters. This includes verifying the correctness of function names, required parameters, and appropriate data types. In contrast, Executable Function Evaluation takes this further by executing the generated function calls to assess their functional accuracy. This evaluation ensures that the functions not only compile but also run correctly, producing the intended outputs, which is vital for real-world applica- tions. In addition to BFCL, we incorporated Fl scores to measure exact matches of API names and parameters in order to evaluate the models on alternative benchmarks (Abdelaziz et al.. 2024). In these scenarios, APIs are highly specific, and execution is only feasible if all aspects, such names, parameters, and input/output formats, strictly conform to the API specifications.
5.2 OVERALL PERFORMANCE ON VARIOUS BENCHMARKS
We first evaluate Hammer series on BFCL. Table 2 indicates that within the BFCL framework, our Hammer series consistently achieves corresponding sota performance at comparable scales, par- ticularly Hammer-7B, whose overall performance ranks second only to the proprictary GPT-4. In addition, we evaluated our Hammer series (1.5b, 4b, 7b) on other academic benchmarks to further show our model's generalization ability. Upon observing Hammer's performance across various
7

---

Published as a conference paper at ICLR 2025
Table 4: Detailed performance comparison of different models using Abstract Syntax Tree (AST) evaluation with regard to four function-calling styles on BFCL (as of date 09/20/2024).
Model
AST Summary
Simple
Multiple
Purailel
Parailel Multiple
GPT-4-0112-Previoe (Prempt)
85.50
78.82
88.44
91.00
83.75
GPT-4-1106 1-011666-HH-- Preview (Prompt)
86.31
78.75
89.12
94.12
83.25
GPT-4-0613 (Prompt)
84.66
78.76
91.75
82.67
Hammer-7B (FC)
78.70
69.31
82.52
78.88
84,08
GPT-4-tubor 2024 (Prompt)
85.41
80.47
88.81
88,
84.25
GPT-4o-mini-2024442 12t-mimm---024-7-111 (Prompt)
80.52
75.88
81.64
85.12
79.42
Functioary-Mesiun--33.11 1-70 (PC)
81.06
74.34
87.59
81.62
80.67
Funetionry-Smaall-V (FC)
78.64
xLAMM7B & (FC)
72.77
70.28
78.18
74.12
68.50
Gorilla-OpenPunciiotss (FC)
73.18
70.81
79.47
75.75
66.67
Sunciionry-Smaallv33.---88 (FC)
76.16
69.50
81.50
80.12
73.50
FireFunction-222200 (FC)
74,20
74.11
81.49
73.62
67.58
1oniee-20B-FunciinnCalin
66.73
65.27
73.05
60.75
67.83
Hammer-4B (FC)
69.59
62.58
77.72
69.12
68.92
ALAM-L.3B-Ic (HC)
Herrmes Pro- Plam Llama-3 (FC)
72.09
66.29
73.49
70.25
78.33
Hammer-l 1.5 (FC)
58.75
Command R Plus (FC)
66.32
64.25
72.45
66.25
62.33
Permes-2--Po.. Jama- 1-8B (FC)
64.18
62.32
74.96
61.62
57.83
Drrmms-2-Pro-Msstra-.
60.82
60.98
71.49
60.38
50,42
Tatiia----Tht--Laamm- (FC)
61.08
58.53
67.82
59.62
58.33
benchmarks unrelated to the <lam-function-calling-- Datasets, as shown in Table 3 we find that Hammer demonstrates remarkably stable performance, which indicates the robustness of Hammers. Considering that the demand for on-device applications is the primary motivation behind search, Appendix H further illustrates the non-functional metrics and hardware contigurations of Hammer-7B model when deployed mobile devices.
5.3 DETAILED PERFORMANCE ON DIFFERENT TYPES OF FUNCTION CALLING
In this section, we closely examine the performance of Hammer across different types of function- calling tasks, exampled in Figure 4 and detailed in Appendix
Simple
Multiple
Paraliel
Parallel Multiple
Irrelevance
Query Ws
Query Candidates
Query
Query
Query What
Wh and
Candidates
Candidates:
Candidates:
(nde]jntt w, Int bi]
[adeifint a, ini b),
[adellint a, int bi)
(wddlint a, int b),
[multjjnt w, ini bi]
mulbjinta
multlint
Answer [acid[a=], b-2),
Currect Answer
[addja=1, b-2]]
[add/a=1, 2-2]]
[addia-1, b-2),
add(a=d, b-4]]
mult(anš, b-4]]
Figure Demonstration of different function-calling tasks.
As shown in Table 4 and Table 5] we found that Hammer-7B demonstrates exceptional overall performance across these various tasks. Its AST Summary is second only to the GPT-4 series and Functiooary-MMediume -Mw-iM-eddd---------- 1-- Notably, Hammer-7B even outperformed GPT-4 in the practically relevant Executable Function Evaluation. highlighting the potential of Hammer and the function- masking technique in real-world scenarios. Moreover, observed that Hammer-7B achieved state- of-the-art results in both tables for the most complex Parallel Multiple task. This suggests that the cecetion-masking training approach becomes increasingly advantageous task complexity rises. This aligns with our insight that more complex tasks typically demand a deeper understanding of functions, necessitating models focus more function descriptions.
5.4 ABLATION DIFFERENT BASE MODELS
To further validate the effectiveness of our augmented data and tuning technique, we applied our approach to two different sizes of the deepseek-coder models, in addition to the Qwen series. The results are illustrated in Table 6 Upon examining the results presented in the table, we first note that the fine-tuned Hammer model exhibits notable performance improvement compared to the vanilla Qwen models (Bai al. 2023 Yang et al. 2024) as well as Qwen variants trained with the original xlam-functionn function- function-calling caaa-callnn----- 60k dataset, thereby confirming the efficacy of our data and methodology on the Qwen architecture. Subsequently, we compared the deepseek-coder model (Guo al.) 2024)
8

---

Published as a conference paper at ICLR 2025
Table 5: Detailed performance comparison of different models using Executable Function (Exec.) evaluation with regard to four cunction-calling styles on BFCL (as of date 09/20/2024).
Model
Exec. Summary
Simple
Multiple
Parallel
Parallel Mulliple
GPT-400 Preview (Prompt)
89.25
99.00
96.00
82.00
80.00
GPT-4-1 106-Previow (Prompt)
87.38
99,00
96.00
82.00
72.50
GPT-4-0613 (Prompt)
87.57
98.29
96.00
86.00
70,00
Hammeer 7B (FC)
89.72
91.86
94,00
88.00
85.00
HPTT44-turbo22224-444 (Prompt)
88.13
99.00
96.00
80.00
77.50
GPT-to-mini-20224-77- (Prompt)
87.95
98. 98.29
94,00
82.00
77
Fumctionary-Medding
89.32
75.00
Functionary -3.1 8B (FC)
83.45
87.79
90.00
86.00
70.00
XLAM-7B-fc (FC)
85.68
94.21
88.00
88.00
72.50
Gorilla- Gorilla Saallloppppauutttooss>22221 (FC)
84.97
95.86
96.00
78.00
70.00
Functionury Sm.2l (PC)
83.04
90.64
88.00
86.00
67.50
FireFunction-v2-7088 (FC)
84.23
94.43
88.00
82.00
72.50
Granite-2UB-PunctiooCCaa (FC)
82.97
85.36
90.00
84.00
72.50
Hammer-4BB (RCC)
67.79
77.50
X.3AM-133B-cc (PCC)
80.80
79.21
88.00
86.00
70.00
Hlemnes2-Pro-Liamme-3 (FCC
81.29
80.64
88.00
84.00
72.50
Hammer: 1.5B (FC)
75.86
49.93
92,00
84,00
77.50
Command Rlusus (FC)
77.41
89.14
86.00
82.00
52.50
Hermes-2-Pre-llmm--B3
74.05
68.71
90.00
80.00
57.50
Hermes--PPo-Miistaa- (FC)
74.25
60.50
90.00
84.00
62.50
Henmes-2--Teae Llama-3--aB0- (FC)
72.54
before and after fine-tuning; the fine-tuned variant, referred to as deepseek- coder---eeer,,,,,,r,, strates significant enhancements over the vanilla model, despite the poor performance of deepseek- coder prior to fine-tuning. This suggests that our methodology is not exclusively applicable to the Qwen model. Furthermore, is noteworthy that the performance of the deepseek-coder Hammerr fine-tuned using our approach, significantly surpasses that of the xLAM model, which was also based on deepseck-coder-instrre dataset. This further underscores the superiority of our proposed method.
Hammer, demon-
and obtained through SFT with the
Table 6: Ablation on different base models and benchmarks. We apply the masking tuning process Deepseek- Coder models. Besides, the Qwen-xLA.M series are trained with the original xlam- function-calling-66 60k without masking. For more details see Table and Table 13 Appendix E.
API-Bank I.11
API-Bank 1.2
Tonls (Single-Tool)
Nexus Raven
Model
Tool-Alpaca
Average
In-struc Qwea2-7B-xLLAM
Qwen2-7B Instruct
60.62
49.50
48.11
77.51
63.47
59.84
83.38
66.14
58.18
90.98
72.74
74.48
Hammer--B 7B
85.79
66.40
59.86
91.66
77.35
76.21
Qwen 1.5.4B 4B. Chat
59.78
38.48
16.98
62.32
33.70
42.25
78.64
58.32
55.48
83.28
64,10
68.96
Hammer-4B
81.46
61.01
56.96
92.45
64.89
71.35
Qwen2-1.5B- Innstutttt
63.55
Qwen2 1,5B xLAM
70.93
59.84
53.83
84.10
$4,90
64.72
Hammer-
72.30
53.48
56.88
XLAM-7B-fe (FC)
80.69
64.24
58.96
76.87
$1.50
67.65
Doopseck-Coder-1
56.70
Deepseek Coder 7B Hammer
75.18
60.04
62.95
93.20
83.35
74.94
1I--M- (FC)
83.70
50.58
Deepseek-Codss-1.. B.e-oodo---1---m1e Instrrctt
38.42
24.75
06.06
21.52
8,15
19.78
Deepeek-Coder-11 1.0-0--11.---aammee
77.22
65.97
57.68
88.98
64.68
70.91
5.5 ABLATION ON DIFFERENT MASKING RATIO
To further investigate the impact of various function masking ratios on model performance, de- signed an ablation study focused on the masking ratio. We systematically applied different masking ratios while fine-tuning the Qwen2-1.SB 1.5B model on the Seal-Tools training dataset for one epoch. Subsequently, we evaluated the performance of the models trained with different masking ratios on the test sets of both Seal-Tools and API-Bank. This allowed us to observe and analyze the perfor- mance across both same-task and cross-task scenarios.
Based on the results presented in Figure 5 impede the model's learning speed within the same task scenario, i.e. the test Seal-Tools. Con- versely, the testing results on API-Bank indicate that a larger mask ratio facilitates better general- ization of the model across different scenarios. This observation aligns with previous insights, suggesting that, in the absence of masking, the model may overfit to the training data during fine- tuning, negatively impacting its performance in novel task environments. By enforcing focus
we observe that an excessively large mask ratio can

---

Published as a conference paper at ICLR 2025
Name (Soal-bolss)
Name (Seal-Toots)
0.10
0.0
mask 0.67
mosk
masi
0.4
quer2
ol meai:
6775
musk
Step
Step
Name (API Bank))
Func. (API Banki
maakk
FRESi
Atrr
All mast speen2
Stop
Figure 5: An ablation to evaluate the impact of different masking ratios. For instance, "mask 0.33" denotes that 33% of the instances the training batch are masked, while others remain unaltered. more flexible description content, function masking can mitigate this overfitting to some extent, thereby enhancing cross-scenario generalization performance.
5.6 ABLATION DIFFERENT PROPORTIONS OF IRELEVANGE-AUGMENTTI DATA
To further explore the relationship between these two aspects, we conducted an ablation study on the ratio of aelevvance-augmenten data used during training. In this ablation, we sampled total of 10,000 instances with varying data proportions from the augmented dataset to fine-tune the Qwen2- Instruut model and then exam on the BFCL. benchmark, observing the changes in the model's irrelevance detection and function-calling capabilities across different ratios.
Irrelevance vs Ratio
AST vs Ratio
Performance vs Ratio
0.8
0.5
Overall Accuracy
2ST Summary
035
8:54
1::0
8.55
0.4
0.2
Ratio (70
Ratio 1961
Rahio 0%
Figure 6: Ablation on different proportions of irrelevance-uugmentee data applied, e.g. ratio=30% means 30 percent of the training data is sampled from the seeeance-augmenter dataset with other 70 percent sampled from the clam-function caanctionccallnn--66n calling-60k dataset.
the variation in the proportion of irrelevance- augmented data reveals an inverse relationship between the model's performance in irrelevance de- tection and its function-callin the trade-off between these two aspects. Furthermore, the final panel of [Figure 6) indicates that, within our experimental settings, the Hammer model achieves optimal overall performance when data is approximately 10%. This insight guides us in estab-
As illustrated in the first two panels of Figure 6
capabilities. This finding underscores the importance of balancing
the proportion of preeevance-auggeette lishing the target size for the prelevance-augmentee dataset, i.e., 7.5k. is essential to note that this proportion may require adjustment depending on the underlying model and training dataset; thus, the ratios presented herein are intended as a reference only.
CONCLUSION
In conclusion, our exploration of dunction-calling models reveals significant challenges related to performance inconsistency across different benchmarks, primarily driven by misleading from spe- cific naming conventions. By introducing the Hammer family of models, we provide robust so- lution that enhances generalization capabilities through carefully constructed augmented dataset and innovative function masking techniques. The superior performance of Hammer on variety of benchmarks demonstrates its potential for practical application in real-world scenarios.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
The SJTU team is partially supported by the National Key R&D Program of China (2022ZD01 14804), Shanghai Municipal Science and Technology Major Project (202ISHZDZX0102) and National Natural Science Foundation of China (62322603). Muning Wen supported by the Wu Wen Jun Honorary Scholarship, Al Institute, Shanghai Jiao Tong University.
REFERENCES
Ibrahim Abdelaziz, Kinjal Basu, Mayank Agarwal, Sadhana Kumaravel, Matthew Stallone, Rameswar Panda, Yara Rizk, GP Bhargav, Maxwell Crouse, Chulaka Gunasekara, et al. Granite- function calling model: Introducing function calling abilities via multi-task learning of granular tasks. arXiv preprint arXiv:2407.00121, 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.0686 087744 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Kinjal Basu, Ibrahim Abdelaziz, Subhajit Chaudhury, Soham Dan, Maxwell Crouse, Asim Mu- nawar, Sadhana Kumaravel, Vinod Muthusamy, Pavan Kapanipathi, and Luis A. Lastras. Api- blend: A comprehensive corpora for training and benchmarking api 1lms, 2024. URL Stthttp::
//axxiv.org/abs/2240 2402244222.11111
Aili Chen, Xuyang Ge, Ziquan Fu, Yanghua Xiao, and Jiangjie Chen. Travelagent: An assistant for personalized travel planning, 2024a. htRLitttss///axxi/
abs /20p..2499.. 22409.08069
Wei Chen, Zhiyuan Li, and Mingyuan Ma. Octopus: On-device language model for function calling of software apis. arXiv preprint arXiv:2404.01549, 2024b.
KR1442 Chowdhary and KR Chowdhary. Natural language processing. Fundamentals of artificial intelligence, pp. 603 649, 2020.
Lutfi Eren Erdogan, Nicholas Lee, Siddharth Jha, Sehoon Kim, Ryan Tabrizi, Suhong Moon, Cole- man Hooper, Gopala Anumanchipalli, Kurt Keutzer, and Amir Gholami. Tinyagent: Function calling the edge. arXiv preprint arXiv:2409.006/1 2024a.
Lutfi Eren Erdogan, Nicholas Lee, Siddharth Jha, Schoon Kim, Ryan Tabrizi, Suhong Moon, Coleman Hooper, Gopala Anumanchipalli, Kurt Keutzer, and Amir Gholami. Tinyagent: Function calling at the edge. [attps://bai: berkeley edh/blogg2024/05/222
Ciny-agent// 2024b.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retricval-auge auggenetd generation for large language models: A survey. arXiv preprint ar arXiv:2312.1099) 2023.
Carl A Gunter.
Semantics of programming languages: structures and techniques. MIT press, 1992.
Tom Gunter, Zirui Wang, Chong Wang. Ruoming Pang, Andy Narayanan, Aonan Zhang, Bowen Zhang, Chen Chen, Chung-Cheng Chiu, David Qiu, Deepak Gopinath, Dian Ang Yap. Dong Yin, Feng Nan, Floris Weers. Guoli Yin. Haoshuo Huang. Jianyu Wang, Jiarui Lu, John Pee- bles, Ke Ye, Mark Lee, Nan Du, Qibin Chen, Quentin Keunebrock, Sam Wiseman, Syd Evans, Tao Lei, Vivek Rathod, Xiang Kong, Xianzhi Du, Yanghao Li, Yongqiang Wang. Yuan Gao, Zaid Ahmed, Zhaoyang Xu, Zhiyun Lu, Al Rashid, Albin Madappally Jose, Alec Doane, Alfredo Bencomo, Allison Vanderby, Andrew Hansen, Ankur Jain, Anupama Mann Anupama, Areeba Kamal, Bugu Wu, Carolina Brum, Charlie Maalouf, Chinguun Erdenebileg, Chris Dulhanty, Do- minik Moritz, Doug Kang, Eduardo Jimenez, Evan Ladd, Fangping Shi, Felix Bai, Frank Chu, Fred Hohman, Hadas Kotek, Hannah Gillis Coleman, Jane Li, Jeffrey Bigham, Jeffery Cao, Jeff Lai, Jessica Cheung, Jiulong Shan, Joe Zhou, John Li, Jun Qin, Karanjeet Singh. Karla Vega, Kelvin Zou, Laura Heckman, Lauren Gardiner, Margit Bowler, Maria Cordell, Meng Cao, Nicole
11

---

Published as a conference paper at ICLR 2025
Hay, Nilesh Shahdadpuri, Otto Godwin, Pranay Dighe, Pushyami Rachapudi. Ramsey Tantawi, Roman Frigg, Sam Davarnia, Sanskruti Shah, Saptarshi Guha, Sasha Sirovica, Shen Ma, Shuang Ma, Simon Wang, Sulgi Kim, Suma Jayaram, Vaishaal Shankar, Varsha Paidi, Vivek Kumar, Xin Wang, Xin Zheng, Walker Cheng, Yael Shrager, Yang Ye, Yasu Tanaka, Yihao Guo, Yun- song Meng, Zhao Tang Luo, Zhi Ouyang, Alp Aygar, Alvin Wan, Andrew Walkingshaw, Andy Narayanan, Antonie Lin, Arsalan Farooq, Brent Ramerth, Colorado Reed, Chris Bartels, Chris Chaney, David Riazati, Eric Liang Yang, Erin Feldman, Gabriel Hochstrasser, Guillaume Seguin, Irina Belousova, Joris Pelemans, Karen Yang, Keivan Alizadeh Vahid, Liangliang Cao, Mah- yar Najibi, Marco Zuliani, Max Horton, Minsik Cho, Nikhil Bhendawade, Patrick Dong, Piotr Maj. Pulkit Agrawal, Qi Shan, Qichen Fu, Regan Poston, Sam Xu, Shuangning Liu, Sushma Rao, Tashweena Heeramun, Thomas Merth, Uday Rayala, Victor Cui, Vivek Rangarajan Sridhar, Wencong Zhang, Wenqi Zhang, Wentao Wu, Xingyu Zhou, Xinwen Liu, Yang Zhao, Yin Xia, Zhile Ren, and Zhongzheng Ren. Apple intelligence foundation language models, 2024. URL
https://arxiv> org/ wtwp//g///////a///////////aaaaa/ 2407 21075
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming- the rise of code intelligence. arXiv preprint 24X 2024.
Raisa Islam and Owana Marzia Moushi. Gpt-4o: The cutting-edee advancement in multimodal Ilm. Authorea Preprints, 2024.
Schoon Kim, Suhong Moon, Ryan Tabrizi, Nicholas Lee, Michael W. Mahoney, Kurt Keutzer, and Amir Gholami. An LLM compiler for parallel function calling. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.). Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 24370-24399 PMLR, 21-27 Jul 2024. URL www.s/resssev... ww3p///m2//22222// html
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented Ilms, 2023.
URL https: //2l /wtpxvvvvv....///aa/.aaa/aaaaavv org/ abs /t.oo//3333 2304. 08244
Weiwen Liu, Xu Huang, Xingshan Zeng, Xinlong Hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu. Zezhong Wang, Yuxian Wang, Wu Ning. Yutai Hou, Bin Wang, Chuhan Wu, Xinzhi Wang. Yong Liu, Yasheng Wang. Duyu Tang, Dandan Tu, Lifeng Shang, Xin Jiang, Ruiming Tang, Defu Lian, Qun Liu, and Enhong Chen. Toolace: Winning the points of Ilm function calling, 2024a. URL https: / /aw.iv.vvii..a...a//...a.a.aav.. org/ abs/ www.///40.0 240.2 00920
Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan. Weiran Yao. Zhiwei Liu, Yihao Feng, Rithesh Murthy, Liangwei Yang. Silvio Savarese, Juan Carlos Niebles, Huan Wang, Shelby Heinecke, and Caiming Xiong. Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets, 2024b. URL https:///axxi
org/ 2406. 18518
Shishir G. Patil. Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305. 15334, 2023.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ rorl-world apis. arXiv preprint arXiv:2307.16789 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Venkat Krishna Srinivasan, Zhen Dong, Banghua Zhu, Brian Yu, Hanzi Mao, Damon Mosk- Aoyama, Kurt Keutzer, Jiantao Jiao, and Jian Zhang. Nexusraven: A commercially-permis.. language model for function calling, In NewrIPS 2023 Foundation Models for Decision Making 51c.idddd11Ppppppe.e
Workshop, 2023. URL https: / wwtt:/eeeeeetteeeeeeeeeeeeo.
net wtruooormmm id-
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. Toolalpaca: Gen- eralized tool learning for language models with 3000 simulated cases, 2023.
12

---

Published as a conference paper at ICLR 2025
Muning Wen, Ziyu Wan, Weinan Zhang, Jun Wang, and Ying Wen. Reinforeing language agents via policy optimization with action decomposition. arXiv preprint arXiv:2405.15821, 2024.
Mengsong Wu, Tong Zhu, Han Han, Chuanyuan Tan, Xiang Zhang, and Wenliang Chen. Seal- tools: Self-instruct tool learning dataset for agent tuning and detailed benchmark, 2024. URL
https: ptttss://arxv>> org/ abs/ 2205/2405..m 08355
Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, lon Stoica, and Joseph E. Gonzalez. Berkeley function calling leaderboard. https://gorilla. cat wew.ine_eeaeeebbrr..tttt.
2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.16671, 2024.
Shunyu Yao. Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances Neural Information Pro- cessing Systems, 35:20744 0007744-0777,222 2022.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. r-bench: A benchmark for tool-agent-user interaction in real-world domains, 2024. URL https::/ /tt:://arxiv... org/ abs/ 2406. 12045
Jianuu Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, Zhiwei Liu, Yihao Feng, Tulika Awalgaonkar, Rithesh Murthy, Eric Hu, Zeyuan Chen, Ran Xu, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Silvio Savarese, and Caiming Xiong. xlam: A family of large action models to empower ai agent systems, 2024. URL httpt:tt////aa/
org/ abs 2409. 03215
Nan Zhang, Zhenhua Duan, and Cong Tian. A mechanism of function calls msvl. Theoretical Computer Science, 654, 03 2016. doi: 10.1016ff.ees.216..22200
Zhaowei Zhang, Fengshuo Bai, Qizhi Chen, Chengdong Ma, Mingzhi Wang, Haoran Sun, Zilong Zheng, and Yaodong Yang. Amulet: Realignment during test time for personalized preference adaptation of LLMs. In International Conference on Learning Representations (ICLR), 2025.
URL /ttps:://openreview net forum? fd http://wwwww..eeeeen.ee.eeee..e.
13

---

Published as a conference paper at ICLR 2025
A ANALYSIS OF XLAM MODEL PERFORMANCE ON SEAL-TOOLS AND NEXUS RAVEN BENCHMARKS
This section provides a comprehensive analysis of the factors contributing to the performance degra- dation of the xLAM model the Seal-Tools and Nexus Raven benchmarks. Through detailed examination of failure cases, specific areas where the model encountered difficulties have been iden- tified.
Errors were categorized into three primary types: function selection errors, parameter filling error, and rejection errors. (Table 7 compares the number of cases for each error type, contrasting the Hammer-7b with the xLAM-7b-fo-r.
Table 7: Error type analysis for Seal-Tools and Nexus Raven benchmarks (Hammer-7b XLAM-7b- fc-r).
Benchmarks
Function Error
Parameter Error
Rejection Error
Correct
Total
Scal-Tools
14 7
49
63
3 37
228
187
294
294
Nexus Raven
24
65 21
14 186
215
98
318
318
The data reveal that the xLAM model exhibited notably high rate of rejection errors, particularly the Nexus Raven benchmark. This indicates that xLAM may be overly conservative in rejecting candidate functions, resulting in significant number of missed correct predictions.
Considering the diverse range of parameters the Nexus Raven benchmark, we further analyzed how the number of parameters in function labels influenced the model's performance.
Table 8: Impact of parameter numbers on performance in the Nexus Raven benchmark (Hammer- 7b/xLAM-7b-fc--)
Parameter Number
Function Error
Parameter Error
Rejection Error
Correct
Total
6
0
0
9
23
24
1
7
7
0
61
43
69
69
0
10
1
23
26
11
37
37
0 0
4
0
14
22
26
26
5
22
10
8
7
0
39
33
63
63
10
1
18
2 35
14
35
35
27
0 0
11 0
2 43
30
43
43
All
24 13
65
21
14 186
215
98
318
318
Table 8 indicates that the xLAM model's performance deteriorates significantly when dealing with functions that have five or more parameters. Nearly all such cases experienced incorrect rejections, suggesting that the model struggles with complex functions requiring the management multiple parameters.
B ERROR TYPE ANALYSIS
We have conducted categorical statistics for the various failed cases presented in Figure 2 to provide readers with a more detailed and comprehensive understanding. The specific results are shown as
Table 9
In Table 9) "Correct" refers to cases where the prediction is entirely correct; "Func. Error" indicates cases where the function itself was incorrectly selected; "Param. Error" represents cases where the function was correctly chosen, but parameter filling was erroneous; and "Reject Error" denotes cases where the model incorrectly deemed all candidates irrelevant. The table confirms xLAM's overre- liance on function and parameter names, as the frequency various errors increases significantly when function and parameter names masked.
14

---

Published as a conference paper at ICLR 2025
Table 9: Categorical statisties of number of various failed cases, with "cases with HammerJcases with xLAM" in each grid.
Types
No Mask
Fn Mask
Arg Mask
All Mask
Correct
228
187
213
174
213
148
212
133
Func. Error
14
20
13
23
21
Param, Error
49
63
60
66
57
72
51
70
Reject Error
3
37
38
8
70
Total
294
294
294
294
294
294
294
294
C DATA CONTAMINATION CLARIFICATION
All queries/examples in the irrelevance-ate -aggment-- dataset are sampled from the original xlam- canction-calling training set. For each sampled example, we simply remove the correct function from its candidate list and replace its label with an empty list, which indicates that all candidates are irrelevant. Therefore, the construction process of the irrelevance-augne augmented dataset should not intro- duce any additional data contamination beyond what already exists the cam--fucttoonnee calling 60k dataset. Besides, fan-ffuctionncallinne Seal-Tools benchmarks used in our experiments were released in August and May 2024, respec- tively, both after the release of the training set. Therefore, Hammer's 's performance on these two benchmarks should, to some extent, address these concerns.
was released in January 2024, while the BFCL-V2 and
D POTENTIAL FAILURE OF FUNCTION-MASIING
To give readers a better understanding of the limitations of function function-masking, in this section, we have to point out that function-mans might perform worse than traditional approaches in the following two cases.
The first case is shown in the ablation study in Section 5.3 "A large mask ratio impede the model's learning speed within the same task scenario bui facilitate better generalization of the model across different scenarios." This suggests that when the training and testing sets are from the same source or have similar distributions, function masking might perform worse than tradi- tional approaches.
Another situation arises in cases where there is little description available. In such cases, due to the incomplete information in the descriptions, directly masking all names might be aggres- sive, leading to decline in performance. The comparison on the Tool-Alpaca benchmark suggests this possibility since contains function candidates in the test set with very limited descriptions and Hammers achieve performance very close to baselines. If the description is further simplified, function masking might perform worse than traditional approaches as well.
15

---

Published as a conference paper at ICLR 2025
E
EXTRA EXPERIMENTAL RESULTS
Table 10: Full evaluation of the BFCL leaderboard Yan et al. 2024). (as of date 09/20/2024.)
AST Congory
Essc Catgory
Bunk
Model
Overall Ace
simple
Mulliple
Faralel
Pazaiel Mullipic
Simple
Maliile
Pusllel
Pazalil Mallipi:
Inclevance
Relessnce
4-125 Pueriow (Poontt
78.32
36.00
50.00
Piseeri (Prepii)
GPT 4P7-400135 (Pronp))
54.24
78.76
85
K2.67
56.30 20
96,10
70.00
Hamner Th (PC)
83.93
08.31
K2.52
15.89
54.05
91
96.01
54.00
55.10
72.57
02.68
GPP---ur 20-20444244400--444--0000 (a-29 (Prmpt
63.89
60.47
83.51
$8.12
64.25
00.00
56.00
90.00
77.50
51.82
82,93
GPE nisi 3155--n-11 (Prorgg
K1.35
75
81 A4
15.12
75. 41
25
06.01
K7.00
M0-89
(Poorge)
Fanctioary Median--1 (FC)
82.55
74.34
87.59
94.00
75.00
cIr--- 1-r-4-1ice-ll Proviss (PC)
09 53
154.1
U5.
04.10
15.00
15.3
12.10
K1.91
Measi lam----700 --------nitnn Instne (Poorpe)
81 99
77.87
64.10
77.85
14. 14
94.11
54.00
40.10
54.47
93.64
10
Clasde Opes 20140229 (Prnpp))
80.85
76.65
87.47
78.39
75.17
16.57
94.00
52,00
75.
56.15
85.37
GPE-4-0 15-horice (PC)
90.87
83.33
54.05
54.00
14.05
(Prerpi))
Punctionay Smull
80.21
72.70
72.92
50.00
68.36
16
miand miand--anp 2407 (PC Any)
3417
79.65
B1.01
87.42
90.50
83.50
06.20 20
92.00
96.00
17. 13500
0.34
100.00
15
Gre-4--1044-5- (PP
70.35
70.40
R2.33
$9.00
76.05
68.90
$4.00
$5.00
72. 50
73.50
70.23
16
xLAM-Th-s (PC)
71
76.18
74.12
64.50
58.00
84000
72. 13.50
79.76
30.89
GPE-4o-min 3024-07-18
79.25
67.83
90.15
72.07
70.00
(Prupp)
19
CellssOpefaacttiinn (IC)
74.10
70.81
74.47
75-75
66.7
94 96
56.00
74.00
70.00
84.37
20
GFT-4ube 2024 09 (FC)
79.69
64.21
H2.72
$2.50
15.75
58.71
58.00
56,00
12.50
79.79
70.73
11
Fundiiorr Amalless (1T)
78.06
06.30 50
81. 50
$0.12
73.50
18.00
86.00
63. 50
72.13
90.20
Gr--4o-1144----0 (PC)
73.97
90:01
40.00
8400
large
Juno)
78.78
17:50
2300222-0000222 (Pompp)
50.00
15
FinForation-22 (FC)
77.45
74.11
$1.49
73.62
67.38
04 23
18.00
80.00
72.50
52.94
82.80
26
Gnntt 30t Panciint3lligg (PC)
76.63
65.27
73.05
50.75
07.83
55 36336
54,00
13. 13.50
72:43
95.12
27
Mrind Ne (Pompp)
76.31
72.89
R1.37
$1.50
73.25
02. 20.00
94.00
86.00
50.00
13.25
60.80
Clande 3.5.Smmss 3024002 (Prong))
76.29
76.93
90:27
65.33
20:00
70.00
51.22
29
GPT-3. 3.5 01 (PC)
09.79
55.00
57.50
30
Opes Miard Sevo 2407 [FC Julo)
14977
64.37
79.99
$0.25
74.00
91.3 16
$6.00
96.00
62.50
59.14
65.85
31
<l AM- Ihdo-r (FC)
74.00
69.99
7100
54.00
03.03
79 71
58.00
86.00
70.00
$1:21
95.17
32
Homes 2.Pro Liana 3 -08 (PC)
74.28
06.29
73.49
70.25
78.33
50 64
55.00
$4.00
72.50
53.80
50.49
Geniti
(FC)
56.15
63:50
(Phomp)
75.06
$8.00
Genini 1.5 Presien ba09 (PC)
74.56
55.0
79.43
64.75
70.00
18.00
50.00
12.50
36
GPT-4o 2011- 0211-8800 05 (Poopp)
7411
65.76
75.86
72.12
71.67
70.57
18.01
76.00
15.04
59.56
53.65
37
Commard Ria (Poopp (Orgina)
74.11
68.14
75.13
77.50
62.17
01 29
$6.00
75,00
55.00
59.31
75.61
18
Open Miara Namo
(CC Any
7111
67
83. 46
77.39
70.08
56.00
0.73
100.00
Misual Masual-Maiinn
72.19
63.77
90.22
90.0
55.00
57.50
40
Command- R- Pla (PC) (rrglaal)
73.04
64.25
27.45
60.11
80
16.00
$1.00
$3.50
52.74
91
41
Genini 1.5 Flush Peeige 0514 (PC)
70.25
65.90
83.26
53.87
63.50
57.03
56.00
74,00
75.00
74.00
63.41
42
DBRX laane (Prong))
09.55
09.97
90.35
56.89
$1.50
10 50
56.00
60000
62.50
44.86
82.93
Clank-d 5-8 2040000 (FC)
63.38
73.95
83.00
05 16
44.00
40.00
63.41
GPT Tu 0125 (Porpping)
$7.90
(KC
66.15
62.52
74.96
06.71
90100
50000
$7.50
55.16
46
Hemos Po Miena-78 7 (FC)
63.44
60.95
71.49
50.38
50.42
60.5 00.50
50.0
$4.00
63.50
38.55
75
47
Hernes 2 There Jona 3 (KC)
04.33
56.53
67.43
62
54.3 33
1M 14
18/01
75.00
55.10
$2.60
51.23
48
Mead lame inane (Pronpt)
02.20
50.53
70.36
$5.90
51.25
90
64000
50.00
23.88
7805
Clande-3-Ops 20240220 pols 202404404)
61.89
79.05
20.92
$6.00
30.00
(Ponpe)
51
Chade- 200 (Pomp))
09:34
7464
15.46
$1.08
45.17
19
94.00
51.00
13 50
18.00
85.17
52
Opes Miand 9x22 (FC Any
58.89
73.23
85.42
10.75
63.08
92.57
92.00
24000
47.50
0.34
100.00
55
Mec--Miind-85220 (CC Audo
54:37
56.75
83.35
10.50
K3.35
77 79
92.00
2400
45.00
46:20
85.37
Gemm- 16-Pm-500 (PC)
51:11
64.90
79.40
5.00
3012 Juno)
51.90
$2.00
57
20.00
Kiatml-maall-240
36.42
50.00
57
FinForation-1 (FC)
49.11
72.25
90.37
0.00
0.50
64.30
50.00
0.00
0.00
58.55
95.12
5%
Cluade--Sorne 2054022- (PC mede- 204-000)
47:97
63.79
75.37
8.35
3.35
73
90
0.00
0150
39
57.55
59
Cladk--nttss 2 2.hoorpt)
47.55
$4.50
$5.91
37.75
32.42
57
72.00
39.00
15.00
70.21
46.34
Clasie 3-Hak- 20240307 2023-3024-0000
47003
72.74
78.95
2.33
92.00
0.00
Sooddakeiasee fikie
(Poapj)
43.45
3497
31.79
38.33
25.00
40.00
53
motrd lagg 2407 (Protit)
27.87
19,05
43.11
33.38
23.17
5.71
30.00
18.00
5.00
40.70
58.44
54
Miota--Small-290t (Prop)))
24.44
7.55
38.07
17.28
K92
4.14
12.00
12.00
8.0
65.2
56.10
65
Mistal 2312 (Proupt)
21.17
21.11
25.92
9755
3.50
19.6
3.00
12.00
0.00
92.23
19.51
Despecskoi..5 (Phomp)
11:18
4.7
3000
183
0:00
0.00
0.00
Hemee Thhr Lino 3-706
10.00
0.00
3000
0.00
9.00
0.00
0000
16

---

Published as a conference paper at ICLR 2025
Table 11: Full evaluation of different models on several academic benchmarks. The rank is based on the average F1 score on "Func. Args".
Fl Func-Name
[F Func. Args
Fl Average
Model
API-Bank L-1
API-Bank L-2
Tool-Alpaca
Scal-Tools (Singlo-Tool)
Necus Raven
Func Nome
Func.+ Args
GPT-4-0613 (Prompt)
92,93
84.78
69.60
56.98
58.64
66.67
94,56
93.95
95.73
91.60
88.29
78.79
(Prompt)
Hammer-7B (FC)
93.48
85.79
82.91
66.40
82.31
59.86
97.44
91.66
92.46
77.35
89.72
76.21
Granite-208-Fuunn CaO-l--FuctionCCalll
Hammer-48 (PC)
91.65
81.46
77.39
61.01
85.09
55.96
96.42
99:.45
81.73
64.89
86.50
71.35
XLAM-7B-ft (TC)
90.05
80.69
72.49
64,24
67.26
58.96
78.97
76.87
54,09
57.50
72.57
67.65
Cenilll-OenFunnctionse (FC)
69.21
70.34
48.82
54.69
72.93
51.26
93.20
91.11
72.84
68.41
71.40
67.16
1.3--f (FC)
94,86
83,700
91.80
64.32
64,86
50.58
90.74
80.45
64.43
54.80
81.34
66.77
Qwen2-7B- 1e22t---nnttrectteo (Prompt)
81.55
60.62
95.65
49.50
71.59
48.11
93.88
77.51
87.05
63.47
85,94
59.84
9-9---..--I (Prompt)
74.63
82.08
Qwenl.5-4B Chat (Prompt)
55.33
59.78
46.74
38.48
35.41
16.98
48.44
62.32
29.03
33.770
42.99
42.25
Table 12: AST Evaluation for Hammers and different base models on BFCL.
Ovenal
Madel
Sunmary
Simple
Panllel
Panaliel Multipl:
Irrelevance
Relevanee
72.79
Qwen2-7B-Inee
69.47
68.75
$1.88
60.75
66.50
61.31
97.56
76.36
Quas2-7b-stAM
72.67
Hammer 7B
78.70
69.31
$2.52
78.88
$4.08
72.87
92.68
32.92
Qwent 5-4B-CCha
25.43
24.60
32.09
22.12
22.00
66.56
29.27
71.13
Qwenl 5-45-LAM
67.75
63.88
77.84
62.62
66.67
58.20
97.56
Qwen2-1 Instraci
41.44
50.77
19.38
33.83
22.91
69.42
Ouem2-1.Sb-JLAA St-2.AM
63.39
63.30
73.84
$7.25
67.17
62.27
92
71.16
[aamer 1,50
65.52
62.34
72.84
58.75
68.17
72.18
92.68
75.22
XLAM-7B-4c (FC)
72.77
70.25
78.18
74.12
68.50
79.76
30.49
17.65
Deepseek Coder
3.53
0.25
2.58
99.51
0.00
Despace
Despael-Colai Bat.ls--B-tnttts-HHaaaaa
77.00
20.96
sLAM-11 3B8-2 (PC)
67.37
64.49
73.06
64.00
67.92
61.21
95.12
16.81
Deepinek- 3R-Iasmmer
0.21
0.83
0.00
0.00
0.00
100.00
0.00
Despeesk-Codrr 38--133nec--
Table 13: Full evaluation of the ablation study on different base models and benchmarks.
Academic Benehmarks (FI Fune-Name FI Func. Args)
Fl Average
API-Bank
API-Bank L-2
Seal-Tools (Single-Too
Nexus Raven
Func Name
Func.e
Model
Tool-Alpaca
Args
Qwen2.7B-
81.55 60.62
95.65
49.50
71.59
48.
93.88
77.51
87.05
63.47
85.94
59.84 84
Qwen2-7b-xL.n
90.98
74.48
Hammer-7B
93.48 85.79
82.91
66.40
82.31
59.86
97.44
91.66
92.46
77.35
89.72
76.21
Qwen Dwwel..54B--C 5--BB-hhtt
59.78
46.74
35.41
16.98
48.44
62.32
29.05
33.70
42.99
42.25
Sweul_5-Ib-xLAM
86.68
78.64
73.17
58.32
84.18
55.48
93.19
88.28
81.28
64.10
83.70
68.96
Hammerr-4B
91665 81.46
77.59
61.01
85.09
$6.96
96.42
92.45
81.73
64.89
86.50
71.35
Qwen2. 1,5B instract
74.63 63.55
57.69
33.62
65.76
45.25
82,08
75.49
70.62
45. 55,46
70.16
$2.67
1.h22-1.5bxxLAM
81.61 770.93
80.11
59 59.84
81.84
53.83
92.77
84. 10
76.58
54.90
82.58
64.72
72.30
59.71
88.65
79.87
56.88
XLAM-7B-Ie (FC)
90.05
80.69
72.49
64.24
67.26
$8.96
78.97
76.87
54.09
57.50
72.57
67.65
Deepseek Coder: Instruct
$1.42
56.70
35.51
39.64
11.58
20.08
50.00)
65.47
26.89
46.47
35.08
45.67
Deepseek- Coder-7B.Hammnn Ha.Hare
83.47
75..18
69.17
60.04
83.77
62.95
96.95
93.20
93.75
83.35
85.42
74.94
XLAM-LL3B-fc (PC)
83,70
64.32
64.86
50.58
90.74
80.43
64,43
54.80
81.34
66.77
Deepscel-Coder-1 3.3t-utttuuct
35.23
38.42
20.41
24.75
10.68
06.06
16.46
21.52
4.34
8.15
17.42
19.78
Deepseck, Coder
65.97
$7.68
64.68
17

---

Published as a conference paper at ICLR 2025
F DIFFERENT TYPES OF FUNCTION-CALLING TASKS
Simple:
This query style includes straightforward scenarios where a single function call is made based on the user's input with single provided JSON format API description.
Multiple: In this style, user queries could be answered by one of several function calls. The chal- lenge lies in selecting the most appropriate function from multiple provided APIs. It represents one of the most common real-world use cases.
Parallel: This query style requires executing multiple function calls simultaneously in response to a single user query, which may consist of one or more sentences but with only one API provided.
Parallel Multiple:
This query style combines the parallel and multiple categories, where multiple function and API documents are provided, and each function call might be invoked multiple times based on the query's 's requirements.
Irrelevance: In this query style, no suitable function exists within the candidate options to fulfill users' intent, thus the model should have the ability detect it and decline the task, rather than making incorrect attempts.
Simple
Multiple
Parallel
Parallel Multiple
Irrelevance
Query What 1 2?
Query
Query
Query
Query
What is 1 2 Candidates
What (1 2] and 13 4p?
What is (1 2) and (3 x42
Wh 2
Candidates
Candidates:
Candidates:
Jadclinn a, int bl]
[add[int int
[addjint a, int b))
[add]inta, int b),
[mult( a, int b]]
mult/int a,
multjint a, int
Currect Answer [add(a=1] b-2]]
Currect Answer
Currect Answer
Currect Answer
Currect Answer
[add[aa=]] b-2)
[add/a=1, b-2), add(an3, b-4[]
[add(a=1, b-21, mult(a-3, b-4]]
Figure 7: Demonstration of different function- calling styles.
G IMPACT OF IRRELEVANCE-AUGMENTTI DATA ON MODEL PERFORMANCE
We conducted a thorough analysis to determine the number of failed cases resulting from irrelevant function calls, particularly before and after incorporating the eeeevance-uuggentne dataset into the training regimen for the Hammer model. The analysis provides insights into the model's ability to accurately detect irrelevant function calls.
Table 14: Performance Hammer-P5 7B "with without" irrelevance-augment dataset, where GT
stands for "ground truth".
GT Irrelvann
GT Relevant
Model Predict Irrelevant Model Predict Relevant
145|16
35
370
2450/ 2480
Table 14 clearly demonstrate the significant improvement in irrelevance detection when the Hammer model trained with the irelevance-augmennte dataset. Without this augmentation, the model's performance in detecting irrelevant function calls is poor, with almost all ground truth irrelevant cases being predicted as relevant. This analysis underscores the urgency and importance incor- porating irrlevance-augmeenee augmented data into the training process, which aligns with our motivation for creating such a dataset and highlights its impact on model performance.
H ON-DEVICE FUNCTION CALLING PERFORMANCE AND EFFICIENCY
Our research focuses on the development of compact models (no larger than 7B) that achieve a balance between robustness and performance, with specific emphasis maintaining accuracy
18

---

Published as a conference paper at ICLR 2025
in function calling critical factor for effective task execution mobile devices. This section provides a comprehensive analysis of the efficiency and performance of our model in on-device function calling scenarios, aimed at meeting the requirements for lightweight, robust models suit- able for mobile applications, such as personal assistants. (Table 15 presents a detailed evaluation of non-functional metrics and hardware configurations for Hammer-7B model, after has been quantized using the Q4 K.M method.
Table 15: Performance metrics of the Hammer- 7B model mobile devices after quantization.
Model
Quantizaaion Method
Precidon
Prefil Speed
Decode Speed
Backend
RAM Usage
Pracesor
Mobile Model
Ql.C.
Snapdragon
Quantization using the Q4.K M method results in approximately 2% performance degradation, which remains within an acceptable range. preserving the model's suitability for mobile deployment. These metries demonstrate that the Hammer-7B model can operate efficiently on mobile devices. However, its inference speed still has significant room for improvement due to the current lack of universal and efficient edge-side inference frameworks. Nevertheless, we are confident that our community will address this issue in the near future.
EXAMPLE INPUT TO MODELS WITH FUNCTION MASKING
The prompted inputs to models in our experiment are exampled as:
[BEGIN OF TASK INSTRUCTION]
You are a tool calling assistant. In order to complete the user's request, you need to select one or more appropriate tools from the following tools and fill in the correct values for the tool parameters. Your specific tasks are:
1. Make one or more function/tool calls to meet the request based the question.
2. If none of the function can be used, point it out and refuse to answer.
3. If the given questiion lacks the parameters requi red by the function, also point it out.
[END OF TASK INSTRUCTION]
[BEGIN OF AVAILABLE TOOLS]
"name": "Lx0m64zīyg";
"descripe incerrpption:. "Gets hourly weather forecast information for given geographical coordinates using the RapidAPI service.", " "parameters":
"TDpjPd": [
"description": "The latitude of the geographical location.", "type\", \: "int",
"default": 46.95828
},
"78th2U31Fj";
"description": "The longitude of the geographical location.", "type": "int", "default": 10.87152
"name": "WoDdNSe7e7K5",
" "description". "Fetches weather updates for given city
19

---

Published as a conference paper at ICLR 2025
using the RapidAPI Weather API.", "parameters": "LzžsvxUc"; {
"description": "The name of the city for which to retrieve weather information.", "type": "str", "default"; "London"
"name": "CBrCNmwOERb",
"description": "Fetches the hourly weather forecast for a given location using the RapidAPI service.", "parameters";
"TDEJ. SiopEJ.ZWMt.. {
"description": "The name of the location for which to retrieve the hourly weather forecast.", "type": "str", "default"; "Berlin"
"name": "1YTQVXKwLY",
"description": "Returns an air quality forecast for a given location.".
"parameters": {
"2bkgDA": {
"description": "The latitude of the location for which the air quality forecast is retrieved.", "type": "int", "default"; "35.779"
Sogi.Rez16"; {
"description": "The longitude of the location for which the air quality forecast is to retrieved.", "type": "int", "default"; "-78.638"
"hF.1": {
"description": "The number of hours for which the forecast is to be retrieved (default is 72) ", "type": "int", "default"; "72"
[END OF AVAILABLE TOCLS]
[BEGIN OF FORMAT INSTRUCTION]
The output MUST strictly adhere to the following JSON format, and other text MUST be included. AnThe example format is follows. Please make sure the parameter type is correct. If no function call is needed, please directly output an empty list '[]'
20

---

Published as a conference paper at ICLR 2025
"name": "funo_ namel", "arguments": { "argument:1 ": "valuel", " "argument?2; "value2"
}
(more tool calls as required)
[END OF FORMAT INSTRUCTION]
[BEGIN OF QUERY]
What are the current weather conditions in Sydney?
[END OF QUERY]
The corresponding output generated by models should be:
"name": "WoDdNse7e7k5", "arguments": {
"LzZsvxIC"; "Sydney"
21


Reviews:
[{"review_id": "c95NoiMut7", "reviewer": "Reviewer_peJz", "summary": "This study introduces Hammer, a novel family of foundation models designed to enhance function-calling capabilities in large language models (LLMs). The authors developed a specialized dataset comprising 7,500 instances to improve the models' sensitivity to irrelevant functions. By shifting the focus from function names and parameters to their descriptions, the Hammer models aim to reduce the likelihood of misinterpretation, thereby enhancing generalization across diverse benchmarks. Experimental results indicate that Hammer models achieve superior performance across various benchmarks.", "questions": "1. Could the authors provide a more detailed analysis of function name issues? Specifically, how many failed cases are due to function or parameter name errors? Figure 2 does not clearly convey this information. Authors may consider an error type analysis with detailed tables isolating the number of failure cases due to function name/ arguments name/ other error types.\n2. It would be beneficial for the authors to analyze how many failed cases result from irrelevant function calls. The urgency of this issue is not apparent in the current version. \n3. Based on the results in Table 5, it is difficult to verify the effectiveness of the proposed pipeline compared to xLAM variants. Suggestions could refer to weakness item 3.\n4. While Table 1 shows that Hammer achieves impressive results on AST evaluations, the comparitive performance boosting across other benchmarks in Table 2 does not show the same level of improvement. Is there any analysis to explain this discrepancy? Authors could provide discuss potential reasons for the discrepancy and propose additional experiments or analyses that could help explain the varying levels of improvement observed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The work effectively identifies the issue of naming variability in function-calling LLMs and incorporates training methods to address it.\n2. Hammer models utilize an augmented dataset to handle null function-calling cases.\n3. The study conducts comprehensive evaluations, demonstrating that Hammer consistently performs well across diverse benchmarks, establishing it as a reliable open-source model for function calling.", "weaknesses": "1. The proposed methods may inherit potential biases inherent in LLMs. The sampled \"Masked Data\" can degrade in performance when encountering function names or parameters that differ significantly from those seen during training. This dependency may limit its effectiveness in applications where naming conventions vary widely.\n2. There are potential overfitting concerns. The authors do not provide a thorough data contamination analysis for the newly created dataset.\n3. Whether the performance boosting comes from the effectiveness of the proposed pipeline or the Qwen model itself remains questionable. According to the results of ``API-Bank’’ columns in Table 5, when using the same base model, Deepseek-Coder-7B, it is not as effective as the xLAM variant. Authors could have Qwen models trained with xLAM datasets to further elaborate it.\n4. The presentation of results is not well-organized. Comprehensive tables are difficult to follow and analyze, making it challenging to verify the effectiveness of each component. Authors could break down the long tables into smaller and more focused  ones. Also, for figure 3, author could have the examples shown in function masking to be larger and more compact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nOx8FbHMUI", "reviewer": "Reviewer_zD7o", "summary": "This paper introduces Hammer, a family of lightweight models for function-calling tasks that addresses the problem of inconsistent performance across different benchmarks. The authors propose a function masking technique during training and an irrelevance-augmented dataset to improve model robustness and generalization. Experiments show that Hammer achieves competitive performance compared to larger models like GPT-4 on various benchmarks.", "questions": "1. Have you tried replacing function names with semantically similar ones instead of random strings? This could be more meaningful since function names are important features in practice and descriptions and parameters may sometimes be missing. In some usage scenarios LLMs need to infer functionality from names alone.\n2. Could you provide more insights into other factors contributing to Hammer’s performance? Have you tried using DeepSeek models as base models (just like xLAM) instead of just Qwen? This would be helpful to understand what specific aspects of the base model selection impacted results. Also, it would be beneficial to know if there are other training tricks beyond data augmentation.\n3. For Table 1’s inconsistent performance across benchmarks, I am curious why does xLAM perform well on three benchmarks but poorly on two others? Could the authors provide some error pattern analysis and related insights?\n\nMinor things: the observations in the paper might be related to the code data contamination problem as well, where the memorization of func names / orders could impact the generalization ability. Ref: https://arxiv.org/pdf/2402.05980", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The function masking technique is an innovative solution to reduce model dependency on naming conventions. The empirical results show this approach helps achieve more consistent performance across different benchmarks.\n2. The authors conduct extensive experiments across multiple benchmarks and provide detailed ablation studies on masking ratios and irrelevance detection. The evaluation is thorough and well-documented with clear performance metrics.\n3. The work addresses a real-world problem in function-calling models and provides a lightweight solution suitable for on-device deployment. The improved generalization capability has significant practical value.", "weaknesses": "1. The authors replace function names with random strings, but don’t explore using semantically similar names. This approach may be overly aggressive since function names often contain valuable semantic information that could be preserved while still improving generalization\n2. While the paper demonstrates Hammer’s superior performance, it lacks detailed analysis of other potential contributing factors beyond the masking technique. The choice of base model and other training details could significantly impact the results\n3. More detailed error patterns analysis would be beneficial to understand the model and baseline failure patterns across different benchmarks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wKQ9IVPxyf", "reviewer": "Reviewer_nBzQ", "summary": "In this paper, the authors studied large language models' function calling and tool use abilities. The authors not only identified a critical gap in existing function-calling models, where the model is often misled by specific naming of functions and arguments. \nTo address the issue, the authors present a novel tunning framework that masks the function name and arguments; this results in a family of various new open-source function calling models called Hammer and also a new specialized dataset for irrelevance detection. These models achieves state of the art performance on open benchmark and leaderboards such as Berkeley function calling (BFCL).", "questions": "Please see the weakness part.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The authors identified a very important issue of the current existing open source function calling models such as xlam.\n- Based on the issue, the method that the authors proposed makes a lot of sense and indeed achieves very good performance.\n- The model and dataset will be open source later, which would benefit the community a lot.", "weaknesses": "- The method is specifically designed for function calling problems; it is unclear whether some other domains will also benefit the tunning framework;\n- I am not sure how this strategy is specifically related to on-device language models; It can also be applied to larger models, and it would be really great to see the performance of larger models such as 70B llama if they are applied with this method; \n- For the cases where there is little description, I am not sure if we can still leverage the masking strategy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7czGfeZILW", "reviewer": "Reviewer_95n3", "summary": "This paper introduces Hammer, a novel family of language models specifically designed for function-calling tasks. The authors identify a critical limitation in existing models: their performance varies significantly across benchmarks due to being misled by function and parameter naming conventions. To address this, they propose a function masking technique during training and augment the training data with irrelevant detection examples. Their empirical results show that Hammer models achieve state-of-the-art performance across multiple benchmarks, particularly impressive for their parameter count, with the 7B parameter version competing with much larger models including GPT-4 on certain metrics.", "questions": "How does the function masking technique affect the model's zero-shot generalization to entirely new API schemas or documentation formats not seen during training?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The problem identification is well-motivated and articulated.\n2. The evaluations are comprehensive, covering multiple benchmarks and including detailed ablation studies on both masking ratios and irrelevance detection data proportions.", "weaknesses": "1. The paper doesn't include the comparison with some recent relevant baselines (e.g., ToolBench, API-Bank).\n2. There are no ablation studies on the effect of different description formats/lengths on model performance.\n3. There’s no discussion of potential failure modes or edge cases where function masking might perform worse than traditional approaches.\n4. The random string generation process for masking isn't fully specified - different approaches to generating random strings could affect reproducibility.\n5. The exploration of the trade-off between masking and maintaining semantic meaning in function names is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qh5MDK7RsY", "reviewer": "Reviewer_SUbn", "summary": "The paper addresses the problem of inconsistent performance across benchmarks in current function-calling models, which often arises from different function and parameter names. The authors developed an augmented dataset and proposed a novel function masking method that guide the model to focus more on function descriptions, instead of names, thereby enhancing its robustness and generalization capabilities across diverse benchmarks.", "questions": "see weakness", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper identifies a significant challenge in current function calling models that is both practical and impactful, particularly as the function calling models are increasingly deployed in real-world applications where robustness and accuracy are essential.\n\n2. The proposed method is simple but effective, which addresses the problem thoroughly. The method guides the model to focus on function descriptions rather than names. Also, the method is potentially replicable and offers inspiration for ai safety practitioners in other fields who seek to improve model robustness.\n\n3. The evaluations are thorough and effectively demonstrate the method’s applicability to real-world use cases. \n\n4. The authors consider irrelevance detection, which is essential in practical applicaitons, as it reduces the risk of incorrect or unnecessary function calls and enhances reliability.", "weaknesses": "The paper claims that the model is designed for on-device function calling; however, it lacks sufficient explanations or evaluation results to demonstrate its efficiency or performance in such scenarios. The authors did not discuss the hardware configurations required to run the model on edge devices or address potential resource constraints. Additionally, the authors did not provide evaluation results regarding inference time, which are essential for assessing the model’s practicality in real-world, resource-limited environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)