HammerBench: Fine-Grained Function-Calling Evaluation in Real Mobile Assistant Scenarios
Authors: Jun Wang, Jiamu Zhou, Muning Wen, Xiaoyun Mo, Haoyu Zhang, Qiqiang Lin, jincheng, Xihuai Wang, Weinan Zhang, Qiuying Peng, Jun Wang
Accepted at: acl25

HammerBench: Fine-Grained Function-Calling Evaluation in Real Mobile Assistant Scenarios

Jun Wang*,1    Jiamu Zhou*,1    Xihuai Wang2    Xiaoyun Mo1
Haoyu Zhang1    Qiqiang Lin1    Cheng Jin1    Muning Wen2
Weinan Zhang2    Qiuying Peng*,1    Jun Wang*,1
1OPPO Research Institute, 2Shanghai Jiao Tong University
{wangjun15,zhoujiamu,moxiaoyun,zhanghaoyu1,linqiqiang1,
jincheng,pengqiuying,wangjun7}@oppo.com
{leoxhwang,muningwen,wnzhang}@sjtu.edu.cn

Abstract

Evaluating the performance of LLMs in multi-turn human-agent interactions presents significant challenges, particularly due to the complexity and variability of user behavior. In this paper, we introduce HammerBench, a novel benchmark framework for assessing LLMs’ function-calling capabilities in real-world, multi-turn dialogues. HammerBench simulates diverse mobile assistant use cases, incorporating imperfect instructions, dynamic question-answer trajectories, intent and argument shifts, and the indirect use of external information through pronouns. To construct this benchmark, we curate a comprehensive dataset derived from popular mobile app functionalities and anonymized user logs, complemented by a cost-effective data generation pipeline leveraging open-source models. HammerBench is further augmented with fine-grained interaction snapshots and metrics, enabling detailed evaluation of function-calling performance across individual conversational turns. We demonstrate the effectiveness of HammerBench by evaluating several leading LLMs and uncovering key performance trends. Our experiments reveal that different types of parameter name errors are a significant source of failure across different interaction scenarios, highlighting critical areas for further improvement in LLM robustness for mobile assistant applications1.

1 Introduction

The mobile assistant built on large language models (LLMs), where users interact with agents to provide indispensable context for accurate API calling (Lin et al., 2024), often needs to handle multiple rounds of user interaction to complete complex task requests from users (Yan et al., 2024). For instance, tasks like ticket purchasing typically demand multiple interactions to gather and provide complete context and information for accurate decisions. Such multi-turn interactions pose challenges for LLM-based agents in understanding user intent and making correct function calls due to the uncertainty of user intent and behavior.

To help detect the defects of LLM agents in personal assistant scenarios, many automated dataset evaluation methods driven by LLMs have been developed by the community including multi-turn dialogue (Ou et al., 2023; Bai et al., 2024) and function call evaluation(Yan et al., 2024; Wang et al., 2024b). These function call benchmarks are typically built based on virtual environments to support quantifiable automated evaluation. However, in complex multi-turn interactions, conversations and function calls are often closely intertwined (Wang et al., 2023a), and current benchmarks struggle to evaluate this complex relationship. And we still lack a fine-grained benchmark built in real-world scenarios data to identify the practical application flaws of the agent.

Based on the analysis of logs from anonymized real users, we found that in the real-world scenario, users may frequently change their intent during interactions, requiring the agent to accurately recognize these changes for successful task completion. Current benchmarks are unable to capture this process dynamic and lack the granularity needed to detect process defects or subtle issues in task execution. Additionally, in different scenarios, the references to external information and the differences in the information provided by users vary greatly, so the evaluation needs to cover a broader range of scenarios to help detect these differences. Therefore, We argue that a comprehensive evaluation of mobile personal assistant agent has three key principles:

Authenticity of test data for capturing capabilities in satisfying the needs of real users.
<table>
  <tr>
    <th rowspan="2">Benchmark</th>
    <th rowspan="2">Construction Method</th>
    <th colspan="4">Single-turn</th>
    <th colspan="3">Multi-turn</th>
    <th rowspan="2">External Individual Information</th>
  </tr>
  <tr>
    <th>Perfect</th>
    <th>Imperfect</th>
    <th>Irrelevant</th>
    <th>Intent Shifts</th>
    <th>Diverse Q&A Trajectories</th>
    <th>Argument Shifts</th>
  </tr>
  <tr>
    <td>API-Bank</td>
    <td>Human</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>BFCL</td>
    <td>LLM+Human</td>
    <td>✓</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
    <td>✓</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>NoisyToolBench</td>
    <td>Human</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>ToolSandBox</td>
    <td>Human</td>
    <td>✓</td>
    <td>x</td>
    <td>✓</td>
    <td>✓</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>HammerBench (Ours)</td>
    <td>LLM+Human</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
</table>

Table 1: Comparison between existing benchmarks and ours. Table 2 provides detailed descriptions for each column.

Diversity of scenarios for capturing more situations that users may encounter in daily use.

Granularity of evaluation for capturing potential process defects.

In this paper, we introduce HammerBench, a fine-grained, multi-turn function-calling benchmark. To better reflect real-world user experiences, we curated the dataset based on popular mobile app functionalities from major app stores and queries derived from anonymized user logs. We expanded the dataset with open-source models, creating a cost-effective data generation pipeline. Table 1 shows HammerBench’s superior diversity and coverage across various interaction scenarios. HammerBench supports a wider range of complexities in both single-turn and multi-turn dialogues, addressing key aspects often overlooked in previous benchmarks, such as handling imperfect responses, supporting diverse Q&A trajectories, and dealing with indirect references or vague descriptions. HammerBench further introduces fine-grained metrics to evaluate function-calling performance across individual conversational snapshots. These included metrics such as Function Name Accuracy, Parameter Name Hallucination and Missing Rate, Progress Rate and Success Rate, offering a detailed assessment of the model’s ability to handle function calls and track progress throughout multi-turn interactions.

To validate the effectiveness of HammerBench, we benchmark 10 LLMs with HammerBench and analyze their performance. Our experiments demonstrate that HammerBench can perform more fine-grained evaluations in richer scenarios to assess the agent’s performance in a broader range of real-world tasks. Specifically, HammerBench reveals significant challenges in multi-turn interactions, particularly in tracking argument shifts and handling external information, which affect the accuracy and consistency of function-calling performance across models. Furthermore, HammerBench’s fine-grained metrics, such as Progress Rate and Parameter Hallucination Rate, effectively highlight specific model issues, like hallucinations and parameter name errors, providing deeper insights into model performance across different contexts. These results demonstrate that HammerBench can provide a comprehensive and in-depth evaluation for LLM agents in mobile assistant scenarios and offer valuable insights for optimizing robust function-calling LLMs.

2 Related Works

2.1 LLM Agent Evaluation

Evaluating LLM Agents is challenging due to the open-ended nature of human conversation, making traditional rule-based evaluations difficult. Early dialogue system evaluations, such as topic-based evaluations (Guo et al., 2017), used topic classifiers to assess dialogue quality by evaluating sub-topics individually. With the rise of large language models (LLMs), newer methods leverage these models as evaluators. For example, (Zheng et al., 2023) discussed the advantages of using large models for dialogue assessment. Recent frameworks like MT-Bench (Bai et al., 2024) and MT-Eval (Kwan et al., 2024) apply GPT-based models to evaluate multi-turn dialogues by decomposing them into sub-tasks. MT-Bench evaluates dimensions such as Context Memory, Anaphora Resolution, and Reasoning, while MT-Eval focuses on Recollection, Expansion, Refinement, and Follow-up. Some benchmarks evaluate diverse capabilities of LLM Agents. For example, AppBench (Wang et al., 2024a) evaluates the planning capabilities of LLMs by incorporating combinations of single/multiple apps or APIs with multi-step tool calls across applications. Mobile-bench (Deng et al., 2024) considers the performance of LLM-based mobile agents in multi-APP interactions. Visualwebarena (Koh et al., 2024) evaluates the planning and autonomous exploration abilities of multimodal web agents. Webcanvas (Pan et al., 2024) serves as a dynamic benchmark capable of adapting to the ever-changing land-
scape of web content and UI structures. Similar to the evaluation above, in function-calling scenarios, evaluating dialogue dynamics—rather than just the function call itself—adds a layer of complexity, requiring a nuanced approach to assessment.

2.2 Function Calling Benchmark

Function-calling benchmarks can be broadly categorized into single-turn and multi-turn evaluations. Single-turn benchmarks like ToolAlpaca (Tang et al., 2023), ToolLLM (Qin et al., 2023), and NexusRaven (team, 2023) use automatic data generation based on LLM world knowledge. In contrast, benchmarks such as ToolBench (Xu et al., 2023) and RestGPT (Song et al., 2023) rely on manually labeled data to ensure quality. Some benchmarks, including BFCL-V1 and SealTool (Wu et al., 2024), use the self-instruct method (Wang et al., 2023b) to generate data, while BFCL-V2 refines datasets by cleaning online user logs. Despite these advancements, single-turn benchmarks struggle to capture the diversity of real-world user interactions and fail to fully assess function-calling capabilities.

Multi-turn function-calling benchmarks, such as API-Bank (Li et al., 2023) and NoisyToolBench (Wang et al., 2024b), are manually curated, ensuring high-quality dialogue data. MINT (Wang et al., 2023a) focuses on interactive tools and external tools in question answering, while BFCL-V3 and ToolSandBox (Yan et al., 2024; Lu et al., 2024) offer comprehensive multi-turn evaluations. However, gaps remain in these benchmarks, especially regarding argument shifts.

In the broader landscape of agent evaluations, multi-turn frameworks like AgentBoard (Ma et al., 2024) and Agent as Judge (Zhuge et al., 2024) provide general evaluation systems for agent capabilities. However, these systems do not specifically target the challenges of multi-turn function-calling scenarios. To address the limitations of existing benchmarks, we propose a more targeted evaluation system that focuses on the fine-grained aspects of multi-turn function calling, including argument shifts and external information handling.

3 Principles of HammerBench

Based on the analysis of anonymized user logs and existing benchmarks, we identify three key principles for designing an ideal benchmark: authenticity, diversity, and granularity. Below, we discuss how HammerBench incorporates these principles.

Authenticity of test data: Queries should reflect real user behavior. However, user logs often exhibit a long-tail distribution, where rare but impactful scenarios may be underrepresented. To ensure comprehensive evaluation, we supplement the dataset with additional instances from these rare scenarios. Moreover, traditional API designs prioritize developer-centric parameters, which may not be intuitive for end users. HammerBench prioritizes user-friendly tools, abstracting API complexities to maintain operational integrity while providing a seamless experience. The benchmark dataset is curated from real Apps’ functionalities and anonymized user logs, capturing a wide range of common and rare user intents.

Diversity of scenarios: A diverse range of tools, queries, and user behaviors is essential for assessing LLM agents’ function-calling abilities. HammerBench tests agents across multiple domains through a diverse range of tools, from ticket buying to daily schedules. It also includes various query types, from simple requests to complex tasks, and captures diverse user behaviors, such as detailed vs. vague queries and single-turn vs. multi-turn interactions. This diversity enables the benchmark to evaluate how well the agent adapts to different contexts and user expectations. In short, our benchmark encompasses four categories in multi-turn interactive tasks: 1) Imperfect instructions (e.g., unclear or vague queries), 2) Varied question-answer trajectories (e.g., differing conversational paths), 3) Intent and argument shifts (e.g., changing goals or inputs), and 4) External individual information (e.g., implicit user context or background).

Granularity of evaluation: Granularity refers to the level of detail in the evaluation metrics used to assess system performance. HammerBench adopts fine-grained metrics to evaluate both individual function-calling tasks and multi-turn interactions. Key metrics include Function Name Accuracy, Parameter Name Hallucination and Missing Rate, and Argument Accuracy. These metrics enable a detailed assessment of specific performance aspects, such as correct function invocation and handling of incomplete information. In addition, HammerBench tracks the Progress Rate, which measures how well the model progresses toward task completion, and the Success Rate, which evaluates the final outcome. This granular approach allows us to identify subtle deficiencies in model behavior, such as mismanagement of dependencies or errors in reasoning, providing valuable insights
<table>
  <tr>
    <th>Evaluation Type</th>
    <th>Data Type</th>
    <th>Abbr.</th>
    <th>Description</th>
  </tr>
  <tr>
    <td rowspan="4">Single-turn</td>
    <td>Single-turn perfect instruction</td>
    <td>Perfect</td>
    <td>The user query that clearly gives all required parameter values.</td>
  </tr>
  <tr>
    <td>Single-turn imperfect instruction</td>
    <td>Imperfect</td>
    <td>The user query that only gives few required parameter values.</td>
  </tr>
  <tr>
    <td>Single-turn with pronouns</td>
    <td>External</td>
    <td>There are anaphoric pronouns that refer to external individual information in the user query.</td>
  </tr>
  <tr>
    <td>Single-turn irrelevant</td>
    <td>Irrelevant</td>
    <td>There is no tool in the candidate tools list that can solve the user query.</td>
  </tr>
  <tr>
    <td rowspan="3">Diverse question-answer trajectories</td>
    <td>Single-question-single-answer</td>
    <td>sQsA</td>
    <td>The agent asks and the user answers with parameter values one by one.</td>
  </tr>
  <tr>
    <td>Multi-question-multi-answer</td>
    <td>mQmA</td>
    <td>The agent asks for multiple parameter values, the user also answers the corresponding value.</td>
  </tr>
  <tr>
    <td>Multi-question-single-answer</td>
    <td>mQsA</td>
    <td>The agent asks for multiple parameter values, but the user only answers one value.</td>
  </tr>
  <tr>
    <td></td>
    <td>Single-question-multi-answer</td>
    <td>sQmA</td>
    <td>The agent only asks for one parameter value, but the user answers multiple values.</td>
  </tr>
  <tr>
    <td>Intent shifts</td>
    <td>User intent has changed</td>
    <td>IS</td>
    <td>The agent should output some special tags to terminate the ongoing session when a user expresses a new intent unrelated to the current slot filling process, allowing for the subsequent handling of the new intent.</td>
  </tr>
  <tr>
    <td>Argument shifts</td>
    <td>Slots overriding<br>Multiple slot values</td>
    <td>SO<br>mSv</td>
    <td>The user repeatedly modifies the value of the same slot before tool execution.<br>The user repeatedly modifies the value of the same slot after tool execution.</td>
  </tr>
  <tr>
    <td>External individual information</td>
    <td>Answering with pronouns</td>
    <td>External</td>
    <td>The user does not answers directly, but gives the external individual information pronoun.</td>
  </tr>
</table>

Table 2: Descriptions of test data in HammerBench at different granularities.

![HammerBench construction pipeline: toolset collection, data generation, external knowledge generation, and validation. Blocks with GPT icons indicate the use of LLMs, while orange blocks represent verification modules, and green blocks denote various data types corresponding to each phase.](page_186_573_1077_393.png)

Figure 1: HammerBench construction pipeline: toolset collection, data generation, external knowledge generation, and validation. Blocks with GPT icons indicate the use of LLMs, while orange blocks represent verification modules, and green blocks denote various data types corresponding to each phase.

for further optimization.

4 HammerBench

In this section, we describe the methodology and workflow for constructing the datasets used in HammerBench. To provide a more concise overview, the overall structure of test data generated for HammerBench is summarized in Table 2, with detailed statistics and analysis in Appendix A.

4.1 Test Data Construction

HammerBench’s dataset construction follows a four-stage systematic process: toolset collection, API generation, validation, and manual refinement, as shown in Figure 1.

Tool Collection and API Generation: The toolset is based on functionalities from popular mobile apps sourced from major manufacturers’ app stores. We categorize these apps into 60 functional groups, for which we create prompts (see Appendix E.1) to generate APIs using LLMs. This process involves creating function names, descriptions, parameters, and return values, with human oversight to ensure simplicity, user-friendliness, and minimal redundancy.

Validation and Refinement: Generated APIs undergo manual review for quality, resulting in 1,063 final selections. The review process prioritizes completeness (covering typical usage scenarios) and rationality (clarity and simplicity without losing functionality). Function-calling instances are generated using the self-instruct method (Wang et al., 2023b), producing three types of instructions: (1) complete function calls, (2) incomplete calls (missing key parameters), and (3) irrelevant calls (triggering rejection). This ensures both practicality and usability for effective evaluation.

4.1.1 Multi-Turn Interactions with Imperfect Instructions

As 76% of queries in user logs contain fewer than 10 tokens, we designed a workflow to generate
Figure 2: Examples of four types of test cases in HammerBench: 1) Diverse Q&A trajectories generated by merging user-agent interactions; 2) Intent shifts: agent terminates the session when users change their intent; 3) Argument shifts: three cases of changing slot values during interactions; 4) External individual information: users use pronouns instead of exact details, common in real-world interactions.

datasets that reflect typical query lengths. First, we use LLMs to identify the most commonly used parameters in daily applications, with the prompt for parameter selection provided in Appendix E.3. LLMs then generate queries including these parameters (see Appendix E.2). If parameters are missing or extraneous, the model regenerates the query until it aligns with the preset list.

This process produces imperfect instances that pass basic format verification, but hallucinations remain a concern. To mitigate this, we introduce a semantic validation step (Li et al., 2024), checking if the function call generated by the LLM is consistent across zero-shot and one-shot in-context settings. Consistency is evaluated with two metrics: 1) Rouge-L, using a predefined threshold, and 2) semantic alignment, by re-assessing the function call with the LLM. Instructions that are consistent in zero-shot settings are deemed more reliable, reducing hallucination risk. Instances that pass validation are further refined through manual inspection and sampling review. This results in 3,240 imperfect instances, forming the basis for the multi-turn function-calling scenarios discussed in the following sections, as exampled in Figure 2.

4.1.2 Diverse Question-Answer Trajectories

Building on the imperfect instructions from the previous section, we extend these to multi-turn function-calling dialogues with interactive slot-filling. To assess the model’s ability to handle diverse conversational flows, we design four types of question-answer trajectories:

Single-Question-Single-Answer: The model queries a single argument, and the user provides a corresponding value. This tests the model’s ability to accurately elicit a piece of information.

Multi-Question-Multi-Answer: The model asks multiple questions, and the user answers each. This evaluates the model’s ability to process multiple inputs within a conversation.

Single-Question-Multi-Answer: The model asks one question, and the user gives multiple responses. This challenges the model to interpret multiple answers for a single argument.

Multi-Question-Single-Answer: The model asks multiple questions, but the user only answers one. This scenario tests the model’s ability to handle incomplete information and adapt to unexpected behavior.

We generate these trajectories by first creating single-question-single-answer instances using the prompts in Appendix E.7, which undergo semantic and manual review (Section 4.1.1). The remaining trajectories are formed by randomly combining questions and answers from this dataset, resulting in 2,310 distinct instances for evaluation.

4.1.3 Intent and Argument Shifts

In real-world interactions, users often shift their intentions or modify slot values unexpectedly. The distinction between intent shifts and argument shifts is whether the new user content can be handled by the previously retrieved candidate functions. As the candidate functions retrieved by the initial user query cannot solve the new intentions,
we need to start a new session to handle intent shifts.

Users may abandon a complex request or switch goals mid-conversation. This section evaluates whether the model can recognize intent shifts and respond accordingly, such as by issuing a rejection signal to terminate the conversation. To create the dataset for intent shifts, we begin with Single-Question-Single-Answer instances. We then randomly replace a user input with a function call for a different API, simulating a sudden intent change, resulting in 1,098 intent shift instances.

Additionally, we consider two types of argument shifts: slot overriding and API repurposing.

Slot Overriding: Users may provide different values for the same parameter, with only the most recent one being relevant. We generate these instances by modifying dialogues to include turns where users explicitly override previous arguments, resulting in 1,462 instances.

API Repurposing: Users may update certain parameters of a previous function call while keeping others unchanged. To model this, we add a user input turn after a function call, introducing new values for some parameters while retaining others, yielding 1,066 instances.

4.1.4 References to External Individual Data
In AI assistant interactions, users often refer to external data indirectly using pronouns, instead of explicitly stating specific values. To evaluate the model’s anaphora resolution ability, we propose a method to generate datasets with references to external data. These references can come from any function call instance, with data retrieved from various external sources.

We select nine common information entities (e.g., names, company names, product IDs, license plates, etc.) and use LLMs to identify argument values corresponding to these entities. Instances containing such references are retained for further processing. One identified argument is then replaced by a pronoun referring to the external entity, and a new instance is generated by rephrasing the original text to include the pronoun. All instances are validated semantically and manually to ensure accuracy. Based on our practical experience, this allows the efficiency of constructing this type of data to reach 200 data points within one person-day, which is much higher than the previous method (Basu et al. (2024), Li et al. (2023))—40 data points within one person-day. This process results in 1,175 single-turn and 487 multi-turn instances involving external references.

4.2 Function Calling Snapshots
In complex interactions, models must assess whether the historical context provides enough information to fulfill user intent and identify missing details for API parameters. When information is incomplete, agents should generate follow-up questions to request the missing inputs. A common approach is the Learning to Ask paradigm, where models are fine-tuned to generate questions at each turn and then make a function call once all parameters are gathered.

We suggest an alternative, the Function Calling Snapshots, where models generate a formatted function call, or Snapshot, at each turn, regardless of context completeness. This call is then compared to the function definition using rule-based logic to identify missing information, enabling the generation of relevant follow-up questions. By decoupling the identification of missing data from question generation, this approach allows for more focused information recognition and detailed analysis, such as detecting Missing Parameters or Hallucination. For more details, please refer to Appendix B.3.

We evaluated this mechanism through a comparative experiment by fine-tuning Qwen2.5-7B on both paradigms and testing on 100 out-of-distribution (OOD) conversations with human evaluation. As shown in Table 4, the Function Calling Snapshots approach outperformed Learning to Ask by 16% in task success rate, demonstrating its superior effectiveness.

4.3 Fine-Grained Evaluation Metrics
We use several key metrics to assess function-calling performance:

Accuracy (Acc.): a traditional metric, the accuracy when function and parameter names are all correctly predicted.

Function Name Accuracy (Func. Acc.): the accuracy of predicted function names, without taking the parameters into account.

Function-Irrelevant Accuracy (Irrelevant): the proportion of samples where irrelevant functions are correctly excluded.

Parameter Hallucination and Missing Rate (PHR/PMR): PHR tracks incorrect parameter names, while PMR tracks missing parameters. These measure the accuracy and completeness of parameter predictions.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Overall (Acc.%)</th>
    <th colspan="4">Single-turn (Acc.%)</th>
    <th colspan="3">Multi-turn (Acc.%)</th>
  </tr>
  <tr>
    <th>Perfect</th>
    <th>Imperfect</th>
    <th>External</th>
    <th>Irrelevant</th>
    <th>Diverse Q&amp;A</th>
    <th>Argument shifts</th>
    <th>External</th>
    <th>Intent shifts</th>
  </tr>
  <tr>
    <th colspan="10"><i>Prompt</i></th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>71.4(2.1)</td>
    <td>78.7(1.8)</td>
    <td>78.8(1.4)</td>
    <td>56.2(2.8)</td>
    <td>69.5(2.1)</td>
    <td>73.8(2.5)</td>
    <td>68.3(1.7)</td>
    <td>69.6(2.4)</td>
    <td>76.2(2.5)</td>
  </tr>
  <tr>
    <td>Claude3.5-sonnet</td>
    <td>70.3(2.2)</td>
    <td>79.9(1.7)</td>
    <td><b>79.4(1.6)</b></td>
    <td>55.8(2.8)</td>
    <td>68.2(2.1)</td>
    <td><b>74.9(2.5)</b></td>
    <td><b>73.6(1.6)</b></td>
    <td><b>71.8(2.3)</b></td>
    <td>58.7(2.9)</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td>
    <td><b>71.5(2.1)</b></td>
    <td><b>80.9(1.7)</b></td>
    <td>68.6(1.6)</td>
    <td><b>67.4(2.7)</b></td>
    <td>73.1(2.0)</td>
    <td>65.9(2.7)</td>
    <td>65.2(1.7)</td>
    <td>61.1(2.5)</td>
    <td>89.8(1.8)</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>58.8(2.3)</td>
    <td>75.8(1.8)</td>
    <td>58.1(1.7)</td>
    <td>58.9(2.8)</td>
    <td>41.0(2.2)</td>
    <td>57.6(2.8)</td>
    <td>53.2(1.8)</td>
    <td>53.9(2.6)</td>
    <td>72.0(2.7)</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct</td>
    <td>61.6(2.1)</td>
    <td>76.0(1.8)</td>
    <td>69.5(1.6)</td>
    <td>59.6(2.8)</td>
    <td>12.5(1.3)</td>
    <td>73.1(2.6)</td>
    <td>69.7(1.7)</td>
    <td>69.3(2.4)</td>
    <td>63.0(2.9)</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct</td>
    <td>49.1(2.2)</td>
    <td>73.0(1.9)</td>
    <td>51.1(1.7)</td>
    <td>57.6(2.8)</td>
    <td>8.1(1.2)</td>
    <td>53.5(2.9)</td>
    <td>48.6(1.8)</td>
    <td>50.1(2.6)</td>
    <td>50.6(3.0)</td>
  </tr>
  <tr>
    <td>Minstral-8B-Instruct</td>
    <td>37.4(2.0)</td>
    <td>78.0(1.8)</td>
    <td>32.2(2.6)</td>
    <td>57.7(2.8)</td>
    <td>3.9(0.8)</td>
    <td>39.2(2.8)</td>
    <td>38.7(1.8)</td>
    <td>35.0(2.5)</td>
    <td>14.8(2.1)</td>
  </tr>
  <tr>
    <th colspan="10"><i>Function Call</i></th>
  </tr>
  <tr>
    <td>Hammer2.1-7b</td>
    <td>65.8(2.1)</td>
    <td>68.9(2.0)</td>
    <td>59.3(1.7)</td>
    <td>50.2(2.9)</td>
    <td><b>89.7(1.4)</b></td>
    <td>59.1(2.8)</td>
    <td>55.7(1.8)</td>
    <td>48.7(2.6)</td>
    <td><b>94.7(1.3)</b></td>
  </tr>
  <tr>
    <td>xLAM-7b-fc-r</td>
    <td>50.8(2.4)</td>
    <td>73.5(1.9)</td>
    <td>48.8(1.7)</td>
    <td>48.9(2.9)</td>
    <td>50.0(2.2)</td>
    <td>52.3(2.9)</td>
    <td>46.2(1.8)</td>
    <td>48.3(2.6)</td>
    <td>38.3(2.9)</td>
  </tr>
  <tr>
    <td>ToolACE-8B</td>
    <td>50.5(2.3)</td>
    <td>78.8(1.7)</td>
    <td>51.9(1.7)</td>
    <td>56.9(2.8)</td>
    <td>26.6(2.0)</td>
    <td>49.4(2.9)</td>
    <td>48.4(1.8)</td>
    <td>49.2(2.6)</td>
    <td>43.0(2.9)</td>
  </tr>
</table>

Table 3: HammerBench leaderboard (Single-Turn and Multi-Turn) including Acc. and 95% confidence intervals, where the best performances are highlighted.

<table>
  <tr>
    <th>Training Paradigm</th>
    <th>SR</th>
  </tr>
  <tr>
    <td>Learning to Ask</td>
    <td>68%</td>
  </tr>
  <tr>
    <td>Function Calling Snapshots</td>
    <td>84%</td>
  </tr>
</table>

Table 4: Comparison between Learning to Ask and Function Calling Snapshots paradigms.

<b>Progress Rate (PR):</b> the proportion of correct function calls up to the turns of error, defined as \( PR = \frac{k}{n} \), where \( k \) is the number of correct turns and \( n \) is the total number of turns.

<b>Success Rate (SR):</b> the overall accuracy of function calls across all turns in a conversation, with higher values indicating better final performance.

These metrics—Acc., Func. Acc., PHR, PMR, SR, and PR—provide a comprehensive framework for evaluating function calls in both individual and multi-turn dialogues.

5 Experiment

In this section, we demonstrate the superiority of HammerBench through a series of experiments with 10 commonly used or function-calling-specific LLMs. We provide detailed analyses of the experimental results, followed by in-depth discussions and valuable insights that could be used to guide subsequent model optimization.

5.1 Experimental Setups

To validate the effectiveness of HammerBench, we benchmark the following baseline models: GPT-4o (Hurst et al., 2024), Claude3.5-sonnet(Anthropic, 2024), Llama-3.1-70B-Instruct (Dubey et al., 2024), Qwen2.5-72B-Instruct (Qwen Team, 2024), Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, Minstral-8B-Instruct (AI, 2024), xLAM-7b-fc-r (Liu et al., 2024b) and ToolACE-8B (Liu et al., 2024a). For all models, we impose a strict requirement on the output format in the prompt, as detailed in Appendix D. All open-source models are deployed on 4 A100 80G GPUs, with a temperature setting of 0.0 and bfloat16 precision.

5.2 Results Analysis of Evaluation Types

Table 3 shows the overall result of HammerBench in different evaluation types including single-turn and multi-turn. Models like Qwen2.5-72B-Instruct (Prompt), GPT-4o (Prompt), and Claude3.5-sonnet (Prompt) are among the top performers, with a strong ability to handle both single-turn and multi-turn interactions, while models like Hammer2.1-7b (FC) and Minstral-8B-Instruct (Prompt) show more specialized strengths or weaknesses depending on the task type. Specifically, with the help of diverse scenarios, the following key insights primarily emerge regarding Intent Shifts, Argument Shifts, and External Individual Information.

<b>Intent Shifts.</b> In the scenario of intent shifts, we assess the LLM’s ability to recognize changing user intentions. As shown in Table 3, LLMs with stronger single-turn irrelevance detection capabilities tend to perform better in handling intent shifts.

<b>Argument Shifts.</b> As seen in experiments from Table 5, a major challenge in multi-turn interactions occurs when users modify the same slot multiple times before or after function execution. This tests the model’s ability to track and update slot values. LLMs often struggle to handle such shifts, like when a user changes a date from June 1st to July
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">Diverse Q&A (PR%)</th>
    <th colspan="4">Argument shifts (PR%)</th>
    <th colspan="4">External (PR%)</th>
    <th rowspan="2">IS (Acc.%)</th>
  </tr>
  <tr>
    <th>sQsA</th>
    <th>mQmA</th>
    <th>mQsA</th>
    <th>sQmA</th>
    <th>w/o SO</th>
    <th>w/ SO</th>
    <th>w/o mSv</th>
    <th>w/ mSv</th>
    <th>w/o External</th>
    <th>w/ External</th>
  </tr>
  <tr>
    <td colspan="13"><i>Prompt</i></td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>66.0(4.1)</td>
    <td>73.2(8.3)</td>
    <td>68.5(4.0)</td>
    <td>70.8(3.3)</td>
    <td>67.9(2.9)</td>
    <td>68.7(2.9)</td>
    <td>68.5(2.9)</td>
    <td>56.5(2.1)</td>
    <td>74.4(3.5)</td>
    <td>66.3(3.4)</td>
    <td>76.2(2.5)</td>
    <td></td>
  </tr>
  <tr>
    <td>Claude3.5-sonnet</td>
    <td><b>70.6(4.0)</b></td>
    <td><b>73.7(3.8)</b></td>
    <td><b>68.9(4.0)</b></td>
    <td><b>72.0(3.8)</b></td>
    <td><b>71.7(2.9)</b></td>
    <td><b>72.5(2.9)</b></td>
    <td><b>72.0(2.5)</b></td>
    <td><b>63.9(2.4)</b></td>
    <td><b>76.6(3.4)</b></td>
    <td><b>68.6(3.4)</b></td>
    <td><b>58.7(2.9)</b></td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td>
    <td>56.3(4.2)</td>
    <td>64.8(4.3)</td>
    <td>56.9(4.3)</td>
    <td>64.2(4.3)</td>
    <td>61.1(3.1)</td>
    <td>60.3(3.1)</td>
    <td>63.2(2.6)</td>
    <td>57.4(2.6)</td>
    <td>65.9(3.8)</td>
    <td>59.6(3.7)</td>
    <td>89.8(1.8)</td>
    <td></td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct</td>
    <td>64.7(4.2)</td>
    <td>67.4(4.2)</td>
    <td>61.6(4.2)</td>
    <td>68.0(4.2)</td>
    <td>65.3(3.1)</td>
    <td>62.0(3.1)</td>
    <td>65.7(2.7)</td>
    <td>58.6(2.6)</td>
    <td>72.6(3.7)</td>
    <td>63.5(3.6)</td>
    <td>63.0(2.9)</td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>48.9(4.3)</td>
    <td>53.1(4.4)</td>
    <td>43.6(4.2)</td>
    <td>53.0(4.4)</td>
    <td>50.7(3.2)</td>
    <td>45.8(3.0)</td>
    <td>51.7(2.8)</td>
    <td>41.8(2.4)</td>
    <td>58.1(4.0)</td>
    <td>50.0(3.7)</td>
    <td>72.0(2.7)</td>
    <td></td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct</td>
    <td>38.4(3.7)</td>
    <td>51.2(4.4)</td>
    <td>38.4(3.7)</td>
    <td>50.4(3.4)</td>
    <td>41.5(2.9)</td>
    <td>38.0(2.8)</td>
    <td>42.8(2.6)</td>
    <td>35.0(2.3)</td>
    <td>47.8(3.7)</td>
    <td>43.6(3.5)</td>
    <td>50.6(3.0)</td>
    <td></td>
  </tr>
  <tr>
    <td>Minstral-8B-Instruct-2410</td>
    <td>26.8(3.9)</td>
    <td>28.6(4.0)</td>
    <td>19.0(3.0)</td>
    <td>28.4(4.0)</td>
    <td>27.8(2.9)</td>
    <td>26.2(2.8)</td>
    <td>28.2(2.6)</td>
    <td>22.0(2.1)</td>
    <td>31.3(3.9)</td>
    <td>25.1(1.2)</td>
    <td>14.8(2.1)</td>
    <td></td>
  </tr>
  <tr>
    <td colspan="13"><i>Function Call</i></td>
  </tr>
  <tr>
    <td>Hammer2.1-7b</td>
    <td>52.4(4.3)</td>
    <td>57.6(4.3)</td>
    <td>41.5(3.8)</td>
    <td>57.2(4.3)</td>
    <td>53.9(3.2)</td>
    <td>49.8(3.0)</td>
    <td>54.8(2.7)</td>
    <td>43.9(2.3)</td>
    <td>60.6(4.0)</td>
    <td>46.3(3.4)</td>
    <td><b>94.7(1.3)</b></td>
    <td></td>
  </tr>
  <tr>
    <td>xLAM-7b-fc-r</td>
    <td>39.3(4.4)</td>
    <td>40.7(4.5)</td>
    <td>30.4(3.8)</td>
    <td>41.1(4.1)</td>
    <td>39.0(3.2)</td>
    <td>34.4(3.0)</td>
    <td>39.1(2.8)</td>
    <td>29.1(2.1)</td>
    <td>49.1(4.2)</td>
    <td>39.8(2.7)</td>
    <td>38.3(2.9)</td>
    <td></td>
  </tr>
  <tr>
    <td>ToolACE-8B</td>
    <td>42.6(4.0)</td>
    <td>46.4(4.2)</td>
    <td>25.9(2.8)</td>
    <td>46.1(4.1)</td>
    <td>44.1(3.1)</td>
    <td>40.1(2.9)</td>
    <td>44.8(2.7)</td>
    <td>35.1(2.2)</td>
    <td>51.1(3.9)</td>
    <td>43.6(3.6)</td>
    <td>43.0(2.9)</td>
    <td></td>
  </tr>
</table>

Table 5: Ablation results: The evaluation (PR and 95% confidence intervals) on HammerBench for different multi-turn data types, while “w/o” refers to subsets of sQsA datasets with the same user queries with “w/”. The IS represents the success rate of LLM terminating correctly when encountering intent shifts. This table is a snippet from Table 9.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Func. Acc.(%)</th>
    <th colspan="3">First snapshot(%)</th>
    <th colspan="3">Last snapshot(%)</th>
  </tr>
  <tr>
    <th>PHR</th>
    <th>PMR</th>
    <th>Other Error Rate</th>
    <th>PHR</th>
    <th>PMR</th>
    <th>Other Error Rate</th>
  </tr>
  <tr>
    <td colspan="8"><i>Prompt</i></td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>88.8(1.9)</td>
    <td>8.5(1.8)</td>
    <td>3.4(1.1)</td>
    <td>2.9(1.1)</td>
    <td>6.6(1.6)</td>
    <td>8.8(1.8)</td>
    <td>8.4(1.8)</td>
  </tr>
  <tr>
    <td>Claude3.5-sonnet</td>
    <td>90.2(1.8)</td>
    <td>8.6(1.0)</td>
    <td>2.4(1.0)</td>
    <td>1.5(0.8)</td>
    <td>7.0(1.6)</td>
    <td>8.0(1.7)</td>
    <td>4.8(1.3)</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td>
    <td>85.4(2.1)</td>
    <td>12.8(1.2)</td>
    <td>1.3(0.7)</td>
    <td>1.8(0.9)</td>
    <td>11.0(2.0)</td>
    <td>6.0(1.5)</td>
    <td>4.1(1.3)</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct</td>
    <td>91.6(1.6)</td>
    <td>19.1(2.4)</td>
    <td>2.0(0.9)</td>
    <td>2.0(0.9)</td>
    <td>11.1(1.9)</td>
    <td>2.1(0.9)</td>
    <td>5.9(1.5)</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>82.9(2.2)</td>
    <td>24.7(2.8)</td>
    <td>2.9(1.1)</td>
    <td>1.3(0.7)</td>
    <td>14.6(2.3)</td>
    <td>5.8(1.5)</td>
    <td>5.5(1.5)</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct</td>
    <td>89.3(1.8)</td>
    <td>36.8(3.0)</td>
    <td>1.6(0.8)</td>
    <td>1.2(0.7)</td>
    <td>15.2(2.3)</td>
    <td>22.1(2.6)</td>
    <td>4.2(1.3)</td>
  </tr>
  <tr>
    <td>Minstral-8B-Instruct-2410</td>
    <td>74.6(2.6)</td>
    <td>52.5(3.4)</td>
    <td>2.9(1.2)</td>
    <td>1.0(0.7)</td>
    <td>18.1(2.6)</td>
    <td>7.0(1.7)</td>
    <td>5.4(1.6)</td>
  </tr>
  <tr>
    <td colspan="8"><i>Function Call</i></td>
  </tr>
  <tr>
    <td>Hammer2.1-7b</td>
    <td>83.1(2.2)</td>
    <td>19.1(2.6)</td>
    <td>3.7(1.2)</td>
    <td>1.2(0.7)</td>
    <td>13.3(2.2)</td>
    <td>9.4(1.9)</td>
    <td>4.8(1.4)</td>
  </tr>
  <tr>
    <td>xLAM-7b-fc-r</td>
    <td>88.8(1.9)</td>
    <td>47.2(3.1)</td>
    <td>3.4(1.1)</td>
    <td>1.0(0.6)</td>
    <td>18.1(2.4)</td>
    <td>8.7(1.8)</td>
    <td>4.1(1.3)</td>
  </tr>
  <tr>
    <td>ToolACE-8B</td>
    <td>81.8(2.3)</td>
    <td>29.6(3.0)</td>
    <td>3.9(1.3)</td>
    <td>1.2(0.7)</td>
    <td>25.5(2.9)</td>
    <td>4.9(1.4)</td>
    <td>4.5(1.4)</td>
  </tr>
</table>

Table 6: Evaluations on the first and last snapshots in the sQsA dataset. First snapshot: the first turn of the conversation, Last snapshot: the last turn of the conversation. Other Error Rate: parameter values error rate. The corresponding 95% confidence intervals are in brackets.

8th but the model retains the old value. This issue is particularly evident in scenarios with similar parameters, such as updating train orders. The Slot Overriding task effectively detects this flaw, revealing that many open-source LLMs are slow to adapt to changes, while GPT-4o shows stronger performance in tracking dynamic slot updates. Another challenge arises when users modify a slot after a tool has been executed, leading to issues with maintaining consistency across multiple values. Even GPT-4o, though less affected, sometimes fails to track all updated parameters. This is likely due to the model perceiving the tool call as “complete” after execution, which causes it to disregard prior input. The Multiple Slot Values task highlights this issue, where slot values are inconsistently managed across turns. This task underscores the difficulty models face in maintaining parameter consistency over extended interactions. Appendix C.4 shows examples of these challenges.

External Information. User queries containing external individual information often introduce noise, affecting slot-filling accuracy. The External Information task effectively detects this issue, showing how LLMs struggle with indirect references like pronouns or contextual information, leading to lower accuracy as shown in Table 5 and Appendix C.3. This task reveals the vulnerability of LLMs to external factors like personalized data or context, significantly disrupting function-calling accuracy. It highlights the importance of models being able to separate core inputs from external noise, making it a critical metric for real-world performance. Further results are available in Appendix F.

5.3 Results Analysis of Evaluation Metrics

In addition to the results from diverse scenarios, the fine-grained metric design of HammerBench allows for a more detailed identification of specific
issues in models within particular contexts. We will analyze the effectiveness of the metrics from different scenarios. Some interesting insights are revealed with the metrics defined in HammerBench, with more details in Appendix F.

Progress Rate. In diverse Q&A dataset Table 5, interactions often lead open-source LLMs to hallucinate additional parameters, particularly when addressing multi-question-single-answer (mQsA) scenarios, which results in parameter mispredictions, as elaborated in Appendix C.2. Progress Rate (PR) can help gauge model resilience effectively. For instance, GPT-4o demonstrates minimal degradation in PR, suggesting its robustness against hallucination issues.

PHR/PMR. Based on the results presented in Table 6, the PHR in the initial snapshot, where the context is incomplete, is significantly higher compared to the final snapshot, where the context is complete. This indicates that an incomplete context leads to a higher frequency of hallucinations in parameters, which remains a persistent bottleneck for many open-source LLMs.

6 Conclusion

This paper presents HammerBench, a fine-grained evaluation benchmark for multi-turn function calling tasks, assessing imperfect instructions, diverse question-answer trajectories, intent and argument shifts, and user queries with external individual information. These components reflect real-world user scenarios. We decompose multi-turn slot-filling interactions into a series of function-calling snapshots, allowing evaluation of each conversational turn. Additionally, we introduce random merging of dialogue trajectories to assess question-answer trajectory variability. A method for generating data with external individual information is also released, enabling evaluation of a model’s ability to resolve pronouns and anaphora.

Our experimental results highlight the challenges LLMs face in handling argument shifts and external information, e.g. tracking dynamic slot updates. Additionally, metrics such as Progress Rate and PHR/PMR reveal critical areas of improvement, notably in model resilience against hallucinations and handling evolving user intents, underscoring the need for further advancements to enhance LLM robustness in real-world applications.

Limitations

Although HammerBench can assess certain multi-turn real-world scenarios and facilitates the evaluation of multi-turn interactions in more complex contexts by incorporating multi-dimensional data generation methods, its evaluations are confined to fixed, pre-generated datasets. This limitation inherently restricts the benchmark’s capacity to fully capture the diverse range of behaviors and capabilities that a large language model (LLM) might exhibit in a dynamic, real-world setting. When compared to the approach of using LLMs to generate real-time, dynamic trajectories during the evaluation process (Lu et al., 2024), the scope of trajectories covered by HammerBench is less comprehensive, and it does not assess the model’s ability to self-explore.

The datasets used in HammerBench are automatically generated by synthesizing real-world scenarios, which inevitably introduces some discrepancies when compared to actual user queries. While this benchmark does support the evaluation of multi-turn interactions in more complex scenarios by incorporating data generation techniques from various dimensions, exhaustively capturing all possible user queries remains infeasible. The reliance on fixed evaluation trajectories, while making the evaluation process more convenient and controllable, limits HammerBench to only a partial step towards automated evaluations that more closely mirror real user interactions.

References

Mistral. AI. 2024. Introducing the world’s best edge models. Website. https://mistral.ai/news/ministraux/.

Anthropic. 2024. Introducing claude 3.5 sonnet. Accessed: 2025-02-15.

Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiacheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezhen Ge, Bo Zheng, et al. 2024. Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page 7421–7454.

Kinjal Basu, Ibrahim Abdelaziz, Kelsey Bradford, Maxwell Crouse, Kiran Kate, Sadhana Kumarravel, Saurabh Goyal, Asim Munawar, Yara Rizk, Xin Wang, Luis Lastras, and Pavan Kapanipathi. 2024. Nestful: A benchmark for evaluating llms
on nested sequences of api calls. arXiv preprint arXiv:2409.03797.

Shihain Deng, Weikai Xu, Hongda Sun, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, Rui Yan, et al. 2024. Mobile-bench: An evaluation benchmark for llm-based mobile agents. arXiv preprint arXiv:2407.00993.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Fenfei Guo, Angeliki Metallinou, Chandra Khatri, Anirudh Raju, Anu Venkatesh, and Ashwin Ram. 2017. Topic-based evaluation for conversational bots. NIPS 2017 Conversational AI workshop.

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.

Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvuru, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. 2024. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649.

Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, and Kam-Fai Wong. 2024. Mt-eval: A multi-turn capabilities evaluation benchmark for large language models.

Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. Api-bank: A comprehensive benchmark for tool-augmented llms. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3102–3116.

Yunshui Li, Binyuan Hui, Xiaobo Xia, Jiaxi Yang, Min Yang, Lei Zhang, Shuzheng Si, Junhao Liu, Tongliang Liu, Fei Huang, et al. 2024. One-shot learning as instruction data prospector for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4586–4601.

Qiqiang Lin, Muning Wen, Qiuying Peng, Guanyu Nie, Junwei Liao, Jun Wang, Xiaoyun Mo, Jianmu Zhou, Cheng Cheng, Yin Zhao, et al. 2024. Hammer: Robust function-calling for on-device language models via function masking. arXiv preprint arXiv:2410.04587.

Weiwen Liu, Xu Huang, Xingshan Zeng, Xinlong Hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuqiang Yu, et al. 2024a. Toolace: Winning the points of llm function calling. arXiv preprint arXiv:2409.00920.

Zuxin Liu, Thai Quoc Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, et al. 2024b. Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, et al. 2024. Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities. arXiv preprint arXiv:2408.04682.

Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. 2024. Agentboard: An analytical evaluation board of multi-turn llm agents. arXiv preprint arXiv:2401.13178.

Jiao Ou, Junda Lu, Che Liu, Yihong Tang, Fuzheng Zhang, Di Zhang, and Kun Gai. 2023. Dialogbench: Evaluating llms as human-like dialogue systems. arXiv preprint arXiv:2311.01677.

Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, et al. 2024. Webcanvas: Benchmarking web agents in online environments. arXiv preprint arXiv:2406.12373.

Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv e-prints, pages arXiv–2307.

Qwen Team. 2024. Qwen2.5: A party of foundation models.

Yifan Song, Weimin Xiong, Dawei Zhu, Wenhao Wu, Han Qian, Mingbo Song, Hailiang Huang, Cheng Li, Ke Wang, Rong Yao, Ye Tian, and Sujian Li. 2023. Restgpt: Connecting large language models with real-world restful apis. arXiv e-prints, pages arXiv–2306.

Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. 2023. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. arXiv e-prints, pages arXiv–2306.

Nexusflow.ai team. 2023. Nexusraven: Surpassing the state-of-the-art in open-source function calling llms.

Hongru Wang, Rui Wang, Boyang Xue, Heming Xia, Jingtao Cao, Zeming Liu, Jeff Z Pan, and Kam-Fai Wong. 2024a. Appbench: Planning of multiple apis from various apps for complex user instruction. arXiv preprint arXiv:2410.19743.

Wenxuan Wang, Juluan Shi, Chaozheng Wang, Cheryl Lee, Youliang Yuan, Jen-tse Huang, and Michael R Lyu. 2024b. Learning to ask: When llms meet unclear instruction. arXiv preprint arXiv:2409.00557.
Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. 2023a. Mint: Evaluating llms in multi-turn interaction with tools and language feedback. arXiv preprint arXiv:2309.10691.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023b. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484–13508.

Mengsong Wu, Tong Zhu, Han Han, Chuanyuan Tan, Xiang Zhang, and Wenliang Chen. 2024. Seal-tools: Self-instruct tool learning dataset for agent tuning and detailed benchmark. arXiv preprint arXiv:2405.08355.

Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. 2023. On the tool manipulation capability of open-source large language models. arXiv e-prints, pages arXiv–2305.

Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. Berkeley function calling leaderboard. https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.

Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitri Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, et al. 2024. Agent-as-a-judge: Evaluate agents with agents. arXiv preprint arXiv:2410.10934.
A  Analysis of Benchmark Dataset

The overall structure of the datasets generated for HammerBench is summarized in Table 2. The dataset comprises four single-turn datasets, each with four distinct types, and eight multi-turn datasets, categorized into three types.

<table>
  <tr>
    <th>Statistics</th>
    <th>Perfect</th>
    <th>Imperfect</th>
    <th>External</th>
  </tr>
  <tr>
    <td># of categories</td>
    <td>60</td>
    <td>60</td>
    <td>55</td>
  </tr>
  <tr>
    <td># of tools</td>
    <td>1063</td>
    <td>894</td>
    <td>463</td>
  </tr>
  <tr>
    <td># of queries</td>
    <td>2116</td>
    <td>3240</td>
    <td>1175</td>
  </tr>
</table>

Table 7: Statistics of single-turn datasets.

**Statistics:** Our HammerBench dataset includes 60 functional categories, 1,063 tools, and a total of 6,531 queries, distributed across three types, as shown in Table 7. Figure 3(a) illustrates the correlation between the number of tools and the number of parameters defined in the toolset. For the parts of multi-turn test data, there are 1098 sQsA conversations generated from single-turn imperfect instructions, spanning 59 categories and 494 APIs. If we define the number of turns as the total number of questions asked by the agent, there are approximately 404 conversations with turns greater than 1. Figure 3(b) presents the distribution of conversations according to the number of turns. Because conversations can only be merged with more than one turn, there are 404 conversations in mQmA, mQsA, and sQmA. And the distribution of multi-turn dataset types is depicted in Figure 3(c).

**Quality:** The dataset quality is rigorously ensured through the LLM validation module. Initially, data that fails the LLM check is manually corrected on a case-by-case basis. Data that passes the LLM’s double-check are subject to a random sample review, with 100 instances selected for human recheck. Our quality assurance procedures have shown that the data availability rate exceeds 95%. Given that our fine-grained multi-turn dataset is constructed from single-question-single-answer multi-turn data, the quality of the underlying single-turn instances is of paramount importance. Consequently, we conducted a manual review of 1,098 multi-turn instances to ensure their compliance with the logical flow of everyday conversations.

**Authenticity:** Within the goal of reflecting real-world user experience, we argue that the authenticity of queries and APIs should be interpreted primarily in terms of semantic distribution—i.e., the distribution of user intents and API functionalities—rather than their surface linguistic expressions. To ensure this, we first extracted a seed dataset from a large-scale real user interaction log and data from application markets. This dataset is designed to faithfully represent the underlying distribution of genuine user needs and API capabilities. For example, the distribution of tool categories invoked by users in our seed dataset is as follows: System Settings (32.26%), Alarm (10.84%), Phone Call (10.12%), Weather (7.75%), Music (6.37%), Pay (3.28%), Map (1.97%), and Others (27.41%).

Building on this seed dataset, we employed LLMs to rewrite, expand, and diversify the queries and APIs. This augmentation process does not alter the statistical semantics of the dataset; the overall distribution remains aligned with real-world user behavior. Consequently, our open-sourced dataset retains both the coverage and distribution of real-world usage scenarios.

Moreover, directly releasing raw user queries or APIs could entail legal or ethical risks—such as unintentional exposure of private user information or commercial APIs without permission. Therefore, the rewriting process using LLMs also serves as a protective mechanism, helping us strike a practical balance between open access and responsible data sharing.
Figure 3: Statistics: a)The number of tools corresponding to different parameter counts in our toolset; b)The number of conversations corresponding to different turn counts in sQsA; c)Various number of multi-turn data cases constructed based on the Imperfect and External in Table 7.

B Examples

B.1 Single-turn

normal user query

User: Please help me send the traffic violation record of PDX-1234 in Portland yesterday to Manager Zhang by email.
Agent: {"name":"shareViolationDetail", "arguments":{"city": "Portland", "media": "email", "name": "Manager Zhang", "plate_number": "PDX-1234", "time":"yesterday"}}

missing arguments user query

User: Share the traffic violation information in Portland.
Agent: {"name":"shareViolationDetail", "arguments":{"city": "Portland", "media": "", "name": "", "plate_number": "", "time":""}}

user query with external individual information

User: I found the license plate number mentioned in my notepad and would like to share the traffic violation information I encountered in Portland yesterday with Manager Zhang via email. Is that okay?
EK: At 3pm today, I saw a car with license plate number PDX-1234 in the parking lot. The owner seemed to have forgotten to turn off the lights, and the lights were on until the evening. I am worried about violating traffic rules. If you know the owner, please remind him.
Agent: {"name":"shareViolationDetail", "arguments":{"city": "Portland", "media": "email", "name": "Manager Zhang", "plate_number": "PDX-1234", "time":"yesterday"}}

B.2 Multi Turn

conversation with external individual information

User: Share the violation information.
Agent: Which city did the violation occur in?
User: The violation occurred in Portland.
Agent: Please provide a way to share.
User: Share the link via email.
Agent: Please provide the recipients of the shared content.
User: The recipient was Manager Zhang.
Agent: Please provide the license plate number.
User: The license plate number mentioned in the notebook
EK: At 3pm today, I saw a car with license plate number PDX-1234 in the parking lot. The owner seemed to have forgotten to turn off the lights, and the lights were on until the evening. I am worried about violating traffic rules. If you know the owner, please remind him.
Agent: When is the order opening time?
User: It was yesterday.
Agent: {"name":"shareViolationDetail", "arguments":{"city": "Portland", "media": "email", "name": "Manager Zhang", "plate_number": "PDX-1234", "time":"yesterday"}}

B.3 Function Calling Snapshots

Two snapshot evaluation examples are in the following:

First example: Suppose a tool is missing four required parameters: A, B, C, and D. In this case, we have no predefined script (e.g., should select A first). Instead, asking for any subset of A, B, C, D is considered a correct action. Besides, if the LLM requests parameters outside of this set, it is treated as an incorrect response, often indicative of hallucination. This is directly reflected in an increased PHR score, enabling us to detect such cases effectively.

Second example: Again, assume the tool is missing parameters A, B, C, and D; if the LLM asks for slot B, but the user input provides content for slot A. The key focus here shifts to whether the LLM correctly interprets the information provided by the user during each interaction round. A correct behavior in this context requires the LLM to accurately extract and fill the content into slot A, and then continue querying any subset of the remaining missing parameters B, C, D. We covered this type of case in our Diverse Q&A dataset.
C Examples of Model Output Errors

C.1 Imperfect instruction

Parameter hallucination

User: Help me find a mail.
Agent: {"name":"searchMail", "arguments":{"keyword": "mail", "max_results": "10"}}
The user does not provide any parameter information, but the parameter slots are filled in by LLMs.

C.2 Diverse Q&A trajectories

Parameter hallucination

User: See how much airline points I have.
Agent: What's your rider’s name Provide the rider’s ID number.
User: The passenger’s name is Bob.
Agent: {"name":"getFlightAirlinePoints", "arguments":{"name": "Bob", "passenger_id": "123456789"}}
Due to aske ID number, although the user does not answer, LLMs also mistakenly fills in this parameter slot.

C.3 External Individual Information

Redundant Information

User: Share my taxi ride.
Agent: What time is the order
User: The time mentioned in the Facebook message sent to me by Xiao Ming<EK>:Yesterday evening at 8pm, we had a virtual meeting to discuss next week’s work schedule. Everyone put forward a lot of good suggestions and felt that the meeting was efficient. Hopefully our team can continue to maintain this positive attitude!
Agent: {"name":"shareTaxiOrder", "arguments":{"time": "yesterday evening at 8pm", "media": "Facebook"}}
The user only mentions that the time is in the Facebook, but does not mention that the media is Facebook.

C.4 Argument shifts

slot overriding

User: Is there any way to change the train ticket standby order I applied for previously?
Agent: What number of trips did you book
User: The number I booked is G123. I’m sorry, Booked trips is D789.
Agent: {"name":"updateTrainWaitlistOrder","arguments":{"train_number": "D789", "after_number": "G123"}}
The user provides two values for the same slot, but LLMs fail to recognize.

multiple slot values

User: Find me a way. How do I get there?
Agent: Please provide your departure location name or address If using your current location please confirm directly
User: People’s Square
Agent: Please provide your destination name or address
User: Shanghai Hongqiao International Airport
Agent: {"name":"vehicleNavigationRoute","arguments":{"departure": "People’s Square", "destination": "Shanghai Hongqiao International Airport"}}
User: I also want to see that Name or address of the destination is Bondi Beach
Agent: {"name":"vehicleNavigationRoute","arguments":{"destination": "Bondi Beach"}}
When another destination is given again, the original departure parameter is lost.
D  Prompts of Function Calling

__________________________format prompt__________________________

You have access to the following functions.
<tools>
To call a function, please respond with JSON for a function call. Respond in the format ““json{"name": function name, "parameters": dictionary of argument name and its value},““
Attention! For time parameters, please fill in the format as described in the user request, and do not automatically convert the format! For example:
user: Set an alarm for 8 a.m. tomorrow
assistant: {"name":"UtilityTools.AlarmClock.addAlarm", "arguments":{"time":"8 a.m. tomorrow"}}}

## Note! Please do not hallucinate parameters. If some parameters are not mentioned in the user request, please directly output an empty string ” For example:
user:Check my traffic violation record to see what happened.
assistant:““json{"name": "Navigation.TrafficViolations.viewViolationDetail", "arguments":{"plate_number": "", "city": "", "time": ""}}““.
Never ask to the user for missing parameters!Output tool call!
## If all the above tools are not suitable, you must output: Sorry, no tool is suitable for your request.

Let’s start!

E  Prompts of Dataset Construction

E.1  Tool Generation

__________________________Generating Tools__________________________

# System
You are a tool builder! Your task is to generate realistic and versatile toolkits that will be used by Large Language Models (LLMs) to enhance their ability to solve real-world tasks. You must generate toolkits that are useful, cohesive, complete, and diverse. You should focus on generating toolkits that are commonly used by average users.
### Core requirements
1. Realism requirement: Ensure that the generated toolkit has a realistic design and can be seamlessly integrated with LLMs for practical real-world applications. Specifically, the toolkit should either have a tangible prototype that has been deployed in the real world, such as those developed using existing APIs (such as Weibo or NetEase Mail), or demonstrate strong potential for future development and accessibility through APIs (such as industrial automation toolkits).
2. Diversity requirement: Ensure that the generated toolkit is diverse, meeting a wide range of domains, types, functions, and use cases.
3. Compatibility requirement: Ensure that the generated toolkit is compatible with the textual interfaces of LLMs. In particular, LLMs can only interact with the toolkit through textual and programmatic APIs. Therefore, the tool API should mainly accept and return text as input and output. Therefore, the toolkit’s API should mainly accept and return text as input and output. In cases where the input or output involves alternative types such as files, images, videos, or audio, these should be interfaced through data paths or URLs, rather than raw data.
4. Cohesion requirement: Ensure that the generated toolkit is a cohesive collection of related tools designed to facilitate the completion of a specific core target task. It should contain a variety of tool APIs that serve different purposes within the scope of the intended functionality. For example, the NetEase Mailbox toolkit may include tools for sending, searching, reading, and deleting emails.
5. Completeness requirement: Ensure that the toolkit is functionally complete and self-contained. Each toolkit should contain comprehensive tool APIs that are sufficient to complete its core target tasks without relying on any external tools or resources. In particular, if a toolkit involves operating data resources (such as tasks in the Trello toolkit), the tool APIs should generally support relevant CRUD
(create, read, update, delete) operations on these resources, or similar operations. In addition, these tool APIs should generally operate using unique identifiers of the data source. You should not assume that such unique identifiers will be provided by the user. Instead, there should be a tool API (e.g., a search tool API) in the toolkit for retrieving unique identifiers. A typical design of a complete toolkit is as follows:
- A search tool for retrieving unique identifiers (and possibly primary content) of data resources, such as by keyword search.
- A read tool that takes a unique identifier as a parameter and returns the detailed data resource.
- An update tool that takes a unique identifier and the updated data resource as parameters and updates the data resource, returning the unique identifier of the updated data resource.
- A delete tool that takes a unique identifier as a parameter and deletes the data resource, returning the success status of the deletion.
- A create tool that takes a data resource as a parameter and creates a new data resource, returning the unique identifier of the created data resource.

### Additional Requests
1. Unique target user requirement: Ensure that the toolkit is designed for a specific target user group and that all tools will be used by the same target user group. The designated user group can vary - individual users, enterprise users, developers, etc. For example, in the case of an e-commerce toolkit, all tools should target either individual users or enterprise users, but not both.
2. Compactness requirement: Ensure that the toolkit API is compact, meaning that if tools have similar interfaces and functionality, they should be unified into one tool. For example, tools for searching for different types of items should be unified into one tool API with a parameter for specifying the type of items to search for.
3. Search limit requirement: For toolkits that involve searching the tool API to retrieve data resources, ensure that there is a parameter for specifying the maximum number of results returned by the search tool API. This parameter can be optional, and the default value is specified in the tool description.
4. Avoid unnecessary authentication requirements: In most cases, toolkits are designed for personal use by a single user, and it is reasonable to assume that the user has completed authentication before using the toolkit. In this case, ensure that all tools do not require login, authentication, or personal ID. For example, all tools in a personal banking toolkit should not require a login or a 'user_id' parameter for the tool. This also means that "unauthorized evaluation" should not be considered a potential risk for the toolkit.
5. Avoid unnecessary ID requirements: Ensure that the tool does not introduce unnecessary unique identifiers. Unique identifiers are only necessary when multiple data resource instances of the same type need to be distinguished and operated on by the unique identifier. For example, for an online shopping toolkit, an 'order_id' needs to be introduced because there are usually multiple orders in a user account, and it is necessary for operations on orders such as retrieval, read or cancel. However, unique identifiers are unnecessary when the data source is singular, eliminating the need for distinction. For example, for an online shopping toolkit, a 'cart_id' for a user's shopping cart or a 'profile_id' for the user's own profile are redundant because a user account usually contains only one of each.
6. Stored data access requirements: For cases where some tools in the toolkit require specific user details such as address or payment method, and it is reasonable to assume that the user has previously provided this information to the toolkit, there must be a tool API that can access the stored data. For example, in a toolkit for food delivery or e-commerce, there should be a tool API for accessing a user's saved addresses and payment methods.
7. Operation Status Indication Requirement: For tool APIs that involve operations such as creating, updating, or deleting data resources, it is critical to include the operation status (e.g., a Boolean value of "success") as part of the return value. Other return values should also align with the operation status. For example, if a tool returns a unique identifier for a created data resource, it should return null or an empty string if the operation failed.
8. File/Media Management Requirement: Toolkits that involve file or media resources (such as the
Twitter toolkit that requires media attachments) need to carefully distinguish between local and remote file management. Remote files located on a remote server should be identified and managed by their remote file path or unique identifier. Whereas local files located on the user’s system should be identified and managed by local file paths. Different tools may require remote or local files as input parameters or return values, depending on the specific use case. For example, for the Twitter toolkit, a tool for posting a tweet should require the local path of the media file to be posted, while a tool for retrieving tweets should return the remote file identifier of the media file in the tweet. Additionally, there should be a tool for downloading remote media files to the local system.

### Output Format
Toolkit specifications should be in a valid JSON list separated by “‘json’ and ‘‘’. Each item in the list should contain the following fields:
‘tool’: The name of the tool, which should be in "CamelCase" format.
‘name’: The Chinese name of the tool, which should express the function of the tool as concisely as possible.
‘description’: The summary of the tool, which should be a clear and concise description of the purpose and function of the tool without any ambiguity. It must be in Chinese.
‘parameters’: The parameter list of the tool, each parameter should contain the fields including ‘name’, ‘type’ and ‘description’, ‘required’ (whether the parameter is required). ‘description’ must be in Chinese.
‘returns’: The return list of the tool, each return should contain ‘name’, ‘type’ and ‘description’.

Please note:
1. For tool ‘parameters’ and ‘returns’, ‘name’ should not contain spaces and should be in "snake_case" format.
2. For tool ‘parameters’ and ‘returns’, ‘type’ should be a valid JSON type, i.e. one of ["string", "integer", "number", "boolean", "array", "object", "null"].
3. For tool ‘parameters’ and ‘returns’, ‘description’ should be a clear and concise description of the parameter or return, without any ambiguity. If the parameter or return is subject to some specific format or value constraints, these constraints should be explicitly specified in ‘description’.
4. If a tool ‘parameters’ or ‘returns’ is an object, the exact fields of the object should be explicitly specified in ‘description’.

### Format example
You should output the toolkit as a JSON object, strictly following the structure shown in the following example tool spec:
*WARNING! ** You need to check your JSON format carefully!!! Output the detailed toolkit spec as:
“‘json
<Output toolkit spec follows [format description]>...
...
Try to generate tools that users might use in their daily lives.
Do not output the same string repeatedly!!!
Make sure to meet the above format requirements, especially do not lose "parameters"
Question: You need to generate tools for the following toolkits related to map navigation: Reference message: Turn on navigation sound, turn off navigation sound, change navigation sound, query real-time bus, view my footprints, find charging piles, add common addresses, view common addresses, delete common addresses, add favorite places, view favorite places, delete favorite places, add favorite routes, view favorite routes, delete common routes

E.2 Query Generation

Generating Single-turn Queries
You are now a mobile phone user. Your task is to fill in the corresponding parameters according to the actual scenario based on the provided description and parameters, and then generate a user request
based on the filled parameters.

Please make sure that the generated user requests are different from each other. You can feel free to use different sentence styles, such as instructions or requests, and change the degree of detail as needed.
Related requirements:
1. Use as few questions and polite tones as possible;
2. Natural questions similar to human conversations need to be more humane and colloquial, and in line with people’s daily life
3. The parameters in the generated user request should be as reasonable as possible. For example, if it involves image parameters, the user request needs to have representative information like test.jpg
4. Fill in parameters carefully, output the corresponding JSON format and wrap it with ““json...““
5. Information such as test.jpg contained in parameters also needs to be included in the user request!!
6. All parameters must be filled in all given parameters!! Do not miss any one!
7. The user request needs to be wrapped with <user0>...<user1>. Your task is to generate natural and colloquial user requests, make sure to clearly mention the tool name {name_for_human} in the request, and avoid using stiff or direct imperative tones. For example, do not generate requests like ‘Dianping, show me my group purchases’, but use more humane and daily conversation expressions, such as ‘Help me see the group purchases on Dianping’. Please generate a user request that meets the specifications according to this requirement.
8. You can generate parameters first, and then generate user requests based on the generated parameters.
9. The text descriptions in parameters must be in Chinese!
10. For time parameters, please use Chinese text! And the expressions before and after must be consistent!
11. When no parameter definition is given, please generate a statement that does not contain any parameters! Output Format:
1. Parameters: ““json parameter dictionary““. <user0>[user request]<user1>
2. Parameters: ““json parameter dictionary““. <user0>[user request]<user1>
… Here are two examples for your reference:
description: Modify alarm time, ring mode, weekday settings, custom settings
parameters:

Must include parameters:
Output:
1. Parameters:““json{}““. <user0>I want to modify the alarm time<user1>

description: Modify the alarm time, ring mode, weekday settings, custom settings
parameters:[{"name":    "clock_time","type":    "string","description":    "Alarm time","required":  false},{"name":    "target_time","type":    "string","description":    "Modified target time","required":  true}]
Must include parameters: clock_time, target_time Output:
1. Parameters:““json["clock_time":  "8 o'clock","target_time":  "15 o'clock"]““. <user0>Change the alarm at 8 o’clock to 15 o’clock in the afternoon<user1> The description of the tool and the corresponding parameters are as follows:
description:{description}
parameters:{parameters}
Must include parameters:{param_list} Please refer to the example and generate {num} [user requests] and [parameter dictionaries] that conform to the output format according to the above requirements and tool conditions.
E.3  Parameters Selection

Selecting parameters

Please select the specified number of parameters that best suit your daily habits based on the tool description and corresponding parameter definitions. ## Selection Principles:
1. Simulate daily user usage scenarios and give priority to parameters that are more commonly used in daily life ## Output Format:
1. Output your thoughts first, then output the parameter name you selected, and wrap it with “‘…’”
2. Use commas to separate parameter names ## For examples:
Tool description: Create a new alarm
Parameter definition:
["name": "clock_time_name","type": "string","description": "Alarm name, empty by default","required": false,"name": "clock_time","type": "string","description": "Alarm time","required": true,"name": "kwargs","type": "array","description": "Ring mode, weekday settings, custom settings","required": false]
Output 2 parameters: clock_time, clock_time_name

Start now! Tool description: {tool_desc}
Parameter definition: {parameters}
Output {num} parameters:

E.4  LLM Check Prompt

LLM Check Prompt

## System
You are an expert who can accurately judge the information matching relationship between texts, and you have a keen insight into the information matching between two structured texts.

## Task Description
1. Given a query, model_output and the label of the corresponding tool call, your task is to determine whether the model_output and label both express the same semantic meaning and are derived from the information found in the query. output your judgment True/False

2. If model_output and label have different number of parameters, such as {'A':1,'B':2} and {'A':1}, output False
3. model_output and label do not have to be exactly the same. As long as they express the same meaning and can solve query needs, only True is output.
4. If model_output is the same as label, or only a slight difference between them in terms of singular and plural, output True! such as {} and { }, {"departure": "office"} and {"departure": "Offices"}, you need output True

## Output Format
1. Following Task Description, Only output True/False, Never answer more text! For examples:
1. Different characters but same semantics, output True:
query: Add a note of the address of Starbucks Zhongguancun store on the map, and write "frequently visited writing place"
label: {"address": "Starbucks Zhongguancun store", "remake": "often come to write"}
model_output: {"address": "Starbucks Zhongguancun store", "remake": "frequently visited writing place"}
Output: True
2. model_output contains parameters that are not in label, output False:
query: How many times have you ridden this year?
label: {}
model_output: {"time": "this year"}
answer: False
3. The model_output is missing the parameters of label, output False:
query: I want to know which five-star hotels are there in Nanshan District, Shenzhen
label: {"destination": "Nanshan District, Shenzhen", "checkin_date":"", "checkout_date":"", "price_range":"", "kwargs":"five-star hotel"}
model_output: {"destination": "Nanshan District, Shenzhen", "checkin_date":"", "checkout_date":"", "kwargs":"five-star hotel"}
answer: False
4. Time hallucination occurs, output False:
query: Make an appointment to test drive Xiaomi su7 at 10 am this Saturday
label: {"test_drive_time": "This Saturday at 10 am", "name":"", "phone":"", "kwargs":""}
model_output: {"test_drive_time": "2023-04-20 10:00", "name":"", "phone":"", "kwargs":""} answer: False
5. The time format is different, but it expresses the same time, output True:
query: Make an appointment to test drive Xiaomi su7 at 10 am this Saturday
label: {"test_drive_time": "This Saturday at 10 am", "name":"", "phone":"", "kwargs":""}
model_output: {"test_drive_time": "This Saturday at 10:00", "name":"", "phone":"", "kwargs":""}
answer: True
6. All are empty characters, output True:
query: Open TikTok
label: {}
model_output: {}
answer: True
Start now!
query: {query}
label: {label}
model_output: {model_output}
answer:

E.5 Irrelevant tools selection

Irrelevant tools selection
Given a query and the corresponding tool description and parameter definition, please judge whether the tool can complete the query.
## Output Format:
Only outputs boolean strings: True/False

Start now!
query: {query}
tool:{tool}
answer:

E.6 Generation with external individual information

classify entity
Please determine whether a parameter is in the entity category
## Output Format:
Please select which entity it belongs to. The selected entity category is enclosed in “‘...‘”
entity:{entities}
Parameter name and description:{param_desc}
Output:

pronoun generation
Please rewrite the following content into a sentence in the form of adjective + noun that is more in line with daily habits and has more specific information.
For example:
Input: The time mentioned in the message received from SMS
Requirement: Cannot contain specific time
Output: The time mentioned in the text message sent to me by Tom

Start now! Output only the rewritten 1 sentence!

Input: {entity} mentioned in the message received from {Tool_} Requirement: Cannot contain specific {entity} Output:

________________________external individual information simulation________________________

Please simulate a message in {app} as required

## Related requirements
The content must involve entity;
The content should be as detailed as possible, and no referential content such as that or this should appear. The message should contain a clear {entity}: {param_value} The content must not contain the following: {param_forbid} ## Output format
Only output simulated messages, do not ask or output other content!

Simulated message:

________________________rewrite query________________________

Please generate the corresponding user query according to the label and the corresponding tool function description
## Related requirements
1. Try to be consistent with the language of daily conversation interaction, smooth and concise
## Output format:
Only output the query, do not include other information

Tool description: {desc}
label: {label}
query:

E.7 Multi-turn Prompt

________________________Question prompt________________________

You are a function call agent, currently processing the {description} function call task, but the parameters {miss_args_desc} are missing. Please ask the user for the missing parameter values

**Requirements**
- Do not include the existing parameters: {args_desc}
- Be brief and concise, do not add additional explanations
- Ask only about the missing parameters
- Ask multiple missing parameters together, do not use special symbols to separate, such as colons, newlines, etc.
- Just ask questions, do not tell the user how to answer
F Extra Experiment Details

Our experiments are all conducted in the HammerBench datasets generated by open-source LLMs. When generating these data, we only summarized the behaviors and scenarios based on anonymous user logs, without involving any privacy information of users. And HammerBench’s evaluations strictly follow the license of the corresponding LLMs. Qwen2.5-72B-Instruct, Qwen2.5-7B-Instruct(Qwen Team, 2024) and ToolACE-8B(Liu et al., 2024a) are Apache-2.0 LICENSE; xLAM-7b-fc-r(Liu et al., 2024b) is CC-by-NC-4.0 LICENSE; Llama-3.1-70B-Instruct(Dubey et al., 2024) and Llama-3.1-8B-Instruct are llama3.1 LICENSE; Minstral-8B-Instruct(AI, 2024) is mrl LICENSE. They all allow developers to use their technology for non-commercial purposes and to support research work. And GPT-4o(Hurst et al., 2024) and Claude3.5-sonnet(Anthropic, 2024) are also widely used in research. Without using LLM judge, it takes approximately 90 minutes for the 7B LLMs to inference all single turn(6500+ samples) and multi turn data(6400+ samples) using a single A100 80G GPU. As for GPT-4o, one evaluation approximately cost 75 dollars.

The experiment result of single-turn dataset is shown in Table 8. As for single-turn dataset with missing arguments (Imperfect), PHR is much larger than other single-turn dataset. It reveals that user query with missing arguments can easily lead to parameter hallucinations. In these cases, LLMs tend to fill in missing arguments based on their internal model of the world rather than the actual user input, as detailed in Appendix C.1. Since snapshots are often imperfect, these inaccuracies significantly impact multi-turn success rates, further highlighting the challenges posed by incomplete or unclear user input. As a result, the overall effectiveness of function-calling tasks is diminished when the instructions provided to the model are less than ideal. And the single-turn dataset with external individual information has the lower end-to-end Acc. .

We also evaluate function-calling snapshots across three types for detailed investigation. (1) every turn throughout the conversation in Table 12, (2) the turn when the user changes slot values in Table 13, and (3) the first and last turn of the conversation in Table 6. Some additional observations are as follows.

Fluctuations in Conversation Success Rate Across Diverse Q&A Trajectories: As shown in Table 12, different conversation trajectories introduce varying degrees of disturbance to the overall success rate of function calling. Consistent with expectations, we find that the progress rate tends to be higher when slot values are provided more rapidly within the conversation (e.g. mQmA and sQmA, where users provide multiple slot values at once and thus answer models’ questions faster.), highlighting the impact of the timing and flow of user input on function-calling accuracy.

Generalization and Handling of Argument Shifts and External information: As shown in Table 13, open-source LLMs are generally slow to detect and adjust to slot overriding. Only GPT-4o and Claude3.5-sonnet, with its more robust generalization abilities, is able to perform this task more effectively. As for external individual information, external noise increases Parameter Mismatch Rate (PMR) and causes the model to miss crucial arguments, hindering its ability to fill slots correctly.

Imperfect Instructions and Parameter Name Hallucinations: As illustrated in Table 6, the PHR in the first snapshot is significantly higher than in the last snapshot for all LLMs, with the PHR being positively correlated with the rate of parameter name hallucinations. Furthermore, when all parameter names are predicted correctly, the end-to-end function call is typically accurate as well, i.e. Other Error Rate≈ 0%, emphasizing the importance of precise initial instruction. We can see similar observations from Table 8.
<table>
  <tr>
    <th>Data type</th>
    <th>Model</th>
    <th>Func. Acc.</th>
    <th>PHR</th>
    <th>PMR</th>
    <th>Acc.</th>
    <th>Irrelevant</th>
  </tr>
  <tr>
    <td rowspan="8">Perfect</td>
    <td>GPT-4o (Prompt)</td>
    <td>89.74%</td>
    <td>0.16%</td>
    <td>10.63%</td>
    <td>78.69%</td>
    <td>64.38%</td>
  </tr>
  <tr>
    <td>Claude3.5-sonnet (Prompt)</td>
    <td>89.89%</td>
    <td>0.26%</td>
    <td>7.72%</td>
    <td>79.86%</td>
    <td>69.35%</td>
  </tr>
  <tr>
    <td>Owen2.5-72B-Instruct (Prompt)</td>
    <td>90.50%</td>
    <td>1.98%</td>
    <td>3.92%</td>
    <td>80.86%</td>
    <td>74.59%</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct (Prompt)</td>
    <td>91.87%</td>
    <td>6.64%</td>
    <td>4.27%</td>
    <td>75.99%</td>
    <td>14.19%</td>
  </tr>
  <tr>
    <td>Owen2.5-7B-Instruct (Prompt)</td>
    <td>89.41%</td>
    <td>5.50%</td>
    <td>3.96%</td>
    <td>75.75%</td>
    <td>37.36%</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct (Prompt)</td>
    <td>87.54%</td>
    <td>5.22%</td>
    <td>3.56%</td>
    <td>75.99%</td>
    <td>9.29%</td>
  </tr>
  <tr>
    <td>Ministral-8B-Instruct-2410 (Prompt)</td>
    <td>88.32%</td>
    <td>0.43%</td>
    <td>2.23%</td>
    <td>78.02%</td>
    <td>1.14%</td>
  </tr>
  <tr>
    <td>Hammer2.1-7b (FC)</td>
    <td>80.29%</td>
    <td>1.06%</td>
    <td>7.95%</td>
    <td>68.85%</td>
    <td>89.35%</td>
  </tr>
  <tr>
    <td>xLAM-7b-fc-r (FC)</td>
    <td>83.17%</td>
    <td>0.34%</td>
    <td>5.00%</td>
    <td>73.48%</td>
    <td>49.76%</td>
  </tr>
  <tr>
    <td>ToolACE-8B (FC)</td>
    <td>91.44%</td>
    <td>0.78%</td>
    <td>4.96%</td>
    <td>78.82%</td>
    <td>22.98%</td>
  </tr>
  <tr>
    <td rowspan="8">Imperfect</td>
    <td>GPT-4o (Prompt)</td>
    <td>88.86%</td>
    <td>7.61%</td>
    <td>4.48%</td>
    <td>78.77%</td>
    <td>71.17%</td>
  </tr>
  <tr>
    <td>Claude3.5-sonnet (Prompt)</td>
    <td>88.94%</td>
    <td>8.15%</td>
    <td>2.49%</td>
    <td>79.38%</td>
    <td>67.90%</td>
  </tr>
  <tr>
    <td>Owen2.5-72B-Instruct (Prompt)</td>
    <td>84.59%</td>
    <td>16.27%</td>
    <td>1.68%</td>
    <td>68.64%</td>
    <td>78.97%</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct (Prompt)</td>
    <td>91.20%</td>
    <td>20.87%</td>
    <td>1.90%</td>
    <td>69.47%</td>
    <td>19.04%</td>
  </tr>
  <tr>
    <td>Owen2.5-7B-Instruct (Prompt)</td>
    <td>82.09%</td>
    <td>26.72%</td>
    <td>2.52%</td>
    <td>58.08%</td>
    <td>49.04%</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct (Prompt)</td>
    <td>89.35%</td>
    <td>40.72%</td>
    <td>1.80%</td>
    <td>51.08%</td>
    <td>8.50%</td>
  </tr>
  <tr>
    <td>Ministral-8B-Instruct-2410 (Prompt)</td>
    <td>75.03%</td>
    <td>54.17%</td>
    <td>2.59%</td>
    <td>32.19%</td>
    <td>6.09%</td>
  </tr>
  <tr>
    <td>Hammer2.1-7b (FC)</td>
    <td>81.88%</td>
    <td>24.38%</td>
    <td>3.02%</td>
    <td>59.29%</td>
    <td>89.23%</td>
  </tr>
  <tr>
    <td>xLAM-7b-fc-r (FC)</td>
    <td>86.45%</td>
    <td>41.86%</td>
    <td>2.61%</td>
    <td>48.82%</td>
    <td>34.51%</td>
  </tr>
  <tr>
    <td>ToolACE-8B (FC)</td>
    <td>82.53%</td>
    <td>33.58%</td>
    <td>3.96%</td>
    <td>51.94%</td>
    <td>29.37%</td>
  </tr>
  <tr>
    <td rowspan="8">External</td>
    <td>GPT-4o (Prompt)</td>
    <td>87.91%</td>
    <td>2.81%</td>
    <td>29.11%</td>
    <td>56.16%</td>
    <td>73.02%</td>
  </tr>
  <tr>
    <td>Claude3.5-sonnet (Prompt)</td>
    <td>82.13%</td>
    <td>2.49%</td>
    <td>25.27%</td>
    <td>55.83%</td>
    <td>67.23%</td>
  </tr>
  <tr>
    <td>Owen2.5-72B-Instruct (Prompt)</td>
    <td>93.10%</td>
    <td>5.58%</td>
    <td>12.52%</td>
    <td>67.40%</td>
    <td>65.70%</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct (Prompt)</td>
    <td>93.10%</td>
    <td>5.72%</td>
    <td>12.52%</td>
    <td>67.40%</td>
    <td>4.26%</td>
  </tr>
  <tr>
    <td>Owen2.5-7B-Instruct (Prompt)</td>
    <td>88.51%</td>
    <td>9.90%</td>
    <td>8.94%</td>
    <td>58.89%</td>
    <td>36.59%</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct (Prompt)</td>
    <td>89.87%</td>
    <td>9.94%</td>
    <td>8.52%</td>
    <td>57.61%</td>
    <td>5.96%</td>
  </tr>
  <tr>
    <td>Ministral-8B-Instruct-2410 (Prompt)</td>
    <td>90.12%</td>
    <td>11.89%</td>
    <td>6.70%</td>
    <td>57.70%</td>
    <td>4.34%</td>
  </tr>
  <tr>
    <td>Hammer2.1-7b (FC)</td>
    <td>77.87%</td>
    <td>7.65%</td>
    <td>13.44%</td>
    <td>50.21%</td>
    <td>90.63%</td>
  </tr>
  <tr>
    <td>xLAM-7b-fc-r (FC)</td>
    <td>86.89%</td>
    <td>10.08%</td>
    <td>15.08%</td>
    <td>48.85%</td>
    <td>45.78%</td>
  </tr>
  <tr>
    <td>ToolACE-8B (FC)</td>
    <td>92.25%</td>
    <td>11.71%</td>
    <td>10.70%</td>
    <td>56.93%</td>
    <td>27.40%</td>
  </tr>
  <tr>
    <td rowspan="8">Overall</td>
    <td>GPT-4o (Prompt)</td>
    <td>88.83%</td>
    <td>3.52%</td>
    <td>14.74%</td>
    <td>71.21%</td>
    <td>69.52%</td>
  </tr>
  <tr>
    <td>Claude3.5-sonnet (Prompt)</td>
    <td>86.99%</td>
    <td>3.63%</td>
    <td>11.82%</td>
    <td>71.69%</td>
    <td>68.16%</td>
  </tr>
  <tr>
    <td>Owen2.5-72B-Instruct (Prompt)</td>
    <td>89.40%</td>
    <td>7.94%</td>
    <td>6.04%</td>
    <td>72.30%</td>
    <td>73.09%</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct (Prompt)</td>
    <td>92.25%</td>
    <td>11.07%</td>
    <td>6.84%</td>
    <td>68.34%</td>
    <td>12.49%</td>
  </tr>
  <tr>
    <td>Owen2.5-7B-Instruct (Prompt)</td>
    <td>86.67%</td>
    <td>14.04%</td>
    <td>5.14%</td>
    <td>64.24%</td>
    <td>41.00%</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct (Prompt)</td>
    <td>88.86%</td>
    <td>18.62%</td>
    <td>4.35%</td>
    <td>60.77%</td>
    <td>8.13%</td>
  </tr>
  <tr>
    <td>Ministral-8B-Instruct-2410 (Prompt)</td>
    <td>84.49%</td>
    <td>21.16%</td>
    <td>5.51%</td>
    <td>59.97%</td>
    <td>3.86%</td>
  </tr>
  <tr>
    <td>Hammer2.1-7b (FC)</td>
    <td>80.01%</td>
    <td>11.03%</td>
    <td>8.13%</td>
    <td>59.45%</td>
    <td>89.74%</td>
  </tr>
  <tr>
    <td>xLAM-7b-fc-r (FC)</td>
    <td>85.50%</td>
    <td>17.16%</td>
    <td>7.56%</td>
    <td>57.05%</td>
    <td>50.02%</td>
  </tr>
  <tr>
    <td>ToolACE-8B (FC)</td>
    <td>88.74%</td>
    <td>15.35%</td>
    <td>6.54%</td>
    <td>62.56%</td>
    <td>26.58%</td>
  </tr>
</table>

Table 8: Experiment result for single-turn dataset.
<table>
  <tr>
    <th>Data type</th>
    <th>Model</th>
    <th colspan="2">Func.</th>
    <th colspan="2">Acc.</th>
    <th colspan="2">PHR</th>
    <th colspan="2">PMB</th>
    <th colspan="2">Acc.</th>
    <th colspan="2">PR</th>
    <th colspan="2">SR</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>Func.</th>
    <th>Acc.</th>
    <th>PHR</th>
    <th>PMB</th>
    <th>Acc.</th>
    <th>PR</th>
    <th>Acc.</th>
    <th>SR</th>
  </tr>
  <tr>
    <td rowspan="14">sQaA</td>
    <td>GPT-4o (Prompt)</td>
    <td>89.85%</td>
    <td>9.31%</td>
    <td>5.11%</td>
    <td>72.52%</td>
    <td>66.03%</td>
    <td>54.95%</td>
  </tr>
  <tr>
    <td>Claude 3.5-sommet (Prompt)</td>
    <td>91.08%</td>
    <td>9.02%</td>
    <td>4.28%</td>
    <td>70.64%</td>
    <td>62.37%</td>
    <td>56.55%</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct (Prompt)</td>
    <td>82.17%</td>
    <td>12.29%</td>
    <td>3.55%</td>
    <td>64.59%</td>
    <td>56.28%</td>
    <td>45.55%</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct (Prompt)</td>
    <td>91.83%</td>
    <td>12.86%</td>
    <td>1.71%</td>
    <td>75.05%</td>
    <td>64.72%</td>
    <td>55.19%</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct (Prompt)</td>
    <td>82.67%</td>
    <td>19.50%</td>
    <td>5.23%</td>
    <td>60.08%</td>
    <td>48.90%</td>
    <td>37.62%</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct (Prompt)</td>
    <td>90.90%</td>
    <td>22.80%</td>
    <td>19.76%</td>
    <td>51.33%</td>
    <td>38.43%</td>
    <td>18.56%</td>
  </tr>
  <tr>
    <td>Minstral-8B-Instruct-2410 (Prompt)</td>
    <td>71.28%</td>
    <td>33.17%</td>
    <td>8.11%</td>
    <td>33.36%</td>
    <td>26.26%</td>
    <td>12.96%</td>
  </tr>
  <tr>
    <td>Hammer2.1-7b (FC)</td>
    <td>81.68%</td>
    <td>15.71%</td>
    <td>6.13%</td>
    <td>62.26%</td>
    <td>52.37%</td>
    <td>41.08%</td>
  </tr>
  <tr>
    <td>sLAM-7b-fc (FC)</td>
    <td>91.33%</td>
    <td>29.19%</td>
    <td>6.81%</td>
    <td>58.03%</td>
    <td>39.28%</td>
    <td>32.17%</td>
  </tr>
  <tr>
    <td>ToolACE-SB (FC)</td>
    <td>80.69%</td>
    <td>28.11%</td>
    <td>3.11%</td>
    <td>54.54%</td>
    <td>42.55%</td>
    <td>26.98%</td>
  </tr>
  <tr>
    <td>mQmA</td>
    <td>GPT-4o (Prompt)</td>
    <td>90.84%</td>
    <td>7.12%</td>
    <td>5.60%</td>
    <td>75.63%</td>
    <td>73.22%</td>
    <td>64.35%</td>
  </tr>
  <tr>
    <td></td>
    <td>Claude 3.5-sommet (Prompt)</td>
    <td>90.84%</td>
    <td>7.93%</td>
    <td>4.43%</td>
    <td>75.52%</td>
    <td>73.74%</td>
    <td>65.59%</td>
  </tr>
  <tr>
    <td></td>
    <td>Qwen2.5-72B-Instruct (Prompt)</td>
    <td>81.58%</td>
    <td>10.58%</td>
    <td>2.55%</td>
    <td>61.72%</td>
    <td>64.75%</td>
    <td>57.67%</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama-3.1-70B-Instruct (Prompt)</td>
    <td>91.58%</td>
    <td>12.22%</td>
    <td>2.54%</td>
    <td>73.23%</td>
    <td>69.02%</td>
    <td>54.40%</td>
  </tr>
  <tr>
    <td></td>
    <td>Qwen2.5-7B-Instruct (Prompt)</td>
    <td>82.42%</td>
    <td>20.64%</td>
    <td>5.29%</td>
    <td>58.12%</td>
    <td>53.06%</td>
    <td>44.55%</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama-3.1-8B-Instruct (Prompt)</td>
    <td>90.59%</td>
    <td>23.65%</td>
    <td>10.83%</td>
    <td>57.70%</td>
    <td>51.15%</td>
    <td>40.59%</td>
  </tr>
  <tr>
    <td></td>
    <td>Minstral-8B-Instruct-2410 (Prompt)</td>
    <td>71.28%</td>
    <td>35.32%</td>
    <td>6.48%</td>
    <td>46.99%</td>
    <td>28.91%</td>
    <td>21.28%</td>
  </tr>
  <tr>
    <td></td>
    <td>Hammer2.1-7b (FC)</td>
    <td>81.68%</td>
    <td>14.87%</td>
    <td>6.08%</td>
    <td>60.88%</td>
    <td>57.57%</td>
    <td>43.53%</td>
  </tr>
  <tr>
    <td></td>
    <td>sLAM-7b-fc (FC)</td>
    <td>91.33%</td>
    <td>33.79%</td>
    <td>7.64%</td>
    <td>52.53%</td>
    <td>40.74%</td>
    <td>34.40%</td>
  </tr>
  <tr>
    <td></td>
    <td>ToolACE-SB (FC)</td>
    <td>80.44%</td>
    <td>27.54%</td>
    <td>4.96%</td>
    <td>52.63%</td>
    <td>46.39%</td>
    <td>33.41%</td>
  </tr>
  <tr>
    <td>mQsA</td>
    <td>GPT-4o (Prompt)</td>
    <td>90.34%</td>
    <td>9.17%</td>
    <td>7.88%</td>
    <td>73.00%</td>
    <td>68.52%</td>
    <td>59.15%</td>
  </tr>
  <tr>
    <td></td>
    <td>Claude 3.5-sommet (Prompt)</td>
    <td>90.59%</td>
    <td>9.15%</td>
    <td>6.42%</td>
    <td>73.34%</td>
    <td>68.93%</td>
    <td>59.65%</td>
  </tr>
  <tr>
    <td></td>
    <td>Qwen2.5-72B-Instruct (Prompt)</td>
    <td>81.93%</td>
    <td>13.74%</td>
    <td>5.11%</td>
    <td>63.90%</td>
    <td>56.93%</td>
    <td>47.27%</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama-3.1-70B-Instruct (Prompt)</td>
    <td>91.83%</td>
    <td>15.31%</td>
    <td>3.30%</td>
    <td>74.47%</td>
    <td>61.86%</td>
    <td>52.37%</td>
  </tr>
  <tr>
    <td></td>
    <td>Qwen2.5-7B-Instruct (Prompt)</td>
    <td>82.17%</td>
    <td>25.62%</td>
    <td>7.76%</td>
    <td>54.44%</td>
    <td>62.62%</td>
    <td>42.42%</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama-3.1-8B-Instruct (Prompt)</td>
    <td>90.59%</td>
    <td>27.01%</td>
    <td>19.78%</td>
    <td>48.59%</td>
    <td>38.42%</td>
    <td>19.05%</td>
  </tr>
  <tr>
    <td></td>
    <td>Minstral-8B-Instruct-2410 (Prompt)</td>
    <td>71.28%</td>
    <td>45.38%</td>
    <td>7.79%</td>
    <td>32.94%</td>
    <td>19.02%</td>
    <td>8.91%</td>
  </tr>
  <tr>
    <td></td>
    <td>Hammer2.1-7b (FC)</td>
    <td>81.68%</td>
    <td>26.43%</td>
    <td>11.26%</td>
    <td>52.63%</td>
    <td>41.50%</td>
    <td>25.24%</td>
  </tr>
  <tr>
    <td></td>
    <td>sLAM-7b-fc (FC)</td>
    <td>91.33%</td>
    <td>33.79%</td>
    <td>7.75%</td>
    <td>52.63%</td>
    <td>41.09%</td>
    <td>34.65%</td>
  </tr>
  <tr>
    <td></td>
    <td>ToolACE-SB (FC)</td>
    <td>80.44%</td>
    <td>27.00%</td>
    <td>4.06%</td>
    <td>51.68%</td>
    <td>46.16%</td>
    <td>32.43%</td>
  </tr>
  <tr>
    <td>w/ SO</td>
    <td>GPT-4o (Prompt)</td>
    <td>90.68%</td>
    <td>8.52%</td>
    <td>4.07%</td>
    <td>72.98%</td>
    <td>68.65%</td>
    <td>57.51%</td>
  </tr>
  <tr>
    <td></td>
    <td>Claude 3.5-sommet (Prompt)</td>
    <td>90.44%</td>
    <td>8.83%</td>
    <td>3.58%</td>
    <td>72.54%</td>
    <td>67.01%</td>
    <td>50.17%</td>
  </tr>
  <tr>
    <td></td>
    <td>Qwen2.5-72B-Instruct (Prompt)</td>
    <td>85.08%</td>
    <td>13.62%</td>
    <td>5.08%</td>
    <td>65.40%</td>
    <td>60.25%</td>
    <td>52.09%</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama-3.1-70B-Instruct (Prompt)</td>
    <td>91.47%</td>
    <td>16.17%</td>
    <td>1.56%</td>
    <td>70.25%</td>
    <td>62.01%</td>
    <td>52.09%</td>
  </tr>
  <tr>
    <td></td>
    <td>Qwen2.5-7B-Instruct (Prompt)</td>
    <td>82.88%</td>
    <td>23.21%</td>
    <td>3.92%</td>
    <td>54.21%</td>
    <td>45.78%</td>
    <td>33.05%</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama-3.1-8B-Instruct (Prompt)</td>
    <td>90.10%</td>
    <td>27.30%</td>
    <td>14.56%</td>
    <td>46.44%</td>
    <td>37.24%</td>
    <td>22.54%</td>
  </tr>
  <tr>
    <td></td>
    <td>Minstral-8B-Instruct-2410 (Prompt)</td>
    <td>74.28%</td>
    <td>37.75%</td>
    <td>10.44%</td>
    <td>40.22%</td>
    <td>26.21%</td>
    <td>10.99%</td>
  </tr>
  <tr>
    <td></td>
    <td>Hammer2.1-7b (FC)</td>
    <td>82.85%</td>
    <td>19.92%</td>
    <td>5.03%</td>
    <td>57.85%</td>
    <td>49.83%</td>
    <td>36.83%</td>
  </tr>
  <tr>
    <td></td>
    <td>sLAM-7b-fc (FC)</td>
    <td>89.89%</td>
    <td>34.70%</td>
    <td>6.51%</td>
    <td>48.37%</td>
    <td>34.44%</td>
    <td>24.94%</td>
  </tr>
  <tr>
    <td></td>
    <td>ToolACE-SB (FC)</td>
    <td>81.51%</td>
    <td>32.96%</td>
    <td>3.05%</td>
    <td>50.25%</td>
    <td>40.07%</td>
    <td>26.87%</td>
  </tr>
  <tr>
    <td>w/ mSv</td>
    <td>GPT-4o (Prompt)</td>
    <td>90.68%</td>
    <td>7.84%</td>
    <td>6.45%</td>
    <td>61.98%</td>
    <td>56.45%</td>
    <td>18.53%</td>
  </tr>
  <tr>
    <td></td>
    <td>Claude 3.5-sommet (Prompt)</td>
    <td>90.02%</td>
    <td>8.02%</td>
    <td>5.36%</td>
    <td>69.68%</td>
    <td>63.94%</td>
    <td>42.89%</td>
  </tr>
  <tr>
    <td></td>
    <td>Qwen2.5-72B-Instruct (Prompt)</td>
    <td>88.44%</td>
    <td>12.35%</td>
    <td>4.21%</td>
    <td>60.97%</td>
    <td>53.20%</td>
    <td>32.06%</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama-3.1-70B-Instruct (Prompt)</td>
    <td>91.62%</td>
    <td>14.44%</td>
    <td>5.21%</td>
    <td>68.87%</td>
    <td>58.64%</td>
    <td>41.39%</td>
  </tr>
  <tr>
    <td></td>
    <td>Qwen2.5-7B-Instruct (Prompt)</td>
    <td>83.06%</td>
    <td>21.07%</td>
    <td>4.85%</td>
    <td>51.85%</td>
    <td>41.75%</td>
    <td>18.15%</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama-3.1-8B-Instruct (Prompt)</td>
    <td>89.84%</td>
    <td>25.19%</td>
    <td>13.85%</td>
    <td>48.30%</td>
    <td>34.97%</td>
    <td>15.80%</td>
  </tr>
  <tr>
    <td></td>
    <td>Minstral-8B-Instruct-2410 (Prompt)</td>
    <td>75.25%</td>
    <td>33.46%</td>
    <td>4.81%</td>
    <td>32.77%</td>
    <td>22.16%</td>
    <td>7.53%</td>
  </tr>
  <tr>
    <td></td>
    <td>Hammer2.1-7b (FC)</td>
    <td>83.86%</td>
    <td>17.78%</td>
    <td>4.81%</td>
    <td>52.83%</td>
    <td>48.87%</td>
    <td>24.46%</td>
  </tr>
  <tr>
    <td></td>
    <td>sLAM-7b-fc (FC)</td>
    <td>89.18%</td>
    <td>26.90%</td>
    <td>15.51%</td>
    <td>43.23%</td>
    <td>29.09%</td>
    <td>4.80%</td>
  </tr>
  <tr>
    <td></td>
    <td>ToolACE-SB (FC)</td>
    <td>81.93%</td>
    <td>29.82%</td>
    <td>3.82%</td>
    <td>45.74%</td>
    <td>35.07%</td>
    <td>11.57%</td>
  </tr>
  <tr>
    <td>w/ External</td>
    <td>GPT-4o (Prompt)</td>
    <td>90.81%</td>
    <td>5.52%</td>
    <td>13.36%</td>
    <td>69.64%</td>
    <td>66.34%</td>
    <td>49.18%</td>
  </tr>
  <tr>
    <td></td>
    <td>Claude 3.5-sommet (Prompt)</td>
    <td>90.40%</td>
    <td>6.22%</td>
    <td>9.18%</td>
    <td>71.83%</td>
    <td>68.62%</td>
    <td>54.28%</td>
  </tr>
  <tr>
    <td></td>
    <td>Qwen2.5-72B-Instruct (Prompt)</td>
    <td>84.48%</td>
    <td>12.36%</td>
    <td>8.98%</td>
    <td>61.12%</td>
    <td>59.58%</td>
    <td>45.91%</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama-3.1-70B-Instruct (Prompt)</td>
    <td>92.10%</td>
    <td>12.41%</td>
    <td>2.61%</td>
    <td>69.43%</td>
    <td>63.79%</td>
    <td>47.79%</td>
  </tr>
  <tr>
    <td></td>
    <td>Qwen2.5-7B-Instruct (Prompt)</td>
    <td>81.22%</td>
    <td>19.73%</td>
    <td>6.35%</td>
    <td>53.87%</td>
    <td>50.04%</td>
    <td>34.89%</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama-3.1-8B-Instruct (Prompt)</td>
    <td>89.79%</td>
    <td>22.70%</td>
    <td>14.95%</td>
    <td>50.14%</td>
    <td>43.56%</td>
    <td>25.51%</td>
  </tr>
  <tr>
    <td></td>
    <td>Minstral-8B-Instruct-2410 (Prompt)</td>
    <td>73.67%</td>
    <td>33.52%</td>
    <td>9.88%</td>
    <td>35.00%</td>
    <td>25.06%</td>
    <td>14.08%</td>
  </tr>
  <tr>
    <td></td>
    <td>Hammer2.1-7b (FC)</td>
    <td>82.85%</td>
    <td>15.74%</td>
    <td>17.72%</td>
    <td>44.33%</td>
    <td>46.32%</td>
    <td>25.50%</td>
  </tr>
  <tr>
    <td></td>
    <td>sLAM-7b-fc (FC)</td>
    <td>89.18%</td>
    <td>26.80%</td>
    <td>8.18%</td>
    <td>48.39%</td>
    <td>39.96%</td>
    <td>15.91%</td>
  </tr>
  <tr>
    <td></td>
    <td>ToolACE-SB (FC)</td>
    <td>83.87%</td>
    <td>28.83%</td>
    <td>5.45%</td>
    <td>49.15%</td>
    <td>43.61%</td>
    <td>26.93%</td>
  </tr>
  <tr>
    <td>Overall</td>
    <td>GPT-4o (Prompt)</td>
    <td>90.67</td>
    <td>7.96</td>
    <td>7.21</td>
    <td>71.40</td>
    <td>67.14</td>
    <td>52.00</td>
  </tr>
  <tr>
    <td></td>
    <td>Claude 3.5-sommet (Prompt)</td>
    <td>90.67</td>
    <td>8.13</td>
    <td>5.73</td>
    <td>73.92</td>
    <td>70.15</td>
    <td>58.84</td>
  </tr>
  <tr>
    <td></td>
    <td>Qwen2.5-72B-Instruct (Prompt)</td>
    <td>83.45</td>
    <td>12.32</td>
    <td>5.24</td>
    <td>65.03</td>
    <td>59.90</td>
    <td>49.19</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama-3.1-70B-Instruct (Prompt)</td>
    <td>91.67</td>
    <td>13.59</td>
    <td>3.16</td>
    <td>71.56</td>
    <td>63.70</td>
    <td>52.96</td>
  </tr>
  <tr>
    <td></td>
    <td>Qwen2.5-7B-Instruct (Prompt)</td>
    <td>83.37</td>
    <td>21.46</td>
    <td>5.37</td>
    <td>55.75</td>
    <td>48.03</td>
    <td>34.13</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama-3.1-8B-Instruct (Prompt)</td>
    <td>90.37</td>
    <td>24.72</td>
    <td>14.90</td>
    <td>51.42</td>
    <td>42.12</td>
    <td>25.88</td>
  </tr>
  <tr>
    <td></td>
    <td>Minstral-8B-Instruct (Prompt)</td>
    <td>72.61</td>
    <td>36.41</td>
    <td>6.17</td>
    <td>38.34</td>
    <td>25.15</td>
    <td>16.15</td>
  </tr>
  <tr>
    <td></td>
    <td>Hammer2.1-7b (FC)</td>
    <td>82.36</td>
    <td>17.77</td>
    <td>9.80</td>
    <td>56.54</td>
    <td>49.81</td>
    <td>34.11</td>
  </tr>
  <tr>
    <td></td>
    <td>sLAM-7b-fc (FC)</td>
    <td>90.45</td>
    <td>32.18</td>
    <td>9.79</td>
    <td>49.87</td>
    <td>36.39</td>
    <td>25.24</td>
  </tr>
  <tr>
    <td></td>
    <td>ToolACE-SB (FC)</td>
    <td>83.37</td>
    <td>31.81</td>
    <td>4.95</td>
    <td>48.93</td>
    <td>39.95</td>
    <td>23.80</td>
  </tr>
</table>

Table 9: Overall evaluation on HammerBench for different metrics of different data types.
<table>
  <tr>
    <th>Data type</th>
    <th>Model</th>
    <th>Func.</th>
    <th>Acc.</th>
    <th>PHR</th>
    <th>PMR</th>
    <th>Acc.</th>
    <th>PR</th>
    <th>SR</th>
  </tr>
  <tr>
    <td rowspan="12">w/o SO</td>
    <td>GPT-4o (Prompt)</td>
    <td>89.14%</td>
    <td>8.65%</td>
    <td>5.68%</td>
    <td>72.35%</td>
    <td>67.90%</td>
    <td>58.76%</td>
    <td></td>
  </tr>
  <tr>
    <td>Claude-3.5-sommet (Prompt)</td>
    <td>90.51%</td>
    <td>8.69%</td>
    <td>4.96%</td>
    <td>76.19%</td>
    <td>71.18%</td>
    <td>64.39%</td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct (Prompt)</td>
    <td>84.86%</td>
    <td>12.70%</td>
    <td>5.52%</td>
    <td>65.73%</td>
    <td>51.11%</td>
    <td>41.82%</td>
    <td></td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct (Prompt)</td>
    <td>91.75%</td>
    <td>14.34%</td>
    <td>1.97%</td>
    <td>73.88%</td>
    <td>65.34%</td>
    <td>58.41%</td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct (Prompt)</td>
    <td>82.88%</td>
    <td>20.27%</td>
    <td>5.17%</td>
    <td>59.71%</td>
    <td>50.66%</td>
    <td>42.06%</td>
    <td></td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct (Prompt)</td>
    <td>90.76%</td>
    <td>25.08%</td>
    <td>15.26%</td>
    <td>52.66%</td>
    <td>41.54%</td>
    <td>27.69%</td>
    <td></td>
  </tr>
  <tr>
    <td>Ministral-8B-Instruct-2410 (Prompt)</td>
    <td>74.22%</td>
    <td>35.16%</td>
    <td>10.42%</td>
    <td>42.28%</td>
    <td>27.27%</td>
    <td>21.27%</td>
    <td></td>
  </tr>
  <tr>
    <td>Hammer2.1-7b (FC)</td>
    <td>82.68%</td>
    <td>16.92%</td>
    <td>6.58%</td>
    <td>61.83%</td>
    <td>53.87%</td>
    <td>44.46%</td>
    <td></td>
  </tr>
  <tr>
    <td>xLAM-7b-fc-r (FC)</td>
    <td>89.69%</td>
    <td>30.83%</td>
    <td>6.55%</td>
    <td>55.64%</td>
    <td>39.01%</td>
    <td>33.26%</td>
    <td></td>
  </tr>
  <tr>
    <td>ToolACE-8B (FC)</td>
    <td>81.58%</td>
    <td>28.32%</td>
    <td>3.88%</td>
    <td>54.23%</td>
    <td>44.08%</td>
    <td>32.71%</td>
    <td></td>
  </tr>
  <tr>
    <td>GPT-4o (Prompt)</td>
    <td>90.37%</td>
    <td>8.52%</td>
    <td>4.90%</td>
    <td>72.98%</td>
    <td>68.65%</td>
    <td>57.93%</td>
    <td></td>
  </tr>
  <tr>
    <td>Claude-3.5-sommet (Prompt)</td>
    <td>90.44%</td>
    <td>8.83%</td>
    <td>3.58%</td>
    <td>76.49%</td>
    <td>72.54%</td>
    <td>65.01%</td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct (Prompt)</td>
    <td>85.08%</td>
    <td>13.62%</td>
    <td>5.08%</td>
    <td>65.40%</td>
    <td>60.25%</td>
    <td>50.17%</td>
    <td></td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct (Prompt)</td>
    <td>91.62%</td>
    <td>14.85%</td>
    <td>2.03%</td>
    <td>73.94%</td>
    <td>63.15%</td>
    <td>54.84%</td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct (Prompt)</td>
    <td>82.88%</td>
    <td>23.21%</td>
    <td>3.92%</td>
    <td>54.21%</td>
    <td>45.78%</td>
    <td>33.05%</td>
    <td></td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct (Prompt)</td>
    <td>90.10%</td>
    <td>27.30%</td>
    <td>14.56%</td>
    <td>48.84%</td>
    <td>37.99%</td>
    <td>22.54%</td>
    <td></td>
  </tr>
  <tr>
    <td>Ministral-8B-Instruct-2410 (Prompt)</td>
    <td>74.22%</td>
    <td>37.75%</td>
    <td>3.60%</td>
    <td>40.29%</td>
    <td>26.21%</td>
    <td>19.93%</td>
    <td></td>
  </tr>
  <tr>
    <td>Hammer2.1-7b (FC)</td>
    <td>82.68%</td>
    <td>19.92%</td>
    <td>6.58%</td>
    <td>57.31%</td>
    <td>49.46%</td>
    <td>36.56%</td>
    <td></td>
  </tr>
  <tr>
    <td>xLAM-7b-fc-r (FC)</td>
    <td>89.89%</td>
    <td>34.70%</td>
    <td>6.51%</td>
    <td>48.37%</td>
    <td>34.44%</td>
    <td>24.94%</td>
    <td></td>
  </tr>
  <tr>
    <td>ToolACE-8B (FC)</td>
    <td>81.51%</td>
    <td>32.96%</td>
    <td>3.05%</td>
    <td>50.25%</td>
    <td>40.07%</td>
    <td>26.87%</td>
    <td></td>
  </tr>
  <tr>
    <td>GPT-4o (Prompt)</td>
    <td>88.80%</td>
    <td>8.37%</td>
    <td>5.98%</td>
    <td>72.28%</td>
    <td>68.47%</td>
    <td>60.11%</td>
    <td></td>
  </tr>
  <tr>
    <td>Claude-3.5-sommet (Prompt)</td>
    <td>90.21%</td>
    <td>8.59%</td>
    <td>5.32%</td>
    <td>75.71%</td>
    <td>71.99%</td>
    <td>65.09%</td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct (Prompt)</td>
    <td>85.88%</td>
    <td>12.73%</td>
    <td>5.07%</td>
    <td>67.83%</td>
    <td>63.15%</td>
    <td>54.84%</td>
    <td></td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct (Prompt)</td>
    <td>91.62%</td>
    <td>14.85%</td>
    <td>2.03%</td>
    <td>73.94%</td>
    <td>63.15%</td>
    <td>54.84%</td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct (Prompt)</td>
    <td>82.88%</td>
    <td>23.21%</td>
    <td>3.92%</td>
    <td>54.21%</td>
    <td>45.78%</td>
    <td>33.05%</td>
    <td></td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct (Prompt)</td>
    <td>89.65%</td>
    <td>25.98%</td>
    <td>13.73%</td>
    <td>52.98%</td>
    <td>42.76%</td>
    <td>31.23%</td>
    <td></td>
  </tr>
  <tr>
    <td>Ministral-8B-Instruct-2410 (Prompt)</td>
    <td>75.25%</td>
    <td>35.55%</td>
    <td>5.29%</td>
    <td>42.89%</td>
    <td>28.17%</td>
    <td>23.61%</td>
    <td></td>
  </tr>
  <tr>
    <td>Hammer2.1-7b (FC)</td>
    <td>83.16%</td>
    <td>17.25%</td>
    <td>6.79%</td>
    <td>61.90%</td>
    <td>54.76%</td>
    <td>46.00%</td>
    <td></td>
  </tr>
  <tr>
    <td>xLAM-7b-fc-r (FC)</td>
    <td>89.89%</td>
    <td>34.70%</td>
    <td>6.51%</td>
    <td>48.37%</td>
    <td>34.44%</td>
    <td>24.94%</td>
    <td></td>
  </tr>
  <tr>
    <td>ToolACE-8B (FC)</td>
    <td>81.93%</td>
    <td>28.45%</td>
    <td>4.23%</td>
    <td>54.24%</td>
    <td>44.81%</td>
    <td>34.90%</td>
    <td></td>
  </tr>
  <tr>
    <td>GPT-4o (Prompt)</td>
    <td>90.68%</td>
    <td>7.84%</td>
    <td>6.45%</td>
    <td>61.98%</td>
    <td>56.45%</td>
    <td>18.53%</td>
    <td></td>
  </tr>
  <tr>
    <td>Claude-3.5-sommet (Prompt)</td>
    <td>90.02%</td>
    <td>8.02%</td>
    <td>5.36%</td>
    <td>69.68%</td>
    <td>63.94%</td>
    <td>42.89%</td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct (Prompt)</td>
    <td>86.17%</td>
    <td>12.23%</td>
    <td>5.66%</td>
    <td>64.98%</td>
    <td>57.35%</td>
    <td>42.05%</td>
    <td></td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct (Prompt)</td>
    <td>91.62%</td>
    <td>14.44%</td>
    <td>5.21%</td>
    <td>68.87%</td>
    <td>58.64%</td>
    <td>41.39%</td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct (Prompt)</td>
    <td>83.06%</td>
    <td>21.48%</td>
    <td>4.85%</td>
    <td>51.85%</td>
    <td>41.75%</td>
    <td>18.15%</td>
    <td></td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct (Prompt)</td>
    <td>89.65%</td>
    <td>25.19%</td>
    <td>15.48%</td>
    <td>48.38%</td>
    <td>39.16%</td>
    <td>15.89%</td>
    <td></td>
  </tr>
  <tr>
    <td>Ministral-8B-Instruct-2410 (Prompt)</td>
    <td>75.25%</td>
    <td>33.64%</td>
    <td>4.41%</td>
    <td>36.47%</td>
    <td>22.00%</td>
    <td>7.53%</td>
    <td></td>
  </tr>
  <tr>
    <td>Hammer2.1-7b (FC)</td>
    <td>83.25%</td>
    <td>17.78%</td>
    <td>10.81%</td>
    <td>52.81%</td>
    <td>43.87%</td>
    <td>17.49%</td>
    <td></td>
  </tr>
  <tr>
    <td>xLAM-7b-fc-r (FC)</td>
    <td>89.18%</td>
    <td>26.90%</td>
    <td>15.51%</td>
    <td>43.23%</td>
    <td>29.09%</td>
    <td>4.80%</td>
    <td></td>
  </tr>
  <tr>
    <td>ToolACE-8B (FC)</td>
    <td>81.93%</td>
    <td>29.82%</td>
    <td>3.82%</td>
    <td>45.74%</td>
    <td>35.07%</td>
    <td>11.57%</td>
    <td></td>
  </tr>
  <tr>
    <td>GPT-4o (Prompt)</td>
    <td>89.38%</td>
    <td>5.22%</td>
    <td>3.64%</td>
    <td>77.95%</td>
    <td>74.36%</td>
    <td>67.34%</td>
    <td></td>
  </tr>
  <tr>
    <td>Claude-3.5-sommet (Prompt)</td>
    <td>90.40%</td>
    <td>4.44%</td>
    <td>4.35%</td>
    <td>80.35%</td>
    <td>76.60%</td>
    <td>70.04%</td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct (Prompt)</td>
    <td>88.06%</td>
    <td>8.88%</td>
    <td>2.99%</td>
    <td>69.92%</td>
    <td>65.61%</td>
    <td>67.55%</td>
    <td></td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct (Prompt)</td>
    <td>92.44%</td>
    <td>9.76%</td>
    <td>1.30%</td>
    <td>79.57%</td>
    <td>72.55%</td>
    <td>66.93%</td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct (Prompt)</td>
    <td>81.42%</td>
    <td>13.10%</td>
    <td>3.91%</td>
    <td>64.92%</td>
    <td>58.13%</td>
    <td>49.99%</td>
    <td></td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct (Prompt)</td>
    <td>89.79%</td>
    <td>17.30%</td>
    <td>16.30%</td>
    <td>58.26%</td>
    <td>47.80%</td>
    <td>32.65%</td>
    <td></td>
  </tr>
  <tr>
    <td>Ministral-8B-Instruct-2410 (Prompt)</td>
    <td>73.76%</td>
    <td>28.16%</td>
    <td>3.97%</td>
    <td>47.81%</td>
    <td>31.67%</td>
    <td>17.39%</td>
    <td></td>
  </tr>
  <tr>
    <td>Hammer2.1-7b (FC)</td>
    <td>82.65%</td>
    <td>12.33%</td>
    <td>4.06%</td>
    <td>66.19%</td>
    <td>60.62%</td>
    <td>52.85%</td>
    <td></td>
  </tr>
  <tr>
    <td>xLAM-7b-fc-r (FC)</td>
    <td>88.77%</td>
    <td>22.59%</td>
    <td>5.69%</td>
    <td>63.09%</td>
    <td>49.07%</td>
    <td>43.46%</td>
    <td></td>
  </tr>
  <tr>
    <td>ToolACE-8B (FC)</td>
    <td>83.87%</td>
    <td>23.38%</td>
    <td>2.68%</td>
    <td>60.84%</td>
    <td>51.12%</td>
    <td>39.99%</td>
    <td></td>
  </tr>
  <tr>
    <td>GPT-4o (Prompt)</td>
    <td>90.81%</td>
    <td>5.52%</td>
    <td>13.36%</td>
    <td>69.64%</td>
    <td>66.34%</td>
    <td>49.18%</td>
    <td></td>
  </tr>
  <tr>
    <td>Claude-3.5-sommet (Prompt)</td>
    <td>90.40%</td>
    <td>6.22%</td>
    <td>9.18%</td>
    <td>71.83%</td>
    <td>68.62%</td>
    <td>54.28%</td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct (Prompt)</td>
    <td>84.48%</td>
    <td>12.36%</td>
    <td>8.76%</td>
    <td>61.12%</td>
    <td>59.58%</td>
    <td>45.91%</td>
    <td></td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct (Prompt)</td>
    <td>92.44%</td>
    <td>12.36%</td>
    <td>1.77%</td>
    <td>79.57%</td>
    <td>72.55%</td>
    <td>66.93%</td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct (Prompt)</td>
    <td>81.22%</td>
    <td>19.73%</td>
    <td>6.35%</td>
    <td>53.87%</td>
    <td>50.04%</td>
    <td>34.89%</td>
    <td></td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct (Prompt)</td>
    <td>89.79%</td>
    <td>22.70%</td>
    <td>14.95%</td>
    <td>50.14%</td>
    <td>43.56%</td>
    <td>25.51%</td>
    <td></td>
  </tr>
  <tr>
    <td>Ministral-8B-Instruct-2410 (Prompt)</td>
    <td>73.67%</td>
    <td>33.52%</td>
    <td>9.88%</td>
    <td>55.00%</td>
    <td>25.06%</td>
    <td>14.08%</td>
    <td></td>
  </tr>
  <tr>
    <td>Hammer2.1-7b (FC)</td>
    <td>82.65%</td>
    <td>15.51%</td>
    <td>4.06%</td>
    <td>66.19%</td>
    <td>60.62%</td>
    <td>52.85%</td>
    <td></td>
  </tr>
  <tr>
    <td>xLAM-7b-fc-r (FC)</td>
    <td>88.77%</td>
    <td>26.38%</td>
    <td>13.66%</td>
    <td>48.30%</td>
    <td>39.76%</td>
    <td>25.91%</td>
    <td></td>
  </tr>
  <tr>
    <td>ToolACE-8B (FC)</td>
    <td>83.87%</td>
    <td>28.83%</td>
    <td>5.45%</td>
    <td>49.15%</td>
    <td>43.61%</td>
    <td>26.93%</td>
    <td></td>
  </tr>
</table>

Table 10: Ablation on HammerBench.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">Diverse Q&A</th>
    <th colspan="3">Argument shifts</th>
    <th rowspan="2">Intent shifts</th>
  </tr>
  <tr>
    <th>Overall</th>
    <th>sQsA</th>
    <th>mQmA</th>
    <th>sQmA</th>
    <th>SO</th>
    <th>mSv</th>
    <th>External</th>
  </tr>
  <tr>
    <td>GPT-4o (Prompt)</td>
    <td>72.00</td>
    <td>72.52</td>
    <td>75.63</td>
    <td>73.00</td>
    <td>74.05</td>
    <td>72.98</td>
    <td>61.98</td>
    <td>69.64</td>
    <td>76.22</td>
  </tr>
  <tr>
    <td>Claude3.5-sonnet (Prompt)</td>
    <td>72.02</td>
    <td>76.62</td>
    <td>75.52</td>
    <td>73.34</td>
    <td>73.94</td>
    <td>76.49</td>
    <td>69.68</td>
    <td>71.83</td>
    <td>58.74</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct (Prompt)</td>
    <td>68.12</td>
    <td>64.59</td>
    <td>67.82</td>
    <td>63.90</td>
    <td>67.40</td>
    <td>65.40</td>
    <td>64.98</td>
    <td>61.12</td>
    <td>89.79</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct (Prompt)</td>
    <td>70.49</td>
    <td>75.05</td>
    <td>73.20</td>
    <td>70.74</td>
    <td>73.52</td>
    <td>70.25</td>
    <td>68.87</td>
    <td>69.29</td>
    <td>63.02</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct (Prompt)</td>
    <td>57.78</td>
    <td>60.08</td>
    <td>58.12</td>
    <td>54.40</td>
    <td>57.70</td>
    <td>54.21</td>
    <td>51.85</td>
    <td>53.87</td>
    <td>72.04</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct (Prompt)</td>
    <td>51.49</td>
    <td>51.53</td>
    <td>57.70</td>
    <td>48.59</td>
    <td>56.22</td>
    <td>48.84</td>
    <td>48.30</td>
    <td>50.14</td>
    <td>50.63</td>
  </tr>
  <tr>
    <td>Ministral-8B-Instruct (Prompt)</td>
    <td>35.39</td>
    <td>43.19</td>
    <td>40.29</td>
    <td>32.94</td>
    <td>40.18</td>
    <td>40.29</td>
    <td>36.47</td>
    <td>35.00</td>
    <td>14.75</td>
  </tr>
  <tr>
    <td>Hammer2.1-7b (FC)</td>
    <td>61.31</td>
    <td>62.26</td>
    <td>60.86</td>
    <td>52.63</td>
    <td>60.65</td>
    <td>57.85</td>
    <td>52.81</td>
    <td>48.73</td>
    <td>94.71</td>
  </tr>
  <tr>
    <td>xLAM-7b-fc-r (FC)</td>
    <td>48.43</td>
    <td>58.03</td>
    <td>52.53</td>
    <td>46.00</td>
    <td>52.63</td>
    <td>48.37</td>
    <td>43.23</td>
    <td>48.30</td>
    <td>38.34</td>
  </tr>
  <tr>
    <td>ToolACE-8B (FC)</td>
    <td>48.19</td>
    <td>54.54</td>
    <td>52.63</td>
    <td>38.55</td>
    <td>51.68</td>
    <td>50.25</td>
    <td>45.74</td>
    <td>49.15</td>
    <td>42.98</td>
  </tr>
</table>

Table 11: The evaluation (Acc.%) on HammerBench for different multi-turn data types. It is a snippet from Table 9.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">w/o (PR) | w/ (∆ PR)</th>
    <th colspan="2">Argument shifts</th>
    <th rowspan="2">External</th>
    <th rowspan="2">IS</th>
  </tr>
  <tr>
    <th>mQmA</th>
    <th>sQmA</th>
    <th>sQmA</th>
    <th>SO</th>
    <th>mSv</th>
  </tr>
  <tr>
    <td>GPT-4o (Prompt)</td>
    <td>+6.03% | +7.18%</td>
    <td>+6.03% | +2.48%</td>
    <td>+6.03% | +4.76%</td>
    <td>+6.79% | +0.745%</td>
    <td>+6.47% | -12.01%</td>
    <td>+7.46% | -8.023%</td>
    <td>76.22%</td>
  </tr>
  <tr>
    <td>Claude3.5-sonnet (Prompt)</td>
    <td>+7.04% | +3.10%</td>
    <td>+7.04% | +1.72%</td>
    <td>+7.04% | +1.97%</td>
    <td>+7.11% | +0.831%</td>
    <td>+7.19% | -0.848%</td>
    <td>+7.66% | -7.976%</td>
    <td>58.74%</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct (Prompt)</td>
    <td>+5.62% | +2.70%</td>
    <td>+5.62% | +0.72%</td>
    <td>+5.62% | -3.09%</td>
    <td>+6.12% | +3.282%</td>
    <td>+5.34% | -3.333%</td>
    <td>+5.70% | -7.057%</td>
    <td>63.02%</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct (Prompt)</td>
    <td>+6.47% | +2.70%</td>
    <td>+6.47% | +1.72%</td>
    <td>+6.47% | +3.282%</td>
    <td>+5.34% | -3.333%</td>
    <td>+5.70% | -7.057%</td>
    <td>+7.25% | -9.076%</td>
    <td>63.02%</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct (Prompt)</td>
    <td>+4.89% | +1.59%</td>
    <td>+4.89% | -5.280%</td>
    <td>+4.89% | +4.138%</td>
    <td>+5.066% | -4.878%</td>
    <td>+5.171% | -9.961%</td>
    <td>+5.813% | -8.093%</td>
    <td>72.04%</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct (Prompt)</td>
    <td>+3.84% | +1.27%</td>
    <td>+3.84% | -0.012%</td>
    <td>+3.84% | +11.92%</td>
    <td>+4.54% | -3.546%</td>
    <td>+4.276% | -7.786%</td>
    <td>+4.780% | -4.244%</td>
    <td>50.63%</td>
  </tr>
  <tr>
    <td>Ministral-8B-Instruct-24h (Prompt)</td>
    <td>+2.68% | +1.80%</td>
    <td>+2.68% | -7.762%</td>
    <td>+2.68% | +1.589%</td>
    <td>+2.776% | -1.554%</td>
    <td>+2.87% | -6.172%</td>
    <td>+3.134% | -6.280%</td>
    <td>14.75%</td>
  </tr>
  <tr>
    <td>Hammer2.1-7b (FC)</td>
    <td>+3.78% | +1.45%</td>
    <td>+3.78% | -8.921%</td>
    <td>+3.78% | +1.793%</td>
    <td>+3.901% | -4.568%</td>
    <td>+3.914% | -10.05%</td>
    <td>+4.097% | -9.311%</td>
    <td>38.34%</td>
  </tr>
  <tr>
    <td>xLAM-7b-fc-r (FC)</td>
    <td>+3.28% | +1.45%</td>
    <td>+3.28% | -8.921%</td>
    <td>+3.28% | +1.793%</td>
    <td>+3.901% | -4.568%</td>
    <td>+3.914% | -10.05%</td>
    <td>+4.097% | -9.311%</td>
    <td>38.34%</td>
  </tr>
  <tr>
    <td>ToolACE-8B (FC)</td>
    <td>+4.25% | +3.843%</td>
    <td>+4.25% | -16.68%</td>
    <td>+4.25% | +3.554%</td>
    <td>+4.088% | -4.010%</td>
    <td>+4.418% | -9.745%</td>
    <td>+5.121% | -7.510%</td>
    <td>42.98%</td>
  </tr>
</table>

Table 12: Multi-turn fine-grained evaluation across different data types for all snapshots. Each cell consists of two items: the absolute value of the baseline sQsA dataset and the change (∆) in the metric after encountering the corresponding situations. So that we can more clearly observe the impacts of these situations.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">SO</th>
    <th colspan="4">External</th>
  </tr>
  <tr>
    <th>PHR</th>
    <th>PMR</th>
    <th>PR</th>
    <th>PHR</th>
    <th>PMR</th>
    <th>PR</th>
  </tr>
  <tr>
    <td>GPT-4o (Prompt)</td>
    <td>7.555% | +0.501%</td>
    <td>8.251% | -2.508%</td>
    <td>68.86% | -0.996%</td>
    <td>5.492% | +0.477%</td>
    <td>5.492% | +18.76%</td>
    <td>75.06% | -18.91%</td>
  </tr>
  <tr>
    <td>Claude3.5-sonnet (Prompt)</td>
    <td>7.594% | +0.801%</td>
    <td>6.845% | -3.348%</td>
    <td>74.43% | -0.034%</td>
    <td>4.690% | +2.987%</td>
    <td>7.129% | +7.851%</td>
    <td>75.54% | -15.61%</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct (Prompt)</td>
    <td>11.88% | +2.471%</td>
    <td>6.804% | -1.116%</td>
    <td>66.32% | -4.742%</td>
    <td>9.716% | +6.643%</td>
    <td>8.097% | +6.217%</td>
    <td>66.83% | -13.97%</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct (Prompt)</td>
    <td>11.76% | +2.239%</td>
    <td>2.023% | -0.858%</td>
    <td>74.63% | -8.213%</td>
    <td>8.302% | +5.055%</td>
    <td>1.291% | +2.419%</td>
    <td>78.94% | -15.71%</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct (Prompt)</td>
    <td>16.58% | +7.338%</td>
    <td>6.02% | -2.006%</td>
    <td>60.75% | -1.706%</td>
    <td>12.80% | +9.826%</td>
    <td>3.31% | +3.391%</td>
    <td>62.60% | -10.10%</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct (Prompt)</td>
    <td>16.57% | +7.345%</td>
    <td>5.054% | -3.11%</td>
    <td>51.31% | -9.73%</td>
    <td>12.28% | +9.826%</td>
    <td>3.31% | +3.391%</td>
    <td>62.60% | -10.10%</td>
  </tr>
  <tr>
    <td>Ministral-8B-Instruct-24h (Prompt)</td>
    <td>21.67% | +7.864%</td>
    <td>4.54% | -3.440%</td>
    <td>50.72% | -9.175%</td>
    <td>17.56% | +9.367%</td>
    <td>4.683% | +14.28%</td>
    <td>53.16% | -23.74%</td>
  </tr>
  <tr>
    <td>Hammer2.1-7b (FC)</td>
    <td>14.13% | +7.616%</td>
    <td>8.469% | -3.306%</td>
    <td>61.58% | -11.20%</td>
    <td>11.52% | +4.699%</td>
    <td>5.555% | +29.96%</td>
    <td>64.72% | -31.59%</td>
  </tr>
  <tr>
    <td>xLAM-7b-fc-r (FC)</td>
    <td>19.47% | +10.16%</td>
    <td>8.571% | +0.313%</td>
    <td>62.33% | -19.62%</td>
    <td>14.25% | +7.414%</td>
    <td>6.653% | +14.06%</td>
    <td>66.73% | -23.74%</td>
  </tr>
  <tr>
    <td>ToolACE-8B (FC)</td>
    <td>27.29% | +10.78%</td>
    <td>4.461% | -2.101%</td>
    <td>53.67% | -12.50%</td>
    <td>23.38% | +11.29%</td>
    <td>3.225% | +5.040%</td>
    <td>58.06% | -19.42%</td>
  </tr>
</table>

Table 13: Evaluations for snapshots at the moment of slot overriding (SO) and answering with pronouns (External). Each cell in the table includes two items: the baseline absolute metrics under the sQsA dataset, and the changes (∆) in metrics after considering SO/External.

Reviewer paper summaries: The paper introduces HammerBench, a novel benchmark framework designed to evaluate the function-calling capabilities of large language models (LLMs) in real-world, multi-turn dialogue scenarios. HammerBench simulates diverse mobile assistant use cases, incorporating imperfect instructions, dynamic question-answer trajectories, intent and argument shifts, and the indirect use of external information through pronouns. The benchmark is constructed using a comprehensive dataset derived from popular mobile app functionalities and anonymized user logs, augmented with a cost-effective data generation pipeline leveraging open-source models. The authors demonstrate the effectiveness of HammerBench by evaluating several leading LLMs, uncovering key performance trends, and identifying areas for improvement, particularly in handling parameter name errors and tracking argument shifts.
Summary of strengths: HammerBench integrates various data types from previous benchmarks, creating a more comprehensive evaluation framework. This synthesis allows for a broader assessment of LLM capabilities across diverse interaction scenarios.
Summary of weaknesses: 1. The Diverse Q&A Trajectories category, while valuable, is not novel and has been previously explored in benchmarks like AppBench. However, the paper fails to acknowledge or cite AppBench, which could have provided additional context and validation for this aspect of the benchmark.
2. The paper does not fully address the real-world scenario challenges highlighted in the introduction. Most of the queries used in the benchmark are generated by LLMs rather than collected from actual user interactions. Additionally, the APIs for the apps are not real but are instead generated by LLMs based on functional descriptions. This raises questions about the benchmark's ability to accurately reflect real-world usage and the practical applicability of the results.
3. Lack of Distinction Between Argument Shifts and Intent Shifts. The paper introduces both "Argument Shifts" and "Intent Shifts" as separate categories, but they appear to lack a fundamental distinction. Both seem to revolve around changes in user input, with the primary difference being the specific API calls triggered.
Comments, suggestions and typos: N/A
Overall assessment scores: 2