The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models
Authors: Shishir G Patil
Huanzhi Mao
Fanjia Yan
Charlie Cheng-Jie Ji
Vishnu Suresh
Ion Stoica
Joseph E. Gonzalez
Venue: ICML 2025 poster (2025)
Keywords: Function Calling Evaluation
Tool use
Agentic Evaluation
Large Language Models
Primary area: deep_learning->large_language_models

Abstract: Function calling, also called tool use, refers to an LLM's ability to invoke external functions, APIs, or user-defined tools in response to user queries—an essential capability for agentic LLM applications. Despite its prominence, there did not exist a standard benchmark to evaluate function calling abilities, due to two reasons – the challenging nature of evaluating when a function call is valid, and the challenge of acquiring diverse, real-world functions. We present the Berkeley Function Calling Leaderboard (BFCL), a comprehensive benchmark designed to evaluate function calling capabilities in a wide range of real-world settings. The BFCL benchmark evaluates serial and parallel function calls, across various programming languages using a novel Abstract Syntax Tree (AST) evaluation method that can easily scale to thousands of functions. We construct the benchmark using a combination of expert curated, and user-contributed functions and associated prompts. Finally, BFCL benchmark evaluates the ability of models to abstain and reason in stateful multi-step agentic setting. Evaluating a wide range of models, we observe that while state-of-the-art LLMs excel at singleturn calls, memory, dynamic decision-making, and long-horizon reasoning remain open challenges. Since its preview, BFCL has become the defacto standard for evaluating function-calls, and can be accessed at gorilla.cs.berkeley.edu/leaderboard.html.

The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models
Shishir G. Patil
Huanzhi Mao
Fanjia Yan Charlie Cheng-Jie Ji Vishnu Suresh
Ion Stoica
Joseph E. Gonzalez"
Abstract
soning, and creative multimodal tasks. However, agents powering use-cases such as coding, and knowledge discov- cry (Huang et al., 2024; Yao et al., 2022) often require LLMs to interface with external (e.g., web-search) to either retrieve up-to-date information or to enact actions which have real- world consequences. To aid in this, previous work such as Gorilla (Patil et al., 2024) and Toolformer (Schick et al., 2023) introduced techniques to train LLMs to use tools, and since then there has been growing interest in enabling LLMs to leverage external tools, a capability commonly referred to as function calling or tool use.
Function calling, also called tool use, refers to LLM's ability invoke external functions, APIs, or user-defined tools-an essential capa- bility for agentic LLM applications. Despite its prominence, there does not exist standard benchmark to evaluate function calling due to two reasons - the challenging nature evaluat- ing when a function call is valid, and the chal- lenge of acquiring diverse, real-world functions. We present the Berkeley Function Calling Leader- board (BFCL), a comprehensive benchmark de- signed to evaluate function calling in wide range of real-world settings. The BFCL benchmark evaluates serial and parallel function calls, across various programming languages, using novel Abstract Syntax Tree (AST) evaluation method that can easily scale thousands of functions. We construct the benchmark using combination of expert-curated and user-contributed functions and associated prompts. Finally, BFCL bench- mark evaluates the ability of models to abstain and reason in a stateful multistep agentic set- ting. Evaluating wide range of models, we observe that while state-of-the-art L.LMs excel at single-turn calls, memory, dynamic decision- making, and long-horizon reasoning remain open challenges. Since its preview, BFCL has become the defacto standard for evaluating function-calls, and can accessed at https: / //tprr/olllaaaa//// es berkeley. edu/leaderboard. html
Despite substantial progress leveraging function calling in LLMs, evaluating function calling remains challenging. Existing solutions, such Corilla-APlBench (Patil et al., 2024), propose methods for assessing function calls but exhibit notable limitations: APIBench's is evaluation met- rics focus primarily on functional correctness, overlooking semantic errors or variations in API usage. Other bench- Trock--TooBBechh (Guo al., 2024), ToolSandbox (Lu et al., 2024), and the Nexus Function Calling Leaderboard (Srinivasan et al., 2023) fail capture real-world use cases or patterns such as parallel invocations. We discuss some of these limitations in more detail in the related work section.
To address these challenges, we introduce the Berke- ley Function Calling Leaderboard (BFCL), a large- scale, multi-task, multi-turn benchmark that distinguishes funetion-calling capabilities among LLMs by evaluating their ability to invoke the correct function call. BFCL is composed of four parts: (1) 'single-turn' that tests single- turn function-calling scenarios, including parallel function invocations and multiple function candidates; (2) 'crowd- sourced' which consists of 2, 251 curated from more than 67, 000 community contributed real-life function-callin datapoints; (3) 'multi-turn' featuring eight curated API suites and 1000 queries, assessing sustained context manage- ment and dynamic decision-making: and finally (4) 'agentic' spanning applications (Vu et al., 2023). database queries (Gao et al., 2024), and chatbot context management (Packer et al., 2024). This comprehensive structure spans single- function tasks to complex, multi-step real-world use cases, enabling a thorough evaluation of an LLM's adaptability,
1. Introduction
Large Language Models (LLMs) have made significant strides in diverse domains, including conversational AI, rea-
University of Califomia, Berkeley. Correspondence to: Shishir G. Patil <shishirpatile @ berkeley.edu>>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models
correctness, and effectiveness in function invocation.
Evaluating LLMs' function-invocatiinn capabilities poses unique challenges because deterministic validation typically requires executing the corresponding functions which com- plicates large-scale evaluation. BFCL. overcomes this by introducing a novel validation strategy that obviates the need for function execution. Drawing inspiration from pro- gramming language literature, we employ Abstract Syntax Tree (AST) sub-string matching as proxy for actual func- tion execution, thereby facilitating scalable evaluations. To validate this approach, we utilize subset of our dataset to evaluate models using the earlier mentioned execution approach and observe a strong correlation between BFCL's execution and AST metrics.
ARTLOCOO
Crowe Swa-os
Betakans (PO NORE
NC
RD
Paratee
Commac 2011
As models increasingly incorporate function calling capabil- ity, our phased release lets compare if BFCL's 's single-tun turn dataset has possibly leaked into the training data of the latest models. To investigate this, through CharNLL as measure, we compare LLMs' familiarity with the single-turn dataset against that of the crowd-sourced dataset which was released six months apart.
Figure I: This chart visualizes the diverse categories within BFCL. The inner ring represents the four major sections, middle ring specifies their respective evaluation methods, and the outer ring highlights each category. Numbers indicate the total number of datapoints in each category.
In summary, this work makes the following contributions:
1. BFCL is diverse dataset of 5, 551 question-function answer pairs across multiple programming languages, including Python, Java, JavaScript, REST APIS, and SQL.. This diversity ensures a comprehensive assess- ment of LLMs' function-calling ability across range of domains and use cases.
with external systems for knowledge retrieval (Sasaki et al., 2024) and real-time interaction. Furthermore, advancements like the I.M.Comiller (Kim et al., 2024) optimize this pro- cess through parallel function execution, improving both efficiency and accuracy. This integration of function calling represents a crucial step towards more capable and versatile LLMS and subsequently unlocks broad agentic behaviors.
2. A novel application of Abstract Syntax Trees (AST) based sub-string matching to serve as proxy for func- tion execution and enabling scalable, deterministic val- idation of function calls.
Benchmark for Function Calling. Quite few benchmarks have been proposed to test LLM's ability to perform func- tion calling, in this work, we focus on those designed to eval- uate a model's ''s native function-calling capabilities, rather than methods that rely on prompt-hased or code-generation approaches (c.g., Nexus Raven (team, 2023) uses a prompt- based protocol, and App World (Trivedi et al., 2024) empha- sis code generation capability). Among benchmarks that do evaluate native function calling, many such as App Blend (Basu et al., 2024) and API Bench (Patil et al., 2024) focus solely on single-turn interactions. Although TinyAgent (Er- dogan et al., 2024) addresses nested function calls, it does so by using placeholder variables instead of letting the model see the actual execution output of earlier calls; effect, still operates under a single-turn framework.
3. The first inclusion of community-contribtted, real- world user queries and functions in function-calling evaluations, providing a more accurate representation of practical complexities and use cases.
2. Related Work
Language Models Using Tools. Function calling (Schick et al., 2023) extends the capabilities of LLMs beyond its own knowledge base by enabling them to interact with ex- ternal tools and APIs. Unlike structured output (Zhong & Chen, 2021), function calling allows LLMs perform tasks requiring real-time data extemnl computation (At- touche et al., 2024). Models like GPT-4 (OpenAl, 2024) have demonstrated early ability to generate structured JSON for function invocation, prompting research into leveraging API calls as functions. More recent models have natively integrated function calling, empowering them to interact
For benchmarks that truly cover multi-turn interactions, most are constrained by narrow domain scope or a limited set of functions. TauBench (Yao et al., 2024), for example, supports only 28 functions spanning two domains (airline and retail), and RestBench (Song et al., 2023) offers see- narios solely within the TMDB and Spotify domains. The
2

---

The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models
narrow coverage (< 150 entries) is more prone to overfit- ting and does not sufficiently reflect the breadth of real-
Multiple Functions
Parallel Functions
Function Relevance Detection
world function-calling scenarios. Furthermore, benchmarks like ToolSandBox (Lu et al., 2024) and TauBench (Yao et al., 2024) rely on L.MMs to simulate user queries. De- spite careful attempt to control the user simulator's 's behav- ior, LLM-based users remain prone to hallucination and insllwction-folloiingg errors, which confound evaluation.
Prompt: What 2 37
Promat: 12 and 14 517
Promot: What 15 2133
Function:
int bo, mult (int b)
[Add tint.
Function:
Function: lint int bi
(andlint int bi
Agent: add (am2, bw3)
Agent: [add (am2, b-3)
Agent:
Beror. The user asks for adding but have multiplicatiin.
add (ame, bw5)
Other works, such as ToolBench (Qin et al., 2023), depend solely on Rapid APIs that are subject to high variance in performance, making reproducibility a challenge. While the subsequent StableTToolBench (Guo et al., 2025) version mitigates this by caching or simulating API responses, it continues to rely on L.L.-based evaluators for determining response solvability, thus risking model-induced biases and undermining objectivity (e.g., GPT-family models tending to favor responses from their own model (Panickssery et al., 2024)). Similar issues exist in T-Eval (Chen et al., 2024), which also depends on LLM-based evaluation.
Figure 2: Examples of single-tum turn function function-calling calling scenarios to illustrate multiple, parallel, and irrelevance entry types outlined in Section 3.1.
LLMs; data transformation augments data with incom- plete queries, and data validation ensures consistency and correctness through comprehensive unit-tests.
3.1. Single-turn Dataset
We classify single turn function-calling scenarios into five types based on the number of available tools and their in- vocation patterns. Simple involves one tool with a single invocation, whereas Multiple includes several tools each in- voked once. Parallel scenarios feature multiple invocations of a single tool, and Parallel Multiple combines multiple tools with multiple invocations. Irrelevance refers to cases where tools are available but not invoked. Detailed defini- tions are provided in Appendix C.
Lastly, current benchmarks uniformly employ LLM-curated user queries, limiting their ability to accurately reflect gen- uine user interactions.
Our benchmark directly addresses these shortcomings by incorporating deterministic evaluation metrics, an exten- sive and diverse set of robust multi-turn interactions, and a unique multilingual dataset derived from real-world, user- contributed queries that have more than 15 languages repre- sented in user queries, including Chinese, French, Japanese, and Korean, etc. Additionally, on top of Python and REST (which are commonly covered in existing benchmarks), we also have entries in Java and JavaScript for more diverse programming languages.
We evaluate function calling through two methods: AST- based substring matching and executable tests. For AST- based evaluation, we curate functions in Python, Java, and JavaScript from popular GitHub repositories, filtering out trivial ones. For executable tests, we include (1) Python functions covering mathematical and physical computations, and (2) API-wrapped functions emulating real-world ser- vices (e.g., currency exchange and geocoding). Functions are standardized into schema to avoid inconsistent docu- mentation and ensure fair model comparisons. We further increase complexity by generating multiple and parallel calls, adding distractor functions, and testing robustness with missing parameters. See Appendix E for details.
Collectively, these enhancements enable a more compre- hensive, fair, and reproducible assessment of LLM function calling, setting our benchmark apart from existing efforts.
Impact: Since it's preview, BFCL has become the de- facto evaluation for function calling used by all leading labs developing large language models (MetaAI, 2024; Team, 2025; Cohere, 2025). The leaderboard is constantly evolving and is in it's current iteration (v4). This paper collapses the timeline and condenses all the learnings
3.2. Crowd-sourced Dataset
The crowd-sourced dataset contains 64,517 real single-turn user queries collected between 2024-02-26 and 2024-04-01, representing genuine function-call interactions from users. We remove duplicates via ROUGE-L and embedding based similarity, and exclude queries from public sets to avoid contamination. Human experts minimally edit queries for clarity and adherence to our function schema, preserving their original semantics. See Appendix E.2 for details. This dataset has the same dataset structure and categories as Section 3.1.
3. Berkeley Function Calling Leaderboard
BFCL employs a structured data curation pipeline to con- struct our benchmarking dataset across different categories. The pipeline follows five stages: data collection sources functions from online repositories, APIs, and user queries; data pre-processing extracts and structures key function attributes; data generation standardizes functions and user queries into a schema that can be presented to the
3

---

The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models
3.3. Multi-turn Dataset
participates in sequence of consecutive conversations that gradually reveal user-specific facts. After each domain-level dialogue block, we record memory snapshot. Evaluation queries are issued with an empry chat history but access to the stored snapshot, probing whether the model can accu- rately retrieve, add, overwrite, or delete information that was mentioned minutes (short-term) or many turns (long-term) earlier. This design lets us measure how well the model maintains and updates memory across both topic boundaries and temporal gaps, mirroring real deployment scenarios where sustained, personalized assistance is critical.
Multi-turn covers an entire conversation between the user and the assistant. Each conversation includes multiple turns new user message-ann each turn may involve one or more steps ceeps-individdue interactions between the as- sistant and a tool or environment.
Our multi-turn dataset evaluates the LLMs's s ability to han- dle queries that evolve over multiple turns. It is divided into four categories: Base category covers the basic multi-turn user queries asking everyday tasks and providing all nec- essary information. Missing Parameters tests the model's ability to recognize when critical parameter information is missing from the user request and cannot be inferred from the system. Missing Functions tests the model to identify when no available function can fulfill the user request. Long Context challenges the model's 's ability to maintain accuracy in multi-turn long context queries function call results. Detailed definitions are provided in Appendix D.
3.4.3. SQL DATASET
Each question in the SQL category can be succinctly trans- lated into an SQL query. Traditional text-to-SQL. tasks typically involve prompting language models to generate a valid SQL query, then evaluating the result via exact string matching or by querying a predefined database. In BFCL, however, we adopt a more structured approach by supplying the LLM with a JSON-based schema that defines funda- mental SQL operations (e.g., SELECT, INSERT, UPDATE, DELETE). Each operation includes detailed structured pa- rameters needed for a complete SQL query (e.g., WHERE, LIMIT, JOIN). This schema allows for deterministic translation of function calls into SQL queries and supports nested or more complex queries through the composition of multiple function calls.
To construct the dataset, we developed a custom API code- base across diverse domains like vehicle control, ensuring full transparency of API state and design. Data generation involves task generation of multi-turn queries that describe real-world tasks, along with specified initial API state con- figurations and function documents. Human annotators label ground truth trajectories from the generated multi-turn queries. Data validation includes question completeness, initial state verification, and function call sequence align- ment with -abeled ground truth. Detailed data gener- ation and validation processes are outlined in Appendix F.3.
4. Evaluation Methodology
BFCL employs tailored evaluation protocols for each dataset category. The Single-Ture category use both AST-substring matching (Section 4.1) and execution-respoons matching (Section 4.2), whereas the Crowd-Sourced category re- lies solely on the AST matcher. The Multi-Ture category combines a state-based and a response- -based checker (Sec- tion 4.4), and the Agentic category is evaluated with a strict exact-match criterion (Section 4.5).
3.4. Agentic Dataset
The Agentic dataset is divided into three categories. We de- scribe them in detail below. They share similar data curation pipeline as the single-turn dataset in Section 3.1.
3.4. 1. WEB SEARCH DATASET
For this category, the LLM has two tools: DuckDuckGo search function that retrieves webpage titles, snippets, and URLs, and a fetch function that extracts full webpage con- tent. To ensure fairness across models with varying knowl- edge cutoffs, questions focus on recent but stable informa- tion, such as the 2024 TIME Person of the Year, rather than constantly changing data like stock prices.
We later quantify the agreement among the AST substring- matching and execution-respons. matching metrics (Sec- tion 4.3) to validate the reliability of the AST approach, highlight differences in inference strategies for prompt-only versus function-calling models (Section 5.1), and, finally, leverage perplexity ratios on the single-turn and crowd- sourced settings to detect potential data contamination in the BFCL single-turn corpus.
3.4.2. MEMORY DATASET
Our memory dataset spans five distinct domains, each chosen to reflect a practical, real-world use case for LLMs-college advising, customer support, medical as- sistant, etc. Within every domain, the model is first given concise setting prompt (e.g.. "You are an academic advi- sor helping sophomore plan their coursework") and then
4.1. AST Substring Matching
Evaluating function calls by direct execution can be chal- lenging due the limited availability of executable func- tions and the laborious process of manual implementation. which curtails the diversity of functions available for testing.
4

---

The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models
if it passes both checks in all turns.
Models
GPT 4 1106 preview
0.9
Trend Line
State-Basch Evaluation compares the system's 's final state after each turn (i.e., after all function calls) with the ground- truth state. Multiple sequences of function calls can achieve the same result, but the final state must match the labeled out- come. This approach captures modifications to the system (e.g., creating files or removing stocks from a watchlist).
Summary
0.8
Variance
0.7
Gemini-1.5 p.pppo
0.6
0.5
0.4
Desponse-Based Evaluation verifies that the model follows the necessary sequence of function calls (the minimal vi- able execution result path) to produce the requested output. This is critical for read-only requests (c.g., retrieving stock prices), where we want to ensure the model calls the appro- priate functions rather than guessing the result.
0.4
0.5
0.6
0.7
0.8
Summary
Figure 3: The Abstract Syntax Tree (AST) based evaluation (AST Summary) strongly correlated with the evaluation by executing the functions (Exec Summary) validating AST as a reliable off line evaluation methodology.
While state-based evaluation is a powerful technique, cannot detect whether non-state- functions (e.g.. get_zi wtw.:iipcdebbb.iiee ty or est aatimate.d stance) were actually invoked. We need response-based -ased checks to confirm the model is reasoning through the task reliably (e.g., calling get wtw.:ippooebb..citt...... (City-Name) ) be- fore get_weathherbyy siyyde (City_ _attyycddedd ). By combining both types of evaluation, BFCL provides deeper insight into the model's correctness and decision- making process.
To address this, we introduce an Abstract Syntax Tree (AST) substring matching approach that preserves alignment with execution-base evaluation without actual execution.
We restrict the model's output to Python-callable function calls using prompt-based instructions, then extract function names and parameters through Python's ast module. In- stead of requiring exact parameter matches, we verify that each parameter belongs to a predefined set of valid values. function call is correct if the function name matches ex- actly and if all parameter values fall within their respective possible answers. For details on the AST matching rules, please refer Appendix H.
4.5. Exact-Match Evaluation
During evaluation, the model is given explicit formatting in- structions through the system prompt detailed in Appendix J. We evaluate only the dedicated answer field with strict exact-match criterion. Focusing on this individual field pre- vents spurious positives that would occur the reference phrase appeared incidentally inside a longer, unclear sen- tence. For example, consider a yes/no question: a reply such as "I am not because relevant information was found" contains the token "no," but the model has not ac- tually committed to the negative answer. By isolating the answer field, such cases are not erroneously marked as correct responses.
4.2. Execution Response Matching
Execution response matching involves validating function calls by executing them and comparing the results against expected outcomes. There three ways compare the response. For functions that output deterministic results, we check for exact-match of the response. For functions whose outputs are time sensitive, we execute the ground truth func- tion call and the model's output function call simultaneously and match the results, accounting for real-time value fluctu- ations. For nested lists or dictionaries, matching, which only checks the length of the list and the presence of the dictionary key.
perform structure
Before matching, both candidate and reference answers are -ommaiieed-covvertedd lowercase and stripped of punctuation the same procedure in our AST-based evaluation. A prediction is deemed correct if and only if the normalized strings are identical.
4.3. AST Matching Performance
By comparing the scores and relative rankings the BFCL single-turn dataset, evaluated using AST and Execution in Figure 3, we observe a strong correlation between AST scores and execution-based performance. This suggests that AST matching serves as a reliable indicator of model effectiveness in real-world scenarios.
5. Results and Analysis
5.1. Accuracy
Table presents the evaluation results of various LLMs on BFCL. While the pop-efforming models excel in single- tur,, crowd-sourced, and hallucination-relate metrics, there remains significant room for improvement in multi-turn and agentic tasks, particularly in memory management.
4.4. State & Response Based Evaluation
For multi-tumn tasks, we employ two checks after each tur:: state-based and response-based An entry correct only
5

---

The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models
Table Evaluating different LLMs on BFCL. The categories are defined in Section 4
SypiTin
Oree Sverel
Agiicc
Feaiit
Tayl
26.1
613
1u4
30.5
109
10.1
790
11.5
61.5
614
41.
17.4
6-3
21.
666
14.
10.3
86.
383
613
315
11.8
18:
16.3
30.0
133
042
01.3
613
216
633
18.3
6-8
FA4
218
812
814
908
18.5
636
611
823
03%
321
1340
1888
313
170
3-1
536
02.
13.
180
61.
071
$15
124
114
16.3
161
116
113
110
100
10
68.1
133
05.
745
117
11.3
T1A
915
318
TLA
250
60
450
Sesaentttee (NC)
1-3
174
115
141
135
760
760
135
122
132
150
150
50
03
195
471
230
130
Overt thenes
365
903
K15
436
163
763
715
1803
310
1.0
430
014
311
384
341
180
140
945
K25
265
K58
013
113
191
114
135
0.4
383
815
133
868
173
765
811
T14
765
133
160
140
69
15
663
458
125
74
064
015
155
173
735
161
810
78
-28
114
283
K15
113
173
135
A6T
133
31
R3
110
6.3
14
264
180
265
110
623
134
154
154
ANT
1.15 Phopp
-14
114
965
618
111
66T
47
1803
48
20
213
378
76
358
756
623
13
125
18
Vien visaac)
113
Cocee FentseCaling 2g 200
264
114
215
14
515
113
615
481
153
765
15
43
13
013
125
453
15
Larad Chieat
865
948
Uiaa 1.18
253
460
517
15
53
Qveil 16 PaanehPappp
313
118
563
834
143
785
233
461
43
03
15
24
04
98
19
1--4 Boll Phopp
411
215
63
481
13
11-77-- (FC)
363
815
419
M8
108
134
M3
360
08
60
Grama 2-bb (heag)
318
613
211
965
419
41
304
133
739
634
48
714
05
69
Quee'' It bosutiec
714
291
18
56-
345
Queetl
11
661
63
133
559
163
1NT
164
08
60
Lime 111 doror IT
314
81
365
155
159
150
363
305
15
53
134
111
350
465
186
46
48
tho was c
9-3
115
115
158
Models that support native function-calling (FC), such as GPT-4, Gemini-1.5-Pro, Claude-3.5-Sonnet can run BFCL directly by supplying all function definitions in their t tools input field. In contrast, most models lack built-in function- calling capabilities. For these models, we use a prompt- based workaround: we guide them to produce structured function calls through the system prompt (detailed in Ap- pendix A), placing the function definitions in the system prompt rather than in dedicated tools field, Through- out this paper, we refer to models that have their native function-calling feature enabled as "FC models" (or operat- ing in "FC mode"), and those for which we rely on system prompts to trigger function calls as "prompting models" (or in "prompting mode").
function calls in FC mode, whereas it can in prompting mode. In addition, when handling other programming lan- guages (e.g., Java or JavaScript), models in prompting mode often outperforms FC mode.
Prompting models exhibit on average three times more de- coding issues than FC models (412.93 vs. 182.5 out of 4,251 total entries), aligning with the observation that structured FC-mode outputs are easier to parse. However, among successfully decoded responses in the multiple function- call category, FC models show more incorrect function-call counts on average (77.5 vs. 21). A similar trend appears in the parallel multiple category, indicating that prompting models are generally more flexible in complex scenarios.
a model supports both FC and prompting modes, we find that the FC mode outputs tend to be structured responses that lower parsing errors. However, these structural con- straints of the FC mode can limit the flexibility of model in complex function calling scenarios. We therefore see more capable models often performing better in the prompt- ing mode. For instance, Claude cannot execute parallel
5.2. Dataset Composition Difference
We construct the data generation pipeline for single-turn based on our understanding of the composition of real-life function-calling scenarios from our experiences building a function-calling LLM, and from function-calling documen- tations (OpenAI, 2025b) and community forums (OpenAI, 2025a). At that time, there were no formal crowd-soureed
6

---

The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models
datasets, After we collected the crowd-sourced part of this dataset, we found that there are quite few differences between the single turn and the crowd-sourced.
with our ground truth answers. The second is LLM-as-a- Judge approach to better understand the root-causes behind the the emrors.
In crowd-sourced, there are significantly more scenarios on multiple and much less parallel function calling scenar- ios. This observation reflects how most users interact with function calling: high demand for the feature of having to intelligently choose between functions and lower demand for making parallel function calls in single turn.
5.4.1. 1. DETERMINISTIC ERROR ANALYSIS
We classify the errors shown by our benchmarked models into 5 categories: Empty Turn Response Error, Instance State Mismatch, Execution Response Mismatch, Force Ter- mination, and API Error. An Empty Response Error refers to when a model does not make any function call for one or more turns in the conversation. An Instance State Mis- match refers to discrepancy between the model's 's internal representation of the API state and the expected ground truth state. An Execution Response Mismatch error occurs when the responses generated by the model for a specific turn do not include all the expected responses defined in the ground truth for that turn. The Force Termination error occurs when the model's processing is abruptly stopped dur- ing inference. This happens in cases when the model uses many steps within a turn in an attempt to answer the user's question leading to termination of its efforts. Lastly, the API Error simply groups all the cases the model's provided API endpoint failed to run on an entry in the dataset due to it being down the token length of our questions going over the maximum allowable input token length.
Crowd-sourced dataset also differ from single turn in that they contain multi-lingual user prompts, multi-lingual func- tion docs, as well as user prompts that contains lots of redun- dant information, etc. As an example of the rich diversity we observed in our dataset, we even include a classification function triggered through function calling.
On average, each entry in crowd-sourced contains 3 func- tion choices, with the maximum one having 37 function choices. Each function has an average of 4 parameters, with the maximum one having 28 parameters. Here are some statistics and distributions.
5.3. Parallel Funetion Call Ability
When a question requires multiple function calls (whether to the same function or different ones), the model can issue them all at once in single turn, or sequentially across multiple turns. For tasks that have interdependencies among calls, issuing parallel function calls in single turn significantly reduces latency. For instance, checking the stock prices of 20 different stocks simultaneously is far more efficient than making 20 separate requests.
2024 12 37
Zhen Tars lnvineeee feeennad
claucted S-wermet- 30243022--6
spe-to 2024- 20-c
gemins 2 nifeccheeppCC
We observe notable shifts in the evlolution of models' abili- ties to generate parallel function calls. Early models (e.g.. the Claude Sonnet) lacked any function call capability. Later iterations introduced partial support for parallel calls-albed with suboptimal performance. Interestingly, with the re- lease of flagship versions such as 122-022-11221------ and claude-33 3- 5- -oonnet-20241022 lel function call feature appears to have regressed or even removed! We hypothesize that this is because although par- allel calls can be more efficient for anniinterdepenent tasks, they may adversely affect accuracy when function calls are chained. In many real-world scenarios, each subsequent call relies on information returned by the previous one. As a result, generating calls one at a time-waiting for each exe- cution's 's result-might might ultimately be both faster and more accurate for these use cases.
DeepSeet V3
Limo 3.1-76
400
Figure 4: Error distribution across models on the BFCL Multi- Turn Dataset showing the counts of different error types: Empty Response Error, Execution Response Mismatch, Force Terminated. Instance State Mismatch, and API Error. The total number of entries in the dataset is 800.
the paral-
5.4.2. LLM-AS-A-JUDGE ERROR ANALYSIS
In addition to the mechanical error analysis, we employed LLMs-as-uudees to classify and analyze the root causes of failures in multi-turn interactions. We utilized few-shot structured prompting method query the LLMs about error cases. The prompts included detailed multi-turn conversa- tion logs, initial configurations, and user queries. The judges were tasked with explaining and categorizing failures into predefined types: 1) Failed to Understand Environment State: Errors stemming from inaccurate assumptions or hallucinated environment states. 2) Failed to Understand User's Request: Misinterpretation of user request specifi-
5.4. Multi Turn Error Analysis
We analyze the errors made by models in two ways: The first is through deterministic algorithmic approach for classifying the different ways model's results don't align
7

---

The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models
Distribution of Error Types Across Models
Table 2: While models display consistently high perplexity single-turn dataset then its crowd-sources counterpart, the relative difference entails the model's familiarrty to the single-turn which is open to be trained on as evaluation entries.
100
Ers Tadte to Lnndenttnd Ensirommnt State
Number of Errors
80
Feeled Underttand Functinn Decumentation
Tailed Lindenttnd Users Requect
60-
Model
PPL (single-turn)
PPL (crowd-sourced)
40-
LJama2-7B
3.47
2.56
Openfunctions-v2
3.09
2.49
20-
CodeL-lmma-7B
3.45
2.49
Caude Sornet 20243022 2020222222222 (FC)
ppt-4o-2224-11220 (FC)
c1-2024-12-17 (FC)
Amazon Neva Pro-11 (FC)
mistral-large-ae 2407 (FC)
Gemini- C--Flssh--ee :Eap (FC)
Owenz 5- 72B -istrittttcttt (Pronpe))
Liama 3 708 inet-ntttttt (Prompt)
DeepSeek-V/ (FC)
Meta-Ilama-3 8B
3.58
2.63
Mistral-7B
4.39
2.92
Functionary-7B
3.81
2.73
Salesforce xLAM-7B
3.67
5.09
Table 3: Character- level negative bigliikeldhood (Char-NLL) met- ric provides insights similar to those from Table 2. outputs of the function calls are structured, Char-NLL effectively captures model uncertainty and predictive performance at the token level across both single-turn and crowd-sourced settings. Lower values indicate better modeling of structured output.
Model Name
Figure 5: The bar chart shows the frequency of three error types-failed understand environment state, failed to under- stand function documentation, and failed to understand the user's request- acress-acrooss various Al models. Error types are color-coded, illustrating differences in model performance.
Model
Char-NI.I. (single-turm)
Char-NI.I (e.od((doo--sorrec
Llama2-7B
0,344
0.264
Openfunctions-22
0,295
0.239
cations. 3) Failed to Understand Function Documenta- tion: Errors due to misinterpretation misuse of provided function documentation. Full judge prompt template is in Appendix F
CodeLlama 7B
0,342
0,256
Meta-LJama. 8B
0.322
0.245
Mistral-7B
0.295
Functionary- 7B
0.337
0.254
Salesforce xl. ALMA-7BB
0,340
0.427
The most prevalent failure mode across models was Failed to Understand Environment State as can be seen in Figure These errors occurred when the model either hallucinated or assumed incorrect environment state information, includ- ing attempting actions in incorrect directories or failing to execute necessary steps due to premature termination of actions due to state misalignment. Failed to Under- stand User's Request was the second most frequent error type. These failures typically arose when the model misin- terpreted user intent, such as returning unsorted data instead of sorted content or failing to execute requested multi-step operations in the correct sequence user requested.
Tables and present the perplexity and char-NLL metrics, respectively, for several open-source models on the origi- nal single-turn benchmark versus the novel crowd-sourced dataset (wherein lower values signify superior predictive performance). It is observed that the majority of models achieve comparable or improved performance, i.c., lower perplexity and char-NLL on the to single-turn. For instance, Llama 2-7B (Touvron et al., 2023) exhibits perplexity of 3.47 on single-turn compared to 2.56 crowd-sourced, and char-NLL of 0.344 versus 0.264. Similarly, Openfunctions--2 and CodeL.lmaa demonstrate slightly enhanced performance on the crowd- sourced queries. This trend implies that these models did not derive an unfair advantage from memorization on the original benchmark. This finding supports the inference that their strong benchmark scores are attributable to gen- uine capability rather than exposure to test solutions. In contrast, Salesforce XLAM-7B (Zhang et al., 2024) shows an increase in perplexity from 3.67 to 5.09(char-NU NLL from 0.340 to 0.427) on crowd-sourced, representing a notable performance degradation.
data relative
also
5.5. Measuring Data Contamination using crowd-sourced
Traditional benchmarks inadvertently compromise if model's training data overlaps with the evaluation set, lead- ing to artificially low perplexity and Negative log likelihood on those benchmarks. By comparing language modeling metrics between crowd-sourced and single-turn, we can diagnose possible data contamination or overfitting, In par- ticular, an abnormally low perplexity or character-level neg- ative likelioodo (char-NLL) on the static single-turn benchmark, coupled with a significant performance drop on crowd-sourced, would signal that the model may have memorized the former. On the other hand, consistent per- formance across single-turn and crowd-sourced suggests genuine generalization rather than training exposure to the test answers
These performance decrements underscore the utility of the crowd-soureed dataset as a "stress test" for models: any model that has primarily memorized benchmark solutions or was overly tuned to the static test distribution is likely to be exposed through a significant score regression on the new dataset. In the instances of xLAM,, the crowd-sourced evaluation reveals vulnerabilities that the original static test masked. It is noteworthy that both are specialized or smaller-
8

---

The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models
scale models, which may have been exposed limited
(Live)
data variety; consequently, when confronted with genuinely novel queries, their performance markedly declines.
XLAM-78
Overall, the perplexity and char-NLL metrics across single- turn and crowd-sourced present a coherent pattern. Figure 6, and visualize these results, plotting each model's perfor- mance on single-tur against its performance on crowd- sourced. The majority of data points are situated near the diagonal, signifying comparable proficiency both datasets. In contrast, the outlier models deviate sharply up- ward, indicating higher perplexityNNLL on crowd-sourced This analysis substantiates the practical value of the BFCL Live methodology. By incorporating a real-world test set, more robust indicator of contamination or overfitting is obtained. Models cannot rely on memorized answers for crowd-sourced queries; thus, any substantial discrepancy in metrics serves as an immediate flag for potential eval- uation inflation. In summary, these comparative metrics confirm that top-performing models maintain strong gen- cralization on fresh data, whereas models exhibiting any indication of contamination or poor robustness are clearly identified by the performance gap between their single-turn and crowd-sourced scores.
Mistral-78
Lama2-78
Codel, ama-78
Fanctionary-88
Lama-33 8B
opentunctions. v2
0.00
0.15
0:20
0.30
0.40
ChanNLLL
Figure 7: Character-level NLL on crowd-sourced dataset.
information. While some models handle this correctly, most skip listing the keys and instead attempt to guess a key when calling retrieve leading to mismatches and errors. Even worse, many models give up after single failure, mistakenly concluding that the information unavailable rather than attempting to look up valid keys.
Character-leve NLL (ilge-t)
6. Conclusion
XLAM-15
Mistral-7B
The Berkeley Function Calling Leaderboard (BFCL) estab- lishes a new standard for evaluating large language mod- els' (LLMs) ability to invoke and manage external tools and APIs. By introducing a comprehensive benchmark that spans single-turn, crowd-sourced, multi-turn, and agen- tic scenarios, BFCL provides robust and scalable frame- work for assessing the function-calling capabilities critical to agentic Al systems. Its use of Abstract Syntax Tree (AST) based evaluation ensures reproducibility and avoids the scalability limitations of execution-based methods. The inclusion real-world, multilingual user queries further enhances benchmark's 's practical relevance.
Uama2-76
eLlama-70
tionary-78
inctionsv22
0:00
0.05
0.10
Figure 6: Character-level NLL on single-turn dataset.
Our analysis reveals that while many LLMS perform well on simple single-turn tasks, they often falter in more complex agentic and memory-intensee the gap between current LLM performance and the demands of real-world, long-horizon reasoning tasks. Moreover, our comparative evaluations with crowd-sourced data expose po- tential overfitting static benchmarks and highlight BFCL's role in stress-testing generalization.
5.6. Memory Management Category Analysis
Current models struggle with memory tasks; even the bench- mark leader, openai ol 11-2024-12--2 12-17 (FC). reaches only 12% accuracy. We observe models demonstrate faulty behaviors in memory management and organization. For instance, when preserving the fact that "the user is fourth-yeal male CS major," some models save as one key (user profil le), others split it (ma jor, year, gender). Split- ting quickly exhausts key space, demands frequent merges, and complicates retrieval.
scenarios. This underscores
As function-calling becomes a foundational capability for L.L ...---oee applications, BFCL. serves essential tool for the community. We hope this benchmark not only drives advancements L.M architectures and training but also promotes transparent and reproducible evaluation prac- tices in the development of functionally robust Al agents.
Models often hallucinate during key retrieval. Because the key-value store requires exact matches, models should first call 11 keys s to view existing keys before retrieving

---

The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models
Impact Statement
Erdogan, L. E., Lee, N., Jha, S., Kim, S., Tabrizi, R., Moon, S., Hooper, C., Anumanchipalli, G., Keutzer, K., and Gho- lami, A. Tinyagent: Function calling at the edge, 2024, URL https: htppivxxiii.v../.../.a org/ abs/ 220...409.000000 00608.
The introduction of the Berkeley Function Calling Leader- board (BFCL) represents a sigmificant advancement in the evaluation and benchmarking of large language models (LLMs) for function-calling capabilities. Function calling is an increasingly critical skill for LLMs, enabling them to in- tegrate seamlessly with external systems, perform complex tasks, and reason effectively in stateful, multi-turn interac- tions. Despite its importance, existing benchmarks inad- equately capture the diversity, complexity, and real-world applicability of function-calling scenarios.
Gao, D., Wang, H., Li, Y. Sun, X., Qian, Y., Ding, B., and Zhou, J. Text-to-sql empowered by large lan- guage models: A benchmark evaluation. Proc. VLDB Endow., (15):1132-1145, January 2024. ISSN 2150- 8097. doi: 10.147778/66412244.444422 URL https: wtt..o///...17//3661111101 3641221.
Ge T., Chan, X., Wang, X., Yu, D.. Mi, H., and Yu, D. Scaling synthetic data creation with 1,000,000,000 personas, 2024. URL https: http://xxxvvvoo./o/o
BFCL addresses these challenges by presenting multi- faceted benchmark that evaluates L.LMs across single-turn, multi-turn, crowd-sourced, and agentic datasets. By lever- aging innovative techniques such as Abstract Syntax Tree (AST) substring matching for scalable and deterministic evaluation, and incorporating real-world user-contributed data, BFCL sets new standard for evaluating LLMs' capa- bilities.
Guo, Z., Cheng, S., Wang, H.,, Liang, S., Qin, Y., Li, P., Liu, 7., Sun, M., and Liu, Y. Stabletoolbench: Towards stable large-scale benchmarking on tool learning of large language models, 2024.
This benchmark has the potential to significantly shape the development of next-generation LLMs by providing researchers and practitioners with a comprehensive tool to assess and improve function-calling performance. More- over, it paves the way for more robust, adaptable, and ethically-aligned L.M deployments in diverse domains such as healthcare, finance, and education.
Guo, Z., Cheng, S., Wang, H., Liang, S., Qin, Y., Li, P., Liu, Z., Sun, M., and Liu, Y. Stablctoolbench: Towards stable large-scale benchmarking on tool learning of large language models, 2025, URL http:: //tpxv..xg////////// abs 2403. 07714.
Huang, S., Zhong, W., Lu, J., Zhu, Q., Gao, J., Liu, W., Hou, Y., Zeng, X., Wang, Y., Shang, L.,, Jiang, X., Xu, R., and Liu, Q. Planning, creation, usage: Benchmark- ing LLMs for comprehensive tool utilization in real- world complex scenarios. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 4363- 4400, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 18663//112024. findings-acle 251. URL https: / accanthology.. org/2024 findingn acl. .259/.
Ultimately, BFCL contributes to the broader goal of making LLMs more effective, and reliable in real-world applications, fostering innovation and ensuring responsible Al use.
References
Attouche, L., Baazizi, M.-A Colazzo, D., Ghelli, G., Sar- tiani, C., and Scherzinger, S. Validation of modern json schema: Formalization and complexity. Proceedings of the ACM on Programming Languages, 8(P5PL):1451- 1481, 2024.
Kim, S., Moon, S., Tabrizi, R., Lee, N., Mahoney, M. W., Keutzer, K., and Gholami, A. An LLM compiler for parallel function calling. In Forty-first International Conference on Machine Learning, 2024. URL https: / //openreview. net/ forum fow.mmidduuuuuuuFFFFFF jnF.
Basu, K., Abdelaziz, I., Chaudhury, S., Dan, S., Crouse, M., Munawar, A., Kumaravel, S., Muthusamy, V., Kapani- pathi, P., and Lastras, L. A. Api-blend: A comprehensive corpora for training and benchmarking api Ilms, 2024. URL https: / /aa //aaxtxa aaxiiv www..org/aaa.o at/////s/222 /2402.222222 15491.
Lin, C.-Y. ROUGE: A package for automatic evalua- tion of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Asso- ciation for Computational Linguistics. URL https: //aclanthologe . wwgg//00-0013.3
Chen, Z., Du, W., Zhang, W., Liu, K., Liu, J.. Zheng, M., Zhuo, J., Zhang, S., Lin, D.,, Chen, K., and Zhao, F. T- eval: Evaluating the tool utilization capability of large language models step by step, 2024. URL https:// arxiv.org abs //1//2211.1110
Lu, J., Holleis, T., Zhang, Y., Aumayer, B., Nan, F., Bai, F., Ma, S., Ma, S., Li, M., Yin, G., Wang, Z., and Pang, R. Toolsandbox: A stateful, conversational, interactive evaluation benchmark for Ilm tool use capabilities, 2024. URL https: //arx/iaavv.//////////a/ear //ta//aiv.rgg/bbaae../. 24682.068.84... 04682.
Cohere, T. Command a: An enterprise-ready large lan- guage model, 2025. URL https: /ttp::///axxvoooo org/ abs/2504. 00698.
10

---

The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models
Marten, R., Vu, T. Cheng-Jie Ji, C., Sharma, K., Dimakis,
Srinivasan, V. K., Dong, Z., Zhu, B., Yu, B., Mao, H., Mosk-Aoyama, D., Keutzer, K., Jiao, J., and Zhang, language model for function calling. In NeurlPS 2023 Work- shop on Instruction Tuning and Instruction Following, 2023. URL https:/ wpwpppeeeeeeeeeeeeeeeeeee net/ forum? id-Md6RUrG267.
A., and Sathiamorrthy, M. Curator, January 2025.
J. Nexusraven: commercially-permissi
MetaAL. The Ilama 3 herd of models, 2024. URL https:
//arxiv. abs/ 2407. 21783,
OpenAI. Gpt-4 technical report, 2024. URL https:// arxiv. wwg.orrg///..... 22t../////3/.... 287744
team, N. Nexusraven-v2: Surpassing gpt-4 for zero-shot function calling, 2023, URL htttt:t///xxuuufll
OpenAI. Openai community forum, 2025a. URL https: //community openai com. Community of devel- opers building Al-powered applications.
si/blogs/ravenv2.
Team, Q. Qwen3 technical report, 2025. URL https:
OpenAl. Function calling, 2025b. URL https:
///aaxiv.rrg/ab//2222 /tt..//b...55500088... 09388.
//plat form. wow..oonna.......... com/ docs / gui des/ function- calling. Documentation for integrating OpenAl models with custom code and external services.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C.,, Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, S. Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Moly- bog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subra- manian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I, Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Sto- jnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https: /at::://axxv..v/ www.xiii.r.g.....o abs 000.309788.....
Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., and Gonzalez, J. E. Memgpt: Towards Ilms as operating systems, 2024. URL https://arxix 2310. 08560.
Panickssery, A., Bowman, S. R., and Feng, Lim eval- uators recognize and favor their own generations, 2024. URL https: //rxxiv
org/ abs/a/ss///20 13076.
Patil, S. G., Zhang, T., Wang, X., and Gonzalez, J. E. Go- rilla: Large language model connected with massive apis. In The eiiht Annual Conference on Neural Information Processing Systems, 2024. URL https: //openreview net www./neeut/teeeeeee
Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., Zhao, S., Hong, L.., Tian, R., Xie, R., Zhou, J., Gerstein, M., Li, D., Liu, Z., and Sun, M. Toolllm: Facilitating large language models to master 16000+ real-worid apis, 2023. URL https:/ /ates//axxiin www.wiwww./.w..t.aa........ooore htgg/b//////a/
Trivedi, H., Khot, T., Hartmann, M., Manku, R., Dong, V., Li, E., Gupta, S., Sabharwal, A., and Balasubramanian N. Appworld: A controllable world of apps and people for benchmarking interactive coding agents, 2024. URL https: /atp:://axivvvvii org/ a2/////s//24/ 2404.189711
abs/ 2307 16789.
Sasaki, M., Watanabe, N., and Komanaka, T. Enhancing contextual understanding of mistral Ilm with external knowledge bases. 2024.
Vu, T., lyyer, M., Wang, X., Constant, N., Wei, J., Wei, J., Tar, C., Sung, Y.H.,, Zhou, D., Le, Q., and Luong, T. Freshlims: Refreshing large language models with search engine augmentation, 2023. URL https://aaa //arxiv.. org/ abs / 2210.230111 03214.
Schick, T., Dwivedi- Yu, J., Dessi, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves to use tools. In seveth- Conference on Neural Information Processing Systems, 2023, URL https::////. wttp://wwweeeeeeeeeeeeeeeeeeeeep net/ forum? id=Yacmp254TH.
Yao, S., Chen, H., Yang, J., and Narasimhan, K. Web- shop: Towards scalable real-world web interaction with grounded language agents. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, vol- ume 35, pp.
Curran Associates, Inc., 2022.
Song, Y., Xiong, W., Zhu, D., Wu, W., Qian, H., Song, M., Huang, H., Li, C., Wang, K., Yao, R., Tian, Y., and Li S. Restgpt: Connecting large language models with real-world restful apis, 2023. URL https: : arxii.. org/abs/ abs 2306 6, 06624.
Yao, S., Shinn, N., Razavi, P., and Narasimhan, K. bebench: A benchmark for tool-agent-user interaction in worldworld domains, 2024. URL https:/ aarxiv. org/ /ba/ 2406. 12045.

---

The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models
Zhang, J., Lan, T., Zhu, M., Liu, Z.,, Hoang. T., Kokane, S., Yao, W., Tan, J., Prabhakar, A., Chen, H., Liu, Z., Feng. Y., Awalgaonkar, T., Murthy, R., Hu, E., Chen, Z., Xu, R., Niebles, J. C., Heinecke, S., Wang, H., Savarese, S., and Xiong, C. xlam: family of large action models to empower agent systems, 2024, URL https:// arxiv. org/ abs /b////4/..22.....//// 2409. 03215.
Zhong. Z. and Chen, D. A frustratingly easy approach for en- tity and relation extraction. In Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tuu Tur, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., and Zhou, (eds.), Pro- ceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, pp. 50-61, On- line, June 2021. Association for Computational Linguis- tics. doi: 10.18653/11/002111 naaell..aalamaiiaa URL https: / //aclantholog/ org/ 2021, naac 1mmmmminn.... 5/.
12

---

The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models
A. System Prompt for Prompting Models
To enable prompting models to execute function-calling actions, we use the following universal system prompt, where [functions] placeholder is replaced with the function documentation(s):
You are an expert in composing functions. You are given question and of possible functions. Based the question, you will need to make one more function tool calls to achieve the purpose.
If none of the functions can be used, point it out, If the given question lacks the parameters required by the function, also point it out.
You should only return the function calls in your response.
If you decide invoke any of the function(s) (s), you MUST put it in the format of Junc_names params namel=pamam , params name2=parmms.vdlue munc_name21 params)]
You SHOULD NOT include any other text in the response.
At each turn, you should try your best to complete the tasks requested by the user within the current turn. Continue to output functions to call until you have fulfilled the user's request more functions and proceed to the next lurn or lask.
the best of your ability. Once you have call, the system will consider the current turn complete
Here is a list of functions in JSON format that you can invoke. (aleee/toffns
B. Data Augmentation on Function Documents
Parallel Functions Category: We created an additional user query that invoked the same function but with different set of parameter values. This allowed us to expand the dataset by having multiple questions that invoked the same function, each with different input parameters. For example:
Parallel Functions Augmentation
queryl + [f' name' funcl', 'description': order takeout' }] -> ansl query2 (which contains queryl) [[' name": ' funcl', 'description': 'order takeout }] -> [ansl, ans2]
The key transformation here is introducing query2 and generating ans2 based different set of parameter values.
Multiple Functions Category: In this category, we combined several function documents from different base entries to introduce distractor functions. However, the user query remained unmodified. These distractor functions were meant to test whether the model could accurately select the relevant function and not be misled by additional, unrelated function docs. GPT was used to ensure that none the distractors could be alternative solutions to the function call. For example:
Multiple Functions Augmentation
query [( name': funcl', ddscription' 'order takecut')) }] -> ansl query + [f name': funel', ddscciiption' forder takeout' { name"
func2', 'description: 'get weather'
-> [ansl]
Here, the distractor func2 is added to test the model's ability to focus on funcl and avoid being distracted by irrelevant functions.
Multiple Parallel Functions Category: We combined multiple user queries and function documents from different base
13

---

The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models
entries, ensuring that function docs combined than user queries. This transformation tested the model's ability to handle multiple function calls and filter unused functions. Some of the functions remained unused in the process, creating more complex multi-query scenario. For example:
Multiple Parallel Functions Augmentation
queryl + [funcl] -> anal
queryl + query2 [funcl, func2, func3] [ans1,
ans2]
The transformation here introduced query2 and added func2 and func3 while testing how the model handles the multiple function calls.
Function Parameter Removal: Based on the base entry, asked GPT to remove from the prompt, while keeping the function document unchanged. In this the model expected to either ask follow-up question to clarify the missing information return eror message instead of making function call (which would be considered hallucination). This will be used in the irrelevance category. For example:
pieces of parameter information
Function Parameter Removal Augmentation
queryl + [funci] -> ansl queryl' (missing parameter info) [funcl] for clarification.]
[No Function Call,
Model asks
The key transformation involved removing the necessary parameter info in queryl and testing whether the model responded with clarification етгог message.
Function Removal: In this case, we removed one or more invoked functions from the function list in the augmented multiple parallel entries (from the previous step). The model was expected to either ask for more information on the missing function or produce an error indicating the absence a relevant function for query. This will be used in the irrelevance category. For example:
Function Removal Augmentation
queryl + query2 [funcl, func2, func3] [ansl, ans2]
queryl + query2 [funcl, func3] >> [No Function Call, Model asks for clarification.]
The transformation involved removing func2 from the list and verifying whether the model recognized its absence, producing the appropriate error message.
C. Single Turn Dataset Formal Definition
C.1. Dataset Structure
Each entry in the dataset consists of user query, list of candidate functions, and corresponding expected model output:
AST-based functions: Each entry is represented as:
(g. F, A), where is the user query, F = (f1, ......) is the function set, and A is the labeled funcciin and valid parameter set.
(1)
Executable functions: Each entry follows:
(q. F. fP)), where f(P) is the expected executable function call, and
(2)
P [p1. is the set of parameters used in the function.
14

---

The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models
C.2. Dataset Categories
We categorize function-calling scenarios based on the number of available tools and the type of invocation:
Simple (S): Single available tool, single function invocation.
|F| 11, [f(P)=
(3)
Multiple (M): Multiple available tools, single function invocation.
11 1, [f(P)==1
(4)
Parallel (P): Single available tool, multiple function invocations single turn.
|F| = 1, [f(P) >
(5)
Parallel Multiple (P.)): Multiple available tools, multiple distinct functions called in parallel.
| 1, [f(P))
(6)
Relevance (R): At least function call is expected.
Fl 2 1,1f(P) 21
(7)
Irrelevance (Z): Single or multiple available tools, but no function invocation is expected.
1F f(P) =
(8)
D. Multi-turn Dataset Formal Definition
D.1. Multi-turn Dataset Structure
Dataset Structure
Each dataset entry consists of multi-turn user query sequence and corresponding ground truth function trajectory:
(Q,T) T),
where Q = (2,,,1,,...0 12, (n) represents the user queries,
(9)
(T1. T2, ..., J represents the assistant's expected function call trajectory per turn.
D.2. Dataset Categories
The dataset is divided into four categories:
Base (B): Standard multi-tura turi tasks where each turn has at least one expected function call.
Vi,
(10)
Missing Parameters (MP): One turn contains an underspecified query, and the expected assistant response is natural language follow-up.
E s.t. T; 0, a++1 resolves missing parameters.
(11)
Missing Functions (MF):: One turn requests function that is unavailable, prompting the assistant to respond with clarification request.
3i s.t. = 0, g++1 provides missing function information.
(12)
LLong Context (CC): User queries and assistant responses in each turn contain a high number of tokens.
Vi, Igdl + |m|
(13)
15

---

The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models
E. Data Generation Pipeline Implementation Details
E.1. Single-turn Dataset
For the single-turn tasks, we divide the data collection into two categories based on their evaluation method: AST categories that use Abstract Syntax Tree, and Execute categories that evaluate by execution. The evaluation methodology is discussed in 4.1 and 4.2.
Data Collection
AST: We collect functions from popular GitHub repositories (top 100 starred) in Python, Java, and JavaScript. These functions are well-documented, making them ideal candidates for our downstream tasks. We exclude trivial functions such as _init_..__e_........ and functions with fewer than two parameters (excluding the self parameter) to ensure complexity and relevance.
Execule: The category is divided into two sub-categories based on the backend type. 1) Pure Python Functions: We manually constructed functions inspired by common math and physics calculations. These are purely executable Python functions that don't rely external APIs. 2) Python Functions Wrapped APIs: This sub-category includes functions that invoke API calls from popular public API providers such as ExchangeRate API, OMDb API, and Geocoding API. We focused on GET requests, as they are the most common in real-world scenarios. These functions demonstrate the model's ability to generate executable REST API calls through complex function documentation, using zequests. get () along with the API's hardcoded URL and a description of the function's 's purpose and parameters.
Data Preprocessing
We pre- pee-proesss them extract useful context for downstream data generation tasks.
AST: We extract function names, descriptions, parameter names, types, and default values directly from signatures and docstrings.
Execute: For executable functions, we use Python's requests get () as function document template. The schemas included base URLs, query parameters, path parameters, and body parameters.
Data Generation We transform the extracted function information, such as docstrings from python functions and API documentation from ExchangeRate, into well-formatted function documents. This transformation ensures consistent formatting, including proper descriptions of parameters, types, and default values, making them compatible with downstream evaluation pipeline. Once the function documentation was generated, realistic user questions were created based on these documents and their use in the original codebase or API context.
Data Transformation To introduce complexity and mimic diverse real-world function-calling scenarios, we expand the dataset through various transformations detailed in B. These transformations included augmenting the entries to simulate different function calling patterns, such as parallel and multiple, and introducing scenarios with queries having incomplete missing information to test the model's behavior.
Data Validation We ensure 1) function documentations adhered to the BFCL format, including all required function schema fields. 2) The function parameters are precisely defined and correctly categorized. 3) User prompts were relevant, clear, and properly aligned with the corresponding function documentation. We've instructed three human experts
E.2. Crowd-sourced Dataset
Data Collection For the crowd-sourced dataset, 64,517 real-world user queries are collected between 2024-02-26 and 2024-04-01 via our hosted model endpoint.
Data Preprocessing
To preprocess the collected data:
Deduplication: We applied the ROUGE-L (Lin, 2004) score and OpenAl's text-embedding models to remove duplicate queries and function does.
Exclusion of Public Datasets: We filtered out any queries from public test sets such as those from Nexus Function Calling Leaderboard to prevent contamination.
Data Parsing: The valid function documentation was then parsed into a JSON format compatible with the BFCL evaluation pipeline.
16

---

The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models
Data Generation
The expert-curated dataset doesn't I have data generation phase, because all entries are authenticate user-contributed data. The result from the pre-processing phase go directly into the transformation phase.
Data Transformation Minimum Edit Transformation: Using the minimum edit principle, human annotators applied necessary corrections to improve clarity, precision, and consistency without changing the core content of the function docs or user prompts.
Data Validation In addition to all the data validation step used in the single-turn section, we also make sure that 1) The transformed function doc and prompt preserve their original semantic meaning. 2) Any sensitive information in user prompts was replaced with placeholders to maintain privacy, and ambiguous content was clarified.
E.3. Multi-turn Dataset
Data Collection The multi-turn dataset began with the creation of a custom API codebase that spanned eight domains, including Vehicle Control, Trading Bots, Travel Booking, File System, Messaging, Twitter, Ticket Booking, and Math. Each API was designed simulate real-world multi-turn function calls.
Data Preprocessing We constructed graph of function dependencies, where each function represented node, and edges mapped output dependencies. This setup allowed us model realistic multi-turn interactions across different APIs and domains.
Data Generation
The data generation process for multi-turn interactions involved:
Task Generation:
We prompt GPT-4o-0806a to invoke a series of function calls and then derive a natural langugage query that requires the function trajectories. The questions vary in tone and style to simulate different user personas and interaction scenarios.
Precisely, we adopted the dataset from Persona Hub (Ge et al., 2024) to generate diverse evaluation dataset with different personas ranging from people with different occupations, age groups, etc. For example, a persona can look like:
High school physics teachers Science historians Elderly hermits Each persona would have a unique style to phrase the request.
Function Lists: For each task, we provided list of available functions from both primary and companion APIs.
Initial Configurations: We set up initial states (e.g., pre-authenticated sessions) to avoid unnecessary interactions and focus meaningful multi-turn tasks.
Human-labeled Ground Truth: Expert human labelers reviewed and labeled each data point with ground truth for each multi-turn interaction.
Data Transformation
During data transformation, we scaled the dataset by sampling execution paths through the graph. Additionally, incomplete tasks were fixed by introducing additional configurations and function calls maintain coherence.
Data Validation Validation in multi-turn interactions involved:
Question Validation: Ensuring that the questions were specific and complete.
Ground Truth Validation: Verifying that the multi-turn function call sequences matched the ground truth.
Initial Configuration Validation: Ensuring that the initial configurations were complete and relevant to the multi-turn
tasks.
Function List Validation: Checking that all necessary functions were included in the task's 's function list.
API Code Validation:
Using unit tests and format checkers to ensure that the API code was consistent and complied with the required standards.
17

---

The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models
F. LLM Judge BFCL Error Analyzer Implementation Details
use GPT-40-08-06 as a judge, using bespoke curator (Marten et al., 2025) for structured synthetic data generation. The judge prompt formatted as follows,
### ** Error Analysis Prompt**
* :o
You are an *serror analysis expert** tasked with identifying and classifying failures in the Al assistant's 's responses. Your goal is to determine if and where the assistant failed, categorizing the root cause as one of the following:
F*ilee to Understand Function Documentation **
F*ilee to Understand User's Request s*
**Failed to Understand Environment State **
**No Failure
** Inntrructions:
Carefully analyze the provided =*multi-tumt conversation identify any failures and their underlying causes.
### aniniall Configuration and User Queriesus
**Initial Configuration:s (intratiioiiiiiiiiiiiimmmm {initial_config] R*Related Function Documentations::*
**List of User Queries:** (user-queriee
### ** Evaluattio Process
Use the following guidelines critically evaluate the multi-ture responses:
1. *ompprre the model's function call traces** with the ground truth function call Iraces to identify any discrepancies in API usage.
2. =*Compare the end state*s with the ground truth state to determine if the model achieved the correct outcome.
3. **Pay attention to any mechanistic errors*s reported by the state checker, as these indicate validation failures.
### w* **Entry ID: {entry-iee id) ** #### **Turn-by by-urr Breakdown*s *
(turnsbrreakdown)
### ** State Checker Results**
18

---

The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models
### ** Failure Categorization
A failure occurs **ONLY** if:
The environment's 's **final *tate** differs from the **ground truth state **, OR
The assistant **failed to produce the minimum required trajectory *s.
Exploraaion Steps**
The assistant may take exploration steps, which may result in execution errors.
**Do not penalize*s the assistant for exploration steps.
=*#nnly mark a* the critical step that leads to the failure.
If the ssstate checker reports an error**, a failure s*definitely ** occurred. If the esstate checker reports "None"**, check if the model failed to produce the ssminimum required trajectoryss. **.
### *=Example Failure Analyses** #### **Example 1**
** Context: Fuel tank has gallons of gas initially. Max capacity is 50 gallons
User:*# Fill the fuel tank until we are able to reach Rivermist. Oil costs money just need to reach there, I don't need full tank.* Assistant Response: 'fillFuelTank (fmuutttm
** Ground Truth:** "displayCarStatus 'fuel") ") fillFuelTank (fuelmmmun=mmm))
Failure Analysis:
=*Falr Type:** =Failed to Understand Environment State* **Description: The model filled the tank to maximum capacity without checking the current fuel level first. **Turn DD: '0'
#### ExExmle 2**
** ==Coteext:** User has a trading account with $10,000 balance. User:** #I want to buy some Apple stock, but first tell its current price and make sure I can afford at least 5 shares Assistant Response:** 'execute trade "oymbol='APPL: quantity=5, order type=' market', side='buy')"
Ground Truth:**
wew.stock.piice (symbol='AAh) ='APLL
Faiilue Analysis:
=*Failure Type:** =Failed Understand User's Requests
*Deescipption: The model executed trade immediately when the user only requested price information and affordability check.
**Turn 1D:** '0'
### ** Final Task
Now, identify the failures using the above categorization. **Only identify ONE root cause of the failure.*. **
19

---

The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models
If there is *NO failure in ANY turn, wereport exactly ONE entry ** with **Faaiure Type:** 'No Failure*
**Description::s 'No Failure Occurs
**Root Cause:** 'No Failure Occurs
**Turn 1D:* '-1'
### ** Your Failure Analysis:
G. Function Calling Model Performance over time
PLease refer to Figure 7.
Function Calling Model Performance Over Time
OpenAl
GPT-4: 85%
90%
Anthropic
85%
Mistral
Expert-curated AST Accuracy
Mistral-Large-1: Claude-3.5 3.5ronttt-oo 75%
Mistral-Large-2: 73%
75%
70%
Claude-3-Opm 3-Opus: 69%
65%
Demini-1.5-pree.
55%
50%
45%
Gemini-1.0ppro: 41%
2023-06-01
2023-07-31
2023-09-29
2023-11-28
2024-01-27
2024-03-27
2024-05-26
2024-07-25
2024-09-23
Figure 8: Models from 2023 and early 2024 struggled with reliable function calling scale. As model sizes grew and function calling became post-training objective, their capabilities improved significantly.
H. BFCL AST Substring Matching
Parallel Tool Calls
By definition, parallel tool calls execute simultaneously; therefore, their order is irrelevant. If strict sequencing required, the model must emit one function call time and wait for its completion before producing the next.
Given predicted sequence A [a,, .2. Um] and a ground-truth sequence B [b,, be...., buj.
we do not require positional alignment;
any predicted call a, may match any ground-true truth call by.
The evaluation follows an all-or-nothing rule: even a single ground- truth call is unmatched, the entire prediction fails. This ensures the model identifies all required calls, regardless of order.
20

---

The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models
Possibile Arsswr
Ourput Function
Description
Error: Parameter Hallucination
No.
No unexpected
parametors?
Yes
Yes
Errrr: Type Mismatch
No.
Parameter Types Match?
Enror: Funation Mismatcch
No-
Funcsion Masch?
YES
Dist Checker
List Checker
Saring Checker
Extract Funciion Parameters
Error: Required
Parameters Missing
No
Requied Parameters Presenk?
Error: Paramater
Values Mismatch
Value Match?
Valid Function Call
Figure 9: BFCL AST Substring Matching procedure. The model response is parsed to extract function information in the systematic manner outlined above.
Parameter Values
INTEGER Vs. FLOAT
Python only: an int may be supplied where float expected (Python auto-converts).
Java & JavaScript: when documentation specifies a float, the model must output literal float (e.g. 5. 0); an int such is incorrect.
Supplying float for an int parameter is invalid in all languages.
LIST AND TUPLE
Order matters: [1,2,3] # [2, 1, For order-agnostic questions, all permutations of the correct answer are enumerated.
Type matching is recursive for nested structures; outer and inner element types must satisfy the specification.
STRING
Comparison is case-insensitive.
All strings are standardized before checking:
whitespace removed,
punctuation -/--* (note: and ") stripped.
Examples
Possible dates: ["20th June", "2023-06-20", "06/20/2023", "Jun.20 2023"] Possible locations: ["New York City", "NYC"]
DICTIONARY (DICT)
Key presence and value correctness are checked.
Key order is ignored (dictionaries are inherently unordered).
21

---

The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models
LIST DICTIONARIES
The list order of dictionaries matters.
Within each dictionary, key order does not matter.
Cross-Language Notes
For Java and JavaSeript, strings representing code constructs converted Python equivalents using Tree-Sitters before evaluation. During conversion, parameter types are also validated (e.g. Java long must end with L).
I. Function Parameters Distribution
Mean: 1.68
Mean: 2.74
Expert-Curated
0.4
Proportion
0.4
Proportion
0.3
0.3
0.2
0.2
0.6
2.97
Crowd-Sourced
0.5
Mean: 3.78
Crowd-Sourced
0.5
0.4
Proprtion
0.4
Proportion
0.1
0.1
0.0
0.0
10
15
20
25
30
10
15
20
25
Figure 10: Distribution functions (left) function parameters (right) in BFCL dataset between single-turn and crowd sourcdd categories. The histograms reveal that crowd-sourced data entries broader range and higher mean both functions and parameters count compared to single-turn scenarios. It's worth noting that crowd-sourced, we have entries with 37 functions and functions with 21 parameters, where that max number for single-turn is only 3.36 and 3.69, respectively.
J. Format Instruction Prompt for Agentic Task
This is the additional system prompt that models would receive on agentic dataset entries:
For your final answer to the user, you must respond in this format: ('answer': A short and precise answer to the question, "contett' A brief explanation of how you arrived at this answer or why it is correct]. If you do not know the answer, respond with ('answer': 'I do not know', 'context', 'I do not know } If you think the question cannot be properly answered, response with { answer': 'I cannot answer this question', 'context'; A short reason explaining why this question cannot be answered).
22


Reviews:
[{"review_id": "pn6PIqNbYI", "reviewer": "Reviewer_fXJo", "summary": "This paper proposes a new benchmark FCL for LLM’s function calling, which contains a ‘single-turn’ dataset, a ‘crowd-sourced’ dataset, a ‘multi-turn’ dataset, and an ‘agentic’ dataset.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time to review our paper and the comments. We emphasize the motivation, purpose, and contribution of our work. \n\n1. **Need more discussions on the comparisons and differences to existing benchmarks**\n\nA: Please see our rebuttal comment to reviewer krTJ. \n\n2. **Uneven score distribution across different dataset subsets. Near-perfect scores for single turn while extremely low score for agent memory category**\n\nA: Single-turn tasks remain critical for two reasons - first, they form the foundational building blocks for more complex, multi-turn, and agentic workflows. Second, they help weed out the “weaker” models. \nDespite rapid progress in the LLM field, our updated results show that smaller models (e.g., 27B, 7B, 3B, 1B) still struggle on single-turn entries, indicating the ongoing need for these categories to differentiate model capabilities and guide their development. Even “flagship” models occasionally fail on certain single-turn tasks (e.g., claude-3.7 can’t do any parallel tool calls, and  Llama-3.3-70B cannot refrain from tool call when the query is asking for weather information while the only function available is get_coordinate_by_city), demonstrating that near-perfect scores in one subset do not guarantee universal mastery.\n\nAs for the performance gap within the Multi Turn or Agentic category, these tasks are deliberately much more challenging, requiring context retention, dynamic planning, and advanced reasoning skills. Our findings reflect that while many models excel at basic single-turn interactions, they still struggle with multi-turn scenarios. The best model in multi_turn_base can only get 60% correct, and even lower on the more advanced miss_func or miss_param settings, and only 12% in the agentic memory subcategory. \n\nWe believe these results underscore the value of our benchmark: it captures the full spectrum from fundamental single-turn tasks to complex agentic use cases. To further illustrate this progression, we have updated [Table 1](https://anonymous.4open.science/r/FCL/Table_1.jpg) with more model sizes. And Figure 7 in the appendix shows how performance on single-turn tasks has steadily improved across multiple checkpoints for flagship models. Together, these additions demonstrate the ongoing relevance and evolving nature of our benchmark across different model scales and capabilities.\n\n3. **Insights derived seem obvious**\n\nA: The insight is not just the relative position of models (e.g., 4o better than 4o-mini), but rather to unearth interesting patterns, such as the fact that Claude-3.7 fails to handle parallel tool usage… These kinds of fine-grained observations have proven to be valuable to developers of large language models. Indeed, leading LLM providers have cited this work to identify and focus their post-training efforts based on the observations of this benchmark. For example, Google refined Gemini’s parallel tool call mechanism based on our findings, while Meta enhanced Llama’s handling of precise parameter value types by analyzing traces from our dataset. This evidence demonstrates that our work generates concrete, practical insights, rather than just confirming that larger models outperform smaller ones.", "claims_and_evidence": "The authors claim that \"Despite the importance of function calling, there isn’t a standard benchmark to evaluate function calling abilities in a wide range of real-world settings.\"\n\nI believe that the authors make an unfair comparison in Section 2, Related Work. For example, while Appworld is a simulated API, it provides high-quality human-annotated executable environments and test cases, ensuring fairness in execution and enabling reinforcement learning training. Additionally, ToolBench includes 126,486 instances and 16,464 real-world APIs that can be used to evaluate zero-shot tool calling abilities.", "methods_and_evaluation_criteria": "I believe that the evaluation methods proposed in this paper are generally reasonable and can reflect the ability of LLMs to call tools.", "theoretical_claims": "None.", "experimental_designs_or_analyses": "From Table 1, it is evident that the score distribution between different dataset subsets is very uneven. Many models in the Single Turn, Crowd Sourced, and Hallucination Measure categories score close to 100, which raises doubts about whether these scores can effectively differentiate model capabilities (especially given the rapid progress in the LLM field). In contrast, the best model under the Agentic category for the memory column scores only 12 points, which calls into question the benchmark's validity.", "relation_to_broader_scientific_literature": "This paper introduces a new benchmark and a new evaluation metric using AST, which could be beneficial to the development of the community.", "essential_references_not_discussed": "ToolBench and StableToolBench are large-scale datasets for tool learning using real APIs. These should be fairly compared and discussed in relation to the contributions of this paper.", "other_strengths_and_weaknesses": "I believe this paper does not have significant advantages over existing benchmarks, and the insights derived from its evaluations appear to be obvious (e.g., GPT-4o is stronger than GPT-4o-mini).", "other_comments_or_suggestions": "This paper should include a conclusion section. The asterisks in lines 238-239 seem to be a markdown structure output from LLMs and should be removed.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HVlDDJwJVM", "reviewer": "Reviewer_VP8g", "summary": "This paper describes a benchmark for evaluating function calling capabilities of large language models. It includes different function calling settings such as single-turn, multi-turn, multiple functions in a single turn, parallel functions etc. The evaluation setup defines execution and non-execution based metrics and includes performance comparison of several open weight and closed weight models on the different function calling settings.\n\n## Update after rebuttal: \nThe FCL benchmark is valuable for the function calling research community. However, the information about so many tasks and their details is scattered in multiple git repos and blogs. This paper would benefit by being a single source of all the information that exists in v3+agentic, so while I appreciate the extra pointers in the rebuttal, I would encourage adding those to the paper. I will keep my score with the current form of writing.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer VP8g for the time to review our paper and the comments. We address your questions as follows.\n\n1. **The Proposed Settings are Too Similar to Those From BFCL**\n\nA: In the interest of double-blind reviewing, we refrain from commenting on this. \n\n2. **Need More Discussions on the Comparisons and Differences to Existing Benchmarks**\n\nA: Please see our rebuttal comment to reviewer krTJ. \n\n3. **Insufficient Dataset Statistics and Details**\n\nA:\nWe utilized GPT-4o-0806 throughout the data generation process and incorporated a hybrid validation strategy—combining manual, programmatic, and LLM-based methods—at various stages of data curation. For the FCL benchmark data curation across both single-turn and multi-turn categories, we provide a more detailed description of the methodology here https://anonymous.4open.science/r/FCL/DATA_CURATION_METHOD.md. Additionally, we present comprehensive multi-turn dataset statistics and visualizations https://anonymous.4open.science/r/FCL/DATASET_STATISTICS.md, including distributions of turn counts, input token usage, and other key characteristics across multi-turn categories.\n\n4. **Where to find the dataset**\n\nA: You can find all the datasets and the codebase here: https://anonymous.4open.science/r/FCL\n\n5. **Clarifying What Makes the Agentic Dataset Unique**\n\t\nA: \nApart from the statistics that Agent Dataset has a much higher number of turns needed and that more write (as opposed to read) operations are involved, the Agentic Dataset uniquely evaluates tools frequently integrated with LLMs by specifically focusing on core agentic capabilities, independent of their narrow functional uses. Unlike general-purpose function calling evaluations, which typically assess context-free functions (e.g., a 'pause_music' command relevant primarily to virtual assistants like Siri or Alexa), the Agentic Dataset targets fundamental, broadly applicable agentic skills such as web browsing, memory management, and database querying. Take memory management as an example, the fact that it spans across multiple sessions is something you would never see in a normal/standard multi-turn dataset, yet it is critical for true agentic behaviour.\n\nThese abilities constitute the backbone of effective LLM agents but have not yet been systematically examined at scale. While there are attempts to evaluate efficient incorporation of web search knowledge or text-to-SQL, our work is unique in systematically treating these functionalities as integral tool calls within the broader agentic paradigm rather than standalone tasks.\n\n6. **Defining “Memory” and Its Role in the Memory Dataset**\n\nA: Memory refers to how the model retains and utilizes information from previous interactions so that it can provide coherent, contextually relevant responses over multiple sessions. In practical deployments (e.g., chatbots), relevant conversation data will be stored in an external database. For example, a healthcare agent might store a patient’s reported allergies or lab results for reference in future conversations. This stored information is generally not visible to the user, but they will be given a “wipe memory” command.\n\nThe Memory Dataset in our work is designed to test the model’s ability to:\n- Identify which pieces of information from past interactions need to be stored.\n- How to find the right balance between multiple contexts when the memory database is limited in size or entry count.\n- Retrieve those details as necessary to respond to new queries in a later session.\n\n8. **Evaluating Read-Only Calls: Response-Based vs. State-Based Approaches**\n\nA: \nState-based evaluation is limited for read-only requests because these requests do not fundamentally alter the environment. For instance, if a user wants to list files in a directory, using commands like `cd` and `ls` repeatedly does not change the underlying filesystem state in a way that would distinguish one action from another, so a traditional state-based metric offers little insight.\n\nMoreover, incorporating outputs of read-only calls directly into the state essentially converts the evaluation into a response-based evaluation. \n\nWe want to measure whether the model invokes the right tools at the right times, instead of relying on outdated intrinsic knowledge. By focusing on response-based evaluation for these scenarios, we hope to more accurately gauge the model’s real-world tool usage and ensure it is actually calling necessary functions rather than guessing or leveraging intrinsic information.\n\n9. **How the Model Receives and Understands Environment State**\n\nA: The model won’t be directly informed of the entire state of the environment. Instead, we equip the model with functions that can query partial or entire states of the environment, such as ‘ls’ in a file system or ‘display_tire_pressure' in a vehicle control system. We want the model to not take the state information for granted, but instead actively explore the state and retrieve as needed.", "claims_and_evidence": "This is a benchmark paper, so most claims have been made based on the evaluation on the dataset.", "methods_and_evaluation_criteria": "The propose a set of evaluation metrics such as AST based accuracy, state-based comparison and they make sense.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experimental design for prompting the LLMs to generate function calls and evaluating those on different function call setups described is as expected for this task.", "relation_to_broader_scientific_literature": "Function calling capability of large language models is an important area of research. There are existing benchmark datasets for this task such as [ToolBench](https://paperswithcode.com/dataset/toolbench) and [BFCL](https://gorilla.cs.berkeley.edu/blogs/13_bfcl_v3_multi_turn.html). The proposed benchmark seems to include additional categories of function calling settings, but most of the proposed settings are similar to those from BFCL.", "essential_references_not_discussed": "A detailed comparison with existing benchmarks such as StableToolBench, ToolSandbox and many others covered [here](https://github.com/quchangle1/LLM-Tool-Survey) would be very helpful to understand the novelty of this benchmark. The introduction as well as related work section mentions some but is missing many. For example, StableToolBench does capture real-world use cases and has a good number of data samples with single as well as multiple API invocations.", "other_strengths_and_weaknesses": "1. While the benchmark is useful and the experiments section is comprehensive, it would be good to understand the detailed differences/similarities with existing benchmarks.\n2. Detailed statistics of the dataset are missing. For example, for the 200 multi-turn long context examples, what is the average number of turns, or what is the maximum context length?\n3. Information about the API domains is missing. For example, appendix E.3 says the multi-turn dataset \"spanned eight domains, such as Vehicle Control, Trading Bots, Travel Booking, and the File System.\" - what are the other 4?\n4. Details of data generation are missing. For example, appendix E.3 says \"Task Generation: We prompt the model\" - which model is this?\n5. Details of data validation are missing. For example, is the data validation for multi-turn dataset manual/programmatic/LLM-based?\n6. Data sources information is missing. For example, what is the source of the SQL dataset? There are many text2SQL datasets available, is it one of them? What is the source of the database?\n7. No information on where to find this dataset is mentioned.", "other_comments_or_suggestions": "1. Minor typo: Line 260 - For an explanation of the (FC) and (Prompt) annotations in the model names, please refer to Section 5.1. : This is in section 5.1.\n2. The paper ends abruptly.", "questions_for_authors": "1. What makes the Agentic dataset different from others? They are all function calls after all?\n2. Memory dataset: What is memory here? external database? So in an agentic setting, we expect the end user to be aware of the memory component and ask for using it for some utterances? Can you give some examples of this dataset?\n3. For Execution Response Matching, how do you take care of necessary imports etc.?\n4. One of the arguments for Response-Based Evaluation is that the state based evaluation does not work for read-only requests. Why? If the state for read-only calls can include the output of the call, state based evaluation can be applicable? \nDoes it matter that the model generated the result using intrinsic knowledge vs tool calls as long as it is correct?\n5. for the case \"Failed to Understand Environment State\" - how is the environment state provided to the model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fpcG2KKK6i", "reviewer": "Reviewer_dutu", "summary": "* The paper proposes a benchmark for the task of function calling (\"FC\") i.e., given a prompt and a set of available functions, to predict the correct sequence of function calls along with necessary parameters that accomplishes the prompt requirement.\n* The benchmark evaluates FC ability along various categories: single-turn vs. multi-turn, relevance, various programming languages (e.g., python, java, REST), etc.\n* An important aspect of the paper is the proposed evaluation methodology. Here, the paper introduce various hand-crafted methods to evaluate depending on the scenario (e.g., primarily AST for \"simple\" calls, state- and response-based evaluation for multi-turn calls)\n* Finally, the paper evaluates existing LLMs (both with/without native support for FC) and presents some interesting analysis e.g., performance across categories, breakdown of errors in various categories.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer dutu for the time to review our paper and the comments. We address your questions as follows.\n\n1. **Need More Discussions on the Comparisons and Differences to Existing Benchmarks**\n\nA: Please see our rebuttal comment to reviewer krTJ. \n\n2. **Concern Over Potential Train/Test Data Contamination**\n\nA: \n\nThe scores reported are all run by the authors. We make sure all the scores are reproducible, and publicly archive all the results & scores we obtained, which the community finds valuable.\n\nAs the reviewer noted, the Single Turn dataset has indeed been publicly available for some time, making it difficult to prevent developers from potentially training on these test sets. Nonetheless, our Crowd-Sourced dataset was designed not only to capture more authentic user interactions but also to detect potential data contamination. By updating the Crowd-Sourced dataset on a monthly or quarterly basis, we make it considerably harder for model providers to anticipate and train on the test set. In principle, if a model has not been trained on our public test sets, it should exhibit comparable relative performance across both the Single Turn dataset and successive checkpoints of our Crowd-Sourced dataset.\n\nLastly, in the case of open-source models, we can calculate perplexity to identify signs of data contamination (see Section 5.5 for more details).\n\n3. **Unclear AST Substring Matching Procedure for Predicted vs. Ground Truth Sequences**\n\nA: \n\nBy definition, parallel tool calls are considered to execute simultaneously and, therefore, the order of calls does not matter. If order does matter (e.g., strictly sequential calls), the model must emit one function call at a time and await completion before issuing the next. Consequently, for a predicted sequence A = [a_1, a_2, ..., a_m] and a ground-truth sequence B = [b_1, b_2, ..., b_n], we do not require strict positional alignment. Instead, any predicted call a_i can be matched with any ground-truth call b_j. Our evaluation then adopts an “all-or-nothing” criterion: if at least one ground-truth call is not matched by any predicted call, the entire evaluation is considered a failure. This ensures the model correctly identifies all necessary calls, regardless of order.\n\nOn the parameter values, \nFor integer, float:\n- For Python tests only, we allow the use of `int` values for Python parameters expecting `float` values to accommodate the Python auto-conversion feature from `int` to `float`.\n- For non-Python languages (Java and JavaScript), if the function documentation specifies `float` parameter, then it should be a `float` in the model output (such as 5.0); an `int` (such as 5) will not be correct.\n- Supplying `float` value for `int` parameter is not allowed in any language.\n\nFor List, Tuple:\n- Order matters and the elements must match exactly. For example, `[1,2,3]` is not equal to `[2,3,1]`. So for questions where the order in the list doesn't matter, permutations of the possible answer are used to accommodate this situation.\n- Note that the type match extends recursively for nested data structures (List or Tuple), where both the outer type and the inner types of elements must match the specified requirements.\n\nFor String:\n- The evaluation process is case-insensitive.\n- All strings will be standardized before checking. This applies to both the model output and the possible answers.\n  - All white space is removed.\n  - A subset of punctuations `,./-_*^` are removed to make the evaluation more robust and accurate.\n- For example, \n- Possible Date `[\"20th June\", \"2023-06-20\", \"06/20/2023\", \"Jun.20, 2023\"]`\n- Possible Location `[\"New York City\", \"NYC\"]`\n\nFor Dict:\n- The evaluation focuses on the key presence and the accuracy of associated values as per the possible answers.\n- Ordering within dictionaries is not considered due to they are inherently unordered.\n\nFor lists of dictionaries:\n- While the ordering of dictionaries is considered (since it's a List), the order of key-value pairs within each dictionary is not.\n\nThis method works for other languages as well. For Java and JavaScript, the evaluation procedure uses tree sitters to convert the string version of a Java/JS type into their corresponding Python format, since we perform checking with Python data types. For example, a `HashMap` in Java will be converted to a `dict` in Python. During this process, the converter will also perform the type checking for those parameters; e.g. if the parameter should be a `long` type in Java, the converter will check to make sure that the string input does have an `L` at the end (because otherwise, it wouldn't be a valid Java `long` type).", "claims_and_evidence": "The claims are generally quite clear. The only issue I have is multiple claims that downplay complexity of existing benchmarks. For instance :\n* \"other benchmarks fail to capture ... parallel invocations\" (L17): Many recent benchmarks appear to consider more complex invocations compared to the proposed leader e.g., ToolBenchv2, TinyAgent consider invocations as a directed graph\n* \"failure to evaluate abstention\" (L105): Once again, many benchmarks do consider this e.g., ToolBench has >16K APIs, majority of which are irrelevant", "methods_and_evaluation_criteria": "The paper does not propose a method per se. Rather, the emphasis of methodological contributions would be the hand-crafted rules to verify. One bit I found somewhat unclear:\n* AST substring matching (Sec. 4.1): Although this section appears clear for the case of Python calls in \"simple\" category, it's unclear beyond. Specifically, if we have predicted sequence $A = [a_1, a_2, ..., a_m]$ and GT sequence $B = [b_1, b_2, ..., b_n]$, it's unclear how the association is done.", "theoretical_claims": "No theoretical claims in the paper.", "experimental_designs_or_analyses": "This is generally clear and I appreciate the paper's breakdown/analysis of >10 models on the proposed leaderboard. I have a major concern here:\n* Did the authors run the evaluation of the models in Table 1, or was is run by the developers of the models? Because, the proposed FCL data has been publicly available for some time now (esp. simple, multiple, ...). I'm concerned of train/test contamination i.e. the evaluated models were trained on parts/entirety of the proposed data. To be honest, this is challenging to verify, since numerous models evaluated (esp. the top-performers) are proprietary and do not make available details on the training methodology. Nonetheless, if this is possible, I fear the analysis of the paper might lead to conclusions on overfit data.", "relation_to_broader_scientific_literature": "The main contribution of the paper is the diversity of FC scenarios considered. For instance, compared to some related benchmarks:\n* ToolBench: considers complex FC calls, but restricted to REST APIs (here, it considers python, java, REST)\n* TinyAgent: also considers complex FC calls, but restricted to single-turn pythonic calls\n* AppWorld: requires long and stateful multi-turn interactions, but it's within a sandbox environment", "essential_references_not_discussed": "Essential references are discussed. However, another major concern: a more elaborate comparison and discussion of the proposed leaderboard compared to dozens of other FC leaderboards. For instance, see Table 1 in \"Tool Learning with Large Language Models: A Survey\", Qu et al., 2024. I believe the authors in the related work discussion (Sec. 2), only present a small subset.", "other_strengths_and_weaknesses": "No other comments here.", "other_comments_or_suggestions": "Nitpicks on writing:\n* L261: mentions please refer to Section 5.1, within Section 5.1.\n* L252 \"from some other people's blog posts\": cite please", "questions_for_authors": "Important questions that influence my score (repeating from above):\n* Since the evaluated models (e.g., gpt-4o) are proprietary, and the dataset has been publicly available for some time now, how can we ascertain that there was no overlap between the models' training data and the proposed dataset?\n* Given that there are >10 FC benchmarks (see Table 1 in \"Tool Learning with Large Language Models: A Survey\", Qu et al., 2024), what is the novelty introduced by this benchmark? After all, there are already benchmarks that are multi-turn (e.g., AppWorld), require real-world APIs (e.g., ToolBench), require parallel invocations (e.g., TinyAgent).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4XRcIe03dD", "reviewer": "Reviewer_krTJ", "summary": "This work introduces function-calling benchmark, a wide-coverage benchmark that evaluates LMs' ability of invoking correct function calls. Multiple parts and types of function calling are included: single-turn, crowd-sourced, multi-turn and agentic tasks. Furthemore, an evaluation method based on Abstract Syntax Tree (AST) substring matching is introduced to provide an appropriate way to evaluate LMs' performance.\n\n\nUpdate after rebuttal:\nDetails as discussed in the authors' rebuttal should be added to later versions of this work. I will maintain my original score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer krTJ for the time to review our paper and the comments. We emphasize the motivation, purpose, and contribution of our work. We address your questions as follows.\n\n\n1. **Need more discussions on the comparisons and differences to existing benchmarks**\n\nA: Although in recent times quite a few benchmarks have been proposed to test LLM’s ability to perform function calling, in this work, we focus on those designed to evaluate a model’s native function-calling capabilities, rather than methods that rely on prompt-based or code-generation approaches (e.g., Nexus Raven uses a prompt-based protocol, and AppWorld emphasis code generation capability). Among benchmarks that do evaluate native function calling, many such as App Blend and API Bench focus solely on single-turn interactions. Although TinyAgent addresses nested function calls, it does so by using placeholder variables instead of letting the model see the actual execution output of earlier calls; in effect, it still operates under a single-turn framework.\n\nFor benchmarks that truly cover multi-turn interactions, most are constrained by a narrow domain scope or a limited set of functions. TauBench, for example, supports only 28 functions spanning two domains (airline and retail), and RestBench offers scenarios solely within the TMDB and Spotify domains. The narrow coverage (<150 entries) is more prone to overfitting and does not sufficiently reflect the breadth of real-world function-calling scenarios.\nLastly, benchmarks like ToolSandBox and TauBench rely on LLMs to simulate user queries. Despite careful attempts to control the user simulator’s behaviour, LLM-based users remain prone to hallucination and instruction-following errors, which can confound evaluation results.\n\nOther works, such as ToolBench, depend solely on Rapid APIs that are subject to high variance in performance, making reproducibility a challenge. While the subsequent StableToolBench version mitigates this by caching or simulating API responses, it continues to rely on LLM-based evaluators for determining response solvability, thus risking model-induced biases and undermining objectivity (e.g., GPT-family models tending to favour responses from their own model). Similar issues exist in T-Eval, which also depends on LLM-based evaluation.\n\nLastly, current benchmarks uniformly employ LLM-curated user queries, limiting their ability to accurately reflect genuine user interactions.\n\nOur benchmark directly addresses these shortcomings by incorporating deterministic evaluation metrics, an extensive and diverse set of robust multi-turn interactions, and a unique multilingual dataset derived from real-world, user-contributed queries that have more than 15 languages represented in user queries, including Chinese, French, Japanese, and Korean, etc. Additionally, on top of Python and REST (which are commonly covered in existing benchmarks), we also have entries in Java and JavaScript for more diverse programming languages. \n\nCollectively, these enhancements enable a more comprehensive, fair, and reproducible assessment of LLM function calling, setting our benchmark apart from existing efforts.\n\nYou can find all the datasets and the codebase to run them here: https://anonymous.4open.science/r/FCL. Our evaluation scripts support almost all popular models (100+ proprietary models, and 85+ open-source models). \n\n2. **Need more descriptions and examples of how different types of data instances look like, especially on multi_turn** \n\nA: A set of example test entries and corresponding ground truth outputs for each test category is available here: https://anonymous.4open.science/r/FCL/TEST_EXAMPLES.md", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "The benchmark and evaluation methods make sense.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "It provides a reasonable benchmark to evaluate LM's function calling abilities.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n- This work introduces a nice benchmark to evaluate LM's function-calling abilities.\n\nWeaknesses:\n- More discussions on the comparisons and differences to existing benchmarks (such as Gorilla) should be included.\n- There is a lack of clear descriptions of how different types of data instances look like. Figure 2 only seems to show some examples of single-turn ones, while it will be more informative to provide more examples and comparisons of different parts.\n- Some parts of the presentation of this work can be further improved, for examples, some tables (Table 1) and figures (Figure 1) are difficult to read.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper is about Berkeley Function Calling Leaderboard, a well known and popular benchmark for evaluating the function-calling capabilities of Large Language Models. Given this is a blind review, it was confusing to few reviewers on how this work compares to well known BFCL leaderboard. All other concerns/questions from reviewers are addressed to a good extent in the authors rebuttal. Therefore, I recommend accepting this work.
Decision: Accept (poster)