Towards Reliable Benchmarking: A Contamination Free, Controllable Evaluation Framework for Multi-step LLM Function Calling
Authors: Seiji Maekawa
Jackson Hassell
Pouya Pezeshkpour
Tom Mitchell
Estevam Hruschka
Venue: ICLR 2026 Poster (2026)
Keywords: Function calling
Tool-augmented LMs
Too-use
Primary area: generative models
TL;DR: FuncBenchGen is a contamination-free, controllable benchmark for multi-step function calling via hidden-DAG traversal. We systematically identify the factors that affect LLM function calling performance.

Abstract: As language models gain access to external tools through structured function calls, they become increasingly more capable of solving complex, multi-step tasks. However, existing benchmarks for tool-augmented language models (TaLMs) provide insufficient control over factors such as the number of functions accessible, task complexity, and input size, and remain vulnerable to data contamination.
We present FuncBenchGen, a unified, contamination-free framework that evaluates TaLMs by generating synthetic multi-step tool-use tasks to stress-test TaLMs.  
The key idea is to cast tool use as traversal over a hidden function-dependency DAG where nodes are function calls and an edge between nodes represents one function consuming the output of another.
Given a set of external function schemas, initial variable values, and a target variable, models must compose the correct call sequence to compute the target variable.  FuncBenchGen allows users to precisely control task difficulty (e.g., graph size, dependency depth, and distractor functions) while avoiding pretraining/test-time leakage.

We apply our FuncBenchGen framework to evaluate seven open and closed LLMs on tool use tasks of varying difficulty.  Reasoning-optimized models consistently outperform general-purpose models with GPT-5 significantly outperforming other available models. Performance declines sharply as dependency depth increases.  Furthermore, connected distractors---irrelevant functions sharing type-compatible variables with relevant functions---prove especially difficult to handle. 

We find that strong models often make syntactically valid function calls but propagate incorrect or stale argument values across steps, revealing brittle state tracking by LLMs in multi-turn tool use.
Motivated by this observation, we introduce a simple mitigation strategy that explicitly restates prior variable values to the agent at each step.  Surprisingly, this lightweight change yields substantial gains across models. e.g., yielding an improvement in success rate from 62.5\% to 81.3\% for GPT-5, without modifying the underlying architectures or training.

Published as a conference paper at ICLR 2026
TOWARDS RELIABLE BENCHMARKING: A CONTAMI- NATION FREE, CONTROLLABLE EVALUATION FRAME- WORK FOR MULTI-STEP LLM FUNCTION CALLING
Seiji Mackawa Jackson Hassell
Pouya Pezeshkpour
Tom Mitchell
Estevam Hruschka
Megagon Labs
jackson, pouya, tom, estewammeeeegago. ai
ABSTRACT
Existing benchmarks for aog-euugeetee language models (TaLMs) lack fine- grained control over task difficulty and remain vulnerable to data contamination. We present FuncBenchGen, a unified, contamination-free framework that evalu- ates TaL.Ss by generating synthetic multi-step tool-use tasks to stress-test .s.. The key idea is to cast tool use as traversal over a hidden function-dependenee DAG where models must infer the correct sequence of calls to compute a tar- get value. FuncBenchGen allows precise control over task difficulty (e.g.. graph size, dependency depth, and distractor functions) while avoiding pretraining/test time leakage. Our evaluation demonstrates tently outperform general-purpose models with GPT-5 significantly outperform- ing other available models. Performance declines sharply as dependency depth increases. Furthermore, connected iistractors-irrelle compatible variables with relevant funetions-provv especially difficult handle. Also, strong models often make syntactically valid function calls but propagate incorrect or stale argument values across steps, revealing brittle state tracking by LLMs in multi-turn tool use. Motivated by this observation, we introduce a simple mitigation strategy that explicitly restates prior variable values to the agent at each step. Surprisingly, this lightweight change yields substantial gains across models. e.g., yielding an improvement in success rate from 62.5% to 81.3% for GPT-5.
models consis-
functions sharing type-
Code
INTRODUCTION
Large Language Models (LLMs) equipped with external tools ("tool use") have become central to complex real-world applications, as they enable interaction with external environments and access up-to-date information (Luo et al. 2025). A growing body of work investigates different facets of this capability, including the ability to handle large collections of callable APIs (Kwak et al.) 2025) multi-step function calling (Zhong al. 2025 Song et al. (025). long-horizon tool use (Kate al.
2025). and broader evaluations of tool-use skills (Li et al. 2023; Patil et al. 2025 Sun et al. 2024)
Despite recent advances, progress in this field is impeded by two key issues. First, while prior stud- ies have curated realistic API sets and constructed tasks to assess function-calling abilities (Li et al., 2023 Qin et al.) 2024), these benchmarks often exhibit limited function-set diversity due high curation cost, and they are vulnerable to data contamination from pretraining overlap and test-time web search (Han et al. 2025), as benchmark question- question-answer answer pairs may be publicly accessible. Second, existing benchmarks provide little fine-grained control over task difficulty, e.g., the number of required functions, the function dependency depth, and the presence of irrelevant functions with variables that are type-compatible with required functions. While some benchmarks (Patil et al.) 2025 Qin et al. 2024 Kate et al. (025) allow limited control over certain aspects of task complex- ity, such the number of required functions, they lack comprehensive control across the function dependency depth and the
of irrelevant functions. These limitations reduce the generality of empirical findings and prevent from understanding which factors most

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of previously proposed function-calling testbeds with FuncBenchGen. Func.
indicates function.
Contamination- Free
Task Complexity (Controllabilit)
Required Func. Size Dependency Depth
Inrelevant Func. Type
BPl-BBnk et al. 2023
X
X
BFCLv4 (Pant
ToolBenchi (Qin et 2034
X
X
CumplexFuncBBcnee (Alo et
2025
X
X
LongFuncEval (Kate
X
X
FuncBenchCien (ours)
/
significantly impact model performance. We summarize the existing benchmarks and compare them with our proposed framework in Table
In this paper, we aim to isolate and analyze the core capabilities and failure types of tool-augmented LLMs (Tal .s)) in multi-step function calling scenarios. To this end, we propose FuncBenchGen, framework for automated generation of contamination-fie function calling tasks with controllable difficulty to enable systematic evaluation and analysis of TaLMs, as illustrated in Figure [] The key idea is to represent function dependencies as a directed acyclic graph (DAG) and frame multi-step function calling
graph traversal problem. Given set of input variables with known values, target variable, and a set of external function schemas, the task requires an agent to determine the value of the target variable by executing appropriate sequence of external function calls.
Our key contributions are as follows:
We introduce FuneBenchGen, novel evaluation framework for tool- augmented language models (TaLMs). The framework automatically generates contamination-ffes function- calling tasks with controllable difficulty, specified by parameters such as the number of required function calls, the number and types of input/output variables, and the number and connectivity of irrelevant functions.
Using FuncBenchGen, conduct extensive experiments with seven state-of-the-arr open and closed I.MMs. Our analysis yields several important findings: reasoning-optimize models consistently outperform general-purpose ones, yet even GPT-5 struggles with longer function call sequences. For example, it achieves only a 15% success rate when 20 function calls are required. We also observed that external functions that are irrelevant to solving the problem, but that are "connected" to the solution DAG (i.e., that share type- compatible variables with functions involved in the solution) severely degrade performance for all LLM models. Moreover, the task success is strongly influenced by graph structure, with shallower and more sequential dependency chains being easier to solve; and although most models invoke functions with correct syntax, they frequently fail to propagate argu- ment values across calls.
Finally, identifying key failure types in models function calling capability, we propose simple augmentation mitigation strategy that restates variable values from prior calls, which significantly improves success rates across models for example, yielding an improvement in success rate from 62.5% to 81.3% for GPT-5.
2
RELATED WORK
Multi-step Reasoning in LLMs Recent LLMs demonstrate impressive multi-step reasoning ca-
pabilities across various domains, including mathematical problem solving (Davoodi et al. 2025 multi-document reasoning (Maekawa al. 2025), and commonsense reasoning (Yu et al. 2025
Other work has focused on the nigor and logical validity of this reasoning. For example, FOLIO
(Han et al.) 2024) and Lugti-LogiEval (Patel et al., (024) evaluate complex multi step logical rea- soning using first-order logic. Despite these advances, the robustness of their multi-step reasoning capabilities remains unclear when models are required interact with external tools.
Code Generation and Reasoning The ability to use tools is fundamentally linked to a model's ability to reason about code execution. This foundational capability has been evaluated by recent
2

---

Published as a conference paper at ICLR 2026
Graph
Creation
Execution
Function Depaedeency DMG
Shufted Function Scherias
Iopar Nokes
Target tar
Farettion
Input Parameters
Pumsaiion
LLM
Agent
7
haee
Grsmantutoos Mettem
Figure 1: Overview of FuncBenchGen. d, nome, no and rilis indicate the dependency depth and the numbers of core nodes, connected irrelevant nodes (CINs), and disconnected irrelevant nodes (DINs), respectively.
benchmarks (Gu et al. 2024, Chen et al. 2025). While these benchmarks focus on reasoning about code, our work evaluates the applied task of using code black-box external tools.
Models such as TooILLM (Qin et al. 2024). Gorilla PPttl et al. 2024), and ToolACE (Liu al. 2025a) highlight the importance of equipping LLMs with access to vast API collections to tackle real real-world use cases. Tool-Planner (Liu et al. 2025b), ToolDial (Shim al. 2025). HammerBench (Wang et al.) 2025) further extend these capabilities by incorporating planning and multi-turn dialogue for more sophisticated tool-use by using LLMs' multi-step rea- soning capabilities. Meanwhile, LongEuncEvval (Kate et al., 2025) and ComplexFuncBench (Zhong at. 2025) focus on evaluating LLMs' ability to handle long-context and multi-step function calling scenarios. Despite these advances, the evaluation datasets used often lack generality and controlla- bility, limiting their utility for systematic analysis.
Function Calling in LLMs
Contamination and Robustness in Evaluation Robustness and contamination issues in bench- mark construction pose significant challenges. Datasets such as (Mirzadeh al. 2025 Shojaee iet al. 2025) have addressed the limitations of LLM reasoning evaluation due to dataset contami- nation or task leakage, e.g.. when the LLM training set includes specific tool or task included in the test set. HHnn et al. (2025) have shown data contamination can happen during web search tool- cnntamination-llmieee benchmark that
use. White et al. (2025) have introduced LiveBench, refreshed regularly and spans multiple domains such as math, reasoning, and instruction following tasks. Though these studies emphasize the importance of creating bias-free and contamination- free tasks, they do not specifically address the challenges in multi-step function calling scenarios. In contrast, our work seeks to bridge this gap by offering a principled, automated framework for function set and task generation, supporting controlled stress-testing and robust analysis of TaLM capabilities.
3 THE EVALUATION FRAMEWORK FOR FUNCTION CALLING IN LLMS
We introduce FuncBenchGen, a benchmark generation framework designed to evaluate the multi- step function calling capabilities of LLMs. FuncBenchGen automatically creates synthetic, contamination-free function sets and tasks with controllable difficulty, enabling systematic analy- sis of the factors that affect model performance without the confounding influence of data bias or leakage. We formalize multi-step function calling as a graph traversal problem defined over a di- rected acyclic graph (DAG) of function dependencies. Importantly, LLMs are given only the list of generated functions and input variables, rather than the dependency graph itself, requiring them to infer the correct call sequence. The framework overview is illustrated in Figure]
3.1
TASK DEFINITION: MULTI-STEP FUNCTION CALLING
Given of functions F {f1, ......... fn). where each function fa takes set of input variables Vin and produces a single output variable yout along with a set of input variables Vinput (11, [ with known values and target variable vr, the LLM agent is tasked
3

---

Published as a conference paper at ICLR 2026
Dependecy Depth(d) = the aen
Dependocy Depth(d) m are mee n
Dependecy Depih(d) =
2,
n =S, 0, =
Legand
DN
Target
Examples of the different kinds of graphs that can be generated with 5 core nodes. CIN indicates connected irrelevant nodes and DIN indicates disconnected irrelevant nodes. noore is the number of core nodes, n nn is the number of CINs, and n is the number of DINs.
Figure 2:
to determine the value of vy by iteratively executing a sequence of function calls. For each exe- cution step, the LLM agent call any number of functions from F and then the system returns function outputs based on the input values specified by the L.LM agent. The process ends when no further calls made, at which point the agent's final output is parsed to obtain the answer.
3.2 BENCHMARK GENERATION PROCESS
FuncBenchGen is designed based on two core principles to address limitations of existing bench- marks, as discussed in Section [ 1) Contamination--Fees The framework generates synthetic function sets and tasks at evaluation time, ensuring that pretraining test-time leakage cur. 2) Controllable Task Complexity: The framework allows us to precisely control multiple dimensions of task complexity by taking inputs the number of functions, the function dependency depth, and the amounts and type of irrelevant functions. This enables systematic analysis of how different complexity factors affect model performance by isolating each factor.
To ensure contamination-ffet evaluation, FuneBenchGen generates synthetic function sets and tasks evaluation time. Also, control task complexity. we represent function dependencies as a di- rected acyclic graph (DAG), where nodes represent functions and edges represent the dependencies between functions, and formulate multi-step function calling as a graph traversal problem. This idea allows us precisely control multiple dimensions of task complexity by manipulating the underly- ing graph structure. Formally, function dependency graph G (F,E) is DAG where each node fi FF represents function with input variables V and output variable wiui and each directed edge (G) E & indicates that function f, can consume the output from function fi. To generate the function dependency graph G, the framework takes as inputs the number n nore of functions that will be required to solve the task, the dependency depth d, and the number of irrelevant distractor functions that connected the solution DAG (oon and the number of irrelevant distractor functions that are disconnected from the DAG dnsd)). Recall that we define function f to be con- nected to the solution DAG and only if there exists a dependency edge between, f and at least one function in the solution DAG. We call the connected irrelevant modes/functions CIN nodes and the disconnected irrelevant nodes/functions DIN nodes for brevity.
The benchmark generation process consists of two main steps: 1) Graph Structure Generation: Given parameters specifying the desired graph characteristics that meet the target task difficulty, the framework constructs a DAG that satisfies these constraints including the number of required func- tions, the dependency depth, and the presence and type of irrelevant external functions. 2) Function Schema Creation: Each node in the dependency graph is converted into function schema with randomly generated name. This on-the-fly fll generation ensures contamination-fres evaluation.
Graph Structure Generation The framework takes two-stage approach: 1) core node creation and 2) irrelevant node addition. First, we create node sequence length to ensure that the gener- ated graph contains a valid path from input variables to the target variable with the required number of function calls. Then, we iteratively add the remaining = - (d + 1) core nodes by randomly adding parent nodes to the existing core nodes, while ensuring acyclicity and dependency depth. Second, we add irrelevant nodes according to the specified connection type. For connected irrelevant nodes (CINs), randomly select nodes from the existing core nodes and add the irrelevant nodes
4

---

Published as a conference paper at ICLR 2026
as their children. For disconnected irrelevant nodes (DINs), we create isolated nodes and then add at most [n /2] edges between them while ensuring acyclicity, to simulate the possible function dependeney between irrelevant nodes. We provide illustrative examples of the generated graphs and more graph generation details in Appendix A.I
Function Schema Creation For the model input, each node in the dependency graph is converted into a function schema with: (1) Function name: A randomly generated identifier (e.g., func_ seec)__p).. (2) Input parameters: Variables with randomly assigned type and subtype annotations. (3) Output variable: A single variable that can serve as imput to dependent functions. And, (4) Description: Natural language explanation of the function's purpose.
Functions are linked through semantic type and sub-type matching rather than variable names, serv- ing as a lightweight proxy for the semantic reasoning that connects functions in real-world scenarios. In practice, if function's 's input type and sub-type matches another's output type and sub-type, the two are connected. See B.2 for details.
3.3 REAL-WORLD APPLICABILITY
Many real-world function calling scenarios can be mapped directly to DAG structures. For ex- ample, user may ask LLM agent to find a sightseeing tour near their hotel tomorrow and reserve taxi for them an hour after the tour ends (example from ComplexFune (Zhong al. 2025)). Here, an agent must: 1) find the hotel's location, 2) use the location to find nearby tours, 3) filter nearby tours by date and availability, and 4) reserve a taxi at the correct time and place. Figure 3 illustrates the function dependency graph for this exam- ple. This workflow naturally forms a depen- dency graph where each function depends specific outputs from previous calls can- not be searched for without the hotel's location, results, etc. Our framework abstracts this complexity into a controllable evaluation setting while preserving the essential challenge: models must infer these dependencies from function signatures alone, just as they would when interacting with unfamiliar APIs in deployment.
Taxi Reservation Tool Structure
DIT Joonet
TESOYV
Find
Shar
Figure 3: Graphical representation of a real tool- calling example sequence. Many tool-calling ap- plications can be represented as DAGs in this way.
cannot be selected without the tour search
3.4 IMPLEMENTATION DETAILS
Each variable in the system is assigned a three digit random integer value. Functions implement deterministic logic: they return the correct output value only when provided with the exact expected input values, which the agent must discover by calling the parent functions of that node. They return random incorrect outputs when given invalid inputs, simulating realistic API behavior where invalid parameters lead to silent failures or unhelpful responses. For the sake of simplicity, we set the number of output variables for each function to one.
We implement an interaction protocol between the LLM agent and the function execution environ- ment. At each step, the model can make multiple function calls, then receive their outputs, and continue until it makes no further calls. An example of the model input is shown in Appendix B.3
4 EXPERIMENTS
We apply our FuncBenchGen framework to conduct extensive experiments to explore the following research questions: (RQ1) How do LLMs perform in function calling tasks as the size of the core function set varies? (RQ2) How do irrelevant functions affect model performance? (RQ3) How does the function dependency depth impact performance? (RQ4) How do larger function sets and thinking budgets affect performance of the best performing models? (RQ5) What are common failure types in function calling tasks and how can they be mitigated?
5

---

Published as a conference paper at ICLR 2026
Table 2: Success rates and average number of function calls (ACs) made by models. Results are aggregated across all test configurations: number of irrelevant functions {0. 5, 10, 20), irrelevant node connectivity type [Connected, Disconnected, Half-and-Halhe {I, iI - 1}, with 5 random trials per configuration. ACs (Succ.) and (Fail.) denote the ACs for successful and failed trials, respectively. "_" indicates that there are no successful trials.
and graph dependency depth
# core nodes
5
10
20
Success Rate
ACs
ACs (Fail.)
Success Rate
ACs
ACs (Fail.)
Success Rate
ACs (Succ.)
ACs
(Succ.)
(Succ.)
(Fail.)
GPT-5
72.5%
5,4
7.6
38.2%
11.5
13.0
15.0%
22.0
23.7
GPT-5-mini
16.0%
5.0
7.9
7.6%
10.0
14.2
4.2%
20.0
23.4
Gemini-2.5-Prn
46.5%
5.5
6.3
14.4%
10.6
13.2
6.0%
22.6
24.4
Bermini-2.5-Flassh
31.5%
13.8%
10.3
7.2%
1.3
Qwen3
11.0%
5.4
8.2%
11.2
11.1
3.8%
24.1
19.3
GPT-4.1
12.0%
5.2
4.1
2.2%
10.3
7.1
0.2%
21.0
12.0
GPT-4.1-ming
11.0%
5.2
5.0
0.0%
9.0
0.0%
18.9
4.1 SETUP
To investigate our research questions, we test LLMs under various controlled conditions varying graph size, number of irrelevant functions, and the depth of required function call sequences. For graph size, we vary the number of core nodes in {5, 10, 20). We set the number of added irrelevant nodes to {0, 10, 20, 40). For each irrelevant node setting, we test three different connection types: 1) Connected: all irrelevant nodes are connected to the core nodes in the dependency DAG, 2) Disconnected: all irrelevant nodes are disconnected from the core nodes, and 3) Half & half: half of the irrelevant nodes are connected to the core nodes and the other half are disconnected. We also vary the dependency depth between and nooe for graphs with 5 and 10 core nodes. For graphs with 20 core nodes more, set the dependency depth for every 10% increment starting from 1, i.e. 3, J.., 7, 19 for 20 core nodes. To obtain reliable results, we generate 5 different graphs for each setting of core nodes, irrelevant nodes, and dependency depth and report the average results.
Models We evaluate various LLMs with reasoning-optimized and ceneral-purpose capabilities. Reasoning models include both closed and open models: GPT-5, GPT-5-mini, Gemini-2.5-PH Gemini-2.5-Flash. Qwen3-235B22A (Qwen3 for short). General models include GPT-4.1 and GPT- 4.1-mini. The model details are summarized in Appendix [.I We set the thinking budgets of all the reasoning models their default, e.g., medium for GPT-5, and set the temperature and top-p parameters to 0.0 and 1.0, respectively, for all our experiments
Evaluation Metrics We evaluate the models based on both success rate and efficiency function calling. A function call sequence is considered correct if it produces the expected output for the task. To measure efficiency, we record the number of function calls generated by each model. Given budget constraints, cap the maximum number of calls twice the minimum required number. A more detailed analysis function-calling sequences is provided in Section 4.5
4.2 MAIN RESULTS (RQ1)
Table[[]soove the overall results of all models across different numbers of core nodes. The results averaged over extra and 10, 20, 40 irrelevant nodes (functions) of three connectivity types with five generated graphs for each setting.
GPT-5 outperforms all other models by a significant margin, but it reaches only 15% tended function-call sequences, while general-purpose models fail to reliably solve even sim- models, e.g., GPT-5 and Gemini-2.5-Pie
pler tasks. Reasoning-optimmize general-purpose models (GPT-4.1, GPT-4.1-mini at every core size. With 5 core nodes, GPT-5 attains 72.5% success rate versus 12.0% for GPT-4.1. However, success rates fall sharply as sequence length grows: GPT-5 drops from 72.5% (5 core nodes) to 15.0% (20 core nodes), and Gemini-2.5-Pr from 46.5% to 6.0%. These trends suggest limited effective planning depth and only modest self-correction via reflection as tasks require longer function-call sequences.
consistently surpass
Since GPT-5 and GPT-5-mini do support the temperature parameter, use their default settings.

---

Published as a conference paper at ICLR 2026
Connected
Half
Disconnected
Success Rate
GPT-5
GPT-S-mini
Gemini-2.5-PP P-5-P0
Gemini-22 2.5-flash
Qwen3
GPT-4.1
GPT-4. GPT4411mini 4.1-mini
Figure 4: Success rate under different irrelevant node connection types. The results are averaged over (5, 10, 20) numbers of nodes and {0, 10, 20, 40} irrelevant nodes. Connected, Half & Half, and Disconnected indicate that all, half, and none of the irrelevant nodes are connected to the core nodes, respectively.
GPT-5 invokes 10% more calls than necessary even when successful. While GPT-5 demon- strates strong performance, it still exhibits inefficiencies in function calling. For example, with 10 and 20 core nodes (i.e., required function calls), GPT-5 averages 1..5 and 22.0 function calls respec- tively for successful cases. This indicates that even the best- best-performing models struggle to optimize their function call sequences, often making unnecessary calls.
The trends of average calls failed cases depend on the model. We observe distinct patterns in the average number of function calls (ACs) for failed cases across different models. For example, GPT-5, GPT-5-mini, Gemini-2..5 2.5-i tend to make more function calls in failed cases than successful ones, indicating that they struggle to find the correct path once they deviate. In contrast, Gemini- 2.5-Flash invokes significantly fewer function calls in failed cases compared successful ones, suggesting that it gave up when it could not find the correct path.
4.3 EFFECT OF IRRELEVANT FUNCTIONS (RQ2)
Connected irrelevant nodes (CINs) severely degrade performance for all models. Figure shows that CINs have the most negative impact model performance compared other types across most tested LLMs. We speculate that this is because CINs share variables with core nodes, making it challenging for models to distinguish relevant function paths from irrelevant ones. The
similar trend of the performance drop is observed in other reasoning tasks (Shi et al.) 2023 when distracting information is present.
Interestingly, the effect of disconnected irrelevant nodes (DINs) varies across models. For instance, GPT-5 achieves better performance in the "Disconnected" setting than in the "No Extra" setting. suggesting that the existence of DINs may prompt GPT-5 to carefully consider its function calls, potentially leading better performance the presence of irrelevant functions. In contrast, Gemini- 2.5-Pro performs worse in the "Disconnected" setting than the "No Extra" setting, suggesting that it may struggle to ignore irrelevant functions even when they are disconnected from the core nodes.
We also observe that "Half & Half" settings often yield intermediate performance between the "No Extra" and "Connected" settings, indicating that the presence of some CINs is sufficient to significantly confuse the mod- els.
GPT-S
Gemin F-iish Owenll
GP74
Gam---mmin
1-min
Genirii 2.5 Pro
0.0
Success
4.4 EFFECT
0.4
OF FUNCTION DEPENDENCY DEPTH (RQ3)
0.2
We break down the results by the dependency depth in We also show the results with 95% confidence intervals and more numbers of core nodes in Appendix
Dependency Depth
Figure 5
Figure 5; Success rates by the depen- dency depth. The number core nodes is set to 10. The results averaged over all irrelevant node settings.
Lower dependency depth is more manageable. Lower dependency depth leads to higher success rates across all models. For instance, with 10 core nodes, GPT.

---

Published as a conference paper at ICLR 2026
5 achieves near 90% success rate when the dependency depth is set I, where graph has star structure (see Figure 2) but this rate drops sharply to less than 30% when the dependency depth increases, i.c., between 4 and 8. This pattern is consistent across other reasoning -optiiing-- models like Gemini-2.5-PM and GPT-5-mini, suggesting higher dependency depth poses significant challenges for L.MSs in function calling tasks.
Fewer branches in the function call sequence are less harmful. We observe that GPT-5, Gemini-2.5-Pro, and Qwen3 show a slight performance improvement when the dependency depth is set to 8 and 9, where graph has path structure, compared to dependency depth between 5 and 7. This suggests that having fewer branches in the function call sequence may be less harmful, as it reduces the complexity of decision-making in smaller models like GPT--5-mini and 1omini-2.5--lashe indicating that this benefit may be more pronounced in larger, more capable models.
each step. However, this trend is not observed
4.5 DISCUSSION (RQ4)
To better understand the behaviors of the best performing model, we conduct experiments using GPT-5 with larger function sets and different thinking budgets.
Connected
Half & Half
Discornected
1.0
0.0
Success
Larger sets of required nodes are generally more chal- leging even for the disconnected setting. Figure 6 shows results for each number of core nodes, where re- sults are averaged across the numbers of irrelevant nodes.
0.4
0.2
0.0
5
10
20
40
Nodes
Surprisingly, even in the disconnected setting where irrel- evant functions are not type-compatible with core func- tions, GPT-5's performance degrades significantly (lower than 10% for 40 core nodes) as the number of core nodes increases. This poor performance of GPT-5 indicates that even the best models are not ready to be used over a large function sets which is happening with existing MCP servers
Figure 6: Success rate of GPT-5 with various numbers of core nodes. The re- sults are averaged across {0, 10, 20, 40} irrelevant nodes.
Anthropic 2025
Sufficient thinking budget is necessary in complex function calling tasks. To investigate in detail how
3.0
Medium Thinking
Success Rate
0.8
Minimal Thinking
the thinking budget affects the performance of GPT-5,
3.6
we compare the medium thinking budget i.e. our de- fault setting in other experiments, with the minimal think- ing budget in Figure 7 We observe a significant drop from the medium thinking budget setting, indicating that the minimal budget severely limits the model's ability to reason through multi-step function calling tasks. Only for the no extra irrelevant nodes setting, GPT-5 with the minimal thinking budget obtains higher than 50% suc- cess rate. In other settings, the success rates are less than 20%. This result highlights the importance of providing sufficient reasoning capacity for LLMs to effectively navigate complex function calling scenarios.
0.4
0.2
0.0
20 Connected Irrelevant Nodes
10
40
Figure 7: Thinking budget comparison for GPT-5 with 5 core nodes, averaged over all irrelevant node settings.
4.6 FAILURE ANALYSIS AND MITIGATION STRATEGY (RQ5)
In analyzing model failures in function calling, we use complete execution traces from our in-house executor, which logs all calls, arguments, returns, and state updates, enabling deter- ministic, annotation-free attribution. The executor enforces four sequential, machine-checked predicates-nant name resolution, schema conformance, dataflow availability, and value consis- designe to be minimal and collectively exhaustive. We categorize the failure cases into four types: 1) Function Not Found: The model attempts to call a function that does not exist in
While GPT-5 has four options for its thinking budgets, high, medium (default), low, and minimal, we fucus on the medium and minimal thinking budgets in this paper due to the budget constraint.
8

---

Published as a conference paper at ICLR 2026
Table 3: Failure types. The results are averaged (5, 10, 203 core nodes and all irrelevant node settings, with 5 trials each.
Failure type
GPT-5
GPT-5-mini Gemini-2.5-Pro Gemini-2.5-Flash
Qwen3
GPT-4.1
GPT-4.1-mini
Function Not Found
0.0%
0.0%
2.4%
0.0%
0.0%
0.0%
0.0%i
Wrong Number of Inputs
0.0%
0.0%
0.2%
0.9%
0.1%
0.0%
0.0%
Value Not Yet Known
79.6%
66.8%
69.1%
81.3%
74.0%
73.2%
66.8%
Incorrect Value
20.4%
33.2%
28.3%
17.9%
25.8%
26.8%
33.2%
Total errors
6,054
13,
5,756
235
9,472
3,597
8,560
The model provides more or fewer in- put arguments than the function schema allows. 3) Value Not Yet Known: The model tries to use a variable whose value has not been established through prior function calls or initial inputs. 4) Incorrect Value: The model uses variable with incorrect value, which has been established through prior function calls or initial inputs. These failure types cover all systematically detectable errors, as they correspond the hierarchical prerequisites required for any valid function execution: 1) function existence errors, 2) argument mismatch errors, 3) incorrect call sequence errors, and 4) state- trate-trackig errors. These failure types can be considered in sequence (1-4) using if- (f-bracches, and any error arising from function call will fall into one of these categories.
the provided function list 2) Wrong Number of Inputs:
Models attempt to use unknown variable values most frequently. Table summarizes the fail- ure types of models, where the results are aggregated over function calls made by each model. We observe that the most common failure mode across all models is attempting use variable values that are not yet known, accounting for over 66% of failures in every model. This indicates that mod- els often struggle to accurately track which variables have been established through function calls. While recent models have shown strong value retrieval capabilities (Hsieh et al., (024)) from given long documents, they still face challenges in maintaining context over multiple steps in function calling scenarios.
Interestingly, most models rarely attempt call non-existent functions or provide too many inputs, suggesting that they generally understand the function schemas provided.
4.6.1 MITIGATION STRATEGY
To address the most common failure types of using unknown variable values and in- correct values, we propose a simple mit- igation strategy. Instead of each function just returning the value of its output vari-
success rate
1.0
Original w/ Mitigation
able, it also returns the list of all known variable values as well.
GPT-5
GPT-5-mini
Gemini-2.5-PP 5-Pro
Gemini 2.5-il225--Faah F-ass
Owen3
GPT-4.1
GPT4 1-mini
This provides no extra information to the model - it simply restates the values of all the variables the model has already discov- ered, including variables with wrong val- ues that have been discovered through in- correct function calls. This provides the model with explicit context about which variables are available for use in function calls. This lightweight approach does not rely on the FuncBenchGen framework, and can easily be implemented in real-world scenarios with simple wrapper around the provided functions.
Figure 8: Comparison between the baseline and the proposed mitigation strategy. The number of core nodes is set to 5. The results are averaged across {0, 10, 20, 40) CINs, with trials each.
Simple variable reminders dramatically improve performance. Figure 8 compares the suc- cess rates of all models with and without the above mitigation strategy. observe that the miti- gation strategy improves performance over both reasoning and general models. Gemini-2.5-Fiaa and Qwen3 have a small improvement or a slight performance drop, suggesting that these models may struggle to effectively utilize the additional context provided. Overall, this result indicates that even a simple reminder of known variable values can significantly enhance LLMs' function calling capabilities by reducing emrrr related to variable usage.

---

Published as a conference paper at ICLR 2026
S 10 n 2 0,seed
5, 10. fonn 10, 20,ndt 0,seed
Care
Core
(a) A failure example in which GPT-5 calls irrelevant functions and invoke core functions with wrong input function calls and attempt to call the target function values.
(b) A failure example in which GPT-5 misses required
with irrelevant values.
Figure 9: Analysis of GPT-5 function calling errors. The generated graphs feature depth of 5, comprising 10 core nodes, 10 CINs, and DINs across various random seeds. The number at the top-right of each node denotes the sequential order of function calls made by GPT-5, e.g., "2" indicates the second function called. The color of the number indicates the result: black for correct calls, green for irrelevant nodes (no error), and red for errors.
4.7 CASE STUDY OF FUNCTION CALLING FAILURE
To illustrate how LLMs fail function calling tasks, we present a case study of GPT-5's failure examples in Figure ? In Figure Pa GPT-5 incorrectly calls irrelevant functions such as func_pbb and Func_ped in the seventh and eighth calls, respectively. Then, we observe that GPT-5 invokes the core function func_nss with wrong input values in the ninth call, i.c., it uses correct variable values from func_puf and func_ení but also uses an incorrect value from func_pbb instead of func_hoj though it was successfully called in the fifth call. In Figure 9b GPT-5 misses required function calls such as func_rgy and func__sq (the blue nodes without a number at the top-right) throughout the entire function calling sequence. As result, GPT-5 attempts call the target function func_xrx with irrelevant values from func_qcu and func_otq in the fifth and ninth call. These examples highlight the challenges LLMs face in accurately navigating function call sequences, particularly in avoiding irrelevant functions and ensuring correct input values for core functions. We provide full function calling sequence of the example Appendix C.4
CONCLUSION
We present FuncBenchGen, a novel framework for generating multi-step function calling bench- marks that address key limitations in existing evaluations. By enabling controllable complexity and contamination free tasks, FuncBenchGen provides a robust platform for systematically assessing ILMSs function calling capabilities. Our extensive experiments reveal significant performance gaps between reasoning-optimmzze and general-purpose models. Results also highlight challenges posed by irrelevant functions and long call sequences even GPT-5 struggles with longer function call sequences. We identify common failure types, and based on this analysis propose a simple yet ef- fective mitigation strategy that significantly improves performance across various models. Overall, FuneBenchGen offers a valuable tool for advancing research into LLM function calling capabili- ties. Incorporating more complex control flows such as conditional logic and iteration would be our interesting future direction.
REPRODUCIBILITY STATEMENT
We have made several efforts ensure the reproducibility of our work. First, the design principles, methodology, and evaluation setup for FuncBenchGen are detailed in Section 3 All models used in
the experiments well-documented in Appendix B. with the prompt example in Appendix B.3
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
We made use of Al tools such ChatGPT and Copilot to support coding and refining this paper, but all content was carefully reviewed and edited by us to ensure it adheres to our standards and aligns with our research objectives.
REFERENCES
Anthropic. Introducing the model context protocol. Technical report, 2025. URL httpps//www. anthropic.
Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. Reasoning runtime behavior of a program with llm: How far are we? In 2025 IEEF/ACM 47th International Conference on Software Engineering (ICSE), pp. 1869-1881. IEEE, 2025.
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261 2025. URL httpps///axxi.cr wwtp:///.gggbb//////e//// 06261/
Arash Gholami Davoodi, Seyed Pouyan Mousavi Davoudi, and Pouya Pezeshkpour. LLMs are not intelligent thinkers: Introducing mathematical topic tree benchmark for comprehensive evaluation of LLMs. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume Long Pupers), pp. 3127-31 40, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 077799-889766 189-6. doi: 10.18653/ v1/2025.naae www..-alal...o.... URL https::///// htttps://aclanthologg/ org/ 2025. nwann-lalllaaaaaa 161/
Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. Cruxeval: benchmark for code reasoning, understanding and exccution. In Proceedings of the 41st International Conference Machine Learning, pp. 16568- 16621, 2024.
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng. Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle-Solano, Hannah Sz- abó, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander Fabbri, Wojeiech Maciej Kryscinski, Semih Yavuz, Ye Liu, Xi Victoria Lin, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Rex Ying, Arman Cohan, and Dragomir Radev. FOLIO: Natural language reasoning with first- order logic. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 22017-22031, Mi- ami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/ w1/2024.cmmlp-maie 1229. URL https: htttps::/aclanthhl..gg. org/
/ 2024 wttp//224..mmmmmmmmaaa......aaa. emnlp-main. 1229/
Ziwen Han, Meher Mankikar, Julian Michael, and Zifan Wang. Search-time data contamination. arXiv preprint 250i 13/80, 2025.
13180
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What's the real context size of your long-context language models? In First Conference on Language Modeling, 2024. URL http:/tss//oopeevviii..eeee wtwpreewwwwieeeeeoommrioiiie kloBbc76Sy
Kiran Kate, Tejaswini Pedapati, Kinjal Basu, Yara Rizk, Vijil Chenthamarakshan, Subhajit Chaud- hury, Mayank Agarwal, and Ibrahim Abdelaziz. Longfunceval: Measuring the effectiveness of long context models for function calling. arXiv preprint arXiv:2505. 10570, 2025. URL https://arxiv.. org/ 2tb.//555..... 10570
Beong-woo Kwak, Minju Kim, Dongha Lim, Hyungjoo Chae, Dongjin Kang, Sunghwan Kim, Dongil Yang, and Jinyoung Yeo. Toolhaystack: Stress-testing tool-augmented language mod- els in realistic long-term interactions. arXiv preprint arXiv:2505.23662, 2025. URL https: 7/arxiv. //arxiv.org//bs//550...
11

---

Published as a conference paper at ICLR 2026
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang. and Yongbin Li. APl-bank: A comprehensive benchmark for tool-augmented LLMs. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3102-31 Singapore, December 2023. Association for Computational Linguisties. doi: 10. 18653//12022..emmlpmm 187. URL attps://acllantholgg. wwwpnnhhoooo.oor///0/00 202. wwtgg//2....nmmmmm.a......... smnnp-main. 187/
Weiwen Liu, Xu Huang, Xingshan Zeng, xinlong hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, Zezhong WANG, Yuxian Wang, Wu Ning, Yutai Hou, Bin Wang, Chuhan Wu, Wang Xinzhi, Yong Liu, Yasheng Wang, Duyu Tang, Dandan Tu, Lifeng Shang, Xin Jiang. Ruiming Tang, Defu Lian, Qun Liu, and Enhong Chen. ToolACE: Winning the points of LLM function calling. In The Thirteenth International Conference on Learning Representations, 2025a. U htttss://ppereeieeeee wtt.://eoooooomm/---8B8aoo000
Yanming Liu, Xinyue Peng, Jianman Cao, Shi Bo, Yuwei Zhang, Xuhong Zhang. Sheng Cheng, Xun Wang, Jianwei Yin, and Tianyu Du. Tool-planner: Task planning with clusters across multiple tools. In The Thirteenth Interaational Conference on Learning Representations. 2025b. URL https: http://////eeeeeeeeeee///////nna www.ee/oooooooooooooioiiiio
Ziyang Luo, Zhiqi Shen, Wenzhuo Yang, Zirui Zhao, Prathyusha Jwalapuram, Amrita Saha, Doyen Sahoo, Silvio Savarese, Caiming Xiong, and Junnan Li. Mep-universe: Benchmarking large lan- guage models with real-world model context protocol servers. arXiv preprint arXiv:2508. I4704, 2025.
Seiji Maekawa, Hayate Iso, and Nikita Bhutani. Holistic reasoning with long-context LMs: A bench- mark for database operations on massive textual data. In The Thirteenth International Conference on Learning Representations, 2025. URL
Seyed Iman Mirzadeh, Keivan Alizadch, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. GSM-symbolic: Understanding the limitations of mathematical reasoning in large language models. The Thirteenth International Conference on Learning Representations, 2025. /wRK/tttps////peeeeeeiee..eeeeee http://wwwwwweeeeeeor/oo////deee jXKRZIvjB
OpenAl. Model release blog: Introducing gpt-4. I in the api. Technical report, 2025a. hhtths httpe/aai..oom//de//ggggttt
OpenAl. Model release blog: Introducing gpt-5. Technical report, 2025b. URL https:///ppenai. htw.inde//introducinnge.
Nisarg Patel, Mohith Kulkarni, Mihir Parmar, Aashna Budhiraja, Mutsumi Nakamura, Neeraj Varsh- ney, and Chitta Baral. Multi-LogiEval Towards evaluating multi-step logical reasoning abil- ity of large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 20856-20879, Miami, Florida, USA, November 2024. Association for Computational Lin- (11533/1/2024.mmmll
guistics. doi: Smnlp-maing 1160/
1160. URL https://accaantollogy hwwppttoooooooooo///0000 or2//024.
Shishir G Patil, Tianjun Zhang, Xin Wang. and Joseph Gonzalez. Gorilla: Large language model connected with massive APIs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL httpps://ppenreiiett http://wwwww..etooooo/ttttttttte
Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (BFCL): From tool use to agen- tic evaluation of large language models. In Forty-second International Conference Machine Learning, 2025. http:httt///ppeeeeeeee
nwwt/frumm wiwpummmidddiimmmmm 2GmDdhBBdd
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li, Zhiyuan Liu, and Maosong Sun. TooILLM: Facilitating large language models to master 16000+ real-world APIs. In The Tweifth International Conference on Learning Representations, 2024. | /tttpp:///ppereeeeeeeeee http://pwrreiiieeeeeefooooonaaaa wtwt:/oorumii-dddgg220000
12

---

Published as a conference paper at ICLR 2026
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pp. 31210-31222 31227. PMLR, 2023.
Jeonghoon Shim, Gyuhyeon Seo, Cheongsu Lim, and Yohan Jo. Tooldial: Multi-turn dialogue gen- eration method for tool-augmented language models. In The Thirteenth International Conference on Learning Representations, 2025. URL
fttt//formmmid_J]. J5eGJskZZ
Parshin Shojaee, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, and Mehrdad Farajtabar. The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity. arXiv preprint (ikiv:2506.0694411 2025. URLhttps. //arxiv.org/ante 2506. 06941
Yewei Song, Xunzhu Tang, Cedric Lothritz, Saad Ezzini, Jacques Klein, Tegawendé F Bissyandé, Andrey Boytsov, Ulrick Ble, and Anne Goujon. Callnavi. challenge and empirical study on Ilm function calling and routing. arXiv preprint arXiv:2501.05555 2025. URL httpt:
org/abs/2501. 05255
Jimin Sun, So Yeon Min, Yingshan Chang, and Yonatan Bisk. Tools fail: Detecting silent errors in faulty tools. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 14272-14289,
Miami, Florida, USA, November 2024, Association for Computational Linguistics. doi: 10. http:/2222..mmmmmaaaaaa
URL httpps://aclaattiooyyy.rr///////
main. 790/
Qwen Team. Qwen3 technical report, 2025. https:/ttts///
09388
Jun Wang, Jiamu Zhou, Xihuai Wang. Xiaoyun Mo, Haoyu Zhang, Qiqiang Lin, Jincheng Jincheng, Muning Wen, Weinan Zhang, Qiuying Peng, and Jun Wang. HammerBench: Fine-grained function-calling evaluation in real mobile assistant scenarios. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Com- putational Linguistics: ACL 2025, pp. 3350-3376, Vienna, Austria, July 2025. Association for doi: 10.18653/v1/2055. finni.iiins.... finninngs-acl.1755 175. URL httpps://aclanthhllgg... / 2025. (.1.:2....inndiio findings-ane acl. 175/
Computational Linguistics. ISBN 979-8-89 8-81176-25-5..
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha Venkat Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. Livebench: challenging, contamination-limiie LLM benchmark. In The Thirteenth International Conference on Learning Representations, 2025. URL https:://pennr
Jianxing Yu, Shiqi Wang, Hanjiang Lai, Wenqing Chen, Yanghui Rao, Qinliang Su, and Jian Yin. Generating commonsense reasoning questions with controllable complexity through multi-step structural composition. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.), Proceedings of the 31st International Confer- ence on Computational Linguistics, pp. 2221-2222222 Abu Dhabi, UAE, January 2025. Association for Computational Linguistics. URL httttp::///allanhooloo.. org/ 2025. coling main. 155/)
Lucen Zhong, Zhengxiao Du, Xiaohan Zhang, Haiyi Hu, and Jie Tang. Complexfunchench: ex- ploring multi-step and constrained function calling under long-context scenario. arXiv preprint 1rX1v32501..0132.
httttp:://aaxxi.oo///a///////// 10132
A ADDITIONAL DETAILS OF GRAPH GENERATION
A.1 GRAPH GENERATION PROCESS DETAIL
As described in Section the graph gener- ation consists of two main steps: core node creation and irrelevant node addition. In the node creation step, we first generate a initial graph that is a sequence of nodes based on the specified number of dependency depth shown in the left in the figure. Then, iteratively add remaining
We illustrate the graph generation process in Figure 10
13

---

Published as a conference paper at ICLR 2026
Core Node Creation
Irrelevant Node Addition
Longest Chain Core Nodes
Remaining Core Nodes
Irrelevant Nodes
Legend
CN
DIN
Target
Dependecy Depth(d) = n core
5, n3
comn
2, 72 dis
Figure 10: Graph generation process.
nodes to the graph until reaching the specified number of core nodes while ensuring that the max- imum dependency depth not exceeded (see the middle in the figure). To increase the diversity, we randomly add up to 2nr edges between core nodes while maintaining the acyclic property of the graph. If adding the new node would exceed the maximum dependency depth, we discard that choice and try again. In the irrelevant node addition step, we add irrelevant nodes based on the speci- fied connection type (see the right in the figure). For connected irrelevant nodes, we randomly select parent nodes from the existing core nodes and link the new irrelevant nodes to it as their children. For disconnected irrelevant nodes, we simply add the new irrelevant node without linking to any core nodes.
A.2 EXAMPLE GENERATED GRAPHS
d= s m 10 ds 0,seed
d= tore fon = n0 ni seed 100
(a) Seed=0.
(b) Seed=100.
Figure II: Example DAGS with depth 5, 10 core nodes, 10 CINs, DINs, and different random seeds.
To provide concrete examples of our generated graphs, we present figures illustrating DAGs with varying depths and numbers of core and irrelevant nodes used in our experiments. Specifically, Figure displays generated DAGs with depth 5f 10 core nodes, 10 CINs, and 0 DINs, generated using different random seeds (0 and 100 in Figures Tla and [Tb respectively). These examples demonstrate how the overall structures the DAGs can vary significantly-for instance, one graph features two first-layer core nodes while the other has four-while maintaining the same depth and total number nodes. In our experiments, we use five different random seeds and aggregate the results to ensure robustness against these structural variations.
Next, we provide generated DAGs with depth and 11, 20 core nodes, 5 CINs, and DINs in Figure [2] to demonstrate how the structure changes with varying depths while keeping the number of core and irrelevant nodes constant. Figures [2a and [26] illustrate these graphs, respectively. The depth 3 graph exhibits wider structure, while the depth 11 graph shows longer chain of
14

---

Published as a conference paper at ICLR 2026
d 3, offe 20, o 5.
seed
dm a piten n. sesed m
(a) Depth=3.
(b) Depth=11.
Figure 12: Example DAGs with different depths, 20 core nodes, CINs, and 5 DINs.
These examples highlight the flexibility of our graph generation process in creating diverse structures that can effectively test the function calling capabilities of LLMs under different complexity levels.
dependencies.
B ADDITIONAL EXPERIMENT DETAILS
B. MODEL DETAILS
Model
Size
Context
HoggingFace API
Liceuse
GPT-5 OpenAA 20256
400k
upt-5 2925-88-07
OpenAl Service Terms!
GPT-5-mmr (ppenrr
M255
4000
1opt---aini-2275-666
OpenAl Service Terms
Gemini-2 5.P
1M
gemini-2. S-pro
Gemini Additional Terms Service'
Gemini-2 Flaah Comanici 6t al.
2025
1M
genini -2. 5-flahh
Gemini Additional Terms Service"
Qwen-3 Tean 2023
235B
128k
Qwen/Qwen3 21---2350--2
Apache license 2.0
GPT-4.1
2025g
1M
upt 1- 11-2825-44-11 04-14
OpenAl Service Terms
OpIAA
1Bt-4.11mini-222--66n
OpenAl
Table 4: Models used in experiments. Model sizes are not publicly disclosed (-).
We summarize the details of the models used in our experiments in Table 4 All models are used via their respective APIs except for Qwen3, which is accessed through HuggingFace. We set the temperature and top-p parameters that do not support the temperature parameter, use their default settings.
and 1.0, respectively, for all our experiments. For models
B.2 TYPES AND SUBTYPES
We observed that if functions were linked based on exact variable names, the problem became too easy. In real-world scenarios, even if a function consumes the output of another function, the names of those variables are rarely the same. Instead these relationships are captured through semantic meaning and data types. For example, if function A outputs variable of type SQLQuery and function B has an input of type SQLQuery, that often means that function B is designed to consume the output of function A. To reflect this, FuncGenBench links functions based on types, rather than direct variable names. The only exception to this are the input variables listed in the model prompt. Those are linked to function schemas based on direct variable name matching instead.
Additionally, each variable is given both type and subtype. A subtype is unique to that variable to allow for unambiguous linking between two functions, but types may be shared across many dif- ferent variables of completely separate functions. We include both type and subtype to better reflect real-world scenarios where many functions operate on the same kind of data, but only some functions able to consume the outputs of others in meaningful way. For example, many functions could accept an argument type "Employee", but some functions (such as num managed emppoeeess would expect that "Employee" to have subtype "Manager."
15

---

Published as a conference paper at ICLR 2026
Function Linking Strategies
Linking based on variable names:
Func_yep processes variable mfmjsy to produce variable tcok.
Func_ayj processes variable tcok produce variable arpl. Linking based types:
Func_yep processes variable mfmjsy (type_uxe with subtype_muv) to produce variable aargww (type_beo with subtype_dej)
Func_ayj processes variable riivq (type_ber with subtype_dej) to produce variable sjyav (type_wdc with subtype_uqq)
B.3
EXAMPLE MODEL. INPUT
We provide example of the model input used in experiments below.
Example Model Input
User Prompt:
Using the tools your disposal, functions until you are able give the correct value of variable bujxye.
Variable mfmjsy = 731
You have all the information you need to get the correct result.
Tool Prompt:
[(' "tppe: 'function',
'functionn: ('name': 'func_yep's
"description' Processes variable of (type_uxe with subtype_muw) to produce (type_beo with subtype_dej)', 'strict'; : True,
"parameters': ('type': 'object', properties': ('mfmjsy': ('type': 'integer']]
required': ['mfmjsy'], Paditiiiillrpetttiees False}]),
]
C ADDITIONAL RESULTS
C.I RESULTS OF NO EXTRA SETTING
Table 5 shows the results of all models in the no extra irrelevant node setting. We observe that the trends of success rates and ACs are consistent with those Section while the "No Extra" setting obtains higher success rates in general.
C.2 RESULTS OF DEPENDENCY DEPTH WITH ERROR BARS
Figure 13 shows the results for each dependency depth with error bars in which the results averaged across the numbers of noisy connected nodes.
C.3 RESULTS OF OTHER DEPENDENCY DEPTIIS
Figure 14 shows the results of all models for each dependency depth with 20 nodes. We observe similar trends to those Figure Lower dependency depth leads to higher success rates across models.
16

---

Published as a conference paper at ICLR 2026
Table Success rates and average function calls (ACs) that were actually made by models. Results are aggregated across only no extra irrelevant node configurations and graph dependency depth fI....n 1}. with 5 random trials per configuration. ACs (Suec.) and (Fail.) denote the ACs for successful and failed trials, respectively. indicates that there no successful trials.
# core nodes
10
20
Success Rate
ACs (Succ.)
ACs (Fail.)
Success Rate
ACs (Succ.)
ACs (Fail.)
Success
ACs (Succ.)
ACs
(Fail.)
GPT-5
80.0%
5.0
5.0
28.9%
10.4
9.6
14.0%
21.0
17.6
GPT-5-mini
20.0%
5.0
4.7
8.9%
10.0
8.9
4.0%
20.0
17.5
Gemini-2.5-PHr
65.0%
5.0
5.0
20.0%
10.0%
20.0
24.6
Gemini-2.5-Fii Flash
35.0%
5.0
0.5
22.2%
10.0
0.3
10.0%
20.0
0.0
Qwen3
10.0%
5.0
5.0
4.4%
10.5
9.9
4.0%
21.0
18.2
GPT-4.1
15.0%
5.0
4.0
6.7%
10.0
6.9
2.0%
21.0
12.2
GPT-4.1-mini
10.0%
5.0
4.3
0.0%
8.6
0.0%
15.0
GPT-5
Gemini-2.5-Flash Qwen3
GPT-4.1
GPT-5-mini
GPT-4.1-mini
Gemini-2.5-Pro
1.0
Success Rate
0.8
0.6
0.4
0.2
0.0
1
3
4
6
9
Dependency Depth
Figure 13: Success rates by the dependency depth with error bars. The number of core nodes is set to 10. Error bars indicate 95% confidence intervals.
Figure shows the results GPT-5 for each dependency depth with 40 core nodes. Since the core node size large, the success rates are 0 when dependency depth is greater than equals 9.
Figure 16 shows the comparison between the baseline and the proposed mitigation strategy with 10 core nodes. We observe similar trends those in Figure
C.4 FUNCTION CALLING SEQUENCE EXAMPLE
Figure 17 shows a full function calling sequence example of the case study in Figure 9a
17

---

Published as a conference paper at ICLR 2026
GPT-5
Gemini-2.5-Flash
GPT-4.1
GPT-S-mini
Qwen3
GPT-4.1-min
Gemini-2.5-Pro
1.0
Success Rate
0.8
0.6
0.4
0.2
0.0
1
5
7
11
13
15
17
19
Dependency Depth
Figure 14: Success rates of all models by the dependency depth. The number of core nodes is set to 20 and the results are aggregated.
1.0
0.8
Success Rate
0.6
0.4
0.2
0.0
5
9
13
17
21
25
29
33
37
Dependency Depth
Figure 15: Success rates of GPT-5 by the dependency depth. The number of core nodes is set to 40.
success rate
Original
0.8
w/ Mitigation
0.4
0.0
GPT-5
GPT-5-mini
Gemini-2.5-PP P-rPPr
Gemini-2.5 5 5-Flash
Qwen3
GPT-4.1 1
GPT-4.1-mint
Figure 16: Comparison between the baseline and the proposed mitigation strategy. The number of core nodes is set to 10. The results are averaged across (0, 10, 20, 40} CINs, with 5 trials each.
18

---

Published as a conference paper at ICLR 2026
Context
Using the too's st disposal, functionis) to compute and the correct value variable Variable earjy Variable exykad 565, Variable ryql 253, Variable ulkykp 943, Variable vpfw 744. You have information you need to get the correct result.
Function Call 1 func_ssc [Core] (Input Subtype:
*fno")
Arg: ("eahjjr";
361 (subtype: "opa")
Function Call func cdl [Core] (Input Subtype: "dzagf"
)
Arg: ("rygl": 253)
diuurey 17 (subtype: "nzywkr")
Function Call 3 func_puf [Core] (Input Subtype: "nzywkr",
"cpa")
("tco": "Ewcyk"" 361
xvebz 852 (sublype: "zefat")
Function Call func_eni [Core] (Input Subtype: "pogo", "cpa
Arg: (*vofw": 744, "hrj 3513
loovrr 853 (sublyype: "Injarm")
Function Call 5 func_hoj [Core] CInput Subtype: "jgze", "cpe" ")
Arg: ("ulkykp": 943, "bunula": 361)
xfz 152 (sublype: "sxwyrg")
Function Call 6 func_zip [Corc] (Input Subtype: "tti", "zefat")
Arg: ("exykod": 565, "suydf": 852)
Physvi
blype: "whd")
Function Call 7 Func_pbb [CIN] (Input Subtype: "sxxyrg") "
Ang: ("nsxyf"; 152)
exdd 932 sublype: "gvv""
Function Call 8 func_pped [CIN] (Input Subtype: "cpa")
('ztzot": 361)
brnvi
508 (sublype: "vak")
Function _nss [Core] (Input Subtype:
"sxwyrg"
"tnjsmm") )
The second
variable is expected to be 152
Arg: ("cef": 852,
"jaieu": 932.
"Itikr"s 1 853)
hjmqkg - 241 (subtype: "oo?)
Error: Incorrect Value
Function Call 10 func..ytt [Cone] (Irput Subtype-
"tiozz"
"whd", "nzywkr", "cos")
Arg: ("wzqzw": S58, "dpnvuu": 932)
"ajuth": 486, "fhahi": 17,
ufocyi 770 (subtype: "aqt") Value
No function has returned a variable in subtype "tiozz" yet
Function func__nv [Cono] (Irput Subtype:
"coj",
"naynkr")
Ang: ("Fak": 241, *snv1": : 17)
wkgyr 637 (sublype: "loozz") Error: Value Not Yet Known
Function Call 12 func_yyrt [Cora] (Irput Subtype:
"tiozz", "uhd"
"nzyukr",
"coj*)
Arg: ("wzqzw": 637, "donvuu": 241)
*ajuih": 486, "Fhahi": 17,
ufocyi
106 (subtype: "aqf") Value
Function Call 13 [Cona] (Irput Subtype:
"whd")
Arg: ("ieioss": 165,
496)
+ zly 19 (subtype: "uqujo")
Error: Value Not Yot Known
Function Call 14 func_ezw [CIN] (Irpus Subtype: "aqf")
Arg: ("fovex": 186)
luewxy S94 (sublype: "amovi") Error: Value Not Yot Known
Detailed function calling sequence. The left block shows the input to the function call which the model generates, and the right block shows the output from the function call. Core nodes highlighted in blue, connected irrelevant nodes (CINs) in green, and failed function calls in red. Errors are annotated below the output values.
Figure 17:
19


Reviews:
[{"review_id": "DSDYmdpOy8", "reviewer": "Reviewer_ncD9", "summary": "This paper proposes FuncBenchGen, a benchmark framework for evaluating LLMs on multi-step function-calling tasks under controlled and contamination-free conditions. The benchmark allows fine-grained control over task complexity (e.g., function count, depth, and distractors) and reveals key limitations in models' state tracking. A simple mitigation strategy, restating known variables, significantly improves performance.", "questions": "1. Since function schemas and variable types are randomly assigned, how do you ensure that the generated tasks remain semantically coherent or representative of real-world function-calling scenarios?\n2. How do you validate that randomly generated functions produce diverse yet meaningful reasoning chains, rather than trivial or redundant ones?\n3. The process for generating CIN and DIN nodes appears partly heuristic. Could you clarify how these nodes are chosen and connected in practice?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper proposes a novel benchmark framework that addresses a critical and under-explored challenge, evaluating LLMs on multi-step tool use in a contamination-free, controlled setting.\n2. Experiments are comprehensive, covering a range of models and various settings.\n3. The proposed mitigation strategy, while simple, is effective and general.", "weaknesses": "1. The paper suffers from significant formatting and presentation problems. For example, the abstract is split into three paragraphs, disrupting its coherence.\n2. The paper does not provide concrete examples, making it difficult to understand what the actual data instances look like. Without such examples, it is unclear how complex the generated tasks truly are, what kind of reasoning is required at each step, or how function inputs and outputs are represented.\n3. Function schemas and variable types are generated entirely at random without semantic constraints, which reduces interpretability and realism. As a result, the benchmark evaluates abstract structural reasoning rather than meaningful function-calling behavior, limiting external validity.\n4. Tasks are purely synthetic, with limited connection to real-world API use or natural semantics.\n5. The proposed \"variable reminder\" mitigation, while effective, is arguably a hack rather than a principled solution. It sidesteps the problem rather than enabling models to learn better internal state tracking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AT5l9UAa3i", "reviewer": "Reviewer_UX5z", "summary": "This paper proposes FuncBenchGen, a contamination-free evaluation framework designed to address the key limitations in existing benchmarks for multi-step function calling of TaLMs. Current benchmarks face the risk of data contamination and lack fine-grained control over task complexity factors such as function dependency depth, the number of core functions, and the presence of irrelevant functions. FuncBenchGen resolves these issues by formulating multi-step function calling as traversal over a hidden function-dependency DAG. The framework dynamically generates synthetic functions and tasks during evaluation to ensure no data leakage, while allowing users to precisely adjust complexity parameters, including the number of core functions, dependency depth, and the type and quantity of irrelevant distractor functions. Extensive experiments on seven state-of-the-art large language models show that reasoning-optimized models consistently outperform general-purpose models, with GPT-5 achieving the highest success rate but performing poorly in long-sequence tasks. CINs severely impair performance by interfering with relevant paths, and model performance declines sharply as dependency depth increases. The most common failure mode is the use of \"unknown variable values,\" while syntax errors are relatively rare. A simple mitigation strategy, explicitly restating all known variable values at each step, can significantly improve performance.", "questions": "1.After a task is split into m core nodes, will m function APIs actually be built as tools for the large model to call? If querying addresses, web searches, etc., are involved, how is each of these tools constructed? \n2.When the model is only provided with the meta-information (function description, type + subtype annotations) of synthetic functions without knowing their preset \"input→output\" mapping rules, how does the model ultimately obtain the expected target variable value? How does the model infer the correct function call sequence without knowning the semantic meaning? If the aforementioned questions and shortcomings are explained, a reassessment of the score may be conducted.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.Proposes an innovative DAG-based FuncBenchGen framework enabling fine-grained complexity adjustment.\n2.Conducts extensive experiments on 7 state-of-the-art LLMs with rigorous variable control and repeats several times, yielding reliable results.\n3.Presents a practical lightweight mitigation strategy with significant performance gains without model modification/training.", "weaknesses": "1.Given a task, how can a framework automatically generate a test suite with a specified depth of n? How should the functionality of each core node be constructed? It is crucial to ensure that exactly n nodes are sufficient and necessary to complete the task and that each node is feasible; however, this is not mentioned in the text.\n2.There is a discrepancy between the \"type + subtype\" matching of synthetic functions and the parameter semantic matching of real APIs. The framework's migratability in real API scenarios has not been verified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KfTK4wiQlW", "reviewer": "Reviewer_Hw9C", "summary": "The paper introduces FuncBenchGen, a benchmark for tool-augmented language models. The authors synthetically generate function-calling tasks in the form of DAG's, with each node taking in some input/output variables and edges representing function composition. The authors benchmark seven LLMs on FuncBenchGen and propose a mitigation strategy to improve performance on their benchmark.", "questions": "- How would fine-tuning on the task affect the performance on the benchmark? One advantage of the synthetic setup is that it is very easy to generate more examples of the task to study something like this.\n- While you mention is an abstraction of a more complex real-world task, do you have any evidence performance on this task correlates in some way to performance on a real-world function calling task? Often, tasks that are equivalent down to some abstraction can have large differences when evaluating a model on them.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed benchmark is synthetic, which allows the authors to study various properties in a controlled manner. The authors vary multiple dimensions such as the number of irrelevant premises, the size of the graph, and so on.\n- An aspect of tool-use is measured: being able to retrieve the correct functions and use them in a way that allows the LLM to solve the execution prediction problem.\n- The paper is presented clearly and the evaluation is sound. Most SoTA models are tested on the benchmark, and quite a few of them struggle to complete the task.", "weaknesses": "- While this synthetic task is interesting from an analysis point of view, it is unclear how it applies to real-world function calling scenarios. While the authors briefly address this in Sec 3.3, I find the abstraction to simplify the task a lot and would like to see at least one of the described real-world scenarios evaluated in a more realistic setting to validate the simplified formulation being a reliable proxy.\n- The papers fails to mention a large relevant body of literature focused on following the flow of programs and logic chains (e.g. https://arxiv.org/abs/2209.00840, https://arxiv.org/abs/2401.03065, https://arxiv.org/abs/2403.16437). These tasks, in spirit, are very similar, as they relate to being able to follow relevant chains of logic in a graph. Many of the research questions have been studied in similar contexts (e.g. RQ2 in https://arxiv.org/abs/2302.00093, RQ3 in https://arxiv.org/abs/2406.17169).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Uer9mOqUrD", "reviewer": "Reviewer_cs9L", "summary": "This paper presents FuncBenchGen, an automated contamination-free benchmark for testing tool-augmented LMs on multi-step function calls with fine-grained control over difficulty. It treats tool use as walking a hidden function-dependency DAG and generates synthetic function schemas at eval time to prevent train/test leakage. The setup lets you tune depth, function count/types, and add distractors. The authors test seven leading LMs, analyze where they fail, and show a simple fix—explicitly restating variables—significantly boosts success.", "questions": "- Could you share more about how FuncBenchGen samples DAGs? In particular, how do you make sure the sampled graphs are truly representative for a given size and depth?\n- How well does the synthetic type-matching approach hold up with real-world function calls where types are deeply nested and often ambiguous? What is the suggested way to use real API sets to stress-test realism and semantic complexity?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The framework generates tasks and functions on the fly, preventing pretraining/test-time leakage and removing the data-contamination issues that plagued prior function-calling benchmarks.\n- FuncBenchGen exposes fine control over function-set size, dependency depth (DAGs), and irrelevant distractors (connected/disconnected), enabling targeted evaluation along key axes (Table 1).\n- The study is broad and careful. The leading LMs are tested across graph sizes, complexities, and distractors; The results (Table 2; Fig. 3–4) give a multi-view of behavior, and the executor breakdown (Table 3) shows variable-tracking errors dominate—pointing to clear fixes.\n- The proposed fix is simple and effective. Based on the reported results, for example, GPT-5 show gains from 62.5% to 81.3%. Besides, modeling multi-step tool use as DAGs and enforcing schema/type matching aligns with real API workflows and applications.", "weaknesses": "- The missing prior work and limited experiments. The study does not directly discuss or compare to the prior work on contamination-free or fine-grained function-calling benchmarks (HammerBench [1], LiveBench [2], etc.). This may significantly limits novelty and contributions. This paper should include the missing work also in Section 2, so to more competently introduce the related work and explain how FuncBenchGen is distinct in controllability, expressivity, or realism.\n- Shall include more examples to better show the motivations and potentials/limitations of FuncBenchGen. For example, the paper may show qualitative examples of actual failed call traces or state-tracking failures, such as incorrect call propagation, classic state-tracking errors. Besides, it shall be better if this paper shows more complex control flow, such as conditional logic, iteration, probabilistic outputs, which is commonly appearing in real-world systems.\n[1] Wang, Jun, et al. \"Hammerbench: Fine-grained function-calling evaluation in real mobile device scenarios.\" arXiv preprint arXiv:2412.16516 (2024).\n[2] White, Colin, et al. \"LiveBench: A challenging, contamination-limited LLM benchmark.\" arXiv preprint arXiv:2406.19314 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8CgHTciOey", "reviewer": "Reviewer_Lq7b", "summary": "This paper presents FuncBenchGen, a evaluation framework for tool-augmented LLMs. the main contributions are: (1) proposing a framework that can generate contamination-free function-calling tasks. Besides, they conduct extensive experiments with 7 LLMs. They have revealed some interesting findings. Finally, they proposed a simpel augmentation mitigation strategy that restates variable values from prior calls.", "questions": "See above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Overall, the presentation is good. The paper is well-written and easy to follow.\n2. The contribution is significant, and many experiments are conducted for demonstration. They proposed a evaluation framework that is contamination-free, requiring many task complexities including function, depth and irrelevant function types. Their experiments with 7 LLMs, including the most-up-dated GPT5.", "weaknesses": "1. failure analysis and mitigation strategy are not extensively studied. For example, how do you find these 4 types of these failures? how it widely spreading in the cases? How do you define the boundary of each failure type? Section 4.6.1 mitigation strategy, is there more experiments to demonstrate its effectiveness?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)