Benchmarking LLM Tool-Use in the Wild
Authors: Peijie Yu
Wei Liu
Yifan Yang
Jinjian Li
Zelong Zhang
Xiao Feng
feng zhang
Venue: ICLR 2026 Poster (2026)
Keywords: benchmarking
automatic evaluation of datasets
evaluation methodologies
evaluation
metrics
reproducibility
statistical testing for evaluation
Primary area: datasets and benchmarks
TL;DR: WildToolBench reveals that what truly challenges LLM tool-use is not artificial complexity, but simple, realistic user behaviors.

Abstract: Fulfilling user needs through Large Language Model multi-turn, multi-step tool-use is rarely a straightforward process. Real user interactions are inherently $\textbf{wild}$, being intricate, messy, and flexible. We identify three key challenges from user behaviour: $\textit{compositional tasks}$ that demand efficient orchestration of tool-call topologies, $\textit{implicit intent}$ spread across dialogue turns that require contextual inference, and $\textit{instruction transition}$, which mixes task queries, clarifications, and casual conversation, forcing LLMs to adjust their policies on the fly. Existing benchmarks overlook these behaviors, making the apparent progress of LLMs on tool-use spurious. To address this, we introduce $\textbf{\textit{WildToolBench}}$, an LLM tool-use benchmark grounded in real-world user behavior patterns. Comprehensive evaluations of 57 LLMs reveal that no model achieves an accuracy of more than 15\%, indicating a substantial gap in the robustness of LLMs' agentic ability. Controlled experiments and in-depth analyses further indicate that the real challenge for LLM tool-use lies not in artificially complex tasks, but in the wild nature of user behavior, emphasizing the need to reconsider the interactions among $\textit{LLMs}$, $\textit{users}$, and $\textit{tools}$.

Published as a conference paper at ICLR 2026
BENCHMARKING LLM TOOL-USE IN THE WILD
Peijie Yu', Wei Liu', Yifan Yang', Jinjian Li', Zelong Zhang' Xiao Feng', Feng Zhang'
'Tencent HY, "King' College London (peijieyu cetrieeyu@eennet.com
ABSTRACT
Fulfilling user needs through Large Language Model multi-turn, multi-step tool-use is rarely straightforward process. Real user interactions are inherently wild, being intricate, messy, and flexible. We identify three key challenges from user behaviour: compositional tasks that demand efficient orchestration of tool-call topologies, implicit intent spread across dialogue turns that require contextual inference, and instruction transition, which mixes task queries, clarifications, and casual conversa- tion, forcing LLMs to adjust their policies on the fly. Existing benchmarks overlook these behaviors, making the apparent progress of LLMs on tool-use spurious. To address this, we introduce WildToolBench, an LLM tool-use benchmark grounded in real-world that no model achieves an accuracy of more than 15%, indicating substantial gap in the robustness of LLMs' agentic ability. Controlled experiments and in in-depth analyses further indicate that the real challenge for LLM tool-use lies not in artifi- cially complex tasks, but the wild nature of user behavior, emphasizing the need reconsider the interactions among LLMs, users, and tools. The benchmark is publicly available at https://github. http:://hhhhbboooooooogggg/////t
behavior patterns. Comprehensive evaluations of 57 LLMs reveal
INTRODUCTION
Opua
3/DospGuah
65%
42%
269
WildToolBench
BFCL-V4
BFCL-v3
BFCL-v2
T-Bench
1--Bench(Telecnnn
Figure 1: Session Accuracy comparison among tool-use benchmarks. See details in Appendix B
Large language models (LLMMs) are evolving rapidly, and agents built on them have become promising direction (Google| 2024 DeepSeek-Al et al. 2025 Zeng et al. 2025 Yao al. 2023 Liu ct al. 20244] 2026 Qian et al. 2024b) Qian et al.j 2024a). These agents interact with the real world through various tools, opening up avenues for Al applications. Developing benchmarks that evaluate the tool-use capabilities of large language models reliable way has become increasingly important.
Current mainstream LLM tool-use benchmarks follow a multi-turn multi-step paradigm: LLMs function assistants and engage in multi-turn dialogues with users complete coherent tasks. Each

---

Published as a conference paper at ICLR 2026
+
But Challenging Tasks for LLM
Find Optimal Tool Use Orchestration for Quick Response
Infer information and Intert From Multi-Turn Dialogue
Switch Among
Various Policies On the Fly
Single Woather
Parollol Wouther
Auk for Calriication
Single Search Tool Use
Arswear vil Tool Use
Loe Angalle
Saple
weekend the wouthe for weekend.
How 2001 Suni Seath wine.
Now. 2814 PPC-1PCC
tarribb Check and Las Vegin
Rany 1ut---rt seerrvrd
Tany
Viggin
(howss Sheeet
Whech ety's perforrances edterisedd
Vegas
potieen
11-32) thai "Loes
(2054-
Tel
Aeterion rock bend
The Exginn
dattydaayy. ErC-unn
ferowdl lour.
Ramay
Simple Task
Folfow Up Yeak with Mulippe Requinements
Emquering
Start Now
Tasi
Just a Natural Dialogue for User
Figure 2: WildToolBench poses three characteristics that seem easy and natural for the user, but challenging for the L.M tool-use.
task typically requires multi-step tool-use. However, existing benchmarks (Huang al. 2024a Qin ett al. 2024; Du et al. 2024 Yao et al.] 2024 Ji et al., 2024b) are overly idealized and neglect the complexity of multi- multi-step settings in real- world scenarios. From large-scale analysis of real user logs, we identify three salient properties of how human users employ LLMs to solve tasks with tools: 1) users tend to deliver Compositional Tasks that contain multiple simple requirements, demanding tool orchestration beyond simple chaining to respond on time. 2) Users' implicit intention spread within dialogue, requiring LLMs to infer it from context. 3) In conversation, users naturally transition between different types of instructions, such as task-giving, follow-up, explanation, and casual chatting modes, demanding LLMs to adapt their policies on the fly.
These three characteristics embody the design philosophy of WildToolBench, "What truly challenges LLMs' tool-use capabilities not artificially constructed complex scenarios, but simple yet realistic user behaviors". namely, the compositionality, vagueness, and variability of user instructions. In WildfoolBench, through a carefully constructed data pipeline combined with human verification and annotation, we curate 256 scenarios with 1024 tasks. As shown in Figure [ while prior tool-use benchmarks tend to be saturated, WildToolBench remains highly challenging. Our results show that even the most advanced language models struggle achieve satisfactory performance, with most models reaching no more than 15% session accuracy. A further breakdown of experiments on 57 LLMs reveals that in-the-wild task settings severely degrade model performance, underscoring that the future evaluation of LLMs' agentic ability cannot rely on simple, idealized benchmarks but must instead account for the inherent complexity of real-world user behaviours.
2 RELATED WORK
LLM agents have emerged as a prominent research direction, with their core competency rooted in the ability to utilize external tools. Tool-use benchmarks have, to some extent, shaped the evolution of LLMS' agentic ability, from simple QA to multi-turn, multi-step. long-horizon autonomous tool-use. T-EVAL, UltraToool, and MetaTooll(Chen et al.] 2024 Huang et al. 2024a b) assess various sub-capabilities of tool-use, but treat tool invocation as a simple question-answerriig task, which fails to capture the multi-turn interactive nature of the LLM agent loop. WorfBench and TaskBench(Qiao et al. 2025 Shen et al. 2024) took a step forward by introducing single-tun lingl-turn multi-step tool invocation and emphasizing planning capabilities, but are constrained by annotating only single optimal path and relying on similarity- based metrics, which can be imprecise in evaluation. ToolBench, Any ToolBench,, and TooleeeTooBeBchOOOO
al. 2024 Du et al. 2024 Guo al. 2024) also focus single-turn
2

---

Published as a conference paper at ICLR 2026
multi-step tool-use, but their proposed tasks are synthesized by LLMs and generally exhibit low level
of difficulty. On the other hand, BFCL-VI and BFCL-V2 (Ji et al. (024a) pioneered the evaluation of parallel tool-use but were still limited to single-turn scenarios. BFCL-V3 (i et al. 2024b) introduced multi-turn evaluation and assessed the sequential multi-step capabilities of L.LMs. However, its tasks lack semantic correlation, with each task being independent and identically distributed, and provided with complete intention and information, which is unnatural compared with real-world user behaviours. Therefore, r-Bench and 7²-Bench( Yao et al., 2024 Barres et al. 2025) introduce the design of LLM-as- User. To some extent, user simulators better approximate real environments (e.g., requiring an LLM agent to proactively ask questions rather than merely execute tool calls reactively). However, LLM- simulation still diverges significantly from real user behavior. For instance, LLMs tend to behave an unrealistically flawless manner, making tasks too easy to solve. Moreover, reliance on LLM simulation also leads to unstable evaluation results. Through a human-in-the-looy annotation process, WildToolBench explicitly incorporates three real user behaviors (compositional tasks, implicit intent, and instruction transition), thereby setting new standard for evaluating LLM tool-use. A comparison between Wild'foolBench and previous benchmarks is provided in Table [T]
3 WILDTOOLBENCH
3.1 FORMULATION
We formalize the interaction between a user and LLM as multi-turn dialogue, denoted as
D {ui, @11 U2 ....... uN, aN}
where u; is the i-th user message and a; is the corresponding LLM assistant response. Within this N-turn dialogue, there are M user tasks {gi...., 9M } which are scattered throughout the dialogue. For each user message u, there may exist a task gj. and the LLM needs to detect the user's intention and solve the task in the response ag If solving this task requires tool usage, the LLM will first engage in a process of multi-step tool invocation, which can be regarded as the LLM conducting several rounds of interaction with the external environment (e.g.. local database or a MCP server), denoted as T {af c,, a2 62, a5; es), where a² is the L.M'' tool call, and e the corresponding environment feedback after executing this call. Once this S-step tool invocation T is completed, the LLM gathers information from feedback and generates the user's ' response to the task with 0;.
In real scenario, user intentions are varied in one dialogue session, and user messages are mixed with various types of tasks g. such as asking questions, requesting follow-upe ups, seeking improvements, explaining themselves, or just chatting. The LLM needs to apply different policies for correct reactions, which may include 1) L.M just replies without any tool usage (S = 0), such as in response to task that needs clarification Jetarify or a task that does not require tool Jchat. 2) LLM adapts a single-tool invocation policy (S = 1) for simple task Jsingle: or 3) LLM performs multi-step tool invocations (S > 1) for a hard task gmults. From the LLM's perspective, the dialogue unfolds as a Markov Decision Process (MDP), where the state at each step is the full dialogue history (including u, a, aT, and e), and the actions are the tokens that formulate different policies. Under this formalization, WildToolBench faithfully reflects the complexities and challenges inherent in applications for real-world users, where 1) the user task g is compositional, consisting of multiple sub-requirements, necessitating effective tool orchestration. This implies that T may be tree rather than a simple chain-like execution. 2) User tasks {91,... 9M } are contextually interrelated, requiring the LLM uncover latent context from historical observations, including user messages {u} } and assistant messages {a}. 3) User intentions transition in each message u, and the LLM must switch its policies accordingly to give correct response a.
3.2 DATA CURATION
The data curation pipeline of Wild'oolBench follows three steps as shown in Figure 3 summarized seed application scenarios and user behavior patterns from large-scale real user logs, and iteratively rewrote and expanded them into 256 scenarios. Then, following ToolAlpaca (Tang et al. 2023), we collected more than 1,600 publicly available APIs from the internet, carefully verified and cleaned them into tool set. For each scenario, picked reasonable tool subset and generated 4 tasks based on it. Finally, we employed GPT-4o (OpenAll (022a) to construct a multi-agent system simulating the roles of user and assistant, generating initial trajectories under the given task
First, we
3

---

Published as conference paper at ICLR 2026
1. Scenario Curation
3. Multi-Agent Simulation
Seed Scenatios
and User
Behaviour Patterns
Generate
Refire
256 scenarios
lcentff
Real User Logs
User Agent
Assistant Agent
Raw Trajectory
2. Task Curation
Human Check and Refine
Public APls
llumen
WildToolBench
1600+ Tools
1024 Tasks
Figure 3: The data curation pipeline of WildToolBench.
and tool subset. Each tool invocation in the trajectory was manually examined and annotated as ground truth, producing the final dataset. The detailed process is described in Appendix [ Each stage of the data curation pipeline involved manual annotation and validation to ensure accuracy and diversity. Furthermore, in the manual inspection of tasks, we emphasized three aspects: task compositionality (3..) contextualized intention (3.4). and instruction transition (3.55). reflecting
the inherent complexity of real user behaviors. Finally, we present comprehensive statistics of
WildToolBench in 3.6 The unique design WildToolBench is highlighted Table [
Brack and
Invergue Bern
Crease Drewlloee
dependency creating a survey
OP: Net 3-stega
b) Enumerate possible tool ececution trajectory
Match LLM's tool
Score LLM's trajectory with AP OP
Figure 4: Visualization of the mattherrtte-mtcc-ccorrc pipeline for evaluating the LLMs' tool orchestration ability in WildToolBench.
3.3 CHALLENGE 1: TooL ORCHESTRATION FOR COMPOSITIONAL TASK
Real-world user instructions do not always present very hard tasks, but multiple simple requirements are combined into single instruction. We meticulously constructed tasks under common scenarios (e.g., document operations or weather inquiries), but with compositional forms that better reflect real user instructions (e.g., searching for popular movies generate survey slide, multi-city weather inquiries intertwined with travel planning). Compared with simple and well-defined tasks found in previous benchmarks, these are "in-the-wild" tasks that require LLM possess strong planning capabilities to identify tool dependencies and construct an efficient tool-calling topological graph, thereby improving TTFT (Time to First Token). To accurately measure whether an L.LM can
4

---

Published as a conference paper at ICLR 2026
effectively construct an efficient tool-calling topology. WildToolBench measures not only the final task accuracy but also more fine- aie-grained metrics such as the optimal path rate and task accomplishment progress, in a simple three-stage manner: enumerate, match, and score.
First, the adjacent tool dependencies are manually labeled by human experts. Then, we apply a depth-first topological sorting algorithm (see Appendix [D foor details) generate all possible legal tool execution paths that obey the adjacent dependencies. Our approach enumerates all possible tool execution paths, rather than restricting to limited suboptimal paths (Qiao et al. 2025 [Shen et al. 2024). Such an enumeration generates a decision tree set that considers all branching and parallel scenarios. For example, in Figure 4 a), the
Enumerate
then-investigate branch and the slide branch can be executed in parallel. leading to five possible paths as shown in Figure bb.
Match Every time the LLM executes a tool, we use an incremental path matching strategy locate this tool call in the previously enumerated decision tree set. Each tool call either terminates the path if mismatched or takes a step into the corresponding sub-tree.
Score By matching and locating the LLM's tool call in the enumerated decision tree set, we can evaluate the quality of the LLM's is current tool execution topology. Whenever the tool executed by the LLM terminates or completes a path, we calculate whether this path has the minimum depth among all enumerated decision trees. so, indicates that this decision tree is not only valid but also possesses optimal efficiency, and we can calculate the Optimal Path Rate (OP Rate). Furthermore, the LLM often fails on many tasks, with tool-calling nodes generated midway that do not fall within the valid decision tree set. We calculate the Accomplish Progress Rate (AP Rate) based the proportion of its successful nodes. These two fine-gras one-grained metrics, OP and AP, are used to measure tool orchestration.
Tuan
should dff marite
Tao gent aieao ynri coni Wou Oow 301
Comfee Tuxe Sport cire an eay
zp1
car not by conmared
flomion Eredees
Turn
Leeg Rangy Drpentenry
snices vhic dencral Ii's acall)1?
Task SengleTool
No Trandtice
Task 2
Trandiice
Task 3
Toans tony
Task 4 Minud Muli-Tool
Single-To
Clarrty then Fundlad Mulii-Tol
Figure 5: Examples for Challenges on Hidden Intention $3.4 and Instruction Transition |3.5 These challenges arise from the very nature of real user behavior: from the user's perspective, the interaction is a coherent dialogue rather than a series of isolated task submissions.
3.4 CHALLENGE 2: INFER HIDDEN INTENTION THROUGH DIALOGUE
Previous research (Chiang et al. follow up with additional questions and may modify or omit contextual information, which aligns with our observations. The LLM must infer the user's latent intentions from the multi-turn conversation, gather the necessary information and even proactively request clarifications. In WildToolBench, we utilize three strategies to construct tasks that demand multi-turn context inference, as shown The current user message i4 contains only a subset of the information required to complete the task, while the omitted information is present in previous user or assistant messages {wi, .i,..., 1 0;-1). J.. 2) Coreferential Reference: The current user message contains the full information, but the subject is expressed only via pronouns or ellipsis, referring back to entities mentioned in earlier user or assistant messages. 3) Long-Range Dependency: Similar to partial information, except that the missing information is located in distant dependencies; that is, u, depends on {wi, .1,...., My az with i j 2.
2023 Su al. 2019) reveals that in sequential tasks, 80% of users
in Figure 5] 1) Partial Information:
5

---

Published as a conference paper at ICLR 2026
3.5 CHALLENGE 3: ADAPTABLE POLICY SWITCH FOR INSTRUCTION TRANSITION
When interacting with an LLM assistant, most users treat the interaction as a natural conversation rather than a series of independent task submissions. Users frequently initiate tasks across multiple turns, ask follow-up questions, provide explanations, engage in casual dialogue, and interrupt or resume tasks at will, continuously transition among different instruction types. As illustrated in Figure 2 and Figure what appears to users as an ordinary conversation in fact involves multiple transitions. Such flexible and frequent instruction transitions require the L.LM to adapt its policy appropriately, making suitable choices among strategies such as tool-use, direct question answering. or proactive inquiry.
In constructing WildToolBench. we categorize all tasks into four types: tasks solvable with single tool call (Raingle), tasks requiring multiple tools and multi-step calls (fmutt), conversational or tool-free queries (gehat), and tasks that require the assistant to ask for clarification (Selarity). For each scenario, we carefully curated the proportions these four task types as well as their switching frequency, ensuring that WildfoolBench faithfully reflects the phenomenon of instruction transition observed in real user behavior. This setup benchmarks LLMs' ability to accurately track evolving user intentions in natural dialogue and generate appropriate responses.
3.6 STATISTICS
Toot
Lneg Fange
9,29
3.89
Corls
200
Disrication Types
2:18
Oeciea Dioriibation Toal Call Per
Dioribution Diliegan Tures
Disribaatio Tool
Figure Key statistics for WildTooolBench.
Figure 6 presents detailed statistics of WildToolBench. We constructed 256 scenarios, each consisting of a multi-turn dialogue with four user tasks, resulting in a total 1,024 tasks. We evaluate whether the LLM generates the ground-truth tool calls within the dialogue, measuring both task-level accuracy and session accuracy, i.e., whether all four tasks in a dialogue are correctly completed.
The key observations from these statistics are follows: 1) the four task types (gsingles Imultis Ichat Getarity) and various forms of hidden user intention are well balanced, ensuring diversity and challenge within each dialogue; 2) tool parameter types are highly diverse, and the tool type covers 8 major categories and 24 subcategories, all of which correspond to commonly encountered real scenarios; 3) the average dialogue length is 5.27 turns and the average number of tool-call steps is 1.92. which is significantly higher than BFCL (3.75 turns, 1.68 steps). We highlight the wild nature of WildToolBench in T
EXPERIMENTS
We present a detailed experiment by benchmarking 57 mainstream LLMs on WildToolBench, ranging from proprietary to open-source LLMs, from general to specialized models, and from instruction-tuned
to large reasoning models. The experiments and analysis are organized as follows: $4.1 gives an overview of benchmarking results and key takeaways. 44.2 investigates how well LLMs orchestrate tool calls to handle the compositional user instructions in WildToolBench, 5 examines the inference ability of LLMSs when users omit hide their intentions and information across multiple turns in

---

Published as a conference paper at ICLR 2026
Table I: Comparative analysis of the WildToolBench against other tool-use benchmarks.
Contextual Multi-Task
Hidden Info
User Instruction
Sequential Tool-Use
Purallel Tool-Use
Mixed Tool-Use
Benchmark
in Context%
Transition%
Wenchhh
BFCL v3
Paril or
3025
15.7%
39.7%
BFCL
Par aE
0.0%
0.0%
BFCL
0.0%
0.0%
ToolBench
Qin 8
2024
0.0%
0.0%
x
Any ToolBeecc
bench
(Barres
2015
r r-heech
Yao et al 21/4
T-EVAL
2074
0.0%
0.0%
UltraToof
Huann 0i
2024g)
0.0%
0.0%
X
dialogue, and $4.4 presents how frequent instruction transitions affect the LLM's s ability to make correct decisions. Finally, in 4.5 we provide empirical analysis of the errors that LLMs made in WildToolBench.
Table 2: WildToolBench Results.
Categorized by Task Type
Categorized by Task Order M
Overall
Models
Usingle
Omoitt
Hetarit
gchat
2
3
4
Task
Session
Proprietary General Models
G Gemini-2 2.0-Thinking
56.64
40.23
52.34
94.92
78.13
63.67
51.95
50.39
61.04
14.45
Gemini-2.5-Pro
55.08
36.33
46.88
86.72
70.31
56.64
53.52
44.53
56.25
14.06
Claude-4-Sonnet
41.80
71.09
52.73
44.53
$6.54
12.50
54.30
39.06
48.05
93.75
69.53
60.94
55.86
48.83
58.79
12.11
@ GPT-4o
60.16
41.80
39.45
78.13
72.66
55.08
46.09
45.70
54.88
11.72
0 Grok-4
59.38
41.41
33.59
66.02
63.67
52.34
42.97
41.41
10
10.16
@ GPT-5
46.09
34.38
31.64
84.38
62.11
50.00
45.31
39.06
49.12
5.86
Open-Souree General Modela
# GLM-4.5
57.81
40.63
44.53
81.25
70.70
60.16
50.78
42.58
56.05
12.11
Kimi-K2
54.30
33.98
39.84
86.72
68.75
57.03
48.83
40.23
53.71
10.55
DeepSeck-R1
56.25
41.02
43.75
80.08
74.22
54.30
48.83
43.75
55.27
9.38
V3
58.98
38.67
33.59
79.30
75.39
53.91
41.02
40.23
$2.64
9.38
Qwen3-32B-Thinking
53.52
28.91
37.11
80.86
62.50
52.73
46.48
38.67
50.
7.81
Open-Source Specialized Models
xL.AM-2-70B
64.45
36.72
28.91
64.84
64.06
51.56
42.58
36.72
48.73
7.81
ToolACLs2-BB
62.11
37.89
33,98
84.38
72.27
59.38
46.88
39.84
$4.59
7.42
Watt-8B
61.72
22.66
78.13
47.27
39.06
35.55
47.66
4.69
Hammer2.1-77
40.23
21.88
30.47
94.92
61.72
46.88
40.63
38.28
46.88
4.69
4.1 OVERALL PERFORMANCE
We evaluate three categories of models, including Proprictary General Models (OpenAl
2025
Anthropic 2025 Mistral 2024 Doubao 2025 OpenAl 2024b Google 2024 Seed 2025 Open-
Source General Models Qwen 2025 DeepSeck-Al et 2024 Yang al. 2024 DeepSeek-Ai
et al. 2025), and Open-Source Specialized Models (Zeng et al. 2025 Liu ct al. 2024b) Lin et al.
2024 Shi al. 2024). We employ each model's 's native Function Call format achieve optimal performance. Table presents the overall performance of lop-performing models. Full results on 57 models are provided Appendix E.2
In terms of overall performance, none of the mainstream LLMs achieve a session accuracy higher than 15%, and most models fall below 60% in task accuracy, highlighting the difficulty of WildhooolBechh. Proprictary LLMs generally outperform open-source ones, and maasoning-orientte models consistently surpass non-reasoning models. The best-performing open-source models, such as GLM4.5 and Kimi K2, achieve performance comparable to the top three proprietary models, while the remaining open-source models still lag considerably behind.
We further conducted drill-down analysis of task accuracy along two dimensions: task type and task order. For task type, when the user's intention is casual chat or tool-free answering, most LLMs can reliably recognize the intention and respond appropriately. However, when the intention involves
7

---

Published as a conference paper at ICLR 2026
clarification eliciting task details through counter-questions, LLMs frequently misfire by executing a function call. Moreover, multi-step tool-use exhibits substantially lower accuracy than single-step tool invocation. For task order, within dialogue, tasks appearing later exhibit greater dependence preceding information, and model performance deteriorates accordingly.
4.2
LLMs PERFORM Pool ON TooL ORCHESTRATION
Table 3: WildfoolBench tool orchestration evaluation result.
Task Accuracy
AP Rate
OP Rate
Models
S+P Amalsi
Overall
S+P 9 mallsi
Overall
S+
Overall
Proprictary General Models
G Gemini-2 2.0-Thinking
54.14
25.00
16.67
40.23
45.28
39.89
40.37
53.50
16.67
40.66
G Gemini-2.5-pp
49.04
25.00
14.29
36.33
47.17
39.15
39.87
43.31
11.90
32.37
A Claude-4-Sonaa
54.78
31.25
25.00
43.75
60.38
46.32
47.57
52.87
23.81
42.74
3 ol
50.96
12.50
21.43
39.06
35.85
37.50
37.35
50.32
20.24
39.83
@ GPT-4o
53.50
31.25
21.43
41.80
41.51
45.40
45.06
51.59
21.43
41.08
0 Grok-4
54.14
18.75
21.43
41.41
41.51
46.51
46.06
53.50
21.43
42.32
@ GPT-5
43.31
37.50
16.67
34.38
49.06
38.42
39.36
42.68
13.10
32.37
Open-Sowre General Models
# GLM-4.5
51.59
31.25
21.43
40.63
67.92
48.90
50,59
49.68
20.24
39.42
M' Kimi-K2
45.86
12.50
15.48
33.98
52.83
34,93
36.52
43.95
15.48
34.02
DecpSeck-R1
53.50
18.75
21.43
41.02
41.51
44.12
43.89
52.87
20.24
41.49
DeepSeck-V3
52.87
25.00
14.29
38.67
43.40
32.54
33.50
51.59
14.29
38.
<5 Qwen3-32B -h-Tinnning
42.04
12.50
7.14
28.91
41.51
28.31
29.48
40.13
7.14
28.63
Open-Source Specialized Models
xI.AM-2-70B
49.68
12.50
16.67
36.72
43,40
44.85
44.72
26.75
7.14
19.92
ToolACE2-8B
47.77
31.25
20.24
37.89
50.94
43.01
43.72
26. 11
14.29
21.99
Watt-8B
44.59
6.25
1.19
28.13
22.64
21.87
21.94
44.59
1.19
29.46
Hammer2.1177
33.12
12.50
2.38
21.88
24.53
13.24
14.24
31.85
2.38
21.58
We further analyzed whether LLMs can correctly orchestrate tool-call topologies to handle com- positional tasks. We divided compositional tasks into three categories according their required tool topology: sequential multi-step tool-use (o parallel multi-step tool-use (o and mixed tool-use combining both sequential and parallel structures (g As shown in Table 3 the highest task accuracy is merely 43. 3,75%, falling just 25% for g tasks, indicating that compositional tasks with multi-turn interactions remain a significant challenge for LLMS.. Similarly, the peak optimal path (OP) rate reaches only 42.74%. suggesting that current LLMs have substantial room for improvement in tool execution efficiency. See full results of 57 LLMs in Appendix E.3 Specialized tool-use models perform significantly worse than general-purpoct models, indicating limited generalization despite their intended focus. Claude-4-Sonae tool orchestration, outperforming other proprietary models. The Gemini series reveals strong bias, excelling in parallel but dropping sharply in mixed one. Among open-source models, GLM-4.5 excels sequential and mixed tasks, even surpassing leading proprietary models. Furthermore, we observe that reasoning- eeasoning-enabled enabled model variants outperform their non-reasoning counterparts within the same series, indicating that additional reasoning leads to better tool-call orchestration for compositional tasks. These results refute the conclusion in previous work Zhou et al.) (2025) that reasoning model does not outperform a non-reasoning model on tool-use, highlighting limitations in previous evaluations.
shows clear advantage in complex reasoning for
4.3 LLMs STRUGGLE TO INFER INTENTION ACROSS DIALOGUE
Figure reports the accuracy of LLMs on three types of user tasks, in which user intention and information are partially hidden or omitted across multi-turn contexts. We find that long-range dependency tasks are the most challenging, with model achieving accuracy above 50%. By contrast, tasks involving partial information or coreference are relatively easier. The results reveal clear specialization across models: reasoning models such ol and gemini-2.00 2.0-thinking excel at inferring omitted information and intent in partial information tasks, while Claude-4-Sone 4-nneet leads on coreferential reference tasks, indicating that no single model outperforms others across all aspects.
8

---

Published as a conference paper at ICLR 2026
Gurie Hlannng
55.3
56.7
57.1
66.4
A
Deuceefddereee
S1.7
93.4
$8.3
41.3
80
Gereni2.5
05
69.0
47.3
16.2
42.5
Gaude- Scroet
A
Grde-
GFTG
64.8
46.1
Cteek
GPTS
G.N-15
21.2
51.8
GIN-15
K
Kei-k
48.7
48.4
35.3
Deepdeck
SeasSeek
62.0
40.3
56.2
Doopbeck Vy
45.0
45.0
22655-228-Th (
5 Owond-322 328 Tlasking
66.0
49.3
69.3
42.6
46.1
Toe ACE2
68.7
46.8
Marmeenc
Hammer2.3170
61.9
45.3
45.8
35.0
Long
Figure 7: LLM's performance under different hid- den information strategies.
Figure 8: LLM's performance goes down as the instruction transition goes more frequently.
Long-range dependency tasks remain the weakest dimension overall, with scores clustered between 30 and 45, while also exhibiting the largest performance gap (17.3), making them a key differentiator among models. Mid-tier models demonstrate notable strengths despite lower overall averages; for instance, ToolACE2-8B (Ref 56.2) and MI.5-4.5 (Ref 57.9) approach top-tier performance. Model capability generally correlates positively with model size, illustrated by the Qwen2.5 series results in Table In general, reasoning models demonstrate stronger capabilities in inferring hidden intent and retrieving omitted information within multi-turn contexts.
4.4 USER INSTRUCTIONS CHANGE, AND LLMs LAG BEHIND
To further investigate the impact of user instruction transitions on LLM decision-making, we analyzed the performance of all models on tasks in WildToolBench with varying transition frequencies. As stated in Section $3. I we categorize the tasks into four types: Isingles Imalti Jehat, and Gelarify. corresponding to tasks solvable with a single tool call, multi-step tool calls, direct question answering, and tasks requiring clarification, respectively. An instruction transition is defined as a change in task type between two consecutive tasks within a dialogue. Given that each scenario contains up to four tasks, most three transitions can occur. As shown in Figure g across open-source and proprietary models, general-purpose and specialized models, as well as reasoning and non-reasoning models, task accuracy decreases the number of transitions increases. In some cases, the drop reaches as much as 30% in accuracy. Our analysis indicates two main factors underlying this trend. First, tasks with frequent transitions reflect more flexible and in-depth user demands (e.g., task requiring clarification followed by a follow-up query for further information). Such tasks more closely resemble real user scenarios and are inherently more difficult. Second, LLMs exhibit self-conditioning (Sinha et al.) 2025), whereby previous responses bias subsequent decisions. For example, if model previously used a tool call, tends to continue using tool calls; if it previously executed parallel tool calls, is biased toward repeating them. This interference prevents the model from selecting the appropriate response. Essentially, this arises because the long conversational context dilutes the model's 's attention the current task, historical user and assistant messages accumulate. This problem is particularly pronounced when the current task requires recalling past interactions (as noted in Section $3.4 further exacerbating the interference from historical context.
4.5 ERROR ANALYSIS
reveals that the primary challenge in LLM tool-use has shifted from syntactic correctness to semantic and logical reasoning. The data indicates two divergent failure philosophies: a "cautious" profile, exemplified by rather than risk an incorrect action (8.02% Wrong Name error), and an "eager" profile, seen in models like Grok-4, which minimizes refusals (3.72%) at the cost of significantly higher propensity to select the wrong tool (24.07% Wrong Name error). Across the spectrum, "Wrong Name Missing Info" and "Redundant Call" (23.06% in Gemini-2.00 2.0-hhiking) emerge as the most prevalent errors, highlighting systemic deficits in intent understanding and context management. This problem is particularly pronounced in specialized open-source models like xLAM--270B and Watt8B,, where "Wrong Name"
Table reveals
which prefers to refuse a task (24.56% Refusal rate)

---

Published as a conference paper at ICLR 2026
Table 4: Error Analysis in WildToolBench.
Action Errors
Parameter Errors
Models
Refisal
Wrong Name Missing
Wrong Retusal
Redundant
Call
Easl Termination
Param Type
Param Hallucination
Param
Propretury Generai Models
Gemini-2 hinniig
24.56%
8.02%
3.26%
23.06%h
18.05%
4.76%
1.50%
4.51%
12.28%
Gemini-2.. 2.5-Pro
33.93%
7.81%
3.79%
16.74%
14.51%
5.13%
1.12%
6.17%
10.49%
M Claade 4 Sonnet
9.44%
19.55%
11.24%
16.40%
12.13%
6.52%
1.80%
8.09%
14.83%
3.55%
8.77%
8.06%
11.37%
GPT-4o
5.41%
21.68%
12.12%
14.50%
11.47%
7.58%
3.46%
9.96%
13.85%
Grok-4
3.72%
24.07%
17.03%
17.81%
10.18%
5.68%
2.94%
6.46%
12.139
GIT--5
10.17%
Open-Snurce General Models
GLM-4.5
10.89%
19.33%
10.67%
18.89%.
15.33%
6.00%
1.33%
4.67%
12.89%
Kimi-K2
21.31%
13.50%
7.17%
16.24%
11.00%
6.54%
2.74%
6.75%
DeepSeck-RI
13,54%
14.41%
11.14%
20.96%
11.79%
6.33%
1.53%
8.52%
11.79%
DeepSeek-V3
10.52%
21.68%
10.93%
15.88%
16.49%
5.15%
1.65%
7.42%
10.31%
QQwen3-3283 Thinking
20.35%
9.20%
19.18%
19.18%
4.31%
Openn-Sourre Speetillised Models
xLAM-2-70B
30.67%
16.19%
5.71%
1.71%
5.14%
ToolACE:2
10.11%
28.60%
8.60%
6.67%
18.28%
6.02%%.
3.23%
3.08%
14.84%
Wair-8B
5.97%
30.97%
10.45%
7.09%
23.13%
4.29%
1.87%
5.78%
10:45%
15.26%
errors exceed 30%, suggesting that specialization can lead to brittleness. Conversely, parameter- errors such as "Param Type Error" or "Param Hallucination" are consistently lower across all models. This suggests that the frontier of agentic Al development now lies in improving higher-order planning and reasoning rather than basic syntactic generation. The prevalence of "Redundant Call" errors reveals a widespread deficiency in long-range planning for most capable models, indicating that they struggle with context management over time. However, the deceptively low rates of this error in some specialized models can misleading, as this "pseudo-capability" often masks a more fundamental failure to initiate tasks correctly, evidenced by catastrophic "Wrong Name" error rates.
5 LIMITATIONS
WildToolBench uses human annotations to ensure data quality, diversity, and alignment with real user behaviour distribution. However, this limits the scaling potential of the data size. What's more, the dual objectives of maintaining data quality and traversing all policy transition types concurrently limit the feasible length of tasks. Despite this, experimental results reveal significant trends model performance, leading robust conelusions the gap in current LLMs' tool-use ability. We also working on combining human-annotated rubrics with a fully automated synthetic environment scaling pipeline for both training and evaluation of the Agentic Model, which is the foundation for the next scaling trend in the Al era.
6 CONCLUSION
WildToolBench, grounded in real user behavior patterns, identifies three major challenges for LLMs performing multi-turn, multi-step tool-use: compositional instructions, hidden intent, and instruction transitions. Unlike prior evaluations that focus solely on increasing the complexity of tool-call procedures, WildToolBench emphasizes assessing L.M tool-use capabilities in the context of realistic user scenarios. Benchmarking nearly all mainstream models, WildToolBench reveals fundamental limitation in current LLM development: for effective tool-use, model cannot merely function as tool executor; it must also possess the capacity to understand users. This capability depends deeper foundational skills of large models, including instruction following, long-contex comprehension, and theory of mind-essentia abilities for future agentic models. Beyond serving leaderboard, WildToolBench provides structured rubrics that guide model developers interpreting user behaviors from multiple perspectives, facilitating more effective model iteration.
10

---

Published as a conference paper at ICLR 2026
7 REPRODUCIBILITY STATEMENT
WildfoolBench provides all the datasets, evaluation scripts, and all 57 LLMs evaluated trajectories to support 100% reproducibility. See these materials in the Github Repository.
REFERENCES
Anthropic. claude-3.7. https///ww...tttrrrr...cc/////// 2025.
Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan, 7 Evaluating Conversational Agents in Dual-control Environment. arXiv preprint arXiv:2506..7733 2025.
Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, and Feng Zhao. T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step. In Lun-Wec Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics |Volume I: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 9510- 9529. Association for Computational Linguistics. 2024. doi: 10/16655W1/2224AALL-LLMM URL phttps:///ol: org/ 10. 1865186533 3/ v /2024 ac ong. 5155
We Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An Open-source Chatbot Impressing GPT-4 with 90% ChatGPT Quality, March 2023. URL
https:///msys: http:///...rrooooo///o///// 2ttp://3lg/222-0333333 http:p//////ss..rrr//a///2222222 30000c0-v--a
DeepSeek-A1, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang
Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui
Ding. Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang. Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang. Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang. Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang. Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L.. Xiao, and Wangding Zeng. DeepSeek-V V3 Technical Report. CoRR, abs/2412. 1943777..1444 2024. doi: 1044550/ARXIV.2412.200 httpss://dii
URL
/ /ww. wwg.ioorg11.... 48540455000/0 /ttp//axiivv/... 2412 19437
DeepSeek-AL, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo. Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li. Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao. Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li. Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge. Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, and S. S. Li. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. CoRR,
2025. doi: 10.48500ARXIV2250..2220 URL
https://dot wttp.////....////.......o.o.
25 12948

---

Published as a conference paper at ICLR 2026
Doubao. Doubao http://///......aa.aa.aoo//////a 2025.
pro
Yu Du, Fangyun Wei, and Hongyang Zhang. AnyTool: Self-Reflective, Hierarchical Agents for Large-scale API Calls. In Forty-first International Conference on Machine Learning. ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. http:tts: httpttssss///peeeeeeertt net/
forum? "erum?id=qfe gF F.F.FFTbbiTT TkWW
Google. Gemini. htttas/e//eeepttaa.ooooooo//ooee
2024.
Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. StableToolBench: Towards Stable Large-scale Benchmarking on Tool Learning of Large Language Models. In Lun- Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting. August 11-16, 2024, pp. 11143-11156. Association for Computational Linguistics, 2024. doi: 13. 11111/11111633/12222222
2211665//1/2024411DDDD MANNDINGS-ACL.664 URL https:: Eindings-acc.666
Shijue Huang, Wanjun Zhong, Jianqiao Lu, Qi Zhu, Jiahui Gao. Weiwen Liu, Yutai Hou, Xingshan Zeng, Yasheng Wang, Lifeng Shang, Xin Jiang, Ruifeng Xu, and Qun Liu. Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACI. 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 4363 4400. Association for Computational Linguistics, 2024a, doi: 10. FFGES//W/22222FINNNNGGGG..
Complex Scenarios. In Lun-
URL httphttts://ttt org. 10. 10w118663/vl1 /2244. winddnggsaaclnnn.n 259
Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun. MetaTTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024b. net httttooummeeeeeeeet Rdw.ummdddRRRoeeeeoe
URL httpttt::///peeeeeiiittee
Charlie Cheng-Jie Ji, Huanzhi Mao, Fanjia Yan, Shishir G. Patil, Tianjun Zhang, Ion Stoica, and Joseph E, Gonzalez. Gorilla OpenFunctions v2, https::///ori:ii
cs www.keeey.eee................ee. edu/
/blogs/?_open_funee
sassss_22hhmmmmm 2024a.
Charlie Cheng-Jie Ji, Huanzhi Mao, Fanjia Yan, Shishir G. Patil, Tianjun Zhang, Ion Stoica, and Joseph E. Gonzalez. Gorilla BFVL V3. https:://orill as. cs wwlaa.cs.berkkeeee www.weee..eeeeeoeooooo.e...eee leaderboard. html 2024b.
Qiqiang Lin, Muning Wen, Qiuying Peng, Guanyu Nic, Junwei Liao, Jun Wang, Xiaoyun Mo, Jiamu Zhou, Cheng Cheng, Yin Zhao, Jun Wang, and Weinan Zhang. Hammer: Robust Function-calling for On-device Language Models via Function Masking. CoRR, abs/2410.045877 2024. doi: 10. URL doi org/10 48550/ arwiv .1l.i..2441000 2410. 04587
48550/ARXXIV.410.0555
Wei Liu, Chenxi Wang. Yifei Wang, Zihao Xie, Rennai Qiu, Yufan Dang. Zhuoyun Du, Weize Chen, Cheng Yang, and Chen Qian. Autonomous agents for collaborative task under information asymmetry. Advances in Neural Information Processing Systems, 37:2734 2765, 2024a.
Wei Liu, Peijic Yu, Michele Orini, Yali Du, and Yulan He. Hunt instead of wait: Evaluating deep data research on large language models. arXiv preprint vrXiv:2602.02039, 2026.
Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng. Rithesh R. N., Liangwei Yang, Silvio Savarese, Juan Carlos Niebles, Huan Wang, Shelby Heinecke, and Caiming Xiong. APIGen: Automated Plpeline for Generating Verifiable and Diverse Function-calling Datasets. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15. 2024. 2024b. URL http: http:://ppapeee nips cc cc/paper files/paper -atttsteeet-tttseteee...
and Benchmarks - html
12

---

Published as a conference paper at ICLR 2026
Mistral. Au Large.
2024.
OpenAl. GPT-4o. httpps//eeeaa..ccoomo/////lleeee
OpenAl. ol and ol-mini.
2024b.
OpenAI. GPT-4.1.
2025.
Shishir G. Patil, Huanzhi Mao, Charlic Cheng-Jie Ji, Fanjia Yan. Vishnu Suresh, lon Stoica, and Joseph E. Gonzalez. The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models. In Forty-second International Conference on Machine Learning, 2025.
Chen Qian, Zihao Xie, YiFei Wang, Wei Liu, Kunlun Zhu, Hanchen Xia, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, et al. Scaling large language model-basee multi-agent collaboration. In The Thirteenth International Conference on Learning Representations.
Chen Qian, Yufan Dang, Jiahao Li, Wei Liu, Zihao Xie, Yifei Wang, Weize Chen, Cheng Yang, Xin Cong, Xiaoyin Che, et al. Experiential co-learning of software-developing agents. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long 2024a.
Papers), pp.
Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. Chatdev: Communicative agents for software development. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), pp. 15174-15181 2024b.
Shuofei Qiao, Runnan Fang, Zhisong Qiu, Xiaobin Wang. Ningyu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Benchmarking Agentic Workflow Generation. In The Thirteenth International Conference Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https:: http::::////peeeeveeeeeee net.eewwwwwwwwweeeeee.e.e....e forum? SawerummidiuuuPoooo XOFmoi
Yujia Qin, Shihao Liang Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong. Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li. Zhiyuan Liu, and Maosong Sun. TooILLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL
Qwen. http:paeeeeeeeoeoeero.oooooooodd
httpss://penneeiew..ee www.nneee/ooouiieeoeeeee
2025
2025.
Seed. doubao-think.
2025.
Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng
Li, and Yueting Zhuang. TaskBench: Benchmarking Large Language Models for Task Automation. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NewrIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024, URL nips CC/ Satpapper___leeepppp__eeee 085 and Bww.hhenchhaaaeeee.eeaaa wrakkks__rcchhhhh.
Wentao Shi, Mengqi Yuan, Junkang Wu, Qifan Wang, and Fuli Feng. Direct Multi-turn Preference Optimization for Language Agents. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing. EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pp. Linguisties, 2024. doi:
Association for Computational URL UtRL:ttts:://ot http///oooor//////////////////// org/10
10055//V12202.EMNIPP.M.M 18653/v 16653/v1/220 //024 emn 1 mmm..nn.. 138
Akshit Sinha, Arvindh Arun, Shashwat Goel, Steffen Staab, and Jonas Geiping. The Illusion of Dimin- ishing Returns: Measuring Long Horizon Execution in LLMs. arXiv preprint wxXiv2550900967. 2025.
13

---

Published as a conference paper at ICLR 2026
Hui Su, Xiaoyu Shen, Rongzhi Zhang, Fei Sun, Pengwei Hu, Cheng Niu, and Jie Zhou. Improving Multi-turn Dialogue Modelling with Utterance ReWriter. In Anna Korhonen, David R. Traum, and Lluís Märquez (eds.). Proceedings of the 57th Conference of the Association for Computational Linguisties, ACI. 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pp. 22-31. Association for Computational Linguistics, 2019. doi: 10. 1..1865//11/19--1 1003. URL
1003
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. ToolAlpaca: Gener- alized Tool Learning for I Language Models with 3000 Simulated Cases. CoRR, abs/2306.05301, 2023. doi: 10448550/RXXW.220600552
URL https: wrw.orgg/110 0rt//00.4855/arxx11
2306. 05301
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 Technical Report. CoRR, abs/2412.15115, 2024. 4B5500/25500/21 2412
doi: 1044550/ARXIV2242..5111 URL https://d//d
15115
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. ReAct: Synergizing Reasoning and Acting in Language Models. In The Eleventh Inter- national Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL/pttpas//ppeeeeevew http:ew.eeeeoooooeeeeeeeo wor.:/orummieoo Poooummiiddw.. vluYOL-X
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 7-bench: A Benchmark for -oge--gent Interaction in Real-world Domains. International Conference on Learning Representations, 2024.
Xingshan Zeng, Weiwen Liu, Xu Huang, Zezhong Wang, Lingzhi Wang, Liangyou Li, Yasheng Wang, Lifeng Shang, Xin Jiang, Ruiming Tang, and Qun Liu. ToolACE-R: Tool Learning with Adaptive Seff-refinement. CoRR, abs/2504.01 400, 2025, doi: 10.4550/ARXI..550000000 URL
1a1/50/rrxxvv.v 2tw.ll...204..00000 01440
Xueyang Zhou, Guiyao Tie, Guowen Zhang, Weidong Wang, Zhigang Zuo, Di Wu, Duanfeng Chu, Pan Zhou, Neil Zhenqiang Gong, and Lichao Sun. Exploring the necessity of reasoning in Ilm-based agent scenarios. arXiv preprint 2rx3 2025.
14

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS (LLMs)
For the paper writing, we employed LLMs solely for grammatical correction at the writing stage. The LLM itself did not contribute experimental design, idea development, manuscript writing.
Other uses of LLMs (such as benchmark construction) have been clearly stated in 3.2 and
B BENCHMARK COMPARISON
Since the representative benchmarks we compiled span multiple time periods, not all models reported results for every benchmark in their original papers. Therefore, we collected evaluation results from multiple sources. Figure I primarily demonstrates that previous LLM tool benchmarks have tended toward saturation, while WildToollBench remains challenging. The information we compiled is mainly drawn from official reports of GLM4.5, Kimi K2, GPT5, and BFCL leaderboard well report by an independent third-party organization, Artificial Analysis P
The details are shown in Table
Table 5: Performance Comparison across Different Benchmarks
Models
BFCL-v4
BFCL-v3
BFCL-v2 7-Bench
PeBnhhhh (telecom)
Gemini2.5 pro
14.1
25.0
64.0
62.5
54.1
GPT4o
11.7
42.5
70.5
51.6
25.1
Claude-Sonnet
12.5
71.7
54.8
81.1
70.3
45.2
Claude-Opul
57.9
81.5
70.5
57.0
@ GPTS
5.9
28.5
58.3
72.3
96.7
Grok
10.2
68.9
36.1
74.4
67.5
47.9
GLM4.5
12.1
64.9
65.6
81.7
70.1
24.6
K2
10.6
42.1
48.8
80.8
62.6
65.8
@r DeepSeek RI
9.4
32.0
44.5
80.9
58.7
36.5
or DeepSeek V3
9.0
27.4
33.0
79.9
57.6
32.5
C DATA CURATION
Figure 3 gives an overall preview of the data curation for WildToolBench. First, we analyzed large collection of real user logs extract suitable seed scenarios and summarize behavior patterns. Second, we build our toolset by leveraging tool descriptions from public API following the approach introduced by ToolAlpaca. This GitHub repository is continuously updated and now contains more than 1400 tool lists, but to stay consistent with ToolAlpaca, we use 400 of these tool lists, covering around 1600 APIs in total. Then, selected corresponding tool subset for each seed scenario. In particular, we enumerated all possible simple and complex parameter types (String, Integer, Float, Boolean, Enum, Array, Object, Nested) enhance the diversity and complexity of tool parameters. Third, five human experts specializing in LLM agents inspected and refined these tool sets, mainly by correcting unreasonable tool combinations and parameter specifications, thereby improving the logical coherence and interoperability of tools. This process yielded 256 realistic scenarios, and for each scenario, we get a diverse and reasonable tool subset.
After obtaining the scenarios, we prompted User Agent to generate initial first-round user tasks based on the scenario and tool subset. Based on the four task types defined in this paper (fsingles Imaltii Ielarify Schat), used controlled generation type. To enhance diversity, we varied across five dimensions: sentence structure, linguistic style, task background, task length, and task difficulty. We then used the three omission types defined in Challenge 2 (User Hidden Intent), including Partial Information, Coreferential Reference, and Long-Range Dependence, together with real user questions few-shot examples, guide the User Agent in generating the subsequent three tasks. For each step, multiple candidate tasks were generated, from which human experts selected the highest-quality ones and refined them to better match human distributions, resulting in the final user tasks.
produce the first-round tasks for each
hhtps://gorlllasss.coo http://////llllls..oooooooaaaaaa berreclly.cctt/eaadrbbadddd
"https//ariffciaaamllsisslll https://gthub.coo/// http://hww....mooolllllaaaappppp
15

---

Published as a conference paper at ICLR 2026
Once the expert-refined user tasks were obtained, the Assistant Agent executed the Agent Loop for tool calls until producing& a summary. Each tool call the trajectory was then automatically checked for issues such function hallucination, parameter hallucination, type errors, and redundant calls. Subsequently, five human experts inspected the full trajectory, corrected errors (e.g.. in tool planning parameter values), and annotated tool-call dependencies (used construct DAGs for calculating optimal path rates in Challenge 1). This process was repeated until all task trajectories were generated.
Finally, ensure data quality, conducted multiple rounds of discussion-based optimization. Several human experts randomly sampled 20% of the data, annotated potential errors, and initiated a review session where annotators collectively resolved issues. This process was repeated with different pairs of experts and different 20% samples each round, continuing until the detected error rate dropped to zero and every data point had been checked at least once with no conflict. After four such iterations, the data quality improved from 62%, 78%, 86%, and 94% to a final 100%, yielding the completed WildtoolBench. human experts took one month to finish the whole data curation process.
D DETAILS OF TooL ORCHESTRATION EVALUATION
Algorithm T shows the pseudo code for enumerating all possible tool orchestration paths. The main design idea this algorithm is to enumerate all possible tool execution paths a directed acyclic graph (DAG) using depth-first search with backtracking. At each step. identifies the set of nodes with zero indegree, generates all non-empty subsets to simulate parallel execution, appends the selected nodes to the current path, and updates the indegrees of their successors. The algorithm recursively explores all possible paths and finally classifies them into optimal and suboptimal sets based on path length, systematically accounting for both serial and parallel execution combinations (mixed multi-tool).
E COMPLETE EXPERIMENTAL RESULTS
E.1
HYPERPARAMETER SETTINGS
To further enhance the reproducibility of our dataset, hereby introduce the hyperparameter settings used during model inference. Specifically:
For Proprietary Models, adopted the default hyperparameters from the official website without making any changes to hyperparameters such as temperature. top-p. and top-k.
For Open-Source Models, if official API is available, we utilize it with its default hyperparameters. Otherwise, the model is deployed via the Hugging Face library, where tool-calling functionality is implemented according its chat template. For generations, its default typerparameters. setting only max new new_tokens to 512. The version of Hugging Face used was 4.51.0, and no other modifications were made.
the model. geder.geeeeate method with
E.2 WILDToou FULL RESULTS
We provide all the benchmarking results of 57 models shown in Table including 16 Proprietary General Models, 30 Open-Source General Models, and Open-Source Specialized Models trained for tool-use.
E.3 WILDTOOLBENCH FULL TooL ORCHESTRATION RESULT
We provide the detailed results on tool orchestration of 57 models as shown in Table 7 including 16 Proprictary General Models, 30 Open-Source General Models, and 11 Open-Source Specialized Models trained for tool-use.
E.4 WILDTOOLBENCH FULL ERROR ANALYSIS
We provide all the detailed error analysis results of 57 models as shown in Table 8 including 16 Proprictary General Models, 30 Open-Source General Models, and Open-Source Specialized Models trained for tool-use.
16

---

Published as a conference paper at ICLR 2026
Table 6: WildToolBench Full Results.
Categorized Task Type y
Categorized by Task Order
Overall
Models
9uingte
Imnutts
Iciorify
9chat
2
Task
Session
Proprictary General Models
Gemini-22 2.0-Thinking
56.64
52.34
78.13
63.67
50.39
61.04
Gemini-2.5..
55.08
36.33
46.88
86.72
70.31
56.64
53.52
44.53
56.25
14.06
Bludee4-Sonnee
60.16
43.75
41.80
80.47
71.09
57.81
52.73
44.53
56.54
12.50
ol
54,30
39.06
05.05
93.75
69.53
60.94
$5.86
48.83
58.79
12.11
GPT-4o
60.16
41.80
78.13
72.66
55.08
46.09
45.70
54.88
11.72
57.81
39.06
41.41
63.28
60.55
50.00
48.05
42.97
50.39
11.33
2 o3
61.72
39,45
44.92
81.64
73.83
60.94
49.22
43.75
56.93
10.16
Grok-4
59.38
41.41
33.59
66.02
63.67
$2.34
42.97
41.41
50.10
10.16
Claude-4.1-Opul
55.86
55.08
54.98
GPT-4.1
57.42
44.14
34.38
81.25
69.53
58,20
46.88
42.58
54,30
8.98
2d. Mistral-Large
56.25
36.33
37.50
68.75
67.58
48.83
44.14
38.28
49.71
7.03
Doubuo-11 1.6
55.86
40.23
31.25
63.28
69.14
48.83
40.23
32.42
47.66
7.03
Doubuo-1 Taabboo--.-Thhikeee 5-Thhiniing
60.16
22.66
26.95
75.39
65.63
47.66
40.23
31.64
46.29
GPT-5
46.09
34.38
31.64
84.38
62.11
50.00
45.31
39.06
49.12
5,86
Doubuo-1 1aapp--1.---1e 6-Thinking
57.42
34,38
18.75
47.27
$7.03
39.06
31.25
30.47
39.45
3.13
Doubuo-1.5
58.59
24.61
9.38
34.38
39.45
28.91
29.30
29.30
31.74
0.78
Open-Source Source General Models
GL.M-4.5
57.81
40.63
44.53
81.25
70.70
60.16
50.78
42.58
56.05
12.11
Kimi-K2
54.30
33.98
39.84
86.72
68.75
$7.03
48.83
40.23
53.71
10.55
Owen3-30B-A3B
48.05
69.92
$1.56
48.05
51.66
Swen3-14B-Thininng
56.64
30.47
37.11
88.67
69.53
54,30
50.39
38.67
53.22
9.38
DeepScck-RI R1
56.25
41.02
43.75
80.08
74.22
$4,30
48.83
43.75
55.27
9.38
DeepSeck-V3
58.98
38.67
33.59
79.30
75.39
53.91
41.02
40.23
52.64
9.38
Dwen3-88-Thhinkin
56.64
DeepScek-V3.1
44.92
40.63
33.98
81.25
61.33
51.56
45.70
42.19
50.20
8.98
19 Qwen3-32B
57.81
33.20
39.84
79.69
69.53
$4,30
46.48
40.23
52.64
8.59
Owen2.5.144B-Insmme
50.39
27.34
32.81
83.20
66.41
48.44
40.63
38.28
7.81
Qwen3-14B
56.25
31.25
71.09
$4,30
$4.10
7.81
Soonn333B-Thinking
53.52
28.91
37.11
80.86
62.50
$2.73
46.48
38.67
50.10
7.81
19 Dwen33-4B-Thinkingg
60.16
28.91
38.67
87.89
68.75
59.38
44.14
43.36
53.91
7.81
Swwen..5772-IInsie
58.98
32.03
34.38
82.81
70.70
50.00
46.48
41.02
$2.05
Qwen3-88
60.16
26.95
48.44
50.39
24.61
38.67
89.45
71.88
49.22
44.14
37.89
50.78
6.25
1 Qwen3-1.7B% 1.nn3--...-Thikiii
49.22
24.61
30.08
84.38
68.75
46.09
41.80
31.64
47.07
6.25
Swwen.55-3B-InnSie
53.91
38.67
36.72
82.81
69.14
$4,30
48.83
39.84
$3.03
5.86
Qwen3-4B
51.17
35.96
49.22
39.06
Qwen2.5-7B.
52.73
25.39
28.91
73.05
64.06
45.31
37.50
33.20
45.02
4,30
19 Qwen2.5.33 Swven..5-3H-Innee
-nn--rrrttttt
48.83
18.36
20.70
73.83
$5.86
41.41
33.59
30.86
40.43
4,30
Qwen3-1.711
48.05
19.53
26.95
81.64
67.58
44.14
33.98
30.47
44.04
4,30
Qwen2.5 Sowen2.5-1.55
5-1_St-Instr
36.72
13.28
39.06
1.eem33-06B-Thinkiii
44.92
16.02
23.05
87.11
66.02
40.23
35.55
29.30
42.77
3.91
1 Qwe63-0.61
46.09
7.81
12.11
78.13
53.91
34.38
26.56
29.30
36.04
3.52
Qwen2.5
23.83
4.69
15.23
82.03
43.75
29.69
26.95
25.39
31.45
0.78
Llama-3.3-3P 3.3tuut-nsst
0.00
17.19
16.31
0.00
0.00
0.39
61.33
17.58
18.75
14.06
11.33
15.43
0.39
Ilama-3 Llama-3.3-7005 1.3-70B--Instrun
3.52
0.00
0.00
$2.73
17.97
14.06
12.89
11.33
14.06
0.00
(.nmaa-,,3-88--meee
0.39
0.00
0.39
69.14
25.39
19.14
14.06
11.33
17.48
0.00
Open- Spenn-Suuce Specialized Models
XLLAM-2-32B
60.94
34.77
38.67
69.92
69.53
$2.73
43.36
38.67
51.07
8,20
XLAM-2-70B
64.45
36.72
28.91
64.84
64.06
51.56
42.58
36.72
48.73
TooLACE2-8B
62.11
37.89
33.98
84.38
72.27
59.38
39.84
54.59
7.42
XLAM-228B
62.11
29.30
24.22
54.30
52.73
44.14
39.06
33.98
42.48
5.08
Watt-8B
61.72
28.13
22.66
78.13
68.75
47.27
39.06
35.55
47 66
4.69
Hammer2.1-78
40.23
21.88
30.47
94.92
61.72
46.88
40.63
38.28
46.88
4.69
XLAM-2-IB
53.13
17.19
15.23
65.23
50.39
40.63
25.78
37.70
XLAM-2-3B
52.34
23.44
16.02
57.03
42.97
43.36
34.77
27.73
37.21
1.17
Bammer2.113B
32.81
18.36
11.33
91.02
34.77
46.48
40.63
31.64
38.38
0.39
Hammer2.1-1.55
4.69
1.56
1.17
96.09
26.17
26.17
26.56
24.61
25.88
Hummer2.1-00
1-0.5B
9.77
2.34
3.13
86.72
25.39
29.69
25.39
21.48
25.49
0.00
17

---

Published as a conference paper at ICLR 2026
Table 7: WildToolBench Tool Orchestration Result.
Task Accuracy
AP Rate
OP Rate
Models
Pumli
S+P Funlts
Overall
Fonnts
S+P Fmalti
Overall
Omalti
S=P
Overall
Proprictary General Models
G Gemini-2.0-Thin 2.0-Thinking
54.14
25.00
16.67
40.23
45,28
39.89
40.37
53.50
16.67
40,66
Gemini-2.5-Pa.
49.04
25.00
14.29
36,33
47.17
39.15
39.87
43.31
11.90
32.37
Claude-3. 3.7Soneet
43.95
62.50
25.00
39.06
86.79
61.40
63.65
0.00
0.00
0..00
1toude--SSonet
54.78
31.25
25.00
43.75
60.38
46.32
47.57
$2.87
23.81
42.74
A Claude-4, 1-pus
50.96
43.75
17.86
39.84
62.26
48.35
49.58
50.32
16.67
38.59
50.96
12.50
21.43
39.06
35.85
37.50
37.35
50.32
20.24
39.83
48.41
31.25
23.81
39.45
66.04
54.60
55.61
0.64
0.00
o4-mini
39.49
31.25
16.67
31,64
52.83
37.68
39.03
0.00
0.00
1 GPT-4o
53,50
31.25
21.43
41.80
41.51
45.40
45.06
$1.59
21.43
41.08
3 GPT-4.1
58.60
25.00
20.24
44.14
49.06
45.77
46.06
$6.69
19.05
43,57
GPT-5
43.31
37.50
16.67
34.38
49.06
38.42
39.36
68
13.10
32.37
Grok-4
54.14
18.75
21.43
46.06
$3.50
1 Mistral-Large
47.77
25.00
16.67
36.33
45.28
40.44
40.87
45.85 86
15.48
35.27
Douban- 1.5
35.03
12.50
7.14
24.61
37.74
29.41
30.15
9.55
1.19
6.64
Doubao-1 1.5 T--Thinking
31.21
18.75
7.14
22.66
56.60
23.35
26.30
28.03
7.14
20.75
Doubao-1
50.96
25.00
22.62
40.23
50.94
47.79
48.07
50.96
22.62
Doubao- 66-Thinking
46.50
12.50
15.48
34.38
52.83
39.34
40.54
43.95
15.48
34.02
Open-Source General Models
XLAAM-2-70B
49.68
12.50
16.67
36.72
43.40
44.85
44.72
26.75
7.14
XLAM-2-32B
45.86
25.00
15.48
34.77
58.49
40.26
41.88
25.48
5.95
18.67
XLAM-2-8B
40.76
25.00
8.33
29.30
43.40
27.57
28.98
26.75
3.57
18.67
XLAM-2-33
33.12
12.50
7.14
23.44
24.53
23.35
23.45
15.92
3.57
11.62
XLAM-2-IB
27.39
0.00
1.19
22.64
17.46
17.92
10.83
0.00
ToolACE2-8B
47.77
31.25
20.24
37.89
50.94
43.01
43.72
26.11
14.29
21.99
Watt-BB
44.59
6.25
1.19
28.13
22.64
21.87
21.94
44.59
1.19
29.46
Hammer2.11 1-77B
33.12
12.50
2.38
21.88
24.53
13.24
14.24
31.85
2.38
21.58
Hammer2.
24.84
12.50
7.14
18.36
32.08
18.01
19.26
24.20
7.14
18.26
Hammer2.1- 1-1.5B
2.55
0.00
0.00
1.56
1.89
1.29
1.34
2.55
0.00
1.66
1ammer2.110.50
3.18
0.00
1.19
2.34
9.43
10.48
10.39
3.18
0.00
2.07
Open-Source Specialized Models
09 Llama-3.3-70B. -nnstruett
0.00
0.00
0.00
0.00
15.09
2.02
3.18
0.00
0.00
0.00
09 Llama-3.3-8B-Innse Instructtt
0.00
0.00
0.00
0.00
3.77
0.18
0.50
0.00
0.00
0.00
Llama-3.3-33 3.0--33..-B--sttmee
0.00
0.00
0.00
0.00
3.77
0.18
0.50
0.00
0.00
Llama-3.3-11 3. Instruct
0.00
0.00
0.00
0.50
0.00
0.00
6 00 IInstrcttt
44.59
25.00
9.52
32.03
41.51
30.33
31.32
42.04
7.14
29.88
15 Qwen2.5-32B IIsttrcct
52.87
43.75
10.71
38.67
56.60
24.08
26.97
$2.23
10.71
37.76
5 Qwen2.5 1-4--55--8----- IIstrucct
39.49
18.75
5.95
27.34
26.42
19.67
20.27
36.31
4.76
25.31
Owwen2557BBInne
38.22
6.25
4.76
26.47
33.12
69 -ww.n.5-3B-IIsiiie
28.66
6.25
1.19
18.36
15.09
12.13
12.40
27.39
0.00
17.84
3 Qwen2.5-1 1.5.5-11.B-nnttreee
21.66
0.00
0.00
13.28
9.43
4.96
5.36
21.66
0.00
14.11
1 Qwen2.5-0.55
7.64
0.00
0.00
4,69
11.32
3.68
4.36
7.64
0.00
4,98
Owwen-33B-AA0)
42.04
6.25
5.95
25.00
25.13
40.76
5.95
6 Qwen3-32B
46.50
12.50
11,90
33.20
47.17
30.70
36
43.95
9.52
31.95
1 Q-en 14B
44,59
25.00
7.14
31.25
41.51
28.86
29.98
44.59
7.14
31.54
1 Qwen3-8B
38.85
0.00
7.14
26.17
15.09
25.18
24.29
38.22
5.95
26.97
Qwen3-4B
35.67
6.25
1.19
34.39
1.19
8 Qwen3-1.77 1.7B
31.85
0.00
0.00
19.53
18.87
16.91
17.09
31.85
0.00
20.75
0 Qwen3-0.66 3-0.6B
12.10
6.25
0.00
7.81
22.64
9.19
10.39
11.47
0.00
7.47
0 Qwen3-300 1-----3----3--hinlee
36.94
6.25
4.76
24.61
18.87
22.43
22.11
35.67
3.57
24.48
3wwen-330B-hhme Thinkknnnn
42.04
12.50
7.14
28.91
41.51
28.31
29.45
40.13
7.14
28.63
9 Qwen 14h-Thiiing
43.95
5,95
30.47
45.28
31.62
32.83
43.95
5.95
30.71
4 Qwen3-SB-Thinke
47.13
12.50
11.90
33.59
28.30
31.99
31.66
47.13
10.71
34.44
1 Quen33-4--Thinking
42.68
18.75
4.76
28.91
39.62
23.16
24.62
42.68
4.76
29.46
10ee3--..B-Tbinkiie
38.85
6.25
1.19
24.61
26.42
11.40
12.73
38.85
1.19
25.73
9 Qwen3-0.6B-Thinnee
26.11
0.00
0.00
16.02
18.87
9.74
10.55
25.48
0.00
16.60
$ G-.-4.5
51.59
31.25
21.43
40.63
67.92
48.90
50,59
49.68
20.24
39.42
M. Kimi-K2
45.86
12.50
15.48
33.98
52.83
34.93
36.52
43.95
15.48
34.02
DeepSock-R1
53,50
18.75
21.43
41.02
41.51
44.12
43.89
$2.87
20.24
41.49
DeepScck-V3
52.87
25.00
14.29
43.40
32.54
33.50
$1.59
14.29
38.59
or DeepSeck-V3.1
53.50
25.00
19.05
40.63
52.83
37.68
39.03
47.77
14.29
36.10
18

---

Published as a conference paper at ICLR 2026
Table WildToolBench Full Error Distribution Analysis.
Models
Action Errors
Parameter Errors
Refusal
Wrong Name Missing Info
Wrong Refusal
Redundant Call
Call Erroe
Early Termination
Param Type Eror
Param Hallucination
Value Error
Param
Proprictary
Gemini-2 O-Thinking
24.56%
8.02%
3.26%
23.06%
18.05%
4.76%
1.50%
4.51%
12.28%
G Gemini-2 2.5-222-PP S-Pro
33.93%
7,81%
3.79%
16.
14.51%
5.13%
1.12%
6.47%
10.495
Clinde-3.7-Son 3.7-Smmet
11.02%
16.73%
17.91%
16.34%
16.54%
1.57%
6.30%
12.01%
Claude-4-Snmet
831%
Clinde-411 1-pppe
15.18%
17.79%
9.76%
18.22%
12.15%
3.25%
2.17%
8.46%
13.02%
@ ol H od
30.57%
8.53%
3.55%
21.33%
8.77%
8.00%
1.42%
6.4%%
11.37%
10.66%
17.46%
9.98%
13.15%
17.01%
4.31%
1.30%
10.43%
15.65%
@ GPT-40
5,41%
21.68%
12.12%
14.50%
11.26%
7.38%
2.50%
10.2%
13 83.8%%
@ GPT-4.1
11.97%
21,58%
8.55%
18.90%
9.83%
6.41%
1.71%
9.62%
11 11.54%
① GPT-5
15,93%
13.05%
6.91%
31.67%
10.17%
3.65%
1.15%
10,94%
6.53%
Grok-1
Mistral-Large
8,93%
24.08%
15.34%
8.10%
14.76%
5.05%
1.94%
8.16%
13.40%
D Doubao-1 1.5
15.16%
31.47%
21.32%
2.29%
11.30%
2.72%
1.72%
8.44%
5.58%
0 Doubau- 1.5- 1.5-Thinking
17.27%
27.64%
11.45%
5.82%
13.09%
1.36%
1.82%
11.45%
7.09%
25.37%
Doubao-Lo Thinking
3.87%
30.97%
21.77%
16.77%
8.87%
1.68%
0.97%
6.13%
5.97%
Open Source General Models
09 Jama-3333--20 3.0--..------ntttrr
62.27%
4.32%
5.11%
19.66%
0.11%
0.34%
1lama-5.3--00
0.00%
09 1llma-3.3330 Enstruct
78.30%
0.00%
0.00%
21.70%
0.00%
0.00%
0.00%
0.00%
0.00%
09 Liama 3.3- Eastust
80.14%
0.00%
0.00%
19.86%
0.00%
0.00%
0.00%
0.00%
0.00%
3 Owen2 5. Instrrcc
12.42%
21.59%
8.90%
15.00%
12.02%
8.76%
2.24%
7.54%
11.20%
Owen2
Owen2 Instractt
17.42%
15.91%
8.14%
19,9%
12.50%
8.71%
2.08%
6.25%
9.09%
15 Owen2 S. Instract
11.01%
21.49%
12,204
11.61
20.96%
5.51%
1.44%
6.75%
7.82%
1 Owen2 5 3B Instract
14.92%
20.66%
10.0%
13.28%
18.36%
6.23%
5.25%
9.18%
Owen2
15.57%
16.07%
Qwnn2 5B-Sstrtccccc
27.92%
11,82%
6.59%
20223%
141.6%%
2.99%
1.00%
6.81%
5.70%
1 Owen3: 32B
10.72%
19,59%
202114
16.91%
5.57%
1.24%
6.50%
8.66%
15 Owen3: 10em3-30B-A33
16.77%
14.95%
5.15%
30.10%
14.31%
1.65%
1.41%
4.65%
7.38%
Qwen3-888
8.79%
27.31%
9.00%
6.39%
19.40%%
7.53%
1 1.69%
9.98%
9.42%
15 -wwn3-33
15.71%
15.13%
6.51%
11.37%
18.01%
6.90%
1.31%
11.69%
10.34%
15 Qwan3- L.7
14.49%
21.12%
8.03%
9.42%
24.08%
5.24%
1.05%
6.63%
9.95%
Qwen3-0.B
29.37%
14.48%
1.79%
15 Qwen3-14B 3-1---1-1----0 14B-Thinking
15.03%
17.12%
5.64%
1983%
13.99%
5.85%
1.88%
10.23%
10.23%
15 Qwen 3- SiB-Thinkin
11.32%
19.44%
6.84%.
14.10%
17.52%
6.84%
2.14%
10.47%
11.32%,
V. Dowem3-4B-Thinkii
12.71%
6.14%
11.23%.
Qwen3-1 3.B-TTiini
15.31%
20.66%
7.20%
12.73%
16.05%
7.01%
2.03%
7.20%
11.44%
15 (B-om---nBBThThkkn
18.26%
19.45%
3.63%
15.70%
25.04%
4.78%
1.02%
4.11%
7.51%
4 Gi M-4.5
10.89%
19.33%
10.67%
18.89%
15.33%
6.00%
1.11%
4.8%
12.89%.
K Kimi-K2 K2
21.31%
13.50%
7.17%
16.24%
11.60%
6.54%
2.53%
6.90%
14.14%.
DeepSock-R1
13.54%
14.41%
11.14%
20.96%
11.79%
1.31%
or DeepSeck-V3
10.52%
21.63%
10.9%%
15.88%
16.40%
5.15%
1.44%
7.63%
10.31%
or DeepSeck-V.. V3.1
20.00%
11.70%
9.41%
25.29%
13.92%
3.73%
0.98%
5 5.69%
9.22%
Open-Source Specialioed Models
<LAMM-277B
6.48%
30.67%
17.14%
4.38%
16.19%
3.71%
0.95%
12.57%
x1-AM-2-178
8.9%%
21.76%
15.37%
8.38%
18.56%
4.19%
1.60%%
5. 3.30%
15.77%.
x1 <1AAM-2-88
5.60%
27.39%
19.86%
3.00%
19.52%
5.09%
2.21%
4.92%.
11.21%.
<LAM-2-38
21.00%
XLAM-22118
9.40%
24.61%
13.95%
22.57%
1.8%%
3.61%
10.65%
RooAACE--BB
10.11%
28.60%
8.00%
6.67%
18.29%
6.02%
2.80%
4.0%%
14.82%
Walt-BB
5.97%
30.97%
10.45%
7.09%
23.13%
4.29%
1.49%
6.16%
1045%
4.41%
Hommer2.1-30
Hammer2.1-1.50
60.47%
3.20%
1.32%
32.15%
1.32%
0.26%
0.00%
0.26%
0.92%
30.80%
17.69%
4.46%
22.15%
13.11%
2.75%
0.13%
6.82%
1.97%
19

---

Published as a conference paper at ICLR 2026
Algorithm Enumeration of All Serial and Parallel Execution Paths
Require: Directed acyclic graph G E); annotated length L
Ensure: All paths P, divided into optimal and suboptimal sets
1: Compute indegree(u) for all E V
2: Initialize visited <- false for all w E
CurrentPath < 0, P PP-
4: function ZEROINDEOREE(indggr,, visited)
5: return {u V indegree uj A -visiteduu]]
6: end function
7: function COMBINATIONS(22
8: return all non-empty subsets of Z
9: end function
10: procedure DFS(indegree, visited, CurrentPath)
11:
Z 4- ZEROLNDEGOREEindeere, visited)
12: if Z i then |urreetPaah]] = |V| V CurrentPath] = then Add copy of CurrentPath to P end if return
13:
14:
15:
17: end if
for all C' (OMBINATIONS( do
19:
Backup indegree, visited, CurrentPath Append C to CurrentPath
21:
Mark all v E C' as visited
22:
for all edges (v, u) with v E do indegree(u) -
indegree(uj end for DFS(indegree, visited, CurrentPath) Restore backup 27: end for
23:
24:
25:
26:
28: end procedure
DFS(indegree, visited, CurrentPath)
30: L t- min P
Papt f- P: Ip]
32: Paub + T Popt
33: return P,Popts Paub
PROMPTS
F.I
PROMPT FOR SINGLE-TOOL CALLS SEED TASK GENERATION
We show the role prompt of the single-tool calls task generation in Figure
F.2 PROMPT FOR SEQUENTIAL MULm-TToL CALLS SEED TASK GENERATION
We show the role prompt of sequential multi-tool calls task generation in Figure 10
F33 PROMPT FOR PARALLEL MULn-TOoL CALLS SEED TASK GENERATION
We show the role prompt of parallel multi-tool calls task generation in Figure
F.4 PROMPT FOR MIXED MULT--ToL CALLS SEED TASK GENERATION
We show the role prompt of mixed multi-tool calls task generation in Figure 12
20

---

Published as a conference paper at ICLR 2026
F.5 PROMPT FOR CLARIFY SEED TASK GENERATION
We show the role prompt of the clarify task generation in Figure 13
F.6 PROMPT FOR CHAT SEED TASK GENERATION
We show the role prompt of chat task generation in Figure
F.7 PROMPT FOR CONTEXT SEED TASK GENERATION
We show the role prompt of context task generation in Figure 15 and Figure T6
G
ERROR CASES
Figure presents several typical enror examples discussed in the main text.
21

---

Published as a conference paper at ICLR 2026
Single Moole-Toool Calls task Generation Prompt.
Please act as user interacting with super intelligent agent.
This super intelligent agent has access range of external tools and can use these tools to solve the tasks you propose.
Next, please propose tasks that you need the super intelligent agent solve based on the [Requirements].
All 5 tasks must require the use of {{(tool))) from the [Tool List] to be completed, and each task should only require a single call to {{{too())).
The tasks should be specific and diverse.
Finally, please output the final result according to the [Format] without generating any extra text.
The required parameters for tool {f[tool}) are: [[[tool_required] } and the optional parameters are: {[(tool_no cecttoll_orrree he_roorrrree]]]]
[Requirements]=""" -."
The description of the user's task must include information on all the required parameters needed to call { }. For other optional parameters, please add them as you see fit, using natural language.
2. The user's tasks should use different types of sentence structures: imperative, declarative, interrogative, etc.
3. The user's tasks should include different tones: colloquial, formal, polite, direct, etc. Ensure that the length of the user's tasks varies, gradually increasing from short to long.
5. Ensure that the user's tasks involve different themes/instaaness, different scenarios, and different roles.
6. Extract common entities that appear in all descriptions from the [Tool List] and ensure that these entities appear in the user's tasks. Do not explicitly specify the tool {{{lool)) in the user's tasks.
Tool List]=""" [[(too)))}
[Forma]]
"task 1": "xxx".
"task 2": "xxx".
"task 3": "xxx".
"task 4": "xxx".
"task 5": "xxx".
Figure 9: Single-Tool Calls task Generation Prompt.
22

---

Published as a conference paper at ICLR 2026
Sequential Multi-Tool Calls task Generation Prompt.
Please act user interacting with super intelligent agent.
This super intelligent agent has access to range of external tools and can use these tools to solve the tasks you propose.
Next, based on the [Requirements], please propose tasks that you need the super intelligent agent to solve.
These tasks must require the combined use of tools from the [Tool List] (including: [[[al_ttol_name]] }) to be completed.
The tasks should be specific, diverse, and require the sequential invocation of multiple tools to solve.
Finally, please output the final result according to the [Format] without generating any extra text.
{ (oouioooorquiire_iinn)))
[Requirements]="" -".
The description of the user's task must include all the required parameters needed to invoke the tools, while other optional parameters can be added as you see fit, using natural language. The user's tasks should use different types of sentence structures: imperative, declarative, interrogative, etc.
3. The user's 's tasks should include different tones: colloquial, formal, polite, direct, etc.
4. Ensure that the length of the user's tasks varies, from short to long, gradually increasing in length.
5. Ensure that the user's tasks involve different themes/instances, different scenarios, and different roles.
Based the descriptions of all tools in the [Tool List], extract the common entities that appear in all descriptions and ensure that these entities appear in the user's s tasks.
There must dependencies between the multiple tools invoked, meaning that tool must be called and completed before tool B can be run, i.e., tool В must be invoked after tool A. The difficulty of the tasks is divided into easy, medium, and hard levels. Easy represents simple, medium represents moderate, and hard represents difficult. Ensure that the 5 tasks you generate are all of medium difficulty or above.
9. Do not explicitly specify the names of the tools to be used in the user's tasks. nPm
Tool List]=""" ((oolss)]
[Format]="""
"task 1": "xxx",
"task 2": "xxx",
"task 3": "xxx".
"task 4": "xxx",
"task 5"; "xxx",
Figure 10: Sequential Multi-Tool Calls task Generation Prompt.
23

---

Published as a conference paper at ICLR 2026
Parallel Multi-Tool Calls task Generation Prompt.
Please
interacting with super intelligent agent.
This super intelligent agent has access to range of external tools and can use these tools to solve the tasks you propose.
Next, based on the [Requirements], please propose tasks that you need the super intelligent agent solve.
These 5 tasks must require the combined use of tools from the [Tool List] (including: { [[all_ttol_namm]] }) be completed.
The tasks need to be specific, diverse, and require parallel invocation of multiple tools to solve.
Finally, please output the final result according to the [Format] without generating any extra text.
{
[Requirements)="""
The description the user's task must include the required parameters needed to invoke the tools, while other optional parameters can be added you fit, using natural language. 2. The user's tasks should use different types of sentence structures: imperative, declarative, interrogative, etc.
3. The user's ss tasks should include different tones: colloquial, formal, polite. direct, etc. Ensure that the length of the user's tasks varies, from short to long, gradually increasing in length.
Ensure that the user's tasks involve different themes/instances, different scenarios, and different roles.
Based on the descriptions of all tools in the [Tool List], extract the common entities that appear in all descriptions and ensure that these entities appear in the user's tasks.
There must no dependency between the multiple tools invoked. A dependency between invocations means that tool B can only be run after tool A is completed. No dependency means that tool A and tool can be invoked parallel.
8. The difficulty of the tasks is divided into easy, medium, and hard levels. Easy represents simple, medium represents moderate, and hard represents difficult. Ensure that the 5 tasks you generate are all of medium difficulty or above.
9. Do not explicitly specify the names of the tools to be used in the user's tasks.
[Tool List]=""" T"M
{[(oools)
[Format]=""" ""M
"task 1": "xxx".
"task 2": "xxx".
"task 3": "xxx".
"task 4"; "xxx".
"task 5"; "xxx",
Figure 11: Parallel Multi-Tool Calls task Generation Prompt.
24

---

Published as a conference paper at ICLR 2026
Mixed Multi-Tool Calls task Generation Prompt.
Please act as user interacting with super intelligent agent.
This super intelligent agent has access to range of external tools and can use these tools to solve the tasks you propose.
Next, based on the [Requirements], please propose tasks that you need the super intelligent agent solve.
These 5 tasks must require the combined use of tools from the [Tool List] (including: { [[aal_ttoo_nmmm]] to be completed.
The tasks should be specific, diverse, and require both serial and parallel invocation of multiple tools to solve.
Finally, please output the final result according to the [Format] without generating any extra text.
{I Lol_itooo_rquued_initieet
[Requirements]=" -"*
1. The description of the user's task must include all the required parameters needed to invoke the tools, while other optional parameters can be added as you see fit, using natural language.
2. The user's tasks should use different types of sentence structures: imperative, declarative, interrogative, etc.
3. The user's tasks should include different tones: colloquial, formal, polite, direct, etc. Ensure that the length of the user's tasks varies, from short to long, gradually increasing in length.
5. Ensure that the user's tasks involve different themes/instances, different scenarios, and different roles.
6. Based on the descriptions of all tools in the [Tool List], extract the common entities that appear in all descriptions and ensure that these entities appear in the user's tasks.
7. There should be dependencies between some of the tools invoked, while others should not have dependencies. A dependency between invocations means that tool B can only be run after tool A is completed. No dependency means that tool A and tool B can be invoked in parallel.
8. The difficulty of the tasks is divided into easy, medium, and hard levels. Easy represents simple, medium represents moderate, and hard represents difficult. Ensure that the 5 tasks you generate all of medium difficulty above.
9. Do not explicitly specify the names of the tools to used in the user's tasks.
[Tool List]=""" """ [[(oolls) mie
[Format]=""'
"task 1": "xxx",
"task 2": "xxx".
"task 3": "xxx".
"task 4": "xxx",
"task 5"; "xxx".
Figure 12: Mixed Multi-Too Mooli-Tooloooo Calls task Generation Prompt.
25

---

Published as a conference paper at ICLR 2026
Clarify task Generation Prompt.
Please act as user interacting with super intelligent agent.
This super intelligent agent has access range of external tools and can use these tools to solve the tasks you propose.
Next, based on the [Requirements], please propose tasks that you need the super intelligent agent to solve.
These tasks must require the combined use of tools from the [Tool List] (including: }) to be completed.
All 5 tasks must require the use of {i [tool]] } from the [Tool List] be completed, but will leave the super intelligent agent unclear on how to fill in some of the required parameters of {{ [tool)] ]] ), and should be diverse.
Finally, please output the final result according to the [Format] without generating any extra text.
The required parameters for tool [((tool)) } are: ([(ooo_requiree) }}, and the optional parameters are: [[ [[(loo__o_rquurrrd)))
[Requirements]=" -...
The description of the user's task must lack all the necessary information for calling II [[ttol]) }, leaving only the optional parameter information, which you can add you see fit, using natural language descriptions. Note that tool parameters allow for some parameter inference, meaning that if the tool parameters be inferred from the user's task description, it does not count as lacking necessary information. Lacking means that even through inference. the parameter values cannot be obtained.
2. The user's tasks need to use different types of sentence structures: imperative sentences, declarative sentences, interrogative sentences, etc.
3. The user's s tasks should include different tones: colloquial, formal, polite, direct, etc. Ensure that the length of the user's tasks varies, from short long, gradually increasing in length.
5. Ensure that the user's tasks involve different themes/instances, different scenarios, and different roles.
Based on the descriptions of tools the [Tool List], extract the common entities that appear in all descriptions and ensure that these entities appear in the user's tasks.
Task difficulty is divided into casy, medium, and hard levels. Easy represents simple, medium represents moderate, and hard represents difficult. More difficult tasks require more steps to execute. Ensure that the tasks you generate are all medium difficulty or above. 8. Do not explicitly specify the tool {{ [[tooo]] }} in the user's tasks. mne
Tool List]=""" [[ [
[Format]=""
"task 1": "xxx".
"task 2": "xxx".
"task 3": "xxx".
"task 4": "xxx".
"task 5": "xxx",
Figure 13: Clarify task Generation Prompt.
26

---

Published as a conference paper at ICLR 2026
Chat task Generation Prompt.
Please act as user interacting with super intelligent agent.
This super intelligent agent has access range of external tools and can use these tools to solve the tasks you propose.
Next, based on the [Requirements), propose 5 casual conversation tasks that you need the super-intelligent agent to solve.
These 5 casual conversation tasks should not use any tools from the [Tool List], but should have some thematic relevance.
Finally, please output the final result according to the [Format] without generating any extra text.
The required parameters for tool [[(tool] are: [f[tool_requiee }}, and the optional parameters are: [[toool_o_required]]]
[Requirements]="""
1. The user task is a casual conversation task, which must be unrelated to the functions of the [Tool List], but should have some thematic relevance.
2. User tasks need to use different types of sentence structures: imperative, declarative, interrogative, etc.
3. User tasks should include different tones: colloquial, formal, polite, direct, etc. Ensure that the lengths of the user tasks are different, ranging from short to long, with gradually increasing length.
5. Ensure that the user tasks involve different themes/examples, different scenarios, and different role identities. mnm
Tooo List]=""" {(tools)))
[Format]="""
"task 1": "xxx".
"task 2": "xxx".
"task 3"; "xxx".
"task 4": "xxx".
"task 5": "xxx",
Figure 14; Chat task Generation Prompt.
27

---

Published as a conference paper at ICLR 2026
Context task Generation Prompt, Part 1.
Please act as user interacting with super intelligent agent.
This super intelligent agent has Planner, an Agent assistant, and range external tools that can be used to solve the tasks you propose, as detailed in the [Tool List].
Based on the information in [Historical Conversations), you have already proposed your task, and the super intelligent agent has solved it for you.
Therefore, next, please continue to propose new tasks based on the reply from the Agent assistant in the round of [Historical Conversations], referring to the [TTurn Type Information] and [Example], and the new tasks you propose must require the use of { (omobrrmm)rr]] } tool from the [Tool List] to solve.
Finally, output according to the [Format].
[all|_tool reqqurrd info]
[Tool List]=""" """
[[[tools]]]
[Turn Type Information)=""" [[urn type [
Figure 15: Context task Generation Prompt, Part
28

---

Published as a conference paper at ICLR 2026
Context task Generation Prompt, Part 2.
When actually generating tasks, one of the following types will be substituted into the prompt placeholder (1 (atp__tt____fo]]]]
Partial Information: The new task generated needs to omit some content from previous conversations, without having to state the full semanties. The omitted content be any sentence component, including: subject, attribute, attrihute value, modifier, etc.
2. Coreferential Reference: The new task generated requires reference some content from previous conversations, which can be: 1) Ordinal reference, such as: the second point, the last point, etc. 2) Pronominal reference, such as: he, this sentence, which one, etc. 3) Vague reference, such as: xxx this model, etc.
Long-Range dependency: The new task generated needs use content from previous conversations (excluding the last round), for example, something mentioned in the first round, something mentioned before.
[Example]=""" [Historical Conversations)=*** ([history)))
[Output]=****
$##
[Historical Conversations)="""
[history) }}
[ormat]="""
"task 1": "xxx",
"task 2": "xxx",
"task 3": "xxx".
"task 4": "xxx",
"task 5"; "xxx",
Figure 16: Context task Generation Prompt, Part 2.
29

---

Published as a conference paper at ICLR 2026
yanarday.
round re
Yask 3.2: Corgforental Retinosce
Arteel
Actie:: Tullt Lnmoodal www.aaaaaaaaaaaooaaooooo Cilvetorg
The LBS auraeiic aeficaa lo ascen
News bhom IBBC (Deccrsee 2104)
Acties:
X
ausoeac ochiced be Ca Loon meae for Lottod anvena.reedd
CTimata Ehange bew.aatrryyceeeeee..........o combeeeeeiict17799
seartphocelligg
Usef
Fullisaion 23, 2004
Vesienaa
Informstiinn
Contore etsesus seabysi
v
Makraise the ineetion to theck the und unc the uoda ansadil
Usar
kaaw about attiiies the sulhan those utiiies
mxtadira eriiles Alio,
Tark Lone Ranze
Depordotry
Actiee::
What information would you lhe sugure obout yeaterday's tiSte??
Inieet dait occess tecamecty 2EERT
X
Audeen da Acsaue
Andben Tol Dac Decether
News fuon ESPPeDeerreer 23, 2005)
the NBA Caraiteaa Uhe garnes are upprecciglltppee/ee /wwwe cipe.oomiitaal htw.oossot
Anadlvee NFE plapoff anattonhhpp epp.ommatti affi reabiiyy
Actiee:
rago
cod.alaia the es ther'ss bosed tie cacculicn sonalla, and qaery thia
oh fins mends antidles
vest to chiui the uxcadaats of oue theas
Yaok 1: Coralivential Ralonesiz
Clanify
Coeld you plase provide me
specitc amile you intemeeed
indoraatio ussatall alle eu zares condes
lic search for
X
Figure 17: Typical error examples discussed the main text
30


Reviews:
[{"review_id": "cgZjXUUIdq", "reviewer": "Reviewer_Cyfy", "summary": "The paper presents WildToolBench, a benchmark designed to evaluate LLMs’ real-world tool-use abilities. Built from 256 human-verified scenarios reflecting authentic user behaviors—compositional tasks, implicit intents, and instruction transitions—it exposes key weaknesses in current LLMs’ reasoning and planning. Benchmarking 57 models shows none exceed 15% session accuracy, revealing significant gaps in robustness and adaptability.", "questions": "1. In Appendix C, the paper mentions that behavior patterns were summarized by analyzing real user logs — does this imply a potential information bottleneck, where instead of fully leveraging authentic user dialogues and scenarios, the authors mainly relied on manual summarization combined with LLM synthesis?\n\n2. The same appendix states that human experts conducted four iterative rounds to raise data quality from 62% to 100%. How exactly was this data quality measured, and what were the main issues identified during the intermediate stages?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper is highly original in redefining LLM tool-use benchmarking through real-world user behavior modeling, capturing compositionality, implicit intent, and instruction transitions that prior datasets overlook.\n\n2. The study demonstrates strong methodological rigor and clear exposition, combining large-scale human-verified data with systematic evaluation of 57 models to yield reproducible and insightful findings on LLM agent robustness.", "weaknesses": "1. Heavy reliance on human-verified curation (256 scenarios / 1,024 tasks) constrains scale and may bias domain coverage; moreover, this limitation implies that such a data-construction pipeline is difficult to extend for model training or optimization purposes.\n\n2. The evaluation emphasizes function-call correctness and orchestration (OP/AP) but underplays live constraints—API latency, rate limits, outages, and cost; adding live (or high-fidelity simulated) tool execution with cost/latency/robustness metrics and failure-injection studies would better reflect real deployment trade-offs and stress agent reliability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wZCA6HoAXP", "reviewer": "Reviewer_6GJ2", "summary": "This paper introduces WildToolBench, a benchmark designed to evaluate Large Language Models' (LLMs) ability to effectively use tools in real-world, multi-turn, multi-step dialogue settings. Unlike existing benchmarks that focus on artificially complex tasks, WildToolBench addresses the complexity of user behavior in real-world interactions. The authors identify three key challenges that current LLM tool-use capabilities fail to handle effectively: compositional tasks that demand complex tool orchestration, implicit intention spread across dialogue turns, and frequent transitions between task queries, clarifications, and casual conversation. The paper benchmarks 57 LLMs, revealing that none of the models achieve higher than 15% accuracy in session completions, underscoring a significant gap in LLMs' agentic capabilities. The experiments and analysis demonstrate that LLMs struggle not with artificially difficult tasks, but with the \"wild\" nature of real user interactions.", "questions": "referred to weakness paragraph.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "By focusing on multi-turn, multi-step tool-use that mimics actual user interactions, the benchmark is highly relevant to practical applications of LLMs in domains requiring tool integration. The insights provided from this evaluation have the potential to inform the development of more robust, user-aware AI systems. Meanwhile, this paper effectively highlights the specific areas where LLMs currently struggle, such as handling compositional tasks, inferring implicit user intentions, and adapting to frequent task transitions. These findings are essential for guiding future model development and addressing fundamental weaknesses in tool-use.", "weaknesses": "Although the paper introduces a useful benchmark, there is a lack of transparency regarding the usage api/document. The author didn't provide any scratch version of the benchmark, which limits the reproducibility and potential impact of the paper on the community.\nAdditionally, a significant portion of the benchmark creation was done manually, which limits its scalability and increases costs. The current manual curation process is resource-intensive, making it difficult to expand the dataset further without significant additional effort.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z9AGYS6oTx", "reviewer": "Reviewer_E7mX", "summary": "This work proposes WildToolBench -- a tool-calling evaluation dataset that simulates real-user behavior when interacting with tool-calling models. This work identifies three user characteristics that LLMs find challenging, namely single tool-use, inferring information and intent from multi-turn dialog, and switching intents on the fly. This work creates 256 scenarios, with each scenario containing 4 tasks for a total of 1024 tasks. Evaluation of 57 models finds that the best performance on a session is 15%, indicating that the benchmark is hard to solve for current models.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The underlying motivation to build a dataset mimicking realistic user conversations is a meaningful contribution. Results showing that models struggle greatly to fulfill user requirements in a session demonstrate the utility of this dataset as a good benchmark.\n- Extensive experimentation covering 57 models, ranging from proprietary models to open-source general and function-calling specific models.", "weaknesses": "- I found the process of data curation hard to understand from the main text. The paper would benefit from reordering of the content by moving sections describing data curation from the Appendix into the main text\n- Similarly, adding a section comparing the WildToolBench and other function-calling benchmarks to show how the other benchmarks fail to capture the cases that WildToolBench does will help set the motivation.\n- Section 3.2 mentions that the first step to data curation is analyzing large-scale real user logs, but no information is provided about the source or the characteristics of these user logs. Similarly, the 1600 APIs collected are also not described.\n- Section 4.2 title should most likely read \"LLMs Perform Poorly on Tool Orchestration\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)