OrchestrationBench: LLM-Driven Agentic Planning and Tool Use in Multi-Domain Scenarios
Authors: Aelim Ahn
Sooyeon Lee
Hyosun Wang
Chiwan Park
Daeryong Kim
Jihyeon Roh
Kichang Yang
Wonjun Jang
Hwang Woosung
Min Seok Kim
Jihoon kang
Venue: ICLR 2026 Poster (2026)
Keywords: OrchestrationBench
Workflow-based Planning
Constraint-aware Tool Use
Primary area: datasets and benchmarks

Abstract: Recent progress in Large Language Models (LLMs) has transformed them from text generators into agentic systems capable of multi-step reasoning, structured planning, and tool use. However, existing benchmarks inadequately capture their ability to orchestrate complex workflows across multiple domains under realistic constraints. To address this, we propose OrchestrationBench, a bilingual (English/Korean) benchmark that systematically evaluates (1) workflow-based planning and (2) constraint-aware tool execution. OrchestrationBench spans 17 representative domains with nearly 100 realistic virtual tools, covering scenarios that require sequential/parallel planning and compliance with business constraints. Unlike previous work, it explicitly disentangles planning evaluation from tool execution evaluation, which assesses tool selection, argument extraction, validation, and rejection handling. Constructed entirely through manual annotation with cultural adaptation, the benchmark ensures authenticity, diversity, and freedom from model-specific biases. Extensive experiments across state-of-the-art models show that function calling performance is relatively consistent, whereas planning capabilities exhibit substantial variation across models, emphasizing the need for structured planning evaluation. As a living benchmark, OrchestrationBench is designed to expand toward new domains, tools, and integration enabling rigorous, cross-cultural, and service-ready evaluation of LLM orchestration capabilities. The benchmark is publicly available.

Published as a conference paper at ICLR 2026
ORCHESTRATIONBENNCH: LLM-DRIVEN AGENTIC PLANNING AND TOOL USE IN MULTI-DOMAIN SCENARIOS
Aclim Ahn", Sooyeon Lee", Hyosun Wang, Chiwan Park, Daeryong Kim, Jihyeon Roh, Kichang Yang, Wonjun Jang, Hwang Woosung, Min Seok Kim", Jihoon Kangl Kakao Corp.
ABSTRACT
Recent progress in Large Language Models (LLMs) has transformed them from text generators into agentic systems capable of multi-step reasoning, structured planning, and tool use. However, existing benchmarks inadequately capture their ability to orchestrate complex workflows across multiple domains under realistic constraints. To address this, we propose Orchestration bilingual (En- glish/Korean) benchmark that systematically evaluates (1) workflow-boel -bssdd plan- ning and (2) constraint-aware tool execution. OrchestrationBench spans 17 rep- resentative domains with nearly 100 realistic virtual tools, covering scenarios that require sequential/parallel planning and compliance with business constraints. Unlike previous work, it explicitly disentangles planning evaluation from tool ex- ecution evaluation, which assesses tool selection, argument extraction, validation, and rejection handling. Constructed entirely through manual annotation with cul- tural adaptation. the benchmark ensures authenticity, diversity, and freedom from model-specific biases. Extensive experiments across state-of-thh-art models show that function calling performance is relatively consistent, whereas planning ca- pabilities exhibit substantial variation across models, emphasizing the need for structured planning evaluation. As a living benchmark, OrchestrationBencch is designed to expand toward new domains, tools, and integration enabling rigor- ous, cross-cultural, and service-ready evaluation of LLM orchestration capabili- ties. The benchmark is publicly available.
INTRODUCTION
Large Language Models (LLMss) have advanced rapidly recent years (OpenAl 2022 2023 Deep-
Mind 2025a Anthropic 2025 Although initially regarded primarily powerful text generators, recent research has demonstrated their capacity to operate as versatile agents that can interact with external tools (Yao al. 2023), perform multi-step reasoning over complex instructions, and assist users in various real-world applications (Shi et al. 2024). This evolution signifies a paradigm shift: from passive text generation toward the active orchestration of tasks, positioning LLMs as potential service-ready agents in both consumer facing and enterprise domains.
Despite this progress, substantial challenges remain for real-world deployment. In practice, user requests often involve sequences of interdependent subtasks that must be coordinated effectively (Huang et al., 2024 Yao et al. 2024). These tasks frequently span heterogeneous domains, require integration with external systems, and must adapt to dynamic constraints that evolve during user interaction. However, existing benchmarks operate largely in simplified or domain-isolated settings and thus do not capture the orchestration capabilities required for service-ready LLMs (Zhong al. 2025
Mialon et al. 2023).
'Equal contribution.
1 CConesponding authors. Conntact: {ei uu, marko.k)8kakaccorppp

---

Published as a conference paper at ICLR 2026
Dessmptinn
(Mlll-Spep))
Tim
Main LLM
Exccution
1. Tesk Crchastratin
Mottr
Uncerstanding Workdfoww Exeoulion
Taols 1111
Toote 11i1
Stopt: Bub LLM
Interaction
Execuatinn
Appregaie Heepemmeee
Thals
3 Dynamic Workfow Adeptation
Med inerupp, of Generate new worklow
Orchestration setting of OrchestrationBenchh, where main LLM decomposes user quests into workflows and assigns subtasks to sub-LLMs, which perform tool calling and return results for aggregation.
Figure 1:
To address these gaps, we introduce OrchestrationBencll bilingual benchmark explicitly de- signed to evaluate LLMs in realistic service environments. The benchmark defines a comprehen- sive evaluation protocol that emphasizes two complementary dimensions: workflow planning and constraint-awan tool execution. For workflow planning, the evaluation is formalized as workflow construction. Each workflow is represented dependencies, execution states, and agent assignments. For constraint-awar tool execution, the evaluation goes beyond the syntactic correctness of tool calling. It assesses three aspects: tool se- lection, argument extraction, and value validation against domain-specific constraints. Semantic correctness is adjudicated with auxiliary judges based on LLM. (Qin et al. 2023).
Directed Acyclic Graph (DAG) that encodes task
Figure illustrates the orchestration setting that motivates our benchmark. A main LLM first de- composes a user request into workflows and assigns subtasks to sub-LLMss These sub-I.LMs then perform tool calling with refined parameters, and their outputs aggregated by the main LLM before being returned to the user. This iterative process supports multi-step workflows, parallel or sequential subtask execution, clarification, and workflow revision.
The dataset underlying OrchestrationBencch covers 17 representative domains and nearly 100 realis- tic virtual tools, encompassing single-domain tasks, multi-domain orchestration, constraint valida- tion, and dynamic user revisions. The dataset was initially developed in Korean, leveraging abundant cases and thorough review ensure realistic interaction patterns and reliable evaluation. It was then expanded into English with comparable scale and domain coverage, with both versions val- idated for effectiveness as evaluation data. This design not only enables bilingual evaluation but also captures cultural differences in interaction styles, offering unique insights into orchestration across diverse service environments. Together, these properties allow OrchestrationBench to sys- tematically evaluate orchestration performance across languages, domains, and interaction patterns, moving beyond toy tool calling sets-ups to service-ready assessments.
In summary, this work makes the following contributions: (1) We introduce Orchestration the bilingual benchmark for evaluating L.LM orchestration in realistic multi-domain settings. (2) It separates orchestration into workflow planning and tool execution, with structured metries such (3) The benchmark includes a manually annotated dataset of 17 domains and nearly 100 tools, covering constraint validation and dynamic revisions. (4) Experiments reveal consistent tool execution but substantial variation in planning, highlighting the need for structured evaluation. (5) OrchestrationBench is designed as a living benchmark, extensible to new domains, tools, and deployment contexts.
as Graph Edit Distance.
"Code and dataset available httpa//wwwaa/ggoo....//////aaaaa
2

---

Published as a conference paper at ICLR 2026
Together, these contributions establish OrchestraationBench rigorous and extensible framework for advancing the study of service-ready LLM orchestration.
RELATED WORK
As LLM orchestration systems have evolved to coordinate multiple specialized models for complex tasks, evaluation methodologies have advanced correspondingly. We categorize related works into four primary areas based on their evaluation focus to highlight the distinct contributions of our benchmark.
Tool execution benchmarks assess an agent's 's ability to decompose tasks and invoke appropriate tools or APIs. Early benchmarks such BFCL (Patil et al. 2025), API-Bank (Li et al., 2023),
T-Eval (Chen et al., 2024), and ToolBench (Qin et al.) 2023) established foundational assessment through function-callee cealiinn leaderboards, API integration testing, and execution- bbasd evaluation.
Single-agent task performance benchmarks evaluate an individual agent's 's ability to complete tasks in specific environments. TaskBench (Shen et al., 2024) systematically assesses a single LLM's capacity for task decomposition and its accuracy in invoking predefined APIs (tool calling). Com- prehensive orchestration evaluation in various domains emerged as limitations became apparent, with benchmarks such as 7-bench (Yao et al., 2024). GAIA (Mialon et al. 2023). and Ultra- Tool (Huang et al. 2024) revealing that state-of-the-an agents achieved less than 50% success rates in dynamic interactions. Many benchmarks have also been developed to assess single agent's s performance on complete, realistic tasks, particularly within web, OS, and software environments. OfficeBench (Wang et al.] (2024) evaluates a single agent's s ability to perform long-horizon tasks by switching between office applications (Word, Excel, Email, Calendar). WebArena (Zhou et al.) 2023) evaluates web agents through online interaction with realistic websites (e-commerce, forums, development platforms), where a single agent must navigate web pages by performing browser actions. OSWorld (Xie et al.) (2024) evaluates end-to-end OS-level automation where single mul- timodal agent controls computers via raw mouse and keyboard inputs across Ubuntu, Windows, and macOS. Other notable works in this area include Mind2Web (Deng al. (023) TheAgentCom- pany (Xuet et al. 2024), and SWE-bench (Jimenez et al. 2024). Think (Shabbir et al.) (025) is a domain-specific benchmark for remote sensing, operating beyond web and OS environments, where single agent performs visual-spatial reasoning on satellite imagery with ReAct-style environment- driven replanning based on tool execution observations. The aforementioned benchmarks are valu- able in that they measure the agentic task execution capabilities of LLMSs across multiple domains. However, our work is distinct from these prior works, as it primarily focuses on L.M-tt-LLLMM col- laboration rather than individual agent performance, specifically designing systems where a main model orchestrates and invokes specialist LLMs.
Tool Safety benchmarks ToolEmu (Ruan et al. 2024) provides a crucial evaluation of safety, assess- ing whether a single agent can identify and refuse to execute high-risk or harmful requests, with its focus on the safety alignment of individual tool calls within emulated environment rather than on the agent's 's ability to perform complex, multi-step planning; R-Judge (Yuan et al.] (024) and Safe ToolBench (Xia al. 2025) assess risk awareness in multi-turn agent interactions. Instead of refusing harmful requests, OrehestrationBencl tests whether a main LLM can refuse infeasible re- quests by correctly understanding the functional descriptions and constraints of available sub-LLMs, addressing a distinct dimension of functional feasibility rather than safety.
Agentic planning benchmarks evaluate planning and coordination capabilities across varying levels of abstraction. PlanBench (Valmeekam et al. 2023) focuses on the abstract planning of a single
LLM using PDDL, but does not involve external tool execution. TimeBench (Chu al. 2023) addresses capabilities related to scheduling, but its focus is fundamentally different it statically evaluates the internal temporal reasoning of single L.LM through a guestion-answering format. AgentBench is a multi-turn agent evaluation framework which evaluates a single generalist L.LM's reasoning and Decision-making ability across diverse environments. AgentBoard (Ma et al. 2024) provides analytical evaluation of planning and tool-using for multi-turn tasks, but fundamentally assesses a single LLM agent interacting with predefined APIs or environments. These benchmarks represent an 'LLM-to-API' paradigm. Multi-agent collaboration assessment represents the latest advancement, with MultiAgentBench (Zhu et al., (025) which evaluates both collaboration and competition across eight diverse tasks (negotiation, social deduction games, coordination) with 2-
3

---

Published as a conference paper at ICLR 2026
2025 evaluates multi-agent coordination in planning and scheduling across diverse domains (logistics, disaster relief, events, optimization) with dynamic disruption handling. UltraTool (Huang al. 2024) evaluates LLM tool orchestration across a six- stage pipeline (planning, tool creation awareness, tool creation, tool usage awareness, tool selection, and tool usage) across 22 domains, but unlike work, it focuses evaluating single L.M perfor- mance rather than multi-agent orchestration systems. While these benchmarks test an agent's 's abil- ity to generate plans call tools, OrchestrationBencc introduces a hierarchical 'LLM-to-LLM' orchestration challenge where a main LLM acts as an orchestrator that dynamically coordinates specialized sub-LLMs based on natural language capability descriptions. We provide diagnostic evaluation that decouples workflow planning from constraint-aware execution, offering unique as- sessment of orchestration capabilities required for commercial chatbot deployments.
10 agents. REALM-Bench (Geng & Chang
Table 1: Benchmark Comparison
Benchmark
Multi-Agent
Func Call
Tool Inv.
Multi-turn dataset
Planning
Bi/Multi-lingul
ToolEmu
x
PlanBench
X
x
REALM-Bench
X
TaskBench
X
V
X
API-Bank
V
V
X
/
BFCL-v4
X
/
/
X
UltaTool
X
/
/
X
OrchestrationBench
V
V
THE ORCHESTRATIONBENCH FRAMEWORK
3.1 THE COMPLEXITY OF EVALUATING LLMs IN REAL- WORLD ENVIRONMENTS
Evaluating modern AI systems real-world contexts poses challenges far beyond simple question answering. A user request such as "Book flight to Seoul, find hotel near COEX, and share my itinerary with my team" requires multi-step planning and dependency management, since sharing the itinerary depends on completing prior bookings.
Realistic interactions also demand dynamic adaptation. Users may refine or change their goals during conversation, requesting an early morning flight, then revising due schedule conflict, or deciding to receive summary before sharing the itinerary. The model must flexibly update its reasoning and maintain overall task coherence.
Another critical aspect is constraint validation. When a user asks to schedule meeting at "4:10 PM", the system should recognize service constraints specified by the tool for example, meetings allowed only on the hour or half-hour-and and propose a valid alternative such as "4:00 PM", ensuring both compliance and usability.
These scenarios highlight the intertwined challenges of planning, adaptation, and constraint-aware execution that existing static benchmarks fail to capture. more comprehensive evaluation frame- work is therefore required to assess AI performance in realistic service environments. Detailed examples are provided Appendix
3.2 ORCHESTRATIINNENNC ARCHITECTURE
Orchestration introduces comprehensive evaluation framework with bilingual datasets in En- glish and Korean. In the following sections, we present how our henchmark is designed to capture diverse aspects of service orchestration, including planning, tool use, and multi-domain environ- ments.
3.2.1 ADVANCED PLANNING AND COORDINATION
We formalize orchestration as structured workflow schema that defines each task's 's execution state, dependency relations, and levp- planning (Table This structure enables evaluation of whether models can manage sequential and parallel execution, handle wnter-workflow dependencies, and adapt user interactions during execution.
4

---

Published as a conference paper at ICLR 2026
Table 2: Workflow planning schema
Field
Description
status
Exccution state of workflow: Values: pending, ruining, waiiting_or_jpppn completed. poused, canceled.
type
Depeadeacy type of the workflow. Values: independent, dependent.
depend.cn.
Specifies prerequisite workllows
completed before current workflow sequence of tasks within workflow. Each defined by:
steps
status
Progress of step.
Selected LLM for execution.
red que ry
Normalized user query.
Example
User request:
need go Scoul tomorrow for business Please book a flight, find hotel the COEX and shore my tiinerary with my team."
YAMI. representation:
workflow_2:
status: pending
Type: independent
ateps:
siatus: pending
name: travel_agent
refined_quezy "Boo a flight to Secul for Lemorzew For
business trip''
workflor_2: pending -ype: incependent steps:
status: pending
name: tzavel_ageat
refined_query: "pind and book hotel CUEX Center for tamorrow'
work[low_3r
status: peading
zype: depeadent
depend_an: ("Morrflox__'', *workflox_2"""
steps:
pending
name:
es endar_agent
refined_querys "SShare completed itinerazy with my Leam!"
In real-world settings, queries often evolve dynamically rather than following static task plans. OrchestrationBencc evaluates whether models can flexibly adjust workflows by generating new ones when additional tools are required and splitting workflows when explicit confirmation branching into subtasks is needed. For instance, a user modifies a booking request or adds new conditions mid-conversation, the model should update or extend the workflow while maintaining consistency with previous steps.
Clear criteria determine when workflows should be split unified. Independent requests (e.g., asking for both a flight schedule and hotel recommendation) or tasks requiring intermediate con- firmation (e.g., approving recommendation before booking) are treated as separate workflows. Conversely, tasks that contribute to single coherent goal remain within one workflow-ffr exam- ple, identifying celebrity's birthday before determining their zodiae sign.
This framework enables fine-graines evaluation of a model's ability not only to plan but also to coordinate, interrupt, and resume workflows in alignment with real-world interactive scenarios.
3.2.2 COMPREHENSIVE TOOL USE
The execution evaluation extends beyond verifying tool call accuracy to encompass the entire service-level interaction process. assesses whether models can not only invoke tools correctly but also decide when tool use is necessary, when information be provided directly, and when user inputs are insufficient or ambiguous and proactively request clarification, a behavior behaviorrepreesente by the AWA.T.FOR.USER..O.
UT signal.
Beyond syntactic correctness, real-world services require strict adherence to domain-specifit busi- ness rules. Before invoking tool, the model performs pre-execution validation and issues TOOL CONSTRAINT VIOLATION when constraints are unmet. Such validation includes main- taining logical consistency (e.g., rejecting flight booking where the return date precedes departure)
5

---

Published as a conference paper at ICLR 2026
and enforcing resource limits (e.g., budget or quantity restrictions). Only after successful validation should the model exccute the tool with correctly formatted arguments.
Model performance is then measured through call/reject classification metrics, where AMAIT FOR USER INPUT and TOOL CONSTRAINT WWI.VIIOLLIOOO represent rejection cases, and successful executions evaluated separately using function-calling performance measures. Fur- ther details are provided in the subsequent evaluation section
3.2.3 MULTI-DOMAIN TOOL ENVIRONMENTS
OrchestrationBencch defines 17 representative service domains that are extensible to real-world ap- plications while remaining independent of specific service dependencies. Each domain is built around realistic yet generalized scenarios, enabling evaluation of a model's intrinsic orchestration and cntrructioo-follwinng capabilities.
To reflect the complexity of world interactions, the benchmark includes 97 tools in English and 99 in Korean, with slight differences arising from culture-specific services such as address roman- ization and fortune telling. Unlike prior benchmarks with simplified tool abstractions, Orchestra- tionBench incorporates domain-specifee constraints and realistic behaviors, providing fine-grained coverage of diverse tasks and a faithful simulation of practical service environments.
These domains collectively represent three common types of user workflows: (1) inquiry and in- formation tasks (e.g., checking the weather, finding places, reading news), (2) action and transac- tion tasks (e.g., booking flight, purchasing items), and (3) planning and coordination tasks (e.g.. scheduling meetings, sending messages, arranging deliveries). This categorization highlights that OrchestrationBennch primarily reflects everyday consumer services. while remaining extensible to utility and productivity contexts.
All virtual tools were carefully designed to capture the nuanced characteristics of each domain and to ensure comprehensive task coverage. A complete list of tools is provided in Appendix
3.3 DATASET CONSTRUCTION
The OrchestrationBenchh dataset was designed capture the complexity and realism of real-world service orchestration. To ensure authenticity and quality, all conversation sessions, workflows, and tool calls were manually created by trained annotators following detailed construction guidelines, rather than generated synthetically. This approach ensures that conversation flows, tool usage, and constraint handling faithfully reflect realistic user-service interactions rather than artifacts of any specific model.
The overall construction pipeline-including domain selection, virtual tool design, and manual re- view and validation-is summarized in Table 3 Each stage represents distinct phase of data creation, from domain and tool specification to workflow refinement and multi-annotale validation. All scenarios were cross-valddttd by at least three independent annotators ensure consistency and accuracy. To enable controlled and interpretable evaluation, we excluded ambiguous or multi- solution cases and constructed data only from tasks with clear, well-defined dependencies. Through this rigorous, multi-stage process, AichestrationBencl achieves high reliability while remaining in- dependent of any single model or proprietary API.
Building upon this rigorous and model-independent construction process, OrchestrationBench ex- tends coverage to both English and Korean service environments, capturing diverse linguistic and cultural contexts.
By encompassing two distinct languages and service ecosystems, the benchmark enables evalua- tion of orchestration performance in bilingual and bicultural settings. This is particularly significant given the scarcity of evaluation resources for planning and tool use in Korean. By faithfully mod- eling the complexities of real-worke service interactions across both languages, OrchestrationBench provides a dataset that linguistically and culturally diverse, free from model dependency, and firmly grounded in realistic service orchestration scenarios.

---

Published as a conference paper at ICLR 2026
Table Overview of the Orchestrationneench dataset construction process.
Stage
Main Activities
Selection
are closely related to everyday extensible world service applications (e.g. travel, finance, scheduling, shoopingg)
representative
2. Viriual Tool Design
Design domain specific virtual lools by detining tool constraints. Initial tool descriptions are generated using GPT-lo and refined by annotators for accuracy consistency. Once defined. these lools
parameters, and realistic service level
across scenarios within the same domain ensure consistency and efficiency. The agent card schema is adapted from publicly available A2A examples (Google [025 tailored to our benchmark setting.
3. Scenario Construction
Anmolators design realistic user assistant dialogues across diverse categories such as single domain, multi-domain, constraint validation, clarification, and dynamic revision. Representative examples shown in Table
4. Workflow & Tool-call Defi- nition
Construct structured workflows and tool calls in YAML format, specifyyng execution states, depen- dencies, and argumeat structures.
5. Validation & Refinement
Tool-call results are generated using GPT-4o and iteratively reviewed across multi-tum dialogues. Each cross validated at least ihree independent amnotatons ensure accuracy and herence.
Table 4: Representative examples of constructed scenarios in OrchestrattionBenchh
Scenario Type
Representative Example
Single-domain
there sabway goes straight to the Statue of Liberty?"
dolti-domain orchestration
"I'm traveling next Saturday, Please book me taxi from airport to wy reserved timed with flight arrival."
Constraint violation with rection
Book dentist appointment 4:10 PM."
(System: "Appointments can only be scheduled on the hour or half-hour. Would you like me to set it 4:00 4:30 instead?"
User: "Please schedule it for 4:30 PM.")
User clarification request
"Send money Minji""
(System:: "Maltiple contacts named Minji ure in your address book. Could you specily the account or phone number?")
3.4 DATASET SCALE AND DISTRIBUTION
The dataset includes both English and Korean subsets, which comparable in scale. The English subset contains 219 conversation sessions, 317 planning cases, and 706 tool call instances, while the Korean subset contains 222 sessions, 324 planning cases, and 730 tool call instances, reflecting slight variations due to language-specife differences. (see Appendix D Table 7).. Both datasets span 17 representative service domains with intentionally asymmetric tool distributions: broader domains such as Places or Entertainment contain more tools, while narrower domains such as Weather or News remain compact to reflect realistic usage frequency.
At the workflow level, most sessions involve 2-3 workflows and 2-3 domains. although some ex- tend up to steps or span 4+ domains. This indicates that a single session typically requires multiple rounds of planning, with some including as many as seven planning steps. Moreover, the frequent inclusion of two or more domains reflects realistic multi-domain scenarios where users transition across heterogencous services. In terms of tool invocation, the dataset is dominated by sequential and parallel call structures rather than single isolated calls, demonstrating the complexity of orches- tration required to complete
tasks. (see Appendix D Figure
Together, these distributions demonstrate that OrchestrationBencce covers a wide range of real-world orchestration patterns, enabling fine-graid fine-grained evaluation of model planning, tool invocation, and adaptive reasoning capabilities. This highlights that the benchmark goes beyond evaluating isolated question answering or toy tool callings, and instead enables assessment of orchestration performance in realistic, constraint-awar service environments.
4 EVALUATION
Current end-to- end-to-end benchmarks Liu al. 2023): Mialon al. (2023): Jimenez et al. (024); Yao
al. 2024) offer flexibility but often obscure failure points in complex multi-step tasks. To address this limitation, we employ stepwise evaluation that isolates and tests each component independently.
7

---

Published as a conference paper at ICLR 2026
Our evaluation distinguishes two primary phases: Planning and Tool execution. We further decom- pose tool execution into two sequential assessment criteria to capture the nuanced behaviors of sub-LLMs: Call/reject classification accuracy and Function calling performance.
Models We evaluate the following state- of- are language models, including OpenAl GPT mod- els (gpt-4.1, gpt-4o, gpt-5) (OpenAl 2022 2025), Anthropie Claude models (claude-sonnet-4) (An-
thropic 2025). Google Gemini models -eeminn-2---ro-preevee
gemini-2.5-flash (lass-preview) (Deep- Mind 2025a b), Alibaba Qwen models (Qwen3 series) (Yang et al., 2025), and other Korean open-
source models (A.X-4.0 (Lab) 2025), kanana-1.5 (Team al. 2025) ). EXAONE-4.0 (Research al.,
2025
All reasoning models are configured with low reasoning effort settings.
Evaluation Protocol
We design our evaluation with the following principles:
Each target LLM receives complete conversation history up to the evaluation point
Parallel-executed LLMs operate with isolated histories to prevent information leakage
Sequential LLMs access cumulative conversation history including previous model outputs
Main-LLM workflow generation is triggered exclusively by user input
Sub-LLMs process refined queries from the main-LLM and user-provided clarifications
Each scenario is run three times per model with temperature 0.2 to ensure robust evaluation.
4.1 EVALUATION METRICS
Planning Assessment We measure workflow generation quality using Graph Edit Distance (GED), which quantifies structural differences by calculating the minimum edit operations needed to transform one graph into another, following Gabriel et al. (2024). GED is normalized to 0-1 scale based on the maximum possible edit distance, where higher values indicate more edits required and thus worse performance. To align with conventional reporting where higher scores reflect better performance, we report 1-GED throughout our evaluation. Our workflow representation includes workflow structure, step assignment (sub-LLM selection), and execution status. We conduct hier- archical workflow score evaluation with structural score measuring workflow topology correctness and component score evaluating step-level assignments. We assign higher weight to selection errors (0.8) than status errors (0.2), reflecting the intuition that choosing the wrong tool is generally more detrimental to task success than misidentifying tool execution status, though we acknowledge this weighting is not empirically derived.
Tool Execution Assessment To comprehensively evaluate tool execution capabilities, we exam- ine critical aspects: the model's ability to make appropriate calling decisions and the quality of actual function executions. Call/reject classification accuracy measures the proportion of correct de- cisions including both appropriate rejections and successful function call attempts out of total cases. Function calling performance evaluates the correctness of actual function calls through three spe- eific metrics: tool selection F1, key F1, and argument F1 among cases that successfully proceeded to the function calling stage.
For function calling parameter validation, we employ a three-stage approach: exact match compar- ison, type/pattern validation against tool descriptions, and semantic validation for remaining cases. To reduce model bias, we use an ensemble of three L.M judges (GPT-4.1, Claude Sonnet 4, and Gemini 2.5 Flash) with temperature of 0.3, averaging their scores by taking the arithmetic mean. The LLM judge classifies true/false positives and negatives, with these assessments integrated into F1 calculations. To further ensure reliability, we measured the inter-rater agreement between the human annotators and the LLM judge, which yielded a Cohen's 's Kappa score of 0.63, indicating substantial agreement. To maintain compatibility with function calling training (JSON output for- mat), we implement call rejection and information requests using XML output format.
All detailed results presented in Appendix
8

---

Published as a conference paper at ICLR 2026
Model
Plan
C/K
FC
Score
0.850
0.724
0.836
0,803
0.706
0.778
1000
clande-sonnet-4
0,713
0.868
0,885
gpt gpt 2.2.1-20225-0411
0.744
0.860
0.861
0.822
3pt-Ro-2024-11-2
0,771
0,806
1110
8:00
CMP1
3108
gpt-4.1 mini 2025
minimmm25
0.678
0.801
0.841
0.774
3pt-5-2025-08007
0.583
0.791
0.736
0.462
0.725
0,752
0.646
6100
Qwen3-32B
0.792
0.857
0.808
0.819
1140
1110
Qwen3-14B
0.786
0.834
0.843
0.821
Qwen3-2358-A228
0.768
0.876
0,880
0.842
Qwen3-8B
0.675
0.830
0.776
Qwen3-90B-A3B
0.746
0.847
0.829
0.807
A.X-4.0
0.707
0.781
0.832
0.773
Kanana--15-322. 1t-2.iiis-tt
0.667
0.766
0.737
EXAONE-400-128
0.057
0.653
0,534
0.415
Mapiic
Figure 2: Model performance on English dataset.
Plan
C/R
Score
geminni 2.5. prev-reevewwweee 06 05
0.828
0.813
0.875
0.839
0.807
0.767
0,847
0.807
somnel
0.797
0.759
1-1-441-2022904.1
0.749
0.786
0.891
0.809
3pt-4o-2024-11-20
0,800
0.667
0.883
0.784
gpp-t_1mmin-2225-000-1
0.737
0.587
0.732
Spt-5-2025508-11
0,524
0.748
0,855
0.709
1649
1100
gpt 1-mmm-200550044111
0,451
0.486
0,791
0.576
Qwen3-32B
0.795
0.807
0,816
0.806
Qwen3-14B
0.772
0.789
0.841
0.801
-wen3-2358-A228
0.791
Owen3 8B
0.682
0.658
0,833
0.725
OweR3-30B-A3R
0.449
0.662
A.X-4.0
0.672
0.483
0,818
0.658
kamana- 1.5.32.5h-insnn
0.589
0.433
0.784
0.602
EXAONEE-.0-322
0.529
Figure 3: Model performance on Korean dataset.
Note: Left: detailed performance metrics across planning workflow score, call/rejoct classification (CR) function calling (PC), and overall score. Right: tool usage heatmap showing performance distribution
evaluation metrics. Best performance is marked bold. and
second best performance is underlined. Claude models were evalusted through AWS Bedrock with model version hropio.cc. aude-sonne sonde-t taamet-4-200 -4-2225551--- 4-v1 Our Workflow Score computed as I-GED, where higher values indicate hetter performance. Detailed evaluation results are provided in Appendix[E
4.2 EVALUATION RESULTS
Based on the comprehensive evaluation results presented in Figures 2 and along with correla- tion analysis examining the relationships between different evaluation metrics, several key insights emerge regarding model performance in agentic planning and function execution tasks.
Open-Source Model Viability Open-source dense models achieve competitive performance, with models like Qwen3-235BB-A228 reaching scores comparable proprietary alternatives (0.8404 En- glish, 0.8044 Korean). Dense architectures consistently outperform mixture-ou mixture-of-expert of-eeeers- variants in planning tasks.
Model-Specific Specializations Each model family exhibits distinct strengths independent of size. Gemini models excel in planning (0.8504 English, 0.8278 Korean) but show relatively weaker func- tion calling. Slaude-sonnet-4 demonstrates strong function calling capabilities (0.8821 English, 0.9084 Korean), while GPT-4. variants show balanced performance. Notably, workflow generation exhibits relatively larger performance variation between top-tier and lower-performing models in both English and Korean datasets (see Appendix E Figure 7) highlighting planning as the discrim- inative capability among the evaluated tasks.
Planning-Exeesn Execution Gap Function calling scores represent performance only among cases with correct call/reject decisions. The correlation analysis reveals relatively weak link between plan-

---

Published as a conference paper at ICLR 2026
ning and decision outcomes compared to other measures. This suggests models may generate good workflows but struggle with execution decision-making.
Language-Depent Dependenn Performance Rankings vary substantially between languages, with Claude improving in English decision-making (0.7586-+0.866) -06882) while Gemini maintains stronger Korean performance. This indicates magguaee-ppccfiie training effects and the importance of biligual evaluation.
These findings underscore the need for task lask-specific and language- aawrr aware model selection, with tention to the glanning-execution gap that may limit real-world agentic performance despite strong individual capabilities.
Table Correlation Analysis Between Task Components
Call/reject Classilication
Workllow Scure
Function Calling
English Dataset
Call/rejett Classification
1,0000
Workflow Score
0,5830
1.0000 0.9215
Function Calling
0.7256
1,0000
Metric
Call/rjjctt Classification 1,0000
Workllow Score
Function Calling
Call/rejoct Classification
Korean Dataset
Workflow Score
0,4480
10000
Function Calling
0,5773
0.7751
1,0000
The evaluation results reported here correspond to fixed experimental snapshot. Orchestra- tionBench is maintained as a living benchmark, and the public repository may include additional models and updated evaluations beyond those presented this paper.
CONCLUSION AND FUTURE WORKS
This work introduces OrchestrationBennch. the first bilingual (English/Korean) benchmark for evalu- ating LLM orchestration capabilities in realistic multi-doman service environments. By separating orchestration into workflow planning and tool execution components, our evaluation framework provides detailed insights into model performance across different aspects of agentic reasoning.
Our comprehensive evaluation reveals that open-source dense models achieve competitive perfor- mance in agentic tasks. However, two critical findings emerge: workflow planning shows substan- tially larger performance gaps between models compared to function calling, requiring careful model selection for orchestration tasks. Additionally, while models execute function calls effectively, they struggle with call/reject dlasificationnddetrmminnggg when function calling is appropriate given real- world tool constraints. These findings suggest that current training approaches do not adequately address the decision decision-making complexities essential for practical agentic deployment.
Our evaluation covers 17 domains in English and Korean, which may not capture all orchestration scenarios or generalize to other languages. The current benchmark uses predefined workflows and virtual tools, limiting exploration of more flexible, end-to-end workflow generation and real-world tool integration through frameworks like MCP (Model Context Protocol). Additionally, our turn-by- turn evaluation assumes successful execution each step. potentially inflating overall performance metrics. In practice, an end-to-end evaluation where eπгоrs propagate across turns would likely yield lower success rates, as failures early stages would cascade subsequent steps. Future work incorporating true end-to-end evaluation with real tool integration would provide more realistic performance assessments and reveal the robustness of orchestration systems under error conditions.
Key directions include expanding domain coverage and bilingual support, enabling more flexible end-to-end workflow exploration, integrating real-world multi-domain tools through frameworks like MCP, developing training methods to address the planning-execution gap, and supporting more sophisticated multi-agent coordination patterns. As a living benchmark, OrchestrationBencch will continuously evolve with new domains and tools based on community feedback and deployment needs.
OrchestrationBenchh establishes foundation for systematic evaluation of service-ready LLM chestration, moving beyond isolated tool-calling toward comprehensive multi-agent coordination assessment.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This research was supported by Kakao Corp.. which provided institutional support. We sincerely thank the annotators of Linkage Lab for their contributions to data construction: On-yu Park, Sang- min Lee, Jung-ah Han, Seungkyun Ro, Woon Jo, and Hyeji Lee.
REFERENCES
Anthropic. Claude opus & claude sonnet 4 system card, 2025. anthropic. com/ claude waaaudeeoonneeee sonnet
URL htttps:/wwal
Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning I. Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang. Dahua Lin, Kai Chen, and Feng Zhao. T-eval: Evaluating the tool utiliza- tion capability of large language models step by step. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume I: Long Pupers), pp. 9510-9529, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 1...8663/112222.clllloooo
URL
5157]
Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Haotian Wang, Ming Liu, and Bing Qin. Timebench: A comprehensive evaluation of temporal reasoning abilities in large language models, 2023.
org/ abs 2311.1111171 17667
Google DeepMind. Gemini 2.5 flash model card, 2025a. URL
googleapis
s /ww./igeennnniiieeeieeiiiiiiirae 2. flash.pdl
Google DeepMind. Gemini 2.5 pro model card, 2025b. URL htttps:/ 2.5-pro..pd
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and
Yu Su. Mind2web: Towards generalist agent for the web, 2023.
Adrian Garret Gabriel, Alaa Alameer Ahmad, and Shankar Kumar Jeyakumar. Advancing agentic systems: Dynamic task decomposition, tool integration and evaluation using novel metrics and dataset. arXiv preprint arXiv:2410.22455 2024.
Longling Geng and Edward Y Chang. Realm- A real-world planning benchmark for ilms and multi-agent systems. arXiv preprint arXiv:2502.18886, 2025.
Google. Agent2agent (a2a) samples repository.
http::://tttuu.cco///22222
aaa-samplen les 2025.
Shijue Huang, Wanjun Zhong, Jianqiao Lu, Qi Zhu, Jiahui Gao, Weiwen Liu, Yutai Hou, Xing- shan Zeng, Yasheng Wang, Lifeng Shang, Xin Jiang, Ruifeng Xu, and Qun Liu. Planning, creation, usage: Benchmarking LLMs for comprehensive tool utilization in real-world complex scenarios. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Asso- ciation for Computational Linguistics: ACL. 2024, pp. 2024. Association for Computational Linguistics. doi: 10. 1011886//11222/ 120.://////22......n---aaa mandings-acl.259 URL https://cclanthhooooo.og////// wtw.:o/gg//222...iiinni........e acl.
Bangkok, Thailand, August
Carlos E Jimenez, John Yang, Alexander Wettig. Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE--bnch: Can language models resolve real-world github issues? In The Twelfth International Conference Learning Representations, 2024. httpttts:: htt.htttts::/pppee////e
fett/oorumiid-ie 8yNQM66
SKT AI Model Lab. A.x 4.0, 2025. UR//tttsss//ugggiaaaae..e////
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. API-bank: A comprehensive benchmark for tool-augmented LLMs. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing. pp. 3102-3116, Singapore, December 2023. Association for Computational Linguistics. doi: 10. (006633/11222..mmmml main. 187. URL https: //acl antho www.ooy......................... wrgp.orrg/2333 www.pmnmmmmmme 1877

---

Published as a conference paper at ICLR 2026
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng. Zhengxiao Du Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating llms as agents. arXiv preprint arXiv: 2308.03688, 2023.
Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujuu Yang, Yaohui Jin, Zhenzhong Lan, Ling- peng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn IIm agents, 2024.
Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: benchmark for general ai assistants, 2023. URL https://arxiv,
abs/ /211 12983
OpenAl. Chatgpt, 2022. URL Ittppttts///chht.pppnaa com
OpenAL GPT-4 technicalll CoRR, ao2R. doi: 1048550/ARXXVV2300.07777 URL
OpenAL.
Introducing
2025. Accessed: [2025-09-22].
Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (BFCL): From tool use to agen- tic evaluation of large language models. In Forty-second International Conference on Machine Learning. 2025. URL
net forum? id= 2mDG2mDDDDDD
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023. URL htttts://aaxiiiiii org/ abs 2307 16789
LG AI Research, :, Kyunghoon Bae, Eunbi Choi, Kibong Choi, Stanley Jungkyu Choi, Yemuk Choi, Kyubeen Han, Seokhee Hong, Junwon Hwang, Taewan Hwang, Joonwon Jang, Hyojin Jeon, Kijeong Jeon, Gerrard Jeongwon Jo, Hyunjik Jo, Jiyeon Jung, Euisoon Kim, Hyosang Kim, Jihoon Kim, Joonkee Kim, Seonghwan Kim, Soyeon Kim, Sunkyoung Kim, Yireun Kim, Yongil Kim, Youchul Kim, Edward Hwayoung Lee, Gwangho Lee, Haeju Lee, Honglak Lee, Jinsik Lee, Kyungmin Lee. Sangha Park. Young Min Paik, Yongmin Park, Youngyong Park, Sanghyun Sco, Sihoon Yang. Heuiyeen Yeen, Sihyuk Yi, and Hycongu Yun. Exaone 4.0: Unified large language models integrating non-reasoning and reasoning modes, 2025. URL http::: //rxsi oorgiiiiii.r/vo.oooooo.ooooooo.r
abs/2500 2507. 11407
Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J Maddison, and Tatsunori Hashimoto. Identifying the risks of im agents with an Im-emulated sandbox. In The Twelfth International Conference on Learning Representations, 2024.
Akashah Shabbir, Muhammad Akhtar Munir, Akshay Dudhane, Muhammad Umer Sheikh, Muham- mad Haris Khan, Paolo Fraccaro, Juan Bernabe Moreno, Fahad Shahbaz Khan, and Salman Khan. Thinkgeo: Evaluating tool-augmented agents for remote sensing tasks, 2025. URL
https: httpaxxxvvv..g....a abs 250./2235522222
Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Yueting Zhuang. Taskbench: benchmarking large language models for task automation. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS '24, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 9798331314385.
Zhengliang Shi, Shen Gao, Xiuyi Chen, Yue Feng, and Lingyong Yan. Chain of tools: Large language model is an automatic multi-tool learner. Computation and Language, 16533v6, 2024.
Kanana LLM Team, Yunju Bak, Hojin Lee, Minho Ryu, Jiyeon Ham, Seungjae Jung, Daniel Won- tae Nam, Taegyeong Eo. Donghun Lee, Doohae Jung, Boscop Kim, Nayeon Kim, Jaesun Park, Hyunho Kim, Hyunwoong Ko, Changmin Lee, Kyoung- Syoung-Woon On, Seulye Baeg, Junrae Cho,
12

---

Published as a conference paper at ICLR 2026
Sunghee Jung, Jieun Kang, EungGyun Kim, Eunhwa Kim, Byeongil Ko, Daniel Lee, Minchul Lee. Miok Lee, Shinbok Lee, and Gaeun Seo. Kanana: Compute-efficien bilingual language models, 2025.
www.vvorg/aaaa abs 2t0pp//552..199333
Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kamb- hampati. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. Advances in Neural Information Processing Systems, 36:38975- 38987, 2023.
Zilong Wang, Yuedong Cui, Li Zhong, Zimin Zhang, Da Yin, Bill Yuchen Lin, and Jingbo Shang. Officebench: Benchmarking language agents across multiple applications for office automation, 2024. URL http///axvvv.../ org/ /00p/240000 19056
Hongfei Xia, Hongru Wang, Zeming Liu, Qian Yu, Yuhang Guo, and Haifeng Wang. SafeTool-- Bench: Pioneering prospective benchmark to evaluating tool utilization safety in LLMs. In Christos Christodooulppoulss, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.). Find- ings of the Association for Computational Linguisties: EMNLP 2025, pp. 17643-176600 Suzhou, China, November 2025, Association for Computational Linguistics. ISBN 979-8-89176-335-7. doi: 10. 10.18653/v1/2025.1 htt.://222...iiinsssieeeee iindings-emnl..95. URL https: /attps://aclanttiii org.
E Firdings-emnipp 958/
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024.
Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang. Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang. Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou, and Graham Neubig. Theagentcompany Benchmarking Ilm agents on consequential real world tasks, 2024.
URL http:hhtt////aattt www.:iii.orrg/bsso..o at////////44/4 14161
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025, URL /tappvi//aaxiv.../ org/ abs5 09388
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. ICLR, 2023.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. r-bench: A benchmark for tool-agent-use interaction in real-world domains. Artificial Intelligence, 2406.120453 2024.
Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijic Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, Rui Wang, and Gongshen Liu. R-judge: Bench- marking safety risk awareness for Ilm agents. arXiv preprint arXiv:2401. 2024.
Lucen Zhong, Zhengxiao Du, Xiaohan Zhang, Haiyi Hu, and Jie Tang. Complexfunchench: Ex- ploring multi-step and constrained function calling under long-context scenario, 2025. URI.
wwwrg//////aa. htttps://arxie org/ abs awwp///s...... 11.500111111
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307. 33854, 2023. URL htt.::tts//// /tttap:::/wwbareeeee rena. dev
Kunlun Zhu, Hongyi Du, Zhaochen Hong. Xiaocheng Yang. Shuyi Guo, Zhe Wang. Zhenhailong Wang, Cheng Qian, Robert Tang, Heng Ji, and Jiaxuan You. MultiAgenttBnch: Evaluating
13

---

Published as a conference paper at ICLR 2026
the collaboration and competition of LLM agents. In Wanxiang Che, Joyce Nabende, Ekate- rina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8580-8622, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176- 251-0. doi: 10. 06653//1/2025.alllloooo acl 101--am 1al--ong.440 4217
URL https: / //acl ortgheoooo...r///23...
14

---

Published as a conference paper at ICLR 2026
Appendix
A CHALLENGING HEAL-WORLD EXAMPLES
User
User
Book tickets for eurliett Artana Grande reserve restauram hours before add
Book ar appuintment el XXX Dental ludkey el 4:10 PM.
Entertainment
Place loday al 4:10
Check Ariana Grande's concert schadule and information
Entertainment
Place
Book 2 VIP tokkts the earilest Arlana Grande concert
tool: reservePlace
concert venuue where e people can dirnet hours before the show
Constrations
The reservation can only be made on the hour or the half hour.
Place
Make dinner reservatio for people near the compart venuee 2 hours before the show
LLM
I'm sorry, reservations can only be acheduled on the hour half part. Would you like me to book it for 4:00 instead 4:107
Calendar
Chec Artana Grande's concert so'edulic and Information
User
User
Can recommenc pair of wireless earbucs as birthday gil for frien?? like therr, order right
What does nekse cance ling do?
LLM
Shopping
Razom mond
earburs
airthday gitl
Shopping
for friend
Recommend wireless carbuds birthday gift for friend
Internption
$
Shopping Order wire ess earbuus as bithday gift for a friend
caloecanoelln feature
Shopping wireless earbuds $100 bithcay gift friend
User
expensive... Please recommmnn some hing under $100.
User
LLM
wircless carbuds, suggest some other gift 00.
Shopping
Recommend wireless earbuds birthday gift friend
LLM
Chango
Shopping
Search
Order wireless earbuce birhday gift for a friend
Search noise canoallnn
Recommond wirolass carbuds undor5100 as birtday git for friend
Shopping
Cancolation!
Dicior wiraless earbuds under 5100 as birthay gift for Mend
Shopping
Shopping
Recommmend binilday gills under $100 for lr end, excluting Wimieess sartude
Figure 4: Illustrative workflow scenarios in real-world service environments. (1) Multi-step orches- tration involving concert ticket booking, restaurant reservation, and calendar scheduling, demon- strating sequential and dependent workflows. (2) Constraint-aware execution where invalid requests (e.g., appointment at 4: 10 PM) are negotiated into valid alternatives (c.g., 4:00 PM). (3) Dynamic adaptation to evolving user preferences, including refinement, change, interruption, and cancella- tion during gift recommendation. These examples highlight the need for evaluation frameworks that capture planning, coordination, and robustness realistic service contexts.
15

---

Published as a conference paper at ICLR 2026
B AN ILLUSTRATIVE EXAMPLE OF THE WORKELOW GENERATION PROCESS
Turn 1: Initial Request
Recommend wircless carbuds as birthday gift. I'll order if like them
workflow_ Independent
Status: Feading
Product recomendation: wircless earbuds ($105-$190)
workflow 2
Fending
Dependent depends workflow I
Order execution (awaiting selection)
Search wircless earbuds results
LLM
Turn 2: Requirement Modification
Too expensive. Please recommend something under $100.
workflow 1 Independent
Status: Carspleted
Product recommendation: wireless earbuds ($105-$190)
workflow __11___11___-__
Sulus: Pending
Independent
Product recommendation with price constraint: earbads
(<$100)
workflow
Statu:: Paused
Dependent - depends on: workflow 1-1
Order execution (awaiting user selection)
Scarch carbuds results $$10
LLM
Turn 3: Context Interruption
What does noise cancelling do?
workflow 1 Independent
Status: Completed
Product recomncndation: wircless carbuds ($105-S190)
workflow 1-1
Status: Completed
Independent
Product recommendation with price constraint: earbads
(<5100)
interrupted workflow 2-1
Status Pending
Independent
Search for noisc-cancellling technology
Paused
workflow 2
Dependent - depends on: workflow [-]
Order exccution (swaiting user selection)
Figure 5: Example of the Proposed Workflow Generation Process
16

---

Published as a conference paper at ICLR 2026
C
FULL LIST OF TOOLS IN ORCHESTRATIINNEENG
Table 6: Full list of domains and tools OrchestrationBench
Domain
Tools
Shopping
searchProducts, recommendGifts, sendGifts, orderProducts, getOrder- Status, cancelOrder, modifyOrder, exchangeProducts, refundProducts recommendRestaurants, searchPlaces, reservePlaces, getPlaceReser- vationInfo, cancelPlaceesservttin,, modiffylaaceeservatiinn, ge- IRealEstateInfo
Places
Transportation
Iat getDirections, gerTransportnto, getParkknglnio,, call- Taxi, DailDesggnatedDriveer,,, bookRentalCar, gerTransitSchedule, book- TransifTicket
Logistics/Deliver
bookDeliveryServicc, trackDelivery
Weather
DelDomesticWeatheer getGlobal
Finance
getStockPrice, getCryptoPrice, getExxhangeRate, gttesrsssttss,,,,,,sss,s,, get- GoldPrice, searchFinancelnfo
Travel
findFlightlnfo, bookFlight, getFlightReservatiinn, changeFlight, can- celFlight, getAccommodationnnff. getAccommodattoneservatii bookAccommoodttion, cameelAccommdaation modify tion, planTravel, getPopulatPlaccennoo
Entertainment
gerlVvrogramlnno, getMovielnto, Moveeee,e,ickke,, getMovieBook- ing, cancelMovieBooking modifyMovvieBookingg, getFxhiibitionInfo. BookExhhbiiiionTikke.,, gedExhhibitoonResevaaiillll tionTicket, modify ExhibitionTicket getPerformancelnfo, bookPerfor- mance, getPerformanceBookiin cancePerforrmanceBBoking modi- JyPerformance search Video, www.MMuiiinnf.geeeebbeeee
cancelExhibi-
Life Information
MeLLotteryInfo, get getPostalCode, geuPheneeNumbernnto
Calendar
getCalendar( LunandddrL,una///olaar createSchedule,, getSchedule, cancelSched- ule, modifySchedule remindSchedule
Sports
getSpor(Gamelnto, getSportRank
Person
getProfile, getPersonNews
Counseling
getCounseling, geiZodiaclnio, getCompatiblitylnno, getStarSignInto,
getMbtilnfo
Search
searchInfo
News
searchNews, summarizeNews
Message
sendMessage transferMoney, getAccountBalance, getl getl.oanB,, create Transfer, getAuutoTrasffer.iss,, modifyAutoTransser Tutoela fer converroEngliishdddess, conderToooaaddddrress,,,, getSajulnfo
Personal Banking
Services (Korean only)
Cultural
17

---

Published as a conference paper at ICLR 2026
D
DATASET OVERVIEW
Language
Sessions
Plannings
Tool Callings
English Korean
219
317
706
222
324
730
Table 7: Dataset statistics for English and Korean subsets.
Entertainment
Shopping
13%
Places
12%
Transportation
16%
10%
7.%
Delivery
9%
7120
Sports
Travel
3.
Message
7%
News
5%
%
Life_Info
Calendar
Weather
Counseling
PersonalBanking
Person Finance
Search
Domain-wise distribution across 17 service domains
121
100
100
100
Count
73
50
35
50
-88
46
50
11
6
15
11
2 3 4
6 7
2
3
4 5
A
B
C
D
Workflow per session
Domain per session
Workflow Structure
(b) Workflow count, domain count, and workflow structure
Figure 6: Dataset characteristics: (a) domain coverage and (b) workflow-level properties. In work- flow structure, the x-axis abbreviations denote workflow structures: A=single, B=parallel only, C=sequential only, and D=parallel+sequential.
18

---

Published as a conference paper at ICLR 2026
E DETAILED EVALUATION RESULTS
1.2
1.2
1.0
1.0
0.8
0.6
Score
0.6
0.2
0.7
0.0
-0.2
-02
Workflow
Call/Reject
Function Call
Workflow
Call/Reject
Function Call
Ka)an Dataset
(b)English Dataset
Figure 7: Distribution of model performance across different tasks shown as violin plots. The width of each violin represents the density of models at different performance levels, with wider sections indicating more models achieving those scores.
19

---

Published as a conference paper at ICLR 2026
Table 8: Workflow Generation Performance
(a) Workflow Generation Performance on English Data
Model
Overall Planning Score
Structural Score
Component Score
gemini 2.5 preview
0.8504
0.8141
0.8962
gemini-2.5-ffas--rrviiew
0.8077
0.7998
0.8571
claude-sonnet-4
0.7727
0.7409
0.8408
gpt-4.1-2025-04-14
0.7444
0.7552
0.8386
gnt-4o-2024-11-20
0.7709
0.7552
0.8208
spt-4,1-mini-005-444
0.6784
0.6780
0.7740
gpt-5-2025-08-07
0.5827
0.6839
0.7536
1pp=4-1-nan-2025--04114
0.4623
0.5332
0.5449
Qwen3-3BB
0.7923
0.7925
0.8377
Qwn3--1B
0.7858
0.7665
0.8189
Owen3-255B-A221
0.7679
0.7744
0,8392
0.7225
0.7129
0.8029
Qwen3-8B
0.6749
0,7573
0.7765
Qwen3-30B-A3B
0.7508
0.7632
0.7915
Qwen3-30B-A3B
0.7455
0.7676
0.7994
A.X
0.7074
0.7679
0,8068
kanana- 1..55.55.-iinteeeee
0.6670
0,7074
0.7335
EXAONNE-40-32B
0.0574
0.6784
0.7260
(b) Workflow Generation Performance on Korean Dataset
Model
Overall Planning Score
Structural Score
Component Score
gemini-2.5 2.5 - 1ee-pro-ppeeeee pro-prrvieww
0.8278
0,7842
0.8819
gemini-2.5-flas--rrviiew
0.8067
0.7983
0.8584
claude-sonnet-4
0.7974
0.7583
0,8603
gpt-4.1-2025-044
0.7488
0.7488
0.8334
ept-4o-2024-1--20
0.7999
0,7807
0.8446
gpt-4.1 110-444-mini2202-04441
0.7372
0.7260
0.7913
gpt-5-2025-08-07
0.5235
0.6386
0.6883
gpt-4.
0.4507
0.5454
0.5344
Qwen3-32B
0.7946
0.7893
0.8214
Qwen3-14B
0.7722
0.7687
0.8029
Wwen3-235B-A22B
0.7911
0.7669
0,8366
(ecm3-235B-A22B-Imsse
0.7345
0,7234
0.7999
Qwen3-8B
0.6822
0.7643
0.7703
Owen3-30B-A3B.
-nnttutttnnttrtctt
0.7317
0.7385
0.7789
Qwen3-30B-A3B
0.7336
0.7239
0.7986
A.X-4.0
0.6721
0.6219
0.7714
kanana-1.55 1.5-32.5b-instruct
0.5890
0.6649
0.7061
EXAONE-4.0-32B
0.1181
0.2268
0.2274
Note: Table shows workflow generation performance results. The Overall Score includes all cases and assigns 0 points to completely failed workflows. Structural Score and Component Score metrics only evaluate success- fully generated workflows, excluding failures, which leads different score distributions. Our Workflow Score computed I-GED, where higher values indicate better performance. We report the Overall Planning Score as the Planning Score Figures and 3
20

---

Published as a conference paper at ICLR 2026
Table 9: Tool Execution: Call/Reject Decision Performance
(a) Call/Reject Classification Performance on English Dataset
Model Name
Call/Reject
Rejection F1
FC decision FI
Classification Accuracy
gemini-2.5 S.riini-..-pr-pppee 2.S-pro-preview
0.7241
0.6775
0,8826
gemini-2.5 2.5-illshh prreeveww
0.7064
0.6518
0,8668
claude-sonnet-4
0.8682
0.6841
0.9175
gpt-4.1-2025-04,
0.8595
0.6406
0,9134
gpt-4o-2022-11--2
0.7956
0.3765
0.8784
gnt-4.1-mini-2025-444111
0.8012
0.3468
0.8827
gmt-5-2025-08-007
0.7905
0.6236
0.8557
ggl-4_lnnano-2255-44111
0.7248
0.1039
0,8380
Qwen3-32B
0,8570
0.6965
0,9072
Qwen3-14B
0.8339
0.6807
0,8888
Qwen3-235B-A22P
0.8761
0.7449
0.9193
Owen2-235D-A222--mmmll
0.7934
0.2646
0,8808
Qwen3-8B
0.8299
0.5952
0,8931
Dwen3-30B-A3B-Imsee
0.7635
0.0212
0,8671
Qwen3-30B-A3B
0.8472
0.6453
0,9034
A.X-4.0
0.7808
0.1326
0.8755
kanana- 1.tmaa--15-322 1.5-22.5b-instruct
0.7663
0.0000
0.8682
EXAONE-4.0-32B
0.6533
0.3198
0.7689
(b) Call/Reject Classification Performance on Korean Dataset
Model Name
Call/Reject
Rejection F1
FC decision FI
Classification Accuracy
gemini-2.5 2.5-pio-rr-ooo--- ---r--rreviiew
0,8134
0.7195
0,9072
gemini-2.55 fla- preeeiewwpreeview
0.7671
0.6627
0,8715
claude-sonnet-4
0.7586
0.6058
0.9113
gpt-4.1-2025-04:
0.7864
0.6543
0.9184
gpt-4o-2024-11-22
0.6675
0.4405
0.8944
gnt-4.1-mini-22255004444
0.5866
0.2907
0,8825
gnt-5-2005-08--07
0.7482
0.6292
0.8671
gpt-4_Innano-2055.44411
0.4859
0.1347
0.8370
Qwen3-32B
0.8072
0.7023
0.9121
Qwen3-14B
0.7894
0.6786
0,9003
Qwen3-235B-A22B
0.7587
0.6051
0,9124
Qwen3-235BB-A22B-nnmt
0.5916
0.3017
0,8815
Qwen3-8B
0.6581
0.4311
0,8852
Qwcn3-30B-A39
---------B-- Instruct
0,4366
0.0088
0,8645
Qwen3-30B-A3B
0.4494
0.0294
0.8694
A.X-4.0
0.4833
0.0960
0.8706
kanama-1.5-32.5b-
1.5-322.5-iinstruot
0.4333
0.0000
0,8667
EXAONE-4.0-32B
0.5293
0.1961
0.8625
Note: Call/reject classification accuracy represents overall decision accuracy including all renuerejection True wareefffccttnncteee calls) total cotal_cases, where failed cases are counted incorrect decisions. Re- jection Fl and FC decision Fl measure class-specific performance using precision and recall for each decision type separately, excluding cases that failed to produce valid classification outputs. We report the Call/reject Classification Accuracy the Call/Reject Classification (C/R) in Figures and
21

---

Published as a conference paper at ICLR 2026
Table 10: Tool Execution: Function Calling Performance
(a) Fuction Calling Performance on English Dataset
Model Name
Key Score
Value Score
Function Name
Overall
(F1)
(F1)
(F1)
Score
gemini-2.5-prot 2.5-pro- prooprevvew
0.8406
0.8056
0,8611
0.8358
gemini-2.5-faashppeeee
0.8270
0.7859
0.8509
0.8213
claude-sonnet-4
0.8999
0.8374
0.9176
0.8850
gpt-4.1-2025-N--14
0.8815
0.8111
0,8902
0.8609
gpt-4o-2024-11-20
0.8653
0.8017
0.8862
0.8511
sp-44.1-mini20055-0--
0.8598
0.7885
0.8747
0.8410
8pt55-2025-08-07
0.8227
0.7826
0.8072
0.8042
1p--4l-nano-22055-04-1
0.7598
0.6952
0.8015
0.7522
Qwen3-32B
0.8084
0.7854
0.8301
0.8080
Qwen3-14B
0.8568
0.7856
0.8856
0.8427
Qwen3-23SB-A22B
0,8952
0.8270
0,9188
0.8803
Qwen2-235BBA222B-mms
0,8176
0,8095
0.8344
0.8205
Qwen3-8B
0,8371
0.7636
0,8711 1
0.8239
0.8151
0.7582
0.8519
0.8084
Qwen3-30B-A3B
0.8335
0.7953
0.8574
0.8287
A.X-4.0
0,8532
0.7773
0,8654
0.8320
kanana- 1.5-32.5b
0.7756
0.7116
0,8477
0.7783
EXAONE-4,0-32B
0.5352
0.5206
0.5463
0.5340
(b) Fuction Calling Performance on Korean Dataset
Model Name
Key Score
Value Score
Function Name
Overall
(FI)
(FI)
(F1)
FC Score
gemini-2.5
0.9060
0.8363
0.8823
0.8749
gemini
Semini-2.5-ffae 2.5-flash
0.8665
0.7917
0.8825
0.8469
claude-sonnet-4
0,9289
0.8369
0,9291
0.8983
gpt-4.1-2025-04
0.9237
0.8324
0.9179
0.8913
gpt-4o-2024-11-22
0,9211
0.8191
0,9102
0.8835
3p--411mini-2025--0-11
0.9076
0.7793
0,8920
0.8596
$pt-5-2025-08-07
0.9054
0.8309
0.8298
0.8554
gn=-4.1nnano-0255-04444
0.8189
0.7462
0.8086
0.7912
Qwen3-32B
0,8410
0.7629
0.8440
0.8160
Qwen3-14B
0.8754
0.7659
0,8809
0.8407
Qwen3-235B-A223P
0.9114
0.8208
0.8471
0.8598
Qwen332255BA222B-mmme
0.9109
0.8038
0.8451
0,8533
Qwen3-8B
0.8821
0,7413
0.8769
0.8334
Dwee33-30B-A3B-Innite
0,8653
0,7723
0.8603
0.8326
Qwen3-30B-A3B
0,8171
0.7358
0,8603
0,8044
A.X-4.0
0.8678
0.7775
0.8659
0.8371
kanana- 1.5-32.5b-instruet
0.8335
0.7099
0.8772
0.8069
EXAONE-4.0-32B
0.7170
0.7406
0.7509
0.7362
Note: Table shows function calling performance where evaluation metries are computed only for successful function calls. Key score, Argument score, and Function ponent function calling execution. Overall Score overall function calling performance score. Bold indicates highest performance, underline indicates second-highest performance. We report the Overall FC Score as the FC Score in Figures and
represent Fl performance for each
22

---

Published as a conference paper at ICLR 2026
F WORKFLOW GENERATION PROMPT
AI Orchestrator Prompt Intelligent Workflow Routing for LIM Agents
## System Information system_info:
issystom_infoss
# Current Workflows (workflows)
Input Classification Protocol
You are highly-skilled AI Workflew Orchestrator.
Your mission is to route user input to appropriate agents using the logic below. Always follow this 2-step decision tree when attempting any workflow creation:
## Classification Decision Tree
1. **Chitchat, no agent executicn requiredex
Condition: The input is chitchat or can be answered directly based on prior conversation history, without invoking any agents. Output: ''joon
"status": "SJCCESS", "content"; "Proper message. ex) Query handled without workflew orchestration."
...
2. +*Task Requires Exccution:
Condition: The input involves task that must performed by invoking one of the agents defined the "Agents' Information' section
(e.g., information retrieval, preduct ordering, place search, schedule lookup, etc.)
Action: Initiate update structured WORKFLOW described belox.
Workflow Design Schema
## status enum:
"pending" (waiting): Morkflow or task has not yet started, waiting to be executed
"running" (in progress): Current workflow is actively being executed. You shculd never use this status when you generate the new workflow component.
"waiting_for_inpu"" (awaiting input): Waiting for input from external
user or system, requires input to proceed to next step "completed" (finished) Al1 tasks have been successfully completed "pausea" (temporarily stopped) Workflow has temporarily suspended
# workflow type_ enum:
"independent": A self-contained werkflow that runs independently. It does not rely the output of any other workflow.
"dependent"; A follow-up workflow that depends on prior workflows. Use 'depends_on" on to reference previous workflox IDs.
"interrupt":
temporary workflow triggered by the during the execution of an ongoing workflow.
It pauses the parent workflow and performs an interim task. Once completed, the original workflow can resume. Example: While generating report, the user asks to check urgent emails.
23

---

Published as a conference paper at ICLR 2026
**Step Workflow**
### **Stop**
*+Definiticna:: sequence of tasks that are performed continuously and automatically without intervention in order achieve single objective.
**Characteristicsxx Steps progress sequentially within a single workflow and do not require user input between them.
*Example**:
..
Step 1: Check reservation
Step 2: Modify reservation
### **Norkflow**
**Definitions:: A logical grouping of tasks that are split when user input, confirmation, or branching is required. +Chaaacteristiess:: Split inte separate workflows when multiple tasks requested, user confirmation is needed, the step depends on the outcome the previous one.
Morkflow Design Guidelines
## 1. **Sqquential Execution Without User Input**
Condition::: If the process can proceed without any user interaction How to configurea:; Add all steps sequentially within single xorkflow.steps list
All steps are placed in single workflow Example: check reservation details modify the reservation
## 2. +ssser Input Requi red didway+*
Condiitions: If the process requires user input decision intermediate point
**How to configure**:
Split into separate workflows
Use 'depend_on' to indicate dependency between workflows
Each workflow should be independently executable
Downst workflows are triggered only after the completion of their dependencies
### Example: |examples)
# 3. **User Interrupts Ongoing Flew (Temporary Detour)
Cooniitionn:: If the user temporarily diverges from an active workflow to perform. separate task
**How to configure**:
Pause the current workflow and its steps (set status to 'paused') Name the new workflow as otrerppp_Lirrgggaaaooooooooooooo
1-2
etc. (e.g., peterruup_xoofflww
The original workflow may later be resumed from its paused state
### Example: (examples]
24

---

Published as a conference paper at ICLR 2026
# 4. +*User Modifies Previous Request While Preserving Downstream WorkflowS*A
*CConditinnx: When the medifies the content earlier workflow, but subsequent workflows should remain active and continue their execution
**How to configure**:
Create **new workflow* with medified content (naming: '{ [
Update the 'depend_on' field in dewnstream workflows to reference the new workflow ID
Preserve the execution chain while replacing only the modified portion
### Example: [examples}
#f# Naminn Conventions**
Regular :orkilows**: 'workflow_1" w'', 'workflox_2 2', etc.
Interrupt workflows**: 'interrupt Sinterrutt.occttoo.oooo wow.toow
interrupt.woxxtLow workflow htw.lo/w/orrgiaaalaaaaoo id)-2', etc
**Relaacement workflowsee: (werkfflow_name)11 (workflow_name) -2', etc.
# Task Consolidation Guidelines for Workflow Design
When single request includes multiple search conditions using the same tcol, **do not split into separate workflows handle them within one workflow and a single step.**
in:
Express a single refined_query that includes conditions *TTool Execution**: Whether make single or multiple calls determined by each agent based on its tool specifications not by the orchestrator.
Examples:**
(examples)
Final Notes
Always return outputs in strict JSON format.
to demonstrate how users may see responses. Use "prompt_exanpl
All workłlows not yet initiated must be marked '"status; "pending").
+*eevr include any additional text outside the JSON structure.**
whenever you generate new workflow, always include the full
definition of the previous workflow_1
the start, then append any new or modified steps. Do not discard or omit workflow_lit must be carried forward into every newly created workflow.
Do not split into separate workflews, when single user request includes multiple search conditions using the tool.
## AGENTS' INFORMATION
(a11 LLM descriptions)
25

---

Published as a conference paper at ICLR 2026
G FUNCTION CALL PROMPT WITH PROPER REJECTION HANDLING
You are Tool-Oriented JSON/XML Response Agent.
Your job is to return strictly formatted outputs reaponse input. You may use external tools when recessary, such as for real- time data, calculations, or file operations.
## TOOL CALLS
When you need use a tool, return CNLY the tool call JSCN format with no additional text
#F IMPORTANT: Parameter Extraction Rules
1. When extracting parameters, enly extract conditions that explicitly stated in the user utterance.
2. If the user utterance specifies multiple conditions for the same parameter, refer to the tool description:
If the parameter is of array type, represent it array.
the parameter is not an array, structure the output to invoke the tool multiple times, once for each condition.
# XML RESPONSE FORMATS
For all other responses (not tcol calls), return exactly one of the following XML formats:
### NOLL_CONTRAINTLLOLLATIO
Use when the user's s request violates tool usage constraints or limitations written in descriptions. This takes priority over BMAITING_USERIINPU]
<response>
Eecaatu>>TOLL.COSTTAAAI VIOLATION</status>
<constraaint..yyeeOONNTEA. / cowpeeotreeooosarantttt
<ifolaation.eessaeeeppllaaaaaaa of why the request cannot be processed
</violation__essage>>
ssugestelaateenateeeltteetttttt buggestee_aatteraatt..
approach if availables/
</response>
## SWAITING_USER_IIN
Use when you' re missing required information for tool or task (only if no constraint violations exist).
<response>
<required info>field name< ramer/equrrd info>
srroopt_message>Maal specific information do you need?</prompl message>>ssee </ </response>
## CRITICAL RULES
Tool calls: Return ONLY the JSON object, additional text
Other responses: Use ONLY the XML format, no additional text Do not mix formats or add explanatory text outside the specified structure
**FRIORITY ORDER**; Check for constraint violations FIRSI, then missing information
Handle constraint. violations using 1001_CONSTTAINNT_IIOOO ION format.
highest priority)
Hancle missing information using AWAITING_USKA_IN INPUT format (only if violations)
26

---

Published as a conference paper at ICLR 2026
H LLM USAGE
We used Claude, Gemini, GPT-4.1, and GitHub Copilot to generate synthetic data, which encom- passed the production of virtual tool results, well for Korean-English translation drafts, lan- guage editing assistance, and code development support. These models were also used to assist in drafting the paper. All Al-generated content served only as preliminary drafts and was subsequently reviewed, revised, and validated by human researchers.
27


Reviews:
[{"review_id": "lMLg5BX41m", "reviewer": "Reviewer_rgNq", "summary": "The paper introduces a bilingual (English/Korean) benchmark designed to rigorously evaluate large language models (LLMs) as agentic systems capable of orchestrating complex, multi-step workflows across 17 domains and nearly 100 virtual tools under realistic constraints. The benchmark separately evaluates workflow planning and tool calling. The dataset is manually authored and cross-checked to ensure cultural authenticity and avoid biases that can arise from synthetic (LLM-generated) data. Experiments reveal insights like a) relatively weaker performance in function calling than in planning and b) higher variance in planning abilities across models.", "questions": "1. It seems that the benchmarks assumes that there will be sub-LLMs called for executing the individual tasks. Will it make any difference if the main LLM also calls the tools and executes the tasks, which is the case in many settings?\n\n2. Why will higher Graph Edit distance be better (lines 353-354) when it is the \\textit{minimum} edit operations needed to transform one graph into another?\n\n3. The results are not represented well - the table has too many numbers and is a bit difficult to parse. I would recommend adding a plot with the results from representative LLMs to illustrate the main insights.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. By explicitly disentangling the evaluation of planning (task orchestration) from tool execution (function calling and validation), the benchmark allows for targeted diagnosis of LLM strengths and weaknesses, which is what helped reveal the aforementioned insights.\n\n2. The manual annotation of the dataset and rigorous cross-checking will ensure that the data is free from hallucinations and biases that can arise from LLM generated data.", "weaknesses": "1. The benchmarks is currently limited to only 2 languages so it can't really claim to be multilingual and handling the complexities/variations across multiple languages at the moment.\n\n2. Often there can be multiple ways (workflows) for getting to the same answer. It is not clear to me if the benchmark considers that in evaluating LLM responses or if it insists on the responses matching that in the dataset.\n\n3. It is also not clear to me if the benchmark evaluates the latency of workflows - for e.g. Does the LLM call a slow tool when a fast one would do? Does it make any unnecessary tool calls?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wrGRbjkNgx", "reviewer": "Reviewer_ehb1", "summary": "This paper introduces OrchestrationBench, a multilingual (English/Korean) benchmark focused on evaluating large language models (LLMs) as orchestrators in real-world multi-domain service scenarios. It manually constructs 17 service domains and nearly 100 virtual tools, explicitly separating “workflow planning” (represented as DAGs) from “constrained tool execution” (including call/reject decisions, parameter extraction, and decoupled validation/constraint checking). It employs metrics such as Graph Edit Distance (1-GED) and conducts experiments across numerous mainstream models, providing detailed experimental results and analysis.", "questions": "1. How is the DAG structure constructed during the planning phase? How are parallel subtasks or multi-solution scenarios handled?\n2. What do the two graphs in GEM computation refer to, and how are they obtained?\n3. How is the “manually constructed & multi-round review” dataset specifically implemented? For example, how is broad applicability across model architectures and deployment contexts ensured? How is scenario design refined to cover a wide range of user requests? Has a reference methodology been established?\n4. What tasks are handled by human annotators versus LLMs in data construction? How is model contamination avoided?\n5. What does “living benchmark” mean in Contribution (5)?\n6. Compared to existing tool-use or llm orchestration benchmarks, what makes OrchestrationBench unique?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "1. **Clear separation of planning and execution**: By decoupling “planning” from “function calls/constraint verification”, this approach enables more granular evaluation than “end-to-end” configurations. This facilitates pinpointing specific model deficiencies, proving highly valuable for driving subsequent methodological improvements.\n2. **Multi-domain and cross-language dataset**: OrchestrationBench manually annotated nearly 100 tools across 17 domains, covering a broad range of real-world service areas. This makes it highly relevant to consumer and enterprise-level LLM deployments, positioning it as a practical benchmark. It includes both English and Korean cultural contexts, filling a gap in Korean-language benchmarking.\n3. **Comprehensive model and metric evaluation**: Experiments were conducted across numerous closed-source and open-source models, incorporating precise, interpretable metrics such as 1-GED for workflow planning similarity, multiple F1 variants for tool execution, and consistency checks between human and LLM evaluators (Cohen's Kappa 0.63). Experimental tables clearly present results and support the paper's conclusions.\n4. **Insightful empirical findings**: As shown in Tables 3 and 4, empirical results reveal significant gaps between different models' tool execution and planning capabilities. This demonstrates models' deficiencies in planning, confirming the benchmark's structured evaluation as a crucial discovery.", "weaknesses": "1. **Lack of related work comparison**: The paper fails to provide explicit comparisons with similar benchmark studies, making it difficult to position OrchestrationBench within the broader field and assess its contributions. For instance, REALM-Bench focuses on intensive planning in real-world domains, and ThinkGeo incorporates planning and tool evaluation. The paper lacks direct discussion and comparison with these works (not limited to the mentioned above). Lack of a comparative table highlighting uniqueness and contributions.\n2. **Main contributions lack focus**: The paper divides contributions into five items, lacking a central thread. Some ideas (e.g., separating planning and execution in ThinkGeo) have precedents, and the proposed GED metric does not sufficiently justify its uniqueness and design rationale.\n3. **Writing emphasis imbalance**: Substantial text is devoted to textual case descriptions (case study diagrams are recommended for clarification), while critical details like benchmark construction (Section 3.3) are underrepresented. For instance, Appendix E mentions using large models for data synthesis, yet Section 3.3 omits how these models were applied in dataset construction.\n4. **Lack of supporting diagrams**: Visual diagrams to aid understanding of the benchmark architecture, data flow, and evaluation process are absent. Existing figures are overloaded with text, failing to enhance comprehension and resulting in poor readability.\n5. **Insufficient explanation and analysis**: The authors use 1-GED to quantify workflow similarity but do not clarify the correlation between GED and real-world task success rates. Similarly, Section 4.1 lacks explanation and quantitative analysis for weight settings like selection errors = 0.8 and status errors = 0.2.\n6. **Overall logical organization is disorganized**: Section transitions are loose, and the benchmark architecture, construction, and evaluation fail to form a coherent narrative chain, undermining readability and credibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EyhCUlH17m", "reviewer": "Reviewer_M4wd", "summary": "This paper proposes OrchestrationBench, a multilingual benchmark that systematically evaluate workflow-based planning and constraint-aware execution. OrchestrationBench spans 17 domains with roughly 100 tools and is constructed through manual and independent efforts. Authors also provide evaluation results for the mainstream open & closed LLMs on the proposed OrchestrationBench.", "questions": "1. Authors mention that “to ensure quality, multiple rounds of review and cross-checking were conducted, with inter-annotator agreement maintained at a consistently high level”. What are the statistics? For example, how many rounds and what is the average inter-annotator agreement level?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper studies how to properly evaluate LLM performance in complex and dynamic real-world deployment scenarios, which is a critical topic in current research.\n2. OrchestrationBench is manually constructed by human experts and covers 17 domains with around 100 tools.\n3. This paper properly illustrate and motivate the ideas with sufficient examples. The writing is clear.", "weaknesses": "1. As a multilingual dataset, OrchestrationBench only has examples from English and Korean, which seems limited.\n2. In the evaluation, authors employs LLM judge with GPT-4.1 to evaluate selected models including GPT-4.1 and other variants, which introduces evaluation bias and makes the evaluation results less convincing.\n3. Another observation is that the best model performance achieved on OrchestrationBench has already approached to 90%. One concern is how quickly this benchmark will be saturated and how long it can serve to guide the development of the next generation’s LLMs.\n4. Some critical system designs lack sufficient justification. For example, authors mention “weight selection errors more heavily (0.8) than status errors (0.2)” without any ablation results to justify the choice which leads to concerns about how robust the OrchestrationBench evaluation results are to this config.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FhWIi2rgzd", "reviewer": "Reviewer_rczq", "summary": "This paper propose an multilingual (English/Korean) benchmark to  evaluate LLMs’ workflow-based planning and constraint-aware tool execution across 17 domains with ~100 virtual tools. It differs from prior work by separating planning and tool execution (assessing selection, argument handling, etc.), uses manually annotated, culturally adapted content to avoid model biases. Experiments on SOTA models show consistent function calling but varied planning capabilities—highlighting the need for structured planning evaluation.", "questions": "please refer to weakness", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "*   The paper targets the \"evaluation of LLM agent planning and tool use,\" addressing a key gap in the current field.&#x20;\n*   Through experimental analysis, the paper draws important conclusions. For instance, it  identifies Gemini as the model with the strongest planning capability. Such findings can directly guide real-world applications.", "weaknesses": "*   Validation relies solely on GPT-4.1 as judge, which may introduce implicit biases toward the outputs of specific models, leading to deviations in evaluation results. It would be better to  calculate the consistency between multiple judges (Claude-sonnet-4, and human annotation) to enhance the reliability of assessments.\n*   The current benchmark uses virtual tools and does not integrate real enterprise APIs. It cannot simulate API-related constraints (e.g., error feedback) and dynamic environments in real scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kx2UQ7ogbJ", "reviewer": "Reviewer_rKrV", "summary": "- introuces a (offline) benchmark for orchestration and tool use, both korean and english\n- manually constructed dataset of 17 representative domains and nearly 100 virtual tools\n- define formal evaluation scores based on graph edit distance F1 score for tool execution accuracy\n- run evaluation for multile models, including GPT-4/5, Claude, Gemini, Qwen", "questions": "1. Workflow diversity and taxonomy: Could you provide a clearer taxonomy or summary table of workflows? It would help readers understand what orchestration capabilities are actually being tested.\n2. Alternative valid workflows:  How does the benchmark handle cases where a model produces a different but functionally valid workflow? Is there any provision for multi-reference or semantic equivalence scoring beyond Graph Edit Distance?\n3. Dataset scale and representativeness:  Given that the dataset includes roughly 200 sessions per language, do you consider this sufficient to generalize across 17 domains? Are there plans to expand or semi-automate data generation to increase coverage?\n4: Offline vs online evaluation – Have you considered integrating OrchestrationBench with an interactive or simulated execution environment (e.g., MCP or API sandbox) to test dynamic replanning and feedback-based adaptation?\n5. Extensibility in practice: The paper describes OrchestrationBench as a “living benchmark.”. Wat does this mean in practice?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- well motivated and important proble. bencmarking for more complex and reaslistic settings is a key gap\n- clear method formulation with separation into tool use and orchestration\n- comprehensive emperical study with relevant (commercial), state of-the-art models\n- practical significance, and splitting into tool use and orchestration leas to more actionable insights", "weaknesses": "- Offline evaluation only. while they have a manually designed gold workflow as reference for orchestration, I can imagine many cases where different workflows lead to similarly good outcomes. The offline evaluation particularly for orchestration seems limiting and is penalizing equally good workflows (I'm less concerned about the tool use aspect). Having an online environment would be deisrable (see e.g. OfficeBench, which clearly should be cited as related work)\n- No clear taxonomy of workflows that are covered. I would love to see a clear overview/taxonomy/figure which type of workflows are included in the benchmark. It is hard to understand the empasis. Is it consumer user workflows like the travel booking example, business/enterprise workflows, what domains, ...?\n- while relevant, it's limited novelty, and the paper misses to cite key references such as OfficeBench, WebArena, OSWorld, and ToolEmu, which already test interactive or multi-application orchestration. A clearer discussion of how OrchestrationBench differs from these would help position its unique contribution.\n- Limited dataset scale. Although the dataset covers 17 domains, the total size (≈220 sessions per language, ≈700 tool calls) is relatively small for evaluating general orchestration capabilities. This limits statistical robustness. larger or semi-synthetic expansion would make the benchmark more representative and useful for training-time evaluation. I'm not convinced about the usefulness in it's current state.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)