ResiliBench: Evaluating Agentic Workflow Adaptation in Stochastic Environments
Authors: Ruicheng Ao
Zeping Min
Tingyu Zhu
Wotao Yin
Xinshang Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Workflow execution
LLM robustness
Probabilistic Tool Behavior
Primary area: datasets and benchmarks
TL;DR: We introduce ResiliBench, a benchmark that evaluates LLM workflow execution under probabilistic tool failures and flawed instructions.

Abstract: We introduce ResiliBench, a benchmark that evaluates LLM workflow execution under simulated realistic conditions of instruction quality variability and tool execution uncertainty. Unlike existing benchmarks that encounter these challenges incidentally, our work makes uncertainty the primary focus of systematic study. The benchmark incorporates three key aspects: (1) modeling of probabilistic tool behaviors through parameterized error models that simulate real-world API failure patterns, (2) provision of MDP-derived workflows that maximize expected success rates, and (3) systematic evaluation of model robustness through controlled perturbations of workflow instruction quality. Our construction pipeline generates 5,040 tasks from a tool library of 30 APIs. The evaluation conducted across widely used large language models under conditions of probabilistic tool failures and varying instruction quality reveals notable performance differences. Specifically, MDP-optimal workflow prompts achieve an average success rate of 62.1\%, Chain-of-Thought prompts yield an average success rate of 50.8\%, and flawed workflow prompts result in an average success rate of 54.3\%. Our benchmark is available at https://github.com/Archer222arc/ResiliBench.

Published as a conference paper at ICLR 2026
RESILIBENCH: EVALUATING AGENTIC WORKFLOW ADAPTATION IN STOCHASTIC ENVIRONMENTS
Ruicheng Ao! Zeping Min Tingyu Zhu Wotao Yin Xinshang Wang²
'Institute for Data, Systems, and Society, Massachusetts Institute of Technology. 2 Allibaba Group US, DAMO Academy, SIORR BAIR, UC Berkeley
ABSTRACT
We introduce ResiliBench, a benchmark that evaluates LLM workflow execution under simulated realistic conditions of instruction quality variability and tool ex- ecution uncertainty. Unlike existing benchmarks that encounter these challenges incidentally, our work makes uncertainty the primary foeus of systematic study. The benchmark incorporates three key aspects: (I) modeling of probabilistic tool behaviors through parameterized error models that simulate real-world API failure patterns, (2) provision of MDP-derived workflows that maximize expected suc- cess rates, and (3) systematic evaluation of model robustness through controlled perturbations of workflow instruction quality. Our construction pipeline gener- ates 5,040 tasks from a tool library of 30 APIs. The evaluation conducted across widely used large language models under conditions of probabilistic tool failures and varying instruction quality reveals notable performance differences. Specifi- cally, MDP-optimal workflow prompts achieve an average success rate of 62.1%, Chain-of- Mhain-of-Thhugis Thought prompts yield an average success rate of 50,8%, and flawed workflow prompts result in an average success rate of 54.3%. Our benchmark is available at httptts:/// /ttp:://gtttuu www/Arooee//eeaeeeeccccee
Res Res.lieeillleehheeeee iBench
INTRODUCTION
Large language models (LLMs) have shown capabilities in reasoning, planning, and tool utilization across diverse domains (Brown et al., 2020 Achiam et al. 2023). These models have demon-
strated proficiency in applications that combine natural language understanding with tool orches-
tration (Schick et al.) 2023) Qin et al. 2023 Parisi et al. 2022). Recent advances in agent-based
systems have further expanded the scope of tool-use applications (Wang al. 2023 Xi al. 2023 Sumers et al. 2023). Workflow execution represents one area where LLMs must navigate sequences of tool interactions while maintaining coherence and achieving specified objectives, with applica-
tions ranging from software development (Fried et al. 2023 computing (Lewkowycz et al. 2022).
Nijkamp et al. 2022 to scientific
When LLMs are deployed for workflow execution in production environments, they encounter three primary categories of challenges: (1) Tool reliability issues, where APIs exhibit probabilistic fail- ures including timeouts, service outages, validation errors, and resource limitations that require adaptive error recovery strategies; and the model lacks higher-level information about tool reliabil- ity, for example, an LLM may not know the specific cause of API interruption. (2) Instruction quality variations, where users provide instructions that may be incomplete, ambiguous, contra- dictory, contain logical inconsistencies due to limited domain knowledge communication con- straints; and (3) Complexity challenges, where processes involve intricate tool dependencies, dy- namic multi-agent coordination, and other highly complex tasks, and the number of dialogue turns or token limits restricts the model's exploration behavior.
Recent research has produced various tool-use benchmarks that evaluate LLMs' capacity for API interaction, multi-step reasoning, and workflow execution (Zhuang et al. 2023 Qin et al. 2023 Li
al.
2023
Patil al. 2023), with specialized benchmarks emerging for mathematics (Hendrycks
et al.
2021
Cobbe et al. 2021 code generation (Chen et al. 2021 Austin et al. 2021), and multi-
Equul contribution

---

Published as a conference paper at ICLR 2026
Objective: Process
data from file validated and aggregated report
Executing Step I: fill__operation___r reader
Data loaded,
Task Type: data_ pipeline
Task Specification
Required Tools:
Executting Step 2: catto_processing__aee validator
reader.
data_ pwwcepssinggpprseeeeeee.. porser...)
Fallure: OPERATION FAILED. Input is not in the expected parsed format.
Execution
oww.ottoosssstieeee reader
data Reasoning The validation failed becouse the The provided plan is incornect. correct the sequence by running the porser next.
processing, validator Error Injected
Execute cntta_prceessing
Flawed
Execute bett__prccessing proeessingagggeennoo
Workflow Prompt
Execute operattortttn
Executing: pasta_pracessing_poe porser
Success, Data porsed.
Figure 1: This figure demonstrates the ResiliBench setup. On the left, we provide task specifications, a tool library, and workflow prompts with injected errors. On the right, the LLM is expected to identify errors in the provided workflow, and execute with correction.
modal reasoning (Lu et al. 2022) These frameworks have made significant contributions to under- standing I.LM capabilities (Zhou et al. 2022 Wei et al., 2022), establishing important baselines for tool-use performance under controlled conditions.
We introduce ResiliBench[] a benchmark that systematically evaluates LLM workflow execution capabilities under realistic deployment conditions involving tool reliability issues and instruction quality variations (Liu et al., 22023, Jimenez et al. (202)). While existing benchmarks (Qin et al.)
2023 Li et al.) [223 Huang et al. (023) may encounter probabilistic tool failures and instruction variability as natural consequences of working with real-world APIs and diverse instruction gen- eration, these works typically filter and curate APIs instructions minimize such issues. Our work makes two distinct contributions: Firstly, we make these uncertainties the primary focus of systematic study rather than attempting filter them out. We deliberately introduce controlled perturbations through systematically generated flawed workflow instructions and probabilistic er- ror models. This creates environments that require adaptive strategies, allowing us measure the model's 's ability to recover from API errors and adaptively replan workflows to complete tasks.
Secondly, we provide analytically derived optimal workflows through MDP optimization. Under re- alistic deployment constraints, there exist three theoretical upper bounds for success rates: (i) 100%, which may not be achievable due to round/retry limits; (ii) the upper bound achieved by policy knowing tool call procedures and able to anticipate tool errors (e.g.. knowing the random seed of ev- cry tool operation); (iii) the upper bound achieved by a policy knowing tool call procedures but only aware of the probability of tool failure. Our MDP formulation provides a tractable approximation to bound (iii), computing workflows that maximize expected success rates by reasoning over known reliability statistics.
ResiliBench consists of three primary components: (1) task specifications spanning multiple types and complexity levels, (2) tool registry modeling 30 APIs with probabilistic error behaviors and dependency constraints, and (3) multi-variant workflow guidance including both optimal execution
"ResiiBeench was originally named PilotBench. The name changed due to naming conflict with another paper at ICLR 2026.
2

---

Published as a conference paper at ICLR 2026
plans and perturbed variants. Each task is accompanied by four distinct prompt types baseline, Chain-of- Thought, MDP-optimal workflow prompt, and flawed workflow prompt designed to sim- ulate low-quality prompts in real scenarios.
Our evaluation framework shows model families and size exhibit performance differences in current LLMs when confronted with realistic workflow environments. Assessments suggest performance changes when models encounter flawed instructions and probabilistic tool failures, with success rates that may correlate with both workflow instruction quality and tool reliability levels. These findings could inform approaches for improving LLM robustness in practical workflow execution scenarios.
The primary contributions of this work are multifold:
1. Benchmark Design and Automated Construction: We present ResiliBench, an evalu- ation framework that systematically assesses LLM workflow execution capabilities under instruction variability and tool uncertainty through automated task generation from struc- tured tool libraries.
2. MDP-Based Workflow Generation Framework: We develop Markov Decision Pro- cess (MDP) framework that generates theoretically optimal execution workflows maximiz- ing expected success rates, along with seven types of systematically perturbed variants, enabling controlled evaluation of model robustness to instruction quality variations.
3. Evaluation Findings: Our evaluation reveals that models exhibit dramatically different robustness patterns when confronted with flawed instructions: GPT-4o-mini maintains rel- atively stable performance (optimal: 67.7%, flawed: 62.2%, 5.5 percentage point drop), while Gemini-2.55Fllas shows substantial degradation (optimal: 60.1%, flawed: 20.0%, 40.1 percentage point drop). We also observe emergent abilities in workflow execution where multi-step tool-use proficiency appears suddenly at certain parameter thresholds.
4. World API Integration:
We extend ResiliBench with real-world task set that in- tegrates live public APIs, serving as a complementary evaluation component that directly assesses model performance on actual API interactions. Real-world experiments reveal patterns consistent with the observations from the simulated API experiments.
The remainder of this paper is organized as follows. Section 2 details the core components, evalua- tion methodology, and benchmark statistics. Sectionßdescribes our automated construction pipeline and MDP-based workflow generation methods. Section A presents experimental results and analysis of current LLM capabilities. Section reviews related benchmarks.
BENCHMARK SETUP
In this section, we present the setup of ResiliBench. We begin by describing the core components of the benchmark data and their organization, followed by the task types and evaluation methodology used to assess agent performance. A figure demonstrating the benchmark setup is provided in Figure
1
2.1 BENCHMARK DATA ORGANIZATION
The ResiliBench dataset structured around three primary components: task specifications, tool registry, and reference workflows.
Task Specifications. The benchmark comprises 5,040 unique tasks organized across multiple di- mensions of type and complexity. Each task specification includes natural language description, structured input/output requirements, a list of required tools, execution constraints, and comprehen- sive metadata. Detailed examples of task specifications are provided in Appendix B.
Tool Registry. The benchmark includes a comprehensive Tool Registry modeling 30 canonical software APIs with probabilistic behavior. Tools are systematically organized into six functional categories: data processing, file operations, network, computation, integration, and utility, with five tools per category ensuring balanced coverage.
3

---

Published as a conference paper at ICLR 2026
Each tool definition specifies functional parameters with type constraints, structured return schemas, and explicit error models that enumerate possible failure modes. The system models five primary failure types: imput validation failures (INVALID INPUT), operational failures (OPERATION FAILED), timeout conditions (TIMEOUT), calculation errors (CALCULATION ERROR), and re- source overflow conditions (OVERFLOW) These error types reflect common failure patterns in real- world scenarios, Representative tool examples and detailed specifications are provided in Appendix
BI
Reference Workflows, Each task is accompanied by four distinct prompt types (detailed in Section 3.3; ; baseline, Thhiih,f-Thouuht, MDP-optimal workflow prompt (optimal with respect our MDP reward function), and flawed workflow prompt with systematic eπror injection. These variants enable evaluation of both essruction-followingg fidelity under high-quality instructions and robustness under flawed instruction quality.
2.2 TASK EXECUTION AND EVALUATION
Execution Environment. ResiliBench employs simulated execution environment that provides realistic tool behavior. When agents invoke tools, probabílistic simulator calculates success rates based on tool dependencies, execution history, and failure patterns, then generates appropriate suc- cess or failure responses. The simulator implements a base success rate 0.8. Complete imple- mentation details are provided in Appendix [B.3]
Evaluation Methodology. Task outcomes are categorized into three distinct levels of success. A task is considered full success if all specified tools executed correctly and in the proper sequence. Partial a success reflects substantial task completion that satisfies most requirements but does not achieve perfect execution. Fai Failure indicates that task completion is insufficient due to critical tool failures or breakdowns in execution.
The evaluation framework relies on four primary assessment criteria. Required Tools Coverage mea- sures the proportion of tools successfully executed from the task's required tool list, with full.- success requiring 100% coverage. Sequence Correctness assesses whether tools are executed in the exact order dictated by task dependencies. Output Generation verifies the successful execution of output-pun producing tools (e.g., writers, exporters, savers) as evidence of meaningful task comple- tion. Explicit Completion Signals examines the conversation history for indications from the LLM that the task has been completed. Achieving full.success requires success all four these criteria.
Additionally, two considerations are also applied. Minimum Tool Execution: This defines task-type- specific thresholds for considering partial success. A task may be rated partial.succes it meets at least two conditions, such as exceeding minimum tool execution coverage while pro- ducing the expected outputs. Termination Conditions: The evaluation also accounts for premature task termination, such as when a task experiences an excessive number of consecutive tool failures or becomes trapped in repetitive loop, which typically results in fai faiuur rating. Complete implementation details are provided in Appendix B.4
2.3 BENCHMARK STATISTICS
ResiliBench contains 5,040 unique tasks systematically distributed across multiple dimensions to ensure balanced coverage of workflow scenarios. The benchmark incorporates 30 canonical tools organized into six functional categories, providing a controlled yet diverse environment for evalu- ating LLM workflow execution capabilities. Here we present some statisties about tasks and tools. For more detailed statistical information, please see Appendix B.
Task Distribution. We classify tasks using two approaches: task type classification and complex- ity classification. Task types include five categories: complex validation pipeline tasks, complex network integration tasks, basic file processing tasks, advanced computation pipelines, and simple data transformation tasks. Complexity classification spans three levels: easy, medium, and hard. Complex validation pipeline tasks (1,520 instances, 30.2%) form the largest category, followed by complex network integration tasks (1,360 instances. 27.0%), basic file processing tasks (1,200 in- stances, 23.8%), advanced computation pipelines (640 instances, 12.7%), and simple data trans- formation tasks (320 instances, 6.3%). Tasks span three complexity levels: easy (1,520 instances.
4

---

Published as a conference paper at ICLR 2026
30.2%), medium (2,880 instances, 57.1%), and hard (640 instances, 12.7%). The distributions for both classification approaches are shown in Figure 2a and 2b
Tool Library Composition. We organize tools into six categories: computation, data processing, file operations, integration, network, and utility, with each category containing five tools, as shown in We also conduct statistical analysis of tool-required parameters, with results presented
Figure 2c
in Figure 2d The most common required parameter is opt ions, which is required by 25 tools.
Gingarka
WE
Due
77%
(a) Task type
Task complexity
(c) Tool category
(d) Required parameters
Figure 2: ResiliBench statistics showing (a) task type distribution, (b) task complexity distribution, (c) tool category distribution, and (d) required parameter distribution.
BENCHMARK CONSTRUCTION
ResiliBench employs an automated construction pipeline that generates diverse workflow tasks from systematic tool library. The construction process consists of three main stages: tool library gener- ation, task creation, and prompt generation. This automated approach enables scalable benchmark expansion while maintaining task quality and diversity.
3.1 TOOL LIBRARY GENERATION
The tool library is constructed using a two-layer categorization system that balances systematic coverage with realistic workflow dependencies. The foundation consists of category-operation matrix spanning 6 functional categories protep.oo,sssnng, file_operations, network, computation, integration, utility) and 5 operations each, yielding 30 distinct tool types. Each tool follows the {category)- booprry---rattttoooo raeiiant ]on] naming convention and includes standardized parameter templates, return schemas, and error handling specifications.
The second layer introduces semantic operation types that group tools by their workflow roles: sources (readers, fetchers), processors (parsers, transformers, analyzers), aggregators, outputs (writ- ers, posters), and utilities. This semantic grouping enables dependency modeling where processors depend on sources, aggregators depend on processors, and outputs depend on aggregators. More detailed information can be found in Appendix C.I
3.2 TASK GENERATION METHODOLOGY
Tasks are automatically generated through semantic matching between predefined operation se- quences and the tool library. The system defines five task types: basic file processin, simple data transformation, complex validation pipeline, complex network integration, and advanced compu- tation_pipeling Each task type follows standard operation sequence, such ['read', 'validate", 'transform', 'aggregate', 'write"] ] for complex validation pipelines.
The generation process uses RAG-based semantic matching to map operation steps to appropriate reader network.-
tools. For example, a 'read' operation can match to file_operre fet cher based on semantic similarity. Task variation is introduced through multiple tool choices per operation and different templates. An optional LLM enhancement phase improves task descrip- tions while preserving logical consistency, ensuring both natural language quality and structural integrity. More detailed information can be found in Appendix C.2

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of prompt types in ResiliBench.
Prompt Type
Base Components
Additional Elements
Evaluation Focus
Task description,
Basic instruction following
Baseline
Input/oulput specs,
None
Tool instructions
Task description, Input/output specs, Tool instructions
Explicit reasoning instructions
Reasoning and planning capabilities
Chain-of-Thouge
Task description, Input/output specs, Tool instructions
MDP-Optimal Workflow
Detailed execution plan
Workflow adherence
Task description, Input/output specs, Tool instructions
Perturbed execution plan with systematic errors
Error detection and robustness
Flawed Workflow
3.3 WORKFLOW PROMPT GENERATION
Each task has four distinct prompt types to evaluate different aspects of agent capability. Table vides comprehensive comparison of these prompt variants. Baseline prompts contain essential task information including description, input/output specifications, and tool usage instructions. Chain- of of-Thought (CoT)) prompts enhance baseline prompts with explicit CoT instructions that encourage step-by-step analysis. MDP-Optimal workflow prompts incorporate detailed execution plans derived from MDP formulations that account for tool dependencies and success probabilities.
The MDP-optimal workflow prompts are generated using a MDP framework that formally defines optimality in terms of expected cumulative reward for tool sequence selection under uncertainty. Our MDP formulation uses composite state representation capturing tool execution states, progress tracking, etc. The action space consists of structured tool invocations. The system implements two-phase adaptive reward strategy: an initial coverage-focused phase prioritizing tool discovery and usage, followed by a Policy optimization employs Proximal Policy Optimization (PPO) with Transformer based neural networks, mixed-preciste seexeddprecision training, and curriculum learning across five difficulty stages. The trained policy generates tool sequences that are optimal with respect to the learned reward function and state transition probabilitics, producing workflows with maximized expected success rates given the MDP's modeling assumptions. More details can be found in Appendix Flawed workflow prompts are systematically generated by introducing controlled perturbations to MDP-optimal workflows across seven categories: sequential ordering errors, tool misuse, parameter configuration errors, missing critical steps, redundant operations, logic discontinuity, and semantic drift. More detailed information be found in Appendix C.3
phase emphasizing execution order and efficiency.
3.4 REAL-WORLD API INTEGRATION
To complement simulated environment, extend ResiliBench with real-world task set inte- grating live public APIs that directly assess practical deployment capabilities.
APT Selection and Characterization. We source candidate APIs from the publ 1 ic-apis GitHub repository To identify APIs exhibiting probabilistic behaviors aligned with our benchmark design, conduct empirical reliability assessment: each candidate API is invoked 20 times measure success rate and response latency distributions. APIs are selected based on two criteria: (1) Suc- cess rate variability -enermeemmiiitic behavior with occasional failures (timeouts, rate limiting, service interruptions), and (2) Latency variability cariablity--uubstattiaa substantial variation in response times reflecting real-world network dynamics. This yields APIs naturally exhibiting stochastic behaviors that mirror our simulated error modes.

---

Published as a conference paper at ICLR 2026
Task Construction. We design 8 sequential workflow tasks based 23 selected APIs' func- tionalities and output characteristics. Tasks are constructed by analyzing each API's imput re- quirements, output schemas, and semantic capabilities to create realistic multi-step workflows. As an example, the content: creat i on task requires LLMs to sequentially call four real-world APls fPls-fetching random fact, joke, a programming quote, and stoie quote-then then compile them into a social media post draft.
Framework Integration. We align real-world components with our existing infrastructure through: (1) constructing MCP-compliant tool registrations matching our simulated tool library schema (Sec- tion 3.1 including standardized parameters, return schemas. and error classifications, and (2) for- matiing tasks to match our task specification structure (Section 3.2h. This enables uniform workflow prompt generation and evaluation methodology across simulated and real-world components. The specific tools and tasks are updated to our benchmark repository
4
EXPERIMENTS
4.1 EXPERIMENTAL. SETUP
We evaluate several models spanning proprictary LLMs (GPT-4o-mini. 03-0416-Global. etc.) and open-source LLMs (DeepSeek-V2) V3, Qwen2.5-32BB etc.). Each model is tested with 4 prompt variants per task (baseline prompt, Chain-of-Thought prompt, MDP-optimml workflow prompt, and flawed workflow prompt). We use base success rate Pbase 0.8 for probabilistic tool execution.
The framework employs interactive multi-tum execution environment with up to 10 versational turns per task. Key implementation details include: (1) Interaction protocol and execution: Models ttcol_seerch/qqeerry/tooo......
<toel_info>tool
syntax: the system en- forces one-tool-per-umm execution with dependency management. (2) Automated feedback and parameter handling: The system generates feedback messages with execution results, error in- formation, and progress indicators; format detection reminders are provided when responses lack proper syntax. (3) Error simulation: Tool failures follow predefined error templates specific to each tool's MCP specification, including modes such as TIMEOUT, DEPENDENCY _ERRO,, and INVALID UT. For details, see Appendix B In addition, provide testing interaction exam- ples in Appendix A
name</tocl.ing
and
Our experimental analysis of LLM workflow execution capabilities demonstrates:
Different robustness patterns to instruction quality: Models exhibit dramatically dif- ferent robustness patterns when confronted with flawed instructions. For example, GPT- 4o-mini maintains relatively stable performance across instruction quality variations (opti- mal: 67.7%, flawed: 62.2%, 5.5 percentage point drop), while Gemini-2.5-FFan shows substantial degradation (optimal: 60.1%, flawed: 20.0%, a 40.1 percentage point drop) (Section 4.3
Emergent abilities of workflow execution: Through experiments on the Qwen2.5 series, we observe emergent abilities in workflow execution, where the model's multi-step work- flow execution ability appears suddenly at certain parameter thresholds rather than scaling smoothly (Section 4.4).
Real-world validation:
We extend our evaluation with a real-world test set integrating live public APIs. Real-world experiments reveal patterns consistent with the observations from the simulated API experiments (Section 4.5 and Appendix E.3
4.2 OVERALL PERFORMANCE RESULTS
T the performance results using baseline, Chain-of Thouugh,,, and MDP-optimal work- flow prompts. When the models are presented with the MDP-optimal prompt, success rates range from 56.8% to 67.7% across both proprietary and open-source models. Among proprietary models, GPT-4o-mini achieves 67.7% success rate, followed by GPT-5-mini (60.7% (0) and Gemini-2.5-Flasi (60.1%). Open-source models show competitive performance, with 3lama-33.3-05 reaching 66. %.
B

---

Published as a conference paper at ICLR 2026
Table Performance: Baseline, Chain-of-Thought, and MDP-Optimal Workflow prompting.
Baseline
Chain-of-Thouue
Optimal Workflow
Model
Full
Partial
Fail
Full
Partial
Fail
Full
Partial
Fail
GPT-4o-mini
50.5
46.5
3.0
56.1
43,9
0.0
67.7
31.2
1.1
03-0416-Global
52.7
44.9
2.4
48.9
45.6
5.6
58.5
35.1
6.4
Gemini-2.5-Flash
54.3
44.5
1.1
51.1
44.7
4.2
60.1
36,7
3.3
GPT-5-mini
52.0
46.0
54.3
45.7
0.0
60.7
35.5
3.8
Llama-3.3-70B
47.8
42.5
9.6
43.6
43.6
12.7
66.1
30.9
3.0
Qwen2.5-32B
52.5
43.8
3.7
51.7
45.0
3.3
65.0
31.9
3.1
DeepSeek-V3
50.0
50.0
0.0
50.0
46.9
3.1
56.8
39.0
4.2
Avg
51.4
45.5
3.1
50.8
45.1
4.1
62.1
34.3
3.6
4.3 SENSITIVITY ANALYSIS
50
0774
74.2%
77
75.2%
00:7%
72.4%
Uamte 3
67.4%)
57.1%
Rate
56.9%
57.4%
50.4%
51.55
$2.9%
Lande Chainol Mising Regrrtaa Sarart
Logk Sesaetic Metuncent
Simgl Dato
Conr par cabee
ERIk F8s Types
Complic SEouS Marnoeoorrgaa
(a)
(b)
Figure 3: Performance analysis across representative models: (a) Prompt robustness analysis across 7 categories of flawed workflow prompts, and (b) model performance across different task types.
We conduct sensitivity analysis to examine model performance across two dimensions: instruction quality variation and task complexity progression. This analysis evaluates performance under the MDP-optimal workflow and flawed workflow prompts, and five task complexity levels.
Instructions Quality Sensitivity. Table 3 shows that MDP-optimal workflow prompts achieves higher average success rate (62.1%) compared flawed workflow prompts (54.3%). Besides, models exhibit dramatically different robustness patterns when confronted with flawed instructions. Specifically, GPT-4o-mini maintains relatively stable performance with only 5.5 percentage point drop (optimal: 67.7%, flawed: 62.2%), suggesting implicit eтror correction capabilities. In con- trast, Gemini-2.5-Flash shows substantial performance degradation with 40.1 percentage point drop (optimal: 60.1% flawed: 20.0%). This indicates that tolerance to flawed instructions repre- sents distinct capability dimension. To analyze the specific failure modes, examine seven types of systematic perturbations in Figure 3a revealing that advanced models are particularly resilient to ordering and parameter errors but more vulnerable semantic drift.
Task Complexity Sensitivity. Figure 3b demonstrates consistent performance degradation pat- terns as task complexity increases. Representative models show performance decline from simple content analysis tasks to complex computation pipelines: GPT-4o-mini (7..4% to 53.7%). GPT-5- mini (69.9% to 53.7%). Qwen2.5-32B (73.5% to 53.7%), and Llama-3.3-70B (67.4% to 52.8%). This degradation reflects the increasing cognitive demands of workflow execution. where advanced computation pipelines require more sophisticated reasoning about tool dependencies and execution chains compared to basic file processing tasks.
8

---

Published as a conference paper at ICLR 2026
4.4 SCALING ANALYSIS
We conducted additional experiments on the Qwen2.5 series to investigate the relationship between model size and workflow execution capabilities. The Qwen2.5 series exhibits non-linear scaling patterns (Table 4p, with performance varying substantially: 0.5% (3B). 63.5% (7B). 65.0% (32B), and 65.0% (72B).
The substantial jump from 3B to 7B (63.0 percentage point increase) suggests the emergence of basic tool comprehension capabilities around this scale, while performance plateaus from 32B 72B indicate diminishing returns. This pattern reflects emergent abilities in workflow execution, where tool-use proficiency appears suddenly at certain parameter thresholds rather than scaling smoothly. This observation aligns with discussions such as (Berti et al. 2025).
Table 3: Performance comparison: Optimal vs. Flawed Workflow prompting.
Optimal Workflow
Flawed Workflow
Model
Full
Partial
Fail
Full
Partial
Fail
GPT-4o-mini
67.7
31.2
62.2
34.6
3.2
03-0416-Global
58.5
35.1
6.4
53.8
39.2
7.0
Gemini-2.5-Flash
60.1
36.7
3,3
20.0
12.8
67.3
GPT--5-mini
60.7
35.5
3.8
63.5
36.1
0.4
1.ama-3.3-70B
66.1
30.9
3.0
59.5
36.2
4.4
Qwen2.5-32B
65.0
31.9
3.1
62.9
35.9
1.2
DeepSeek-V3
56.8
39.0
4.2
58.4
39.7
Avg
62.1
34.3
3.6
54.3
33.5
12.2
Table Qwen2.5 series scaling analysis with detailed performance metrics.
Model Size
Full Success Rate
Partial Success Rate
Failure Rate
Qwen2.5-3B
0.5%
0.5%
99. %
Qwen2.5-7B
63.5%
30.2%
6.3%
Qwen2.5-32B
65.0%
31.9%
3.1%
Qwen2.5-72B
65.0%
32.0%
3.0%
4.5 RESULTS ON REAL-WORLLD TASK SET
To assess the transferability of our simulation findings, also evaluate the models real-world tasks. Table 5 presents performance across 8 tasks using 23 live public APIs. Consistent with our simulation results, models exhibit different robustness patterns when confronted with flawed instruc- tions. Specifically, GPT-4o-mini shows moderate degradation (optimal: 42. %. flawed: 34.3%, a 7.8 percentage point drop), while Gemini-2.5-Faash exhibits more substantial degradation (optimal: 55.3%, flawed: 34.1% 1% a 21.2 percentage point drop). We present the specific distribution of API error rates and types occurred in our experiments Section E.4
RELATED WORK
Recent studies on large language models (LLMMs) have produced various benchmarks and frame- works for evaluating tool-use reasoning, workflow execution, and LLM robustness in complex tasks. For example, ToolQA (Zhuang et al. 2023) analyzes tool-use reasoning by differentiating between knowledge-based and tool-reliant questions but is limited to single tool invocations.
Some advanced benchmarks attempt to evaluate LLMs in more realistic scenarios. ToolBench (Qin et al. 2023) extends evaluations a broader set of APIs and enable both single- and multi-tool tasks with curated APIs and instructions. MetaTool (Wang et al.] (024) uses meta-task augmentation improve tool-use knowledge transfer. ToolAlpaca (Tang al. 2023) studies generalization in

---

Published as a conference paper at ICLR 2026
Table 5; Real-word Task Set Model Performance
Optimal Workflow
Flawed Workflow
Model
Full
Partial
Fail
Full
Partial
Fail
GPT-4o-mini
42.1
38.6
19.3
34.3
34.3
31.4
Gemini-2.5-Flash
55.3
41.2
3.5
34.1
33.5
32.4
GPT-5-mini
47.6
45.2
7.1
36.6
36.3
27.0
Llama-3.3-70B
39.6
27.1
33.3
25.4
25.4
49.3
Qwen2.5-32B
55.3
44.7
0.0
42.3
38.8
18.9
DeepSeek-V3
40.2
39.3
20.5
36.0
35.4
28.6
Avg
46.7
39.4
13.9
34.8
34.0
31.3
Table 6: Comparison of workflow-orienter LLM benchmarks. Comparison involves 5 aspects: Planning:
benchmark requires the LLM to design sequence of tool calls. Tool Choice: benchmark measures correct selection of the appropriate tool/API. Tool Call: benchmark checks syntactic and semantic correctness each invocation. Multi-Step: tasks demand two or more consecutive calls. MCP Protocol: tools are specified with the Model Context Protocol (or equivalent structured schema).
Benchmark
Planning
Tool Choice
Tool Call
Multi-Step MCP Protocol
ToolQA ToolQA(Zhuan et al. 2023
No
No
ToolBench(Qnn
Yes
Yes
Yes
No
MetaTool Wang 2024
No
Yes
No
No
ToolA|paca [ang |
No
No
Yes
No
No
API-Bank (Pl-Bank(LLicta))
2123
Yes
Yes
Yes
Yes
Yes
SOP-Benehs [Nundi 2025
Yes
Yes
Yes
Yes
No
Multi-Mission 1ool Bench Yu ct al. 2025
Yes
Yes
Yes
Yes
No
TheAgentCompang (Xu et al. 2024
Yes
Yes
No
Yes
No
Ours (this work)
Yes
Yes
Yes
Yes
Yes
smaller models, mainly focusing on simple tool use. Broader benchmarks such as API-Bank (Li
et al. 2023). SOP-Bench (Nandi al. 2025). and Multi-Mission Tool Bench (Yu et al., 2025) examine planning, procedural adherence. and adaptability, yet mostly emphasize dialog, industrial
SOPs, changing missions. TheAgentCompany (Xu et al. (024) studies real-world digital tasks but does not deeply examine workflow quality and its effect on decision-makinn We present detailed comparison between our benchmark and others in Table
CONCLUSION
We introduce ResiliBench, benchmark with 5,040 tasks from 30 APIs that systematically evaluates LLM workflow execution under tool reliability issues and instruction quality variations. Our evalu- ation reveals dramatically different robustness patterns across model families and emergent abilities where multi-step proficiency appears suddenly at certain parameter thresholds.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, all components the ResiliBench benchmark are publicly available. The complete tool library, the full set of 5,040 generated task specifications, and the source code for our automated construction pipeline are accessible at our anonymous GitH-ub repository: https htggtttuuuu com. haam/Archer2222a extensive details of our methodology in the appendices.
l Bench Furthermore, have provided
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical
10

---

Published as a conference paper at ICLR 2026
report. arXiv preprint arXin:2303.08774,
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Mich Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07700 2021.
Leonardo Berti, Flavio Giorgi, and Gjergji Kasneci. Emergent abilities in large language models: A survey. arXiv preprint arXiv:2503.05788. 2025.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.033: 107.03377 2021.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprini arXiv:21 10.142680..1166, 2021.
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang. Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. In International Conference Learning Representations, 2023.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint wrXiv:2103.03877 2021.
Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, et al. Metatool benchmark for large language models: Deciding whether to use tools and which to use. arXiv preprint arXiv:2310.03 128, 2023.
Carlos E Jimenez, John Yang. Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.006700 2023.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag. Theo Gutman-Solo. et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206. 2022.
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for cugl-nu llms. arXiv preprint wrXiv:2304.002444 2023.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating Ilms agents. arXiv preprint arXiv:2308. 03688, 2023.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
Subhrangshu Nandi, Arghya Datta, Nikhil Vichare, Indranil Bhattacharya, Huzefa Raja, Jing Xu, Shayan Ray, Giuseppe Carenini, Abhi Srivastava, Aaron Chan, et al. Sop-bench: Complex indus- trial sops for evaluating I1m agents. arXiv preprint arXiv:2506.08119 2025.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203. 13474, 2022.
Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models. arXiv preprint arXiv:2205. /2255, 2022.
11

---

Published as a conference paper at ICLR 2026
Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334. 2023.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, al. Toolllm: Facilitating large language models master 16000+ real-world apis. arXiv preprint arXiv:2307. 16789, 2023.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539- 68551, 2023.
Theodore Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas Griffiths. Cognitive architec- tures for language agents. arXiv preprint arXtv:2309.02427, 2023.
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. Toolal- paca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05300 2023.
Guanzhi Wang. Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305. 16291, 2023.
Xiaohan Wang, Dian Li, Yilin Zhao, Hui Wang, et al. Metatool: Facilitating large language models to master tools with meta-task augmentation. arXiv preprint arXiv:2407.12877 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thoughh prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24877 2022.
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: 2023.
A survey. arXiv preprint arXiv:2309.0788.
Frank F Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, et al. Theagentcompany: benchmarking 1lm agents on consequential real world tasks. arXiv preprint arXiv:2412.14161 2024.
Peijie Yu, Yifan Yang, Jinjian Li, Zelong Zhang, Haorui Wang, Xiao Feng, and Feng Zhang. Multi- mission tool bench: Assessing the robustness I1m based agents through related and dynamic missions. arXiv preprint arXiv:2504.00623. 2025.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuur- mans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint 2205:22055.1622 2022.
Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. Toolqa: A dataset for I1m question answering with external tools. Advances in Neural Information Processing Systems, 36: 50117-50143, 2023.
12

---

Published as a conference paper at ICLR 2026
A EXECUTION EXAMPLES
To provide concrete insight into the testing process, we present two actual execution examples from evaluation logs for the gpt-4o model with optimal prompt. These examples illustrate both successful and challenging scenarios that models encounter during advanced computation pipeline tasks.
A.1 SUCCESSFUL EXECUTION EXAMPLE
In the following example, we provide an MDP-optimal workflow prompt to the LLM. The basic file processing task documentary presented as follows.
Task Type: content_analysie Prompt Type: optimal
Task Inatance:
Required Tools: htcple__ppraaios___e__eee reader', Satta_proeessin__areeess "dtta_proossiin__fieerr
Description: Load server log files (raw text/JSON) and recent customer feedback forms from the database, parse and normalize timestamps and service identifiers, and convert both sources into common CSV schema. Apply business rules filter by production services, map error codes to severity levels, and discard debug-level noise. Aggregate response times, error counts, and complaint keywords by hourly and daily intervals, compute rolling averages and spike detection, and correlate spikes with customer complaints. Deliver a trend analysis CSV and one-page summary highlighting recurring anomalies, peak degradation windows, and suggested next steps for the IT ops team.
The prompt that we provide to the LLM consists of the description of the task, detailed guide on how to perform tool search and tool call, and recommended optimal sequence for executing the tools, given as
Workflow Execution Plan:
1. Execute file_operraioon__redeer
2. Execute data_proccssinngljaeeer
3. Execute data_processing_ffleee
The execution flow in this example begins with Turn 1,
Assistant: <tool_search>file operations readers/tool_seardh
This tool search request from the agent triggers the search engine to provide a set of tool search results:
file file_operationns_redee,,, file_operationn__iiieer
file mile_operattlons_coveeett flll_oopeatiionssccnneee file tile_operatiions_coppeees
As an example, the full information file.operations.reddee returned by the tool search
operation presented as follows.
1. file atte__pprraions__eeeee reader Category: fi file_operations
13

---

Published as a conference paper at ICLR 2026
Description: This tool is specifically designed to read and retrieve data from files.
Relevance: 0.70
Paraneters:
source (string): Source location identifier [REQUTRED] - options (object): Additional options [OPTIONAL] default-
Returns:
success (boolean): Whether operation succeeded data (object): Retrieved or parsed data
metadata (object): Operation metadata Possible Errors:
INNPL: Input validation failed OPERATION_FAILLS Operation could not be completed
TIMEOUT: Operation timed out
FILE_ NOT_FOUND: Specified file not found
SERMISSION_DENIEE Insufficient permissions
The assistant then applies the tool call operation
followed by success message indicating that the LLM should move to the next execution step:
User: file_ see__oprration_reedeeee executed successfully.
Output: {
"status": "completed",
"tool": "
Inrecutiio_contert
"step": 0,
Current progress: 1 tools executed. Continue with the next step.
After several turns of tool search and tool call, the final successful execution history consists of the following records:
Step 1: ('tool': "file_operations_reedde 'output': "('status': 'completed', 'tool': 'execution_context| ('step': 0, _fille_oeerations__ee 0}}", 'error': None)
'success': True,
Step 2: ('tool:: Matta_prcesssing parser'', 'success'; True, 'output': "('status'; 'completed', 'tool'; idata__pocesssnng irtaal_ttols_xxecutet
inrecutioncconteti ('step': 1, 1))", 'error': None)
Step 3: ( ('tool': 'dtta_processing
'success'; True,
'output'; "['status'; 'completed', 'tool': 'data saapcccessingfiiteeeeeeeeeeeeeee 'total_tcol ' 2))", 'error': None)
context': ('step': 2,
A.2 CHALLENGING EXECUTION EXAMPLE
To illustrate common execution difficulties, also present challenging scenario where typical failure patterns occur.
The task instance is given
14

---

Published as a conference paper at ICLR 2026
Task Instance:
Required Tools: ('network (ertoork_fetchrr. 'data wttpppprrsssssaiiiaaeaeeeeee (ata__rrcesssnng_arrses)
Description: Pull active customer records from the CRM API and subscriber lists from the email marketing platform, and retrieve shipping-rate quotes for each customer address via the shipping rate API. Normalize and convert JSON/CSV fields, deduplicate by email, and map CRM fields to email platform fields (first_name, last_name, email, suiscription_status, lifetime_ f shipping_class) Apply business rules: prefer the record with the latest CRM last_ past_updated timestamp, set shipping_clann class "fast" if lowest quote $10, and mark Aifetimevvalue > $200 as VIP. Push updates to both systems via their APIS and produce a CSV of synchronized records plus small syne report with counts and errors.
The reference workflow:
1. Execute network_fetcher Execute data_processing Requires: data pasta_processing_paaaee
2.
3. Execute data_ patt__poccssing_ppaa
The execution results are provided as follows. For simplicity, we omit numerical details, keeping only the tool search and tool call decisions made by the model, and the execution result records.
Turn 1:
Assistant: stool_searcchhnetwrrk fetcherx/tool_searct>> User: Tool Search Results: network fetcher, network_postee network network_moniton monitor, fetwork_router, atility_tracker (along with tool information details)
Turn 2: Assistant: stoo_ealDnetook_.ecceer//oo....s User: network_fetcher executed successfully.
Turn 3:
Assistant: staol_searchddate processing valldatorr/tool_sarca User: Tool Search Results: p
network_validatee data_processing filter, data_ data_ppooessinngaggregtooo; @ata_processing_patt
Turn 4:
Assistant: <tool_call>data ttoolccalldatttpprceesin..o.ooo arreessia.aalldtoorrrooo..lll User: late_processinng_allldeet failed due to dependency issues.
DEPENDENCY PROBLEMS:
Missing dependency: 'data satta_prooesssn__aareee must be executed
first
RECOMMENDATION:
Execute the required dependencies first, then retry this tool.
Turn 5:
Assistant: <tool @tool_search-date processing parser</tool_eeachh User: Tool Search Results: drta_processingggaarstt
Sata_processinngaarregtoorr data_pproeessnnn_ttaafffooooomm fata, data_processnng_vvlldttooo
15

---

http://dtt...rroeeeoeoooooaaeeee Published as a conference paper at ICLR 2026
Turn 6:
Assistant: <tool_ @ool__cal>ddta_aae httpD//att_preeesssooaasrrr////s
User: sace_poocessing_aarreee failed.
Error: TIMEOUT: Operation timed out (after 26 seconds)
Turn 7:
Assistant: <tool_search>data processing agpregator</tooo_eerrh User: Tool Search Results: data bata_prroessinngaggrrgggooor
data_processinngffleee,
date_pprccessing__erreer,
dete_prroeesiinggttrrnfforrss. sata_eroceesinng_valddttoo
Turn 8:
Assistant:
User: dat__processinnn_gggeettto failed due to dependency issues.
DEPENDENCY PROBLEMS:
Dependency catta_processing_areee failed cannot proceed safely
RECOMMENDATION:
Execute the required dependencies first, then retry this tool.
Turn 9:
Assistant:
User: data_processing_pree failed. Error: INVALID_INPH Input validation failed (expected: JSON format)
The agent's failure stemmed from two critical errors. First, in Tum 4, it overlooked the prerequisite must
waw.cessing..llliaattt...ooooooi leading to an execution failure. Second, when aasa.ppoocssinng.allli. timeout Turn 6, the agent attempted an improper fallback. It invoked data.procsssing- aggregator, tool for which it lacked the necessary information to correctly execute, causing another exccution error. These cascading failures ultimately caused the agent to exhaust its allotted turns and fail the task.
that data-pra ppwcessing.parseeee...p
encountered simulated
B BASIC TASK FUNCTION COMPONENTS
The most basic component of the ResiliBench collection of tasks, presented as items task library. In the procedure of interacting with and evaluating LLM, each task mapped multiple types of prompts, guiding the LLM to complete the task. The LLM is expected to perform correct tool calls, check the returned results, until all steps of the task correctly finished.
In the remainder of this section, we present the task specifications. including their structure and the completion process for LLMs. We describe the tool library provided to LLMs and the simulated en- vironments that facilitate tool execution and task completion. Finally, we explain how task execution results are generated and evaluated.
B. TASKS AND TOOLS
B.1.1 PARAMETERS AND STATISTICS OF TASKS
This section provides comprehensive overview of the task library, collection of 5040 unique tasks designed for various purposes. The library is generated from multiple source files and encompasses range of task types and complexities.
16

---

Published as a conference paper at ICLR 2026
Library Composition and Statistics. The tasks within the library categorized by type, com- plexity. Table 7 provides a detailed breakdown of these distributions.
Table 7: Task library statistics.
Category
Subcategory
Number of Tasks
Task Type
basic file processing
1200
paaple_data_transfooo
320
1520
complex network integration
1360
advanced computattono pipllii
640
Complexity
easy
1520
medium
2880
hard
640
The tasks are generated based on several templates defined through the generation process. Each Task Type corresponds specific generation logic that determines the complexity and nature of the task.
basic file processing: These are fundamental tasks generated to represent simple data pro- cessing workflows. They typically involve small number of tools (2-3) chosen randomly from the available tool library. The complexity is generally set to easy, focusing on straightforward input, processing, and output steps.
simple data transformmtiion This category includes tasks that use either one two tools. Single-tool tasks are created for each tool category, often paired with optional validation or logging tools. Dual-tool tasks combine two randomly selected tools for two-step process. These are also considered easy in complexity.
complex complex.validdee validatii on pipeline: These tasks simulate multi-stage data validation pipeline. They are constructed by semantically selecting a chain of tools that follow a logical data flow: read- ing/parsing, transformin//validainn and writingeexporting. These tasks are assigned a medi um complexity.
complex network integration: This task type is designed simulate interactions with network APIs. The generator selects tools to create a sequence of fetching data from an endpoint, validating the response, and potentially posting data back. These also of medi complexity.
advanced computation pipeline: These represent the most complex tasks. They are gen- erated by identifying longer, more intricate chains of tools based on their dependencies. The gener- ator attempts to find paths in the tool dependency graph, resulting in tasks that require sequence of multiple tools to complete. These tasks are designated as hard complexity.
Task Parameters. Each task in the library defined by set of parameters that describe its char- acteristics, requirements, and expected outcomes. Table 8 provides an overview of these parameters.
Table 8: Description of task parameters.
Parameter
Type
Description
instance-id task type
String String
A unique identifier for the task instance. The category of the task (e.g.. balec_file.- processing,
file.-
description inputs
String Object
A brief, human-readable description the task. An object containing the input data and options for the task. This often includes a source file path and various processing opti
expected.outputs
Object
An object describing the expected outcome of the task, which typically includes success boolean and may contain me tadata about the results.
17

---

Published as a conference paper at ICLR 2026
Table continued from previous page.
Parameter
Type
Description
required_took
Array
A list of strings, where each string is the name of a tool required complete the task.
constraints
Object
An object specifying any constraints on the task execu- tion, such timeout in seconds and the maximum number of max max_retr's retries.
complexity
String
The complexity level of task, categorized as easy, medi um, or hard.
metadata
Object
An object containing metadata about the task generation, such as the template used, generation timestamp, and whether it was LLM-generated.
descriitio
String
The original, more detailed description of the task before any enhancements or modifications.
B.1.2 EXAMPLE OF TASK
To directly provide an idea of what task might look like, we present an example of "basic file processing task" in the task library.
The following task requires the LLM to execute sequence of file operat ions reader, data processing parser and data processing filter tools to process the input data.
"instance_id" : "task_des2d02d",
"task_ :ype: "basic_fflleprooessnnn""
"description"; "This task retrieves data from CSV file, parses it into a structured format, and filters it based on specified criteria. The final result will be a refined dataset that meets the filtering conditions.", "inputs":
"source": "data/input_flle.cm
"options":
"filter"; "true"
expeccted_otpuuts" { "success"; true, "metadata"; {
100, [Filtered_reccrdeee 80
"ooquiree_toos""". [
"fille_operatiinss_reede"""" "daca_processsingpparre"""" "data_prooessingggflleerrr
"conttraint""; {
"timeout": 300, "max meax_retries":
complexity"; "easy", metadata": {
" "template"; "basic fileffileee pro.essssse.ssiigg
"generated at"; "DD2-07111000422255503337777
"timeout": 300,
18

---

Published as a conference paper at ICLR 2026
" "semantic "semanticc.enneaation" true,
"11m geaeingeeeerddd true, "inputs_ grrriiss_eeerteeddde from": "1lm"
B.1.3 PARAMETERS AND STATISTICS OF TOOLS
The tool library consists of a set of 30 canonical tools, each designed to perform specific operation. These tools are categorized based on their functionality, such as data processing, file operations, and network interactions. This section provides detailed breakdown of the library's composition and the parameters used across the different tools.
Tool Library Composition. The tools are grouped into six distinct categories. The distribution of tools across these categories is uniform, with each category containing five tools. This balanced distribution ensures wide range of capabilities within the library. Table summarizes this distri- bution.
Table Distribution of tools by category.
Category
Number of Tools
computation
file ooperrtiionns
integration
5
network
utility
Common Tool Parameters.
The tools library share a common set of parameters that define their inputs and behavior. These parameters allow for consistent interaction with the tools, regardless of their specific function. Table 10 provides a description of the most frequently used parameters.
Table 10: Common tool parameters and their usage.
Parameter
Data Type Occurrences
Description
options
object
25
flexible object for passing additional options.
source
string
5
The source location or identifier for data input. The data payload to be processed or sent by the tool.
data
object
5
timeout
number
5
The timeout duration for the operation, in seconds.
retry eetry_count
number
5
The number of times to retry the operation upon failure.
destination
string
4
The target location or identifier for the output.
inputfformat
string
4
The format of the input data (e.g., JSON, CSV).
ourput_format
string
The desired format for the output data.
schema
object
4
The validation schema check the data against. The numerical precision for computational tasks.
precision
number
4
Tool Construction Parameter Details. Each tool is defined by a structured set of parameters, turn values, and potential errors. The following table, Table I1 provides a comprehensive overview of the fields that constitute a tool's 's definition in the library.
Table 11: Detailed description of tool definition fields.
Field Name
Type
Deseription
name
String The unique name of the tool.
19

---

Published as a conference paper at ICLR 2026
Table 11 continued from previous page
Field Name
Type
Description
description
String
A human-readable summary of the tool's purpose and functionality.
parameters
Array
An array of objects, where each object defines an in- put parameter for the tool. This includes the parameter's name, type, description, and whether it is required. An array objects describing the possible return values from the tool. Each object specifies the name, type, and description of the return value.
returns
Array
errors
Array
A list of possible errors that the tool might throw, includ- ing an error code and description for each.
dependencies
Array
A list of other tools that must be executed before this tool can run.
dependency metadata
Object
Metadata related the tool's dependencies, including its execution level and order in a workflow. General metadata about the tool, such as its category, op- eration type, and creation timestamp.
metadata
Object
canonical name
String
The official, unique name for the tool, used resolve any aliases.
aliases differentiation
Array Object
A list of alternative names that refer to this tool. An object containing details that distinguish this tool from others, including its unique purpose, key differen- tiators, and usage keywords.
original_escripttn differentiation enhanced
String Boolean
The initial, unenhanced description the tool. A flag indicating whether the tool's differentiation infor- mation has been enhanced by an L.M..
B.1.4 EXAMPLE TOOL
We also provide an example to demonstrate what tool might look like. Recall that a tool named file.operat. readee is required in executing the task example. Here we demonstrate its registry in the tool library.
"file Ffill_operations :
"name": "file_operation" _aaiins____rrr__________
"description": "This tool is specifically designed read and retrieve data from files.", "parameters":
"name": "source",
"type": "string",
"description": "Source location or identifier",
" "required"; true,
"default"; mull, " "constraints" {}
"name": "options",
"type": "object",
"description": "Additional options", "required"; false, "default": (),
"constraints": (
"returns":
20

---

Published as a conference paper at ICLR 2026
"name": "success",
"type": "boolean", "description": "Whether operation succeeded"
"name": "data", "type": "object", "description": "Retrieved or parsed data"
"name": "metadata",
"type": "object",
"description": "Operation metadata"
"errors":
"code":
"INVALID LINVALID_INPUT",
"description": "Input validation failed"
"code":
"OPERATION DITTTTIONFFALL"",
"description": "Operation could not be completed"
"code": "TIMEOUT",
"description": "Operation timed out""
"code":
"FILE NOT FOUND",
OPILE_NOT_LL
"description": "Specified file not found"
"code": "PERMISSIONDENIED",
"description": "Insufficient permissions"
"dppendencies": [],
dependency metadata";
"level": 0, "execution_order 0, "category"; "file_operations"
metadata";
"category"; "file_ "oeeltooprratooss",
"operation": "reader",
"version": "1.0.0",
"created at";
"dependency_ levell: 0,
"execution_orderr 0
"cannnical name": : "file
"fille_operatiinssrreddd""""
In summary, the registration of a tool in the tool library. also often referred to as MCP (Model Context Protocol), is presented externa tools and services. Rather than a custom protocol implementation, it uses generated JSON schemas and client libraries that define how AI agents interact with various tools. Specifically,
standardized interface system that provides structured access to
21

---

Published as a conference paper at ICLR 2026
the schemas specify (1) the input parameters (given parameters in the following) that agent needs to provide to the tool, and (2) the possible outputs of executing the tool, including return values (given returns) and enror messages (given as errors). The schema also specifies other information, including the dependencies of the tool. If a tool has other dependencies, all dependencies must be successfully executed before the tool can be called.
Provided with the task to complete, and the tool library for usage, the LLM is required to perform tool calls, and check their return values to ensure task completion.
B.2 PROMPTS
We provide the LLMs with multiple types of task-related prompis, carrying necessary information regarding the task. The LLMs are prompted to identify and execute tools to complete the task. The prompts of ResiliBench are classified into 4 categories:
Baseline prompts: Only contains basic information, such as task description, input and output, tool call instructions, etc.
2. CoT (Chain of Thought) prompts: Baseline prompts enhanced with CoT reasoning instruc- tions.
3. MDP-optimal workflow prompts: Baseline prompts enhanced with a detailed workflow execution plan.
4. Flawed workflow prompts: Optimal prompts with various flaws injected to the workflow execution plan description.
B.2.1 AN EXAMPLE OF BASELINE PROMPT
Execute a simple task.
Task: Leverage innovative multi-tool workflows to metamorphose ambiguous input into strategic insights, enhancing operational efficacy and fostering data-driven decisions, thus unlocking untapped business potential through streamlined transformation.
Input Data:
input_data: numeric array with 5 values
Expected Output: processed data: processed data with success status
Tool Search Available:
You have access to a comprehensive tool library with specialized tools for various operations.
To find relevant tools, the search command: <tool_ search query</tool_search
Examples of tool searches: <tool_search>file reader writerc/toolsseacch
-xtool_searchhdataa validatin parser</tool_seardn
<<tool_ api fetchs/tool_seardh
After finding the tools you need, execute them using: ttoo_
Instructions:
1. Analyze the task requirements
2. Search for appropriate tools based what you need to do
3. Execute the tools in the correct order
4. Complete the task and indicate when finished
22

---

Published as a conference paper at ICLR 2026
Use appropriate tools to complete the task.
B.2.2 COT REASONING INSTRUCTIONS
**Think step by step about which tools to use and why.**
Please:
1. First explain your reasoning about which tools to use
2.
Then execute the tools in the order you determined Format tool calls as: aaoo__aalbttooo_aaeeeeeeeee
3.
Begin with "Reasoning:" followed by your thought process.
Use appropriate tools to complete the task.
The instructions are directly appended to the baseline prompt form CoT prompt.
B.2.3 AN EXAMPLE OF MDP-OPTIMAL WORKFLOW PROMPT
We provide and example of workflow execution plan with summary:
## Workflow Execution Plan
1. Execute data processing__ransfooe
Reason: Semantic match for "full_descriptionn operation
Requires: data__rocessingg_prrr
2. Execute data_processing_fiitee
Reason: Semantic match for 'full_ddecrription" operation
3. Execute attaappoocessig__a____ Reason: Semantic match for 'full "full_ddscripttion" operation
4. Execute computation. iottttiiionaaayzerr
Reason: Semantic match for 'full_description" operation
Requires: data_prccessinn_aarree, data_processinggaaarraoooo
5. Execute sile_operations__ccneer Reason: Semantic match for 'full_descriptionn operation
6. Execute file file_operraee ttt_pprration__redeee
Reason: Semantic match for 'full_descripttonn operation
7. Execute data_prcessinngaaliatooo
Reason: Semantic match for 'full_description" operation
Requires: data_processirggaareee
8. Execute computatiion_calculttoor
calculator
Reason: Semantic match for 'full_description' operation
Requires: data_pprccessnn__aarss.
network_validator
9. Execute network_monito::
Reason: Semantic match for 'full_ddscriptton" operation
10. Execute data_processnnnaaggeeettee
Reason: Semantic match for dfull_ddscripttoon operation Requires: data prta_pprcessing_paarss
### Analysis:
Critical tools identified: deta_prrocsssinnfflleee,,, lata_proessiinngtraaaffooomm data_processing_aarser
44# Execution Strategy:
1. Follow the recommended sequence for optimal results
2. Use alternatives if primary tools fail
23

---

Published as a conference paper at ICLR 2026
3. Pay special attention to critical tools
Use appropriate tools to complete the task.
The workflow execution plan is attached directly to the baseline prompt to form an MDP-optimal workflow prompt. In section we will explain how the MDP-optimal workflow prompt con- structed.
B.2.4 FLAWED WORKFLOW PROMPT
There are several ways to perturb an MDP-optimal workflow execution plan to make it flawed. For example, we can introduce flaws the order of the tool execution sequence. We can also introduce missing steps or redundancies. An ideally robust LLM is expected to discover and correct the flaws.
To test the capability of the LLM, we include Sequential Ordering Errors, where tools executed in an incorrect sequence, and Tool Misuse Errors, which involve selecting an inappropriate tool that appears suitable. Parameter Configuration Errors occur from incorrect or omitted parameters, while Missing Critical Steps involves the strategic removal of essential operations. We also observe Redundant Operations, where unnecessary tools are added; Logic Discontinuity, where a tool's output is incompatible with the next tool's 's input; and Semantic Drift, gradual deviation from the intended workflow caused by replacing tools with functionally different but semantically similar alternatives. We refer to for details of the flaws.
B.3 TOOL RESULT SIMULATOR
Once the LLM performs tool call, a simulator provides a return message that is either a success or type of failure. For simplicity, create unified simulator for every tool, which calculates the success rate of a specific tool execution, then randomly samples a success or failure.
The specific simulator implementation is decomposed as follows:
Success rate calculator. The success rate calculator assigns success rate p [0. 1] for each specific tool execution. Specifically, the calculator first assigns base success rate of po 0.8. Then, dependencies of the current tool is checked. For each dependency that has not been previ- ously called, penalty of 0.5 multiplied to the success rate, For each dependency that has been previously called but not successfully executed, penalty of 0.7 is multiplied to the success rate. Further, failure histories of the current task decreases the success rate of the current tool execution. Each history failure record induces penalty of 0.9 multiplied to the success rate. In summary, have
I N. 0.7 0.
Pauceess
Pbase
where N, is the number of dependencies not called, Ny is the number of dependencies (called but) successfully executed, and N₁, is the number of historical execution failures.
Result simulator. Given the success rate p, the result simulator first randomly simulates "suc- cess" or "failure". the result is "success", the simulator creates the success-redd related return message provided in the tool's documentation. If the result is "failure", the simulator randomly selects type of failure (e.g. TIMEOUT, EXECUTTON FAILURE) from the tool's documentation.
B.4 TASK RESULT EVALUATION
The evaluation of task outcomes is performed by the Interact Exeexoo,, which determines the success of a task based on a detailed analysis of the execution history. This section outlines the different levels of task success and the criteria used for their assessment.
24

---

Published as a conference paper at ICLR 2026
B.4.1 LEVELS OF TASK SUCCESS
full.success: This level indicates that the task was completed perfectly. requires all of the specified tools to be executed successfully and in the correct sequence.
partial.success: ssuccess: This level represents task that was completed to a significant extent but did not meet all the criteria for full success. This could mean, for example, that a majority of the required tools were executed, that valid output
generated despite some tools failing.
failure: This level is assigned when the task could not be meaningfully completed. This typ- ically occurs critical number of tools fail, no output generated for essential tasks, or the execution gets stuck in loop.
B.4.2 EVALUATION CRITERIA
The executor evaluates the success of a task by assessing several key factors from the execution state. The final success level is determined by a combination of these criteria:
Required Tools Coverage: This is the most critical metric. The evaluator checks what per- centage of the tools listed the task's requi list were executed successfully. Full success requires 100% coverage.
Sequence Correctness: For a task to be considered a full_success the required tools must not only be executed but also be called in the precise order specified in the task definition.
The evaluator checks whether any tool that is expected to produce an output (e.g., tools with names like 'writer', 'exporter', 'saver') successfully executed. The generation of an output is an indicator of full success and partial success, especially for dipeline-oriented tasks.
Output Generation:
Explicit Completion Signal:
The conversation history is scanned for signals from the lan- guage model indicating that it considers the task complete (e.g., "task completed", "finished executing").
Minimum Tool Execution: For each task type, there is minimum number of success- fully executed tools required to be considered for partial.success For instance, advanced addanccdd.comuttteee www.eiioonnppppii.eeeee line requires more successful tool calls than s waw.aatttaaaooaaaatt.......
The evaluation also considers reasons for premature termina- tion, such an excessive number of consecutive tool failures or getting stuck in repetitive loop, which would typically lead to a fai Failure rating.
Termination Conditions:
A task is rated as part partial_success if it meets at least two of the conditions (e.g., has over Minimum Tool Execut ion tool coverage and generates an output). If the conditions for either full or partial success are not met, the task is marked as a failure.
C BENCHMARK CONSTRUCTION
In this section, we introduce how the tasks automatically constructed. Specifically, this includes the generation of the task documentation, tool library and prompts.
C.1 TASK AND TOOL CONSTRUCTION
The generation framework employs a category-mediatte architecture where tool categories serve as the primary interface between tool capabilities and task requirements, ensuring that generated tasks are both reasonable and logically correspond to the underlying tool ecosystem.
C.1.1 TOOL GENERATION METHODOLOGGY
We first generate the tool library, which is independent of tasks, and generated via a LLM-free rule- based process. This process highly relies categories, where the basic layer of category consists
25

---

Published as a conference paper at ICLR 2026
of a category-operation matrix consisting of 30 distinct tool types. Then, introduce semantic operations and semantics matching that re-categosi these tools, order to (1) introduce aliases to flexibly create more tools and (2) facilitate and interface with task creation.
Layer 1: Category-Operation Matrix (Tool Definition Layer). The system defines 6 categories, each with specific operations, creating 6 matrix 30 distinct tool types. The categories and their operations are given as
data-processing: parser, transformer, validator, aggregator, filter
file opeerattatonn: reader, writer, scanner, compressor, converter
network: fetcher, poster, monitor, validator, router
computation: calculator, analyzer, optimizer, simulator, predictor
integration: connector, authenticator, mapper, queue, scheduler
utility: logger, cache, notifier, tracker, helper
Within the matrix, each cppegory-operation pair (e.g.. drtaappoccesinngppiie parser, network fetcher) gen- erates unique tool with:
Specific parameter templates
Specific return value templates
Specific error handling templates
Category-specifie behavior patterns
The
matrix-based templates generate the following information of tools:
Naming Convention. Tools follow systematic (category) (operation) pattern,
ensuring consistent identification and categorization.
Parameter Assignment. The system uses rule-based logic to assign appropriate parame- ters based operation semantics. For example, reading operations (e.g., reader, parser, scanner) receive source parameters, and transformation (e.g., transformer, converter, mapper) operations receive input.format and output_format specification param- eters.
Return Value Generation. Return values are systematically assigned based on operation type. For example, operations for receiving data, e.g., reader, fetcher, scanner, returns data values. Other than the operation-specine return values, all tools provide success indicators and metadata.
Another part of return values the error information, where we also create general errors such as INVALID.ING INPUT and operation- errors such OVERFLOW for computa- tion operations.
Layer Semantic Operation Types (Workfoo/Depeendency Layer). The system then regroups the 30 operations into 5 semantic operation types based on their role in data processing workflows:
operation_types
sources': [' ('reader', 'fetcher', 'scanner', lauthenticatoo! ) "processors': ['parser', 'transformer', 'validator', 'analyzer', 'calculator'),
'aggregators': ('aggregator', 'combiner', 'merger'), Note: some are aliased
'outputs': ['writer', 'poster', 'exporter', 'notifier',, Note: some are aliased 'utilities'; ['logger', 'cache', 'tracker', 'monitor']
We have the following remarks:
26

---

Published as a conference paper at ICLR 2026
1. Cross-category semantic grouping. Operations from different categories can belong to the same semantic type. For example:
sources includes: reader (file operration))) fetcher (network), scanner (fle_oopeationn), authenticator (integration)
processors includes: parser prata_spocsssing). transformer (data_processing). valida- tor (all.pprocessnnn/eewwork)) analyzer (computation), calculator (computation)
2. Some operations have aliases. The semantic layer introduces some operations not explic-
itly in the original 30:
combiner and merger (aliases for aggregator)
exporter (alias for writer or converter)
3. Workflow-based logic. This semantic grouping enables dependency rules like:
dependency_zules
pprocessors': {
'transformer's ['parser', 'reader') ),, # transformers depend on parsers and readers 'validator': l'parser', 'transformer') ' 'analyzer': ('parser', 'aggregator 'calculator'; ['parser', 'validator']
# more rules
Based on the dependency rules, the prerequisites each tool is configured, facilitating construction of complicated workflows with dependencies.
How the Two Layers Work Together. We now summarize how the two layers of categorization work together generate tool information, and facilitate task construction. First, in the tool gener- ation phase, we use Layer 1 (category operation) to create specific tools. Each tool gets a unique name and category-specific parameter/return template. Next, move the dependency resolu- tion phase, where we use Layer 2 (semantic operation types) to determine logical dependencies.
Lastly, the task construction phase, which will be introduced shortly, combines both layers build workflow workflow-wware aware task templates. This phase uses semantic types to understand data flow patterns, and uses specific
tools as the actual implementation.
C.2 TASK GENERATION METHODOLOGY
This section describes systematic approach for generating computational tasks based on a library of available tools, ensuring that generated tasks are both reasonable and logically correspond the capabilities of the underlying tool ecosystem. The framework employs multi-layered architecture that combines predefined task templates, semantic tool matching, and large language model (LLLM) augmentation to create diverse, executable task instances.
Foundational Task Categories The system first defines five types of tasks, including basic file processing, simple data transformation, complex validation pipeline, complex network integration, and advanced computation pipeline.
Then, for each category of tasks, we define standard operation sequence, given as
cettatiionseuuencee
['input', 'process'], ['read', 'process', 'output hoompleerralttdattooppppllllaa.e ['read', 'validate', 'transfor'', 'aggregate', 'write'), coumplee_netwokk.inneeaaaaa ('fetch', 'parse', 'validate', Itransfor'', 'post'), ladvanced computation_pipelll ['read', 'validate', (transform', 'compute', 'aggregate', 'write'),
27

---

Published as a conference paper at ICLR 2026
The task generation process begins with structured template system that defines the fundamental characteristics of different task types. Each task template serves as a blueprint that specifies the following components:
Task requirements: A TaskRequi rement: object that defines the minimum number of tools needed, required tool operations (such as "reader", "transformer", "writer") and com- plexity constraints. This ensures that generated tasks align with available tool capabilities and maintain logical coherence.
Template structure: Each template includes a task type identifier, descriptive require- ments, and objectives that guide the generation process. The template acts as a constraint mechanism, ensuring that only feasible task combinations are considered.
Tool category analysis: The system performs automated analysis of the available tool library to identify tool categories and their associated operations. This analysis informs template creation, ensuring that templates are grounded in actual tool availability rather than abstract specifications.
Semantic Tool Matching. To connect tasks to the pre-generated tool system, the framework in- corporates semantic matching select appropriate tools for task generation. For example, when read exists in the operation sequence of a task, RAG semantic matching would lead to operations such reader, fet cher and scanner in the tool semantic operation types. This further leads to retrieving specific tools, such as file-operat wet..iinnseeeee..ee To create variations in one task category, the specific tool matching with a task operation has multiple choices (e.g.. read can corre- spond to fi le operati i ons reader or network fet Feech)r). For each task, only one specific tool selected for each operation step. Other sources of variation come from the choice of different templates.
Task Construction Example. structed.
We now provide a simple example of how a task might be con-
Step 1: Choice of tools
paaplex_validatiee pepiiinn l'read', 'validate', 'trrnsform', 'aggregate', 'write']
semantics search result: readd -> choose 'dile_operations_reedee""
'validate' -> choose "deta_proeesiinn_allldttttt'
'transform' -> choose 'derapproessinn_ttaaffoooee
'aggregate' -> choose "dra_ppoeessinn_aarrrgttor
'write' -> choose 'file_operations__wiiee
# final roqus = [a list of tools after dependency baadedency-bbeed sorting]
Step 2: Task construction
# 1. Task ID instance_id id ("task_(uuid uuid4 () ":8]
2. Task description (template-base description f"... #(i+1)"
3. Input and output (based on template and tools) requirements TaskRequi ("processig_confiq"" "Configuration for processing"),
28

---

Published as a conference paper at ICLR 2026
objectives
TaskObjective
[fxecceeee [tool}" for tool in selected_tools), " "output"
4. Difficulty (based on task type and tools) complexity = "easy" if len (tools) <= 2 else "medium"
LLM Enhancement. After the basic RAG tool-selection results task, we can also use LLM APIs to create an instance of a task.
Specifically, we provide to the LLM related RAG semantic search result together with the following prompt:
prompt f"un
You are an expert workflow designer. Based on RAG search results, design (task_desc).
Available tools (sorted by relevance) : (json. dumps (tools_info, indent=2))
Requi
Task type: (task_type)
Complexity: (complexity]
Select 3-6 most appropriate tools based on:
1. RAG relevance scores
2. Logical workflow sequence
3. Tool categories and operations
4. Input/output compatibility between tools
Design a complete task instance with:
1. A clear description of what the task accomplisher
2. A logical sequence of tools that work together
3. Realistic input data that matches the first tool's parameters
4. Expected output that matches the final tool's returns
The LLM enhancement provides a more natural and detailed task description, and ensures consis- tency (e.g., among choices of tools and description) within the task setting.
C.3 WORKFLOW PROMPT GENERATION
Each task has several corresponding prompts that guide the tested LLM to complete the task. The baseline prompts and CoT prompts can be directly transformed from the task profile in the gener- ated task library. The MDP-optimal workflow prompt generated through MDP training (details provided in De. while the flawed workflow prompt is created by inserting deliberate flaws.
We present comprehensive framework for systematically generating flawed workflows from opti- mal sequences. Our approach introduces seven distinct categories of workflow defects with varying severity levels, enhanced by semantic similarity detection through RAG techniques.
Sequential Ordering Errors (Order Flaws). Sequential ordering errors occur when tools are executed incorrect or flawed sequences, violating logical dependencies or temporal constraints.
29

---

Published as a conference paper at ICLR 2026
To inject this enror into the workflow, mutations of adjacent tool pairs. We also use the dependency violation method, which repositions dependent tools before their prerequisites, to inject more logical flaws.
the swap method, i..., creating random per-
Tool Misuse Errors Tool misuse represents the selection of inappropriate tools that appear suitable but lack the required functionality for the specific context.
To inject this error, search to identify tools with similar descriptions but different functionalities. We also use category mismatch method, which replaces tools with alternatives from entirely different functional categories, simulating gross misunderstanding of tool capabilities.
the semantic similarity method. leveraging RAG-based semantic
Parameter Configuration Errors. Parameter errors involve incorrect specification or
omission of required tool parameters.
To inject this error, we use the missing parameters method, which systematically removes required parameters based on tool specifications. We also use the type mismatch method, which introduces parameters with incorrect data types or value ranges. Parameter error injection follows the formula P Perror a required + 3 optional where and severity-dependent coefficients.
Missing Critical Steps. Workflow incompleteness through strategic removal of essential tools or validation steps.
To inject this error, we use the middle step removal method, which eliminates intermediate processing steps while preserving workflow endpoints. We also use the validation removal method, which systematically removes quality assurance and verification tools.
Redundant Operations. Introduction of unnecessary duplicated operations that in- crease computational overhead without adding value.
To inject this error, we use the duplication method, which repeats existing tools within the sequence. We also use the unnecessary addition method, which inserts tools that provide no functional benefit to the task.
Logic Discontinuity. Breaks in logical flow where tool outputs become incompatible with subsequent tool inputs.
the format mismatch method, which introduces incompatible data format transitions between tools. We also use the unrelated insertion method, which adds tools unrelated to the primary task objective.
To inject this error,
Semantic Drift (RAG-Enhanced), Advanced error patterns enabled by semantic under- standing, representing gradual deviation from intended workflow semantics.
To inject this error, use the semantic mismatch method, which replaces tools with mantically similar but functionally inappropriate alternatives. We also use the semantic drift method, which performs progressive replacement where each subsequent tool is lected based on the previous replacement, creating cumulative semantic deviation. The drift function is defined T; arg max (S(1, T:-1) x F(, (" Toriginat)) ))), where S represents semantic similarity and F represents functional equivalence.
D MDP-BASED WORKFLOW GENERATION TRAINING MECHANISM
This section provides comprehensive analysis of the Markov Decision Process (MDP) training framework used for MDP-optimal workflow generation in our system.
D.I MDP STATE SPACE FORMULATION
The MDP state space S is defined by composite representation that captures both task semantics and execution dynamics. Each state s, E S is represented as:
= (T. w; th d,
where 7 and w are task-invariant components that remain constant throughout execution, while d, 6, and & capture the dynamic execution state.
30

---

Published as a conference paper at ICLR 2026
Static Task Components (T, The task identification {task_id, task-type, task tojeccije and semantic feature vector w E I220 encode task requirements (input/output needs, domain type, complexity level) as normalized features. These components extracted once at initialization and provide consistent task context for all subsequent decisions.
Tool Execution States Component (1))
V = (o where E Σ
The tool execution status set D contains:
Σ [NOT ATTEMPTED, QUEUED, RUNNING, SUCCESS,
FAILED, TIMBOUT, DEPENDENCY FAILED)
where T represents the available tool set and each o; tracks the execution status of tool i.
Progress and Workflow Component (6)
{p, ki Machieved, Mexpocted: eseg
where:
p [0, 1] denotes overall task progress
k E N represents the current workflow step
Machieved C &Mespected are milestone sets
Gseq
[1,, .2,,,,, Lk] the tool execution sequence
RAG-Enhaneee Context Component (g) This component integrates retrieval-augmentted gener- ation capabilities:
RRag Eçache, Csemantic
Hoelectoon Teandidates
where:
Reagi RAG search results mapping semantic operations tool-score pairs
Eeache: Pre-computed embedding similarity scores for all tools
: Semantic confidence scores per tool
: Tool selection history with metadata
: Capability-type candidate-tools mapping for fallback selection
State Encoding for Neural Networks The composite state is encoded into a fixed-size vector S, E IRd through concatenation and normalization:
S; Concat(encodo(()) encode (0), encode(().w)
where encode(.) transforms discrete components into continuous representations using one-hot encoding, progress normalization, and embedding lookups.
This multi-faceted state representation enables the MDP to capture both structural workflow depen- dencies and semantic task requirements, facilitating more informed action selection complex tool orchestration scenarios.
D.2
MDP ACTION SPACE DEFINITION
The action space A consists of structured actions that operate on tool orchestration and workflow management. Each action A is represented composite tuple:
(a,1,3,0,c)
31

---

Published as a conference paper at ICLR 2026
where the component refers to action (with respect to the tool), t is the specific choice of target tool, and / is the consequent relevant information. Specifically, we introduce the components as follows:
Action Type Component (a)
The action type a belongs to a finite set of predefined operations:
Q E Asye
{INVOKE TOOL, VALIDATE OUTPUT, RETRY TOOL, RECOVER RECOVER.ERROR, ERROR, CHECK DEP DEPENDENCIES, CREATE SHEATE_CHECKPOING REETEOENNTEEETEEEECCCCEOO RESTTORE_CHECCPOE NO OP, PARALLEL EXECUTE)
Tool Target Component (i) The tool identifier E TU {0} specifies the target tool for execution,
where T represents the available tool set and / indicates specific tool target.
Semantic Enhancement Component (g) The RAG-enhanced component / = (Asem Sare, Tall) contains:
Seem E [0, 1]: Semantic relevance score from embedding- search
Sar E (rule, embedding, hybrid, pattern): Information source type
Tan S T: Alternative tool candidates
Parameters Component (0) Action parameters 0 contain execution-specific configuration key-value mapping, enabling flexible parameterization for different action types.
Confidence Component (c) The confidence score [0, 1] represents the system's 's belief in the action's appropriateness for the current state, computed through multi-factor assessment.
D.2.1
ACTION FILTERING MECHANISM
Valid actions at state s are determined through multi-stage filtering process F S 24;
Avalia F semantic ddepentency o F Fconstraint (A)
where:
Constraint Filter (F constraint): Removes actions violating basic execution constraints:
F Feomstraina (A) = {a A :status(a..) # SUCCESSA
retries(a.1) < 3A
(a) V irunning tools(s:) = 0}
Here a.ı represents the choice of tool action a.
Dependency Filter (F dependency): ) Ensures prerequisite tools are successfully executed:
Faependency (A') {a A' Vd deps(a..), tha SUCCESS)
Semantic Filter (F : Applies multi-source confidence scoring:
F Fscmantic (A") {a E A Ccomposite (St, a) Tibreshold }
The composite confidence combines multiple information sources: Coomposite (S,, a) = Σ wi c,(s,, a)
(1)
with weights w [ratee: Wembeed Waattrm: Wrack] = components:
[0.25, 0.30, 0.25, 0.20] and individual confidence
32

---

Published as a conference paper at ICLR 2026
Crule (8t; a) Jlignamen (w. ops(a.)))
Cembed (8t, a) Ç. eache (a.t)
Cpattern (t,, a) max Pecore l extends pl
pEp
Crask (St, a) preference(r..ask.tyye.. a.t)
D.2.2 FALLBACK ACTION GENERATION
When (satia(Ee)] S
(only NOLOP available), the system employs progressive fallback strategies:
Algorithm Fallback Action Generation
1: if canddidaaess * 0 then
2:
Add top-2 RAG candidates with c = 0.35
3: else if workflow step 10 then
4: Add dependency-free tools with c == 0.30
5:
else
6:
Force recovery actions: a E {RECOVER ERROR, RESTORE CHECKPOINT]
7:
end if
D.2.3 ACTION SPACE CARDINALITY
The total action space size scales as:
|A| AAyyll (I + 1) + Ammeal
where |mmeul represents nol-independeet actions (NOLOP, CREATE.CHECKPOIN., etc.). However, the effective action space at any state is typically [Avatia(ss) < |AA] due to filtering constraints.
This structured action representation enables the MDP to maintain semantic coherence while pro- viding sufficient flexibility for complex workflow orchestration scenarios.
D.3 MULTI-PHAN ADAPTIVE REWARD FUNCTION
The reward function implements sophisticated two-phase training strategy that adapts based on the agent's learning progress. This design addresses the fundamental challenge of learning both what tools to use and when to use them.
D.3.1 ADAPIIV STRATEGY
The core insight is that tool selection learning requires different optimization objectives at different stages:
Reoverage (8t, 8141) if Psuccess <
aduutt
R(st, Ge. 8t+11) Roequence (St, Ot, 81+1) otherwise
where Pruccess represents the current success rate and Aadupt 0.3 is the adaptation threshold.
Phase I: Coverage-Focused Learning (Psuccess < 0.3)
During initial learning, the agent must discover which tools are relevant for different task types. The reward function prioritizes exploration and tool discovery:
Raweerage Resploration + Raiscowery + Reumpletion
Rexploration Olattempt 1 [tool attempted] + Orpovel
1 [firstt attempt)
Raiscovery Brequired 1 E Trequised] [sucesss]
Reompletion Yprugress Ap + Smilestume
|M++1 Mal
33

---

Published as a conference paper at ICLR 2026
with reward weights:
O'amempe
3, nnoell 30, Prequized 50, Vprrogress 100, mmitestone 40.
The key insight is that any successful execution required tools receives substantial rewards, re- gardless of execution order.
Phase II: Eequeence-pttimize Learning (Psnccess 0.3)
Once basic tool usage is learned, the focus shifts to optimizing execution sequences and workflow efficiency:
Resquence
Ronder + Refficiency
Reompletion
15 Teorroot.cc
if perfect sequence order
Rorder
5 |Tnear_seql
if near-correct order otherwise
-5 E. |Pactua(()) (expected(1)
max(0, Kearget
k))
Reffeiincy = T]step
Merror
1 (etorll 0]
kinrget
where inctual (1) and Rexpected(1) represent actual and expected positions of tool in the execution sequence.
D.3.2 UNIVERSAL REWARD COMPONENTS
Several reward components operate consistently across both phases;
Progress Incentives:
Rerogress Ybase Ap + Yearly Ap 1[k < 10] + Nate Ap 1[p 0.8]
RAG-Enhanced Semantic Alignment:
Rsemantic
>rag Srag (t) Apattem Spatten (esog: 6) + Ausk Stask 1)
where Smg(1) (t) is the RAG similarity score, Spatee captures learned sequential patterns, and Smsk rep- resents tosk alignment.
Terminal Rewards:
Upon episode completion, substantial rewards are distributed based on both success and training phase:
150 + 50 Tcoverage
if coverage phase and success
Rierminal
100 + 50 r Tsequence
if sequence phase and success if failure
max (0,50 50 Prima)
where Teoverage
(Tazened Traginal ITT (Tmpinol
and Fsequence measures sequence correctness.
D.3.3 PENALTY STRUCTURE
The penalty system is designed to be adaptive and minimal during early learning:
Rpenalty PRemrr + Prepetition Rstagnation
if Pausesss < 0.1 if of0.1 S Psuccess < 0.3
Remo 2 Aewxal
Aevocal
otherwise
Prepetition (count(l, evey) 1) 1[count(() 2 2]
Ratagnation = 10 1[a = a;-1 = NO 02]
where during coverage phase and 5 during sequence phase.
34

---

Published as a conference paper at ICLR 2026
D.3.4 KEY DESIGN PRINCIPLES
This reward architecture embodies several key principles:
Progressive Complexity: Early learning focuses on tool discovery; later learning opti- mizes execution patterns
Semantic Guidance: RAG-enhanced rewards align tool selection with task semantics
Adaptive Penalties: Error tolerance decreases as competence increases
Terminal Differentiation:
Final rewards depend on both success and current learning ob-
jectives
The phase transition Psuccess 0.3 ensures that agents first master basic tool usage before attempt-
ing to optimize execution sequences, leading to more stable and efficient learning convergence.
D.4 MDP ENVIRONMENT DYNAMICS
This section details the core environment mechanics that govern state transitions, completion crite- ria, and adaptive behavior in our MDP framework. Unlike conventional MDP environments with static rules, our system implements dynamic thresholds and curriculum-adee aaaltiive mechanisms that evolve during training.
D.4.1 ENVIRONMENT STEP EXECUTION
The environment step function orchestrates the complete MDP transition process through three quential phases:
MDPStep(ss, ax) (sc+1 Te, done)
where each component is computed through dedicated sub-procedures:
8 State satteTrranition(f ox)
Tz AdaptiveReward(s:.cu des de, 11+1)
done CompletionCheek((4#:1)
Phase State Transition (#++1 StateTransition dt))
The state transition simulates tool execution through an adaptive reliability model that adjusts suc- cess probability based on training progress:
Psucosss (1, st) = Pbase
'dep Tretry "/smmantic dEdcps(()
II
Ft.
0.955
if Pepisode < 0.1
Prase
0.90
if 0.1
Pepisode 0.3
0.85
otherwise
where Pepisode represents the current episode success rate, "dep = 0.8 penalizes unmet dependencies, Tretry = 0.95 reduces reliability with retry attempts T., and 'ssmmanti E [1.0, 1.3] rewards semantic task-tool alignment. note that this differs from the tool result simulator probability settings B.3 which enables testing environment different from the training environment.
The transition updates multiple state components:
Tool execution states; git+1 Bernoulli(Pucces// (1,
Execution sequence: e.+1 e'oq U (4) if successful
35

---

Published as a conference paper at ICLR 2026
Error tracking: cconscuutive Cotal updated based outcome
Data flow state progression: Show advanced according semantic operations
Phase II: Progress Computation (Pt+1 within St+1)
Progress update adapts to task structure through dual calculation modes:
Texeonal O Tragraal
+
Asequence
if Trequired # M otherwise
Viequeel
Pt+1
0.3 Pmilestone
+ 0.4 Pooo + 0.3 Pstep
where Ssequence 0.1 provides sequence order rewards, Pmilestone
Mastioual MMepecsal
Ptool
min(1.0, (successful looks
5 ), and Paep min(1.0, )
Progress monotonically increasing: P++1 max Peomputed) prevent regression.
Phase III: Reward Calculation (r' = AdaptiveReward at, 8t+1))
The adaptive reward system operates through the two-phase strategy detailed in Section D.3 incor- porating:
Base exploration rewards for any non-NO-OP action
Tool execution rewards scaled by training phase
Progress increment rewards: 100 (pe+1 -Pe)
Required tool coverage bonuses (coverage phase) or sequence order bonuses (sequence phase)
Semantic alignment rewards from RAG-enhanced action selection
Phase IV: Completion Assessment (done = Completion (st+1))
Episode termination employs hierarchical completion criteria:
done (p++1 2 1.0)V
(Troquired S Texocuned) (k: kimax)
(Coonsccutive > emax V (AAvatid 1)
where kmax and max are curriculum adaapiiv thresholds. Success determination considers both completion method and achieved progress:
True
if Pt+1 2 0.95 V Trequired C Texecuted
success
p++1 2 0.5
if timeout deadlock
False
otherwise
This four-phase execution framework ensures consistent state evolution while maintaining curriculum-appropriate difficulty and comprehensive performance assessment.
D.4.2 CURRICULUN-ADAPTIVE TRAINING
Additionally, the training process implements dynamic curriculum adjustment based on performance metrics:
Curriculum Stage Transitions:
stage, +
if p O Oudvance 70 Psuccess < otherwise
stage,41
max(0, (0, stage, - 1)
Oegrress
stage;
36

---

Published as a conference paper at ICLR 2026
(n) where i indexes curriculum update events, and Piuccess represents the success over the last episodes. We have advancement threshold anvancee = 0.7, regression threshold Oegress = 0.3, and minimum episodes mim 50.
Each curriculum stage modifies completion criteria given in table 12
Table 12: Curriculum-dependen completion thresholds.
Stage
Min Progress
Max Errors
Required Coverage
0.1
50
0.2
1
0.3
30
0.4
2
0.5
20
0.7
3
0.7
15
0.9
4+
0.7
10
1.0
D.5 POLICY OPTIMIZATION FRAMEWORK AND MDP TRAINING
The MDP training implements sophisticated optimization framework that combines policy gradi- ent methods with curriculum learning and RAG-enhanced decision making. The optimization pro- cess operates through carefully orchestrated phases that build upon the adaptive reward structure. The policy update process operates directly on the multi-source confidence composition framework established earlier, optimizing the neural network that learns to weight and combine different infor- mation sources for action selection.
Policy Architecture and Parameters:
The trainable policy тe is implemented as a neural network that takes the encoded state representa- tion R and outputs action probabilities over the filtered valid action space (sai
Critically, the policy network learns to implicitly weight the confidence components defined earlier:
DSa. = Te(a, =0 fo Ewy (Zuy- cs(()))))
where ej E {Cmil: Cemhed; Cpattem; Crack } are the confidence components defined in Section D.2. and fo represents the learned non-linean transformation.
Objective Function Correspondence:
The optimization objective directly corresponds to maximizing the expected cumulative reward de- fined in Section D.3.I
T
J(0)= = E
Raduptive (8t, Ge, s141)
Reuverage ( Oe, 8141) Ppssccess < 0.33 + Rsequence (St, 0t, 8141) [Psuceess 2 0.3))
where Reoverage and Rsequence are the phase-specific reward functions detailed in equations (8)-(12) of Section 4.3.
D.5.1 EPISODE REWARD ADJUSTMENT
Post-episode reward adjustment propagates final performance back through the trajectory:
37

---

Published as a conference paper at ICLR 2026
6
r/
Mperformance
+ Aposition (1 7)
+ rra + 6patte
1.5
if scorennal 2 0.9
1.2
if scoreginal
2 0.7
Pperformance
1.0
if scorérinal 2 0.5
0.7
otherwise
where bay rewards RAG-guided selections and 6pauem rewards pattern completion.
Gradient Computation with Experience Replay:
The policy gradient incorporates the episode reward adjustment mechanism:
Vo) = EBrWTT Σ
Ve log Terls)) oa
r/ Tt Mperformance
Aposition (1 T) + Erag Spatto
where:
Tt Radaptive (st, Oz, St+1) is the immediate reward from Section D.3.1 Mperformance {0.7, 1.0, 1.2, 1.5] scales based final episode score
dag provides additional reward when actions align with RAG-enhanced confidence Cembed 1)
Opttem rewards actions that complete successful sequential patterns from P successful
Curriculum-Adaptive Learning Rate:
The learning rate adapts based on curriculum stage and phase transition status:
Opoolcy
Obase
stage curriculum
Sphase
1.2 if recently transitioned to sequence phase
Sphase
1.0
if stable in current phase
0.8
if performance declining
where Obase 3 x 10 and Ycurriculum 0.95.
Integration with State Encoding:
The policy update directly operates on the state encoding defined in Section 4.1:
S: Concat(encode(( encode((:). w. funk encode({.))
Loss(O) log ne (arls:) rf + Bentmpy H (TO(
where H(-) is the entropy term encouraging exploration, with Pentrogy = 0.01 during coverage phase and Bentopy 0.005 during sequence phase.
Convergence and Performance Monitoring:
Policy convergence is monitored through multiple metrics aligned with the curriculum progression:
38

---

Published as a conference paper at ICLR 2026
Coverage Phase:
Convergence measured by Toucurea Taquired
0.8 consistently
[ Troquiee
Sequence Phase:
Convergence measured by sequence correctness score > 0.7 and step efficiency improvement
The parameter update incorporates gradient clipping ( S 0.5) and momentum-base optimization to ensure stable convergence across both training phases.
This policy update framework directly optimizes the multi-source confidence weighting to maximize the chase-adaptive reward signals, enabling the system learn both tool discovery and sequence optimization through unified gradient-based learning.
D.5.2 EXPERIENCE COLLECTION AND REPLAY
The system maintains episode trajectories for pattern learning:
D opisode
{(st, Tt, Tt; Stt1))
D pattems ExtractPatterms opiisoee SCOTe
P Suuccssfull
Pa Ssuccesstul
D patters
Pattern Extraction: Sequential tool patterns are learned from successful episodes:
pattern = sequenceli
i+ A:] Vi, k: (2,3)
Σ Episodos scoréepisade pattern, episode]
scote(patterng)
Expisodes [pattern episode]
D.5.3 GENERATING THE MDP-OPTIMAL WORKFLOW PROMPT
Once an MDP-optimal workflow has been determined by the trained policy, it exists as a structured Python object containing sequence of tools, dependency information, and rich metadata. To make this workflow actionable by a downstream execution agent, it must be translated into comprehen- sive and unambiguous set of instructions. This is achieved through the generate wew.teem..pp...... incp-prompe function, which constructs a detailed prompt formatted according to Multi-Agent Communication Protocol (MCP). The goal of this prompt is not merely list the tools, but to provide a complete operational context, including the strategic reasoning, expected outcomes, and contingency plans. The generation process composed of several automated steps that assemble distinct components of the final prompt.
Core Structure The entire prompt is encapsulated within a root <mcp.task> tag, creating a structured, machine-readable format. This structure is populated by several key sections, each gen- erated by a dedicated helper function that extracts and formats information from the final workflow object.
Execution Plan Generation This the central component of the prompt, outlining the se- quence of actions the agent should take. The script employs intelligent generation method, teteerttesmmrr.eeeccuee ion.plan, to create a detailed, step-by-step guide. For each tool in the optimal sequence, this function populates the plan with rich set of contextual details drawn from the smart.action ions list in the workflow object:
Semantic Rationale: If a tool was selected via semantic search (RAG), its relevance score is included (e.g., Semantic match: 95%). This immediately informs the agent of the confi- dence that tool's applicability.
Generated Reasoning: A concise, natural language explanation for the tool's selection is provided. This is generated by the
www..et.......o.oai..eeeeeeeeeee function, which considers factors like the tool's position in the sequence (e.g., Initial data loading step), its semantic capabilities (c.g., Performs parse, transform operations), and whether was mandatory requirement of the task.
39

---

Published as a conference paper at ICLR 2026
Dependencies and Alternatives: The prompt explicitly lists the direct prerequisites for each tool (e.g., Requires: fil__opertions_reea reader)) and provides a list of alternative tools, ranked by semantic similarity, that be used if the primary tool fails.
Expected Outcome and Confidence: For each step, a high-level expected outcome is stated (e.g., Expected: data validated)) and a composite confidence score, calculated by cclcullate....... ceattteecooppoeee rehens weeveeeooote dence, is displayed to manage the agent's "e ex- pectations about potential failures.
Intelligence and Semantic Insights quality, two sections are generated.
To provide the agent with meta-level awareness of the plan's
The
section provides a top-level summary, including the overall predicted success probability of the workflow, calculated by .calculate.- proccess-probabilie
The <semant incinsici ghts> section, generated by peenerate_semantiine insights, offers quantitative summary of how Retrieval-Augment Generation (RAG) influenced the plan. reports metrics such as the average semantic match score across all tools and the proportion of tools that were selected based semantic relevance versus learned policy patterns.
Contingency and Execution Guidance instructions to ensure robust execution.
The final sections of the prompt provide static but critical
The <fai hrrehanndl ing block provides clear, universal protocol for error handling. It instructs the agent to first attempt the primary tool, then cycle through the provided alternatives upon failure, and finally report the issue if all options exhausted.
The <execut ion.t trackingg> block sets the requirements for the agent's response, in- structing it to report its tool selection rationale, any deviations from the plan, and its confi- dence in each step's outcome. This ensures that the execution results can be used for future learning and analysis.
Through this transformed into rich, self-contained operational directive that guides not just the "what" but also the "why", "how", and "what of task execution.
generation process, simple sequence tools derived from an MDP is
E DETAILED EXPERIMENTAL RESULTS
E. DETAILED MODEL PERFORMANCE ACROSS TASK TYPE
The following table presents model performance across different task types. Performance generally decreases from basic file processing tasks to advanced computation pipelines, though patterns vary across model architectures.
Table 13: Model performance across task types.
Type
GPT-4o-mmii
Clandrr-Senett-
Gemini-J_5-Flehh
DrepSrec--33
Deepheck-RI
Qwen2.5-322
1.amm-3.3-700
Adhonadd processing
data strioval
Batch processing
82.0%
58.7%
64.6%
58.6%
64.0%
Contznt antlysis
77.9%
76.3%
66.0%
70.5%
70.0%
66.1%
82.15
81.7%
Mulii step processing
67.5%
62.1%
31.7%
39.5%
39.5%
47.3%
72.9%
65.3%
E.2 COMPLETE MODEL PERFORMANCE ON WORLD TEST SET
In this section, we present the complete model performance table, including results under the base- line prompt and CoT' prompt.
40

---

Published as a conference paper at ICLR 2026
Table 14: Prompt type performance comparison: Success rates across different prompt types for each model.
Baseline
Chain-of: Thoough
Optimal Workflow
Flawed Workflow
Model
Full
Partial
Fail
Full
Partial
Fail
Full
Partial
Fail
Full
Partial
Fail
GPT-4o-mini
38.9
38.9
22.2
41.7
41.7
16.7
42.1
38.6
19.3
34.3
34.3
31.4
Gemini-2.5-Flash
33.3
33.3
33.3
39.5
39.5
21.1
55.3
41.2
3,5
34,1
33.5
32.4
GPT-5-mini
38.9
38.9
22.2
41.7
41.7
16.7
47.6
45.2
7.1
36.6
36.3
27.0
Llama-3.3-70B
66.7
33.3
25.4
49.3
Qwen2.5-32B
41.7
41.7
16.7
43.9
40.4
15.8
55.3
44.7
0.0
42.3
38.8
18,9
DeepSeek-V3
32.1
30.9
37.0
37.8
35.3
26.9
40.2
39.3
20.5
36.0
35.4
28.6
Avg
33.6
33.4
33.0
36.9
35.9
27.3
46.7
39.4
13.9
34.8
34.0
31.3
E.3 SCALING ANALYSIS ON THE REAL WORLD TEST SET
In this section, we present scaling analysis results for the Qwen2.5 series on the real-world test set under MDP-optimal workflow prompts. Table T5 reveals a dramatic emergence of workflow exe- cution capabilities between the 3B and 7B parameter scales. The 3B model exhibits near-complete failure (93.3% 3% failure rate with only 5.0% full success), indicating insufficient capacity for multi-step API coordination. A sharp capability transition occurs at 7B parameters, where the model achieves 37.7% full success with failure rate dropping to 28.1 %. The 14B model maintains similar perfor- mance (35.0% full success, 30.0% failure), while the 32B model demonstrates the strongest perfor- mance with 55.3% full success and zero failures. This non-smooth scaling pattern characterized by sharp capability emergence between 3B and 7B, followed by steady improvement to 32B--povvide evidence for emergent workflow execution abilities in real-world API interaction settings, corrobo- rating our simulation findings (Section 4.4
Table 15: Qwen2.5 series scaling analysis with Optimal prompt.
Model Size
Full Success Rate
Partial Success Rate
Failure Rate
Qwen2.5-3B
5.0%
1.7%
93.3%
Owen2.5-7B
37.7%
34.2%
28.1%
Qwen2.5-14B
35.0%
35.0%
30.0%
Qwen2.5-32B
55.3%
44.7%
0.0%
E.4 REAL-WORLD API FAILURE DISTRIBUTION
We present a record of 948 total API calls across 23 different live APIs, achieving an overall success rate of 52.7% (500 successful calls, 448 failures). This 47.3% natural failure rate demonstrates the inherent uncertainty in real-world API interactions that our benchmark aims to capture. Unlike the simulated tools in our main experiments, these APIs execute actual HTTP requests live endpoints, exposing agents to real-world system behaviors. The error modes and system behaviors directly come from real-world API interactions. Table T6 shows the distribution of error types observed during our experiments.
Table 16: Error Type Distribution in API Calls
Error Type
Counts
Percentage of Failures
TIMEOUT
337
71.10%
OPERATION.FAILED
41
8.65%
INVALID INPUT
27
5.70%
INVALID RESPONSE
21
4.43%
NETWORK_ERRON BATE_LIMIT_ERG
21
4.43%
1
0.21%
We now present an explanation of all the error types presented in the table. The predominant error type, TIMEOUT, reflects implementation of 30-second timeout threshold for API responses.
41

---

Published as a conference paper at ICLR 2026
This design choice aligns with common industry practices for production systems, where timeout mechanisms are essential safeguards against indefinite waiting and resource exhaustion. Impor- tantly, the occurrence of timeout errors depends entirely real-world network conditions, server load, and backend processing times, making these failures authentic reflections of the unpredictable nature of distributed systems rather than artificial constraints. OPERATION OPERATION.FATIES FFI_LEE errors result from Python exceptions during response processing, such as JSON parsing failures or encoding er- rors. EWPALID.INUU are mapped from HTTP 400 status codes returned by APIs when request parameters are missing or malformed. INVALID WWW.IIDREEPOOO RESPONSE errors occur when APIS return un- parseable content, such as empty response bodies or data that doesn't match expected schemas. NETWORK ERROR are triggered by issues including DNS resolution failures and connection re- fusals. RATE LIMIT ERROR directly map to HTTP 429 responses when API rate limits are ex- ceeded.
F LIMITATIONS
Our simulation uses a simplified probabilistic failure model (base success rate 0.8 with penalty adjustments) that does not capture real-world complexities such as input-dependent errors, time- varying reliability, realistic recovery mechanisms like exponential backoff. These simplifications balance reproducibility, scalability, and our focus on instruction quality variation. While the real- world task set (S.ction[3.4) shows simulation rankings correlate with real-world performance, future work could incorporate input-conditioned failure probabilities, explicit rate limiting, and correlated failure patterns.
G THE USE LARGE LANGUAGE MODELS
In adherence the ICLR 2026 policy the use of Large Language Models (LLMs), disclose that an LLM was utilized as a specific component within our benchmark construction methodology.
Role in Code Development. The implementation of our experimental framework, including the automated task generation pipeline and the simulation environment, was expedited with the assis- tance LLM-bbsed coding tools. Specifically, we utilized Anthropic's Claude for generating code snippets, debugging complex logic, and refactoring. These tools acted programming assistants, and all final code was reviewed, validated, and integrated by the human authors.
Role in Manuscript Preparation. Our writing process for the appendices involved collaborative human-LLMM workflow. The initial drafts of all appendix sections were written by the human authors ensure the factual and technical accuracy of the content. These drafts were then processed by an LLM for the purpose of polishing the language, improving formal structure, and enhancing clarity. Following the LLM's revisions. the authors conducted a final, thorough review make critical edits, verify all statements, and ensure the text precisely reflected our methodology and findings.
The core research ideation, experimental design, analysis of results, and the primary drafting of the main manuscript were conducted by the human authors. We take full and final responsibility for all content presented in this paper, including any code and text produced with the assistance of LLMs.
42


Reviews:
[{"review_id": "QYqm8wRtnZ", "reviewer": "Reviewer_NTap", "summary": "The paper introduces a benchmark PILOT-Bench for evaluating LLM workflow execution under probabilistic tool behaviors and variable instruction quality. PILOT-Bench contains 5,040 tasks generated from a library of 30 APIs with explicit probabilistic error modes such as TIMEOUT, DEPENDENCY ERROR, and INVALID INPUT. Experiments show that current LLMs performance differences when models encounter probabilistic tool failures and varying instruction quality.", "questions": "Could the benchmark be extended to adaptive settings where the LLM can retry or re-plan after a failure? Reference settings are like ToolEVO [1].\n\n[1] Learning Evolving Tools for Large Language Models. ICLR 2025.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The benchmark explicitly models stochastic tool behaviors and noisy instructions, which is novel from previous tool learning benchmarks. The probabilistic simulator and MDP-based workflow construction are well-motivated and technically sound.\n2. The automated generation pipeline is clearly described. The inclusion of both optimal and flawed workflows enables the study of robustness to imperfect guidance, which is highly relevant for practical agents.\n3. The experimental evaluation shows results of ten diverse LLMs which is solid and sound.", "weaknesses": "1. All experiments rely on a simulated environment. Demonstrations on real API tools or open-ended workflows would better substantiate the validity.\n2. Writing has room to improve. Some sections  repeat definitions of tool errors and workflow generation, which is redundant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d4LkQoAO3w", "reviewer": "Reviewer_eCNp", "summary": "In this paper, the authors propose a benchmark named PILOT-Bench for evaluating large language models in tool-driven workflow scenarios under uncertainty. The benchmark comprises 5,040 tasks across 30 APIs, testing robustness to flawed instructions and stochastic tool behavior. Experiments across ten LLMs reveal success rates of 71.8% for optimal, 59.4% for Chain-of-Thought, and 61.5% for flawed workflows.", "questions": "Line 352 references the Qwen2.5 series, yet only Qwen2.5-32B results are reported.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. The paper introduces a comprehensive benchmark for evaluating LLM workflow execution under uncertain tool-use conditions, encompassing 5,040 tasks and 30 tool APIs.\n2. The evaluation includes a wide range of models, covering both proprietary and open-source LLMs.", "weaknesses": "1. The result analysis lacks depth—although extensive data are presented (Sections 4.2–4.4), the discussion does not yield clear or insightful conclusions.\n2. The claimed advantages over existing benchmarks (e.g., ToolBench) are limited, as prior works also incorporate probabilistic tool failures and instruction variability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hdx6xVbwH7", "reviewer": "Reviewer_5aFN", "summary": "This paper introduces a new benchmark PILOT-Bench, designed to evaluate the workflow execution capability of LLMs in tool-driven scenarios that more closely approximate real-world complexity. The authors argue that existing benchmarks primarily focus on idealized settings where tool behavior is deterministic and guidance is precise. To address this, PILOT-Bench introduces two core real-world challenges:\n\n1.  Tool Execution Uncertainty: The benchmark simulates probabilistic tool behavior by parameterizing error models, reflecting failure modes that APIs may exhibit in the real world.\n2.  Variability in Guidance Quality: The benchmark provides workflow instructions of varying quality, including \"Optimal Workflow\" prompts and \"Defective Workflow\" prompts, to assess the model's robustness to changes in instruction quality.\n\nThe benchmark comprises a tool library of 30 APIs, from which 5,040 tasks are generated. A significant contribution is that the Optimal Workflows are generated using a MDP framework that considers tool dependencies and stochastic behavior, with policy optimization performed using PPO. The Defective Workflows are created by systematically injecting seven categories of controlled perturbations into the optimal workflows.", "questions": "Please refer the weakness :-)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.  The paper points out the limitations of existing tool-use benchmarks. By focusing on probabilistic tool failures and variability in instruction quality, the benchmark addresses two core challenges for LLM agents in practical deployment.\n2.  The experimental results are insightful. For instance, the high robustness exhibited by some state-of-the-art models when given defective prompts suggests that robustness to defective guidance might be an ability dimension independent of general tool-use capability. Furthermore, the negative impact of task complexity on performance aligns with expectations.", "weaknesses": "1.  One of the most confusing yet undiscussed results in the paper is that the \"Complete Success Rate\" of the Baseline (which only contains the task description) is actually higher than that of the Optimal Workflow for several state-of-the-art models. This directly challenges the premise that the MDP-generated workflow is \"optimal.\" The authors do not seem to discuss this phenomenon. Does this imply that the level of detail and complexity in the optimal workflow actually interferes with or confuses the model, or is my understanding incorrect?\n2.  While the introduction of probabilistic failure is a good mechanism, I believe the simulation is still relatively simple. The testing environment uses a base success rate of 0.8 with a penalty adjustment based on dependencies and historical failures. Real-world API failures are often not independent and can depend on factors like rate limits or specific errors caused by invalid inputs. The current model captures five failure types, but the simulation logic seems to simplify this into a single probability calculation.\n3.  The \"optimal\" workflow is an artifact generated by an MDP agent that maximizes its own reward function within its training environment. It is unclear whether this strategy corresponds to the \"gold standard\" or the truly most efficient workflow as perceived by a human expert. Its optimality is relative to its training process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)