Title: PILOT-Bench: Probabilistic Interaction for LLM Operations in Tool-driven Scenarios
Primary area: datasets and benchmarks
Keywords: Workflow execution, LLM robustness, Probabilistic Tool Behavior

Abstract: Large language models (LLMs) have shown capabilities in tool utilization and workflow orchestration, yet existing evaluation benchmarks focus on idealized scenarios with deterministic tool behaviors and precise guidance, which may not capture real-world deployment complexities. We introduce PILOT-Bench, a benchmark that evaluates LLM workflow execution under simulated realistic conditions of guidance quality variability and tool execution uncertainty. The benchmark incorporates three key aspects: (1) modeling of probabilistic tool behaviors through parameterized error models that may reflect real-world API failure patterns, (2) evaluation of LLMs' workflow execution capabilities in environments where both guidance and tool behavior approximate practical scenarios, and (3) provision of both optimal and perturbed workflow guidance to assess robustness to instruction quality variations. Our construction pipeline generates 5,040 tasks from a tool library of 30 APIs. The evaluation conducted across ten widely used large language models under conditions of probabilistic tool failures and varying instruction quality reveals notable performance differences. Specifically, optimal workflow prompts achieve an average success rate of 71.8\%, Chain-of-Thought prompts yield an average success rate of 59.4\%, and flawed workflow prompts result in an average success rate of 61.5\%. Our benchmark is available at https://github.com/PilotBenchAnonymous/PilotBench.

TL;DR: We propose PILOT-Bench, a benchmark that evaluates LLM workflow execution under simulated realistic conditions with probabilistic tool failures and variable instruction quality.