Hell or High Water: Evaluating Agentic Recovery from External Failures
Authors: Andrew Wang
Sophia Hager
Adi Asija
Daniel Khashabi
Nicholas Andrews
Venue: COLM 2025 (2025)
Keywords: planning
tool-use
TL;DR: Evaluating how well LLMs can find backup plans

Abstract: As language model agents are applied to real world problems of increasing complexity, they will be expected to formulate plans across large search spaces. If those plans fail for reasons beyond their control, how well do language agents search for alternative ways to achieve their goals? We devise a specialized agentic planning benchmark to study this question. Each planning problem is solved via combinations of function calls. The agent searches for relevant functions from a set of over four thousand possibilities, and observes environmental feedback in the form of function outputs or error messages. Our benchmark confronts the agent with external failures in its workflow, such as functions that suddenly become unavailable. At the same time, even with the introduction of these failures, we guarantee that the task remains solvable. Ideally, an agent’s performance on the planning task should not be affected by the presence of external failures. Overall, we find that language agents struggle to formulate and execute backup plans in response to environment feedback. While state-of-the-art models are often able to identify the correct function to use in the right context, they struggle to adapt to feedback from the environment and often fail to pursue alternate courses of action, even when the search space is artificially restricted. We provide a systematic analysis of the failures of both open-source and commercial models, examining the effects of search space size, as well as the benefits of scaling model size in our setting. Our analysis identifies key challenges for current generative models as well as promising directions for future work.

Published as a conference paper at COLM 2025
Hell or High Water: Evaluating Agentic Recovery from External Failures
Andrew Wang Sophia Hager
Adi Asija
Daniel Khashabi"
Nicholas Andrews'
Johns Hopkins University (awangl 16, danielk, @jju.edu
Abstract
As language model agents are applied to real world problems of increasing complexity, they will be expected to formulate plans across large search spaces. If those plans fail for reasons beyond their control, how well do language agents search for alternative ways to achieve their goals? We devise a specialized agentic planning benchmark to study this question. Each planning problem is solved via combinations of function calls. The agent searches for relevant functions from a set over four thousand possibilities, and observes environmental feedback in the form of function outputs or error messages. Our benchmark confronts the agent with external failures in its workflow, such as functions that suddenly become unavailable. At the same time, even with the introduction of these failures, we guarantee that the task remains solvable. Ideally, an agent's performance on the planning task should not be affected by the presence of external failures. Overall, we find that language agents struggle to formulate and execute backup plans in response to environment feedback. While state-of-the-an models are often able to identify the correct function to use in the right context, they struggle to adapt to feedback from the environment and often fail to pursue alternate courses of action, even when the search space artificially restricted. We provide a systematic analysis of the failures of both open-source and commercial models, examining the effects of search space size, as well as the benefits of scaling model size in our setting, Our analysis identifies key challenges for current generative models as well as promising directions for future work.'
1 Introduction
Large language models (LLMs) are capable of reasoning and planning through natural language. As result, they now form the backbone of autonomous language model agents that follow user instructions and carry out actions in the real world. Increasingly, these LLM-driven agents are applied to complex or open-ended tasks, such navigating the internet or automating tasks across a user's 's computer desktop environment.
Naturally, faults may arise that prevent plan from working; categorize faults into two broad classes: (1) internal errors, which arise from mistakes an agent makes such as passing the wrong arguments to function, and (2) external errors that occur for reasons beyond an agent's 's control. As an example of the latter, flight booking API may not have flights fulfilling the desired criteria, requiring the LLM agent try alternate carrier booking APIs. In spite of these obstacles, users generally expect an agent to complete tasks come hell or high water: Prior works have focused on studying the ability of language agents to correct their own mistakes (Shinn et al., 2023; Madaan et al., 2023; Chen al., 2023a). In contrast,
* Equual advising.
I CCode data available here: https://gi thub. com Com/JHU-CLSP/hee http--LLS//eell-o-hhhhhhe- 2The phrase "hell or high water" denotes obligation that should be fulfilled regardless of circumstances beyond one's control.
1

---

Published as a conference paper at COLM 2025
What are the employee ids of employees who report Shelley, and what are their salaries?
salary
Execution
Func "205"
for employees under e maneger specitic
unavailable
. Salarye...
for specitic first name
Fune Artwioyy
furk: & Sraeeerr?
Rowe crainn
selary of employees who
report to a manager
I should use Func A
Agent
specific first name
Func B
use Func
Inputs: first
2
Func C
managertee to search
Large of functions
A gennt Workflow
Figure The agent correctly tries to use Func A, but encounters an external error outside its control: Func A is unavailable. Therefore, it must form backup plan to use the composition of Func and Func We include the full trajectory in Appendix F, Language model agents struggle to identify backup plans when they encounter external errors.
the effect of external faults has remained understudied, despite their prevalence in the real world.
We propose the first benchmark that studies the effect of external errors on language agent performance. Our benchmark satisfies a number of properties that make it suitable for studying this phenomenon. First, our evaluation consistently places agent in situation where the task cannot be completed without using a backup plan, while ensuring that the task remains solvable. Secondly, tasks in benchmark require planning, but straightforward enough that a successful backup plan can be found by interacting with the environment. Furthermore, to actually measure the inherent capabilities of language agents, our tasks are novel relative to the LLM pre-training data, avoiding solutions based on memorized behaviors. Finally, solutions are easily evaluated based on functional correctness requirement, we avoid relying on LLM verification avoid potential biases and facilitate reproducibility and instead employ human-curated datasets where functional correctness can be automatically checked.
challenging enough to
agent's answer must match gold solution. To satisfy this last
Figure shows an example problem from the benchmark developed part of this work. Given a question and access to a library of functions for querying and manipulating the relevant knowledge bases, the language agent must formulate plan-a sequence of one or more function calls-and execute it to obtain the correct answer. Doing so involves not only formulating queries to retrieve the appropriate functions, but also calling these in the right order and with the right arguments. When the language agent executes a function, interacting with an external environment, which can respond in unexpected ways. Upon receiving unexpected feedback from the environment (e.g., if the function was called with the wrong arguments, or was unexpectedly disabled), successful language agent must react accordingly by adjusting its plan. Our benchmark is constructed such that each question can be solved in least two different ways, involving different functions.
In more detail, our proposed benchmark, Hell-or-High-Water consists of 830 questions and 4450 functions that must be executed in appropriate contexts to obtain the right answer. Our benchmark creation process is completely automatic, converting text-to-SQI. datasets into questions and tools. To consistently force model resort to a backup plan, we construct these functions such that every question in the dataset can be solved using either of two non- overlapping sequences of function calls. The first function an agent calls to attempt at solving the problem can be automatically disabled, while always leaving possible alternatives to arrive at the correct solution. While problems in our benchmark are straightforward to solve, involving no more than 3 reasoning steps, the presence of many functions requires the
2

---

Published as a conference paper at COLM 2025
agent to plan. As agents are calling functions to generate the output, rather than generating a natural language output, functional correctness is straightforward for this benchmark.
Using the proposed benchmark, we conduct an extensive empirical study of language model agents which focuses on the following main questions:
Are language agents capable of fault-tolerant planning? All LLMs evaluated have significant difficulty recovering from unexpected errors, including commercial models such as GPT-4 and Gemini 2.0.
How does size of the search space impact fault-tolerant planning? We compare perfor- mance in two distinct settings: (1) when all required functions are included in-context; (2) when the language model agent must generate search queries to retrieve relevant tools. We find that the need to search for relevant tools results in significantly harder challenge. Is fault-tolerant planning property of model size? While larger models perform better in terms of absolute accuracy, all models tested displayed decreased performance in the face of external errors, suggesting that resilience to such errors benefits weakly from model scaling.
Can language agents recognize when no solution is available? In cases where all solutions are disabled (i.e,, are present but return a clear error message), we find that different LLMs exhibit markedly different ability to recognize the lack of feasible solution,
with some exhausting their turn budget.
What is the im impact of question underspecification? In the real-world, users may fail to adequately express their instructions, leading to underspecification. We find that the degree of question underspecification does impact results, with disanbiguating information significantly helping certain models.
2 Related Work
Real-world planning benchmarks. A variety of benchmarks have previously been proposed to test LLM capabilities for real-world planning, with varying levels of complexity between benchmarks. Some, such as API-bank (Li et al., 2023) and ToolQA (Zhuang et al., 2023) specifically test a model's ability to plan, retrieve, and call tool in a single step. Others, such as ToolSandbox (Lu et al., 2024a) and AppWorld (Trivedi et al., 2024) extend those capabilities by adding feedback, allowing for multi-turn interactive dialogues. TravelPlanner (Xie et al., 2024) requires agents to arrange a multi-day travel itinerary, which necessarily requires LLM agents to be able to navigate many multi-step tasks and compose outputs. In our benchmark, we provide multiple distinct solutions for each question with different levels of complexity: one solution solves the question in single step, while the other requires the LLM to break the task into subtasks and call the correct function for each. Having benchmark with multiple solutions, one of which requires more creativity and reasoning from the model, allows us to investigate an LLM's ability to adapt according to feedback. To our knowledge, prior benchmarks do not provide these types of problems.
Tool use. LLMs with the capability to use tools gain significant advantages, such as access to specialized knowledge and multimodal functionalities (Qin et al., 2023b). In particu- lar, many systems have been designed to tools accessed through APIs. Using APIs remains challenging for LLMs to use, as LLMs may fail to correctly call an API on the first attempt (Patil et al., 2023; Qin et al., 2023b). While many methods for creating an agent which can use tools require fine-tuning an LLM (Parisi et al., 2022; Patil et al., 2023; Qin et al., 2023b; Schick et al., 2023; Yang et al., 2023), modern LLMs have strong in-context learning abilities that allow for few-shot tool-use (Hao et al., 2023; Qin et al., 2023a; Lu et al., 2024b). In particular, frameworks such as ReACT (Yao et al., 2023) and CodeAct (Wang et al., 2024b) use alternating reasoning and action steps to allow LLM agents to receive feedback and adjust their use of tools accordingly; for instance, receiving an error message for a failed API call during reasoning step may result in the model adjusting the call and correctly using the tool. Multi-step reasoning processes also allows for tool composition when two or more tools are required: for instance, an agent booking flight might need to consult personal calendar for available times before filtering a list of flights. This requires an LLM to both break task into smaller subtasks and select the correct tool for each subtask. Prior research
3

---

Published as a conference paper at COLM 2025
has shown that LLMs demonstrate limited capabilities in these situations in comparison to single-tool tasks (Qin et al., 2023a; Ye et al., 2025; Hosseini et al., 2024).
Self-reflection for planning. As LLMs display increasing reasoning capabilities, prior work has investigated self-reflection methods which allow LLMs to correct or critique their own responses in order to achieve better results (Madaan et al., 2023; Saunders et al., 2022; Tian et al., 2024a). Self-reflection has also been applied to tool use, as LLMs can use feedback from the tools (e.g. reasons for failure) to amend their outputs (Du et al., 2024; Gou al., 2024; Qiao et al., 2023; Wang et al., 2024a). Using a multi-turn "chain-of-thouge" model to use its reasoning ability to better learn how to call tools (Chen et al., 2023b). This is particularly true when there are silent tool errors, where the tool fails without an error message but produces an incorrect or nonsensical output anyway. Sun et al. (2024) find that LLMSs can frequently detect silent tool errors, for example in embodied settings with object detector module, but often fail to recover from them, leading to cascading errors in multi-tool settings. In contrast, our emphasis is on the ability of LLMs to identify alternate solutions in large action spaces comprising large numbers of functions.
allows a
LLM creativity. Creative problem solving is often broken down into convergent thinking, or finding a single optimal solution to a problem, and divergent thinking, or coming up with multiple unique solutions to problem (Ismayilzada et al., 2024; Lu et al., 2025). Divergent thinking is particularly important for adaptation to new constraints or environments; LLMs tend to struggle with both. Previous benchmarks for weal-world creative thinking have evaluated on tasks such as escape-room puzzles (Qian et al., 2024) or real-world object usage (Tian et al., 2024b). While these benchmarks require creative thinking and compositionality of tools, their settings fall into the narrow category of physical tool use, preventing their use for assessment of virtual tools requiring API calls. Our benchmark examines performance with API requests, Aside from having a realistic setting, this has the advantage of inherent feedback mechanisms such as error message rather than relying on synthetic feedback in the form of human- or L.M-written descriptions of a fictional real-world setting.
3
Hell-or-High-Water. an Interactive Backup Planning Benchmark
3.1 Task Description
The general workflow for any given problem is as follows. The agent is presented with description of the problem and a set of 4450 functions to potentially call. To navigate the large set of functions, we provide a semantic search tool to find and learn about available functions. If the agent uses the search tool by generating a query and executing the search tool, it will be provided with retrieved set of functions and associated documentation relevant to the search query. The agent can then choose to either execute observed functions or look for other functions with a revised search query. If the agent chooses to try any of the observed functions, it must write code to execute those functions. After the code is executed, the agent observes the output and plans the next action accordingly. We provide full trace in SF and more details regarding function search in $A.
In our benchmark, the agent interacts with our environment via the CodeAct framework (Wang et al., 2024b). CodeAct is an extension of ReAct (Yao et al., 2023), a prompting strategy where a language agent alternates between verbalized reasoning and taking actions on an environment. In CodeAct, the actions are concretely defined as executable blocks of code. The workflow is similar to that of using a Jupyter notebook, where a kernel executes one block of code at a time. By working in the modality of code, an LLM is theorized to better adhere to logical patterns of thought (Li et al., 2024). Thus, in our benchmark, each action an agent takes is expressed as a block of code, and each action is preceded by verbalized rationalization and planning. We view this workflow as blending traditional collusee/ressonie approaches with code generation.
Studying external errors. So far we have outlined the agent's workflow under fault-free conditions. Recall that the agent must call the available Python functions to complete the task. If we throw errors during the agent's workflow, then we create external errors thereby inducing conditions requiring backup planning. By controlling when and how we
4

---

Published as a conference paper at COLM 2025
SQL Query: sampled from Spider
Parameterized SQL Query
Decompose to Subqueries
SELECT employen_ id, salary FROM enployees WHERE manager_id (SELECT
SELECT employee salary FROH erployees WHERE managor__ke (SELECT employee_ 1d FROM employees WHERE first
SELECT employee id employees +
SELECT employee salary FROM enployees nanager_ (SELECT FROM scratchpad)
erployee_ id FROM employees MHERE first_nane "Payan')
WHERE First_nare nare
Function wrapping
Func A: Get the ID and salary of employees report
Func B: Get the for specific first name
Func C: Get ID and salary employees manager with a specific ID
manager with specific first name
Function set
Function set
Figure 2:
Benchmark creation pipeline. The "function sets" refer groups of functions that together can be used to solve given question: one direct solution involving single function, and one composition of two or more functions that achieve the same result.
throw errors, we can study the effect of external errors on language agent performance. We describe this process in $A.
3.2 Benchmark Construction
We desire method to automatically create functions with diverse functionality at scale. For this purpose, turn to text-to-SQL. datasets which pair natural language instructions with SQL queries that complete those instructions. In addition to already having instructions which we may reuse, datasets such as Spider (Yu al., 2018) offer thousands of unique SQL queries. Additionally, SQL in these datasets can be queried against actual databases, allowing us to know the expected output for solution verification. Using the Spider dataset, we develop deterministic transformations of SQL to one or more function calls. This procedure results in a large set of functions with varying numbers arguments and return types depending on the underlying SQL commands, which reflect the diverse functionality that a set REST API endpoints might provide, while ensuring functional correctness.
Creating functions from SQL. Using text-to-SQL datasets provides instructions that cor- respond to SQL queries. However, our benchmark calls for executable Python functions with inputs and outputs that an agent may use. At high-level, parameterize these SQL queries then wrap them in Python functions. We first parameterize the queries by substi- tuting literals3 with placeholder variables. This procedure is performed automatically by converting the SQL command into a parse tree*, where each node represents an expression (Appendix C.1). To identify literal expressions, we perform depth first search on the parse tree. DFS yields the nodes of literal expressions in the order they appear-in in the SQL query, allowing us to replace those nodes with that placeholder variable.
By parameterizing the query, we deduce a set of parameters for it. If we wrap the query execution in Python function, those parameters can serve as input arguments. The output of these functions consists of a list of JSON records. Each record is mapping of SQL identifiers to their corresponding value. In other words, the outputs are a subset of the rows and columns of a database, specified by the SQL query it uses. We thus propose pipeline for converting existing SQL queries into Python functions that can accept input and produce output. In SC.1 we walkthough example of our function creation process.
Function composition. These problem types can be solved by set of functions {fr fn), where the input to fr is contained wholly in the output of f 1- Our benchmark creation process is able to produce problems of this nature. In cases where subqueries are present in the SQL query, we use our parser locate and extract the subqueries. Each subquery is converted into tools in the same manner as before. Instructions in the Spider dataset that are mapped to SQL containing subqueries can be solved in least two ways our benchmark.
$i.,, values such as names or numbers that are explicitly expressed in the SQL query itself We Glot https:/ /gtgggu..toom///g http://oooooo/////////// /sqlglt
5

---

Published as a conference paper at COLM 2025
Our process of creating functions makes at least three when subquery is present. Of these three functions, there exist two equivalent combinations of function calls- one involving the subqueries and the other involving the original parameterized query (SC.1).
Function documentation. Having created python functions out of SQL queries, we now prompt a language model create documentation for our tools. We adhere to the docu- mentation format employed by OpenAl for their language model function-calling feature. This format is chosen because a growing number of language models (Dubey et al., 2024) have adopted it as their standard for code generation. Examples of documentation in SD.
Quality control. We removed questions that involved querying empty databases, or other- wise returned no results when the original Spider SQL query was run. Out of concern for context-length and the readability of function outputs, we removed questions where the original Spider SQL query returned an excess of 100 results. To ensure that every problem in our benchmark has valid alternative solution, we use the intuition that SQL containing subqueries can solved in least two ways in our benchmark. We filter out all instructions in the Spider dataset that do not correpsond to a SQL query containing subqueries. After filtering, 413 questions remained. To increase the size of our benchmark, we augmented the remaining questions, which we describe in Appendix C. After augmentation, our benchmark includes 922 problems in total.
Dataset statistics. We portion validation set of 92 examples to perform prompt engineering. This leaves a test set of 830 examples, each with at least known combinations of function calls that yield a correct answer. Our benchmark creation process automatically creates total of 4450 functions. Of these functions 2160 have no arguments, 1661 have one, 568 have two, 46 have 3, and 15 have
4 Main Experiments
4.1 Experimental Setup
We provide empirical analysis of language model agent performance recovering from external errors and identifying valid backup plans. Although there exist multiple ways to achieve the correct answer, we ensure that the agent encounters an external error (SA).
Models. We evaluate wide range of LLMs in various sizes. Open weight models tested include 1ama-3.1-Instrrue Instruct (Qwen et al., 2025). Proprietary models include Gemini-2.0 2.0-ni--.00-aash and GPT-4o. All models were tested using greedy decoding for consistency. An analysis of the effect of model size on performance is in $5
(Dubey et al., 2024), and Qwen-2.5-
Evaluation procedure. The models are evaluated by comparing their final prediction to the reference (human) answer, which derived deterministically from the underlying SQL. Since the reference query produces the same output as correct sequence of function calls, we can verify a solution's correctness simply by comparing to the output of the reference query. In practice, an agent may provide solution that is different from but functionally equivalent the expected solution. For instance, the agent may choose to give the solution "John Doe" when the expected solution is ["Name": "John Doe")]. J. We handle these cases by applying deterministic post-processing to the final answers, in order to avoid unfairly penalizing models that do not strictly adhere to the expected type of the output. For each model report the accuracy and standard error.?
hhugging- quattss
1aama--,1-77B-Instttee
Note that number of tool calls involved in the backup plan can vary. show that this variable does not affect the difficulty of problem
7tandaard error here is derived from bootstrapped confidence interval (0.95) of the accuracy using the scipy library

---

Published as a conference paper at COLM 2025
Model Gemini
Model Liama
Model Qwen
0.8
Acc
False
True
0.4
3
9
27
81
3
9
27
81
3
27
81
Num. of distractors
Num. of distractors
Num. of distractors
Figure 3: Closed world performance with a fixed set of functions provided in the context and no search tool. We show performance as the number of distractors is varied, both with (True) and without external errors (False). While fewer distractors corresponds to better performance, the ability form backup plan in the face of external errors decreases as the size of the search space increases.
4.2
Main Results
The results for the most difficult version of our benchmark, which requires language model agents to execute searches to find suitable functions from among thousands of candidates, are reported in Table 1.
Model
External errors (N)
External errors (Y)
Acc. decrease
Gemini 2.0 (Flash)
71.4 + 1.6
41.1 + 1.7
42.4%
GPT 4o
60.5 + 1.7
38.4 + 1.7
36.5%
Llama 3.3 (70b, Instruct)
64.0 + 1.7
38.9 1.7
39.2%
Llama 3.1 (70b, Instruct)
42.3 + 1.7
23.3 1.5
44.9%
Qwen 2.5 (72B, Instruct)
64.1 1.6
35.3 1.3311 1.7
44.9%
Table 1: Functional correctness (accuracy) in the open-world setting where language agents must use a search function to retrieve relevant tools. All evaluated models exhibit significant degradation in performance in the face of external errors.
Discussion. The difference in performance between Llama-3.3 and Llama-3.1 is interesting, since these models are from the same family and perform quite similarly across certain common benchmark tasks such as MMLU. However, some of the largest improvements for Llama-3.3 are in instructiin-follooing capabilities, coding and reasoning capabilities, achieving 92.1 on IFeval (Zhou et al., 2023), 88.4 on HumanEval (Chen et al., 2021), and 77.0 for MATH (Hendrycks al., 2021) compared 87.5, 80.5, and 67.8 8 respectively for Llama-3.1. These improvements might partially explain the difference performance on our benchmark, which requires robust instruction-followiin and reasoning capabilities.
Closed world experiments. Searching large set of thousands of candidate functions introduces significant challenge, top of the challenge of forming backup plans. As ablation of the impact of search, we experiment with a closed-world setting with a fixed number of functions that are provided upfront in the context. In these experiments, there is no search tool, and all functions necessary to address each question are provided, along with varying number of distractors extraneous functions not involved in the solution to the problem but provided in the prompt. These results are reported in Figure 3. Overall accuracy is significantly higher in this setting, but performance degrades to levels similar to those of the search setting (under 40% for 3 distractors for Qwen2.5) the face of external errors. We note that without distractors (when the number of functions is 3 in the x-axis), Qwen2.5 suffers only slight degradation from the introduction external errors, suggesting that the size of the search space plays an important role in the difficulty of our benchmark.
hht /www. 11 llama. com/

---

Published as a conference paper at COLM 2025
Model Liama
Model Qwen
0.6
acc
0.4
False
0.2
True
0.0
60 parameters (billions)
40
20
parameters (billions)
Figure The effect of model size on performance, where "True" shows accuracy with forced function failures and "False" shows accuracy without function failures. Agents consistently perform worse when external errors are introduced (dashed) regardless of model size.
Further Analysis
Error Analysis. We analyze erroneous agent trajectories and categorize them with the following faxonomy-(1) search failures, where the correct tool never shows up in search results, (2) ID failures, where the correct tool shows up in search results, but the agent never uses it, (3) chaining failures, where the agent identifies one tool but not the other, or uses tools out of order, and (4) tool use failures, where the agent tries the correct tool, but gives up on using correctly.
Model
Total failures
Search
ID
Chaining
Tool use
Gemini 2.0 (Flash)
489
57.1
25.5
12.3
6.14
GPT 4o
511
52.6
24.6
8.61
14.1
Llama 3.3 (70b, Instruct)
496
66.3
11.9
13.9
7.86
Llama 3.1 (70b, Instruct)
495
59.3
18.0
11.7
10.9
Qwen 2.5 (72B, Instruct)
637
60.5
26.4
7.06
6.12
Table 2: Breakdown of agent trajectory errors by type (in percent) for the results in Table Tool searching is the most common point of failure for all models tested.
In the tool search setting, failure to find correct tools accounts for between 53% and 66% of the failures, followed by failure to select those tools when they appear in search results. Search errors may be explained in part by a failure to adequately compose queries reflecting the right information critical capability for agents interacting with large search spaces.
In the closed-world setting, to inves- tigate the performance decrease search space is increased, we apply our taxonomy sults. As the search space increases,
Tools
Total failures
ID
Chaining
Tool use
the Gemini-2 re-
3
211
32.7
16.1
51,2
9
332
45.8
23.8
30.4
the share of identification failures in- creases from 32.7% at tools to 54.8% at 81 tools, suggesting that failure to recognize the correct tool(s) when provided connected to this perfor- mance decrease. These results sug- gest that chaining tools together is less challenging than finding them in the first place.
27
451
54.8
16.9
28.4
81
510
54.9
15.5
29.6
Table 3:
Breakdown of agent trajectory errors by type (in percent) for Gemini results in Figure The percentage of tool searching errors increases with the total number tools the search space. For large search spaces finding the right tools is more difficult than calling them in the right order
Does performance scale with size? We analyze the impact of model size on the ability to find a backup plan. Using the same setup described in $4.1 we evaluate two sizes of LLama-3. 3.1 -I-nstruct and sizes of Qwen-
8

---

Published as a conference paper at COLM 2025
25
setting
correctness
20
Distractors only (Y)
20
False
15
Distractors only (N)
True
turns
turns
10
5
0
Qwen2.5-728
Llama-33 Llamm-3.337700 3.3-70B
Qwen2.5-728 Llama-3.3-700 3.3-708
Figure 5: (Left) A violin plot showing how many turns the agent takes with only incorrect functions (Distractors only Y) and with correct functions included (Distractors only N). The two models exhibit markedly different performance in the Distractors only setting, with Qwen-2.5- 2.5e--..--nstruct particularly inclined to persist with impossible problems. (Right) Where correct functions available, plot how many turns the agent takes for correct solutions and incorrect solutions. In cases where the agent is correct, the agent uses fewer turns, suggesting that additional turns do not help the agent achieve the correct solution.
2.5-Instruct and show the results Figure 4. We find that while absolute model performance tends to improve as model size increases, the model's resilience to failure does not, nor does this appear to be an emergent ability (Wei et al., 2022) of language models. Increasing model size does not address the performance degradation when agents face external errors, and other methods may be needed to give agents the ability to find and execute backup plans. For Qwen in particular, we notice drop in performance for the 30B parameter model relative to the smaller model; examining the execution traces from this model, we notice that it has a tendency to ignore instructions regarding the formatting of responses, which is remedied with further scaling up to 70B. Do agents know when to give up? So far, we have studied the behavior in situations where correct solution is in principle achievable. However, in real-world settings, it may not be possible satisfy user request. In this experiment, we analyze the model's capability to recognize that it cannot solve problem with the provided functions. We constrain the agent's s set possible functions such that only has access to incorrect distractor functions, making the problem impossible to solve. Ideally, the agent should give up upon realizing this problem is impossible. We show the results for this setting in Figure 5, We find that this is not clearly exhibited capability for all tested agents. The readiness of a model to give up varies by model; 1amma-3.3-Instrus exhibits significantly more willingness to give up before exhausting the compute budget, while 2wwen-2.-Innsrrice generally fails to recognize that it cannot solve the task with the given functions. Figure 5 also shows the distribution of correct incorrect answers for the default setting (where the correct functions are provided).
Model
External errors (N) External errors (Y)
Acc. decrease
Llama 3.3 (70b, Instruct)
54.2 - 1.7
36.5 + 1.7
32.7%
Qwen 2.5 (72B, Instruct)
71.1 1.6
47.7 1.7
32.9%
Table 4: What is the impact of underspecification? This table shows results for two open- weight models for a modified version of the prompts where additional problem-specific hints are provided to reduce possible ambiguity (e.g., national elections vs. regional elections). Compared to $4.2, performance for Qwen is significantly improved, while Llama 3.3 actually performs worse. Nevertheless, we still observe significant decrease in performance with the introduction of external errors.
What if questions are underspecified? In real-world settings, users may fail to provide sufficient information for a language agent to fulfill their request. For example, a user may ask a question about election results but fail to specify if they are asking about regional or national elections. In the context of our benchmark, we can control the degree of context included with each prompt to vary the amount of uncertainty for each question, and study the behavior of different LLMs under different degrees of uncertainty. The results from including additional disambiguating information to each question are provided in Table 4.

---

Published as a conference paper at COLM 2025
6 Discussion and Conclusion
Main findings. We find that language agents are not able to reliably incorporate feedback from the environment and identify backup plans. In the context of our benchmark, this is surprising for two main reasons. First, when failure occurs, the feedback from the envi- ronment is quite explicit (e.g., "function_ge 987 is currently unavailable. Please try a different function."" "). Second, our comparisons include LLMs which are specifically developed for applications such as the ones studied in this paper. For example, Google Gemini 2.0 features "action-capabillttis, [...], long context understanding, complex instruction following and planning, compositional function-calling," "9 but, while it performs best among the LLMs we compare- paanre--rroomingggg.o. 11 points better than GPT-4o in the no failure Beeting-Geeint 2.0 still has sharp drop in accuracy when forced to explore alternate solutions. Model scale also does seem to matter; in the category of open-weight models, the 70B parameter configurations perform drastically better than smaller ones.
Future work. Why are language agents unable to adapt effectively to unexpected feedback? Our analysis uncovers some common issues among LLMs (Jiang et al., 2025). Since our benchmark involves new functions, the biases of the language agents may be quite harmful due to the need to adapt to novel situations. For example, Figure 5 shows markedly different behavior for two otherwise similar models, which may suggest that training biases are overriding the behavior of the model, even when faced with clear-cut feedback from the environment. It would also be interesting to explore whether recent thinking reasoning models display improved fault-tolerance. Our preliminary experiments with thinking models suggested that they may require different prompting strategies; however, their increased cost precluded a more detailed analysis.
Memory. We highlight that our context window itself serves as a form of episodic memory for the agent, in which past attempts, mistakes, and reflections are kept in-context over the course of solving a problem for the benefit of the agent. However, agentic workflows can incorporate memory that persists across episodes for the purpose of lifelong learning (Shinn et al., 2023). While we recognize that persistent memory and lifelong learning are an important next-step in the improvement of agentic capabilities, the purpose of our benchmark is to enable research in improving backup planning. Therefore we leave exploration of persistent memory to future research.
Limitations. We do not evaluate the latest commercial models due to cost. For dataset creation, our work builds on the Spider text-to-SQL. benchmark (Yu et al., 2018). Although Spider has been relatively widely used and has undergone improvements and revisions, we find that it is imperfect and contains instances of under-specified questions, as well as data quality issues such as empty tables, which we had to manually filter out. Nonetheless, by basing our benchmark on numan-generatee source data, we can verify generated answers match the reference without being prescriptive about the solution, which is important for our research questions. As previously mentioned, we find that the studied LLMs can be sensitive to the prompt specification. Therefore, we use an established prompting framework based on ReAct, but we acknowledge that further L.M--specffic prompt design could improve the performance of specific LLMs. Additionally, most SOTA models are trained to use specific prompt templates for tool-calling. However, these templates are not standardized across models. To control for these differences, and to make implementation tractable, we use the default instruction following capability of these models instead of custom tool- calling functionalities. We hope that our work motivate future work in improving the fault-tolerance of LLMs, via prompt design or more fundamental training changes.
Acknowledgments
We would like to thank Rachel Wicks, Cristina Aggazzotti, Rafael Rivera-Soto, Aleem Khan, Ashi Garg, Jack Zhang, Ricky Mouser, and Jiefu Ou as well as our anonymous reviewers for their spirited discussion and feedback.
9Gemini 2.0 blog post
10

---

Published as a conference paper at COLM 2025
References
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, and Michael Petrov et al. Evaluating large language models 03374.
trained on code, 2021. URL
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language 05128.
models to self-debug, 2023a. URL http::ss//aaxxiv.org/////////
Zhipeng Chen, Kun Zhou, Beichen Zhang, Zheng Gong, Wayne Xin Zhao, and Ji-Rong Wen. Chatcot: Tool-augmented thain-of-thoughe reasoning on chat-based large language models. arXiv preprint arXio:2305.14323 2023b.
Yu Du, Fangyun Wei, and Hongyang Zhang. Anytool: Self-reflective, hierarchical agents for large-scale api calls, 2024. URL htttt:::://arxi..orr/bbbbs/
04253.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Pandey, Zhiwei Zhao, et al. The llama herd of models, 2024. URL https:////xxxiioorrrr http://wwwwrr/a/////2/222222
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing, 2024. URL https:t/aaxiii..rrr wtt.gr////b/./../////e/// 11738.
Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings. arXiv preprint arXiv:2305.11554 13252.
2023. URL https: / http://axxwv.oggbrb//////a
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint ar arXit:2103.08874 2021.
Arian Hosseini, Alessandro Sordoni, Daniel Toyama, Aaron Courville, and Rishabh Agarwal. Not all Ilm reasoners are created equal, 2024. URL https::/arxxi.orr/bbb//1111 01748.
Mete Ismayilzada, Debjit Paul, Antoine Bosselut, and Lonneke van der Plas. Creativity in ai: Progresses and challenges, 2024. URL
17218.
Dongwei Jiang, Alvin Zhang, Andrew Wang, Nicholas Andrews, and Daniel Khashabi. Feedback friction: Lims struggle to fully incorporate external feedback, 2025. URL
11930.
Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei Xia, and Brian Ichter. Chain of code: Reasoning with language model-augmented code emulator, 2024. URL httpps://axxiis http:////axxvvooo/aabsss// 2312. 04474.
Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A benchmark for tool-augmented llms. Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. URL https://aclanthologg org/2023. emnlp-main, 187/
Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, et al. Toolsandbox: A stateful, conversational, inter- active evaluation benchmark for Ilm tool use capabilities. arXiv preprint ar Xio:2400800662) 2024a. URL h //tp:/../xvvv////oo ooo..rggg/oooo hatgg//b///// abbs/208..a 2408.04682.
Yining Lu, Haoping Yu, and Daniel Khashabi. GEAR: augmenting language models with generalizable and efficient tool resolution. In Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2024b. URL https://arre http:://xxvvvvvrro org/ abs//
Yining Lu, Dixuan Wang, Tianjian Li, Dongwei Jiang, and Daniel Khashabi. Benchmarking language model creativity: A case study code generation. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2025. URL
https://arxiv. wwtpriii.oo//bb////0000 09007.
11

---

Published as a conference paper at COLM 2025
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegr- effe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bod- hisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In Advances in Neural In- formation Processing Systems (NeurIPS), 2023. URL
nips. cc/paper lameerffllee files/ aasrsrrcccoff-staaaatt---------- coneacttt-oooefeneee html.
paper/2023/hashh
Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models. Xiv preprint www:::2255.1225555 2022.
Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis, 2023. URL https://arxiv.. http:/iwww.org///////e// 15334.
Cheng Qian, Peixuan Han, Qinyu Luo, Bingxiang He, Xiusi Chen, Yuji Zhang, Hongyi Du, Jiarui Yao, Xiaocheng Yang, Denghui Zhang, Yunzhu Li, and Heng Ji. Escapebench: Pushing language models to think outside the box, 2024. URL https: /arxivv. org/ /bss/ 2412. 13549.
Shuofei Qiao, Honghao Gui, Huajun Chen, and Ningyu Zhang. Making language models better tool learners with execution feedback. arXiv preprint arXio:2305.13068 2023. URL
https://akkvv.rrr/bb/228... 13252.
Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang. Chaojun Xiao, Chi Han, et al. Tool learning with foundation models. Xrxiv preprint Xxi4:2304.08854, 2023a. URL httpps///aarxi..ogabbb//////
08354.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023b. URL https://arxiv org/abs/ 2307. 16789.
Qwen, ;, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL /tttts:///axiiv htt.::s://axi..or/aabb/////////
15115.
William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators, 2022. URL https: //trxxv.rrg/abs///// htsp:/as//2222 2206. 058022.
Timo Schick, Jane Dwivedi-Wy Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. ToolFormer: language models can teach themselves to use tools. arxiv preprint arXio:2302.04761 2023. URL https: //arxiv. saarxiv.org/abs2222..00 04761.
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. In Advances in Neural Information Processing Systems (NeuriPS), 2023. URL /ttps://arxiv. http://wwwwwg///////////e arg/abss23003. 11366.
Jimin Sun, So Yeon Min, Yingshan Chang, and Yonatan Bisk. Tools fail: Detecting silent errors in faulty tools. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 14272-14289, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 1015533/11222..mm/// m99n...0. URL https::/aclanthol// 2024. emnlp-main.
org/
Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Haitao Mi, and Dong Yu. Toward self-improvement of lims via imagination, searching, and criticizing, 2024a. URL https: //arxiv. barxiv..rgg/ab//4444.1 122522.2.. 12253.
12

---

Published as a conference paper at COLM 2025
Yufei Tian, Abhilasha Ravichander, Lianhui Qin, Ronan Le Bras, Raja Marjieh, Nanyun Peng, Yejin Choi, Thomas L. Griffiths, and Faeze Brahman. Macgyver: Are large language models creative problem solvers?, 2024b. https:://axxx.. http://ww.org//b///////// wtt./r///////////////////// /tlpb///311.99......
Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. AppWorld: controllable world of apps and people for benchmarking interactive coding agents. arXiv preprint ar (iii:24477119000 2024. URL https: /wtp/v/aavvv//aa///a//eeeeaaaa
http:///aawawwa/aa///////// www.://///bbb///////// 18901.
Boshi Wang, Hao Fang, Jason Eisner, Benjamin Van Durme, and Yu Su. Lims in the imagi- narium: tool learning through simulated trial and error. arXiv preprint ar Xrxx::2403. 04746, 2024a.
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better Ilm agents, 2024b. URL https://arxt otg.vi.oo.///oooooooooo org/abs/ 01030.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research (TMLR), 2022. URL h htt.://www...ttoooooooot////////
Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: A benchmark for real-world planning with language agents. In International Conference on Machine Learning (ICML), 2024. URL / pdf/2402.011 016222.
Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt4tools: Teach- ing large language model to use tools via self-instruction. ar Xiv preprint ar XfX::23305.18752, 2023. URL https: //axxvaaa/vvv///////// org/ 18752.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023. URL http:::///ppeeevieeee wtwprrvvie...ne/dd......
id=WE vluYUL-X.
Junjie Ye, Zhengyin Du, Xuesong Yao, Weijian Lin, Yufei Xu, Zehui Chen, Zaiyuan Wang, Sining Zhu, Zhiheng Xi, Siyu Yuan, Tao Gui, Qi Zhang, Xuanjing Huang, and Jiecao Chen. Toolhop: A query-driven benchmark for evaluating large language models in multi-hop tool use, 2025. URL https:://axxii.orrr abs 2abs/2500 255./22566......
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingming Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large- scale human-labeled dataset for complex and cross- Pross-domain semantic parsing and text- to-SQL task. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun' Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3911-3921, Brussels, Belgium, October-November 2018, Association for Computational Linguistics. doi: 10.18653/ 1118663//1/D11// w1///111DDD112// 1425. URL https: //tp::::///aaaatee/ hwwpt//hlll...oo.....//.. org/D18 1008-11111 1425/.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. issructtio-following evaluation for large language models. ar preprint Xix:22311.0777 009911, 2023.
Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. Toolqa: A dataset for Ilm question answering with external tools. Advances in Neural Information Processing Sys- lems, 36:50117-50143, 2023. URL https: /twpe//ediicc/.o...///oo./oo/ose neur ips. cc/ cc/paper files/paper/ _nd Benchmarks. pdf.
2023 2t0p://11e9922222729990000622260 21ta://2290000000222222226221661
13

---

Published as a conference paper at COLM 2025
A Additional Task Information
We here provide further context into the task definition discussed in $3.1.
Function search. To allow an agent to explore what tools are available in our environment, we provide two special functions. The output of each function is observed and processed by the agent in accordance with CodeAct. The first function, search.tools, accepts search query as input and returns a list of relevant tool names with a brief description. The number of tools returned can be specified by the agent. Should the agent wish to learn more about tool, such as the names of arguments, can call the second function get info. This function accepts a tool name as input and returns the full documentation for the tool, which includes a list of required arguments and description of their function. We design our search functions in this way in the hopes of reducing the number of tokens LLM must process in context: documentation can contain many tokens, and therefore should only be returned if necessary.
Studying external errors (continued). For given problem we are given knowledge of Fs, or the set all functions that can lead to a correct solution. We can replace the first function f Eg that the agent tries with modified version f that always throws an error. With the function f effectively unusable, the agent must discover and compose the other functions in Fs over the course of solving the problem. 10 We therefore view our method of introducing inexplicable function errors as means to study the ability of language agents to find successful backup plan when their original plan fails unexpectedly.
B
Additional Analyses
Model
Single Call
Multi Call
Ratio (Single:Multi)
Gemini 2.0 (Flash)
38.3
43.2
355:472
GPT 4o
39,9
37.3
358:472
Llama 3.3 (70b, Instruct)
30.0
49.5
390:400
Llama 3.1 (70b, Instruct)
14.5
40.0
545:285
Qwen 2.5 (72B, Instruct)
38.9
41.8
445:385
Table 5; Accuracy stratified by number of tool calls in backup plan for tool search results in
Table 1. Backup plans involving multiple tools are not substantially more challenging than those involving single tool
Are backup plans involving multiple tools more challenging identify than those involving a single tool? For the results in Table 1, we stratify accuracy based on the number of function calls involved in the available solution, and find that, if anything, single call solutions were more challenging for few models (Llama). Otherwise, we did not observe much performance asymmetry. In addition to the accuracy, we provide the ratio of single function calls to multi function calls and show that ratio is fairly balanced.
Are different models getting the same questions wrong? For the results in Table 1, we calculated Fleiss's Kappa to see whether different models "agreed" on the incorrect questions. Our results indicate agreement of .246, showing "fair" but somewhat low agreement. This suggests that different models tend to get different questions wrong. In other words, model behavior rather than inherent data uncertainty accounts for our results.
C Additional Benchmark Creation Details
We provide further context into the benchmark construction discussed in $3.2.
Recall that in practice, there will always exist multiple subsets of that lead correct solution
14

---

Published as a conference paper at COLM 2025
Function documentation (continued) We provide the language model the code for the function itself, a brief description of the database it accesses, and two in-context demonstra- tions. To ensure the quality of our synthetic documentation, we pass them through series of unit tests, which include checks for the presence of important fields and alignment with code. If the generated documentation fails to pass the checks, the language model has three attempts to correct itself. In practice, we find that documentation which passes the unit tests produced for all tools we create.
Names of functions and arguments. LLMs train on large amounts of code during pre- training. The effects of code seen during pre-training can confound the conclusions of our benchmark- the agent may rely memorization of prior examples to complete the task, code in our
rather than genuine reasoning. To control for the effects of previously benchmark, we give un-descriptive names tools and their input argument. This way, the purpose of a tool cannot be guessed through its name or arguments, forcing the agent to consult documentation.
Function composition (continued). While the process of creating Python functions from SQL subqueries may appear straightforward, deducing the arguments for a function fn based on the outputs of the previous tool fa-1 is not. Tool outputs take the form of list of JSONs, but tool inputs in our benchmark take the form of SQL literals. To reconcile this difference,
extract values from the JSON list by field name, creating mapping field name (str) - values (list). This allows us to set the input arguments for a tool T, to the fields of the previous fn-1, where the expected input for each argument is a list of values. This introduces another assue-inputs are Pythonic datatypes, but the underlying functionality is in SQL.. reconcile the input values of Pythonic datatypes with the wrapped SQL query, we create a temporary SQL table called scratchpad that loads the input values to the function. This and other implementation details can be found in an example of Python function below.
Data augmentation. Consider the instruction "What are the employee ids of employees who report to Shelley, and what are their salaries?" This corresponds to the paramter- ized query SELECT employee.id, salary FROM employees WHERE manager-id employee_id FROM employees WHERE first_name function creation pipeline, we know for certain instructions, the input values for calling the appropriate function are present verbatim in the text (e-g., "Shelley"). We can use this knowledge to alter the value that appears in the instruction with other values present in the same column of the database, We can also parameterize the query with these alternative values and run them against the database to obtain the updated reference solutions. As result we are able to more than double the size of our benchmark.
(SELECT
?). As a byproduct of our automatic
C.1 Function Creation Example
We provide an example for creating executable python functions from SQL commands (discussed in $3.2).
1. SQL We begin the process of automated Python function creation with SQL query.
SELECT employee_id _d. salary FROM employees WHERE manager id (SELECT employee_id FROM employees WHERE
first_name 'Payan')
2. Parsed SQL We convert the SQL command into parse tree using SQLGIot (Figure 3.2), where each node represents expression:
Select(
expressions=[
Colunn(
chiss-ddntifiierrhhhsseeppppp.e quoted=False)),
Co Collunn(
fron=Fromt thiis quoted=False))].
this=Table(
15

---

Published as a conference paper at COLM 2025
thissrreett/iierrrthsseemm.pppp qusted=False))),
where=Where(
this=EQ(
this=Column(
this= thiss-Identifierrhhhssmmaaaa quoted=False)).
this=Select(
expressions=[
Column(
this
quoted=false))1,
Fran=Fran(
this=Table(
bhiss-Identifiierrthiseeeeeeppd quoted=False))).
wheresthere(
this=EQ(
this=Column(
this hhiisstentiifeerrhsssfrrrr.. puoted=false).
popresstoonLieeralthitssseae (s_stringg-rue))))))
3. Query Parametrization We parameterize the query by substituting literals with place- holder variables. This is done deterministically with a septh-finst-search for literal type expressions.
Select(
expressions=[ Column(
thijs-/oentifeerrhhheeeeeeer...e quoted=False)).
Column
thissTdentiiieerrhissssaae quoted=False))],
Fron=From(
this=Table(
this==Identifier (hhisemmeeserrsssssseeepeeeee quoted=False))).
where=Where(
this=EQ(
this#Column(
this:identtffierchhhsmmnaaaer.. quoted=False)),
expressiio=SSuqqeer((
this=Select(
expressions=[
Colunn{
MMlssItdettfiferrrhssseemlyooe. quoted=False)),,
fran=Fron(
this=Table(
thisstreeddiileerhssseeeeeeeee, Fasse))
where=Where(
this=EQC
this=Column(
thissteentiiierrthissiiirs.... quoted=False)),
expreessionPPlacchoolee(([))1]]
We use SQLGlot to convert the modified parse tree back to the following parametrized query:
SCLECT employen_id, salary FROM amployees WHERE manager (SELECT employee_id FROM employees WHERE first_nane ?)
4. Subquery Decomposition In cases where subqueries are present in the SQL query, we use our parser to locate and extract the subqueries. Considered in conjunction with the parametrized query from the previous step, this provides synthesized queries.
SELECT employee_id FROM employees First_nane
SELECT employee_id, salary FROM employees WHERE manager id (SELECT FROM scratchpad)
Function Wrapping Having parametrized the queries and identified the potential input arguments, wrap the queries in Python function. Note that SQL query with single subquery can be made into functions (function 985, function 986, and function function.987) _87). Of these functions, calling function.ge -979 is equivalent to calling function.9es 985 followed by function Sunction_986,,o assuming correct input arguments.
def function. (aw.:oon_95111mmmm 985((ambda_sst signa: str):
APU SETUP Add
16

---

Published as a conference paper at COLM 2025
connect database
db_path wtt.aa//aaaaaaseettttiiitt...e.e
conn sglitee.comeett(b.l connect((bb.pat))
cursor conn.cursor()
query "SELECT employee_id FROM employees WHERE first_name ?"
args (lambda_sia gna,)
cesisorreecuteequeeee. args)
res cursor. fetchalltt
[description[e] description cursor. description]
data = [dict(2ip(columns row)) for row in res]
ccrsor) conn. close()
### *##
if not (en(aaaa) or not len([_ for in res[0] if is not Nonel):
from difflib import get_close_matchh
var var_identifiers
siinadddssiaaa.: "first__aame")
var_aliases [""ambdd__ass [""anbda__iinaa"
var_types ""lanbda_signa"" "str"}
var_data [1anbda_signa]
var_values dict(zia(var_aliase, var_data))
with ("Loon"u teooupp_tbbe_cacheee jas.cccccheejs.... json" 'r') as infile:
colum_data__oo dotka_loooopppp Json. load(infile)
for alias, identifier_ in var var_ her__ieetiilerr.tteesse poolum_data__ookke hrrpphhhrr11 hr_ Ii valuess
hint_data arg_value var_values[alias]
if var var_ tapestyllasil = 'str' and arg_value not in hint_data: hint_data (string.lower():
for string hint_data)
print(f" not find entry with "[alias]" value "(arg_ "[arg_value)"))
close_matches ses_close_aatceesarggvall.. sattches(rr_.aluue..oerrr List(hint__ata.keas)) n=1, cutoff
F.8)
if natches print(f"Did you (hinn__atafcloee_natcceets
data
def
function function_996(benn (etttaa6_e_beplllieeee epstiln:: list):
### PREPROCESSING ##*
organize data list input_data [[beta_ppsill # identifier aliases to names
identifier_info ("EMPLOVEE_IDS "beta_epsilon") identifier_types b'tbeta_eep beta_eepsslon'. "list")
identifier result.. for identifier_alias value in deddetifierriioovallleeee
to data
if type(d) 1ist else [d] for d in input_datal))
input_data):
assert firreerr_tppeiiddniiiieeerrr filassrrrraaaa___eeeaeaaaa_ = str(type(value). maaaaallee..eem........ name__) f"Expected {
identifier Shentifier__tyes_iddee weddennifler_aliiiie but got (type(value)) for (identifier_allas)
transpose result into result
list lest_lenghh-..
result
for range(List_lennth))
temp_record ()
identifier, values ittmss)
[emp__reoodd[idenifiiee values[i]
result. (eaittt.epenn(tem_eeerrrr)
### SETUP 5##
# connect database
db_path
www.ddaaaaaaaeeeer....... iqslltee
sqlite3. www.eecttdbbppaahi..
cursor conn.cursor
# create scratchpad quates *\""
EXISTS scratchpad') remove scratchpad if
temp_quers = f'CREATE TEMPORARY TABLE scratchpad (I(', "Join[[[k for k, in items execute query)
for temp_record in result:
temp_insert INSSERT INTO scratchpad (C(", ".join([k for k, in irsult..items())1 VALUES
((", ".join[[quotes str(val) quotes type(val) str else str(val) for val in temp_record. values()1)))); www.eeeeeeetteepieeeeeeee
AV8 EXECUTE QUERY o##
query "SELECT employee_id, salary FROM employees WHERE manager_id (SELECT FROM scratchpad) cursor.exes
sursoorexeecte execute
fetchall()
columns [description[e] for description in cursor. dessrippestiiiion] data [dict(rippcollums, row)) for cursor)close()
conn. close()
17

---

Published as a conference paper at COLM 2025
not (en(dat.)) or not. len([_ for in res[8] if is not Nonel):
from difflib import get_close_matches
var_identifiers
var_aliases = [
var_types
var_data
var_values dict(zip(var_aliaaee var_data))
with colum_data_
tacle__chehccc jaach"..son".. 'r') as infile:
json. load(iiile))))e)
alias, identifier_
air_identifiers.ffi iters.
hint_data column_data_ lockp_ horppp[""h_"""" hr_ hr_11 valluesss" value [ar_values[alias]
if var var_ -tt_tttssaaaaa]]] hint_data stwing.lowerr(: string for string in hint_data)
"str" and arg_value not in hint_data:
print(f"oould not find entry with "(alias)" value "(arg_value)"
close_matches get_close_ (araosssattceessrgaallllu vareeesarrr.alle.ooerrrrr 1ower(), list(hint_data, keysO), n=t, cutoff
if close_matches.
print(f"Did you cannt_dbtaccloe_mmtcheeee ?")
return data
def
function_ 987 [laamda_eppsilon: str): ## SETUP 8nd connect htterr/aaaaaaaaabeeer////// sqllite"
db_path "spider_data/daee
sqlite3 (am_ecttddb_ptt))) cursor conn.cursor()
QUERY
query "SELECT employee_id, salary FROM employees WHERE manager_id = (SELECT enployee_ id FROM
employees WHERE first_name
args = (lambda_epsilon,, cursor. equeuue args)
res cursor.fetchalle
columns [description[e] for description in cursor, description]
[dict(2ip(columss, row)) cursor.close() close()
### HINT ###
len(data) or len([_
res[0]
Nonel):
from difflib import get_geeee_estttteeeeeo matches
var_identifiers= "Lanbda_epsiloo""" "first_name")
var_aliases ["'labbd__epsioon]]
var_types
"str")
var_data [lambda_epsilon]
var_values = dict(cii(var_aliasee, var_data))
open ("Lookup hatpppttall_eceeeeeee www.laecccheeeeeeeeeeeeeeeeeeeee jsone 'r') as infile:
jso.. loadinie))
for alias, identifier_ in var_ mr__identifierss wtt.etiiieesstteess/
hint_data bolumm_dtt__oockpp"hh hr_ "ir arg_value var jar_valuestalias] var [yr_ttys[[alls]]]] hint_ print(f"Could find entry with "(alias)" value "aagevallue)""
arg_value
hint_data:
string for string in hint_data)
close_matches
()).., (isthhin__dtta.kess()) n=1, cutoff
=.8)
if close_natches: (fint you mean (diint_dbtacccloee.aatchcess ?")
return data
D Examples of documentation
The following are examples of documentation produced automatically via LLM prompting.
Function_985: C
"type": "function", "Function":
"name": "Function_985""
"description": "Get the employee_id from the "employees" table for specific first_nane. The employees table stores information about employees in multinational corporation's human resources management systen"
"arannters":
"type" "object",
"properties":
"lanbda_ sigma":
"type": "str".
"description': "The First_nane query"
18

---

required": "eta_epsilon" Published as a conference paper at COLM 2025
}, required"; [ "Lanbda_signa"
function_9ea0 { "type": "function",
"Function":
"name": "function_s$e 986",
"description": "Get the employee_id id salary from the "hr_1" database for employees wi nanager_id _id natching the provided input. The "hr_1" datahase hunan resources management system storing infornation about employees, departments, job roles, locations, and countries, as well mmpooyeent history and organizational structure.", "arameters";
"type": "object",
"properties":
"eeta_epsiion": {
"type": "list", "description:: "A list of manager_ids to query"
Function_s$1. {
"type": "function",
"Function":
"name": "function_ 987",,
"description': "Get the employee_ id and salary of employees who report to a manager with retrieved fron human resources management systen database."
specific first The
"parameters"
"type": "object",
"properties": "lanbda_epsilan {
"type": "str".
"description": "The first name of the manager to query"
required":
"lanbda_epsilon
}
We include the documentation used for search functions.
search_tore tools: {
"type": "function", "function;
"nane": "search_tools"
"description": "Retrieves descriptions tools that are relevant to a provided search query. This function be used explore what tools are available to use.",
paraneters":
"type": "object",
"properties":
"query":
"type": "string",
"description": "description used search for tools"
tu
"type": "integer",
"description': "number results (nax 9)",
"default:
}.
"required: ["search_query"]
)
get, info:
"Eype": "function", "function":
"name": info",
"description"; "Retrieves the documentation for specific tool. The documentation will include the tool 's nane. description, and usage instructions." "paraneters"; "type": "object",
19

---

Published as a conference paper at COLM 2025
"properties tiess:tiees: Ctol_nanee
"type": "string", ddscriptionn: "name of the tool to retrieve documentation for* b.
"required"; ["tooo_namme]]
}
}
E Prompts
We here provide the prompts we use, both for in-context functions and for function search.
E.1 In-context functions
You are helpful assistant assigned with the task of problen-solving, To achieve this, you will be using interactive coding environment equipped with variety of tool functions to assist you throughout the process.
provide your step-by thinking for solving process should be enclosed using "<thought>" tag, for example: <thought> need to print "Hello World!" </thought>.
turn, you
Your thought
After that, have options:
1) Interact with Pythan programming environment and receive the corresponding output. Your code should be enclosed with the "<execute>" tag, example:
print("Hells World!") </execute>. 2) Stop execution when you have encountered correct solution, Save the final answer to the 'solution' variable and enclose the code with the "<solution>" tag, for example: <solution> solution output </solution>.
verify your solution, your solution should contain the values soriginal keys* returned by the tools
Each tool will return list JSONs, and you may need to perform postprocessing on the output. Print the output how the data structured.
Here are some examples (tool definitions in each example have been truncated for brevity):
((
tool defini tions omitted...
Task:
What is the names movies whose created year after all novies directed by Steven Spielberg?
<thought> I can use function_3ad to get the movies released after Steven Spielberg's 's last movie </
thought>
<execute>
output function_ 363466999666 'Stevnn Spielberg') result liten! movie' for item in output] (rinu(result) </execute>
Observation:
Traceback (nost recent call last):
Cell In[1], line
result moviee' for iten output]
Cell In[1], line 2 in <listcomp>
result [item[" novie' for output]
KeyError: 'novie'
Assistant:
<thought> 'movie' doesn't 't appear be a key in the outputs. To correct the error, I will first try printing aut the output of function_3694 </ thought>
<execute>
output function_3694C 'Steven Spielberg') (rint(output) </execute>
20

---

f"Expected [[field_ feaelldttyypsfffldd__mmmmmm Published as a conference paper at COLM 2025
Observation:
[[[ titll: 'Titanic')), ( (Cttitl': "Avatar']] []]
Assistant:
<thought> The output seens reasonable. The novies released after Steven Spielberg's 's last movie are Titanic' 'Avatar'. will report solution below </thought>
<solution>
solution output
</solution>
{
teol definitions omitted...
})
Task:
Find the average age of students who are living in the dorm with the largest capacity.
Assistant:
<thought> I need to use function_36a 3643 to figure out the largest capacity, then use Junction_6644 to get the average the students living dorn with that capacity. will first functioon_343 thought>
<execute>
output function_3643()
(rint(ou)put) </execute>
Observation:
[E( "#0(ssuueen__ccpccttt) 408))]
Assistant:
<thought> will postprocess the output of Function_3643 the input function_3644
thought>
<execute>
input output[e]] WWW(stueent_ccaacciii
output2 Function_3644(input))
print
</execute>
Observation:
Traceback (most recent call last):
Cell In[1], line
output2
(anction_364(inputt]
File "/sandbox py: in function_3644
assert typeld_tppssfiele ield_nane]]]]]] == str
got ((type(value)) [[field_name))
AssertionError: Expected float
<class 'int'> for xray_tau
Assistant:
<thought> seens my had an error. Based on error, will convert the input from int to a float </thought>
<execute>
input1 output[e]] MA (studeeett__ccpacccc "y
output2 Lunction_Bee(1oaat(eeee) print(autput?) </execute>
Observation:
[[( (''AGG(T1.age)) age))) 19.6)]]
Assistant:
<thought> The result seens reasonable. The average age of the students in the dorm with the largest capacity is 19.6. will report the solution below </thought>
<solution>
solution output2
</solution>
Here are sone the tools you can use for the current task: [FUNCTIONS HERE]
E.2 Function search
21

---

Published as a conference paper at COLM 2025
helpful assistant assigned the of problen-selving. achieve you will using an interactive coding environment equipped with variety of tool functions to assist you throughout the process.
At each turn, you should first provide your step-by-step thinking for solving the task. Your thought process should be enclosed using "<thought>" tag, for example: <thought> need print "Hello Morld!" </thought>.
After that, you have twa options:
Interact Pythan programming environment
corresponding output. Your should be enclosed with the "cexecute>" tag, for example: <execute> print("Hello World!") </execute>, execution you encountered correct solution. the final variable and enclose the code with the "<solution>" tag, for example: <solution> solution output </solution>.
the "solution"
To help verify solution, the tools
solution should contain the values and soriginal keys* returned by
Each tool will return list of ISONs, and you may need perforn postprocessing the output. Print the output to see how the data structured.
some examples (tool definitions each example truncated for brevity):
((
tool tifinitions omitted...
}}
Task: what is the names of movies whose created year is after all movies directed Steven Spielberg?
Assistant:
<thought> find the names to find the year of Steven Spielberg's last movie and then find all the movies that were released the search_tools function to find relevant tools that help with this task. </
created after all movies directed
Spielberg. first need
after. I
thought>
<execute>
search_ (querr novie directors', num_results=9) </execute>
observation:
[' ("(["Function :3599" "Get the number distinct novie titles directed by a specific director. The indb database stores information about novies and TV series, including their casts, crews, genres, keywords, production details.
"(("function_415 "Get the directors from the "movie_1" database who have directed a specific number of movies. The novie_l database stores information about movies, revievers, and their corresponding ratings. "}>".
"[["functio_]F) 74"" "Retrieve list of film titles and their corresponding directars from the cinema
database." "}3'
"{("function_2") 2369"" "Get list of distinct directors from the 'film_rank' database. The film_rank database stores information about films, including their titles. studios, directors, and box office gross, as well as narket estinations for these films in various countries. ")) ("function_3429" "Get the count of distinct novie titles directed by specific director and released before a certain year. indb database stores information about movies and series, including their casts, crews, genres, keywords, and production details. ")]" results truncated]
Assistant:
<thought> These tools don't seen like they would help us solve the task. should refine ny search query to be more specific and see if that returns more relevant tools </thought>
<execute>
search_toolse (qurrr novie directors year', num_results=)) </execute>
Observation:
("(""uunction_Ba "Get the count of distinct novie titles directed by specific director, released
within specified year range. The imdo database
information about novies and TV series, including their casts, crews, genres, keywords, and production details. det 33',
"("function_3429) "Get the count of distinct novie titles directed by specific director and released
before a certain year. The indb database stores information about movies and series, including their casts, crews, genres, keywords, and production details. "Get the after a certain year. The indb database stores information about movies and TV series, including crews, genres, keywords, and production details. "}}", "[[" [[""unction_ 1451"; "Get the distinct directors films released data from the "filn_rank" database, which stores information about films and their market estimations. "))'
[(" ("function_1933)
of distinct novie titles directed by specific director released
their
specific year. function uses
"{" [("Function_3e6) 3692" AGet the latest year of release for movies directed by specific director. The data sourced from "novie_1" database, which information about movies, corresponding patings. "}"" results truncated]
their
22

---

Published as a conference paper at COLM 2025
Assistant:
<thought> can use function_36S3 to find the year of Steven Spielberg's last novie. should find out nore about Function_3692 using the get_info tool </thought>
<execute>
get get_info(" Det_info"'uuctionn.s5 function_3692")
</execute>
Observation:
[(
tool definitions smitted....
}}
Assistant:
<thought> can call function_3692 with the lambda_mu parameter to get the of Steven Spielberg's
last novie </thought>
year = Munnciin_3692(aambbln "Steeen Spielberg') pyeat </execute>
Observation:
[[] (YEA(Y)))):: 1982))]
Assistant:
<thought> Steven Spielberg's last movie released in 1982. I need to find all the movies that were released after. can use the search_tools function to find relevant tools that can help task </thaught>
<execute>
search_tools( novie after year', run_results:3) </execute>
observation:
(((""unntioo_389"" "Get the titles of movies released after a specified year. The movie_1 database information about movies, reviewers, and their corresponding ratings, allowing for the tracking and analysis movie reviews and ratings from multiple reviewers. "))', "[("function_ 3628"; "Get the number of unique novie titles released in specific year. The indb series, including
database stores keywords, and praduction details. "I3' "Get the
casts, crews, genres,
of distinct novie titles directed by specific director after a cer certain year. The indb database stores information about movies and TV series, including their casts, crews, genres, keywords, and production details. ")}" [[("uncctoo_ : "Get movie titles from imdb database stores information about novies and TV series, including their casts, crews, genres, keywords, and production details. ")]"
"(" (("function_18)
released
"imdb" database
a specific genre and release year.
[("Function_Bed ieattioon3688"" "Get the count of distinct novie titles directed by specific director, released within specified year range. The imob database stores information about novies TV series, including their casts, crews, genres, keywords, and production details.")) results truncated]
Assistant:
<thought>
function_ 3693 find the titles nore about function_3693 using the info tool </thought>
released after should find
<execute>
get_ itt_innf((tol__name (aaecttoooollloaeeee Funnniofnnnioo_36533 </execute>
Observation:
((
toal documentation tted.ed...
Assistant:
<thought> can call function_369p with the beta_papa parameter to get the titles movies released after 1982 </thought>
<execute>
titles Function_ [umnctoon_detteee..aaaaaaarllll (AX(EEE))
print(titles) </execute>
Observation: Traceback
call last):
Cell In[l], line titles function_3693(tee (eppbbetaapppayyer([00 (AAX())))) 'J)
"/sandbox_ude" wttooouuullllpppp. wt:lss.p::15922. 159922 function_3693
assert identifier wwwetyyeeeeeniiierrrrr.ee
нu (tr(ttyeevaluee..... (*Expected ([identifier_tyye
Aieettfferr_llass]] but got (vtype(value)) for ([identifier_aliss)
AssertionErron: Expected float but <class 'int'> for beta_papa
23

---

Published as a conference paper at COLM 2025
Assistant:
<thought> It seens my cade had an error. Based on the error, will convert the input. from an int to float </thought>
<execute> titles htwpaa/w...iiltt..e.ooooooaraaaa
print(titles)
</execute>
Observation:
EEC title': 'Titanic'),, ((tiitle:: 'Avatar') 3]]
<thought> output seens reasonable. movies released after Spielberg's last Titanic' and 'Avatar'. will report the solution below </thought>
<solution>
solution = titles
</solution>
Task:
Find the average age of students who are living in the dorn with the largest capacity.
Assistant:
<thought> To find the average age students in dorm with largest capacity, first need find the dorm with the largest capcity and then find the average age students living there. can use the search_tools function to find relevant tools that can help with this task. </thought>
<execute>
search_tools toonttoossssus=taaare students capacity", rus_rsus))s=)) </execute>>
Observation:
[ [("Function_3644 "Get the average age of students living in dorms with specific student capacity. university
The function uses data from 'dorm_1" database, which manages information students, including their personal details, dorn assigrments, and the amenities available in each "))'
"[("function336k 3643"" "Get the maximum student capacity from the dorn information. The dorm information is managed by the "dorm_1" database, which stores data about university students, including their persenal
dorm assignments, (("Function_2(1) "Get the total student capacity across all dorms. The data is retrieved from the dorm_1" database, which nanages information about university students, including their personal details, dorn assigrments, and the amenities lvailable each dorm. "33'
ameni
dorn. "}}',
(("Function_3ek) 33445" "Get the average age of students living in the dorm with the highest student capacity
The function retrieves data from the "dorm_1" database, which nanages information about university students, including their personal details, dorm assignnents, and the anenities available in each dorm."]]"
[[""uunctio__2) 2676"; "Get the count of students and dorm names for students older than a specified age. The data is retrieved from the "dorn_1" database, which manages information about university students, including their personal details, dorn assignments, and the amenities available each d))
results truncated]
<thought>
function 3645 directly the average age students highest capacity. should find out more about function_3645 using the get_info tool </thought>
the dorn with
<execute> get_ famotttoo_nameeuuuciiiiii </execute>
Observation:
((
toal documentation anitted...
<thought> call function_3645 find the age of students living in the with highest capacity </thought>
average_age function_3645() print(average.du) </execute>
Observation: [E( (WG(TT.age)' 19.6)3]
<thought>>
The result seens reasonable. The average age of the students in the dorm with the largest capacity is 19.6. will report the solution below </thought>
<solution> solution average_age </solution>
24

---

Published as a conference paper at COLM 2025
with, you will access tools help you explore other tools that available use.
search_tools =
"type": "function",
"function":
"name", "search_tools tooltttoo",,.,.
"description": "Retrieves descriptions of tools that relevant provided search query.
This function can be used explore what tools are available to use.",
"parantters";
"type": "object",
"properties":
"query":
Type": "string".
ddscrripionn: "description used search for tools"
nun "nun_results ts::
"type": : "integer",
description"; "number of results to return (nax 9)", "default":
"required"; ["search_query")
get__nfoo
"type": "function",
"Function":
"name": "get_info",
"dsccripiion" "Retrieves the documentation for specific tool. The documentation will include the tool 's nane, description, and usage instructions.".
paraneters":
"type": "object",
"properties"
"tool_name":
"type": "string",
"description": "name of the tool to retrieve documentation for"
required": ['tool_name"]
Example Trace (Success)
We include example of successful task completion with Qwen2.5-72B-Instrrce
Task: What are the employee ids of employees who report to Shelley, and what are their salaries? <thought>
salaries employees report Shelley, that can query the employee database and filter by manager name. I can use the search_tools function to find relevant tools. </thought>
the oypl
<execute>
search_tools (query="employee manager salary", nun_results=9) </execute>
Observation:
L' [""function_188" "Get the salary of enployee from the "hr_1" database, which stores information about employees, departments, job roles, locations, and countries, as well as employment history and organizational structure. "5²
report manager with specific first name. The data is retrieved from hunan resources management system database. "}" (("function_ 1460": "Get
(""unction__98"" __1"on____""; "Get employee_ id salary of enployees
manager_id employees who The data is retrieved from 'hr_1" database, humtan resources management system for multinational corporation. "}". (""uuctioo__665 "Get
salary
report specific manager.
phone numbers employees with salaries within specified range. The data is sourced from human resources management system for multinational corporation, "}", ("function_39) 3979"" "Get employee information fron the "hr_1" database where the specified range. The 'hr_l 1 database is a human resources nanagement system for a multinational corporation, storing information about employees, departments, job roles, locations, and countries, as well as employment history and organizational structure."). ")'
falls within
"Get enployee information from the "hr_1" database for employees with specific salary employees,
The "hr_1" database departnents, job roles. locations, and countries.">. ("Function_988" *Get the employee_id nanager_id id natching the provided input. The "hr_1" database storing infornation about employees, departments, job roles, locations, and countries, as well as employment history organizational structure," ")',
resources management system storing information
salary fron the "hr_1" database for enployees with
hunan resources management system
25

---

Published as a conference paper at COLM 2025
"("function_ 849": "Get employee information fron "hr_1" database based salary range, commi percentage, and department ID. The "hr_1" database stores information about employees, depar tnents, job roles, locati...
<thought> can use function_
get the employee_ and salary employees who report manager with specific first name. should find out nore about function_9$ using the get__info tool. </ thought>
<execute>
get eet_inno(tool_aaeee Funnto--
</execute>
Observation:
"type': 'function', function': { I'name': "function_987.
"description', : "Get the employee_id and salary of employees who report to manager with specific first name. The data retrieved from human resources management system database.", paraneters"' ('type' "bjject',
properties': _laanda_epsslon" ('type': 'str', "description" 'The first name of the manager to query )). required":
<thought> I can call Function_987 with the lambda_epst epsilon parameter ter set to "Shelley" to get the employee_id salary of employees report Shelley. </thought>
<execute> result Lariio___87(aadddeeesslllee-eee
print(result)
</execute>
Observation:
Traceback (nost recent call last):
Cell In[1]. line 1
result BerrrinnnB@(Laabaaeeesllloot
File "/sandbox_utiaa" btarbbox_uilllpyyy:.... 42:42 in function_987
raise ValueError (frructoo(" is currently unavailable. Please different function.
ValueError: function_997 is currently unavailable. Please try different function.
<thought> seems function_987 currently unavailable. should try different function. function_988 to get the employee_id and salary of employees with a specific nanager_id. should find out more about function_ 996 using the get_info tool. </thought>
<execute>
get_ toafoottto_aame=nuunettt </execute>
Observation:
[ type ; function',
function': ('name': 'function_985"
"description' 'Get the employee_id and salary from the "hr_1" database for employees with manager_ia input. The "hr_1" database human resources management systen storing information about employees, departments, job roles, locations, and countries, as well as employment history and organizational structure.', paraneters's ('type': "bbject'. propertiess' { 'dscriitiion; 'A list manager_ids query')), rrquured": "'beta_ppsilon'1
natching the
C'type': 'list',
<thought> I can call function_98s with the beta_ beta_epsilon paraneter set to manager_id of Shelley.
However, don't have the manager of Shelley. need to find a function that can get the nanager_id employee their first name. can search_tools function find relevant tools. </thought>
<execute>
search_tool (ooolysquur==emppll manager id", num_results=9) </execute>
Observation:
L'(" ("""unction_1629" "Get information about employees who not belong to depar tments with managers within specifled range of manager IDs. The data retrieved from human resources management system database. "Y'
"["function_98" iisiion__87"".. "Get the employee_id and salary of employees who report to manager with specific resources managenent system database. "Y' "iuuntiion1666" "Get the department_ from the "departments" table for a range of manager iid values. The departments table is
first The data retrieved
of human storing information about employees, depar ampets,,nts, job roles, locations, and countries, as well as employment hi history and organizational structure.")" enployee_id
nanagement system for mul tinationall corporation,
["function_3011) "Get is sourced from a human resources management system for a multinational. corporation." ")" '("function_2 23580" "Get database manages information about music store, including employees and their reporting relationships."" "y"
job_title
employees
department.
the direct reports from the "store_1" database. The store_1
26

---

Published as a conference paper at COLM 2025
("function_588"" "Get employee_1d from "cre_Doc_ Tracking____e De" database for employees with specific roles. The database nanages and tracks documents, including their types, storage locations, and destruction schedules, while also storing information about employees and their roles within organization."
("function_985 986"; "Get the employee_ id and salary fron the "hr_1" database for employees with resources management system storing information about employees, departments, job roles, locations, and countries, as well employment history
nanager_id natching the provided input. The 'hr_1" database
organizational structure. ")", {"function_31881 "Get the first name and job of employees in specific department. The data is retrieved fron human resources management system database. '("function_
"Get...
<thought> I can use function_987 to get the manager_id of Shelley. However, function_987 is currently unavailable. should try different function. can function_152s get the department for a range of manager id values. However, I don't have the manager_id iid of Shelley. I need to find a function that function to find relevant tools. </thought>
the manager
employee by their first name. can use the search_tool tools
(querr manager id first name", nun_resulss=9)
</execute>
Observation:
""ffuctiio_987" *Get employee_
salary employees report manager with specific first name. The data is retrieved from human resources management system database. ")', """fuctioo_311) "Get the first retrieved fron human resources management system database. ")',
and
of employees in specific department. The data
("function_328) "Get the first name, last nane, and hire date of employees in the department given employee. corporation. ")',
data sourced from human resources management system for multinationa]
("fucttion_ 985": *eet the employee_id from "employees" table for specific first_name. The employees table stores information about emplayees in a multinational corporation 's human resources management system")',
function_3e22 "Get the department__i
an employee from the "hr_1" database based on their first nane. The "hr database is huran resources managenent system storing information about employees, depar tments, "rfuncttion111""" "Get employee information from the "hr_1" database for employees with matching first information about employees, departments, job roles, locations, and
roles, locations, countries."
The hr_1 database countr countries, as well as employment history and organizational structure. ")'.
("function_1e 16288" "Get infornation about imployess who do not belong to departments with managers specified range manager system database. ")',
resources management
"uunction_5755 "Get employee information from "hr_1" database employees with matching first name. The "hr_1" database is human resources management system storing information about employees, depar tments, job roles, locations, and countries. "}"
*( ("function_30 " "Get the employee_ and job_title for employees in speci fic department. The data is sourced from human resources management system for multinational corporation. ")']
<thought> I can use function_an (eactinnn_9855 to get the employee_id of Shelley. I should find out more about Function_98s
the get_info tool. </thought>
<execute>
ret_info(tool_nae===Faae
</execute>
Observation:
"type" function',
function': ('name': 'function_985"
"description', "Get the employee_id from the "emplayees" table for specific first_nane. The employees table stores information about employees multiintiona. corporation's hunan management systen",
paraneters': ('type': 'object',
"properties': ('Lanbda_sigma, ('type': 'str', 'description'' 'The first_name to query')), >),
'reuuieed: ('lanbda_signa ']}}}
<thought> call Function_985 with lambda_
"Shelley" get the employee_id
Shelley. </thought>
<execute>
shelley_id = crrrriio_ss@slaaba__siiioee print(shelley_ie)
</execute>
Observation:
[[ <(EMPLOYEE_ID': 285)]
<thought> have the employes_id Shelley, ley, which 285.
call function with the
beta_epsilon parameter set to [205] to get the employee_id and salary employees who report Shelley. </thought>
<execute>
result sitiion__e(beta_eesllllo.
print(resule, </execute>
27

---

Published as a conference paper at COLM 2025
Observation:
[[" ('EEMPLYYE__DD' 286, 'SALARY': B380)]
<solution> solution result </solution>
28


Reviews:
[{"review_id": "gadyu4hUgg", "reviewer": "Reviewer_swx9", "summary": "This paper explores a relatively under-studied but practically important capability of LLM agents: their ability to formulate and execute backup plans when their initial strategy fails due to external errors. To this end, the authors convert a text-to-SQL dataset (Spider) into a planning environment with callable functions and design an interactive benchmark (\"Hell-or-High-Water\") that systematically disables the agent's initial plan by disabling the key function, prompting the model to recover with an alternative. The benchmark includes 830 questions and 4450 functions, each with at least two solution paths. The authors find that across both open-weight and commercial LLMs, model performance drops significantly under such error conditions , indicating a clear limitation in current models' ability to recover from failure. They also investigate the impact of model scale and question underspecification on performance. The study provides useful insights into improving the robustness of LLM agents.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The \"Hell-or-High-Water\" benchmark, which focuses on the capability of LLM agents to formulate backup plans, is reasonably and novelly designed with multiple solution paths and a large function search space. Although some limitations still exist, its advanced benchmark construction scheme nonetheless provides adequate support for exploring this scenario and facilitates subsequent research.\n- The paper reveals that current LLMs, even when faced with explicit error messages, exhibit significant difficulty in formulating and executing backup plans. This offers the latest insights into how to enhance the robustness of LLM agents.\n- The experimental section also achieves in-depth exploration, not only covering a variety of LLMs but also further investigating the impact of factors such as search space size, model scale, and question underspecification on the results.", "reasons_to_reject": "The experimental setup and backup plan design are limited in scope: backup solutions are typically composed of multiple sub-functions derived from decomposing the initial single-function plan, which may introduce a structural bias and uneven difficulty between primary and backup paths. This restricts the generality of the benchmark and fails to fully capture more diverse or logically distinct backup strategies.", "questions_to_authors": "1. Since the backup plans often involve multiple function calls while the initial plan typically uses only one, could this asymmetry affect the observed accuracy drop and confound the comparison between primary and backup strategies?\n2. Have you considered including alternative plans that are logically distinct from the original, rather than just decompositions? Would such designs offer a more comprehensive test of backup planning ability?\n3. Your closed-world experiments (Figure 3) indicate that when backup plans are necessary, performance drops as distractor functions increase. This suggests a key difficulty lies in finding alternative functions during replanning. Could you elaborate on why this search capability seems particularly impaired after an initial plan failure？\n4. You briefly mention that preliminary experiments with thinking models suggested they might require different prompting strategies and that cost precluded a more detailed analysis. Could you provide a little more insight into what initial differences you observed, even if anecdotal?", "extra_scores": {}}, {"review_id": "VT69kAXChi", "reviewer": "Reviewer_DGtT", "summary": "Hell or High Water introduces the first benchmark explicitly designed to test whether language-model agents can recover from failures when their initial tool-use plans break. The authors constructed 830 text-to-SQL tasks from the Spider dataset, each solvable via two distinct tool paths, and systematically disabled agents' initial tool choices at runtime to test adaptive replanning. Evaluations across leading LLMs demonstrated significant drops in performance when initial plans failed, highlighting critical robustness gaps. This benchmark uniquely stresses forced plan adaptation, exposing limitations in current state-of-the-art agent models.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper addresses a critical yet underexplored capability in tool-using agents—robust recovery from failure, which could be a very interesting topic to research community.\n- The paper is well-written and carefully structured, with clear definitions, illustrative figures, and comprehensive experimental analysis.\n- The benchmark is well-engineered, with 4,450 auto-generated, type-checked tools, a clean interface, and full automation of grading.", "reasons_to_reject": "- The paper reports aggregate performance drops when the primary tool fails, but offers limited qualitative or taxonomic analysis of how or why agents fail to recover (e.g., search failures, chaining errors, hallucination, etc.).\n- While the authors use semantic search over 4,000 tools, it remains unclear how feasible or costly it is to run this benchmark at scale in terms of latency, inference tokens, or search efficiency.", "questions_to_authors": "- Does the benchmark control for tool naming conventions (e.g., meaningful vs. obfuscated names)? Could tool discovery be unfairly easy if names leak intent?\n- The name Hell or High Water is memorable, but it might benefit from a more descriptive subtitle for searchability.", "extra_scores": {}}, {"review_id": "VQQs7yGzNS", "reviewer": "Reviewer_3TTA", "summary": "This paper investigates the ability of language model agents to formulate and execute backup plans when their primary plan fails due to external errors such as unavailable APIs. The authors introduce a benchmark called Hell-or-High-Water, built from a transformed text-to-SQL dataset, where each task has multiple valid solution paths involving API-like function calls. They evaluate open-weight and commercial models and find that current LLM agents struggle significantly with fault-tolerant planning. Even when failure is explicitly communicated, agents often fail to adapt or find alternate solutions.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper introduces a new and practical benchmark to evaluate backup planning.\n-  It addresses a crucial capability gap in LLM agents—the ability to recover from failures, which is vital for robust deployment.", "reasons_to_reject": "- Limited agent framework been explored. There are many ways to improve the ReAct, reflection, and even too search methods. These are not explored in this paper. The observed failure is based solely on the CodeAct framework.\n- The effectiveness of reasoning-heavy models or different prompting styles (e.g., self-reflection, tree-of-thought) is only briefly speculated on rather than rigorously tested.\n- Dataset, although clean, does not fully reflect the messiness or ambiguity of real-world APIs.\n- Many recent benchmarks already test tool-use and reasoning; this paper’s unique contribution lies solely in the forced failure and multi-path recovery, which could be seen as an incremental extension.", "questions_to_authors": "- Why do LLMs fail even when the error is explicit and the backup path is simple? Is it due to instruction-following limitations, planning capacity, or search strategy?\n- What are the effects of different prompting techniques? Have you tried ReAct vs. Tree-of-Thought vs. CoT? Any empirical insights? Did you try prompting similar to \"if a tool failed, try another tool\"?\n- How generalizable is this benchmark to real-world tools? For example, can this method be used with actual RESTful APIs rather than synthetic SQL wrappers?\n- Memory should be able to be added in CodeAct framework. Does incorporating memory or trajectory replay help with backup planning? Would a more persistent context help agents recognize and avoid repeating failed paths?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper constructs a dataset to evaluate the ability of LLM-based agents in formulating _backup plans_, i.e., exploring alternative plans when the first plan has failed. The dataset is derived from Spider, a text-to-SQL dataset. Specifically, complete SQL queries from Spider are decomposed and transformed into more atomic Python functions in a deterministic way. Documentation of the functions are generated using an LLM. By construct, each of the tasks has at least two alternative sequences of function calls to complete the goal, originating from different ways of decomposing the original SQL query; this creates the basis for backup planning evaluation. Evaluation on several open and proprietary LLMs show that they suffer a substantial drop when they need to explore backup plans. 

Overall, this is an interesting work. Evaluating the ability of agents in exploring backup plans is an important problem, and it's an interesting idea to derive such evaluation tasks from text-to-SQL that provides some desired properties such as reliability in automated evaluation and a large space of functions. On the other hand, there are certainly various structural biases introduced in the dataset construction. A few examples: 
1. These SQL-derived Python functions are quite different from typical APIs agents deal with in realistic settings. 
2. Even though there are a large number of functions, there could be large semantic overlaps among functions (introduced in an artificial way by construct). 
3. The documentation, which in this setting is probably the most important information for LLMs to select from the functions, is automatically generated by an LLM with some limited unit tests; a human evaluation is likely needed.
4. The alternative plans, by construct, have a very specific relationship between each other. 

As a first attempt at this type of evaluation, the dataset is still a valuable contribution despite the limitations. But it's important to note these limitations; the current Limitations discussion needs to be expanded.
Decision: Accept