{$\tau$}-bench: A Benchmark for \underline{T}ool-\underline{A}gent-\underline{U}ser Interaction in Real-World Domains
Authors: Shunyu Yao
Noah Shinn
Pedram Razavi
Karthik R Narasimhan
Venue: ICLR 2025 Poster (2025)
Keywords: language model
language agent
benchmark
user simulation
rule following
Primary area: datasets and benchmarks
TL;DR: A benchmark where language agents need to interact with simulated users and follow complex rules in realistic domains.

Abstract: Existing benchmarks for language agents do not set them up to interact with human users or follow domain-specific rules, both of which are vital to safe and realistic deployment. We propose $\tau$-bench, a benchmark with two domains (retail and airline) emulating dynamic conversations between a user (simulated by language models) and a customer service agent provided with domain-specific API tools and policy guidelines. We employ a efficient and faithful evaluation process that compares the database state at the end of a conversation with the annotated goal state, and propose a new metric (pass^k) to evaluate the reliability of agent behavior over multiple trials. Our experiments show that even state-of-the-art function calling agents (gpt-4o) succeed on $<50\%$ of the tasks, and are terribly inconsistent (pass^8 < 25\% in retail). Our findings point to the need for methods that can improve the ability of agents to act consistently and reliably follow rules.

Published as a conference paper at ICLR 2025
T-BENCH: A BENCHMARK FOR TOOL-AGENT-U INTERACTION IN WOALD DOMAINS
Shunyu Yao
Noah Shinn
Pedram Razavi
Karthik Narasimhan
Sierra
ABSTRACT
Existing benchmarks do not test language agents on their interaction with human users or ability to follow domain-specific rules, both of which are vital for deploy- ing them in real world applications. We propose 7-bench, a benchmark emulating dynamic conversations between user (simulated by language models) and lan- guage agent provided with domain-specing API tools and policy guidelines. We employ an efficient and faithful evaluation process that compares the database state at the end of conversation with the annotated goal state. We also propose a new metric (pass"k) kk to evaluate the reliability of agent behavior over multiple tri- als. Our experiments show that even kate-of-the-art function calling agents (like gpt-4o) succeed on < 50% the tasks, and are quite inconsistent (pass"8 < 25% in retail). Our findings point to the need for methods that can improve the ability of agents to act consistently and follow rules reliably.
INTRODUCTION
There is increasing excitement around the potential of language agents (Sumers al. 2023 Yao jtt al. 2022 Schick et al.] 2023 Ahn et al. 2022) to enable new levels of automation across various industries. However, their deployment in real-world systems requires several key desiderata to be satisfied. Agents must (1) interact seamlessly with both humans and programmatic APIs over long horizons incrementally gather information and resolve intents, (2) accurately adhere to complex policies and rules specific to task or domain, and (3) maintain consistency and reliability at scale, across millions of interactions. For instance, consider the case of an airline booking agent (Figure [T). When user wants to change their flight reservation to a different destination airport, the agent needs to gather the required information by interacting with the user, check the airline policies using the guidelines provided, and find new flights and (if possible) rebook the user using complex reservation APIs. In addition, the agent should be consistent in its behavior across different kinds of users with the same request, and robust to small changes in the conversation flow that should not affect the end outcome.
Modeling realistic human interaction and rule following in agent evaluation is vital for developing and deploying trustworthy agents in the wild, and tackling challenges like long-contex. reasoning and planning methodical fashion. Existing benchmarks (Yao al. preprint: Zhou et al. 2023
Jimenez et al. [003) Liu et al.) 2203) Ruan et al.] (203) for language agents often feature simplified instruction-folle wowlowwing setups, where the agent autonomously interacts with an environment (web, code terminal, or APIs) given all the information upfront, without any human-in -n-tth--oooop interaction and without the need consult any Nomainn-specii guidelines.
In this work, we introduce 7-bench (short for Tool-Agent- -sser- Interaction Benchmark) to measure an agent's ability interact with (simulated) human users and programmatic APIs while following domain-specifee policies in a consistent manner. 7-bench is built in modular framework with (1) realistic databases and APIs, (2) domain-specifie policy documents, and (3) instructions for diverse user scenarios and corresponding ground truth annotations. As first demonstration, we focus on the realm of customer service and create two different domains where agents need to assist
Work done during internship. Code and data:
https: http::://gittbbbcoom wttp:/.ccc//sieiieeeee researcchh
tau-bench

---

Published as a conference paper at ICLR 2025
(a) r-bench setup
(b) Example trajectory in r-airline
Change flight
gel wuserreeer_e ddetails
book reservation
get reservation dctnils[]K911]]
Tools
cancel_ reservation
updale revervation Nighis
l'cabin:: 'basic economy'.
'created '20240514 1800'
(Read databuse)
Domain polley as system prompt
Current time is 2024-5-15 15:00:00 EST. Basic economy cannot modified.
JK9019 is basic ecunomyy and cannot changed. since within 24h, I can cancel it and book new one. you want do
Basic ceonomy cannot be cancelled after 24 hours booking.... ... (more nules omitted)
Agent
Ob... In that case just cancel it
User instruction system prompr You min li 2017. and want change the your most recent reservation to fly instead of on the same day, change not possible, you want the agent to cancel and rebook... ... You are concise.
cancel reservation[]K99093
User
fr... 'status": "cancelled")
)
(Write database)
Figure I: (a) In r-bench, an agent interacts with database API tools and an LM-simulated user to complete tasks. The benchmark tests agent's ability to collate and convey all required information from/to users through multiple interactions, and solve complex issues on the fly while ensuring follows guidelines laid out in domain-specifie policy document. (b) An example trajectory in т- airline, where an agent needs to reject the user request (change a basic economy flight) following domain policies and propose a new solution (cancel and rebook). This challenges the agent long- context zero-shot reasoning over complex databases, rules, and user intents.
simulated users with diverse requests (7-retail and T-airline). We leverage the generative capabilities of language models (LMs) for data creation and realistic human user simulation (Park et al. 2023) in conjunction with manual annotation and verification.
We constructed 7-bench in three stages, including manual schema and API design, LM-assisted generation of data entries, and manual scenario generation and verification for the user simulator. Our evaluation scheme compares the database state at the end of each episode with the ground truth expected state. This allows for objective measurement of the agent's 's decision making, while also providing room for stochastic variation in the conversation itself, since the user may pose the same request in different ways that result in the same end state of the database. We also introduce the metric of pass"k, which measures the consistency and robustness of the agent across i.i.d. trials.
Our experiments reveal that agents built with simple LM constructs (like function calling or ReAct) perform poorly, highlighting the need for more sophisticated agent architectures. For instance, even state-of-the-art LMs like gpt- 4o achieve low task success rates (pass"1) using function calling (~611% % on r-retail and ~355% on r-airline). With increasing k the chance of consistently solving task drops rapidly, to as low 25% for pass"8 '8 r-retail for the same model. This showcases the fragile nature such agents in handling stochasticity and partial information, which is common in human-agent interaction. Upon analyzing the failure cases, find that current agents struggle with complex reasoning over databases, understanding and following ad-hoe policies, and handling compound (more than one) requests. We hope that 7-bench enables the evaluation and development of more consistent and capable agents for real-world digital tasks involving human interaction.
RELATED WORK
Most existing benchmarks for agents and task-oriented dialogue systems focus on evaluating either conversational tool-use capabilities. 7-bench aims to unify both under realistic settings, while also testing how well agents can follow domain-speciff policies in a consistent manner.
Benchmarks for language agents and tool use. Several benchmarks have been developed to eval- uate agents powered by LMs (Yao et al. preprint: Zhou et al. 2023 Jimenez et al. 2023 Liu et al.] evaluating tool use capa-
2023 Ruan et al., 2023) Recent efforts have focused specifically
bilities of LMs, Le., their ability to generate the right function calls from a set of functions in an
API. Projects like the Berkeley Function Calling Leaderboard (BFCL) (Yan al. 2024). Tool-
2

---

Published as a conference paper at ICLR 2025
Bench (Xu et al. 2023) and MetaTool (Huang et al. 2023) test tool use/function calling in multiple programming languages and propose various methods for evaluating the accuracy of function calls. ToolEmu (Ruan al.) 2023) uses language models themselves emulate the execution of tools, with a focus on exposing potential safety risks when LM agents fail to use tools correctly. However, all these works only contain single-step user interaction, where the human interacting with the agent provides an initial instruction containing all the required information. In contrast, our bench- mark focuses on more realistic setting where the agent has interact with human users to gather information and authorization.
Task-oriented dialogue. Task-oriented dialogue has been a long-standing challenge for NLP, with several efforts over the years to build domain- -ssmciii offline datasets or user simulators. The former types of benchmarks are static and only test the conversational agent on pre-collected conversation trajectories et al., 2021a Budziaaowssii et al. 2018 Andreas et al. 2020). The latter either rely on user simulators that are rule-based or rely on symbolic specifications (Schatzmann et al.) 2007 Gür et al. 2018) or perform tests with real humans through crowdsourcing platforms (He ct al. 2018). Some very recent work explores the use LMs as response raters to train dialogue systems (Hu el. 2023) evaluates their capability for simulating users (eun Yoon et al. 2024). т- bench leverages the powerful text generation capabilities of state-of-te of-thhe-rrt LMs to simulate realistic user utterances and long-context conversations using textual scenario descriptions, with the goal of evaluating agents. The stochastic sampling from LMS allows for diverse yet faithful variations in the dialogue when re-run with the exact scenario this extremely useful for testing agent consistency, as we show in $ |5.1
User simulation with LMs. Our work is also related recent efforts on using LMs simulators of human characters. This includes papers on simulating non-player characters (NPCs) in text adven- ture games (Kim et al.] 2022) multiple agents in human-liks like societies (Park et al.] 2023) or specific collaborative tasks ( (Wu et al. 2023). and enabling human in-the-loop interaction for tasks like on- line shopping (Chen et al. (024) or web search (Zhang et al. 2024). However, all these past works have not used such simulators to benchmark the reliability of agents, instead focusing on showcas- ing the ability of LMs to enable realistic simulations. Our work uses such realistic user simulation to provide an accurate assessment of the reliability and robustness of AI agents for deployment in systems that undertake millions of real-world interactions with humans.
T-BENCH: A BENCHMARK FOR TOOL-AGENT-USSH INTERACTION
Each individual task in 7-bench can be formulated partially observable Markov decision process (POMDP) AS,A, T, with state space S, action space A,, observation space O, transition function T : S A - S x O, reward function R : S > [0, 1]. and instruction space U. The agent interacts with both (1) databases (db) via API tools, and (2) a (simulated) user (user) to complete task, i.e., S = S.b @ Suser. A Aas U Auser. O Oa Ouser. In addition, the agent is provided a domain-specific policy document containing rules it must adhere to one can think of this as partially describing the world model of the domain. We describe each component in more detail below.
Databases and APIs Each benehhh domain has several databases and associated APIs. The contents of the database form the state Sab (Figure Eah, which is hidden from the agent and the user, and can only be read from or written to using API actions adb, which are usually in the form tool_name (**kwargs). When an action is executed on the database, the transition Tas : (Bab, a.a)) + (s'an> am)) is deterministic and implemented as a Python function (Figure 2bj.
Domain policy. Fach domain has a policy (Figure 2c that explains the domain databases, task procedures, and restrictions for the agent to follow its interactions. Some restrictions are imple- mented as checks in the API, c.g., using a payment ID not in the user profile will lead to osb "Error: payment not found", and others not, e.g., the airline policy states different bag- gage allowances for different membership statuses and cabin classes, but the agent needs to fill in the number of baggage items to be paid for in the book given woaldworld agents.
API, similar to the freedom
User simulation. We use a language model (gpt-45 1ort--406111 4-0613) to simulate a human user interacting with the agent. The user state Suser consists of initial system prompt with the task instruction
3

---

Published as a conference paper at ICLR 2025
("order__ ; " "#W2890441",
Return delivered order
"user id": "mei davis davis__935",
After user confirmation the order status will be changed to 'return requested'..
"items": [f
"name": "Water Bottle", "product_id":
"8310926033*, "item_id" id"; "2366567022", "price": 54.04,
Exchange delivered order An order can only be exchanged if its status is 'delivered'...
"options":
"capacity": "1000m1", "material"; "stainless steel",
(c) Domain policy excerpts in 7-retail.
"You are Mei Davis in 80217. You want return the water bettle, and exchange the pet bed and office chair to the cheapest version. Mention the two things together. If you only do one of the two things, you prefer do whatever saves you money, but you want to know the money you can save in poth ways. You are in debt and sad today, but very brief.", "actions":
{ ("instruction":
"color": "blue"
}), ..],, ...)
(a) An orders database entry r-retail.
def ferbur_ddlliere___eo -ite
order_ids str, item ids: dist/stri pemnent_etthddteoo id: str, str:
"name":
def exchange_delivereeee lerreeddeiveeee__reeeee items
"return_dell
order_id: str, stem_ids List [str), new_item_ide ids: List [str], payment_method.
"arguments";
"order id" "#12890441",
"item ids": ("2366567022") "payment
str,
"payment_method___ method id":
str:
"credit_card_10611
|H "outputs"; 1"54.04", "41.64")
(b) An API tool in 7-retail.
(d) User instruction ensures only one possible outcome.
Figure 2: r-bench is constructed in modular fashion with several components: (a) JSON databases, (b) Python API tools, (c) Markdown domain policies, and (d) ISON task instances. The agent can only access API tools and domain policies, and indirectly access databases via API tools. Task annotation is not visible to the agent and is used only for user simulation and evaluation.
(Figure 2dh along with the entire conversation history between the user and the agent so far. The user cannot see the interaction history between the agent and API tools. The agent can interact with the user using any natural language message, e.g., Quser can be "Your reservation has been updated, is there anything else I can help with?". The transition Tuser (Suser: (user) + (suserr Ouser) is stochastic and attaches the agent's message to the chat history followed by sampling a new user message from the LM, e.g., Ouser can then be "Yes, I also want to cance cancel another flight." When the user issues = ##* the episode finishes and the agent is evaluated.
Task instances. As shown in Figure [d] each 7-bench task instance has two parts: an instruction for the user simulation (hidden from agents), and an annotation of the ground truth database write actions (and optionally, ground truth outputs for user questions). The instruction sets up user identity, intent, and preferences in a way that guarantees only one possible outcome under the domain policy. Each task episode consists of the simulated user starting with request, which the agent handles in conversational manner while being able to call tools at any point and refer to the provided policy. Once the episode ends, the datahase state and agent-to-user messages are used to compute the reward.
Reward. The reward of a task episode r Taction X Toupuut {0,1) based on (1) whether the final database is identical to the unique ground truth outcome database (Faction), and (2) whether the agent's responses to the user contain all necessary infor- be
mation (Toupar). So for the task of Figure 2d the agent-user dialogue ied and the agent can call various (read) actions, but the agent is successful if the only database write action is return del vered order items (order id="#w28904
4

---

Published as a conference paper at ICLR 2025
and the user responses contain 04". "41. 64" as substrings. Note that r = might be a nec- essary but not sufficient condition for successful episode e.g., the agent might issue the return without explicit user confirmation, which violates the policy. Nevertheless, our proposed rule-based reward is fast to compute and faithful, and already poses significant challenges for current models and methods as we show in $
iteemidss (M2.l6===26666702222"2
Pass"k metric. For tasks like code generation with good verification techniques (unit tests), the community has defined the pass@k k (pass at k) metric as the chance that at least one out of i.i.d. task trials is successful, which captures the trend of agents enabling discovery of solutions with scaling of inference-tinn compute (Chen et al. 2021b). For real-world agent tasks requiring reliability and consistency like customer service, we propose a new metric pass"k (pass hat k), defined as the chance that all k i.i.d. task trials are successful, averaged across tasks. Therefore, if a task is run for trials and e of those trials end up successful (r 1), unbiased estimates for pass"k and pass @k would be:
11 k: (()))
( (/))
pass"k Erask
pass@k 1ask
Eass
k
In our case, for the same task, the user prompt and database transitions are the same, with just LM sampling of the user and agent messages generating sufficient stochasticity. Thus, pass"k can capture the reliability of the agent at handling variations in conversations with the same underlying semantics while adhering to the domain policies and rules. By default, report the average reward across tasks, pass"1=pas 1=pass @ ]=E[r) = E[e/n], as the main metric for comparing agents.
4 BENCHMARK CONSTRUCTION
r-bench defines domain-agnoste environment and user simulation classes shared by various do- mains, and domain-specific data in terms of database JSON files, database API Python code and documentation, domain policy text, and task instances. Each domain is created in three-stage approach with a mix of LM and code runs, and human labeling and checking.
Stage I: Manual design of database schema, APIs, and policies. We start by co-designing the simplest possible database schemas, APIs, and policies with inspiration (and simplification) from their real-wor world counterparts. Simplicity is important for the logical consistency of various compo- nents and the ease API and task annotation. Still, a minimally realistic domain requires at least tens of schemas, APIs, rules, and turns be challenging enough for existing agents. See $ B. for more.
Stage II: Automatic data generation with LMs. Once the data schema is set up, create an example entry and use gpt-4 to generate a systematic code snippet to sample scalable entries, and manually polish minor bugs in the code. See B.2 for example snippet and more details.
Stage III: Manual task annotation and validation with agent runs. Here, the key challenge is ensure the user instruction leads to unique database outcome, For example, the preferred payment method is not specified, the user might answer differently and cause the final database to be different across trials. So we write an initial user instruction, run trial with gpt-4-turbo function calling agent, polish the user instruction by examining the trajectory, and do this iteratively until we are certain no ambiguities exist (see Figure 7in in A where we run each 7-retail task with 40 gpt -tubb trials and check all tasks with zero or low success rates). We can copy and edit agent actions and outputs for ground truth annotation, which is casier than annotating from scratch.
In practice, we might update minor details database schemas or policies during data or task creation, but the three stages mostly linear, and the constructed data is organized in a modular structure.
4.1 DOMAINS
Using the above procedures, we modularly construct two domains, 7-retail and r-airline. We choose these two domains as they relatively easy synthesize data (e.g., products, prices, flights) and
5

---

Published as a conference paper at ICLR 2025
r-retail
7 r-ailinee
Databases API tools
500 users, 50 products, 1,000 orders
500 users, 300 flights, 2,000 reservations
7 write, 8 non-write
6 write, 7 non-write
Tasks
115
50
Table I: Key statisties from 7-retail and r-airline.
craft policies (e.g., product return, baggage allowance) based on common sense, allow for diverse tasks, and are close to real-world applications. For more capable agents in the future, more advanced domains (e.g., medical, tax, or legal) with more complex data and rules can be studied. Below, we briefly describe the domain policies of two domains (full details of the domains in B.I.
r-retail. In this domain, the agent is tasked with helping users cancel or modify pending orders, return exchange delivered orders, modify (e.g., "Water Bottle" in Figure 2a) has various item options with unique IDs (e.g.. 1000ml, stainless steel, blue). Each pending order can only be canceled or modified once, and each delivered order can only be returned or exchanged once. An item cannot be modified or exchanged for another product type. These constraints simplify task and API design, and challenge agents to follow domain-specifer specific rules, and inform and collect complete information from users before taking actions.
addresses, provide information. Each product
r-airline. Here, the agent has to help users book, modify, or cancel flight reservations, or provide refunds. We construct 300 flights between 20 US cities with realistic durations and prices, and API tools to query direct or one-stop flights. The domain policy is more complex than r-retail, with ad-hoc constraints about combining payment methods, checked bag allowance, flight changes and cancellations, etc. These constraints can also over membership tier and cabin class specific, creating challenging multi-hop reasoning puzzles for the agent.
4.2 KEY CHARACTERISTICS
Realistic dialogue and tool use. Compared to prior task-oriented dialogue benchmarks, r-bench has more complex databases and realistic user simulations thanks to the advances of LMs. Some trajectories can be seen in $ C.2 and $ $D2.2] Notably, even if the user instruction is synthetic, the user utterances generated via LMs are open-ended and natural-sounding.
Open-ended and diverse tasks. Each 7-bench domain's 's data schemas, APIs, and rules are simpli- fied compared to real-world domains, but they are rich enough to support the creation of extremely diverse, open-ended, and sometimes creative tasks (see A
D.2). Importantly, we trade off quantity for quality as $ shows, running small set of high-quality tasks for multiple trials (with pass"k metric) can reliably reveal rich insights into different models, methods, and research challenges.
Faithful rule-based evaluation. Real-world agents are hard evaluate as the trajectory can be ex- tremely diverse for the same task, and success criteria are multi-faceted As result, it often requires human evaluation, e.g., end users to judge task resolution and domain experts to judge rule follow- ing. In r-bench, we trade off slow, careful task annotation for fast, faithful evaluation. By ensuring that only one database outcome is possible based on domain policies and user desires, subjective and noisy human judgments can be replaced by simple and objective database state comparisons.
Modular extension. The codebase structure of 7-bench is modular, and it is easy to add new domains to 7-bench, add or update database entries, domain functionalities, rules, APIs, tasks, and evaluation metrics (given they are consistent with the existing domain data). We release our codebase publicly encourage the community to create tasks and domains for -heench.
EXPERIMENTS
Models. We test various otate-of-the-art proprietary and open language models for agents through OpenAl GPT API (gpt-40, opt-4-turbo, 9pt-4-32k, 5-t-..bbo.rr Anthropic Claude API (claude-3-opus claude-3-sonnet, claude-3 3-abee---hikku)
their APIs:
Google Gemini API (gemini-1) Dremini-1.5-pro-11 -rpoolattst. gemini -innint-..--lfss---eee

---

Published as a conference paper at ICLR 2025
Model
retail
airline
avg
60
gpt-4c
61.2
35.2
48.2
ReAct
40
gpt-4-turbo
57.7
32.4
45.1
gpt-4-32k
56.5
33.0
44.8
3pt-3.5-turbo
20.0
10.8
15.4
(laudc-3-cpus
44.2
34.7
39.5
gpt-4o
gpt-4-turbo
gpt-4-32k
gpt-3.5-turbo
Srsudc-3-sonme
26.3
27.6
27.0
claude-3-haike
19.0
14.4
16.7
Figure 3: pass' across models/methods in r-retail.
gemini-1. 5-pro
21.7
14.0
17.9
gemini-1.5-1
17.4
26.0
21.7
mistral-1aree
30.7
22.4
26.6
80
gpt-4o
mixtral-8x22b
17.7
31.6
24.7
60
get-4-aurbo
gpt-4-32k-0613
meta-llama-3-
14.8
14.4
14.6
40
claude 3 apus
gpt-3 3.5--oubb
Pass"l across models via function calling, except Llama-3 via text-ReAct. Aver- age is weighted by domains, not by tasks.
Table 2:
12
18
32
k-ttriias
Figure 4: pass"k (-) and pass (..) in 7-retail.
Mistral API
(mistral-large
open-mixtral-6x222
AnyScale API
(meta-llame 1at-mma--70--insttte 70---innnstrut)) Only the last two models openly release weights. We do not test small models (7/13B) due to the difficulty of the benchmark.
Methods. Our main method for building the agent through the use of function calling (FC), which natively supported by all tested LMs except Llama-3. In FC mode, the model's system prompt is set to be the domain policy, and at each turn, the model autonomously decides to generate a user re- sponse message or tool call. We also text-formtteed ReAct (Yao al. 2023b) and its Act-only ablation, where the model is instructed to zero-shot generate "Thought: [some reasoning] } Action: (some JSON format action argument]" not suitable for a user-in- in-the-loop setup, c.g., self-reflection (Shinn et al. (003)) is unrealistic as real- world agents only have one chance serve the user, and planning approaches (Yao et al. 2023a) might be too slow to help user in real time.
only the action part. Notably, some agent methods are
We limit each task most 30 agent actions (either tool calls or user responses). For main results (Table 2). we run at least 3 trials per task. The LM temperature is 0.0 for agent and 1.0 for user.
5.1
MAIN RESULTS
Model comparison. From Table 2 we see that gpt-4o is the best model with function calling, and there is wide spectrum of performances among various models. Notably, SoTA open-weight models (llama-3-706 and mistral-8x22b) still have significant gap to cover with respect to SoTA proprictary models (gpt-4o, claude-3-opus). All models are still far from solving 7-bench, especially the more challenging 7-airlinewhere even gpt-4o solves only 35.2% of the tasks. The diversity of model performances (shown Table [ and task difficulties (shown in Figure A as well as large remaining gaps from perfect resolution makes bebech ideal for benchmarking and developing new models for agents, tool use, and dialogue.
Method comparison. Figure 3 shows that natively supported function calling consistently out- performs text-formatted agent methods with the state-of-the-art models. For text-formatted agent methods, adding reasoning traces still consistently helps (compare ReAct vs. Act columns) as it helps bridge the gap between observations and actions that have unfamiliar formats. We have also experimented with adding "think" function for function-calling agents, but it did not boost perfor- mance, perhaps because most FC models have not been trained toward such reasoning.
Agent consistency via ".. As shown in Figure 4 the chance of reliably and consistently solving the same task multiple times significantly drops as the number of trials l increases. Even for the best-performing gpt-4o function calling agent which has > 60% average task success, pass" drops

---

Published as a conference paper at ICLR 2025
to < 25%. In real-world scenarios, it is important and challenging not just to build agents with high average success (pass" 1), but with more robustness and consistency (pass k trend).
Cost analysis. When pair gpt 4o FC agent with gpt user simulation r-retail, the agent / user simulation costs are $0.38 / $0.23 per task respectively, so running one trial per task costs around 200 dollars. For the agent, the input prompt completion output take up 95.9% 4.1%% of the price respectively, so the cost is mainly due to long system prompt (domain policy + function definitions).
5.2 RESEARCH CHALLENGE ANALYSIS
In this subsection, we analyze in both quantitative and qualitative terms the challenges of r-bench, with focus on the 7-retail split and the most advanced baseline: gpt- 4o function calling agent.
Failure breakdown. We sample I 11 gpt-4o FC agent tra- jectories in r-retail trial per task), out of which 40 tasks Upon manual examination
Wrnnng
have failed of these failures, of them are caused by user instruction typo or ambiguity (and then fixed), and the remaining 36 failure cases are agent issues, which are broken down into more detail below and in Figure
Whong Info
38.3%
19.4%
26.0%
Failure Wrong argument or information provided: the challenge of complex database reasoning. For "wrong argument", gpt-4o FC agent usually makes the argu-
Partialy resole
Wronn decision
right type of tool call(s) but fills in one ments incorrectly. In the example shown in CC2.2] the user wants exchange lamp for less bright one and prefers an AC adapter over battery or USB power source. The agent fails reason over the complex inventory of lamps and find the unique option given such a preference. Weaker models and methods struggle with even more basic failures such as hallucinat- ing arguments for example, while gpt- FC agent only makes 0.46 tool calls with non-ex exi-txtstentt user/produuctorddr/iien IDs per r-retail task, gpt-3. 5-turbo FC / Act agents make 2.08 6.34, respectively.
Breakdown of 36 failed gpt--4o FC agent trajectories т- retail.
Figure 5:
For "wrong info", agents omit user-required information (e.g., the user asks for a tracking II) but the agent does not provide it), or calculate the wrong information (e.g.. wrong total price), or provide the user with incorrect information that causes the user request to diverge (e.g., the user might cancel or exchange based on incorrect price information provided by the agent). These failures account for 55% of overall failures and highlight the need for improved common sense and numerical reasoning over complex databases and user intents for future models.
Incorrect decision decision-making: the challenge of domain understanding and rule fol- lowing, While the above failures can be recognized even without referring to the domain policy, "wrong decision-making the domain-specifie knowledge or rules and makes the wrong type of tool call. In the example of |C.22.] the user wants exchange "a couple of items", and according to the domain policy, "Ex- change or modify order tools can only be called once. Be sure that all items to be exchanged are collected into list before making the tool call". However, the gpt-4o FC agent omits the domain knowledge and rule and decides to exchange one item first, resulting in the second item not being exchanged.
Failure 2:
failures (25% of overall failures) occur the agent fails to understand
To further understand how different agents follow rules in different domains, we perform an ablation study by removing the domain policy from the FC agent system prompt. As seen Table 3 in 7-retail where rules are simpler and closer to commonsense, gpt-4o 4o and gpt-3. 5-turbo agents only degrade 4.4% and 5.5% in terms of pass"I, 1, suggesting that their successful cases mostly stem from using tools in an intuitive and common sense way, and that they may not actually be leveraging
r-retail
T-airline
gpt 4c gpt-3.5
61.2 + 56.8
33.2 - 10.8
20.0 14.5
10.8 9.6
Table 3: pass'1 scores degrade when the domain policy is not provided in the agent's system prompt.
8

---

Published as a conference paper at ICLR 2025
User agent strategy
Acc
Errors caused by the agent
Errors caused by the user
Other
Ilm react
0.367
0,980
0.020
0.000
0.300
0.880
0.020
0.100
verify
0.393
0.960
0.040
0.000
reflection
0,406
0.920
0.040
0.040
Table 4: Analysis performed on trajectories from 3 trials on the Airline domain with gpt-4o function- calling as the model and agent strategy. Acc reported as the average accuracy over 3 trials. Agent, user, and other errors are reported as proportions of 50 randomly sampled error trajectories.
the policy documents to the extent possible. In raiilinee where rules are more complex and ad-hoc (e.g., baggage allowance varies for different membership tiers and cabins), removing the policy hurts gpt-4o 4o significantly (-22.4%, but 3pt-3.5 5ururb only slightly (-1.2%), suggesting the for- mer follows rules at times but the latter does not has the capacity process complex airline rules. Overall, r-bench poses significant challenges for function calling agents to follow complex domain dynamics and rules, and showcases there is still work to be done in this direction. Domain-specifii fine-tuning or agent code scaffolding might provide some remedy, which can be important future work.
Failure 3: Partial resolution of compound requests. Lastly, shown in Figure 6 when a task involves many re- quests (represented by the number of ground truth write actions databases), becomes more challenging for function calling agents (19% of cases). Sometimes the agent omits explicit user requests at the beginning of the conservation, hinting at the need for better long-context and memory capabilities, Other times, the agent omits implicit actions, such as in $ C.2.3] where the user wants to fix wrong addresses in all orders, but the agent stops after checking only one order. Agents need to improve in their consistency and systematicity in handling such cases.
gpt-4-turbo gpt-3.5-tune 5--tubo
25
Number API
Figure 6: Retail tasks with more database writes are harder.
5.3 USER SIMULATION METHODS
One challenge of simulating users with a language model is that it could cause occasional halluci- nations/misiirections that may cause episodes fail even if the agent not at fault. We explored alternative user simulation strategies to exhibit better eestructionfoolowiie cent advancements in agent development. We compared the vanilla strategy described above (Ilm)
ability, drawing on re-
to three other strategies: (1) react: write reasoning trace before responding (Yao et al. 2022). (2)
verify: generate response proposals until an LLM verifier returns true, and (3) reflection: iteratively refine response until the LLM verifier returns true (Shinn et al., 2023 Madaan et al.) (023). To
measure the impact of hallucinations and other mistakes caused by each user strategy, we manually evaluate 50 randomly sampled failed trajectories from trials over the entire Airline domain.
From Table we find that all user simulation strategies perform relatively similarly, with reflection performing the best. Further, find that less than 4% of errors are caused by the user simulator across all user strategics.
DISCUSSION
We have presented r-bench, a novel benchmark for evaluating the reliability of agents interacting with humans and tools in dynamic and realistic settings. The benchmark leverages the latest ad- vances in LMs to simulate users, allows for automated testing of agents and provides an assessment of agent's ability to follow domain-specifie rules in consistent manner. Our results show that even SOTA LMs are far from being reliable for use in real-world settings.
Directions for improvement. While 7-bench is step towards dynamic evaluation of agents in real-world scenarios, there are several directions for improvement. The simulated user can have some limitations: (1) the user instruction might contain typos or ambiguities, which annotators can

---

Published as a conference paper at ICLR 2025
examine and fix; (2) the user instruction may not contain all domain knowledge, e.g.. in C.2.1 the user authorizes the single item exchange without knowing that the agent could only issue one exchange action, which reflects real-world users who (rightfully) do not know complex domain policies; or (3) the user simulation I.M might have limited capacity at reasoning, calculation, long- context memorization, alignment with the instruction prompt, e.g., in $ C.2.2 the user authorizes the reeom-rrcomeedded lamp without double checking its features. While these can all be improved in future work, one can also argue that this is indicative of the real world where users can have wide range of skill sets and knowledge, and the onus is on the agents to handle diverse users.
In addition, one can also add more systematic checks the simulator to ensure unique outcomes. The domain policies can also be made more complex to match real-world scenarios. More evaluation metries can added to define agent success (e.g., LMM checks that certain rules are followed). The manual annotation process for the benchmark is difficult and requires deep understanding of both the domain and agent capabilities. There is also some element of implicit bias during the task curation process since we use the gpt 1ere---turie turbo FC agent tune the user's system prompt. Future work can investigate alternative ways of using LMs for improving data curation and user simulation. Finally, while don't 't believe this work has potential negative societal implications directly, it helps real-world agents which can have various consequences for the economy and society the future.
Challenges for agents. At the core, the main results from our experiments demonstrate a critical fact: agents built on top of LM function calling lack sufficient consistency and rule full-following ability to reliably build real-world applications. Solving both of these problems can have outsized impact on automating several real-world tasks and ensuring smoother human-in-the-loww interaction. Other specific features to improve in agents include long-horizon information tracking and memory, as well the ability to focus the right pieces of information in context for the decision at hand, especially when there may be conflicting facts present.
ACKNOWLEDGEMENTS
We thank Clay Bavor, Honghua Dong, Victor Barres, Dongxu Zhou, and Yangjun Ruan for feedback on earlier drafts of the paper, and Nate White for helping set up the different LLM APIs for the experiments.
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, al. Do can, not I say: Grounding language in robotic affordances. arXiv preprint arXiv:2004.01691, 2022. URL
https:://arxiv> org/ abs 2204 01691
Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, et al. Task-oriented dialogue dataflow synthesis. Transactions of the Association for Computational Linguistics, 8:556-571, 2020.
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task- oriented dialogue modelling. arXiv preprint 1eXivv1810..00 2018.
Derek Chen, Howard Chen, Yi Yang, Alex Lin, and Zhou Yu. Action-based conversations dataset: A corpus for building more in-depth task-oriented dialogue systems. arXiv preprint arXiv:2104.00783, 2021a.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis. Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec
10

---

Published as a conference paper at ICLR 2025
Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Me- Grew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code, 2021b.
Sanxing Chen, Sam Wiseman, and Bhuwan Dhingra. Chatshop: Interactive information seeking with language agents. arXiv preprint arXiv:2404.09911, 2024.
Se eun Yoon, Zhankui He, Jessica Maria Echterhoff, and Julian McAuley, Evaluating large language models as generative user simulators for conversational recommendation, 2024.
Izzeddin Gür, Dilek Hakkani-Türr Gokhan Tür, and Pararth Shah. User modeling for task oriented dialogues. In 2018 IEEE Spoken Language Technology Workshop (SLT), pp. 900-906, 2018. doi: 10.1109/SLT.20188 2.1.86666.......
He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. Decoupling strategy and generation in negotiation dialogues. arXiv preprint orXiv:18088 09637, 2018.
Zhiyuan Hu, Yue Feng, Anh Tuan Luu, Bryan Hooi, and Aldo Lipani. Unlocking the potential of user feedback: Leveraging large language model as user simulators to enhance dialogue sys- tem. In Proceedings of the 32nd ACM International Conference Information and Knowl- edge Management, CIKM '23. ACM, October 2023. doi: 10. 10.114553333780036552200 URL
http::/dx..e doi orgg 10.
Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, et al. Metatool benchmark for large language models: Deciding whether to use tools and which to use. arXiv preprint arXiv:23 10.00 /28, 2023.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao. Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 2023.
Minsoo Kim, Yconjoon Jung, Dohycon Lee, and Seung-won Hwang. Plm-based world models for text-based games. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 1324-134,, 2022.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.003688 03688, 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Sean Welleck, Bodhisattwa Prasad Ma- jumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback, 2023.
Joon Sung Park, Joseph C O' O'Brien, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442, 2023.
Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J Maddison, and Tatsunori Hashimoto. Identifying the risks of Im agents with an Im-emulated sandbox. arXiv preprint arXiv:2309. 15817, 2023.
Jost Schatzmann, Daniel Jurafsky, Michael Galley, and David Trevillian. Evaluating agenda-based simulation for reinforcement learning of dialogue management. In Speech Communication, volume 47, pp. 95-121, 2007.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arX 2023.
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and seff-refiection, 2023.
11

---

Published as a conference paper at ICLR 2025
Theodore R Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L Griffiths. Cognitive archi- tectures for language agents. arXiv preprint arXiv:2309.02427, 2023.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen Ilm applications via multi- agent conversation framework. arXiv preprint arXiv:2308.08 08/55, 2023.
Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. On the tool manipulation capability of open-source large language models, 2023.
Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, lon Stoica, and Joseph E. Gonzalez. Berkeley function calling leaderboard. lttpps://ooilla.c Fwnctionnccaaatttcttooeaooaooaae cal wellliingg ieiiinnlaaaae hew__laadrbboradhht...
berkel ley edu /b//bbll/// wlw.bloosssao 8 berke seeerreely_uucctiieeeee ionnttioon.o 2024.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. arXiv preprint arXlv:2210.03629, 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models, 2023b.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real- world web interaction with grounded language agents. ArXiv, volume 35, pp. 20744-20757, preprint.
Erhan Zhang, Xingzhu Wang, Peiyuan Gong, Yankai Lin, and Jiaxin Mao. Usimagent: Large language models for simulating search users. arXiv preprint arXiv:2403.09142, 2024.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. WebArena: A Realistic Web Environment for Build- ing Autonomous Agents. arXiv preprint urXiv:2307.13885 2023.
A ADDITIONAL RESULTS
As shown in Figure T bench tasks have a well-balance and diverse spectrum of difficulties. We use such a plot find tasks with zero success, and examine the task annotations in targeted way.
Figure 7: The success rate of each r-retail task, sorted by qpt-4-turbo success rate, Each task has at least 40 gpt- 4eet-4-turbo trials to ensure reliable per-task success rates.
B BENCHMARK CONSTRUCTION
STAGE I: DESIGN OF DATABASE SCHEMAS, APIS, AND POLICIES
The data schema examples can be in C.1 and D.1
12

---

Published as a conference paper at ICLR 2025
7-retail
r-airline
Databases
users, products, orders
users, flights, reservations
Read APIs
iund_user_id wi..ddddbbeealll find user id by name zip list all all.prrouct.t types get_order sed_errrdeeaiilss details
get.reservation _eetals
get get.user details
list all airports
search dirccchdrecttttie _light
ppppduuctdetattl get_user_details
search www.ooonetttpiiie flight
Write APIs
cancel pending order exchange delivered order items moodify-pending aaddes_sddres modify-pent seey--eeenddddddeeee codggorreeem
book reservation cancel reservation send_certificate update_reservaaton besgggationnaagaages baggages update reservation flights update reservation passengers
modify pending order paymment modify user www.uuueeedeeeee address
detum_delivered_ordl corderddddddrrredreeeeeee _irmeritems
Non-DB APIS
calculate, transfer_to htw.:rtoohmmaaaaaeee
Policies
See B.1
See B.1
Table 5: Overview of r-retail and 7-airline databases and APIs.
API design example Here is the Python implementation of an API in 7-retail.
import json
from typing import Any, Dict, List
def dechange_delivveedd__
data: Dict [str, Any], order_id: str, item item_ids: List|str), new new_item_idss ids: List [strl, payment_methood_id str, str: products, orders, users data ("rodductt data["orders] datal [usar"us
check order exists and is delivered
if order_id not in orders:
return "Error; order not found"
order arderslorder_is
order (edder""tttus") != "delivered";
return "Error: non-delivered order cannot be exchanged"
check the items be exchanged exist
all item ids liteml"item. l for item order "items"
for item id in item_ids: ids:
if item_ids. count (item_id) > all_item count (item_id): return f"Error: (item_id) not found"
check new items exist and match old items and are available if Leneitem_iee != len (ne__itemiide ids): return "Error: the number of items to be exchanged should match"
diff-price
for item_id, new itemitemid in zip (ittemmmddd new hew_item_ids)
item = litem for item in order ("items") if iten[ item("itent ("item_id) == item_id][0] [0]
product_id item ["produc_iid]]
13

---

Published as a conference paper at ICLR 2025
if not new_item_id in products(produeet ("arlattt___"vaaaasttttta
and products [product_id] ("variants") [new [new_iter_id] ["availabie")
f"Error: item (new_item_id) found 4 available"
old price iteml "price"]
new pei_price
products (prodctttt ("variants"] [new_item_ion ["price") diff_price +- new_price cld_price
diff_price (ound 2)
check payment method exists and can cover the price difference gift card
if payment_method_iie not in uscrs (uusr_idd)]] "rppymment_methods""
return "Error: payment methed not found"
payment_methan
users| "user_ j I sttyymen__mehhdsse
[payment__mehood.ae
if payment_methle "source"| == "gift_eard" _crd"" and payment
diff price:
return "Error: insufficient gift card balance to pay for the price difference
modify the order
order("status") "exchange requested" order ("ex="""eeen___ieee""" = sorted
order (""xchange_now_ order order
= serted (nex__itnnin payment_method id diff_price
return json.dumps (order)
exchange_ delivered order items...... info =
"type": "function",
"functicn":
"name";
"description"; "Exchange items in delivered order new items + of the same product type. For a delivered order, return or exchange can be only done once by the agent. The agent needs to explain the exchange detail and ask for explicit user confi ronfiomation (yes/ne) proceed.", "parameters";
"type": "object",
"properties":
"order_id";
"type": "string",
"description": "The order id, such as #00000000". Be - careful there is 'f' symbol at the beginning of the order id.",
h, iitem_ids" { "type": "array", "itoms": { "type": "string",
,, "description": "The item ids to exchanged, each such as '1008292230', There could be duplicate items in the list.",
14

---

Published as a conference paper at ICLR 2025
"new item ids": "type": "array", "items": {
"type": "string",
,, "description": "The item ids
exchanged for, - each such as '1008292230', There could be duplicate items in the 1ist. Each new item id should match the item id in the same position and of the same product.",
b,
beaeyment_mehho_iddee {
"type": "string",
"description": "The payment method id to pay or receive refund for the item price difference, such as 'gift_cate 0000030' or "crodlt_card_0000000 These can be looked up =
from the user or order details.",
b
"required";
"order_id",
"item_ids"
"new seew_item_dde
"payment_method mwwheeeteheooooo. id",
1,
b
Retail policies
# Retail agent policy
As retail agent, you can help users cancel or modify pending orders, return or exchange delivered orders, modify their default user address, or provide information about their own profile, orders, and related products.
the beginning of the conversation, you have to authenticate the user identity by locating ing their user id via email, or via nane + zip code. This has to be done even when the user already provides the user id.
Once the user has been authenticated, you can provide user with information about order, product, profile information, e.g. help the user look up order id.
per conversation (but you handle multiple requests from the same user), and must deny any requests for tasks related to any other user.
You enly help
Before taking consequential actiens that update the database (cancel, modify, return, exchange), you have list the action detail and obtain explicit user confirmation (yes) to proceed.
You should not make providea from the or comments.
information knowledge or procedures net the tools, give subjective recommendations
You should at most make one toel call at a time, and if you take a tool call, you should not respond to the user the same time. If you respond to the user, you should not make tool call.
You should transfer the user to human agent if and only if the request cannet be handled within the scope of your actions.
15

---

Published as a conference paper at ICLR 2025
## Domain basic
-All times in the database are EST and 24 hour based. For example "02:30:00" means 2:30 AM EST.
Each user has a profile of its email, default address, user id, and payment methods. Each payment methed is either a gift card, a paypal account, or credit card.
Our retail store has 50 types of products. For each type of product, there are variant items of different options. for example, for a 't shirt' product, there could be an item with option 'color blue size M", and another iten with option 'celor red size L'.
Each product has an unique product id, and each item has an unique item id. Thoy have no relations and should not be confused.
Each order be in status 'pending', 'processed', 'delivered', 'cancelled'. Generally, you can only take action on pending delivered orders.
Exchange modify order tools can only be called once. Be sure that all items to be changed collected into 1ist before making the tool call!!!
## Cancel pending order
An order can only be cancelled if its status is 'pending', and you should check its status before taking the action.
The user needs to confirm the order id and the reasen (either 'no longer needed' 'ordered by mistake') for cancellation.
After user confirmation, the order status will be changed to cean, and the total will be refunded via the original payment method immediately if it is gift card, otherwise in 5 to business days.
## Modify pending order
An order can only be modified if its status is 'pending', and should check its status before taking the action.
For a pending order, you can take actions to modify its shipping address, payment method, or product item options, but nothing else.
## Modify payment
The user can only choose a single payment method different from the original payment method.
If the user wants the modi fy the payment method to gift card, must have enough balance to cover the total amount.
After user confirmation, the order status will be kept 'pending', The original payment method will refunded immediately if it is gift card, otherwise in 7 business days.
### Modify items
This action can only be called cnce, and will change the order status to 'pending (items fodifed) ' and the agent will not be able to modi fy or cancel the order anymore. So contirm all the details are right and be cautious before taking this action. In particular, zemember to remind the customer to confirm they have previded all items to modified.
16

---

Published as a conference paper at ICLR 2025
For pending order, each item can modified an available new item of the same product but of different product option. There cannot be any change of product types, c.g. medify shirt to shoc
provide payment method pay receive refund of the price difference. the user prevides a gift card, must have enough balance to cover the price difference.
The
## Return delivered order
An order can only be returned if its status is 'delivered', and you should check its status before taking the action.
The needs confirm the crder id, the list of items be returned, and a payment method to receive the refund.
The refund must either to the criginal payment method, or an existing gift card.
After user requested', and the user will receive an email regarding how to return items.
the order status will be changed to 'return
## Exchange delivered order
An order can only be exchanged if its status is 'delivered', and you should check its status before taking the action. In particular, remember to remind the customer to confirm they have provided all items to be exchanged.
For a dclivered order, each item can be exchanged to an available new item of the same product but of different product option. There cannot be any change of product types, e.g. medify shirt to shoe.
The user must provide a payment method to or receive refund of the price ce di fferennce If the user prevides a gi ft card, it must have enough balance to cover the price difference.
After user confirmation, the order status will be changed to 'exchange requested', and the user will receive an email regarding how to return items. There is no need to place new order.
Airline policies
Airline Agent Policy
The current time is 2024-05-15 15:00:00 EST.
As an airline agent, can help users book, modify, or cancel flight reservations.
Before taking any actions that update the booking database (booking, modifying flights, editing baggage, upgrading cabin class, or updating passenger infornation), you must list the action details and obtain explicit contirmation (yes) proceed.
You should not provide any information, knowledge, or procedures not provided by the user or available tcols, or give subjective recommendations or comments.
You should only make one tcel call at time, and if you make a tool call, you should not respond to the user simultaneossly. If you respond to the user, you should not make tool call at the same time.
17

---

Published as a conference paper at ICLR 2025
You should deny user requests that are against this policy,
You should transfer the user to a human agent if and only if the request cannet be handled within the scope of your actions.
## Domain Basic
Each user has profile containing user id, email, addresses, date of birth, payment methods, reservation numbers, and membership tier.
Each reservation has an reservation id, user id, trip type (one way, round trip), flights, passengers, payment methods, created time, baggages, and travel insurance information.
Each flight has a flight number, an origin, destinatien, scheduled departure and arrival time (local time), and for each date:
If the status is "available", the flight has not taken off, available seats and prices are listed.
If the status off, cannet be booked.
"delayed" or "on time", the flight has not taken
If the status is "flying", the flight has taken off but not lanced, cannot be booked.
## Book flight
The agent must first obtain the user id, then ask for the trip type, origin, destination.
Passengers: Each reservation can have at most five passengers. The agent needs to collect the first name, last name, and date of birth for each passenger. All passengers must fly the same flights in the same cabin.
Payment: each reservation can use at most one travel cert ficate, at most one credit card, and at most three gift cards. The remaining anount of travel certificate is not refundable. A11 payment methods must alread be user profile for safety reasons.
Checked bag allowance: If the booking user is a regular member, free checked bag for each basic ecenony passenger, 1 free checked bag for each economy passenger, and 2 free checked bags for each business passenger. If the booking user is silver member, 1 free checked bag for each basic economy passenger, passenger, and 3 free checked bags for each business passenger. If the booking user is a gold member, 2 free checked bag for each basic econony passenger, free checked bag for each economy passenger, and 3 free checked bags for each business passenger. Each extra baggage is dollars.
free checked bag for each economy
Travel insurance: the agent shculd ask if the user wants to buy the travel insurance, which is 30 dollars per passenger and enables full refund if the user needs to cancel the flight given health or weather reasons.
# Modify flight
The agent must first obtain the user id and the reservation id.
Change flights: Basic economy flights cannot be modified. Other be modi fied without changing the origin, destination, and trip type. Some flight segments can be kept, but their prices will not be updated based on the current price. The API does not check these for the agent, so the agent must make sure the rules apply before the API!
reservations
calling
18

---

Published as a conference paper at ICLR 2025
Change cabin: all reservations, including basic econony, can change cabin without changing the flights. Cabin changes require the user to pay for the difference between their current cabin and the new cabin class. Cabin class must be the same acress all the flights in the same flight segment
reservation; changing cabin for just
possible.
Change baggage and insurance: The user can add but not remove checked bags. The user cannot add insurance after initial booking.
Change passengers: The user can modify passengers but cannot modt fy the number of passengers. This is scmething that even a human agent cannot assist with.
Payment: If the flights changed, the needs provide gift card or credit card for payment or refund method. The agent should ask for the payment or refund methed instead.
# Cancel flight
The agent must first obtain the user id, the reservation id, and the reason for cancellation (change of plan, airline cancelled flight, or other reasons)
All reservations can pe cancelled within hours of booking, or if the airline cancelled the flight. Otherwise, basic econory or econony flights can be cancelled only if travel insurance is bought and the condition is met, and business flights can always be cancelled. The rules strict regardless
the membership status. The API does not check these for the agent, so the agent must make sure the rules apply before calling the API!
The agent only cancel the whole trip that is not flown. If any of the segments are already used, the agent cannot help and transfer is needed.
rhe refund will go to original payment methods in 5 7 business days.
## Refund
If the user is silver/gold member has travel insurance or flies business, and complains about cancelled flights a reservation, the agent can offer a certificate as a gesture after confirming the facts, with the amount being $100 times the number of passengers.
If the user is silver/gold member or has travel insurance or flies business, and complains about delayed flights in reservation and wants to change or cancel the reservation, the agent can offer a certificate as gesture after confirming the facts and changing er cancelling the reservat innnr with the amount being $50 times the number of passengers.
Do not proactively offer these unless the user complains about the situation and explicitly asks for some compensation. Do not compensate if the user is regular member and has no travel insurance and flies (basic) economy.
19

---

Published as a conference paper at ICLR 2025
B.2 STAGE II: DATA GENERATION
Below is the Python code to generate the users database in r-retail. As mentioned the paper, the code is mostly generated by gpt-4 with some minor human editing. More data generation code can be seen in htttps://githhii
com/sierra
gessearch/tau-bennte haac///ooocaaaaatcctatatoeooci
We execute the code and use gpt-4 to refine the code based on execution for a few initial iterations, and manually refine the code to polish minor details. The data generation code uses random pack- age to directly sample numeric and categorical entries like dates, prices, flight cabins, and sample from LM-generated lists for textual entries like product types or last names. This approach allows scalable data generation (i.c.,
sample 10,000 users if needed) with minimal human efforts.
import random import json import numpy as np
Updated cities list with corresponding zip code ranges for realism cities_info
"New York": ("NY", [10001, 10292]),
"Los Angeles"; ("CA", [90001, 91607]),
"Chicago": ("IL", [60601, 606571),
"Houston"; ("IX", [77001, 772991),
"Phoenix"; ("Ax", , [85001, 85099]),
phhi""" ("PA", [19019, 19190]),
"San Antonio": ("Tx", [78201, 78299]),
"San Diego": ("CA", 192101, 921991), "Dallas": ("TX", [75201, 753981),
"San Jose": ("CA", [95101, 951961),
"Austin"; ("TX", [78701, 787991),
"Jackscnville"; ("FL", [32099, 32290]),
"Fort Worth"; ("Ix", [76101, 761991),
"Columbus": ("OH", [43085, 43299]), "Charlette"; ("NC", [28201, 28299]),
"San Francisco"; ("CA", [94102, 94188]), "Incianapolis"; ("IN", [46201, 462981), "Seattie": ("WA", 198101, 981991), "Denver": ("CO", 180201, 802991), "Washington"; ("DC", [20001, 20599])
first names ["Emma", "Liam", "Olivia", "Noah", "Ava", "YusuÉ", "Isabella", "Lucas", "Mia", "Mason",
"Sophia", "Ethan", "Aarav", "James", "Amelia", "Lei", "Harper", "Sefia", "Evelyn", "Mohamed", "Yara", "Raj", "Fatima", "Juan", "Daiki", "Mei", "Chen", "Ivan", "Anya", "Cnar"]
last last_names names ("Smith", "Johnson", "Patel", "Nguyen", "Garcia", "Silva", "Kim", "Santos", "Khan", "Li", "Kovacs", "Muller", "Rossi", "Hernandez", "Sanchez", "Ito", 4 "Johansson", "Lopez", "Gonzalez", "Ahmed", "Brown", "Davis", "Wilson", "Anderson", "Themas", "Taylor", 4 "Moore", "Jackson", "Martin", "Lee"]
Punction generate address with city-realistic zip code def generate_address :): steett = ("Maple Drive", "Oak Street", "pine Lane", "glm Street", "Cedar Avenue",
"Aillcrest Drive", "Millow Lane", "Sunset Drive", "River Road", "Lakeview Drive",
"Main Street", "Park Avenue", "Broadway", "Elm Avenue", "Bighland Drive",
+
"Chestnut Street", "Hickory Lane", "Spruce Street", "Cedar Street", "Laurel Lane"]
20

---

Published as a conference paper at ICLR 2025
city, (state, zip range) randem. choice (list (cities info. items ()))
addressl E" (randoom. randint (100, 999) (random. choice (streets)
address2 f"Suite (random.. randint (100, 999) }*
zip_code = str (random. randint (zip_range [0], zip_range (1]))
return
"address1"; addressl,
"address2": address2,
"city": city,
"country": "OSA", "province"; state, "zip": zid_code
def seisson_sanplee (lam, max val) sample =
Use rejection sampling to ensure sample capped max_vai val while sample = 0 or sample >= max_val:
sample np. random. poissen (lam)
return sample
def generate_payment__mee wttppppaaaetttttpeeeee (scu_rt
( $ like
(aaro__423"/"paya/// "ccrr_//00///ppaaaa113/////////3 2aaaaaa___aa"aaa_____
0012"
existing ids [[source)_(randon..... (randem.
rancem id = =" (soouure)
(1000000, 9999999)
[method!"ia]] for user in user protiles.e..ooo.o values for method user "payment mww.hdehhhhhseeeeeee valuuss if random_id not in existing_ids: return random_id
else:
(source)
Generate payment methods
def payment_method();
payment_types ("oredit_card", "paypal", "gift_card")
count_methods poisson_samplett, 5) payment_methodd []
ex existing_methods - set () for in range (count_methods)
payment_saurce random. cendom.cheicee ce (payment_types) if soyymnt__souree
"credit
"credit_card": card":
brand random. ceatsom.choiief// (I"iice "mastercard"))
if ("credit_carce continue payment_methods. .etieett_nnthod..ppennl append "source"; "credit_card",
brand) in existing existing_methods: methods:
"brand": brand,
"last_four": f" ("(addoom, randint (1000, 9999)}"
"id": generate__syymmn__neeooooo.. ("credit ("rredi__card")
H)
exesttng ("retoosssaddrrrrrdrtt...re brand)
elif payment_soure == "paypal";
if ("paypal",) in existing_methods:
continue
syymentt_metoods.appeell "source": "paypal",
"id": generate penerate_payment method id ("paypal")
H)
exist ing_mnthommmssss. add ((paypal",))
else:
if ("gift_card",) in oxisting
continue
payment_methode appendd
"source": "gift_card"
"balance"; random. randint (0, 100),
"id"; generate yenerate_payment mothod id ("gift_card""
H)
21

---

Published as a conference paper at ICLR 2025
existing_metiinee add ("gift_card)
return payment_methods
Generate user profiles with adjusted zip codes user_ user_profiles = ()
for in range (500):
first_name random. choice (first_names)
last_name = random, choice (last_naad) nanes)
raoice
email_suftix, user user_id idutiic
random. sample (range (1000, 10000), 2)
email
f"(first_name. lower () (last_name. lower () (emai __suff_ffii Bexampeeeeepeeeeeeeeeee com"
user_id -
['ffirst_name Laarrss_aame.ooeerreteeeeee (1as_______aaa_ lower () [
addresses aannerate_addrese () for in range pooisson_ssaplae (1,
3)1
payment_methooe payment_methedy
uscr_profiles [user____usr_
"name":
"first_name": first_name,
"last_name": last_name
}
"address": generate_addresst]]
"email": email,
"payment__ethodss" (method! 'id'): 1: method for method + payment_methods),
22

---

Published as a conference paper at ICLR 2025
C RETAIL EXAMPLES
C.I DATA EXAMPLES
Here some examples from wttsss/produuts//rreeeeeeee json respectively. All data is gen- erated by code, and the code is mostly generated by gpt-4, and the gpt-4 prompt is generated by authors.
name": "first name": "James", "last name": "Li" b
aaddress";
"address1"; "215 River Road", "address2"; "Suite 991", "city": "New York", "country"; "DSA", "province"; "NY",
"zip": "10083"
"email":
"james. 11144958xample..om com"",
ceft__card_12259711 { "source"; "gi ft__atd""", "balance": 17, "id": "gi [g11f__ard11129911 1725971
"orders": ["##2611340*, "#N3632959", "#14435622", "#N3638028")
Listing I: An example entry from users database in 7-retail.
"name": "Office Chair",
"product ; "4794339885*,
"variants";
"1793929609";
"item
:ittem__d:: id": "1793929609",
"options":
"material": "fabric",
"color: "black",
"armrest": "none",
"backrest height": "high-back"
"available"; true,
"price": 514.34
4274709903";
"item id": "4274709903", "options"; "material": "mesh",
"color": "red", "armrest": "none", "backrest height"; "standard"
"available"; true,
"price": 544.29
Listing 2: An example entry from products database in 7-retail (more variants omitted).
23

---

Published as a conference paper at ICLR 2025
"order_id id"; "##2611340*,
"user id"; "james 16_5"88",
"address";
"addressi"; "21 River Road", "address2"; "Suite 991", "city": "New York", "countzy": "JSA",
"state": "NY", "zip": "10083"
"items": [
"name": "Wtter Bettle", sproduct_id", id"; "8310926033",
"item id"; =item_id";
"6469567736",
"price": 47.84,
"options":
capacity": "1000m1", "material"; "glass", "color"; "blue"
"name": "Office Chair", "product_id"" "4794339885", "item_id"s "8426249116",
"price": 488.81, "options": {
"material": "fabric", "color": "black",
"armrest"; "fixed",
"backrest height"; "standard"
"ful "ilmetttmnss
"tracking id": ("357962501027")
"item "item_ids"; ("6469567736", "8426249116")
"ssatuss; "processed", "imtieer_hssttry"""
Ecransaction_type"" "payment", "amount": 536.65, 65, "payment method id"; "gift_card__7221
Listing 3: An example entry from orders database in 7-retail.
24

---

Published as a conference paper at ICLR 2025
C.2 TASK AND TRAJECTORY EXAMPLES
Here, tasks are not cherry-picked, and the trajectories are based on gpt-4o function calling agent. Note that the task annotation (in JSON) is hidden the agent.
C.2.1 TASK O: WRONG DECISION
Here, the agent omits the policy that only one exchange per order is possible, and issues one item exchange first without informing user about this policy or recalling such policy itself.
"instruction": "You are Yusuf Rossi in 19122. You received your order #w2378156 and wish to exchange the mechanical keyboard for similar one but with clicky switches and the smart thermostat for one compatible with Google Home instead of Apple Homexit. If there is no keyboard that is clicky, RGB backlight, full size, you'd d go for no backlight. You are detail-oriented and want to make sure everything is addressed in one go.", aactions;
"name":
itemm",,
"arguments";
"order ;i"; "#X2378156",
"item ide": "1 "1151293680", "4983901480"
"new item ids": "7706410293", "7747408585"
iertrent_eeehhd_iiiii id": "credit_card "511926"
Hello there! I've just received my order with the number W2378156, and I'd like to inquire about making couple of exchanges.
assistant: Sure, can help you with that. First, need to authenticate your icentity. Could you please previde your email address?
user: Apologies, but I not comfortable sharing email in chat. the order is Yusuf Rossi and shipping
Rowever, I confirm the zip code as 19122. Would that be sufficient?
assistant: Funct ion (arguments=' ("first_name") "Yusuf", "last_name": "Rossi" "zj =19121"" name= finae=ffin__uuu _urd__uu_____ bd__yd__yoo _a___yy_a____plp
tool: yusut_rosse 9620
assistant:
Thank you, Yusuf. I have successfully authenticated your identity.
Now, let 's proceed with the exchange. Could you please provide the details of the items you want to exchange and the new iters you want in return?
25

---

Published as a conference paper at ICLR 2025
user: Absolutely. starting with the mechanical keyboard from this order, I'd like to exchange it for a similar one but with clicky switches. It's also important that it has RGB backlighting and that it's a full-size model.
assistant:
Function (arguments=' ["order_ id": "4""5"2371120)" name='get_ hame="get_ordess "et__rreerddtailss]
tool: ("order_id"; "IW2378156", "user_id": "yusuf_rossi_9620" "acdress";
("address""; "763 Broadway", "address2"; "Suite 135", "city":
"Philadelphiaa, "country"; "USA", "state": "PA", "zip": "19122"), "items": [["name": "Headohones", "product_id"; "6992792935", ritem_id"; "4202497723", "price": 342.81, "eptions"; ("type": "over-car", "connectivity"; "wircless", "color"; "blue")}, ("namc": "Vacuum Cleaner", "product_id"; "1762337868", "item_id"; "4602305039", "price": 561.05, "options": ("type": "robotic", "bagged/bagless": "bagged", "features":
"cordless")), ("name": "Mechanical Keyboard", pproduct_id", "1656367028",
"price": 272.33, "options": ("switch type":
"Linear", " "RGB", "size": "full size")l, ("nare": "Smart Thermostat", "product id"; "4896585277", "item_ic" "4983901480",
"price": 262.47, "options": ("compatibilityy; "Apple HoneKit", "color"; "black")), ("name": "Smart Watch", "product_id": "6945232052", "iten_id":
"9408160950", "price": 381.26, "eptions"; ("color": "gold", "band material": "leather", "display"; "LCD"))), "fulfillments": [["tracking_ic"" id": ("843053632392"), "item_ide ids": (*4202497723", "4602305039", "1151293680", "4983901480", "9408160950"111, "status"; "delivered", "payment_historry" [[ ( "payett""" "amount"; 1819.92, "payment_methood_id: "eredit_card_991326"11
assistant: Function ("arrentttt""pptttttt_ttee _d" "1663370288
name- name='get_pan get__product_detail_
26

---

Published as a conference paper at ICLR 2025
tool: ("name": "Mechanical Keyboard", "product id"; "1636367028", "variants"; ("9690244451"; ("item_id"; "9690244451", "options"; ("switch type"; "clicky", "backlight"; "RGB", "size": "608"), " "available"; faise, "price"; 236.51), "7706410293": ("item_id"; "7706410293", "options": ("switch type": "clicky" "backlight"; "none", "size": "full size"), "available": true, "price": 269.16), "3616838507": ("iter_id":
"3616938507", "eptions": ("switch type": "tactile", "backlight"; "white" "size": "full size"I, "available": true, "price": 226.111, "8484921793"; i"item_i"" "$484921793", ""sssit"""""""otio"""""""" I"switch type": "Linear", "backlight"; "RGB", "size": "808"1, "available": true, "price": 230.15), "1340995114"; ("item_id"; "1340995114", "options": ("switch type": "tactile", "backlight"; "none", "size": "full size",, "available"; false, "price": 235.13), "6342039236": ("item_id"; "6342039236", "options": ("switch type": "clicky", "backlight"; "white", "size": "full size"), "available": true, "price": 244.91), "1421289881": ("item_id", id":
"1421289881", "eptions": ("switch type": "linear", "backlight"; "none", "size": "808"), "available"; truc, "prico": 268.77;, "7867398203"; ("item_id"; "7867398203", "optiion""""",""""" ("switch type": "linear", "backlight"; "RGB", "size"; "608"), "available"; truc, "price": ? "4648814700"; ("item_id"; "4648814700", "options": ("switch type": "linear", "backlight": "white", "size": "608";, "available": false, "price": 228.84), "5222576926": ("item_id": "5222576926", "options": I"switch type": "Linear", "backlight"; "white", "size": "full size" "available"; false, "price": 249.951, "4402162122": ("item_id") "4402162122",
: type": "tactile", "backlight"; "RGB", "size": "608"), "available": true, "price": 233.9), "1151293680"; ("item_id"; "1151293680*, "optioon",""",""""" ("switch type": "linear", "backlight"; "RGB", "size": "full size"), "available": true, "price": 272.33), "2299424241"; ("item_id") "2299424241*, "aptions": ("switch "clicky""" "backlight": "RGe", "size": "803"), "available"; true, "price": 237.48), "4843487907": ("item_id". "4843497907", "options": ("switch type": "clicky", "backlight"; " "white", "sizc": "sizc""""0*"" "available"; false, "price": 254.84), "9025753381": ("item_id": ("item__"
"9025753381", "options": : ("switch type": "elicky", "backlight"; "RG3", "size": "full size"), "available": false, "price": 231.58), "5439196450": ("item_id": "6439196450", "options": ("switch type": "tactile", "backlight"; "none", "size": "60*", "available": false, "price": 254.56), "9991484137"; {"item_id"; "9991484137*, "options": ("switch type": "tactile", "backlight"; "white", "size": ",08", "available"; true, "price": 240.97), "9665000388"" ["item_id". "9665000388",, "options": ("switch type": "clicky", "backlight"; "none", "size": "808"), "available"; true, "price"; 269.46), "9570044148"; ("iter_id":
"9570044148", "options": ("switch type": "linear", "backlight"; "none", "size": "full size"), "available": true, "price": 231.37), "7658724607"; Fritem_id": "7658724607", "opticns": ("switch type": "tactile", "backlight"; "none", "size": "808"), "available": true, "price": 256.73113
assistant: Here are the available options for mechanical keyboards with clicky switches and RGB backlighting in full-si model:
1. *Ieem ID: 7706410293...
Switch Type: Clicky
Backlight: None
Size: Full Size
Price: $269.16
2. ++ Item ID: 6342039236..
Switch Type: Clicky
Backlight: White
Size: Full Size
Price: $244.91
27

---

Published as a conference paper at ICLR 2025
Unfortunately, there full-size models with clicky switches and RGB backlighting currently available. Would you like proceed with one of the available options, or would you prefer to wait until the desired model is back in stock?
user: Understandable. Amm, that case, I think I'll prioritize the clicky switches and go with the full-size model with no backlight, that is, the Item ID: 7706410293.
assistant: Great choice! Here are the details for the exchange:
Current Item:: Mechanical Keyboard (Item ID: 1151293680)
Switch Type: Linear
Backlight: RGB
Sizc: Full Size
Price: $3
New Item:** Mechanical Keybcard (Item ID: 7706410293)
Switch Type: Clicky
Backlight: None
-iize: Full Size
Price: $269.16
The price difference will be refunded to your original payment method, which is a credit card.
Please confirm if you would like proceed with this exchange.
user: Yes, I can confirm that this exchange is satisfactory. The clicky switches far more significant to than the backlight. Iet's proceed with this exchange.
assistant: Function (arguments- (""rder'd""" ew eiei ids": " "1001110299" "payment method id"; "credit card 9513926" name=' name='exchane exchange delivered order items')
"item_ids"" ("1151166333)3 "nj
tool: ("order_id"; "IW2378156", "user_id"; "yusuf_rossi_960" "acáress";
("addressi"; "763 Broadway", "address2"; "Suite 135", "city"; "Philadelphia", "country": "USA", "state": "PA", "zip": "19122"), "items": [["name": "Headphones", "product_id": "6992792935", "item_id": "4202497723", "price": 342.81, "cptions"; ["type": "over-ear", "connectivity"; "wireless", "color"; "blue")), ( ("name": "Vacuum Cleaner", "product_id"; "1762337868*, "item_id"; "4602305039", "price": 561.05,
"options"; ("type"" "rosotic", "bagged/bagless"; "bagged", "features"; "cordless")), ("name": "Mechanical Keyboard", "product_id"; "1656367028",
"item_ic": "1151293800, "price": 272.33, "options": ("switch type":
"linear", "linear", "backlight": "RGB", "size": "full size" ("name": "Smart Thermostat", sproduct_id"" "4896585277", "item id" "4983901480",
"price": 262.47, "options": ("compatibility": "Apple HomeKit", "color"; "black" ("name": "Smart Watch", "product_id"" id": "6945232052", "item_id".
"9408160950", "price"; 381.26, "eptions"; ("color"; "gold", "band matorial": "lcather", "display": "LCD"))), "fulfillments":
[i"tracking_id"; ( ("843053632392"1, "item_ids"; ("tt=_idd":("4222777777 ("4202497723", "4602305039", "1151293680", "4983901480", "94D8160950*111, "status":
"exchange requested", "payment_history"" [i "payment", "aredit_card__51332
"amount": 1819. 92, "payment_methed_i]
"exchange_ items": ["1151293630"], "exchange new items": ("7000611100000220" 1*7706410293"),
"exchange paypnnttepppaae.teeeeee http:paaaetttttedddeeeeo method "eredit_card. #eredit_car__91122..
"exchange_ppi" price differencee: -3.171
28

---

Published as a conference paper at ICLR 2025
assistant: The exchange request has been successfully processed. Here are the details:
Current Item:** Mechanical Keyboard (Item ID: 1151293680)
Switch Type: Linear
Backlight: RGB
Size: Full Size
Price: $3
N I Mechanical keyboard (Item 1D: 7706410293)
Switch Type: Clicky
Backlight: None
Size: Full Size
Price: $269.16
..Frrce Diffcrence:** $3.17 (will be refunded to your credit card)
You will receive email with instructions how item. Once the return processed, the new item will be shipped to you.
the current
Is there anything else I can assist with today?
user: Yes, there's another item 1'd like to discuss for exchange. Specifically, the smart thermostat from order; it's currently compatible with Apple HomeKit, but I'd like to exchange it for one that's compatible with Google Home.
assistant: mantiion(arruuntttss "pproductidd;; ( name='get "ytt_product_detailss""
tool: ("name": "Smart 'hermostat", "product_id": "4896385277", "variants": ("8722653925"; ("item_id": "8722653925", "options": ("corpatibility": "Google Assistant", "color": "white"), "available": false, "price": 227.8), "8593894906"; ("item_id" id": "8593894906", "optiens": ("compatbbility": "Amazon Alexa", "color": "white"), "available": "aviaal"," faise, "price": 263.111, "2791467853": ("item_id", "2791467853", "options": ("compatibility": "Google Assistant", "color": "stainless steel"), "available"; false, "price": 242.53), "7747408585": ("item_id";
"7747408585", "options"; : ("compatibility"; "Google Assistant", "color"; "aaack"",_"vvvvblbil
true, "price": 249.01), "4953074738"; ("item_id":
"4953074738", "options": { ("compatibilltyy"; "Amazon Alexa", "color": "black"), "black"), "available": true, "price": 226. "4983901480" ("item_id"
"4983901480*, "eptions"; ("compatibilityy; "Apple HomeKit", "color":
"black"), "available"; "aaaaa"""""aiaaaeee"" true,
262.47), "9430266227"; ("itee i"item id": "9480266227", "aptioon"" ( ("compatibilityy" "Apple
"Apple HomeKit", "color":
"stainless steel"), "available": true, "price": 255.98), "6243148452":
("item_id"; "6243148452", "opticns": ("compatibilityy": "Arazon Alexa",
"color": "stainless steei"), "available"; true, "price": 247.0), "3377900078"; ("item id": "3377900078", "options": ("compatibility" "Apple HomeKit", "color": "white"), "available"; true, "price": 68:16
assistant: Here are the available options for smart thermostats that are compatiblc with Google Assistant:
1. **Item ID: 7747408585**
Compatibilis Google Assistantt
Color: Black
Price: $249.01
29

---

Published as a conference paper at ICLR 2025
Unfortunately, this is the only available option that natches your requirement. Would you like to proceed with this exchange?
user: That iten fits the particular requirement I had in mind. Yes, let's proceed with the exchange for the thermostat.
assistant:
Here are the details for the exchange:
Current Item:** Thermostat (Item 4983901480)
Compatibility: Apple Homekit
Color: Black
Price: $262.47
**New Item:** Smart Thermostat (Item ID: 7747408535)
-Cmmattbility:: Google Assistant
Color: Black
Frice: $249.01
The price difference will be refunded to your original payment method, which is a credit card.
Please confirm if you would like proceed with this exchange.
user: Yes, I confirm that I would like to exchange the thermostat as well. The compatibility with Google Assistant is really crucial for so it like the right choice.
assistant:
Function (arguments= ("order :H22338811 ( ("4983901480",, ew_item_idee : ("7777478855"*,, "paymesttt_ met__ crrediit_card__9132222 name- name='excine weecheeechheeeeeee _el ivered_order_iteem
tool:
krror: non-del ivered order cannot exchanged
Function (arguments=' ("surmary" "User Yusuf Rossi wants to exchange smart thermostat (Item ID: 4983901480) for Google Assistant compatible model (Item ID: 7747408585). The system returned an error stating the order is not marked as delivered, but the order status is indeed \'delivered''. Please assist with the exchange.")', name= ttraasffrrtt_oommaee
assistant:
tool: Transfer successful
30

---

Published as a conference paper at ICLR 2025
C.2.2 TASK 7: WRONG ARGUMENT
Here, the agent fails as it is not able reason over complex product inventories find the desired exchange item option.
" "instruction": "You are mei kevacs 8020 (28236) and you want exchange the water bottle and the desk lamp. You want to exchange the water bottle to bigger one, and the desk lamp to less bright one (prefer AC adapter battery > USB) If the agent asks for confirmation, only exchange the desk lamp.", "actions::
"name": oarghanggg_dlliieeeeddrrrrrrrrrr
"arguments":
"order id"; "#76390527",
"item ids": ('8384507844"
"new tnew_item_idee : ["1569765161"),
besyment_method_ide "paypal_76440" 7744869"
user: Hello! bought water bottle and desk lamp from your and I'd like to exchange them both. Could you assist me with it?
assistant: Sure, I help you with that. First, need to authenticate your icentity. Could you please provide your email address?
user: Sure, it's mei__kovass_0020malL.
assistant: Funct ion (argumettst ("email". "mei www__kvvvaaa_0o00 com"
name=! Lind__serrid__y_eee
tool: Error: user not found
assistant: It seems I couldn't find user with that email address. Could you please provide your first name, last name, and zip code instead?
user: Ch, my apologi ogies for the confusion name is Mei Kovaca and my zip code is 28236.
assistant: Function (arguments-* (o ("first_name"; "Mei", "last_name") "Kovacs", "zi p" "28236''''' name= Find__ser_iie iinndLuer_dd_by_naeeeee
tool: mei _si__ovacc__822
assistant: Funct ion (arguments=' ("user_id") id": "mei""""m_aaaaaeee kovacs_8020"))
name= hame='get_uue pyet_ "get_user_details)
31

---

Published as a conference paper at ICLR 2025
tool: ("name": ("first name": "Mei", "last name": "Kovacs"), "address": ('addresss"; "317 street", "address2"; "Suite 461", "city": "Charlotte", "country"; "JSA", "province"; "NC", "zip": "28236"), "email": "mei Novecccccc2222a com", "payment_methods"; ("paypai_764486"" ("source": "paypal", "id": "paypal_7644869") "orders": (*#R6390527", "##7800651", "#R8065207")}
Thank you, Mei. have successfully authenticated your identity. Now, I need to check the details the orders containing the water bottle and desk lamp. could you please provide the order for these items?
assistant:
I'm afraid don't remember the order ID now. unable to provide you with it, I'm sorry.
assistant: Eunction ("order_ide "##6390527")',
name- gett_rdder_detalls))
tool: ("order_id") "##6390527", "user_id":
8020", "adaress";
("address";; "317 Elm Street", "address2"; "Suite 461", "city": "Charlotte", "country": "JSA", "state": "NC", "zip": "28236"), "itens": [["name": "Desk Lamp", "product_id"; "6817146515", "item_id": "8384507844", "price"; 137.94, "eptions"; ("color": "white", "brightness"; "medium", "power scurce": "USB")), ("name": "Biking Boots", "product_id" "7363354090", iitem_id": "1615379700", "price": 253.89, "options": ("size": "10", "material"; "synthatic", "waterproof": "yes") ("name": "Water Bottle", "product_id"; "8310926033", "item_id":
"8538875209", "price"; 45.13, "epticns"; i"capacity": "500m1", "material"; "glass", "color"; "black")]l, "fulfillments": [["tracking_ic" id": ("745984863162"), "item_ids"; ("9384507844", "1615379700", "8538875209"]11, "status"; "delivered", "payrent_histoory"" (""transactionttype"" "payment", "amount": 436.96, "payment__mthhod_cc
assistant: have found the order containing both the water bottle and the desk lamp. Here are the details:
Order ID:AA ##6390527
Status::4 Delivered
IIIems::**
Desk Lamp (Item ID: 8384507844)
Water Bottle (Item ID: 8538875209)
To proceed with the exchange, need the details of the new items you would like to exchange these for. Please provide the iten. IDs of the new items.
user: I don't have the exact item IDs right now. However, could I tell you what I am looking for exactly instead?
assistant: Sure, you can describe the new itoms you want to exchange for, and I can look the item IDs for you. Please provide the details of the new desk lmmp and water bottle you are looking for.
32

---

Published as a conference paper at ICLR 2025
Great! would like exchange my current water bottle for larger one, and the desk lamp for one that is less bright. For lamp, I would prefer one with an AC adapter over a battery-powered one, and definitely over a USB enc, if possible.
assistant: Function ( ("product_id"; "8310926033")",
name= name='get "eeelgggt____ product_detailss
tool: ("hame": "water Bottle", "product_id": _i "8310926033", "variants": ("1434748144"; ("item_id"; *1434748144", "options": ( ("capacity": "1000ml", "material"; Mglass", "ccler": "red"), "available"; false, "price": 49.721, "4579334072": ("item_ id"; "4579334072", "eptions": ("capacity": "750m1", "material"; "glass", "color": "black"), "available": true, "price": 54.85), "6469567736": ("item_idl id" "6469567736", "options"; ("capacity": "1000m1", "material"; "glass", "color": "bluc"), "available"; false, "price": 47.84), "3453331371";
("item_ia" "3453331371", "optiens"; ("capacity"; "500m1", "material";
"stainless steel", "color": "black"), "available": true, "price": 79.
"2439754078"; ("item_id") "2439754078", "options": ("capacity": "1000mi", "material": "stainless steel", "ccler": "red"), "available": true, "price": 49.511, "7843064651": ("item_id", "7843064651", "eptions":
|"capacity": "750m1", "material"; "stainless steel", "color": "blue"), "available": true, "price": 50.141, "7918497119"; ("item
"7918497119", "options": ("capacity": "500m1", "material"; "glass", "color": "blue"), "available": false, "price": 54.51), "5758737025":
("item_id"; "5758737025", "opticns"; ("capacity": "500m1", "materiai": "glass", "color": "green"), "available"; true, "price": 45.09), "7533802601": ("item_id", "7533802601", "options": (" ("capacity"; "500mi", "material": "stainless steel", "cclor": "green"), "available": true, "price": 48.59), "3229676465": ("item_id"; "3229676465", "eptions"; ("capacity": "500m1", "material"; "plastic", "color": "black"),
"available"; true, "price"; 51.94), "2366567022";
("itere ("iten_id";
"2366567022", "options": ("capacity": "1000m1", "material"; "stainless steel", "coler": "blue"), "available": false, "price": 54.04),
"6974536207"; ("item_id"; "6974536207", "options": ("capacity": "750ml", "material": "plastic", "color": "blue"), "available"; true, "price":
49.31, "E777246137": "iitemiid. "6777246137", "options"; l'capacity";
"750m1", "material": "stainless steel", "color": "red"I, "available":
true, "price": 47.761, *8538875209": ("item_id") "9538875209", "options": ("capacity"; "500ml", "material"; "glasss",""""" "color": "black"), "black"""""""""
"available"; true, "price"; 45.13), "9127591879"; ("iter_id":
"9127591879", "options"; ("capacity"; "750m1", "material": "stainless steel", "color": "black"), "available": false, "price": 48.47), "7661609223"; ("item_id": "7661609223", "options": ("capacity": "1000mi", "material": "stainless steel", "coler": "black"), "available": true,
"price": 46.511, "4947921075"; ("item_id"; "4947921075", "options";
I"capacity": "750m1", "material"; "stainless steel", "color": "green"), "available": false, "price": 49.571, "/199146548"; ("item_id": "7199146548", "options": ("capacity": "750ml", "material": "plastic", "color": "black"), "available"; true, "price": 48.0211)
assistant: Function (arguments=' ("product_ : "61771611561111"" name= hame='get__m get_pprceuct____t )
33

---

Published as a conference paper at ICLR 2025
tool: ("name:: "Desk Lamp", "product id": "6817146515", "variants":
("9083642334"; ("item_id"; "9083642334", "options"; ("color": "white", "brightness"; "high", "power source": "USB"), "available"; true, "price": 164.28), "4385534692"; ["item_id"; "4385534692", "options"; {"color": "white", "brightness": "high", "power source": "AC adapter"), "available": false, "price": 138.07), "7624783998"; ("item_id"; "7624783998", "aptions": ("color": "black", "brightness": "high", "power source": "AC adapter"), "available"; true, "price": 154.17),
"1270145486": i"item_id"; "12/0145486", "options": ("color": "white", "brightness"; "high", "power source": "battery"), "available": faise, "price": 144.07), "5320792178": ("iter_id": "5320792178", "options": ("color": "black", "brightness": "medium", "power source": "AC adapter"), "available"; true, "price": 135.24), "5370728469"; ("iter_id""
"5370728469", "eptions": ("color": "silver", "brightness"; "medium", "power source": "USB"), "available"; true, "price": 164.97),
"6805564527"; ("item_iae id"; "6805564527", "options": ("color": "black", "brightness"; "medium", "power scurce"; "USB"), "available"; true, "price": 158.41), "1569765161": ("item_id"; "1569765161", "options"; ("color"; "silver", "brightness": "low", "power source"; "AC adapter"), "available": true, "price": 143.02), "7453605304": ("iter_id": ("ite__r"" "7453605304", "eptions": ("color": "silver", "brightness": "low", "power source": "battery"), "available": true, "price": 150.01), "9190635437":
I"item_ia" "9190635437*, "opticns": "pptooo"" I"color": "black", "brightness"; "Low", "power source": "JSB"), "available": true, "price": 153.23), "4447749792":
id": "447774499)7 "options": ("color": "white", "brightness"; "medium", "power scurce"; "AC adapter"), "available": false, "price": 139.8), "8394507844": ("item_id": "8384507844",
"options": ("color"; "white", "brightness"; "medium", "power source"; "USB"), "available"; false, "price": 137.94)))
assistant: Here are the available options for the water bottle and desk lamp that match your preferences:
### Water Bottle Options:
1. ** Item ID:** 7661609223
Capacity: 1000m1
Material: Stainless Steel
Color: Black
Price: $46.51
2. IIem ID:** 2439754078 Capacity: 1000ml
Material: Stainless steel
Color: Red
Price: $49.51
### Desk Lamp Options:
1. +Iemm ID:* 5320792178
Color: Black
Brightness: Medium
Power Source: Adapter
Price: $135.24
2. Item ID:** 1569765161
Color: Silver
Brightness: Low
Power Source: AC Adapter
Pricc: $143.02
Please confirm which items you would like exchange for, and provide the payment method you would like to use for any price difference.
34

---

Published as a conference paper at ICLR 2025
would Like exchange the desk lamp for the with item ID 5320792178. As for the water bottle, need sone time to think about it. You can proceed with the desk lamp exchange for now.
assistant:
Please note that for delivered order, return or exchange only be done ence by the agent. If you proceed with the desk lamp exchange new, you won't be able to exchange the water bottle later through me.
Would you like to proceed with the desk ammp exchange new, would you prefer to decide on both items before proceeding?
Alright, let's just proceed with the exchange for the desk lamp only. understand the limitaticns, thank you for explaining it to me.
assistant: To proceed with the exchange, here are the details:
Current Desk Lamp:**
Item ID: 8384507844
Color: White
Brightness: Medium
Power Source: USB
Price: $137.94
New Desk Lamp:
Item ID: 5320792178
Color: Black
Brightness: Medium
Fooer Source: AC Adaptor
Frice: $135.24
The price difference will be refunded to your PayPal account.
Please confirm if you would like proceed with this exchange (yes/no)
user: Yes, I would like to proceed with this exchange. Thank you.
assistant:
Function (arguments=' ("ordor_id""""" ("ornttts=tttoordd_dddd, N663905577 "item_ids" (1 ew_item_ie name= exchange dewiveeeeddoreeeeeeeeeeeeee aaaiveerd__odde_iteee..
("8384507844"", "nj sereeenttteth______ee_eee "paypal "paapaa___77668699)
35

---

Published as a conference paper at ICLR 2025
tool: ("order id"; "##6390527", "user id"; "mei kovacs
8020", coovacs_020",,
"address":
("addresss"; "317 street", "address2"; "Suite 461", "city":
"Charlotte", "country": "JSA", "state"; "NC", "zip"; "28236"), "itens": [["namo": "Desk Lamp", "product_id"; "6817146515", "item_id"; "8384507844", "price": 137.94, "eptiens": ("color:: "white", "brightness": "medium", "power scurce": ",SB")), ("name": "Aiking Boots", "product_id": "7363354090", sitem_id": "1615379700", "price": 253.89, "options": ("size" "10", "material"; "synthetic", "waterproof"; "yes")h, I"name": "Kater Bottle", Bottle,,"proauct_ii "product_id"; "8310926033", "iter_id";
"8538875209", "price": 45.13, "epticns"; ("capacity"; "500m1", "material": "glass", "color": "black")]], "fulfillments";
[["tracking_id"; ("7459948831622), "item_ids": (*8384507844", "1615379700", "8536875209"]11, "status" "exchange requested", soayymennhistorr"" "'tramsactiiontyppe.. "payment", "amount": 436.96,
("
ppayment_methen id": "paypal__644863"11 "exchange "exchange_items" items":
("8384507844"), "exchange_nex_itemne jew__eees_tttttt__e_________.. ("5320792178"),
"paraaaa__ "paypaa__76486991 -2.71
assistant: The exchange request for your desk ammp has been successfully processed. You will receive an email with instructions on how to return the current desk lamp. The price difference of $2.70 will be refunded to your PayFal account.
If you need any further assistance, feel free to ask. Have a great day!
user: |UVSTOPHH1
36

---

Published as a conference paper at ICLR 2025
C.2.3 TASK 42: PARTIALLY SOLVE COMPOUND REQUESTS
Here, the task requires the agent to check all orders to fix wrong addresses. However, the agent only fixes the jigsaw order address the user suggests.
"instruction": "Your name is Mei Patel, and you live in 445 Maple Drive, Suite 394, Fort Worth, Texas, 76165. You just created your user id mei_patel__2_ 7272 and ordered seme things, but realized you might have typea your address wrong. You want to check it, and potentially correct all order addresses and your user address. After this, you'd like to check the jigsaw you ordered, and if it's not shipped yet, you want to change it to the easiest jigsaw (easiest level, least pieces) because your kid is too young. By default you use PayPal, You are brief and your memory is not too good scmetimes, but you are polite.", "actions:
"name": "modddyy.eendhggooorreeedessss
"arrgumets";
"order_idi "#79583042", "addressi"; "445 Maple Drive", "address2"; "Suite 394", "city": "Fort Worth", "state": "TX", "country"; "USA",
"zip": "76165"
"name": "noddfy_ooodiigg_orde.dddddss "arguments";
forder_id"; "##4082615",
"addressi": "445 Maple Drive", "address2"; "Suite 394", "city": "Fort Worth",
"state": "IX",
"country"; "USA",
"zip": "76165"
"name": mmodiffy_use_aaddress "arguments":
"user_id"; "mei_patel_7272",
"addressi"; "445 Maple Drive", "address2"; "Suite 394", "city": "Fort Worth", "state": "TX",
"country"; "USA",
"zip": "76165"
"name": "nollfy_pondingooddeetttms!! "arguments";
"order_idd" "#24082615",
"item_ide" ["9779102705"),
tiemmittm__e www_ittem_dds": ids": (*1096508426"),
"payment__method_ide "paypal_4768213>
37

---

Published as a conference paper at ICLR 2025
I think I might have entered my address wrongly during my recent order. Could you please check that for me?
assistant: Sure, I can help with that. First, I need to authenticate your identity. Could you please provide your email address?
user: I'm afraid I don't quite remember which email I usea to register. Can proceed with my user id instead? It's mei p2tel_1272.
assistant: still need to authenticate your identity. Could you please provide your first name, last nane, and zip code? This will help me locate your id.
user: of course, my full name is Mci Fatel and my zip cocc is 76165.
assistant: Function (argumentss" ("frret""""""""ea ("first_name": "Mei", """,,," "last_name"; "Patel", "zipj ": ":"76155"'',
name= find _user id by name_ _ip')
tool: mei_pate1_7272
Thank you for the information. Your id has been authenticated successfully.
assistant:
Could please provide the order id for the order you want me to check the address for?
user: 1 don 't. recall the exact order id, is there way to find my recent orders using my user id?
assistant: Function (arguments=' ("user id":"me" patel_727221 7'72")',
name= name='get use. (eeerddetalss)
tool: ("name": ""first_name": "Mei", "last ; "Patel"), "address": ("addressl": "443 Maple Drive", "address2": "Suite 394", "city": "Fort
Worth", "ccuntry": "USA", "province": "Tx" "zip": "76165"), "email": "mei wppppee11110eexxaeee.ooomm
("credit_card__5000 ["source"; "credit_card" "brand"; "visa", "last_four"; "9904", "id": ("source"; "paypal", "id": 4768213"1,, "orders": 1"#K9583042", "#74082615"
payment_methodee
assistant: found your recent orders. Here are the details:
1. Order ID: #K9583042
2. Order ID: #W4082615
Please 1et me know which order you would like me to check the address for.
user: can't remember exactly which one, but it's the jigsaw that I ordered recently. Could we look into that one, please?
38

---

Published as a conference paper at ICLR 2025
assistant: Function(arguuentee ("order id": "#09593042"1",
name= "gt__order_ettiills/
("order_id"; "#W9583042", "user_id": "mei_patel_7272" "address": ("address1"; "443 Maple Drive", "address2": "Suite 394", "city": "Fort Worth", "country": "DSA", "state": "Tx", "zip": "76165"), "items": I1"name": "Perfume", "product_id" "6858788497", "item_id"; "5421902839", "price": 22, "options": ("scent family": "oriental", "size": "100m_", "gender": "men"I1, I"name": "Dumbbell Set", "product_id" "7233192239",
tool:
"item_ić"; "6245231688" "price": 03, "options": ("weight range":
"30-50 lbs", "material": "iron", "set type": "adjustable"), ("name":
"Luggage Set", "product_id": "5426915165", "item_id": "7160999700", "price": 499.29, "options": ("piece count": "2-piece", "color": "red", "material": "softshell")), ("name": "Garden Hose", peroduct_id": "6679515468", "item_id" id": "3230708338", "price": 99.51, "options": ("longth": "25ft", "material": "latex", "color": "green"))l, "fulfillments": [I, "status"; "pending", "payment_histers [ "payment", "amount"; 1449.08, "payment_methoodic"" "paypal_4768213"")
assistant: Function (arguments=' ["order_idt : "14002611""'' name= cett_order_detaile
tool: ("order_id"; "Iw4082615", "user_id"; "mei_patel_7272" "address": ("address1"; "443 Maple Drive", "address2"; "Suite 394", "city": "Fort Worth", "country": "OSA", "state": "Ix", "zip": *76165"), "items": [["name": "Jigsav Puzzle", "product_id". "1808611083", "item_id". "9779102705", "price": 54.11, "options": ("pieces": "1000", "theme": "art", "difficulty level": "intermediate")) ("name": "Backpack", "product_id"; "2524789262", "item_id"; "5917587651", "price": 212.79, "options": ("color": "grey", "size": "medium", "material": "polyester", "compartment": "laptpp""""""""""""""""""""""" ( ("name": "Grill", "product_id": "6819683" 48", Pitem_id": "3876764226", "price": 981.47, "options": ("type": "electric",
"size": "portable", "features": "side burner")), ("name": "Perfume", "product_id"; "6858788497", "item_id"; "8316205423", "price": 288.75, "options": ("scent family": "woody", "size": "30m1", "gender": "women") 1, sirdduct_idd;; id"; "6945232052", "item_id":
("name": "Smart Watch", "product
"2540052208", "price": 346.42, "eptions": ("color": "gold", "band material"; "silicone", "display"; "LCD"))], "fuifillments"; [], "status"; "pending", "payment_historyy:: [""rransactionttype"" "payment", "amount"; 1883.54, "payment__ethhodiid"" "paypal__7662113"11
assistant: The order with the jigsaw puzzle is Order ID: #M4082615. The eurrent shipping address for this erder
Acaress Line 1:** 443 Maple Drive
Address Line 2:+* Suite 394
**Cit::** Fort worth
**State:* TX
*:Connzzy: USA
-+Ziip Code:** 76165
Is this the address you intended use? not, please provide the correct address, and I can update it for you.
user: That's not right, my correct address should be 445 Maple Drive, Suite 394, Fort Worth, Texas, 76165. Could you please correct my user address and the order address to this?
39

---

Published as a conference paper at ICLR 2025
assistant: update both your default address and the shipping address for the pending order AN4082615 the correct address:
Address Line 1:44 445 Maple Drive
Address Line 2:4* Suite 394
**City:** Fort Worth
**State:** TX
**Countzy:** USA
-+ Code:** 76165
Please confirm this is correct, and will proceed with the updates.
user: Yes, that's correct. Please ahead and update them.
assistant:
Function (arguments=' ""rssnnttt="uuuu__eee"" ("user_id", imei_patel_7272 "addressi";
"445 Maple Drive", "address2"; "Suite 394", "city": "Fort Worth", "state"; "IX", "country"; "USA", "zip": "76165")', name='modifys fy
tool: ("name": ""first_amm"" "Mei", "last_name": "Patel"I, "address": "addresss"; "445 Maple Drive", "address2": "Suite 394", "city": "Fort Worth", "state": "Ix", "country"; "USA", "zip": "76165"1, "email"; "mei wttpppee111eeexxppeeee.eee
"payment_methods"" ( ("oredit_card__50000 ("source"; "credit_card", "brand": "visa", "last_four": "9904", "id": "eredit_card__50300
(paypal_4768211 ("source"; "paypal", "id": ppaypal_476813"11 "orders": (*#69583042", "#94092615"]
Function (rrsssaaeeiiniicceuuuooooooooddr "rs::__(eowW000ddssyyydssssss:" "addressi":"445 Maple Drive", "address2"; "Suite 394", "city":"Fort Worth", "state"; "IX", "country"; "USA", "zip":"76165"" name- come-mmodifiie fyy orderr_d__ddddeeeee
assistant:
tool: ("order_id"; "#w4082615", "user_id": Tmei_patel_7272 "address": ["address1"; "445 Maple Drive", "address2"; "Suite 394", "city": "Fort Worth", "state": "TX", "country"; "USA", "zip": *76165",, "items": [["name": "Jigsaw Puzzle", "product_idf id": "1808611093", sitem_id" "9779102705", "price"; 54.11, "epticns"; ("pieces"; "1000", "theme"; "art", "difficulty level": "intermediate")), ("name": "Backpack", "product_id"; "2524789262", "item_id"; "5917587651", "price": 212. 79, "options": ("color": "grey", "size": "medium", "material": "polyester", "compartment": "laptop")), ("name": "Grill", "product_id": "6819683148", "item_id": "3876764226", "price": 981.47, "options": ("type": "electric", "size": "portable", "features"; "side burner")), ("name": "Perfume", "product_id": "6858788497", sttemid"; "8316205423", "price": 288.75, "options": ("scent family"; "woedy", "size": "30m1", "gender": "women" [ ("name": "Smart watch", "product_id": "6945232052", "iter_id":
"2540052208", "price": 346.42, "cptions"; ("color": "gold", "band material": "silicone", "display"; "IcD"))], "fuifillments"; [], "status": "pending", "payment history": [["transaction_tyy "payment", "amount"; 1883.54, "payment_methodd_1 method id"; "paypal 4768213*)]}
assistant: Both your default user address and the shipping address for the pending order #w4082615 have been successfully updated to:
=*Address Line 1:** 445 Maple Drive
**Adéress Line 2:** Suite 394
**City:** Fort Worth
**atee:** TX
Countzzy:** USA
40

---

Published as a conference paper at ICLR 2025
-**iip C*de:** 76165
Is there anything elso I can assist you with?
Yes, there's a actually one more thing. Could you check the jigsaw that I ordered, and let me know whether it has been shipped yet?
assistant:
The order status for ##4082615 is currently "pending, which means it has been processed er shipped yet.
Is there anything else you would like to know or need assistance with?
user: Yes, I'm considering changing the jigsaw ordered. Could you tell me what is the casiest jigsaw available, like one with the least pieces fit for young kid?
assistant: Function (arguments-' ()'' name llstalllllll
tool: ("Action Camera": "3377618313", "Air Purifier": "3821016478",
"Backpack": "2524789262", "Bicycle"; "9783735446", "Bluetcoth Speaker":
"4768869376", "Bookshelf"; "8600330539", "Coffee Maker": "7996920482",
"Cycling Helmet": "7765186836", "Desk Lamp": "6817146515", "Digital
Camera"; "8940227892", "Dumbbell Set": "7233192239", "E-Reader";
"3801771308", "Electric Kettle": "1075968781", "Electric Toothbrush":
"7352963235", "Espresso Machine": "4354588079*, "Fleece Jacket":
"8560156827", "Gaming Mouse": "5713490933", "Garden Hose": "6679515468",
"Grill": "6819683148", "Headphenes": "6992792935*, "Hiking Boots";
"7363354090", "Indoor Security Camera"; "2985987096", "Jigsaw Puzzle";
"1808611083", "LED Light Bulb": "2696197613", "Laptop": "4760268021",
"Luggage Set"; "5426915165", "Makeup Kit": "5149340237", "Wechanical
Keyboar6"; "1656367028", "Notebock"; "2892623495*, "Office Chair":
"4794339885", "Patio Umbrella": "9743693396", "Perfume": "6858788497*, "Pet Bed": "2747247837", "Portable Charger": "6942297802", "Running
Shoes": "6938111410", "Skateboard"; "1968349452", "Smart Thermostat";
"4896585277", "Smart Watch": "6945232052", "Smartphone": "180172004""
"Sneakers": "747100423"", "133339""" "I-Shirt":
" "Tablet"; "rablett" "8024098596", "rea Kettle"; "9832717871", "Vacuam
cleaner": "1762337868", "Wall Clock": "2344688344", "Nater Bottle";
"8310926033", "Kireless Earbuds"; "9924732112", "Wristwatch"; "5066914160", "Yoga Mat": "4635925001")
assistant: Function (arguments=' ("proouuttt""pruettttttt (aanrrtooncauuttttseeeeeootttcct
: :1888111111))))
name=' cest_product_deeae
41

---

Published as a conference paper at ICLR 2025
tool: ("name": "Jigsaw Puzzle", "product_ic" _id:: "1808611083", "variants": ("3614853563"; ("item_id"; "3614853563", "options": ("pieces": "2000", "theme": "art", "difficulty level": "Intermediato"; "available"; faise, "price": 46.99), "4772738468": ("item_id"; "4772738468", "eptions"; ("pieces": "1000", "theme": "animals", "difficulty level": "beginner"), "available": false, "price": 53.91}, "4068787148": ("iter_id";
"4068787148", "aptions": ("pieces": "500", "theme": "art", "difficulty Levei": "intermediate"), "available"; true, "price": 52.01), "3112842858": ""itemid"" "3112842858", "options": ("pieces": "1000", "theme": "lantasy", "ditticulty Level": "intermediate),, "available"; true, "price": 49.1), "7869640094"; ("item_id". *7869640094", "options": ("pieces": "2000", "theme": "animals", "difficulty level": "expert":, "available": false, "price": 47.59), "1096508426"; ("item_id":
"1096508426", "eptions": ("pieces": "500", "theme": "art", "difficulty level": "beginner"), "available": true, "price": 46.13), "9237024510":
("item_ia"; "9237024510", "opticns": ("pieces": "500", "theme": "animals", "difficulty level": "expert"), "available"; true, "price": 53.53), "5546244844"; ("itom_id": "5546244844", "opticns"; ("picces";
"1500", "theme"; "art", "difficulty level": "intermediate"), "available"; true, "price": 51.59), "1008948180"; ("item_id": "1008948180", "options": ("pieces": "1000", "theme": "art", "difficulty level": "beginner"), "availabie": false, "price": 54.34), "6245746168": ("iter_id": "6245746168", "eptions": ("pieces": "1500", "theme": "aninals", "difficulty Level": "intermediate",, "available": true, "price": 46.0),
"/12/170374"; ("ttem id"; "/12/170374", "options": ("aaas""""""eeeeee" ("pieces": "2000", "theme": "fantasy", "difficulty level"; "beginner"), "available"; faise, "price": 52.03), "9370300555": ("iter_id"; "9370300555", "options"; ("pieces": "1000", "theme": "art", "difficulty level": "expert"), "available": false, "price": 45.9), "9665100170": ("item_id")
"9665100170", "options": ("pieces": "1500", "theme": "animals", "difficulty level": "beginner"), "available"; true, "price": 45.39), "4572024853"; ("item_id"; "4572024853", "options": ("picces"; "1000", "theme": "animals", "difficulty level": "expert"), "available"; true, "price": 53.721, "5645314103"; ("item_id"; "5645314103", "options": ("pieces": "2000", "theme": "animals", "difficulty levei":
"intermediate"), "available": true, "price": 46.19), "9779102705":
("item_ić": "9779102705", "options": ("pieces":"1000 "1000", "theme": "art", "difficulty level": "intermediate"), "available": false, "price": 51.11 "9030221155"; ("item id"; "9030221155", "options": ("pieces": "2000", "theme": "art", "difficulty level": "beginner"), "available"; false, "price": 51.981, "5172162216": ("item_idtl "5172162216", "eptions": ("pieces"; "2000", "theme"; "Landscape", "difficulty level"; "intermediate"), "available"; false, "price"; 48.51H1
assistant: The easiest jigsaw puzzle available, suitable for young kid, is the one with the least pieces. Here the options with 500 pieces:
1. +Ietem ID: 1096508426**
**Theme: ** Art fibiffiuulty level:** Beginner **Price:** $3
2. +IItem ID: 9237024510..
**Theme: ** Animals **Difficulty Level:** Expert **Price:** $53.53
Would you like to exchange your current jigsaw puzzlc for one of these? If so, please confirm which one and provide payment method to handle any price difference.
user: Could we with the Art themed one, Item ID: 1096508426? I generally PayPal for my purchases.
42

---

Published as a conference paper at ICLR 2025
assistant: тo proceed with the exchange, will change the current jigsaw puzzle (Item ID: 9779102705) to the new one (Item ID: 1096508426) and use your PayPal account for any price difference.
Here are the details of the exchange:
**Current ** Jigsaw Puzzle (1000 pieces, Art thene, Intermediate difficulty)
**New Item:** Jigsaw Puzzle (500 pieces, Art theme, Beginner
difficulty)
**Payment M PayPal (ID: paypa1_4768213)
Please confirm if this is correct, and I will proceed with the exchange.
user: Yos, everything looks correct. Flease go ahoad and make the exchange.
assistant:
:"01000066" intem_ids'' (" ("9779102705""; "nj ew ww_item_idde iis": "1096508426" setiment_method__eee id" "pappal "paypal__76621133 4768213")', name='moaity wttplly_eennnnnnneeeeeeeeee order items')
conction (ar
tool: ("order_id"; "IW4082615", "user_id"; "mei_patel17272" "address": ("address1"; "445 Maple Drive", "address2"; "Suite 394", "city": "Fort Worth", "state": "TX", "country": "USA", "zip": *76165"), "items": [["name": "Jigsav Puzzle", "preduct_id" "1808611033", "item_id"; "1096508426", "price": 46.13, "eptions": ("pieces": "500", "theme": "art", "difficulty level": "beginner")), {"name": "Backpack", "product_id"; "2524789262", "item_id"; "5917587651", "price": 212.79, "options": ("color": "grey", "size": "medium", "material": "polyester", "compartment": "laptop")) ("name": "Grill", "product_id": "6819683" 48", Pitem_id": "3876764226", "price": 981.47, "options": ("type": "electric", "size" "portable", "features": "side burner")), ("name": "Perfume", "product_id"; "6858788497", "item_id"; "8316205423", "price": 288.75, "options": ("scent family": "woody", "size": "30m1", "gender": "women") 1, _dd; ("name": "Smart Watch", "product id"; "6945232052", "item_id":
"2540052208", "price": 346.42, "eptions": ("color": "gold", "band material"; "silicone", "display": "LCD")),, "fulfillments"; [], "status"; "pending (item modified)", "payment_history"": [["ranssctionttyp"": "payment", "amount": 1883.54, "payment_methodd_" "paypal_4766213"" "rrransactionttype"" "refund", "amcunt": 7. 7.979999999999997 tetteent__ethe_iiite
assistant: The exchange has been successfully completed. Your order #74082615 new includes the new jigsaw puzzle:
**New Item:** Jigsaw Puzzle (500 pieces, theme, Beginner difficulty)
The payment difference has been handled using your PayPal account, and refund of $7.98 has been processed.
Is there anything else I can assist you with?
user: No, that would be all. Thank you for your help.
assistant:
You'l re welcome! If you have any more questions or need assistance in the future, feel free to reach cut, Have great day!
43

---

Published as a conference paper at ICLR 2025
user: P +
44

---

Published as a conference paper at ICLR 2025
D AIRLINE EXAMPLES
D.1 DATA EXAMPLES
Here are some examples from users. json and flights. json respectively. All data gen- erated by code, and the code is mostly generated by gpt-4, and the gpt-4 prompt is generated by authors.
name": "first name": "Mia", "last name": "Li" h
aadrress";
"addressi"; "975 Sunset Drive", "address2"; "Suite 217", "city": "Austin", "country"; "DSA", "province"; "TX", "zip": "78750"
"email": "mia. www.111111aanmne. 113811eexaaamee.ccomm "dob": "1990-04-05", "payment, "ettent__ethhds":
"credit_cad Bacrii___car__221488551
"source": "credit_card"
"brand": "visa",
four": "7447",
"id"; (credit_card_442111
cortificate -o0rtte33__6388" "source": "certificate",
"amount": 100,
"id":
certificate 7504069"; "source": "certificate",
"amount": 250, "id"; "pertificate__50400
seredit_card_do _ard__955700"; {
"source": "credit_card", "brand": "visa", flast_fourts "1907" "id"; "eredit_car_115700"
"saved passengers": I[ "First_name" "Amelia", "last_name : "Ahmed", "dob": "1957-03-21* "membership"; "gold", "reservations"" ["N06J03", "AIXC49", "HKRG34"]
Listing 4: An example entry from the users database in 7-airline.
45

---

Published as a conference paper at ICLR 2025
"#light_number": "HAT001", "origin"; "PHL",
"destination": "LGA",
"scheduled departure
[esteectleeddeeeprtureeeeee time est": "06:00:00",
"scheduled_arre aaaedddrriilllleeeeeeeeellleeeee time__tmm___________ est": "07:00:00",
"dates":
"2024-05-12";
"status": "Landed",
"actual_departure. timeeturrettte___eee time est": -2002-08-112100:::::00
'actual_arrival_tine tttaaas___memeesse_:_____ est": "2024-95-12106:3390000
"2024-05-13"
"status": "landed",
pactual_departee time est"; 00024-05-11370::00000
"actual_atrrial_ titeeess_mmeeett:::: est": "2024-05-13106:3::000
"2024-05-14";
"status"; "cancelled" )
"2024-05-15";
"status": "landed",
Bectual_departure.. timeeettmm____eeeeeeeeeeeeee _ime__st": 12022-05--1100::011000
"actual time est": "2024-03-15707:30:00
"2024005-6";
"status"; "available",
"available_seats": "basic_economy"" 16, "economy": 10,
"business"; 13 b,
"prices": "basic saraic_ecooomy"" 87, "economy": 122, "business": 471 }
"2024-05-17;
"status"; "available",
"available_seats"
"basic_ecooomy": 16, "economy": 13,
"business": 9 },
"prices": { "basic_economy"" 76, "economy": 189, "business": 498
2224-05-18";
"status"; "available",
"available seats"; ; "basic economy": 2, "economy"; 17,
"business"; 20 b,
"prices": "basic_ pocsi__econom":: 91, "economy": 186, "business"; 321 }
46

---

Published as a conference paper at ICLR 2025
'reservation id": "4W0150",
"user "user_id": "chen_jackson_220", "origin": "DEW", "destination": "LAX", "flight_typp"" "round_trip" "cabin": "business", "flights";
" "origin": "DFW", "destination": "LAX",
"flighh number": "HAT170", "date": "2024-05-22", "price"; 883
"origin": "LAX" "destination": "DFW", "flight : "HAT022", "date": "2024-05-26", "price": 779
},
"passengers";
"first_name"; "Chen", "last =ast__ "Jackson", "dob": "1956-07-07" },
"first name": "Raj", "last name": "Smith", "dob": "1967-04-01" b
"'iirst_naame name": "Fatima", "last name": "Martin", "dob": "1970-01-20" }
hiitinnr__isttty"; It "payment_ich" "gift__aar_33776811 "amount"; 4986 } "created_at"; 0202400--2200110119,
"total bootaalbaggages"" 5,
"nonfree_ bargragee__gggee 0,
"insurance" "no"
Listing 5: An example entry from the reservations database in 7-airline.
47

---

Published as a conference paper at ICLR 2025
D.2 TASK O: SUCCESSFUL TRAJECTORY
Here a successful gpt-4o agent trajectory, where the agent handles dynamic user intents and com- plex flight information the conversation.
"instruction": "You are mia_11_33000 3668. You want to fly from New York to Seattle en May 20 (onc way) You do not want to fly before 11am est. You want to fly in economy. You prefer direct flights but stopover also fine. If there are multiple options, you prefer the one with the lowest price. You have 3 baggages. You not want
insurance, You want to use your two certificat prefer using the larger one, and pay the rest with your 7447 card. You are reactive to the agent and will not say anything that not asked. Your birthday is your user profile so you do not prefer to provide it.", "actions";
to pay. If only one
certificate can be used,
"name": "book "book_reseeer reo__rseeev,,,n"
"arguments":
"user_iar _id; "mia_1i_3668"
"origin": "JFK", " "destination": "SEA", Frlight_tyype;:
"flight type": "ene_way", "cabin": "economy", "flights":
Elighht_nubber "HAT136",
"date": "2024-05-20",
"flight : "HAT039",
"date": "2024-05-20",
}, "passengers": "i'ffrss_naame" "Mia", "last "last_name"; name": "Li", "dob": "1990-04-05")],
"payment__mthodds".
"payment id": "certificate Ccertifiiate_/504466 "amount": 250),
"payment id"; "credit_ card 4421486",
"amount"; 51,
"total batiaal_bagggess 3,
"nonfree baggages"; "nonfree_baggagee": 0,
"insurance": "nc",
b,
user: Hello! I am planning trip and need to baok flight from New York to Seattle. Could you please assist with this?
assistant: Sure, I'd be happy te help you with that. First, need to get your user ID. Could you please provide it?
user: Certainly, my user ID is mia 1i 3668.
assistant: Great, thank you! Next, could you please provide the following details for your trip:
1. Trip type: One way round trip?
2. Departure date (and return date if it's round trip)?
48

---

Published as a conference paper at ICLR 2025
3. Cabin class: Basic economy, eceneny, business?
4. Number passengers and their details (first name, last name, date birth)
5. Do you want to purchase travel insurance for this trip (30 dollars passenger)?
Once I have this information, can proceed with searching for available flights.
user: 1. It's a one-way trip.
2. want to depart on May 20.
3. I'1l be flying in economy class.
4. It's just me, the details should be in my profile.
5. No, do not
purchase travel insurance.
assistant: Function (arguments=' ("user_id":''aaa name= name='get_uscee get_ussrr_eeaalls!
tool: ("name":
"Mia", "last_name": "Li"), "address":
("addressi"; "975 Sunset Drive", "address2"; "Suite 217", "city":
"Austin", "ccuntry"; "USA", "province"; "IX", "zip": "78750"), "email": mnia.1i391Beexxamee.coom
"dob": "1990-04-05", "payment methods": ""creiit_card_4411666 { ("source"; "credit_card", "brand": "visa", "last_four"; "7447", "id": "credit_card_442116""
"eertificate__856333""
("source"; "certificate", "amount"; 100, "id": Saatifiicaae_4566339 Seertificatt__5540665 ("source": "certificate", "amount": 250, "id": "sextifficate__304055"" "credit carrcittcard119557000 (*source"; "credit_card"" "brand": "visa", "last_foure "1907", "id": "credttccar_193370") "saved__passcngers"" [1"first_name": "Amelia", "last_name"; "Ahmed", "dob": "1957-03-21")1, "membership": "gold", "reservations": ["N06J03", "AIXC49", "#КЕс34"])
Function on (arguments-' ""rrrentt"""rrriinnn"" "dest inaiiinn "SKA", "date";" 2024-05-20") name= search direct direct_flightt)
assistant:
tool: [["flight_numberf" "*AT0Е"", "origin": "JFK", "destination": "SEA"
"06:00:00", bechculloLarrrtaaLLttoo_oo"""
"12:00:00", "status"; "available", "available_seats"" :aalsss:"""sssiiii" ("basic_econony"* 17, "econcmy": 12, "business": 3), "prices": ""basic_economy": 51,
"economy": 121, "business": 239}}, ["Elight_mumber": "HAT083", "AATO"" "origin": "JFK", "destination" "SEA", "scheduled Banpduule___pparuur__iiieeeeeee archeauled_arrivvllttt "stt "07:00:00", "status"; "available", serailable_seatt""
"01:00:00",
"ccsicc" 16, "economy": 7, "business"; 3), "prices": ("basic_economy" 87, "ecenomy"; 100, "business"; 276)
assistant: Here are the avai wailable direct flights from New York (JFK) to Seattle (SEA) on May 20, 2024:
1. #liggh HAT069**
Departure: 06:00
Arrival: 12:00 PM EST
Available Seats: 12 in Econeny
Price: $121
2. **Flight HAT083** Departure: 01:00 AM EST
Arrival: 07:00 AM EST
Available Seats: in Econemy
49

---

Published as a conference paper at ICLR 2025
Price: $100
Please confirm which flight you would like to book.
Both of the flights scheduled too early for me. possible, I would like flight that does not depart before 11am BST.
assistant: Function (arguments=' ("origin""""""""""""""""""" "destination"; "date":") 2024-05-20") name- larrssearrr_eeeeeeeeeeee foaarchhhnesttp_flighhh
tool: [[[ "twflighh_uumbbeeeeeeeeeeeeee "HAT057", "origin": "JEK", "destination": "ATL", "scheduled ptaprtuure_timmessteeeeeeeeeeeee "07:00:00", "scheduled arrival_time _st":: "09:30:00", "status": "available", "available_seatt"" ("wasstttss"aaiicooooaaaaaaaaaae ("basic_econony"" 7, "economy"; 3, "business"; 9t, "prices": ( ["basic_ccenony"" 76, "economy": 141, "business"; 477), "date"; "2024-05-20"),
"schedaled_arriis__ttiee
("flight_nuuber"; "HAT039", "origin": "ATL", "destination"; "SEA", petheculee_deepprtureettiiee
"22:00:00", "03:00:00+1", "status": "available", "available_seats"" ("b:::"aaassiii ("basic_ecoe "aeasic_ccooom" ( 18, "economy": 10, "business"; 21, "prices"; ("basic_eccnomy"" 52, "economy": 103, "business": 2831, "date": "2024-05-20"11,
[f"Elight_ (rFllight_number,, number"; "HAT136", "origin": "JFK" "destination"; "ATI", "schecuule___ppprrrree_eieeeee
"19:00:00", "scheulle@Larrivvl____eeee. "21:30:00", "status"; "available", "available_seatts:: ("baiccccoo"""""""""""""oo" 15, "economy"; 14, "business"; 2), "prices"; ("basi__eccoooo"" 52, "economy": 152, "business": 277), "date": "2024-05-20",,
("Elight_number" "HAT039", "origin": "ATI", "destination"; "SEA", detppedd_ddaattte_teeeeee
"22:00:00", "schaduled__arrivva__teee "03:00:00+1", "status"; "available", "available_scats"" ("basic_oconony" 18, "econemy"; 10, "business"; 2), "prices"; ("basic__conomm"" 52, "economy"; 103, "business"; 2831, "date"; "2024-05-20"1).
[#"flight_number:; "HAT218", "origin": "JEK" "destination": "ATI", Secheculle__dpparuueeeeeine
"18:00:00", "schechle__arriva__ttine....
"20:30:00", "status": "available", "available_seatsi" ("basic_est
"atasic_econon"":
12, "economy"; 1, "business": 4), "prices": : ("basic_econory") 76, "economy": 158, "business"; 3421, "date": "2024-05-20"),
["flight_numbee number"; "HAT039", "origin": "ATL", "destination" "SEA", "scheculed departure time time "03:00:00+1", "status"; "available", "available_seats"" (ari:::""aaaaa
time_est""" est": "22:00:00",
arrivia Satral__tmme__se__e__ est":
"basic_econony""
18, "economy"; 10, "business"; 2), "prices"; ("basic_oconomy": 52,
"economy": 103, "business"; 283), "date"; "2024-05-20011.
Defllight_numberri "HAT268", "crigin": "JFK", "destination": "ATL", "07:00:00", "schedulee__rrrivvl__ttee "09:30:00", "status": "available", "available_seats": "ettasicecconoy""
"scheculed_dee henpollee__ddertture_tieeeeeees
17, "econemy"; 19, "business"; 12), "prices"; ("ascccccccm"""" 91, "economy": 101, "business": 453), "date": "2024-05-20"1,
Beeliight_uubberii "HAT039", "origin": "AIL", "destination": "SEA" "scheculed
"22:00:00", "schedilee_aarrriial_ti..ee.eee "03:00: "03:00:00+1", "status": "available", "available_seats". "ibasicjeconony" 18, "economy": 10, "business"; 2), "prices": (*basic_economy": 52, "economy": 103, "business": 283), "date": "2024-05-20"
assistant: Here are some one-stop flight options from New York (JFK) to Seattle (SEA) on May 20, 2024, with departure times after 11:00 AM EST:
1. Fliggh Option 1+*
**iisst Leg:**
Flight Number: HAT136
Departure: 07:00 PM EST from JFK
Arrival: 09:30 PM EST at AIL
Available Seats: 14 in Ecenemy
50

---

Published as a conference paper at ICLR 2025
Price: $152
**cond Leg:**
Flight Number: HAT039
Departure: 10:00 PM EST from ATL
Arrival: 03:00 AM EST (+1 day) at SEA
Available Seats: 10 in Ecenemy
Price: $103
2. + Flight Option 2+*
Firstt Legi**
Flight Number: HAT218
Departure: 06:00 PM EST from JFK
Arrival: 08:30 PM EST at ATL
Available Seats: 1 in Ecenony
Price: $158
**Secon Leg:**
Flight Numbor: HAT039
Departure: 10:00 PM EST from AIL
Arrival: 03:00 EST (+1 day) at SEA
Available Seats: 10 in Ecenemy
Price: $103
Please contirm which flight option you would like to book.
user: I'll go with the first option: Flight HAT136 departing from JFK at 07:00 PM EST and Flight HAT039 departing from ATL at 10:00 PM EST.
assistant:
Here are the details for your booking:
**Trip Type:** One-way
*+riggin:** New York (JFK)
*+Destination:* Seattle (SEA)
Cabbiin Class:** Rconomy
++ iight:*h*****
*srist Leg:**
Flight Number: HAT136
Departure: 07:00 PM EST from JFK
Arrival: 09:30 PM EST at ATL
**Socnd Legi**
Flight Number: НAT039
Departure: 10:00 FM EST from AIL
Arrival: 03:00 AM EST (+1 day) at SEA
*Passenger:
Mia Li (DOB: 1990-04-05)
**Total Price:** $152 $103 $255
C*Checked Bags:** free checked bags (Gold member economy class) ++Travel Insurance:** No
**Pamment Methods:**
have the following payment methods available:
Credit Card ending in 7447
Credit Card ending in 1907
Certificate with $100
Certificate with $250
Please confirm if you would like to proceed with this booking and speci fy the payment msthod(s) you would like to use.
user: Yes, the booking information correct. For the payment, I would like to use my two certificates first. If only one certificate can be used, please use the one with $250, and charge the rest to the credit card enaing in 7447.
51

---

Published as a conference paper at ICLR 2025
assistant: For safety reasons, only one travel certificate can be used per roservation. Therefore, I will use the $250 certificate and charge the remaining amount to the credit card ending in 7447.
Here are the payment details:
#*Gertificate:** $250
*+Creait Card (ending in 7447) *** $5
Please confirm you would like proceed with this payment arrangement.
Yes, confirm. Please proceed with the payment arrangement.
assistant: (arguments="
[Mcclrroceccccccooeooooooooo.o.. " "des
tination":"SEA''
"a " =0240-222200--0"
" ",0_ org.aaahhhh.a...........oooooaar
dbb":"1990-04-033:
9", "2"moutt"
"amount":5) "total, baggages": "noniree_n__eeeeeeeee baggages" "moniree_bagaagss.dd.. "
name='book reseovtiessrvtiioo
tool: ("reservation_idd: "НATHAI", "user_id"; "mia_li_3668" "origin": "JFK", "destination": "SEA", "flight_type"" "one "ee__,aa", "cabin":
"economy", "flights"; [["flight_
#eflight_nunbbr". _numbrrr "HAT136", "date": "2024-05-20",
"price": 152, "origin": "JFK", "destination": "ATI"), ("flight_number"
"HAT039", "date": "2024-05-20", "price"; 103, "origin": "AII",
"destinatica"; "SEA")], "passengers"; [["first_name": "Mia", "last_nane" "Li", "cob": "1990-04-05"i1, "payment_history"" [["payment_id") "certiffiatt__504055" "amount": 250), ("payment_id":
(rredit_carr_44214666 "amount": 5}1, "created_at":
10004-00-11511:::0000,,,,, "total_baggages" 3, "nonfree_baggages" 0, "insurance": "no")
assistant: Your flight has been successfully booked! Here are the details of your reservation:
Reservation ID: HATHAT
TTrrpp Тype:** One-way
**Origin:** New York (JFK)
**lesttnaticnn:: Seattle (SEA)
**Cabin Class:** Economy
**Flighs:***
**First Legi+*
Flight Number: HAT136
Departure: 07:00
from JEK
Arrival: 09:30 PM EST at ATL S*cood Leg:**
Flight Number: НAT039
Departure: 10:00 PM EST from ATL
Arrival: 03:00 AM EST (+1 day) at SEA
**Fasccggr::*
Mia Li (DOB: 1990-04-05)
++Checked Bags:** 3 free checked bags
**Travel Insurance:** No
**Payment Details:**
**Certificate:** $250
**Creait Card (ending in 7447) $5
52

---

Published as a conference paper at ICLR 2025
If you have any further questions or need additional assistance, feel free to ask. Have a great trip!
###STOP***
53


Reviews:
[{"review_id": "1eJNQQL8ng", "reviewer": "Reviewer_EMki", "summary": "This article introduces a novel benchmark called τ-bench, designed to evaluate the ability of language agents to interact and plan with human users and programmatic APIs in real-world scenarios. The main contributions of the article can be summarized as follows:\n- **Introduction of the τ-bench Benchmark**: τ-bench simulates dynamic dialogues between users and language agents equipped with domain-specific API tools and strategy guidelines. It is built on a modular framework, featuring realistic databases and APIs, domain-specific strategy documents, and detailed instructions for various user scenarios, along with corresponding real annotations.\n- **New Evaluation Metric, passˆk**: In addition to measuring the success rate of agents in task completion (passˆ1), the article proposes a new metric, passˆk, to assess the consistency and robustness of agent behavior across multiple independent trials. This metric is particularly important for real-world agent tasks requiring reliability and consistency, such as customer service.\n- **Comprehensive Experiments on Existing Language Models**: The authors conduct extensive tests using τ-bench on a variety of state-of-the-art proprietary and open-source language models, including GPT, Claude, Gemini, Mistral, and Llama. The experimental results indicate that even the most advanced functional calling agents (e.g., gpt-4o) succeed in less than 50% of tasks and exhibit poor consistency (with passˆ8 < 25% in the retail domain).", "questions": "This evaluation framework is likely to benefit from more fine-grained task definition and evaluation, such as information gathering, policy understanding, decision-making, etc., assessing the agent's performance on each subtask. This will provide more comprehensive understanding of the agent's capabilities, offering specific guidance for future research. Additionally, the paper could incorporate more dimensions of human evaluation, such as the agent's understanding of user intent, adherence to policy, and user satisfaction with the agent.\n\nOther questions can refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The τ-bench benchmark effectively addresses the limitations of existing evaluations by providing a more comprehensive framework for assessing the capabilities of language agents in real-world environments. The introduction of the passˆk metric presents a novel approach to measuring the consistency and robustness of agent behavior, while the pass in k demonstrates the potential for using agents in data synthesis.\n2. The dataset construction closely aligns with practical application scenarios. The testing methods and data definitions exhibit scalability, enhancing the utility of the benchmark for future research.\n3. The experiments are comprehensive. The article is well-written and easy to follow.\n4. The qualitative experiments are highly detailed, pointing out promising directions for future research.", "weaknesses": "1. **Limitations of user simulation**: The user behaviors are simulated by LLMs in this study. Although the authors explore methods to improve user simulation, these efforts cannot entirely mitigate the potential issues arising from simulated users. It may result in simulated behaviors that diverge from those of real users, thereby affecting the assessment of the agents.\n2. **Limitations of Evaluation Metrics**: While the passˆk metric assesses the consistency and robustness of agent behavior, it remains a task success rate-based measure. In some cases, an agent may provide valuable information or service even if it does not fully accomplish the intended task. It may make more sense to use more granular evaluation metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MPxqiwrced", "reviewer": "Reviewer_z9mi", "summary": "the paper build a novel agent evaluation benchmark considering the complexity of domain policy, user-agent interactions with  realistic databases and APIs. Based on that, the paper propose new evalaution metric pass^k to determine the robust and consistency of agent, and present insightful analysis about existing LLMs.", "questions": "1. line 349, why self-reflection is not suitable? despite the agent only have one change to serve the user, it still can reflect its own reasoning and API call before it generate the final responses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. the target problem is valuable and important, such as the  complex policies and rules specific to a task or domain that the agent should follow, and consistency and reliability at scale.\n2. the formulation is reasonable and proposed collection pipeline is easy to understand and expand\n3. the experiment and analysis are comprehensive and insightful.", "weaknesses": "1. the contribution is a little overclaimed and the paper does not given enough credits to task-oriented dialogue system (ToD). There are several examples: 1) evaluation scheme in line 78, the dialogue state tracking in ToD also compare the current dialogue state with ground truth expected state; 2) there are many lm-simulated users in ToD (see below) and rule-based user simulator also bring insightful studies; 3) some important works are missing, such schema-guided dialogue dataset which also explore various schema in different domain and build corresponding APIs and collect data via human-to-human ways. Generally, the major contribution of this paper lies in combination of lm-simulated user, real APIs and introduction of domain-specific policy while others such as multi-turn interactions are very similar with ToD. \n\n2. the related work is not fully discussed. 1) line 108-111, there are many works focus on multi-step user interaction, such as API-Bank [1], and ToolUltra [2]; 2) there are many LM-simulated user studies [3,4,5].\n\n\n[1] API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs\n[2] Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarios\n[3] Beyond Static Evaluation: A Dynamic Approach to Assessing AI Assistants' API Invocation Capabilities\n[4] Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems\n[5] PlatoLM: Teaching LLMs in Multi-Round Dialogue via a User Simulator", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YJtKMD3Nbd", "reviewer": "Reviewer_p3GC", "summary": "The paper proposes a new LLM-agent benchmark for two domains (retail and airline), called tau-bench. The gap that the proposed tau-bench aims to fill is LLM-agent benchmarks that focus on User and LLM agent interactions. I.e. measuring how well a LLM agent performs when interacting with a \"human\" user. To this end they propose to use a LLM based user simulator. Furthermore, the paper describes how the benchmark is constructed and based on databases and APIs. Finally, the authors also propose a new metric for the benchmark, pass HAT k (pass ^ k). This metric measures what the chance is that all passes are successful given k i.i.d. attempts.\n\nThe paper has a detailed evaluation of agents performance on the benchmark, detailed analysis of errors of these agents and some cost analysis of both the agent (LLM-based) and user-simulator (LLM-based). The cost of a single evaluation cost $200. There is also some analysis of user simulator performance.", "questions": "1. How do you evaluate the user simulator?\n2. How could you evaluate the user simulator?\n3. What would be the performance of a smaller LLM (7B / 13B) as user simulator?\n4. How do you derive the unbiased estimator?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "The paper is very clear, well written and well designed. The paper shows that much effort has been put into this work. \n\nImportant contributions:\n1. The main contribution is the benchmark itself. Specifically, the databases, function calls and interaction templates.\n\n2. The evaluation of baseline agents and ablation studies are interesting to see. \n\n3. The proposal of the metric pass ^ k (pass hat k, vs. the existing metrics pass @ k) is quite interesting.\n\n4. Good comparison against previous work, including task oriented dialogue systems (well done).", "weaknesses": "The main weaknesses of the paper is the analysis of the **user simulator**, evaluation of it, and especially using less *prohibitively expensive* models such as open source model of the size 7B-13B.\n\n1. The paper evaluated the user simulator based on mistakes of agents (and identified 4% of errors). However, it would be interesting to establish guarantees on the user simulator. \n2. Additionally, it would be important to use such evaluation benchmarks of the user simulator to show how smaller models perform as user simulators to make such a benchmark more usable and available to the community.\n3. Therefore, at this stage it is not clear at all how smaller models would perform as user simulators, whether it is possible or not.\n4. Also, having a benchmark that only works with the highest end APIs is very prohibitive for the research community. (Great projects that could help alleviate this problem for you are: AutoGuide (a RAG mechanism), Act-Re (a finetuning mechanism), StateAct (a prompting mechanism), Optimus-1 (a finetuning & exploration mechanism)).\n\n---\nAn additional weakness is the presentation of the *unbiased estimator* of the pass ^ k (pass hat k) metric is unclear. There is no derivation or explanation where this formula comes from and why it should be an unbiased estimator.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KIFcbLTycM", "reviewer": "Reviewer_D4Hj", "summary": "This paper proposes Tau-Bench, a benchmark for measuring the performance of models on user/agent interactions where the model has the ability to use tools. The goal of each interaction is to learn information from the user, and achieve some end state the user wants the agent to accomplish, such as re-booking a flight or initiating a return through an online store. They also introduce a pass^k (similar to pass@k) metric, measuring how often an agent succeeds at a given task when the task is re-run with slight variations in syntax/semantics/etc, simulating multiple users attempting to accomplish the same goal, to measure how reliable an agent is.", "questions": "My one suggestion would be to explore (in the final revision, or follow up work) how well open models can do when finetuned specifically for these sorts of tasks. The formulation of agent trajectories in this work would lend itself very well to both supervised finetuning on example gold interactions, and preference tuning on positive and negative examples. Because the answers are verifiable, reinforcement learning would also be a good candidate. I'm curious how well specialized models can perform these tasks, especially compared to strong closed models such as gpt4o.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper proposes a novel benchmark targeting a growing area of interest (user/agent interactions to achieve a goal), and provides a strong foundation for future work in this area. The benchmark is well-defined and easy to understand, the proposed pass^k metric seems like a great measure of reliability across a diverse set of users and scenarios, and the tasks in the benchmark are focused on popular, real-world, tasks that are difficult for current models.", "weaknesses": "I think this paper is very strong overall. There are a few additional results I'd be interested to see, but I don't think any are necessarily required:\n* how consistent is pass^k? E.g. if you run pass^16 10 times, what is the variance?\n* what is the effect of changing the policy document? does model performance change much with more, fewer, or different constraints?\n* do in context examples help improve agent consistency?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)