Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis
Authors: Penny Chong
Harshavardhan Abichandani
Jiyuan SHEN
Atin Ghosh
Min Pyae Moe
Yifan Mai
Daniel Dahlmeier
Venue: ICLR 2026 Poster (2026)
Keywords: agent evaluation
metric
LLM agents
error analysis
Primary area: datasets and benchmarks
TL;DR: User-aware agent evaluation approach with automated error analysis

Abstract: Agent applications are increasingly adopted to automate workflows across diverse tasks. However, due to the heterogeneous domains they operate in, it is challenging to create a scalable evaluation framework. Prior works each employ their own methods to determine task success, such as database lookups, regex match, etc., adding complexity to the development of a unified agent evaluation approach. Moreover, they do not systematically account for the user’s role nor expertise in the interaction, providing incomplete insights into the agent’s performance. We argue that effective agent evaluation goes beyond correctness alone, incorporating conversation quality, efficiency and systematic diagnosis of agent errors. To address this, we introduce the TED framework (Talk, Evaluate, Diagnose). (1) Talk: We leverage reusable, generic expert and non-expert user persona templates for user-agent interaction. (2) Evaluate: We adapt existing datasets by representing subgoals—such as tool signatures, and responses—as natural language grading notes, evaluated automatically with LLM-as-a-judge. We propose new metrics that capture both turn efficiency and intermediate progress of the agent complementing the user-aware setup. (3) Diagnose: We introduce an automated error analysis tool that analyzes the inconsistencies of the judge and agents, uncovering common errors, and providing actionable feedback for agent improvement. We show that our TED framework reveals new insights regarding agent performance across models and user expertise levels. We also demonstrate potential gains in agent
performance with peaks of 8-10% on our proposed metrics after incorporating the identified error remedies into the agent’s design.

Published as a conference paper at ICLR 2026
TALK, EVALUATE, DIAGNOSE: USER-AWARE AGENT EVALUATION WITH AUTOMATED ERROR ANALYSIS
Penny Chong', Harshavardhan Abichandani', Jiyuan Shen', Atin Ghosh', Min Pyae Moe', Yifan Mai", Daniel Dahlmeier*
SAP, Stanford University ipenny.chong, chong, harshawardhan abichandani, jiyuan. shen 3 8sap. com. (atin.ghosh, min. moe, d. dahlmeierr)sage com yifanêcs stanford.edu
harshavarrdaan..ahcchaaaaai,
ABSTRACT
Agent applications are increasingly adopted to automate workflows across diverse tasks. However, due to the heterogeneous domains they operate in, it is chal- lenging to create a scalable evaluation framework. Prior works each employ their own methods to determine task success, such as database lookups, regex match, etc., adding complexity to the development of unified agent evaluation approach. Moreover, they do not systematically account for the user's 's role nor expertise in the interaction, providing incomplete insights into the agent's 's performance. We argue that effective agent evaluation goes beyond correctness alone, incorporating conversation quality, efficiency and systematic diagnosis of agent errors. To ad- dress this, we introduce the TED framework (Talk, Evaluate, Diagnose 0 (1) Talk: We leverage reusable, generic expert and non-expert user persona templates for user-agent interaction. (2) Evaluate: We adapt existing datasets by representing subgoals- susch as tool signatures, and responses- as natural language grading notes, evaluated automatically with LLM--as---uudee We propose new metrics that capture both turn efficiency and intermediate progress of the agent comple- menting the user-aware setup. (3) Diagnose: We introduce an automated error analysis tool that analyzes the inconsistencies of the judge and agents, uncovering common errors, and providing actionable feedback for agent improvement. We show that our TED framework reveals new insights regarding agent performance across models and user expertise levels. We also demonstrate potential gains agent performance with peaks of 8- 8-10% our proposed metrics after incorporat- ing the identified error remedies into the agent's design.
INTRODUCTION
Large Language Models (LLMs) agents (Liu et al. 2023 Jang et al. 2025 Koh et al. 2024 are in- creasingly being adopted for many real-world tasks in various domains due to their potential of fully automating mundane workflows and enhancing productivity. However, evaluation of agents remains a challenge today due to the heterogeneous domains the agents operate in. As every domain comes with its own goals, creating scalable unified evaluation framework which reliably assesses agent performance across diverse tasks is non-trivial. Existing works (Qian et al.) 2024 Lu et al.) 2024 Barres et 2025 Chang et al. 2024) each propose their own evaluation methods, e.g., checking database states, tool signatures, or exact matches which differ in scope and assumptions, making unification challenging. Moreover, since agent behavior is heavily influenced by the conversation trajectory with the user, current assessment methods that overlook the user's 's role in the interaction may fail to comprehensively capture agent's performance.
Given that agents are non-deterministic and it is difficult to craft reference conversations, common practice to interact with the agent to dynamically simulate the user responses in the conversation loop with the agent (Yao et al., 2024). This has been adopted as common practice for agent
"Corresponding author.
Code and dataset are available in the repository http://tti..oooSS/aaammmceeerene

---

Published as a conference paper at ICLR 2026
evaluation because static user setups, where user messages are predetermined. do not work. This is because the agent's 's responses to earlier predetermined user inputs may diverge from the reference conversation for which the static messages were curated. However, most works employing dynamic conversation have limitations because they do not systematically separate user persona from task instructions, thus failing to account for the impact of user behavior (independent of the task) on agent performance, providing incomplete insights. This is important as good agents ask clarifying questions when given incomplete input, while poor agents do not; thus, systematic testing essential for fair comparison across agents and tasks. Despite the complexity of agent trajectories, existing works (Qiao al. 2024 Xiao al., 2024 Qian al. 2025) often stop at metric reporting.
To address these shortcomings, we propose the TED framework (Talk, Evaluate, Diagnose). (I) In the Talking stage, we decouple user personas from task instructions and introduce user-aware agent evaluation framework based on reusable, generic persona templates enabling diverse and systematic creation of test scenarios. (2) In the Evaluation stage, we adapt existing datasets by representing subgoals-such as tool signatures, and responses- natural language grading notes, and evaluate them with LLM-as-a-judge. We propose new metrics that capture not only partial progress and task success, but also the efficiency of task progression I in conversational turns. (3) In the Diagnosis stage, we introduce an automated error analysis tool that examines inconsistencies of both agents and L.L--as-a-judge, automatically identifies errors, and offers actionable feedback. We summarize our contributions as follows:
Propose agent evaluation framework applicable across heterogeneous agent domains that is built on reusable, generic expert and non-expert persona templates that systematically assess the impact of users' role on agent performance.
ii) Introduce a benchmark by adapting existing datasets to grading notes-natural language check- lists of subgoals. Grading notes serve as assertion criteria for LLM-as-a-judge, which scores the agent performance based on its trajectory log without requiring access to the environment.
iii) Introduce new metries to accompany the user-aware evaluation setup, which are essential for capturing an agent's ' progress with respect to the number of conversational turns.
iv) Propose an automated error analysis tool that analyzes the inconsistencies of the judge and agents, uncovering common errors, and providing actionable feedback for agent improvement.
RELATED WORKS
Conversation simulation. A majority of the agents today are conversational and involve invok- ing multiple tools to solve a task. With complex tasks requiring human interaction, the literature (Yao al., 2024 Wang et al.) [2023] Xiao et al., 2024) has adopted a dynamic setup using a LLM- simulated user (user proxy), for automated testing of agents. However, existing dynamic evaluation methods face several limitations: some rely on user instruction prompts that are tightly coupled with specific agents, scenarios, and personas (Yao et al. 2024). while others omit user personas alto- gether (Lu et al.] 2024)-bon of which limit the reusability of evaluation methods across different domains. Although agent performance is influenced by the behavior of user proxy, this dependency is rarely analyzed systematically due to user personas being inconsistently defined across samples (Huang et al. (2025). While prior work (Barres et al.) 2025)) introduced systematic evaluation of "easy" and "hard" personas for one of the domains, their telco-specific user prompt templates are not generic and limits reusability across domains. Our TED framework differs from prior work by allowing end-user to systematically test the agent with reusable, generic expert and non-expert personas that are agent- or task-agnostic. We demonstrate this in our experiments on the 7²-bench (Barres et al. 2025) and ToolSandbox (Lu et al. 2024) datasets, which span various domains such as airline booking, messaging, setting reminders, etc., all evaluated using the same user persona templates without any tuning.
To evaluate agent performance, most prior work (Wang et al. 2023 Xie et al. 2024) relies success rate. However, the metric focuses solely on the final outcome and provides only coarse-gran -gaained assessment of agent behavior. This is first addressed by AgentBoard (Chang et al., 2024) that introduced progress rate as a fine-grained metric but in multi-step agent- environment setting without conversation simulation. We extend this to multi-turn settings and propose metrics that combine turn-level efficiency and progress rate. Unlike MINT (Wang al.
Metries and error analysis.
2

---

Published as a conference paper at ICLR 2026
which measures only final success after t interactions, our turn-aware evaluation captures per-turn progress and efficiency, offering a richer measure of agent performance in complex tasks. Given the non-deterministic behavior of agents, Yao et al. (2024) reports the pass@k k and pass k metrics. In line with the passäk metric used to assess the chance of whether at least one out of k trials is successful, we also report new metrics that capture the best-case performance under the stochastic runs. Instead of checking goal attainment via direct database lookups, tool signatures, etc., we represent all subgoals as grading notes. This approach abstracts complex goals, is user- friendly, and does not require system-state access, making our evaluation applicable to both agents that modify the system state and those that do not. While prior work uses natural language for only some assertions (Barres et al. (025). we extend this to cover tool calls and end responses. Similar to Cui et al. (2025), we identify common errors made by LLM agents; however, our approach discovers these errors in an unsupervised manner via automatic analysis of real-time logs rather than relying on predefined categories.
2023).
3 TALK, EVALUATE, DIAGNOSE: TED FRAMEWORK
We define a L.L agent as an automated system that performs tasks via interactions with users, tools, and the environment. Its action space includes tool use, responses users, and internal reasoning. After each action, the agent receives partial state information, such as API responses, or subse- quent user utterance. To systematically evaluate agents, we introduce the TED framework-Talk, Evaluate, and Diagnose- as complementary and interdependent stages. In the Talking stage, diverse user-agent interactions are simulated, study how robust agents complete tasks, for the different type of users, such as non-expert users who require more conversational turns. Traditional metrics like success or progress rates often fail to capture subtleties of turn efficiency, motivating metrics that consider both task progress and turn-efficiency during the Evaluation stage. Moreover, evalu- ation using LLLM-as- j--juugg are subject to stochasticity and potential errors. The Diagnosis stage helps extract meaningful insights from inconsistencies and errors made by both the agent and LLM- as-a-judge. Together, these stages form a unified framework as detailed in the following subsections.
3.1 THE TALKING STAGE
Dynamic evaluation with expert and non-expert user personas. Existing methods that use LLM- simulated user also known as user proxy (aa et al. 2024 Lu et al. 2024) are constrained by either tightly coupled or missing user personas, hindering systematic analysis of the effect of user behavior on agent performance. A tightly coupled task complexity and user persona, makes it challenging to isolate their individual impacts on agent performance. For instance, when an agent answers technical legal questions, the outcome may differ depending on whether the user is an expert layperson, even if the task complexity remains constant. However, if both the task and user expertise as determined by the user persona vary simultaneously. it becomes difficult to determine which factor is driving performance differences. In this work, we propose a scalable, dynamic agent evaluation framework that leverages reusable, generic expert and non-expert user personas simulate realistic user interactions across a wide range of scenarios. Let P (Pexpent; Pnon Pnon-expert] denote the set of persona prompts with different user expertise level, I be the set of task instructions, and U be the set of full user prompt consumed by the LLM-simulated user. We abstract the full user prompt templating process as a function f. combining user persona prompt p. with a task instruction i:
i),
where p P, I, and u E U. The function includes general rules for the user proxy, along with a two-step process-reflee reflection followed by response. For each agent and task instruction sample i, vary only the persona prompt p to generate Vexpert and Unon expert Refer to Appendix A.3 for the prompt f and user persona template An example of task instruction i is shown in Fig.
3.2 THE EVALUATION STAGE
We define the set of grading notes G as natural language text used as assertion -hased ground truths by LLM-as-a-judge Each subgoal is represented by one such grading note Unlike prior work
Subgool is represented by grading note which is natural language text.
3

---

Published as a conference paper at ICLR 2026
that uses keypoints (Hao et al. 2025 or limited natural language assertions (Barres et al. 2025 expand coverage to include tool calls, their order, and key agent responses in G. While we adopt the notion of milestones (key events that must happen) (Lu et al. 2024 for the set G, do not follow their DAG-based construction method. An example of grading note is: Agent should enable Wifi. More examples are in Appendix A.
3.2.1 LLM-AS-A-JUDGE AND MAXPROGRESSRATEG
LLM-as-a-judge. We extend beyond the multi-step agent-environment setting and exact match metric (Chang et al. 2024) by evaluating agents in a multi-tum user-agent setup, where grading notes serve as subgoals assess both intermediate and final states, tool calls, as well as the agent's output responses. Let D = {(i, G,) | E I be the test dataset, where E I is a task instruction, G;= -- 1i.2; Si,n, } be the set of grading notes associated with the task instruction i, and [G, be the number of subgoals, i.e., grading notes. We denote the corresponding agent trajectory, which includes information on tool calls, agent responses and user utterances for the entire conversation up to the final conversational turn, T. For a task sample (i, G.), the progress of the agent given its trajectory T; is defined as the proportion of subgoals achieved:
|Gd
1
progress(i, G T.)
LLM udge (i, Ti);
G
where LLMgeede returns 1 if the subgoal gi. is achieved, and 0 otherwise. We define the progress rate as the average progress across all samples in the dataset D, i.e., progressrate = BG.Ga>P progress(i, G,, Te) ]. Using L---as-a-judge with grading notes reduces the need for custom dataset-specialized evaluation harnesses and infrastructure. In this formulation, the judge is queried once for every subgoal. However, to ensure reliability, we run the judge multiple times and take majority vote as the final score. We discuss the stability of the judge further in Section 3.3 The LLMgudde(() prompt is provided in Appendix A.4
From pass@k to Max PatogressRate@@t behavior, a commonly used evaluation metric is pass@k = Epr.st [1 //x9/(()) Yao et al, 2024). which measures the probability that least one trial succeeds when sampling out of total trials. The notation e denotes the number of trials that are successful. Each trial represents a complete multi-turn conversation, consisting of multiple back-and-forth user-agent exchanges. By this definition, when n = k; the pass@k metric evaluates to 1 if at least one of the k: trials for a given task is successful, and otherwise. The metric then corresponds to the expected maximum success per task, averaged over all tasks, measuring the agent's 's best performance across the trials:
Given the non-deterministic nature of agent
pass@k k- = E(G,G.)-D [max {success(i, G., G., |l |-= I,..,, k]]] where success(-) E {0,
(3)
The notation success(i, G., +/) for given sample (i, G₂) represents whether the agent with trajec- tory π! successfully completes the task on the I-th trial, with a value of 1 for success and O for failure. By taking the maximum success over i trials via the max(.} operator, we capture the agent's best performance across these trials. We then relax the strict success condition in equation 3 Byy defining a thresholded progress-based success criterion:
pass@k- EGG,G,)>P maxx {1{ progress(i, Gi, +/) 2 threshold 1 -1=1,...... ...,, k}]
(4)
where 11-} is the indicator function and the threshold [0
1] defines the minimum progress for a trial to be considered successful. Setting Whreshold = 1 counts only trials with full sub- goals completion (i.e., progress(i, G.:77) = 1) successful, and treats any partial progress (i.e..
progress(i, G., T.) < 1) as failure.
Nonetheless, equation 4 applies a hard threshold treatinng all progress below the threshold as fail- ure and discards agent's fine-graines progress. To retain this information, we define soft ver- sion, M MaxPrr Progress @ale@k to evaluate agent's best performance based on the maximum progress achieved at the final conversational turn, across k trials, averaged over all samples:
M MaarProoressRllee Raleak E,,,,,)o [max {progress(i, G., 1 11= 1..... ......,]]- (5)
4

---

Published as a conference paper at ICLR 2026
3.2.2 PROGRESS AND TURN-LEVEL EFFICIENCY
The turns within each conversational trial are interdependent where errors in the earlier turns can propagate and impact task success, While the MaxProgress Rateek metric in equation S cap- tures -on-determinism by measuring agent's 's best performance across the k trials and evaluates fine- grained progress only at the final conversational turn, it does not assess how quickly progress is made throughout the conversation. This gap in evaluation leads us to consider two distinct scenarios: i) where making early progress matters, and ii) where it does not.
Rate@k
Early progress matters. this subsection, we view progress as function of conversational turns and for notational simplicity, we denote the progress at turn by p(t) progress(i, Gú, rf[1 t)), where T/ [1 tl denotes the segment of the agent trajectory of from the first turn up to turn t. Let p(()= [o, T] - I& represents the discrete progress values at each turn. For computing AUC, we treat the discrete values p(l) as a continuous, monotonically increasing function obtained via linear interpolation. The function measures the agent's ' progress at turn & by the proportion of achieved subgoals, i... grading notes, assuming previously completed milestones cannot be undone. The AUC of the continuous progress function then defined as AUC f" p(t) dt where T is the maximum turns of a conversation. For a given task sample (i, G+) D. we define pi(t) and pz(t) to be the progress functions of two agents, respectively. Consider the case where both agents starts from progress, i.e., pı (0) p2(0) 0 and first agent is strictly more efficient than the second, i.e. P1 (t) 2 P2((), Vt E (0, T). we have:
AUC L p1(t) dt >
[" pa(t) AUC2-
(6)
In this scenario, an efficient agent- compared to less efficient one o achieve higher AUC score. The AUC rewards agent for achieving subgoals early which is crucial for long-horizon tasks such as navigation (Shridhar et al. 22220, Chevalier-Boisveer et al., (018), where finding the right
room or object early often reduces downstream confusion. Likewise, in multi-step planning tasks, like web browsing (Zhou et al. 2023), early retrieval of relevant results significantly narrows the search space, increasing likelihood of success.
ii) Early progress does not matter, While AUC metric favors early progress, one may argue that this is unnecessary in tasks like booking trip, where reserving a plane and hotel are interchangeable subtasks, and order should not affect the outcome. In such cases, completing the simpler subtask with fewer subgoals first, followed by the more complex one (or vice-versa) should not affect the final score, i.e., case where two agents start with zero progress and reach the same progress within the same number of conversational turns, despite the differences in trajectories. To handle scenarios where early progress is not vital, one can weight the increase in progress uniformly by computing the progressperturn (PPT), forming a telescoping series:
1 T-1
-(() P(T) T
PPT T Σ p(t++))-
where p(() is the discrete progress value at turn 1, is the minimum number of conversational turns to reach the final achieved progress p(T), and p(0) O. To align with the PwwppPoorsssaaseee metric from 3 we report both the Mar AUC and MozPPT@L averaged over the task samples, while setting = k. Further details are in the Appendix A.5
3.3 THE DIAGNOSIS STAGE
Automated Error Analysis. Although majority of existing works (Xiao et al. 2024 Qian et al. 2025) stop at reporting final dataset metrics, we argue that evaluation should also include error anal- ysis and actionable improvements. While using grading notes and LLM-as-a-judge simplify our evaluation, the inherent non-determinism of LLMS remains a challenge. Our proposed metrics ag- gregate results using majority vote from judge runs and the best agent performance across k trials. However, the aggregation overlooks consistency an essential aspect of robust agent evaluation. To address this, we further introduce automated error analysis tool that analyzes both judge and agent inconsistencies by plotting sample-level progress expectations and variances, offering deeper insights on top of the final aggregated metrics.
5

---

Published as a conference paper at ICLR 2026
All samples
i) Low-level Error Identificution
il) Semantic Clustering Error Types
fiden
Cluster
Emor MissingBooking
Ertor I: Missing cancel reservation tool invocation
Missing Tool Usage
Toollnoocation
cancel reservalion
3
Error S: Required Cancellation Tool Not Utilozed
Error 2:
Missing Tool Usage
updatic reservation flight 8 --incorrect arguneents
book
felus
Error
Cluster
IncerrotArruurnee
Booking Payment Allocation Failure
Incorrect Flight
update reservation
Selection Argument
lights
Booking Tool
Figure 1:
Our proposed two-step automated emror discovery approach that automatically identifies common errors of the agent based on judge and agent inconsistencies. Identical error colors indicate that similar low-level errors clustered into the same high-level category.
For each subgoal gh,j G3, we define a binary r.v. Zzg, where Z,,; = 1 if the agent achieves the j-th subgoal under the given trajectory, and otherwise. Let the probability of achieving the subgoal Ji,j be Pr(Zi, = 1) = Zi.j. The progress for the sample (i, G.) defined as the proportion of subgoals the agent achieved, i.e., progress(i, G,,
Σ, 2u, Σ,
Its expectation and variance are given by:
[
Σ3 4.g
Varlprogress(i, Gù, T/)) E1 2..))
E Elprogress(i, Gú, T()
(8)
Gil
|G.1
where z.,j = Σタ 1 is estimated by averaging over Q judge runs per subgoal, generalizing the single binary judge output in equation 2 to a probabilistic estimate. Plotting Elprogress(i, G. 7 and Varlprogress(i, Gi +[)] for each task (i, Gi) D, capture judge's inconsistency through the variance, while agent's inconsistency is reflected in the different expected progress values across the k trials.
Building on this, we propose an automated error discovery approach that automatically identifies the common errors of the agent based on judge and agent inconsistencies. Our approach consists of two steps (1) low-level error identification, and (2) semantic clustering of error types. For
every binary score (4) from the judge, there is corresponding explanation e/)) We define ei.- (Q) (c()) ( 2.j } and the subgoals and corresponding explanations where the judge's prediction is inconsistent or indicates the subgoal may not have been achieved. For each candidate error E E, we first perform the low-level error identification step, followed by semantic clustering step:
candidate set & [(or, en,g) Pr(Z,, 1) < 1} be tuple of
fuee(E)); C fem
(9)
where fiden(-) and felus-) is the error identification, and clustering prompt functions, respectively, and x E is the low-level error, and tically similar errors into the same group and provide high-level error summary. We illustrate this two-step process in Fig. D Note that the errors with the same color are merged into one cluster label. We also show preliminary results demonstrating agent improvement by leveraging the identified rors. For a detailed algorithm our automated error analysis method, and the fiden() and felus(.) prompt templates, refer Appendix A.6
cluster label. This clustering step will merge seman-
DATASETS AND EXPERIMENTAL SETUP
We use two agent benchmarks: 72 ---hench (Barres et al. 2025 and ToolSandbox (Lu et al. 2024). For 7"-bench, we utilize 21 and 25 samples from the airline and retail domains, respectively. For the airline domain, we further divide the samples into "easy" and "hard". For ToolSandbox, we select 37 base scenarios and exclude variants with different initial messages or multi-turn conversations, as these can be effectively simulated using our dynamic user proxy- moxy-where both initial and subsequent messages are generated dynamically, and the non-expert user persona effectively simulates multi- turn conversations. Our setup offers greater variability than the original variants with fixed initial

---

Published as a conference paper at ICLR 2026
Table Overall performance different agent models on 72-bench airline and ToolSandbox, using gpt-4.1 as user proxy and LLM-as-a judge. Results are displayed with scores for Expert Persona Non-expert Persona. For metrics with @k, the number of trials n = 20 for 72-bench and for ToolSandbox. Mar ProgressRRaet@h is abbreviated as M Mar MarProg@k.
Agent Model
MeanProg@h
MaxProguu MaxAUC@k
MorPPT
pass@k
72 bench Airline Domain (Easy)
gpt-4.1
0.95
0.82
1.00
1.00
0.99
0.81
0.80
0.50
1.00
1.00
gpt-4o
0.79
0.86
1,00
1.00
0.96
0.86
0.70
0.53
1.00
1.00
gpt-4o-mini
0.70
0.61
0.90
0.90
0.85
0.73
0.60
0.37
0.80
0.80
gpt-5
0.92
0.92
1.00
1.00
0.97
0.88
0.67
0.54
1.00
1.00
mistral-nemo
0.87
0.49
1.00
0.80
0.97
0.67
0.67
0.48
1.00
0.60
mistral-large
0.65
0.53
1.00
1.00
0.96
0.79
0.60
0.42
1.00
1.00
ToolSandbox Dataset
gpt-4.1
0.91
0.87
0.98
0.97
0.96
0,92
0.84
0.73
0.92
0.92
gpt-4o
0.95
0.94
0.99
1.00
0.98
0.96
0.94
0.81
0.95
0.97
gpt-4o-mini
0.91
0.85
0.95
0.93
0.94
0.90
0.89
0.77
0.89
0.84
gpt-5 0.78
0.78
0.97
0.91
0.95
0.84
0.83
0.66
0.95
0.84
mistral-nemo
0.72
0.71
0.92
0.96
0.88
0.87
0.76
0.65
0.84
0.92
mistral-large
0.82
0.79
0.94
0.95
0.93
0.91
0.87
0.75
0.89
0.89
messages. The base scenarios consist a variety of task-oriented domains ranging from contact updates and messaging reminders, currency conversion, etc. We only milestones (key events that must happen) and convert them into grading notes. Importantly, any existing benchmark can be adapted fit into evaluation framework by converting the ground truths into grading notes. We set the maximum number of turns to 15 for -2 -2-bench and 8 for ToolSandbox. Each sample is evaluated over multiple agent trials, n 20 trials for -e-ch and n trials for ToolSandbox. We report metrics at k = trials. We use the gpt-4. model as LLM-as-a-judge for grading the subgoals and for error identification and clustering in our experiments. Unless specified, the user proxy also uses the gpt-4.1 model. More details are in Appendix A.14
5 RESULTS AND DISCUSSION
5.1 MAIN RESULTS
Table summarizes the overall performance of various agent models on 72-bench airline domain and ToolSandbox, with gpt-4.1 serving the proxy. Additional results for r2-bench tail domain are provided in Appendix AA.II] Table [3] On easy airline samples, metrics such as Max Paagrrrrrrsssaaaaae Rate@k and passãk tend to saturate, with most models achieving near-perfect scores. Mean MeanProg@k, Prog@k, which measures the average progress rate across all k trials, captures how consistently agents can achieve the subgoals. However, even MeanProg@k can remain high for strong models making it less effective at distinguishing between top-performing agents. While M (aw.rrorress Rate@k gives the best agent performance over & trials, fails give any mean- ingful distinction between models, especially for easy samples.
By incorporating AU and MarPPT@k, we obtain a more comprehensive evaluation of agent performance. For example, on r²-bench, gpt-4o-mini (expert) and mistral-large (expert) achieve similar M Mean Progresssss scores (differing by only 5%). However, Mar. AUC:AU shows a larger difference of 10% (0.96 vs 0.85) and change in rankings. Further comparison of the M AU with M MarPPTo scores for the two models, suggests that mistral-large achieves greater turn-level efficiency and faster progress in the initial turns, but both models have equal aver- age progress over turns as indicated by the identical MarPPT @k scores. Similar pattern persists in the ToolSandbox dataset, where models such as gpt-5 and mistral-nemo have larger differences on the AarAU CCC and M ax C.PPPTA metrics, when interacting with expert user, but a smaller difference Max Progress Rate@k.
7

---

Published as a conference paper at ICLR 2026
progressti, G))
progress(i: G))
progress(i, G.)
0.75
0.75
0.75
0.50
0.50
0.25
9pt-5
0.25
9pt-
0.25
gpt-4.1
mistral-nemo
mistral-nemo
mistraHarge
0.00
0.00
0.00
3
4
4
5
1
4
5
8
Turn
Tum
(a)
(b)
(c)
Figure 2: Progress curves for selected ToolSandbox samples. mistral-nemo (non-expert, pur-
(a)
search_remi
ple; AUC=0.88, PPT=0.20) gpt-5 (non-expert, blue; AUC=0.61, PPT-0.20), (b)
find.current.city htw.bitttt.ww.bbtteeeeee.e.e mode: mistral-nemo (expert, purple; AUC=0.77)
gpt-5 (non-expert, blue; AUC=0064). (c) htt.rremmner...tteeett...... dad..ad.ddnnnd gpt-4.1 (non-expert, green; AUC-0.50 vs. mistral-large (non-expert, red; AUC=0.34).
1.0
Th
Elprogress(/ G, r])
8 (10aa
0.4
Sample 14
Sample 18
Sample 22
Figure 3: Elprogress(i, G,, d/ (dot) and Varlprogress(t, G3, of) (error bar) on -ench gpt-4o- mini agent using non-expert user proxy gpt-4.1. Each dot is an agent trajectory from a single trial and each task sample from the airline domain is evaluated using k 20 agent trials. We display only three example samples here. Sample belongs to the hard split and others in the easy split.
We also examine the impact of user persona agent performance. The non-expert user simulates an inexperienced user, resulting in agents taking more conversational turns complete the task. This consistently reflected in the M aar Akk scores, which are lower for the non-expert compared to the expert user persona across all models and datasets. is because the expert persona provides clearer and more informative input, enabling the agents to complete tasks faster. The baseline metric M ax Progres ess Rate@k which measures the agent best performance at the end the conversation, overlooks count and thus show similar agent performances when interacting with expert versus non-expert user. The r2-bench agent with gpt-4o-mini achieves the same M az Progress Rate@k score of 0.9 for expert and non-expert users, but more conversational turns are required during the interaction with non-expert user as shown in the lower M MaxAUss AUC@ and Maa PPT@k scores when compared expert user. Another interesting observation in some cases (e.g., gpt-5 vs mistral-large on 72-bench), both models achieve the same MoozProresssatte user personas. However, when examining the AUrAUCO metric, we see the performance gap between the two models is notably larger for agents interacting with non-expert user compared to expert user. This highlights that the type of user interaction can significantly influence agent performance and should be considered
= 1 for both
important dimension when evaluating agents.
Moreover, we know that the AUC metric emphasizes early progress, while PPT weights the in- creases in progress uniformly across turns. To illustrate this, analyze the performance at the sample level. In Fig. 2 we showcase instances where two agents reach the same final progress but differ in the number of subgoals achieved at various turns. For example, in the ToolSa sample search reminder with recency upcoming. both gpt-5 (non-expert) and mistral-nemo (non- expert) achieve PPT of 0.20, yet their AUC scores 0.61 and 0.88, respectively. The progress
8

---

Published as a conference paper at ICLR 2026
rate curves in Fig. 2a demonstrate that mistral-nemo makes rapid early progress, while gpt-5's progress is more gradual. A closer examination of the agent trajectories (see Appendix (A.15) re- veals that mistral- gpt-5 begins by clarifying questions before invoking tools. Since the AUC is sensitive to agents that achieve subgoals earlier in the interaction, mistral-nemo has much higher AUC score than gpt-5, as compared to the PPT metric which simply averages the increase uniformly. Hence, in scenarios where early progress is less relevant, the PPT metric may be more suitable.
executes tool calls in the initial turn and then seeks clarification, whereas
Beyond demonstrating the additional insights provided by our metrics, we also compare the TED framework whole -incuudig metries and user proxy- wwit the evaluation paradigms from x². bench and ToolSandbox. Based on the experimental results in Appendix [A.1]] we infer that the evaluation paradigms from the original papers may not paint a complete picture of the agent perfor- mance as implied by the different observation in model performance rankings, further justifying the need for comprehensive agent evaluation framework.
5.2 ERROR ANALYSIS
Besides reporting the results on the dataset-level, we analyze the expectations and variances of the multiple judge runs and the different l agent trials, as shown in Fig, 3 The different color points for each sample represent the different agent trial runs, which we refer to as trajectories in our discussion. For sample 14, we can clearly see that half of the agent runs have an expected per-sample progress of 0.6, while 35% of them are at 0.4, suggesting that there are one or more subgoals, where the agent consistently fails or succeeds. Upon examining the trajectories, subgoal to be "Agent should cancel reservation K KINW8N." Notably the trials in these two clusters has zero variance in the judge runs, indicating that the judge is highly confident in its assessment. For this sample, only trajectories 5 and 18 show relatively large variance bars, suggesting inconsistency in the judge model itself. We then apply our automated error analysis to identify the common errors
identify the problematic
Task Instruction Sample 14:
Miiiing Chemeekkabee Ame De Rectlobeve
Ther 18
..... Thes you wast clange your reccst reseraation You wast to keep the sare dasos bat warr to chonge to the clssperr Ausiness roued onpy with direct fights or not lhe agent tells apen does not mentonn 13, you want the agont to cancel she corneet ane and a wew one.... .....
you besiz ronncny canser changed ido not membon
Inteenet Osspar Conzanicatins AnmeettaalcPeemeer
Than
Excerpt:
Aguast Intermeniaae Trsjecterin::
Caut
hoping
my resercation
with tha! get Itatedd aesd your
wath axcifying
Brrur 1I
mesoitii 10 woeld tike
<Chen the resernatoon 10 contrmaation nam??
LaerPaacc Yoak. jus want owitch hantnoos cluss hus boog the came dotes Car
Tusldgent Yoar roem sioes TD i ** DNWagee
booking swinchiigg
lire I'sll canedl provisas
icrrect evedl sant het syuem incametty
gaaaee bacbookke wath be oxroced pernat anout Pour buklen..
$2A1200.1 I
chods what dei would look lie?
"2f Wi Will
uae book couraaer Supommrra
InalAasot Here aue the cosilble fiikks your roua aflez soolchisa in buiiees slane
('aae of mentured aboo spre amano and tootuter: sart fuut toac tound str. boocess
-Dwwir the enelietr flghr optine for quace saslege
(ftlpp mother HA daw
Please ceedim if you want praceed with ihe churge and indicate which retum
"HATIO Mew (N05 (thee sine Mihoond fur sunc Stnal bant 1955-11-0)) (hee nand Maai. "as sand Sencher deby 10w.2541 (ia,, mane Lann nand Wilser, ditt
1fiple
dight you'd
chosso.
Uhn can you pick cheapest for way book?1 just want wheteser's
Inaliacuu cleapest
fiihtt optkat
the chepper feulerss
oprlove
X Ence 31
pood
ailnntab poptestttoon solestion
heal on Vall www.://atoottmmuuiimiiioo.
with:44 "Cnnifoorre" 4* 5307 00 *ndd Cand $30100 commoninn belasos)
TanlSanul: Pigment Sammaayy "Tual Con, of New Booking 4 5871.00 PPPdd
X Esar &. Tool call fos book covevvtiion rtices cnee!
Trajectory 6
Trajectory 18
Figure 4: : -2-bench airline sample 14. The blue box shows a truncated task instruction i E I for the non-expert user proxy gpt-4.1 model. The green boxes contain the truncated dialogue for trajectory 6 (left) and agent's 's trajectory 18 (right). The agent model is gpt-4o-mini. The top-right box shows the errors identified. Zoom in for larger view.
made by the agent. Our tool identifies four distinet errors for sample 14 as shown in Fig. In the trajectory 6, the agent did not check the details of the existing flight, which was supposed to be basic economy. Thus, the agent did not cancel the previous flight when attempting to reschedule causing discrepancy in the final payment output. This error was consistently captured by our judge as indicated by the zero variance bar. On the other hand, trajectory 18 involves a different payment-related error whereby the agent hallucinates the value $2613.00, that exceeds the actual cost. This spurious value prevented the agent from calling book reservation, triggering a cascade of three subsequent errors.

---

Published as a conference paper at ICLR 2026
5.3 INCORPORATING IDENTIFIED ERRORS INTO AGENT'S DESIGN
To show the effectiveness of our identified errors in improving the agent, we incorporate these errors into the design of the agent using several strategies on r²-bench's airline special split (selected due to their low progress rate and progress-per-tung and ToolSandbox in Table 2
We experiment with approaches such as: (i) Errors Insert (EI): add the list of identified TED errors into the agent in- struction without manual refinement and (ii) Human Notes (HN): insert manually refined TED error notes into the agent instruction [] The Errors Insert strategy improves several setups- 4o-mini on 72-bench (+9% in Mean WeennProy @rog@k, +5% in M ozr AU Au
gpt- that awareness of common failures helps the agent perform better, with only a few showing declines. In contrast, the Human Notes strategy gives more consistent improvement for more setups compared to Error Insert. We observe in particular a significant gain in Mar P PT of 7-10% for gpt-4.1 using Human Notes ToolSandbox dataset. It is also observed that with better models like gpt-4.1, 1, both strategies perform consistently well on the ToolSandhox dataset. Furthermore, we provide an addi- tional comparison of the two approaches with the HiTEC-ICL. method (Cui et al. (205)) in Appendix
A.10 We emphasize that our work evaluates the effectiveness of TED errors for agent improvement, rather than introducing new in-context learning method.
Table 2: Agent improvement results using gpt-4.1 as user proxy and LLM-as-a judge. Results are displayed for Expert Persona Non-expert Persona. Additional results are in Appendix A.
Agent Model
MeanProg@k
Mar Prog@k
AozAUC@k
Max
2 bench Airline Domain (Special Split: Samples 7, 14, 21, 23, and 29)
gpt-44o-mni + EI
0.37
0.31 10.01
0.66
0.59 0.61::0.00
1:000
0.39 Jo.02 0.27
0.1 1 10.01
gpt-4o-mini + HN
0.30rooz
0.33 10.01
0.63
0.61::0.00
10.00
0.53
0.45 0.4570.04 10.04
0.14 0.24
gpt-4.1 + EI
0.52
0.38+tu.0
0.78
0.77
0.661u.02
0.41 J0.06
0.26
0.10...02
gpt-4.1 + HN
0.53
0.41 10.03
T..8
0.85 10.00
0.66
0.55 70.08
0.27 =0.00
0.14
TootSandbox Dataset
gpt-4o-mini + EI 0.87
1.89
0.98 0.97
0.94.0.0
0.91
0.85.o..0
0.66
gpt-4o-mini + HN
0.88
0.91 10.06
0.96r0.01
0.961a.03
0.95 10.01
0.92
10.02
0.90
0.74,0.03
gpt-4.1 + EI
0.95
0.93
10.06
0.99
10.01
0.99
10.02
0.97
0.93
10.01
0.87
0.76 10.03
gpt-4.1 + HN
0.95 10.03
0.97 10.10
0.98
1.00
0.99
10.02
0.97
0.95
10.03
0.91 10.07
0.83 10.10
5.4 EXTENDED EXPERIMENTS AND HUMAN VALIDATION
We also present results on the full split of ~bbnch airline and retail domains in Appendix A. and include an ablation user model variation in Appendix A.13] To validate evaluation, conduct human studies on the correctness of the user proxy, LLM-as-a-judge. and the identified TED errors reported in Appendices A.7 A.8 and A.9 respectively. Results show that the user proxy behaves correctly in most cases, with only 6-12% errors due to instruction-following issues, while the L.LM-as-a-judee evaluation on gpt-5 agent exhibits minimal error (0-7%), suggesting both approaches remain reliable and cost-effective. More details are discussed in Appendices A. to A.9
6 CONCLUSION AND FUTURE WORK
In this work, we introduced the TED framework that redefines agent evaluation. We showed that including error insights into the agent's design leads to gains, with peaks of 8% for and 10% for MaxPPTak metrics. In the future, we plan to explore the applicability of our metric to non-task-oricnter domains, such as open-ended dialogue with conversational agents, where the expected responses of the agent can be assessed using our grading notes. Limitation of our approach and L.M usage are discussed in Appendix A. and A.2 respectively.
C@k
An example of Human Notes is "You must strictly check and double confirm all requirements for change flight actions before calling the tool. If you are unsure confused, always ask clarifying questions to the user."
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. 7 72-bench: Evaluating conversational agents in dual-control environment. arXiv preprint 2.Xiv:2506607982, 2025.
Ma Chang, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Ling- peng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn IIm agents.
Advances in neural information processing systems, 37:74325-74366 2024.
Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: platform to study the sample efficiency of grounded language learning. arXiv preprint arXiv:/810.082) 2018.
Yue Cui, Liuyi Yao. Shuchang Tao, Weijie Shi. Yaliang Li, Bolin Ding, and Xiaofang Zhou. En- hancing tool learning in large language models with hierarchical error checklists. arXiv preprint arXiv:2506. 00042, 2025.
Yupu Hao, Pengfei Cao, Zhuoran Jin, Huanxuan Liao, Yubo Chen, Kang Liu, and Jun Zhao. Evalu- ating personalized tool-augmented Ilms from the perspectives of personalization and proactivity. arXiv preprint XXXiv:2503.00771 2025.
Kung-Hsiang Huang, Akshara Prabhakar, Onkar Thorat, Divyansh Agarwal, Prafulla Kumar Choubey, Yixin Mao, Silvio Savarese, Caiming Xiong, and Chien-Sheng Wu. Crmarena-pro Holistic assessment of Ilm agents across diverse business scenarios and interactions. arXiv preprint arXiv:2505. 18878, 2025.
Kyochul Jang, Donghyeon Lee, Kyusik Kim, Dongseok Heo. Taewhoo Lee, Woojeong Kim, and Bongwon Suh. Dice-bench: Evaluating the tool-use capabilities of large language models in multi-round, multi partt dialogues. arXiv preprint arXiv:2506.2280 228553...28,,... 2025.
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po- Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents realistic visual web tasks. arXiv preprint arXiv:2401.13649 2024.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men. Kejuan Yang. et al. Agentbench: Evaluating 1lms as agents. arXiv preprint arXiv:2308. 03688, 2023.
Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, al. Toolsandbox: A stateful, conversational, interactive evaluation benchmark for Ilm tool use capabilities. arXiv preprint wxXiv:2408.04682, 2024.
Cheng Qian, Peixuan Han, Qinyu Luo, Bingxiang He, Xiusi Chen, Yuji Zhang, Hongyi Du, Jiarui Yao, Xiaocheng Yang, Denghui Zhang, et al. Escapebench: Towards advancing creative intelli- gence of language model agents. arXiv preprint arXh::2412. 2024.
Cheng Qian, Emre Can Acikgoz, Hongru Wang, Xiusi Chen, Avirup Sil, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. Smart: Self-aware agent for tool overuse mitigation. arXiv preprint arXiv:2502. 11435, 2025.
Shuofei Qiao, Runnan Fang, Zhisong Qiu, Xiaobin Wang, Ningyu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Benchmarking agentic workflow generation. arXiv preprint arXiv:2410.07869, 2024.
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768, 2020.
Xingyao Wang. Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. Mint: Evaluating Ilms in multi-turn interaction with tools and language feedback. arXiv preprint arXiv:2309, 10691, 2023.
Ruixuan Xiao, Wentao Ma, Ke Wang. Yuchuan Wu, Junbo Zhao, Haobo Wang, Fei Huang, and Yongbin Li. Flowbench: Revisiting and benchmarking workflow-guider planning for Ilm-based agents. arXiv preprint arXiv:2406. 14884, 2024.

---

Published as a conference paper at ICLR 2026
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040-52094, 2024.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. T-bench: A benchmark for tool-agent-usee interaction in real-world domains, 2024. URL https://arxiv org/abs/2406. 12045, 2024.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et Webarena: A realistic web environment for build- ing autonomous agents. arXiv preprint arXiv:2307. 13854, 2023.
12

---

Published as a conference paper at ICLR 2026
APPENDIX CONTENTS
Limitation of our approach
14
A.2
LLM Usage
14
A.3
Prompt templates for user proxy
14
Prompt template for LLM-as a--duee
17
A.5
Additional details on progress and turn-level efficiency metrics
19
A.6
Additional details on automated error analysis
19
A.7
Human validation on the correctness of user proxy
23
A.8
Human validation on the correctness of LLLM-as-ajjudg-
23
A.9
Human validation on the correctness of the identified TED errors
24
A. 10 Additional results for agent improvement experiments
25
A.11 Additional experiments the 7 -ennh dataset
27
A.12 Comparison of Different Agent Models using the Evaluation Paradigms from the Original be-ch and ToolSandbox Papers
33
A. 12.1
~bbench Original Paper Evaluation Approach
33
A. 12.2 ToolSandbox Original Paper Evaluation Approach
34
A. 13 Ablation study on the different user models
36
A. Additional information on dataset and experimental setup
37
A.15 Additional details on agent trajectories
42
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 LIMITATION OF OUR APPROACH
Our approach, which uses grading notes and LLLM-as-judge, simplifies evaluation by relying solely on the agent's trajectory, without requiring access to the underlying environment. However, this approach has certain limitations. While we show that multiple judge runs improves reliability and our automated error analysis tool helps in debugging, the method cannot verify whether the database state has actually changed if such modifications are reflected in the trajectory. Consequently, this limits our ability to capture silent failures that produce no observable outputs.
A.2 LLM USAGE
LLM is used assist the writing of UI code for the automated error analysis tool. In addition to that, we use LLM to refine the prompt templates that are used in our experiments. LLM is also used to refine and polish the text the paper to improve clarity and presentation.
A.3 PROMPT TEMPLATES FOR USER PROXY
The following are the reusable, generic expert and non-expert user prompt templates, followed by placeholder
templates for the two-step function f in equation [ The seteer.askssssmnnee corresponds the task instruction E I, and the (agent_desc) placeholder corresponds to the agent description. For the two-step generation process cocusssffrffee f-reflection followed by response, the placeholders {chat history) and {t (terminat ion msg} represent the user-agent chat history up to the current stage conversation and the termination message that the user should produce at the end of the dialogue, respectively. The placeholder (reflect. ieffecciion reeeciiooiiisee ionhissoor' represents the user reflection history up to the current stage of conversation.
Generic expert user persona prompt template:
You are acting as an expert I.M--siulated user who fully understands Al assistant system and goal. Always respond naturally in clear. concise language that fits the expert user role and goal. Provide complete and precise information in your responses. Generate one line at time. Do not give away all the instructions once. Only provide the information that is necessary for the current step.
You are provided with the following user task summary:
[user_tast task ttt.tttsssssaaae summaryl (user_tasee str.ttsssssummrry)
You understand system well and will provide thorough, accurate responses using only information provided in the [user task summary] section.
If the Al assistant returns output in JSON format, respond only the content inside the JSON the format does matter.
The following provides overview of the Al assistant if available. [Al Assistant Description] (agent_desc)
When you expert LLM-simulated user analysing the real real-time chat history, carry out two-step process the user: first, a Reflection Phase, followed by Response Generation Phase.
14

---

Published as a conference paper at ICLR 2026
Generic non-expert user persona prompt template:
You simulating clueless, casual -eN-expert who is interacting with Al assistant. You don't 't fully understand how the Al system works, and you tend give vague incomplete instructions often leaving out key steps or context.
When you respond:
Speak naturally, casually, like someone who's 's unsure how an Al.
Be brief and only provide part of the needed information. not give full picture unless the assistant directly asks for Only share details that are directly related to what was just asked or prompted - not more.
Never proactively explain your reasoning provide background info unless the assistant digs into it.
You are working toward the following general task: [User Task Summaryl
(user.task.sunmary)
But since you're re not an expert, you'll 'll just sort of "feel your way through it" and leave lots of gaps in your instructions. NEVER provide COMPLETE instructions. ALWAYS OMIT some variables and missing key context.
the assistant returns something structured formats like JSON, you can just react casually to content. Treat the format like it doesn't matter.
The following provides overview of the Al assistant if available. [Al Assistant Description]: (agent_desc)
When you as clueless, casual NON-expert user analysing the real-time chat history, carry out two-step process as the user: first, a Reflection Phase, followed by Response Generation Phase.
When simulating your process during the conversation:
You go through two internal steps each time:
1. Reflection Phase (internal thought):
Take quick look at the current chat history. Think yourself: "Okay, what did the assistant just say ask? What should probably say next without overexplain- ing?"
Remember: you're re not confident in how this system works, so don't don't try to be precise.
Response Generation Phase (your reply):
Now write a short, casual message that gives only partial information based on what the assistant asked. Leave things unclear unless the assistant is persistent.
15

---

Published as a conference paper at ICLR 2026
The reflection-step prompt template in the two-step function / equation 1:
The following [Chat History] (if available) provides context and indicates the CURRENT stage of your conversation as LLM-simulated user with the assistant. [Chat History]
{chat_history
Step Reflection
Given the [Chat History] REFLECT carefully the Al assistant's last response and what the LLM-simulated user is trying accomplish based on the [aser.task task.ssumaar))
Briefly address:
Your role as the I.MM-smmulated user.
The current stage of the conversation. You SHOULD NOT skip any user instructions as mentioned in the [user task summaryl.
assistant's last reply the [Chat History].
IMPORTANT CLARIFICATION:
Review the entire [Chat History] and the [user.task_summary and see what should be your next response a LLM-simulated user.
At times, the Al assistant's last message may overlap with or anticipate a future user turn. In such cases, treat it strictly as the Al assistant response, not replacement of the user message
Do NOT generate the I.M--simulated **IMPORTANT remember your user persona as written system prompt (eg: expert user or non-expert) and respond with appropriate reflection.
response yet. RESPOND only with REFLECTION.
TERMINATE ONLY IF the conversation its FINAL STAGE where the agent has completed all the tasks wanted by the user as shown in the [user task summary]. the conversation has concluded, prepare to respond with fterminat nat n mmg} in the next re- sponse generation phase. Otherwise, DO NOT consider termination if the current conversation is at its final stage.
16

---

Published as a conference paper at ICLR 2026
The response-step prompt template in the two-step function / equation
The following [Chat History] (if available) provides context and indicates the CURRENT stage of your conversation as LLM-simulated user with the assistant. [Chat History]
(chat_history.
following the LLM-simulated user reflection. [Reflection]
(raflection.hister)
Step 2: Response Generation Phase
Given the [Chat Historyl and [Reflection), GENERATE the I.I.M-imulated user NEXT RE- SPONSE that:
i) Naturally continues the conversation WITHOUT ADDING NEW TASK that NOT found in the Juser.lask summary]. You SHOULD NOT skip any tasks for the LLM-simulated user.
ii) Avoids revealing repeating the Al assistant's 's answers.
iv) Responds appropriately the assistant's actual reply, even if vague or off-track. If the Al assistant's last message echocs or resembles any part of user message, it's the Al assistant response, NOT a new user turn. Note that suggestions or recommendations by the Al assistant should NEVER be MISTAKEN for actual actions taken.
GENERATE LLM-simulated USER RESPONSE based on [Reflection). Return ONLY the LLLM-simulate user response.
**IMPORTANT* remember your persona written in the system prompt (eg: expert user non-expert) and respond with appropriate response.
TERMINATE ONLY IF the conversation is at its FINAL STAGE where the agent has com- pleted all the tasks wanted by the user shown in the [user task summary]. the conversation has concluded, prepare respond with (termination.msg) in the next response generation phase. Otherwise, DO NOT consider termination if the current conversation is not at its final stage.
A.4 PROMPT TEMPLATE FOR LLM-AS-A-JUDGE
The following the (i.9..1TT)) prompt template from equation The I, the
deder.task-summaree placeholder corresponds to the task instruction (grading.note) placeholder corresponds to the j-th grading notes J.,j for the task instruction
The remaining placeholders, jertorcc (dynamicDialoque represent agent's trajectory, responses output, and the user-agent dialogue, respectively, extracted from 7;-
fagent_responsed
and
17

---

Published as a conference paper at ICLR 2026
Prompt Template for LLM-as-a-judge equation
You provided with sample that contains several key components centered around interaction between agent and simulated user, referred to the user proxy. The user proxy represents a human-in-the-Be -n-ta--ooo--ooooo.....o engaging with the agent by posing questions and guiding the conversation throughout the dialogue.
The [User Summary Instructions] section outlines the user's goals, expectations, and the all task the agent expected to complete. The [Agent Responses Submission] section captures agent's actual responses to the user proxy at each turn the interaction. The [Agent Intermediate Trajectories] section provides a detailed step-by-ster reasoning and actions taken by the agent. Finally, the [Dynamic Dialogue] section presents the full conversation between the agent and the user proxy.
As judge, your task is to determine whether the subgoal specified in the [Ground Truth Sub- goal] has been achieved. To make this assessment, evaluate the [Agent Intermediate Trajectories] and (Agent Responses Submission] ascertain whether the subgoal has been successfully completed.
When the completion of a subgoal depends on the availability of specific tool, and that TOOL ABSENT, the subgoal must be marked as INCOMPLETE, REGARDLESS of the correctness of the agent's responses in [Agent Responses Submission). Tool availability is hard constraint- no amount valid reasoning can compensate acceptable only the subgoal does not require the tool to be used. However, CONCEPTUALIZING or HALLUCINATING a tool call-i.e., falsely implying the tool was invoked-is NOT acceptable, and the subgoal must be marked as INCOMPLETE.
absence. Suggesting or recommending a tool
agent intermediate trajectory are left BLANK or DOES NOT contain rele- vant tools to the subgoal, it means that the ground truth subgoal is NOT MET and the subgoal is considered as INCOMPLETE, REGARDLESS of the correctness of the agent's responses in [Agent Responses Submission]. The thought in [Agett Intermediate Trajectories] CANNOT be considered as substitute for a tool call and the ground truth subgoal considered be INCOMPLETE.
Additionally
[BEGIN DATA] 8#888
[User Summary Instructions]: (user.task.surmaryy
6843858488**
[Ground Truth Subgoal]: (grading note]
[Agent Intermediate Trajectories]: (trajectory)
6***886*****
[Agent Responses Submission]: (agent.ressonses)
****
(dynamicciallon
[END DATAJ
During assessment focus solely on the factual content and the goal completion while disregarding any differences in style, grammar, punctuation, or syntax.
Begin by presenting a concise argument to confirm the validity of your conclusion. Avoid simply stating the correct with your answer formatted 'GRADE: SLETTER' (without quotes) where LETTER is one of or Reply with 'GRADE: C' (without quotes) to indicate COMPLETE the agent has successfully achieved the subgoal. Otherwise, reply with 'GRADE: (without quotes) to indicate INCOMPLETE if the agent did not achieved the subgoal.
the outset. Decide what type of tools is required and then end
18

---

Published as a conference paper at ICLR 2026
A.5 ADDITIONAL DETAILS ON PROGRESS AND LERNL EFFICIENCY METRICS
To align with the Mar ProgressRotess Rate@k metric from equation 5 which evaluates the agent's best performance across & trials, report both the max AUC and max PPT of the k trials averaged over the task samples:
MarAUC@k = B(i.,))-p mmax(AUC |= 1,........ kH]
(10)
PaT@P = B(G.)~Pb Imax(PPT] 1= 1.......
(11)
We show in experiments that the proposed metrics provide interesting insights into the agent behavior that existing metrics failed to capture.
A.6 ADDITIONAL DETAILS ON AUTOMATED ERROR ANALYSIS
The error candidate set & (ggis eig) Pr(Zis 1) 1} can have two situations: i) when all the judge trials consistently score O, ii) judge model has disagreement across multiple judge trials. For the first case, we can select any сi of the trials get the final æ4 Usually in our implementation,
select the first explanation et,y- However, for second case, the fiden will take all eij and apply another selective prompt function fselective to decide the low-level error x. We illustrate the entire algorithm in the below pseudo-code:
Algorithm 1: Automated Error Analysis Method
Input: Judge outputs {(2,9)
Output: High-level error types <ig 9s.3]
(g)
} for samples i, subgoals j. trials 4 1...Q.
<---STEP1: Low-level error identificatiin
Initialize & - 0, +0
for each ji do
if (vg, 7 7.g
0) or (0 Zi Al Z.,) then
E + E U (9ig. e(,))
if Vq. 2
then
// Consistent failure:
// Select the first judge explanation for error identification
Figj + futen (g o.)))
end else
11
// Disagreement across judge trials: // Iterate on all the judge explanation Tmp for to Q do Tmp T Tmp U (Siden(gipp 2
13
15
end Iijj f- fretoctive (Tmp)
19
end
X X U {mir)
end
end
<--- STEP2: Semantic clustering of error types--->
24 < [gt
(i,j) E E
25
C- /2t (X
26 return high-level error types
19

---

Published as a conference paper at ICLR 2026
Prompt Template for Jiden in Automated Error Analysis:
You tasked with summarizing the type in concise and abstract manner based the provided explanation. This explanation generated by a judge model, which evaluates whether the agent's response satisfies the specified subgoals. In the explanation, a grade "C" (Complete) indicates success, while "I" (Incomplete) indicates failure.
Your goal produce
type that:
Clearly captures core failure or issue at an abstract level.
Avoids restating the explanation verbatim.
Is short, specific, and phrased like category label rather than a long sentence.
Does not include sensitive details unnecessary context.
If the error type involves tool usage, explicitly include the tool name in the type.
You will be provided with: * Ground truth subgoals
Judge model's explanation of the agent's response
[BEGIN DATA]
[Ground Truth Subgoals] {subgoals}
[Explanation]
(expLanation)
[END DATA]
Please return your output in the following **strict JSON fo
"error_type" "<error_type>" "explamation"; "<explanation>"
20

---

Published as a conference paper at ICLR 2026
Prompt Template for selective in Automated Error Analysis:
You given multiple independent predictions of the data row. Each prediction includes error type assigned by model. Your task determine the **most probable true error type** using a majority voting approach.
Instructions:
Review all provided types carefully.
Group similar semantically equivalent error types together, even their wording differs. Count how many times cach grouped error type appears.
4. Select the error type with the highest count as the final result. If there tie:
Prefer the error type that
specifie and informative. still tied, choose the one most consistent with the majority wording.
Output ONLY the most probahle error type, without extra commentary.
[BEGIN DATAJ
[Error Types] ferror.type.list)
[END DATA]
Please return your output the following **strict ISON forma
"most_ "mott__robbble_crrroottt
"mmett_rrobbbe_erro___ee
21

---

Published as a conference paper at ICLR 2026
Prompt Template for Jelus in Automated Error Analysis:
You tasked with clustering the following error types based on their semantic similarity. The goal is group related error types under broader, more abstract categories reduce redundancy and improve generalization.
Important:
The cluster_labed should be primarily grounded in the <subgoa provided.
Only you are very certain that error type entirely unrelated to subgoals should you create a new cluster label not derived from them.
Always aim to preserve the subgoal's 'ss intent when naming clusters.
Guidelines:
Error types are not mutually exclusive and may overlap in meaning.
Fach cluster should reflect the most abstract and inclusive lahel that unifies all error types within it.
Do not merge error types referring to different tools into single cluster.
Clusters involving tool usage must be separated tool name.
The blusterlabe
each tool-related cluster must explicitly include that tool's name.
Minimize the number of clusters while maintaining clear and meaningful distinctions.
Avoid overly specific wording- cluster labels should be reusable in other contexts where the same subgoal applies.
[BEGIN DATA]
[Subgoals]
{subgoals}
#***
F Types]: (error.types)
[END DATA]
Please return your output the following strict JSON format:
"clusters":
"c "cluster celsteerlabee''; petneeralizedeerrrrrtco
"error_types": "<<error_type_>>"" "cerror "<error_type_2>> ...,, "error_ida"; ("<error_id_i>") "<error_id_>>> ...]
22

---

Published as a conference paper at ICLR 2026
A.7 HUMAN VALIDATION ON THE CORRECTNESS OF USER PROXY
To ensure the reliability of our user proxy simulation, we manually validate user proxy utterances through human evaluation on 16 randomly selected expert and 16 non-expert user-agent dialogues on 7-bench airline domain and ToolSa dataset. We categorize errors into three types: (1) user role confusion, where the user mistakes their role for the agent's; ss; (2) failure to follow the specified task instructions i E I, termed as missing or violate instructions; and (3) nonsensical or erroneous user responses. Based on Table g] we observe that the user proxy in general behaves as expected except for small number of cases where it does not follow the task instructions. While no AI system can be expected to achieve 100% accuracy, the low number of such errors supports our belief the user proxy's 's inherent potential for agent evaluation.
Table 3: Correctness of user proxy. Both agent and user proxy use gpt4.1 model.
Errors
User Persona
Role confusion
Missing or violate instructions
Erroneous responses
Expert
0.0
0.06
0.0
Non-expert
0.0
0.125
0.0
A.8 HUMAN VALIDATION ON THE CORRECTNESS OF LLM-AS-A-JUDGEE
To ensure the reliability of our a-a-uu---udge evaluation, which uses grading notes (i.e., subgoals) as ground truths, we conducted multiple runs of the judge and used majority vote to determine the final scores. The human validation results as shown in Tables to i are conducted on these majority-vote outcomes. Based on Table 4 for the gpt-5 agent model, we randomly select 10 sam- ples containing both expert and non-expert users from the r2-bench airline domain, and another 10 samples from the ToolSandbox dataset, resulting in 42 and 31 subgoals, respectively. Since each subgoal evaluated independently, we report human validation results at the subgoal level. Hu- "success", "failure", "ambiguous",
man annotators are asked to label each subgoal prediction where "ambiguous" denotes cases that could not be clearly classified as either "success" or "fail- ure". A disagreement with human is defined as mismatch between the human label and the judge's prediction ("success" vs. "failure"). 'Te "ambiguous" category refers to the subgoals prediction labeled as "ambiguous" by human annotators, regardless of the judge's prediction. In Table 4 we observe a slight disgreement between the LLM-as-a-judge and human judgment. Additionally, 6% of the ToolSandbox results are ambiguous even human evaluators, indicating that this not straightforward judgment task for the LLM. The human validation results for the gpt-4.1, gpt-4o, and gpt-4o-mint 4o-mini agents are shown in the Tables 5 and [ respectively. Given the low error rate and only occasional ambiguous samples, these findings reinforce that our approach using grading notes and a-aa--d-de still offers reliable, scalable, and cost-effective alternative other more complex evaluation methods.
Table 4: Correctness of LLM-as-a-judge. The agent uses gpt-5 model while the LLM-as-a-judge and the user proxy use gpt-4.1 model.
Per-subgoal error
Dataset
Disagreement with human
Ambiguous
>e-bench airline (42 subgoals from 10 samples)
0.07
0.0
ToolSandbox (31 subgoals from 10 samples)
0.03
0.06
Table 5: Correctness of LLM-as-a-judge. The agent, LLM-as-judge, and user proxy use gpl-4.1 model.
Per-subgoal error
Dataset
Disagreement with human
Ambiguous
2 bennh airline (37 subgoals from 9 samples) ToolSandbox (40 subgoals from 10 samples)
0.14
0.0
0.20
0.0
23

---

Published as a conference paper at ICLR 2026
Table 6: Correctness of LLM--L-- -uu-uee--judge. The agent uses gpt-4o model while the LLM-as-a-judge and the user proxy use gpt-4.1 model.
Per-subgoal error
Dataset
Disagreement with human
Ambiguous
7 bench airline (27 subgoals from 10 samples) ToolSandbox (32 subgoals from 10 samples)
0.04
0.0
0.09
0.03
Table 7: Correctness of LLM-as-a-judde. The agent uses gpt-4o-mini model while the LLM-as-a- judge and the user proxy use gpt-4.1 model.
Per-subgoal error
Dataset
Disagreement with human
Ambiguous
r--bench airline (30 subgoals from 10 samples)
0.03
0.0
ToolSandbox (35 subgoals from 10 samples)
0,03
0,06
A.9 HUMAN VALIDATION ON THE CORRECTNESS OF THE IDENTIFIED TED ERRORS
To ensure the relevance and accuracy of the errors identified by our TED framework, manually validate them against the reference labels. Likewise, as errors are identified based on subgoals, we report human validation results the subgoal level. Human annotators first identify and catego- rize eπrоrs using only the agent's trajectory and the ground-truth subgoals. These annotated errors serve as reference labels. To measure disagreement, they subsequently evaluate whether the TED- identified errors are semantically consistent with their own reference error annotations. For different agent models, we observe only small disagreement (6-23%) between the errors identified by TED and the reference errors. The detailed results are shown in the Tables 8 and 10 This level of disagreement reasonable, as eтror identification is challenging even for human annotators.
Table 8: Correctness the identified TED errors. The agent, LLM-as-a-judee and user proxy use the gpt-4./ model.
Per-subgoal error
Dataset
Disagreement with human
-eenhh airline (37 subgoals from samples)
0.22
ToolSandbox (40 subgoals from 10 samples)
0.23
Table Correctness the identified TED errors. The agent uses gpt-4o model while the LLM-as- a-judge and the user proxy use gpt-4.1 model.
Per-subgoal error
Dataset
Disagreement with human
7 bench airline (27 subgoals from 10 samples) ToolSandbox (32 subgoals from 10 samples)
0.19
0.09
Table 10: Correctness of the identified TED errors. The agent uses gpt-4o-mini model while the L-MM-as-a-judge and the user proxy use gpt-4.1 model.
Per-subgoal error
Dataset
Disagreement with human
7 bench airline (30 subgoals from 10 samples)
0.23
ToolSandbox (35 subgoals,fro 10 samples)
0.06
24

---

Published as a conference paper at ICLR 2026
A.10 ADDITIONAL RESULTS FOR AGENT IMPROVEMENT EXPERIMENTS
To demonstrate that the identified errors from our TED framework are meaningful and effective, examine several in-context learning approaches to incorporate the found errors into the agent instruction. This study aims to evaluate the usefulness of the errors identified by the TED framework, rather than proposing new in-context learning approach.
We explore simple strategies such as Human Notes (HN where TED errors are manually refined before using them in the agent instructions, and the Error Insertion (EI) approach where TED errors are directly used without modification. Moreover, to demonstrate the effectiveness of the TED errors with other in-context learning approach such as the HiTEC-ICL (abbreviated as HTC) method (Cuil er al. 2025), we conduct a comparison of the authors' generic global errors versus the variant with our TED errors, denoted as HTC in Table
The standard HTC method constructs global and local error checklists, which are then injected as metadata into the user instruction, simulating an artificial conversation loop to better guide the agent. The predefined global errors are highly generic, such Empty Parameter Value Error or Missing Required Parameter Error. After the first round of user prompting with the global error checklist, the user model is also prompted again in the second round with the local error checklist, enabling the agent to recognize local errors for the relevant tools. For the HTC variant, the generic global errors are replaced with TED errors, while the use of local error checklist remain consistent with the original method where detailed tool info are provided to the LLM to generate tool-specific error patterns.
Table 11: Agent improvement results over the baseline agent for Expert Persona Non-expert Per- sona, using gpt-4.1 1 as user proxy and LLM-as-a judge. Blue 1 shows an improvement over the baseline agent, and red 4. shows decrease. We compare the gain in performance for several agent models using the Error Insertion (EI), Human Notes (HN), HiTEC-ICL (HTC) (Cui et al.) 2025 and HTC variant methods.
Agent Model
McanPro@ak
Max
MorAUC@k
Max (PTPP
7 bench Airline Domain (Special Split: Samples 7, 14, 21, 23, and 29)
gpt-4o-mini + EI
10.99
0.31
0.6610.06
0.61 10.00
0.59
0.39 Jo.02 0.2770.03
0.11 jo.01
gpt-40--miii + HN
0.30
0.33
0.63
0.61
10.00
0.53
0.45 10.04 0.24
0.14
gpt-4o-mini + HTC
0.29
0.26.0.06
0.54
0,51
0.50
:0.00
0.32
0. 1..01
gpt-4o-mini + HTC|
0.33
0.33
0.61
0.65
0.57
0.43
0.29
0. 0.100
gpt-4.1 + EI
0.00
0.38.1o.00
0.78
0.77..0.08
0.661o.02
0.41 Jo.06
0.26.0.01
0. 10..0z
gpt-4.1 HN
0.53
0.41 10.03
0.78
0.85
1..00
0.6610.02
0.55 10.08
0.14 10.00
gpt-4.1 + HTC
0.54
0.39
0.02
0.73 1o.12
0.67
0.41 10.06
0.27=0000
0. 10.002
gpt-4.1 + HTC
0.55+0.03
0.40+o.02
0.71 10.03
0.
0.63 p..03
10.04
0.24
12::0.0000
ToolSandbox
Dataset
gpt-4o-mini + EI
0.87
0.89
0.97
0.98
0.94
0.91 0.85
0.66.0.11
gpt-4o-mini + HN 0.88.jom
0.91
0.96
0.96
0.95
0.92 0.90
0.74.j..0
gpt-4o-mini + HTC
0.93
0.92
0.97
0.98
0.961o.02
0.95 0.83 0.94
gpi-4o-mini + HTC!
0.92
0.00
0.98.0.0
0.98
0.97
0.95
0.9310.04
0.8210.05
gpt-4.1 EI
0.95
0.93
10.06
0.99
0.99
10.02
0.97
0.93 10.01
0..7
0.76 10.03
gpt-4.1 HN
0.95
0.97
10.10
0.98
10.00
0.99
10.02
0.97
0.95 10.03
0.91 10.07
0.83 10.10
gpt-4.1 + HTC
0.95
0.95
0.98
0.99
0.95 0.96
0.89
0.75
gpt-4.1 + HTCt
0.95
0.95
1.00
0.99
0.96 0.99
0.92
0.82
Based on Table 11 all four approaches show positive gains in agent performance for a majority of the metrics on ToolSandbox dataset. For the 72-bench special split, we observe mixed trends, with both positive and negative performance gains over the baseline agents among the four approaches. Specifically for gpt-4o-mini, we see there is a significantly larger performance gain >--bench for
'F.g., "You must strictly check and double confirm all requirements for change flight actions before calling the tool. If you are unsure confused, always ask clarifying questions the user."
25

---

Published as a conference paper at ICLR 2026
the HTCt variant using TED errors compared to the standard HTC method that uses predefined generic global errors. However, for gpt-4.1 on r"-bench samples, there is no clear trend suggesting that one performs better than the other. We want to emphasize that, our work only evaluates the usefulness of the TED errors-as as indicated by improvements over baseline agents (blue upward ar- rows)-and and does not propose new in-context learning method to construct better agent instruction prompts.
26

---

Published as a conference paper at ICLR 2026
A.11 ADDITIONAL EXPERIMENTS ON THE T² -B-BENCH DATASET
Besides analyzing the agent performance on the easy airline domain samples, we extend our analysis include hard samples as well.
Table 12 presents the performance of various agent models under easy and more challenging setting on the airline domain. We gain some insights into agent behavior by including hard samples.
First, we observe a general decline across all metrics, reflecting the increased difficulty of the sam- ples. MeanProg@h and pass@k scores drop substantially for all models, indicating that agents are less likely to achieve full task completion on harder samples. While Muz MaxProgre Progress Rate@kk remains relatively high for most models (e.g., gpt-4. gpt-5, mistral-nemo, for expert persona), this metric suggests that most agents can achieve near to completion progress at least one of the n trials.
When shift our focus MarAUC@k the model ranking changes noticeably (eg: gpt-5, mistral- nemo, mistral-large for expert persona). This shift highlights that while many models can achieve completion the difficult tasks, only a few do efficiently. This effect is pro- nounced for the non-expert persona: LLM agents using gpt-4.1 and gpt-4o models maintain high Max MroProgres @ate@k (0.96 and 0.88), but their AarAUC@k scores much lower (0.68 and 0.65).
Another interesting observation is that non-expert persona sometimes has higher MaæProgee MoaaProrressstte@i (e.g., gpt-5, gpt-4o-mini). Upon closer examination, we found in- stances where the expert persona provided all relevant information in the very first turn, which overwhelmed the agent and led to hallucinations. In contrast, the non-expert persona distributes information gradually over multiple turns, allowing the agent respond more effectively. This finding highlights the impact of user interaction style on agent performance.
Table Overall performance of different agent models on r2-bench airline domain, using gpt-4. as user proxy and ALM-as-a-judge. Dataset contains easy and hard samples. Results are displayed with scores for Expert Persona Non-expert Persona. For metries with @k, the number of trials
20.
Agent Model
MeanProg@k
Mar MarProgük MaxAUC&k MorPPTak
Pass@k
7 bench Airline Domain (Easy Hard)
gpt-4.1
0.75
0.67
0.94
0.96
0.85
0.68
0.44
0.25
0.81
0.86
gpt-4o
0.63
0.55
0.91
0.88
0.84
0.65
0.44
0.23
0.81
0.71
gpt-4o-mini
0.53
0.53
0.86
0.88
0.79
0.66
0.43
0.26
0.62
0.76
gpt-5
0.80
0.77
0.96
0.97
0.89
0.77
0.47
0.30
0.91
0.91
mistral-nemo
0.67
0.36
0.96
0.71
0.87
0.56
0.44
0.25
0.86
0.38
mistral-large
0.54
0.51
0.94
0.93
0.87
0.68
0.43
0.24
0.86
0.76
Table 13: Overall performance of different agent models on r2-bench retail domain, using gpt-4. user proxy and auudee. Results are displayed with scores for Expert Persona Non-expert Persona. For metrics with @k, the number of trials 20.
Agent Model
MeanProgoe
Max
MaxAUC@k MoxPPTak
Pass@k
72 bench Retail Domain
gpt-4.1
0.92
0.88
0.99
1.00
0.94
0.83
0.51
0.24
0.92
0.96
gpt-4o
0.87
0.75
1.00
0.99
0.95
0.83
0.54
0.25
1.00
0.96
gpt-4o-mini
0.81
0.76
1.00
0.98
0.95
0.82
0.54
0.25
1.00
0.88
mistral-nemo
0.57
0.58
0.99
0.97
0.86
0.79
0.29
0.23
0.88
0.84
mistral-large
0.80
0.81
1.00
1.00
0.94
0.83
0.49
0.23
0.96
1.00
In Table 13 we further evaluate our framework on the retail domain. In contrast to the airline domain, retail tasks appear more solvable, with most models achieving near-perfect
27

---

Published as a conference paper at ICLR 2026
MazProgress Rate@k. Ratekkk. However, the trend between the personas remains consistent, with expert personas consistently having higher Aar scores than the non-expert personas.
An interesting observation appears for agents with gpt-4o and gpt-4o-mini models in the non-expent user setting. While both models exhibit nearly identical (eeamProgressRaee@he MaxPrrgrrss Rate@k, R, MaxAUCGi and MaxPPT@k scores, the agent with gpt-4o-mini shows a lower passük score compared to gpt-4o. This suggests that gpt-4o-mini frequently reaches state of near-completion (i.e., M ozr Progress Rateek close to 1), but often fails complete the full task as compared to gpt-4o that has a higher passßk score. This further strengthened our justification for the need of fine-graine metries to quantify the agent's progression.
MaxAUC@k MaxProgressRate@k
MaxPPT@4 MaxProgressRate@k
1.0
0.8
0.8
MaxProgressRated
0.6
0.6
0.4
0.4
0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6 MaxPPT@k
0.8
1.0
MaxAUC@k
Figure 5: Scatter plot results on the bench airline domain (easy hard) illustrating the relation- ship between MazAU Ak and Max Progress /aates (left subplot), and between Max MarPPTT0 PT@k and Max Progress Rate@k (right subplot) using the setting n 20 trials.
28

---

Published as a conference paper at ICLR 2026
MaxProgressRRae@k
0.90
0:90
0.80
0.9
0.7
0.6
0.4
0.2
Expert
Non-Expert
gpt-4.1
gpt-4o
gpt-4o-mini
gpt-5
mistral-arge
mistral-nemo
MaxAUC@k
1.0
0.99
0.81
0.96
0.86
0,85 0.73
0.97 0.88
0.96 0.79
0.97 0.67
0.9
0.7
0.6
5core
0.5
0.4
0.2
Expert
Non-Expert
0.0
gpt-4.1
gpt-4o
gpt-4o-mini
gpt-5
mistrai-large
mistrai-nema
MaxPPT@k
1.0
0.80 0.50
0.70 0.53
0.67 0.54
0.67 0.48
0.60
0.60
0.42
0.37
0.8
0.7
Score
0.5
0.3
0.2
Expert
Non-Expert
0.0
gpt-4.1
gpt-4o
gpt-4o-mini
gpt-5
mistral-arge
mistral-nemo
Figure 6: Dataset level performance of different agent models on the > airline do- main (easy), with error bars representing 95% confidence intervals. The top graph shows the Max Paogrrrss Rate@k, middle graph shows AUCA bottom graph shows M aox MarPPT@.
29

---

Published as a conference paper at ICLR 2026
MaxPPogressRateeak
0.98 0.97
0.99-6.99 0.99
0.95
0:93
0.97 0.91
0:94 0:35
0.92 0:96
0.9
0.7
0.6
0.4
0.2
Expert
Non-Expert
0.0
gpt-4.1
gpt-4o
gpt-4o-mini
gpt-5
mistrai-large
mistral-nema
MaxAUC@k
1.0
0:96 0.92
0.98 0.96
0,94 0,90
0.95
0.93 0.90
0.88
0.9
0.8
0.7
0.5
0.4
0.3
0.2
Expert Non-Expert
0.0
gpt-4.1
gpt-4o
gpt-4o-mini
gpt-5
mistra Harge
mistral-nemo
MaxPPT@k
1.0
0.84
0.94
0.89
0.87
0.9
0.77
0.75
0.76
0.73
0.8
0.66
0.65
0.7
Score
0.5
0.3
0.2
Expert
Non-Expert
0.0
gpt-4.1
go
gpt-4o-mini
gpt-5
mistral-large
mistral-nema
Figure 7: Dataset level performance of different agent models on the ToolSandbox dataset, with error bars representing 95% confidence intervals. The top graph shows the Mar MozProgressRatelk middle graph shows M AUCAU bottom graph shows M MaxP PPT@k.
30

---

Published as a conference paper at ICLR 2026
MaxProgressRRteaak
0:94 0:96
0:91 0:38
0.86 0:88
0,96
0.97
0:94 0.93
0,96
0.9
0.71
0.7
0.6
0.4
0.2
Expert Non-Expert
gpt-4.1
gpt-4o
gpt-4o-mini
gpt-5
mistral-argee
mistral-nemo
MaxAUC@k
1.0
0.83
0.89
0.87
0.87
0.79
0.9
0.77
0.68
0.65
0.66
0.56
0.7
0.6
Score
0.5
0.4
0.2
Expert
Non-Expert
0.0
gpt-4.1
gpt-4o
gpt-4o-mini
gpt-5
mistral-large
mistrai-nema
MaxPPT@k
1.0
0.7
0.44
0.43
0.43
0.44
Score
0.5
0.29
0.23
0.26
0.25
0.24
0.3
0.2
Expert
Non-Expert
0.0
gpt-4.1
gpt-4o
gpt-4o-mini
gpt-5
mistral-arge
mistral-nemo
Figure 8: Dataset level performance of different agent models on the 72-bench airline domain (easythard), with error bars representing 95% confidence intervals. The top graph shows the Max Progeroorssstteee middle graph shows Aoz bottom graph shows M MTPkk
31

---

Published as a conference paper at ICLR 2026
MaxProgressRate@k
0.99
1.00
0.99
0.98
1.00
0.99
0.97
0.9
0.7
0.6
5000
0.4
0.2
Expert
Non-Expert
0.0
opt-4.1
9p-40
9pt-4o-mri
mistral- ange
MaxAUC@k
0:94
0:95
0,95
0.94
0.83
0.82
0.83
0.86
0.5
0.79
0.8
0.6
0.4
0.2
Export
0.0
qpt 4.1
3Pc 40
gct miri MaxPPT@k
mistral arge 2411
mistralimmoo
1.0
0.8
0.54
0.54
0.51
0.49
0.6-
0.4
0.24
0.25
0.25
0.23
0.29
0.23
0.2
Expert
get 4.1
300 40
get min
mistral arge 2431
mistoinnmoo
Figure 9: Dataset level performance of different agent models the r--bench retail domain, with error bars representing 95% confidence intervals. The top graph shows the M MorProgress Rate@k, middle graph shows Ma AUC bottom graph shows Mar PPT@k
32

---

Published as a conference paper at ICLR 2026
A.12 COMPARISON OF DIFFERENT AGENT MODELS USING THE EVALUATION PARADIGMS FROM THE ORIGINAL 7² BENCH AND TOOLSANDBOX PAPERS
In the following, we examined the evaluation paradigms from the original r2-bench (Barres all, 2025) and ToolSandbox (Lu et al. 2024) papers to explore their limitations and the potential advan- tages of our TED framework.
A.12.1 -ENCH ORIGINAL PAPER EVALUATION APPROACH
In the Tables 14 and 15 we evaluated the 72-bench agent on the airline and retail domains across different LLM models using the be-hh (Barres et al. 2025) original paper evaluation approach. For a fair comparison with our TED framework, the same set of samples and the same number of trials k 20 are used in the experiments.
Comparing Barres et al. 22025)ss 's approach (Table 14 left column) with our TED framework (Ta- ble [) on the easy split of the 72-bench airline domain, we observe that metries such as pass@k Yao et al., 2024) and pass"k " (Barres et al., 2025) saturate for several models in Table [4, where performances are clustered into two different groups, revealing minimal information on the perfor- mance ranking of the different models. On the pass"4 4 metric which is form of consistency or reliability measure, model gpt-4.1 performs best, followed by gpt-4o, gpt-40-mini, mistral- large, and finally mistral-nemo when evaluated using Barres et al. (2025)'s original approach on the airline easy split. Comparing this performance ranking with TED-evallated results Table] we observe change in rankings for several models. For instance, gpt-4o-mini on some metrics and setups in Ta- ble performs worse than mistral-nemo which contradicts with the ranking in Table [4 In Table mistral-nemo performs better than gpt-4o-mini across the different metrics for the expert For the non-expert user setup, we observe that mistral-nemo performs worse than gpt-4o-mini on the AlaxAUC@L metric (0.67 vs 0.73) but better on the MoxP PPPPPT@ &k metric (0.48 vs 0.37). This suggests that under the non-expert user interaction, the agent with gpt-4o-mini exhibits a steeper per- formance improvement in the early interaction turns before plateauing (reflected by higher AUC), but a lower average performance gain (indicated by a lower PPT) compared to mistral- -emo. This finding highlights the importance of systematically analyzing agent's performance not only based on the underlying LLM models but also based on the conversation quality between user and agent as determined by the user expertise. Additionally, we see the benefit of joint interpretation of multiple metrics (AUC' and PPT) for a better understanding of the progression of agent performance which existing coarse-grained metrics like pass@k and pass"k k fail to provide.
setup.
Table 14: Comparison of different agent models using the evaluation paradigm from the 72 22-bench (Barres et al., (025) original paper and their performances on the passōtk, pass" 44, and pass" k: metrics in the airline domain. The number of trials used is n k 20.
Agent Model
-2-bench Airline Domain (Easy)
R-bbnch Airline Domain (Easy + Hard)
pass@k
pass'4
pass"k k
pass@k
pass'4
pass"k
gpt-4.1
1.00
0.60
0.20
0.76
0.29
0.10
gpt-4o
0.80
0.40
0.20
0.57
0.13
0.05
gpt-4o-mini
1.00
0.28
0.20
0.67
0.10
0.05
mistral-nemo
0.80
0.06
0.00
0.43
0.01
0.00
mistral-large 1.00
0.16
0.00
0.71
0.07
0.00
Similarly, a change in model rankings can be observed for the >2-bench agent on the full airline (easy+hard) and retail domains. In Table [4] (right column), the best performing model on the full airline domain is gpt-4. 1 but this is not true for T'ED-evaluated results Table 12 where the mistral models when interacting with the expert user generally show better on par performance with the gpt-4.1 model on the M AU C'Ok, MarPPT@k and pass@k metrics. A similar behavior can be observed for the mistral-nemo and gpt-4o-mini models. As shown in the right column of Table [4] the agent using gpt-4o-m minm outperforms mistral-nemo, This finding is consistent with our results for the non non-expert user persona setting in Table [2] but it does not align with the expert user setting, where we observe a reversal in the relative ranking, with mistral- outperforming gpt-4o-minn minti..iii.. We further illustrate this with an example in Fig. 10a and 10b by plotting the metric values for the
33

---

Published as a conference paper at ICLR 2026
Table 15: Comparison of different agent models using the evaluation paradigm from the r²-bench [Barres et al. 2025) original paper and their performances the passak, pass"4, and pass' k metries in the retail domain. The number trials used n = 20.
Agent Model
72 bench Retail Domain
pass@k
pass 4
pass"k
gpt-4.1
1.00
0.54
0.20
gpt-4o
1.00
0.43
0.04
gpt-4o-mini
0.92
0.23
0.04
mistral-nemo
0.76
0.08
0.00
mistral-large 0.96
0.33
0.08
two LLM models given sample from the airline domain. Based Fig. 10a the agent using mistral-nemo model, when interacting with an expert user, it achieves the maximum progress value of in the first turn, resulting in AUC 1 which surpasses the performance of gpt-4o-mini (i.e., AUC = 0.982) that completes all the defined subgoals only in the second turn. On the other hand, Barres al. (2025) 's original evaluation approach in Fig. [Obb] aligns with results on the non-expert user setting where the gpt-4o-mini model performs better than mistral- nemo model (AUC 0.714 AUC 0.4892). This further justify the importance of considering the user-agent conversation quality during evaluation.
we observe that
Likewise when comparing results on the retail domain, the model gpt-4.1 using the evaluation paradigm from the 7 r"-bench (Barres et al. 2025) original paper appears to be the best performing model based on Table [5] but is not necessarily the case when comparing with the results obtained from our TED framework in Table [3] These observations show the significance of reporting mul- tiple metrics, e.g., AUC and PPT. to fully understand the task progression the agent and the impact of user's 's role on the agent performance.
progreset G.t)) progresll
1.0
mistral-nemo gpt-4o-mini
0.5
mistral-nemo-exoert got-4o-mini-expert
Metric Value
0.8
0.0
0.6
0.4
mistral-nemo-nnexxpae
gpt-4o-mini-nonexpprt
0.2
0.15
0.5
0.0
0.0
0.0 0.0
0.0
0.0
3
pass@k
pass
pass
pass"K
Turn
Metric Name
(a)
(b)
Figure 10: (a) be-hench airline sample.3 8: mistral-nemo expert (AUC=1.0, PPT=1.0) vs. gpt- 4o-mini expert (AUC= PPT=0.5) and mistral non-expert (AUC=0.482, PPT=0.25) gpt-4o-mini non-expert (AUC=0.714, PPT=0.125) (b) ,2-bench airline samp sample.81 8: Barres et al. 2025 's original evaluation approach with l 20 trials.
A.12.2 TOOLSANDBOX ORIGINAL PAPER EVALUATION APPROACH
We evaluated the ToolSandbox agent across different L.M models using the ToolSandbox (Lu al., 2024) original paper evaluation approach and report the models' performance in Table [6] For fair comparison with our TED framework Table [ the same set samples and the same number of trials A 8 are used in the experiments. We do not report the final graph similarity score, which a composite setup. Instead, consider only the milestone similarity score ScoreM+ and report the best performance of the model by taking the maximum milestone similarity score across the trials as denoted by Max:Soor, ak.
milestone and minefield scores, because minefields are not considered in
34

---

Published as a conference paper at ICLR 2026
Table 16: Comparison of different agent models using the evaluation paradigm from the ToolSand- box (Lu et al. 2024) original paper and their performances on the milestone similarity Scorea+ metric. The number of trials used n
Agent Model
ToolSandbox Dataset
(Base Scenarios)
(Base Scenarios + Variants)
M MoxScoreas Ok
Max:ccoreM Ak
gpt-4.1
0.92
0.90
gpt-4o
0.94
0.91
gpt-4o-mini
0.90
0.88
mistral-nemo
0.87
0.85
mistral-large 0.91
0.89
In Table [6 we present the performance of different agent models on base scenarios (left column), well both base scenarios and variants (right column). Based the table, we observe that both setups exhibit the same performance ranking across models, with gpt-4o achieving the best per- formance, followed by gpt-4.1, and mistral- nimoral-nemo performing the worst. However, this performance ordering differs from our main observation in Table I that is obtained using the TED framework.
Our TED-based evaluation on the ToolSandbox dataset Table [ also shows gpt-4o outperforming gpt-4.1 across different metrics and user personas, but reveals different performance ranking be- tween mistral-nemo and gpt-4o-mini. Based on Table [T] mistral-nemo outperforms gpt-4o-mini on the pass@k metrics for the non-expert user persona setting (0.92 vs 0.84) and vice-versa (0.84 vs 0.89) for expert user. For these two models, only expert user setting Table aliigs with the ranking observed in Table 16 This finding suggests that et al. (2024) 's evaluation paradigm and the Score M+ metric Table 16 may not always paint a complete story, as they do not effectively disentangle user behavior from the evaluation process. These observations highlight the importance of agent evaluation frameworks that account for user behavior, and demonstrate the additional values provided by our TED framework in which existing evaluation frameworks lacking.
35

---

Published as a conference paper at ICLR 2026
A.13 ABLATION STUDY ON THE DIFFERENT USER MODELS
Besides varying the model for agents, we also conducted an ablation study by varying the user models across the gpt family shown in Table 17 We observe similar trend emerges in earlier analyses: the gap across traditional metrics becomes narrower, and the agent performance difference between expert and non-expert user personas is relatively small. This further highlights the impor- lance of our proposed metrics in capturing agent behavior with respect to turns which is beyond what conventional metrics do. expected, the agent interacting with expert user achieves consis- tently higher performance on Mar WfrrAUCii and MaxPPT@A metrics, confirming our hypothesis that agent can resolve tasks more efficiently as expert users tend to understand the system well and provide complete information for the agent. Interestingly, if we use stronger models such gpt-5 for the user proxy, we see a smaller gap between expert and non-expert personas. This suggests that the model capability of the user proxy improves, the model proxy with non-expert persona behaves more like an expert. and achieves performance closer to expert-level outcomes. Based on the current observation, helieve that varying the user model also changes the expertise level, which potentially simulates different user expertise levels.
Overall performance of a gpt-4.1 agent with different user proxy models on the -2 -e-ch airline domain, using gpt-4.1 model as LLLM-as-a-judee Dataset contains easy and hard samples. Results are displayed with scores for Expert Persona Non-expert Persona. For metrics with @k, 20.
Table 17:
number of trials
User Model
Mean (loanPro@kk
MaxProgak
MazAUC@k MaxPPT@k
pass@k
72- Airline Domain (Easy + Hard)
gpt-4.1
0.75
0.67
0.94
0.96
0.85
0.68
0.44
0.25
0.81
0.86
gpt-4o
0.72
0.61
0.95
0.95
0.82
0.66
0.36
0.22
0.81
0.81
gpt-4o-mini
0.73
0.64
0.95
0.95
0.80
0.65
0.35
0.22
0.86
0.86
gpt-5
0.71
0.73
0.92
0.95
0.85
0.83
0.46
0.37
0.76
0.86
36

---

Published as a conference paper at ICLR 2026
A.14 ADDITIONAL INFORMATION ON DATASET AND EXPERIMENTAL SETUP
---bench. For bench, we split the samples in the airline domain into "easy" and "hard" subsets using the pass"k metric, with k 4. We consider samples that always completed in all independent runs as "easy".
The original samples are annotated with tool signatures and natural language assertions. Since these assertions closely align with our grading notes, we use them for evaluation.
ToolSandbox. For ToolSandbox, we do not split the samples as we consider them to be easy samples. We use gpt-4.1 I to convert the milestones into grading notes. Fach data sample scenario consists of set of milestone M = mi, m2, Mn. and may include directed acyclic graph (DAG) of dependencies E {i,j)), where each edge (i. indicates that milestone my depends on m4- The conversion process extracts key information from each milestone, such as required tool calls, expected agent-to-user communications, and ground truth state changes, and assembles this with DAG structure into structured prompt. The prompt template used to produce actionable grading notes, and expresses dependencies using connectors like "before" or "after".
ToolSandhox dataset contains multiple variations of the same scenario for example, the base sce- nario, find s_illl________ holiday has variants like aalddaayyllllliiiiii input message) and find daalssilll_aa..ooo
(which starts with an alternate hiaysssill_hoiiee holiday. mooiday-multipple muultiple Jurerturn (which intentionally provides less in- formation force a multi-turn conversation). These variations serve as crude simulation of user expertise, and their grading notes do not differ significantly from the base scenario. Since we have our own generic user persona templates and these variants share similar milestones, we only use the base scenario (eg., find days till holiday) and ignore the variants. After this process, we manually reviewed and refined the dataset to ensure that the generated grading notes were correct and mean- ingful. Our setup offers greater variability than the original versions with fixed initial messages.
The prompt template used for each scenario is as follows:
37

---

Published as a conference paper at ICLR 2026
Prompt Template Creating Grading Notes section 4:
You creating grading notes for agent evaluation that include dependency relationships. Convert milestones into concise statements about what the agent should accomplish, including any sequence requirements.
SCENARIO: {s cenaari name) DESCRIPTION: eecenario_descripttte TOTAL MILESTONES: (total_mm [motallleeestorss]]
MILESTONE DEPENDENCIES (DAG edges): {mi amilestonns one wee.eeetteeeeeeeeeeeeeeeee. edge.list)
DEPENDENCY ANALYSIS: (human readable description of which milestone dependency
MILESTONES:
Milestone {details including constraint type)
Milestone 1 (details including constraint type)...
RULES:
1. Create MORE natural language subgoals per milestones needed capture all required
actions
Mention specific tool names when relevant: "Agent should call tool_name"
3. Use natural language to describe the purpose: "Agent should call search contaces to find Homer's information"
Include sequence requirements when dependencies exist: "before", "after", "then", "first"
Break down complex milestones into multiple subgoals if needed
Use format: "Agent should [natural action description]"
7. Focus what needs to be accomplished, be specific and actionable
EXAMPLES GOOD NATURAL LANGUAGE GRADING NOTES:
"Agent should call set fi status
off wifi" "Agent should inform the user that wifi turned off"
"Agent should enable cellular service before sending message"
"Agent should enable cellular service before sending message"
"Agent should update contact phone number after finding the contact"
SPECIAL HANDLING FOR COMMUNICATION MILESTONES:
If target_data has SeENeT=AGENT and recRpient-USER with content, the grading note should be: "Agent should inform/tell the [content]"
If target data sender-EXECUTION ENVIRONMENT and recipient=AGENT with tool trace,
focus on the tool call requirement
Focus on what the agent needs to DO COMMUNICATE, not technical database states
CONSTRAINT TYPES:
snapshot similarity: Agent should achieve the target state addition similarity: Agent should add/create the target data
removal.similaree Agent should remove/delete the target data update_similane similarity; Agent should modify/update the target data
RESPONSE FORMAT:
Return a JSON array where each element can be either a single string or an array of strings for that milestone: (json_schema)
Each milestone can have multiple grading notes subgoals. Include dependency relationships when they exist.
38

---

Published as a conference paper at ICLR 2026
Additionally, we provide examples of the generated grading notes for ToolSandbox Lu et al. (2024) below. The full dataset is released together with our code.
Sample: modify contact www.ei/esssssseeeeeeeeeeee
Agent should call get.current..immsteeee
retrieve the current time
Agent should call search.contacts to find the contact information
Agent should call search.message after getting the current timestamp to find the last person the user sent a message
Agent should update the contact's phone number +10293847563 after iden- tifying the person is Homer S.
Agent should inform the user: "The phone number of the person you last talked to has been updated + +10293847563' after updating the contact
Figure 11: Example of generated grading ify contact with message recency.
notes for the ToolSandbox sample mod-
Sample: update contact relationship with relationship twice multiple user turn
Agent should call search.coontacss to find contacts with the relationship 'riend'.
Agent should call modify contact to update Fredrik Thordendal's rela- tionship "enemy' after finding the contact.
Agent should call modi fy ry_contact update John Petrucci's relationship to 'enemy' after finding the contact.
Agent should inform the user: 'Fredrik Thordendal and John Petrucci are now your enemies.
Agent should again call mod fy-contact to update Fredrik Thordendal's and John Petrucci relationship from 'enemy" to 'friend' again.
Example generated grading notes for the ToolSandbox sample up- date contact relationship with relationship twice multiple user furn.
Figure 12:
Sample: find current city low battery mode
Agent should ensure low battery mode is disabled
Agent should enable WiFi
Agent should enable WiFi after ensuring low battery mode disabled
Agent should enable location services
Agent should enable location services after ensuring low battery mode dis- abled
Agent should call get.current.
to retrieve the user's location
Agent should inform the user: You are currently in Cupertino
Figure 13:
Example of generated grading
notes for the ToolSandbox
sample
find current city low battery mode.
39

---

Published as a conference paper at ICLR 2026
Sample: convert.currency
Agent should call convert currency with the arguments: amount=2048, from currency Code='USD'' to eurrency code='CNY"
Figure 14: Example of generated grading notes for the ToolSandbox sample convert_curreet
Sample: search wwwsamessseeaaae with recency oldest
Agent should call get.current..tmmsttam to retrieve the current times- tamp
Agent should call search.messages to find the oldest message
Agent should call search after getting the current timestamp
Agent should inform the user: Your oldest message says 'Hey kid, you want some GPU?".
Figure 15: Example of generated grading notes for the ToolSandbox sample wwthsmeeessseet.eeeee .oldsst.
Sample: search reminder with recency yesterday
Agent should call get..oureent...mmsttan to retrieve the current date and time
Agent should call search-reminder to find reminders for the day before the retrieved timestamp
Agent should call search.reminder. after getting the current timestamp
Agent should inform the user: Your reminder from yesterday says 'Look for Company SF tickets'.
Figure 16:
Example of generated grading
notes for the ToolSandbox sample
search reminder with recency yesterday.
40

---

Published as a conference paper at ICLR 2026
Sample:
yesterday
Answer Al assistant's questions given the following task you (LL.Msimuulted user) want AI assistant to complete: Ask Al assistant to find the content of your (LLM- simulated user's) reminder from yesterday. should say Look for Company SF tickets. Do not leak this information. You do not have any more information.
When Al assistant completed the task, even if you (LLM-simulated user) don't have enough information to validate the correctness, break out of the simulation and end the conversation using the termination message.
Start your first message as: "What's s on my reminder yesterday?"
Sample: send menddmessggee.ee wwtpssaggewwith...m phone hhtppphooouuume number _and__antdntt
Answer AI assistant's questions given the following task you (LLM-simulated user) want Al assistant to complete: Send a message to +12453344098 saying: How's the new album coming along. You do not have more information.Al assistant completed the task, even if you (LLM-simulated user) don't t have enough information to validate the correctness, break out of the simulation and end the conversation using the termination message.
Figure 17: Example of user task instructions for the ToolSandbox samples
Sample:
Domain: airline
Reason for Call: You had a mixup with your assistant and booked multiple flights for the same day.
Known Information: You are Sophia Martin.
Your user id is sophia cat.ilamaatinn44551 4574..
Task Instructions: You want to first check if there are cases like this in your profile. You want the agent fix the situation for you. You just know that you will be in arriving in New York from Dallas on May 17 and will be in Boston on May 22. You want to let the agent figure out which flights should be cancelled. If the agent asks, you might have reservations for other passengers than yourself but you don't 't want to modify those.
The task is considered complete the instruction goal satisfied or you are transferred to another agent or you find yourself in situation in which the scenario does not provide enough information for you to continue the conversation.
Sample:
Domain: airline
Reason for Call: You just faced some money issue and want downgrade all business flights to economy, without changing the flights or passengers.
Known Information:
Your name is Omar Davis.
Your user id is omar.davis.361.
Task Instructions: You are fine with refunding original payment for each reservation. You want to know how much money you have saved in total. You are emotional and bit angry, but you are willing to cooperate with the agent. The task is considered complete if the instruction goal is satisfied you are transferred to another agent or you find yourself in a situation in which the scenario does not provide enough information for you continue the conversation.
Figure 18: Example of user task instructions for the 2 -ench airline samples
41

---

Published as a conference paper at ICLR 2026
A.15 ADDITIONAL DETAILS ON AGENT TRAJECTORIES
Task Instruction
Grading Notes
your M.--mulultted userks upcoming reminder today. should say Buy a nice rich navy bathing dress. IMPORTANT: Do NOT leak this in formation...
Agent call get current timestamp
Agent should call search, remande to find upcoming reminders Agent should call search, rensnder after retreving the curvent timestamp nany bathing dress
Agent inform user: Day
GPT-5 Dialogue Excerpt;
Mistral Nemo Dialogue Excerpt:
Tool Calls you check what my reminder for today!
3 Toal Calls Mude check what reminder for today?! think
LierProay:
UserProsy:
soe
something totally
TauZAgent: What time zone are you in?
InuZAgent coulde't find any reminders that you have set for today, still balieve have reminder, it possible that was set for yesterday tumoerow.
Do want all reminders schediled for today, just nest upooering
UserProgy: Ub. I'm Eastern time, I think. Just, like, whatever's next for
UserProx:: Ub. yealt, maybe be check for tomerruww then? I'm not really sure Inu2Agent coulde't any
Lau2Agent:
soday
Eastem time (cunently UTCC-), pull your reminders, need the idemifier remindess profile.
you still belleve you have reminder, it poesiblle that was ser for yesterday in the future.
PerrPray:
don't really remember email ised, worry think the reminder about buying something" Nox sure what time though. Does that hulp?
UserPraxy Hmm, guess just check any reminders fisture then? Mayle set midom day.
Tau2Awent I found a reminder that go off the future. [eleee
Tau2Agent: and lound this next one:
helps. searched "Bay mee rich navy bothing dress"...
time.
Reminder ID: 51.20be9-cod sac3-de76 Se57. 724e18d34177
Cansent: Buy nice rich navy bathing dress...
Agent dialogue excerpt for gpt-5 and Mistral-Nemo, illustrating how Mistral-Nemo makes tool calls early and quickly satisfies the grading notes, while gpt-5 asks clarifying questions and progresses more gradually. Despite these differing strategies, both agents complete the task in the same number of turns.
Figure 19:
42


Reviews:
[{"review_id": "fdREbM4oh1", "reviewer": "Reviewer_gKqD", "summary": "The authors look at the problem domain of agent evaluation. They argue that existing evaluation frameworks do not take into account a variety of users and that the metrics are too domain specific and are not flexible enough. Therefore the authors propose a framework called TED which they sample personas of \"expert\" and \"non-expert\" users and have a conversation with the agent where the agent has to achieve a series of sub-goals or trajectory. They evaluate is a sub-goal is completed by using LLM-as-a-Judge. Afterwards they have a suite of metrics to represent the trajectory in a numerical value which helps to better categorize if an agent is performing well or not. Additionally they have an automated error analysis method that uses the outputs from their LLM-as-a-Judge to cluster incorrect responses from the agent. They then show that one can come up with a prompt that addresses these errors to feed into the system.", "questions": "Questions\n\n1) You mention that for ToolSandbox you exclude the variants. I understand that these can be simulated by your user proxy but do the variants offer more diverse conversational paths that can be leveraged?\n\nSuggestions\n\n1) For Figure 2 I recommend having a legend or mentioning which colored lines go with which model. It was hard to decipher at first.\n\n2) I suggest you give a brief description of the datasets before getting into detail.\n\n3) I may have missed it but I don't see what LLM you are using for grading. I would make it more clear as to which model you are using for different parts of your pipeline.\n\n4) I like that you have human studies to evaluate user proxy / grading notes. I know that it's in the Appendix but I would give a brief summary of your results in the main paper.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1) The general method described is clear and the authors seem to take a principled approach to designing their metrics.\n\n2) It is interesting to see metrics like meanProg give more details into LLM performance when it comes to agents and uncovering those errors are important.\n\n3) It was good to see the authors use the errors discovered to improve the agents. This leaves future work on how to better incorporate the learnings from their error analysis.", "weaknesses": "1) Both datasets are in the task-oriented domain. Given that these metrics seem to be domain agnostic it would be good to see how these can be translated to other domains.\n\n2) One thing that still isn't clear to me is that while I understand that these metrics can give more insight to an agent are we able to find some correlation between user experience and these metrics? When it comes to evaluation metrics most work show some correlation between the two.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qe9cNKD7Rb", "reviewer": "Reviewer_N12X", "summary": "This work addresses the challenge of agent evaluation automation, where explicit references are difficult to acquire and apply (due to the stochastic nature of agent responses). The authors critique existing evaluation frameworks for lacking both structural clarity—specifically, the conflation of task instructions with user personas—and deeper behavioral insights, relying mainly on surface-level metrics. To address these gaps, the authors propose the TED framework (Talk, Evaluate, Diagnose), which introduces structured user-aware simulations, natural language subgoal grading with LLM-as-a-judge, and automated error diagnosis. Experimental results suggest that the diagnostic feedback produced by TED can lead to measurable improvements in agent performance under the proposed evaluation metrics. While I find the motivation and direction promising, I have concerns regarding the soundness and presentation rigor of the proposed methodology, as detailed in the Weaknesses and Questions sections.", "questions": "1. For Sec.3.2, what do the authors care about multi-turn progression? Aren't the turns independent trials s.t. their orders do not matter? What exactly does the turn-level progression metric tell/reveal?\n2. Why is it necessary to separate user persona and task? If this is the assumption of this work, could the authors clarify the significance with experiments? Essentially answering what's the consequence of NOT decoupling them?\n3. Eqn 5: undefined $success_l$.\n4. The notation of progressive max $max^k_{l=1}$ is somewhat strange and confusing to me, what operation(s) are taken place there? I roughly guess it's somethihng like a \"cumulative max\", but is it exactly? I would advise the authors to clarity for certain, and try to use clearer notations.\n5. Eqn 6: $p(t)$ is contiuous yet for multi-step, steps, are discrete. How would one compute AUC, I'd guess interpolations, but the authors should state clearly what have been done in the paper.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "The paper tackles a timely and important challenge in evaluating conversational or task-oriented agents, where the stochastic and open-ended nature of responses makes it difficult to define reliable ground-truths. I particularly appreciate the exploration of subgoal decomposition and the use of grading notes, which offer a valuable and interpretable approach to fine-grained agent evaluation.", "weaknesses": "My main concerns lie in the soundness and clarity of the paper. In particular, Section 3.2—one of the core components—raises questions about methodological soundness, which in turn affects my confidence in the validity of the experimental design. Beyond this, the motivations behind several key design choices are insufficiently explained, and multiple mathematical notations are either ambiguous or undefined. Further details are provided in the Questions/Concerns section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lPfSLr4sVg", "reviewer": "Reviewer_EiT3", "summary": "This paper introduces a framework for evaluating agents, referred to as the talk-evaluate-diagnose (TED) framework. The framework operates in three stages: talk is about collecting interaction data, evaluate is about automatic evaluation, and diagnose is about making errors human interpretable. As part of the evaluation phase the authors propose a MaxProgressRate@K metric that accounts for progress rather than just overall success (i.e., uses a more fine-grained reward). The authors demonstrate the use of their framework through examples. They observe, in one case, improvements of agent performance by 5%/9% (depending on metric) following changes based on error suggestions generated by the framework.", "questions": "1. How come you decided against releasing the code at the time of submission?\n2. Am I missing an important aspect about the evaluation, that may convince me that there is more than anecdotal evidence of utility?\n3. What are the concrete reasons why the setup needs to be introduced in a single paper? How do steps 2 and 3 necessarily depend on Step 1?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The general sentiment that superficial metrics are insufficient to allow practitioners to build better agents seems well justified.\n2. The proposed framework seems sensible, and follows common practices in building agent benchmarks (i.e. look at the data).", "weaknesses": "1. The writing of this paper could be clearer, parts of the paper can be difficult to follow. In particular the results section could benefit from clearer structuring, e.g. numbering and clearly separating individual insights/observations.\n2. The motivation for the combined three-part approach is unclear to me. Whilst each part on its own seems sensible enough it's unclear to me why the parts need to be introduced together. It's not clear that one part really depends on the other. This disconnect in motivation, in part, makes the paper difficult to follow. It's almost like reading three papers.\n   - Perhaps a paper just focusing on one of the aspects would be easier to follow, and would allow for a more rigorous analysis of that part.\n2. No code or data available at time of submission: since one of the contributions is a benchmark, this lack of code prevents me from fully assessing this part of the work.\n3. Limited evaluation of the tool itself. The reported gains (L461) seem anecdotal, based on improving a single agent based on a single use of their tool, effectively a single datapoint. It is unclear if these results generalise. Potential areas for improvement:\n\t- Evaluation could be improved by focusing more the number of errors found that are missed by other methods\n\t- I realise human studies can be very resource-intense, but a robust study of usage of TED vs competitor tools would be the most convincing.\n\nMinor (no impact on score, no need to respond):\n1. L70 onwards: Is this a proper enumerate environment?\n2. Figure 1: This figure is confusing to follow. Why do the error numbers descend (start 6 going down). How are the arrows to be interpreted? Is Error 5 and 6 on the same datapoint? What are the red rectangles supposed to mean?\n3. Figure 4: Where is error 4? It's in the list but not shown. Also the text is very small, maybe this could be condensed into something easier to read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)