AI Agents That Matter
Authors: Sayash Kapoor
Benedikt Stroebl
Zachary S Siegel
Nitya Nadgir
Arvind Narayanan
Venue: Accepted by TMLR ()

Abstract: AI agents are an exciting new research direction, and agent development is driven by benchmarks. Our analysis of current agent benchmarks and evaluation practices reveals several shortcomings that hinder their usefulness in real-world applications. First, there is a narrow focus on accuracy without attention to other metrics. As a result, SOTA agents are needlessly complex and costly, and the community has reached mistaken conclusions about the sources of accuracy gains. Our focus on cost in addition to accuracy motivates the new goal of jointly optimizing the two metrics. We design and implement one such optimization, showing its potential to greatly reduce cost while maintaining accuracy. Second, the benchmarking needs of model and downstream developers have been conflated, making it hard to identify which agent would be best suited for a particular application. Third, many agent benchmarks have inadequate holdout sets, and sometimes none at all. This has led to agents that are fragile because they take shortcuts and overfit to the benchmark in various ways. We prescribe a principled framework for avoiding overfitting. Finally, there is a lack of standardization in evaluation practices, leading to a pervasive lack of reproducibility. We hope that the steps we introduce for addressing these shortcomings will spur the development of agents that are useful in the real world and not just accurate on benchmarks.

Published in Transactions on Machine Learning Research (05/2025)
Al Agents That Matter
Sayash Kapoor
sayashik@ppinneeton..dd
Deportment of Computer Science Center for Information Technology Policy Princeton University
Benedikt Strocbl
strochl@@rricceton.eth
Center for Information Technology Policy Princeton University
Zachary S. Siegel
Department of Computer Science Center for Information Technology Policy Princeton University
Nitya Nadgir
Center for Information Technology Policy
Princeton University
Arvind Narayanan
anvindn@cs.princettr
Department of Computer Science Center for Information Technology Policy Princeton University
Reviewed on OpenReview: https: opt.revieerr..e....mmmmm net/ forum? id= 2y4uFaMuiZ
Abstract
AI agents are an exciting new research direction, and agent development is driven by benchmarks. Our analysis of current agent benchmarks and evaluation practices reveals several shortcomings that hinder their usefulness in real-world applications. First, there is narrow focus accuracy without attention to other metrics. As result, SOTA agents are needlessly complex and costly, and the community has reached mistaken conclusions about the sources of accuracy gains. Our focus on cost in addition to accuracy motivates the new goal of jointly optimizing the two metrics. We design and implement. one such optimization, showing its potential to greatly reduce cost while maintaining accuracy. Second, the benchmarking needs of model and downstream developers have been conflated, making hard identify which agent would be best suited for a particular application. Third, many agent benchmarks have inadequate holdout sets, and sometimes none all. This has led to agents that are fragile because they take shorteuts and overfit to the benchmark in various ways. We prescribe principled framework for avoiding overfitting. Finally, there is a lack of standardization in evaluation practices, leading to pervasive lack of reprodueibility. We hope that the steps we introduce for addressing these shortcomings will spur the development of agents that are useful in the real world and not just accurate on benchmarks.
1 Introduction
Compound Al systems, or AI agents, are becoming an important research direction. Zaharia et al. (004 argue that "compound AI systems will likely be the best way maximize AI results in the future, and might be one of the most impactful trends in AI in 2024" Over a dozen agent benchmarks have been released, spanning domains such web interaction (Zhou al. 2024), programming (Jimenez et. al. 2023 and tool

---

Published in Transactions on Machine Learning Research (05/2025)
use (Ruan et al. 2024). Many benchmarks developed for LLM evaluation have also been used for agent evaluation.
Agent evaluation differs from language model evaluation in fundamental ways. Agents can be used on tasks that are harder, more realistic, have more real-world utility, and usually don't have a single correct answer. For example, agents can use the command line to carry out tasks; SWE-Agent even includes its own agent-computer interface (Yang et al. 2024). Agents can cost much more than a single model call. For example, the authors of SWE-Agent capped each run of the agent at $4 USD, which translates to hundreds of thousands of language model tokens.
As a result, agent benchmarking comes with distinct challenges. This paper empirically demonstrates these challenges and provides recommendations for addressing them. Specifically, we make five contributions.
1. AI agent evaluations must. be cost-controlled (Section 2). The language models underlying most. Al agents are stochastic. This means simply calling the underlying model multiple times can increase accuracy (Li ct al. 2022 Chen et al. 2024 Li et al. 2024). We introduce three new simple basclinc agents and empirically show that they outperform many SoTA complex agent architectures on HumanEval (Zhong et al. 2024 Zhou et al. 2023 Shinn et al. 2023) while costing much less. Therefore, agent evaluations must be cost-controlled; otherwise it will encourage researchers to develop extremely costly agents just to claim they topped the leaderboard.
2. Jointly optimizing accuracy and cost can yield better agent design (Section 3 Visualizing
evaluation results as a Pareto curve of accuracy and inference cost opens up new space of agent design: jointly optimizing the two metrics. We modify the DSPy framework (Khattab et al. 2023 for joint optimization, lowering cost while maintaining accuracy on HolPolQA (Yang el al.] 2018
3. Model developers and downstream developers have distinct benchmarking needs (Section Through case study of NovelQA (Wang al., 2024a), we show how benchmarks meant for model evaluation can be mislcading when used for downstream evaluation. We argue that downstream evaluation should account for dollar costs, rather than proxies for cost such as the nunber of model parameters.
4. Agent benchmarks enable shortcuts (Section 5).
We show that many types of overfitting to agent benchmarks are possible. We identify 4 levels of generality of agents and argue that different types of hold-out samples are needed based on the desired level of generality. Without proper hold-outs, agent developers can take shortcuts, even unintentionally. We illustrate this with a case study of the WebArena benchmark (Zhou et al. 2024
5. Agent evaluations lack standardization and reproducibility (Section 6 We found pervasive shortcomings in the reproducibility of WebArena and HumanEval evaluations (Table 7). These errors inflate accuracy estimates and lead to overoptimism about agent capabilitics.
The overarching goal of our work is to stimulate the development of agents that are useful in the real world and not just accurate on benchmarks. (1) and (2) above do this by incorporating metrics beyond accuracy into agent evaluation and optimization; (4) and (5) do so by improving precision about what a benchmark aims to measures and ensuring that it actually measures that; and (3) does both.
1.1 What is an Al agent?
In traditional AI, agents defined entities that perceive and act upon their environment (Russell & Norvig 1995). In the LLM era, the term is used in a narrower way (a thermostat would qualify agent under the traditional definition). Many rescarchers have tried to formalize the community's intuitive understanding of what constitutes an agent in the context of language-moddllbassed systems. Many of them view it as a spectrum - sometimes denoted by the term 'agentic' (Ng) 2024) rather than a binary definition of an agent. We agree with this perspective. Since there are already many definitions, we do not provide a new one, but rather identify the factors that cause an AI system be considered more agentic according existing definitions. We found three clusters of factors.
2

---

Published in Transactions on Machine Learning Research (05/2025)
Environment and goals. The more complex the environment c.g. range of tasks and domains, multi-stakeholder,, long time horizon, unexpected changes the more AT systems operating in that environment are agentic (Shavit et al. 2023 Gabriel et al.; 2024). Systems that pursue complex goals without being instructed on how to pursue the goal are more agentic (Shavit et al. 2023 Chan et al. Gabriel et al. 2024/.
2023
User interface and supervision. Al systems that can instructed in natural language and act autonomously on the user's behalf are more agentic (Gabriel et al. (024). In particular, systems that
require less user supervision are more agentic (Shavit al.) 2023 Chan al. 2023 Gabriel al. 2024 We discuss the user supervision aspect in more detail in Section 5.2
System design:
Systems that design patterns such tool use (c.g., web scarch, programming)
planning (e.g,, reflection, subgoal decomposition) are more agentic [Wengg 2023
2024
Systems whose
control flow is driven by an LLM, and hence dynamic, are more agentic [Weng] 2023
Chase 2024
2
Al agent evaluations must be cost-controlled
2.1
Maximizing accuracy can lead to unbounded cost
Calling language models repeatedly and taking a majority vote can lead to non-trivial increases accuracy across benchmarks like GSM-SK, MATH, Chess, and MMLU (Li et al. 2024 Chen al. 2024 Sun et al., 2024).
When the agent environment has casy signals to check if an answer is correct, repeatedly retrying can lead to even more compelling performance gains (Villalobos & Atkinson 2023) Li et al. (2022) showed that the accuracy of AlphaCode increases from close to 0% zero-shot to over 15% with 1,000 retries and over 30% with a million retrics (accuracy mcasured by how often one of the top 10 answers generated by the model correct). Thus, there is seemingly no limit to the amount of inference compute that can increase accuracy, and scaling inference compute has been shown improve performance in various applications (Welleck et al., 2024 [rron et al., 2024). Coding competitions often include signals of correctness, such test cases, which serve verifiers to check if given solution is correct. Agent developers can keep sampling from an underlying model until the solution passes the test cases. Our results below show that this is true for HumanEval.
2.2 Visualizing the accuracy-cost tradeoff using a Pareto curve
In the last year, many agents have been claimed achieve state-of-the-art accuracy on coding tasks. But what cost? To visualize the tradcoff, we ro-ovaluated the accuracy of three agents.
Specifically, we included agents from the HumanFval leaderboard on PapersWithCode that share their code publicly (Chen et al. 2021]: LDB (Zhong et al., 2024). LATS (Zhou et al. 2023). and Reflexion (Shinn et al, 2023 These agents rely on running the code generated by the model, and if it fails the test cases provided with the problem description, they try to debug the code (Zhong al. 2024 look at alternative paths
the code generation process (Zhou et al. 2002)), or "reflect" on why the model's outputs were incorrect before generating another solution (Shinn et al. 2023 Zhou et al. 2023 Zhong al. 2024).
We also evaluated the cost and time requirements of running these agents. In addition, we calculated the accuracy, cost, and running time of few simple baselines.
GPT-3.5 and GPT-4 models (zero shot; no agent architecture)
*Reflexion is absent from the CopersWithCode leaderboard, but it has reported accuracy of 91% (higher than any other agents with publicly available code apart from LDB and LATS), we included top- perfperfoomng agent, did not include link to the code on the benchmark the time our analysis (late April 2024), did it include link to the code in the paper, we did include it.
analysis. AgentCoder, listed the
3

---

Published in Transactions on Machine Learning Research (05/2025)
1.00
0.95
Warming (GPT-4)
LC9 (GPT-4)
Pareto hunieer
Retry (GPT-4)
LDE (Raflexion, GPT-4)
0.90
LDB (GPT-4, GPT-3.5)
GPT-4
LDB (Refiexion, GPT-3.5)
Accuracy
LATS (GPT-4)
Rañaxioo (GPT-4)
0.85
Escalation
0.80
LATS (GPT-3.5)
LDB (GPT-3.5)
0.75
Complex agent
GPT-3.5
Baseline agent model
0.70
130
135
Cost (USD, measured in April 2024)
Figure 1: Our simple baselines offer Pareto improvements over SOTA agents. We rum each agent five times and report the mean accuracy and the mean total cost on the 164 HumanEval problems. Where results for LDB have two models/ radeess/ggent in parenthesis, they indicate the language model or agent used to generate the code, followed by the language model used to debug the code. Where they have just one, they indicate that the same model was used to both generate the code and debug it. Note the nonstandard axes; In Appendix A we show our results with the full y-axis as well as error bars and provide additional details. Robustness checks are contained in Appendix A..2 Section 4 explains why we measure dollar costs instead of using proxics for cost such as the amount of compute used.
We repeatedly invoke model with the temperature set to zero, to five times, if it fails the test cases provided with the problem description. Retrying makes sense because LLMs aron't deterministic even at temperature zero (Appendix A.1
Retry:
This the same the retry strategy, but we gradually increase the temperature of the underlying model with each run, from 0 to 0.5. This increases the stochasticity of the model and, we hope, increases the likelihood that at least one of the retries will succeed.
Warming:
Escalation: We start with a cheap model (Llama-3 8B) and escalate to more expensive models (GPT-3.5, Llama-3 70B, GPT-4) if we encounter a lest. case failure.
We use the modified benchmark version of HumanEval provided with the LDB paper (Zong et al. 2024 since it includes example test cases for all 164 tasks (in the original benchmark, example test cases are provided for only 161 of 164 tasks, as detailed in Section 6
2.3 Two-dimensional evaluation yields surprising insights
Fig. shows our main results for this section. Note that an agent is on the Pareto frontier if there is no other agent that has significantly better performance on both dimensions simultaneously (see Appendix A.1
"State-of-the-art" agent architectures for HumanEval do not outperform simple baselines. There is no significant accuracy difference between our warming strategy and the best rest-performing agent architecture. In fact, not aware of any papers that compare their proposed agent architectures with any of the
We coustrain the Parcto frontier to be convex, because given points and the graph corresponding agents A and can always linearly interpolate between them by creating agent that invokes with probability p and B with probability p. Hence, for instance, zero-shot GPT-4 is not the frontier.
4

---

Published in Transactions on Machine Learning Research (05/2025)
last three of our simple basclines HumanEval (retry, warming, escalation) 0 This finding is supported by several other recent studies showing that scaling inference compute can increase accuracy (Brown et al. 2024 Hassid et al. 2024 Snell et al. 2024).
Agents differ drastically in terms of cost. For substantially similar accuracy, the cost can differ by almost two orders of magnitude. Yet, the cost of running these agents isn't a top-line metric reported in any of these papers. Reflexion and LDB cost over 50% more than the warming strategy, and LAT'S over 50 times more (all these costs are entirely predominantly from calls to GPT-4, so these ratios will be stable even if model costs change). Meanwhile, the escalation strategy strictly improves accuracy while costing less than half of LDB (GPT-3.5).
Lack of clarity on the source of performance gains. There is a widespread belief in the Al community that complex ideas like planning, reflection, and debugging are responsible for accuracy gains on tasks such as HumanEval. However, based on our findings, the qucstion of whether debugging, reflection, and other such "System 2" approaches (Kambhampati et al. 2024) are useful for code generation remains open, in line with other recent findings (Vermaa et al. 2024 Huang et al.] 2023 Pan et al. 2024). The lack of clarity about the efficacy of System 2 approaches is exacerbated by a lack of reproducibility and standardization that we report in Section [6] Failing to identify the sources of empirical gains is longstanding issuc in ML
and related fields (Lipton & Steinhardt, 2018: Henderson et. al.] 2018).
At the same time, models such as OpenAl's ol have reported impressive gains owing to System 2 ap- proaches (OpenAI, 2024a), though this involves training the model to be better at System 2 approaches using reinforcement. learning (OpenAT] 2024bb. Notably, ol outperforms gpt-40 with retry and reflection even when accounting for inference compute (Epoch AT 2024 Similarly, it is possible that System 2 techniques will be useful on harder programming tasks than those represented in HumanEval, such as SWE-bench (Jimenez) et al. 2023).
To summarize this section, useful agent ovaluations must control for cost - even if we ultimately don't care about cost and only about identifying innovative agent designs. Accuracy alone cannot identify progress because it can be improved by scientifically meaningless methods such as retrying.
3 Jointly optimizing cost and accuracy can yield better agent designs
Visualizing the cost and accuracy of agents as a Pareto frontier opens up a new space for agent design: jointly optimizing cost and accuracy, which can lead to agents that cost less while maintaining accuracy. This formalization fully generalizable other desiderata of agent design, such as latency.
The total cost of running agent includes fixed and variable costs. Fixed costs are one-time expenses incurred when optimizing the agent's 's hyperparameters (temperature, prompt, etc.) for given task. Variable costs are incurred each time the agent is run and depend on the number of imput and output tokens. The more an agent is used, the more the variable cost dominates (Appendix B
Joint optimization allows us to trade off the fixed and variable costs of running an agent. By spending more upfront on the one-time optimization of agent design, we can reduce the variable cost of running an agent (e.g., by finding shorter prompts and few-shot examples while maintaining accuracy).
As an illustration of the potential of joint optimization, we modify the DSPy framework (Khattab et al. 2023) and evaluate it on the HotPotQA benchmark (Yang et al.] 2018). We chose HotPotQA because is one of the benchmarks used to illustrate the effectiveness of DSPy in the original paper and has been featured in several official tutorials by the developers. We used the Optuna hyperparameter optimization framework (Akiba et al.] 2019) search for few-shot examples to be included with an agent that minimizes cost while maintaining accuracy. Note that we expect more complex joint optimization approaches vastly outperform our approach. Our results are only starting point intended illustrate the vast, underexplored design space in agent design enabled by joint optimization.
Chen et al. 2024 do compare the effect of retrying the same model multiple times on HumanEval, but they da not test the per performance GPT,4, nor do they argue for a specific complex agent architecture.
5

---

Published in Transactions on Machine Learning Research (05/2025)
3.1 HotPotQA evaluation setup
We implement several agent designs to evaluate performance on multi-hop question-answering using DSPy. For retrieval, we use ColBERTV22 to query Wikipedia based on the HotPotQQ task specification (Yang et al, 2018). Performance is evaluated by comparing whether the agent successfully retrieved all ground-truth documents that are part of the HotPotQA task. We use 100 samples from the HotPotQA training set to optimize the DSPy pipelines and 200 samples from the evaluation set to evaluate the results (this is consistent with the implementation of the DSPy pipelines provided by the developers to illustrate efficacy at multi-hop retrieval). We evaluate five agent architectures:
We do not optimize the agent's prompt or include instructions on formatting Hot queries. Each prompt only contains the instructions for the task and the main content (i.e., question, context, reasoning) but no few-shot examples or formatting instructions.
Uncompiled:
Formatting instructions only: This is the same as the uncompiled baseline, but we add instructions on how to format generated outputs for writing retrieval queries.
Few-shot: We use DSPy to identify effective few-shot examples using all 100 samples from the training set. We include formatting instructions. Few-shot examples are selected based on successful predictions generated on the training set.
We use DSPy's ' random search optimizer a subset of the training data (50 of 100 samples) to select the best few-shot examples based on its performance on the remaining 50 samples. We include formatting instructions.
Random Search:
Joint optimization: We iterate over half the training set (50 of 100 samples) to collect set candidate few-shot examples that improve the model's accuracy. We use the other 50 samples for validation. We jointly maximize accuracy and minimize the number of tokens in the few-shot examples included in the prompt using parameter search. We implement parameter search using Optuna (Akiba et al., 2019). We search over the following parameters to find Pareto-optimal agent designs: (a) the temperature for each module within the agent, (b) the number of few-shot examples, (c) the selection of specific examples, and (d) whether to add formatting instructions. Of the candidate agents selected by Optuna, we pick the one with the best accuracy on the development set as our joint optimization model.
We Lest all five of the above agent designs two underlying models: Llama-3-70B and GPT-3.5.
3.2 HotPotQA results: Joint optimization reduces cost while maintaining accuracy
Fig. 2 shows our main results. We confirm that DSPy offers substantial accuracy improvements over uncompiled baselines, but we find that this comes at a cost. Fortunately, we can mitigate the cost overhead for GPT-3.5, joint optimization leads to 53% lower variable cost with similar accuracy compared to both default DSPy implementations. Similarly, for -lama-3-70B, leads to a 41% lower cost while maintaining accuracy.
Tradeoffss between fixed and variable costs for agent design. Our joint optimization formulation provides way to trade off fixed and variable costs (Appendix B). In particular, find that if used for HotPotQA tasks, the tlama-3-70B as well as the GPT-3.5 joint optimization model both become cheaper (in terms of total cost.) compared to the default DSPy implementation, after 1,350 tasks (Appendix B. For agents used in large-scale real-world tasks, the variable cost. is by far the dominant term compared to the fixed cost, by orders of magnitude, as an agent might be used millions of times.
In summary, joint optimization allows for efficient agent design. This comes at small fixed cost for optimizing
the design, which is insignificant if the agent is used thousands of times.
6

---

Published in Transactions on Machine Learning Research (05/2025)
0.7
Laama-37708 (Tew- Shot)
1ama---7788 (Format. Instr)
L-0 (Joint Opt.)
0.6
Parelo horter
Liama-3-708 (Random Search)
Figure 2: Joint optimization maintains accuracy while significantly reducing cost. All measurements are av- erages of 5 runs on the test set. A figure with error bars and other details about included in Appendix B Note that DSPy has a much higher cost than our baseline agent designs since it always includes up to 8 few-shot examples per prompt (the number being a configurable parameter) without optimizing for cost, whereas the bascline agents include none, and joint optimization uses variable number of examples.
GPT-3.5 (Random Search)
0.5
Accuracy
GPT-3.5 (Joint Opt.) GPT-3.5 (Few-Shot)
0.4
Liama 3:708 (Uncompiled)
empirical results are
GPT-3.5 5 (Incomplled)
0.3
GPT-3.5 (Format. Instr.)
0.2
Join optimication Formating instructione only Unoompieed DSPy detaut
0.1
0..0
0.6 Cost per 100 Inferences (USD, measured May 2024)
0.2
0.4
0.8
4 Model and downstream developers have distinct benchmarking needs
Al evaluations are used by model developers and Al researchers to identify which changes to the training data and architecture improve accuracy (model evaluation) and also by downstream developers to decide which AI systems to use in their products for procurement decisions (downstream evaluation). The difference between model evaluation and downstream evaluation is underappreciated. This has led to much confusion about how to factor in the cost of running AI.
Model evaluation is a scientific question of interest to researchers. Here, it makes sense to stay away from dollar costs, because reporting costs breaks many properties of benchmarks that we take for granted: measurements don't change over time (whereas costs tend to come down) and different models compete a level playing field (whereas some developers may benefit from economies of scale, leading to lower inference costs). Because of this, researchers usually pick a different axis for the Pareto curve, such as parameter count. or training compute.
For model evaluation, controlling for compute is a reasonable approach: if we normalize training compute, we can then understand if factors like architectural changes changes in the data composition are responsible for improvements, as opposed to more compute (Lambert 2024h.
Downstream evaluation is an engineering question that helps inform a procurement decision of which model or agent to use in a particular application. Here, cost is the actual construct of interest. The downsides of cost measurement in model evaluation are exactly what is needed for downstream evaluation. Namely, inference costs do come down over lime, and that greatly malters to downstteam developers. is unnecessary and counterproductive for the evaluation to stay frozen in time.
Proxies for cost are misleading for downstream evaluation. In the context of downstream evaluation, proxies for cost (such as the number of active parameters or training compute) are misleading. For example, Mistral released a figure alongside their latest model, Mixtral 8x22B, to explain why developers should choose it over competitors (Mistral AI team 2024). It used the number of active parameters as a proxy for cost. From the perspective of a downstream developer, this proxy is misleading. For example, as of June 2024, Mixtral 8x7B costs twice as much as I.lama. 13B on compute provider Anyscale. Yet Mistral's figure shows that it costs about the same because it only considers the number of active parameters.
Downstream developers don't care about the number of active parameters when they're using an API. They simply care about the dollar cost relative to accuracy. Mistral chose "active parameters" as a proxy, presumably because it makes their models look better than dense models such as Meta's Llama and Cohere's Command R+. If every model developer picked a proxy that makes their model look good, multi-dimennsonal evaluation would lose its usefulness.
Addressing challenges to cost evaluation. As discussed above, there are several hurdles to cost evaluation. Different providers can charge different amounts for the same model, the cost of an API call might change
7

---

Published in Transactions on Machine Learning Research (05/2025)
overnight, and cost might vary based on model developer decisions, such as whether bulk API calls are charged differently.
These downsides can be partly addressed by making the evaluation results customizable using mechanisms to adjust the cost of running models, i.e., providing users the option to adjust the cost of input and output tokens for their provider of choice to recalculate the tradeoff between cost and accuracy. In turn, downstream evaluations of agents should include looking at the evaluation in the future can instantly recalculate the cost using current prices. We have prototyped simple example of such an interface
token counts in addition to dollar costs, that anyone
4.1 Implications for benchmark design using a case study of NovelQA
The difference between model and downstream evaluation can also lead to challenges in benchmark design. We show how such challenges arise with a case study of the NovelQA benchmark (Wang et al. 2024a The benchmark is motivated by the need to evaluate language models with long context windows. Novel lengths in the benchmark range from 50,000 to over a million words. Each novel has between and 100 questions about it. To evaluate an AI system on NoveIQA, developers submit their model responses to benchmark questions to centralized platform (CodaBench), and top-performing submissions are included in a public leaderboard.
This is good benchmark for model evaluation, but it would be misleading if used for downstream evaluation by developer looking to build bot for answering questions about novels. Like the other benchmarks we have discussed, the NovelQA leaderboard does not measure cost. Nor is it easy to construct such a leaderboard. That's because NovelQA evaluates language models by asking all questions about a novel in one go, right after inputting the novel's content. However, this does not represent how users would ask questions about novels in practice. Even users have many questions about novel, they will likely ask them individually rather than all at once. Such sequential queries would cost orders of magnitude more because the novel has to be re-processed each time. In other words, the task of answering multiple questions about a novel (as implemented by NovelQA) doesn't 't tell us anything about the cost of asking questions sequentially.
In particular, when comparing the two main approaches to novel-based QA - long-context models and retrieval-angmenned gencration (RAG) - NovelQA makes RAG look much worse than it is in a real-world scenario. Specifically, we found that the two approaches are roughly equally accurale, with RAG costing more than 20 times less (Table 6). But on NovelQA, RAG costs half as much (a tenfold overestimate). As result, the NovelQA leaderboard is mislcading for downstream evaluation. To be clear, NovelQA authors don't claim that it is suitable for downstream evaluation. Still, in general it is a common practice for downstream developers to look to model evaluation benchmarks. Instead, we argue that downstream evaluation benchmarks must be separate from, or at least variants of, model evaluation benchmarks.
5 Agent benchmarks allow shortcuts
Benchmarks are useful if they give us an estimate of real-world accuracy. If a benchmark allows shortcuts. accuracy on the benchmark does not translate to the real world (MeIntosh et al. 2024 Mialon et al. 2023 Wagstaff 2012).
Overfitting is one prominent type of shorteut, and serious problem for agent benchmarks, since they tend to be small - typically few hundred samples. This is much more serious problem than LLM training data contamination, as knowledge of test samples can be directly programmed into the agent opposed to merely being exposed to them during training. In principle, lookup table can achieve 100% accuracy agent benchmarks. Overfitting would be obvious to spot if an agent used a lookup table, but other types of overfitting can be much more subtle and hard to detect, which is of course why held-out test sets are crucial in machine learning. Yet, surprisingly, we find that many agent benchmarks do not include held-out test sets. In addition to creating a test set, benchmark developers should consider keeping it secret to prevent LLM contamination or agent overfitting.
many
4See: httppr///aannikkkcrrcchhhhtthtth http:/aghhihiiiaaaateeeee------e
8

---

Published in Transactions on Machine Learning Research (05/2025)
But can go further. What about overfitting to task? For example, if agent scores highly Python programming benchmark but camnot program in any other language, is it a problem? This depends entirely on the agent's purpose and what the benchmark creator desires in terms of the generality of the agent CChllett 2019: Morris et al. 2024). In our survey, we have found four levels of generality:
1. Distribution--peecfic benchmarks are limited to specific task, such as U.S. grade school math problems, and do not account for distribution shifts.
2. Task-specific benchmarks are limited to a specific task such as booking a flight, placing an order on an e-commerce website (Yao et al., 2023), or solving a GitHub issue JJmenez et al. 2023), and account for the possibility of distribution shifts, including drift. After all, an agent that can book flights today but breaks if the flight booking website changes its layout would not be very useful. Web agents have already been found to be sensitive to such small modifications to the GUI (aa et al.] 2024b).
3. Domain-gencral benchmarks aim to mcasure the ability to perform any task in a specific domain, such as web browsing or tool use (Zhou el al. 2024
4. General-purpose benchmarks mcasure the accuracy of agents across different domains, such as the same agent being able to perform web browsing and roboties tasks. It is unclear if such benchmarks are necessary or if aggregating domain-general benchmarks can better serve the purpose.
We propose as a core principle that the greater the intended generality of the agent, the more the held-out set should differ from the training set, as detailed in Table [ For example, if a benchmark is intended to be domain- general but doesn't contain held-out tasks, agent developers may (intentionally or unintentionallyy) take shortcuts that work only for the specific tasks represented in the dataset, resulting in agents that don't. ' work well for other tasks in the domain. Benchmark developers must do their best to ensure that shortcuts are impossible. We view this as the responsibility of benchmark developers rather than agent developers, because designing benchmarks that don't allow shorteuts is much easier than checking every single agent to see if it takes shortcuts. Benchmarks that create a level playing field are the core reason for the rapid Simons Foundation 2019
progress of ML over the last half century (Donoho 2017
Table 1: Appropriate holdouts based on level of generality. See Appendix for details.
Level of generality
What should be held out
Num. benchmarks with appropriate holdouts
Distribution-spocific
In-distribution samples Dut-of-distributio samples Tasks
/ 2
Task-specific
4 10
Domain gemainl
18
Fully general
Domains
0 0/3 3
There are many types of distribution shifts, and benchmark developers can't necessarily model all of them. But they must attempt to identify which distribution shifts are particularly likely for the task in question. Another approach evaluated not just on benchmark tasks but also the corresponding worldwrrld tasks - for example, Amazon shopping for a web shopping benchmark Yao al. 2023).
not always practical is to evaluate sim2real transfer, where leading agents are
We analyzed 33 agent benchmarks and classified them into the four levels of generality (Table 5 Most are either task-specific or domain- domain-general. In many cases, it wasn't clear which level of generality the benchmark developers had in mind, and we made our best guess based on how the benchmark was presented in the paper. This lack of clarity is problematic as it makes it hard to know what we can and can't conclude about the agents that perform well on that benchmark.
We recognize that time and resource constraints may hinder benchmark designers from creating a holdout sot at the correct generality level. Hence we count a holdout as appropriate if there actually exists a holdout at the appropriate level of generality or the benchmark designers indicate an intent. to create such a holdout.
9

---

Published in Transactions on Machine Learning Research (05/2025)
However, as shown in Table I and Appendix C held-out set, including 17 that have no hold-out and no indication that holdout set will be added in future editions of the benchmark.
the majority of benchmarks do not include an appropriate
Note that in traditional machine learning research, held-out test sets are usually at the level of in-distribution or ont-oo-distribuution samples (the first two rows of the Table D. This sufficient because models are specific to a single task, say image classification or spam classification. But LLMs and domain-general agents are expected to handle tasks that are not known ahead of time and may be specified in natural language in some cases. This motivates the need for held-out tasks during evaluation.
5.1 Case study of the STeP agent on WebArena.
Web agents can be evaluated on many capabilities: navigating to a website, scrolling, selecting the right web element etc. There are many different types of websites that can be used such benchmarks: e-commerce, social media, information search etc. WebArena is an agent benchmark that aims to evaluate agents on tasks on the web (Zhou et al., 2024p. Tt. includes clones of six different websites (GitLab, Reddit, Wikipedia, OppsStrectMaps, an e-commerce platform, and a content management system) and two tools (calculator and scratchpad). It has 812 different tasks that involve interacting with these websites, such as "find the address of all US international airports that are within driving distance of 60 km to the Niagara Falls" and "post question on a subreddit related to New York City".
WebArena's 's core stated selling point seems to be realism, which means that it should be difficult to find shortcuts. If we consider WebArena a task-specific benchmark, the key type of distribution shift is drift: agents should be robust to changes made to a website over time. However, WebArena does not model drift. To be clear, it is challenging to model drift. Benchmark developers would need to find changes made to published websites and incorporate those into the test. sets. Further, as we discuss above, the held-out set. would need to be kept secret, since agent developers could otherwise overfit to the specific changes in the benchmark. Still, we view these steps as necessary for mcaningful evaluation.
Consider the top agent on the WebArena leaderboard, called STeP (Sodhi et al. 2024 p. Tt has an accuracy of 35.8%, more than double the accuracy of the top-performing baseline agent introduced in the WebArena paper, and over 10 percentage points more than the next-best agent (Drouin et al., 2024). How does STeP achieve this high accuracy? It turns out that STeP hardcodes policies to solve the specific tasks included in WebArena. For example, several WebArena. Reddit tasks involve navigating to user's profile. The STeP policy for this task is to look at the current base URL and add a suffix 1/user/user_nama patassrrreee___aae This is brittle: the policy would no longer be effective if the website updates its URL structure (an example of drift). Even if the probability of an individual policy failing is small, an agent might need to call different policies dozens of times for cach task. The overall probability of failure compounds quickly.
To be clear, the STeP developers' goals are orthogonal to the benchmark developers' goals creating composable policies for accomplishing fixed tasks that are known apriori. From this perspective, STeP's design choices make sense. Yet the leaderboard accuracy WebArena (such the accuracy of the ST&P agent) is misleading from the perspective of downstream developers, who might. be using the WebArena leaderboard to understand the accuracy of web agents on real-world tasks and make decisions about which agent to adopt in an application.
Things become even more problematic if we consider WebArena a domain-general benchmark. This can be justificd based on the claim that for previous web benchmarks, "the functionality of many environments is a limited version of their real-world counterparts, leading to a lack of task diversity" (Zhou et al., 2024). This suggests that the WebArena developers aim to stimulate the development of agents that can accomplish many different tasks on the web.
Unfortunately, WebArena lacks a held-out test set for evaluating whether an agent can perform well on unseen web tasks. (It is hard to confirm if building a Somain-generas benchmark is indeed their main objective. This lack of clarity is problematic as it makes it hard to assess whether benchmark developers deliver on their promises and what the leaderboard accuracy truly reflects.) Note that if the held-out set contained different tasks (such samples from completely new and unseen websites) compared to those in the training set,
10

---

Published in Transactions on Machine Learning Research (05/2025)
the accuracy agents like STeP would be drastically lower, because none of the hardcoded policies would be effective.
5.2 Agent benchmarks don't account for humans in the loop
The degree of human supervision, feedback, and intervention required for an agent to perform a task can be seen as a spectrum. Consider a data analysis task. On one end of the spectrum, the analyst might use a chatbot to help with tasks like debugging. Here the user is firmly in control and verifies all chatbot outputs. Or the analyst might ask the agent to write and execute code for certain data visualization tasks. The analyst might not review every line of code but only intervene if something appears to be wrong. At the other extreme, an analyst night give the agent a dataset and a lask (e.g. identify how the presence of a swimming pool impacts property value using a home sale database), giving it full autonomy over all aspects of the task such as how to perform causal inference.
Current evaluations predominantly focus on these two extremes: they either evaluate the capacity of chatbots to answer questions correctly (e.g-: MMLU (Hendrycks al. 2021)), whether agents can perform task with no supervision (e.g., the agent benchmarks in Table 5).
However, these evaluations do not reflect how people use chatbots and agents in the real world. For example, the humans using a chatbot might steer it towards the correct answer, give feedback on where went wrong, or ask it to change something about the output. Similarly, an agent might have the capacity to take an action, but require user confirmation for consequential actions. For example, users read emails using ChatGPT's Zapier plugin by connecting their email accounts, but sending an email requires confirmation from the end user.
Human feedback might greatly increase accuracy. For example, Shi et al. (022) find that simple feedback increases the performance of GPT-4 from 0% to over 86% on challenging programming problems from entirely useless to almost perfect. Thus, the lack of human-in-the-logg evaluation of agents might lead us to underestimate their usefulness (whereas the lack of held-out, test sets leads to overoptimism).
Of course, human-in-the-loop evaluation is costly and tricky. The agent's accuracy will depend not just the system but also on the skill level of humans interacting with it. While addressing these concerns with
evaluations is beyond the scope of this paper, it is important direction for future rescarch. Tbrahim et al. (2024) provide a detailed overview of implementing such evaluations in the context of safety evaluation.
6 Inadequate benchmark standardization leads to irreproducible agent evaluations
During our experiments, we identified several shortcomings in the reproducibility and standardization of agent. benchmarks and evaluations. Our analysis is based on widely accepted definition of reproducibility: that the code and data accompanying a paper should be enough to reproduce the results that it reports (National Academies of Sciences, Engineering, and Medicine 2019). Without reproducible agent evaluation, it is hard to distinguish genuine improvements in agent designs from artifacts of the differences in ovaluation choicos. Since agents are often intended to be used by downstream developers, lack of reproducibility also misleads developers adopting agents in real-world applications. Finally, irreproducible results impose a huge time cost on researchers trying to build on claims of state-of-the-art results.
We identified five rool causes for the lack of standardized and reproducible agent evaluations, all of which arise from the differences between LLM evaluation and agent evaluation. Since these issues relate to standardization, we view them as primarily the responsibility of benchmark developers rather than agent developers.
1. Evaluation scripts make assumptions about agent design that aren't satisfied by all agents. Many evaluation shortcomings result from the lack of a clear standard for releasing evaluation scripts alongside benchmarks (Biderman et al.) (2024). Agent developers need to implement their own evaluation for their agents either because the evaluation script does not account for different agent designs or because the script provided by the benchmark developer itself has bugs. This leaves open the possibility for
11

---

Published in Transactions on Machine Learning Research (05/2025)
non-standard evaluation scripts leading to incomparable results across agent developers evaluating their agents on the same benchmark.
2. Repurposing LLM evaluation benchmarks for agent evaluation introduces inconsistencies. In some cases, even if benchmark developers provide evaluation scripts, agent developers need to re-implement these scripts. This is because many prominent benchmarks have been designed for language model evaluation. Using these benchmarks to evaluate agents can require changes to the benchmark design and evaluation.
For example, HumanEival does not include example test cases for 3 of the 164 problems. In addition, the example test cases are included in the docstring accompanying a problem, rather than being machine readable and easily extractable. Both of these design choices are suitable for evaluating language models, since models can be evaluated by prompting them with the included doestrings, even if there are no example test cases. But this benchmark design does not work for agents that rely on using the included example test cases and regenerate solutions if they are incorrect (such as LDB, Reflexion, LATS, and our simple baselines in Section 2p. As a result, developers evaluate agents using varying subsets of benchmarks adding example test cases to the original benchmark. For example, the Reflexion (Shinn et al. 2023
and LATS (Zhou et al.] 2023) modified HumanEval by removing the problems without example lests. LDB (Zhong et al.) 2024) added example tests to the 3 problems which did not originally include them and convert examples to a machino-readable format (we use the version provided by the authors of LDB). But agent and model evaluations are clubbed together on leaderboard websites such as PapersWithCoo despite the varying evaluation methods. Aggregators of benchmark results (such as PapersWithCode) must ensure that evaluations on the same leaderboard were all conducted using the same methods.
3. The high cost of evaluating agents makes it is hard to estimate confidence intervals. Agents might call the underlying languago models hundreds (or thousands) of times. This means that evaluating agents can be extremely expensive. For example, SWE-bench (Jimenez et al. 2023 consists of over 2,000 programming tasks. The authors of SWF-Agent (Yang et al., (024)) set a cost. limit of USD) per task. Evaluating SWE-Agent on the entire benchmark could cost over USD 8,000 for a single evaluation run. The high cost makes it infeasible to run evaluations multiple times, and perhaps evaluations are rarely accompanied by error bars. This makes it hard to understand the variance of reported results. We found that many reported accuracy scores were above the maximum of five runs that we performed in our reproduction attempts, and the reported baselines were in some cases lower than the minimum of five runs we performed.
result, agent
4. Agent evaluation relies external factors such interacting with an environment which can lead to subtle errors. LLM benchmarks typically consist of input strings and rely on strings as outputs, whereas agents often involve dynamic interactions with environments such as the web or a command line, which are not easily reducible lo static inpuls and outputs. This can lead to incorrect assumptions. For example, one common assumption in evaluation is invariance to the order in which tasks in a benchmark are evaluated, as tasks are prosumed be independent. This assumption fails for WebArena (Zhou el al., 2024). One of the websites included in the benchmark (a clone of Reddit) has rate limits for certain agent actions, such as posting content. As result, if the tasks involving Reddit posts are evaluated one after another, they are much more likely to fail. This affects the evaluation of the STeP agent (Sodhi et al. 2024
5. The lack of standardized evaluation leads to subtle bugs in agent evaluation and development. Perhaps due to the issues above, we encountered several bugs with agent developers' implementation of their agents and their evaluations. For example, both LATS (Zhou et al. 2023) and STeP (Sodhi et al.] 2024) marked some incorrectly completed tasks as correct. Similarly, both agents removed small nunber of tasks from the benchmark (1 and 8 tasks for LATS and STeP respectively).
The need for a standardized evaluation framework. These shortcomings stem from three distinct (but related) reasons. First, as of yet, there are no clear standards for providing agent evaluation scripts (shortcoming 1). As result, the differences between model and agent benchnarks are not appreciated (shortcomings 1-3). Finally, due to the lack of community norms on evaluation, there is scope for bugs to
12

---

Published in Transactions on Machine Learning Research (05/2025)
creep in during agent development and evaluation (shortcoming 5). We include examples and more details each in Table 7 Shortcomings with standardization have also been observed for L.M evaluation. Evaluation frameworks like HELM (Liang et al., 2023) and LM Evaluation Harness (Biderman et al. 2024) address these shortcomings for model ovaluations by providing standardized evaluation results. But as we have seen, these Irameworks don't suffice for evaluating Al agents. Developing an agent evaluation framework is a ripe area for future work.
7 Conclusion
AI agent benchmarking new and best practices haven't yet been established, making hard distinguish genuine advances from hype. Our thesis is that agents sufficiently different from models that benchmarking practices need to be rethought. We have taken the first steps toward principled approach to agent benchmarking, resulting in recommendations including cost-controlled comparisons, separating model and downstream evaluation, preventing shortcuts using appropriate hold-outs, and greater standardization of evaluation practices. While developing a full solution for these concerns is beyond the scope of this paper, we hope these sleps will raise the rigor of AI agent evaluation and provide a firm foundation for progress.
References
googlerreearcchccdttseess//ppppp Datasets at Hugging Face, April 2023. URL https://huggingaace.com co/
Mohamed Aghzal, Erion Plaku, and Ziyu Yao. Can Large Language Models be Good Path Planners? A Benchmark and Investigation on Spatial-temporal Reasoning, February 2024. URL btttp://arxiv.oo/
abs/2310. arXiv:2310.03249 [cs].
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A Next- generation Hyperparameter Optimization Framework. In Proceedings of the 25th ACM SIGKDD Interna- tional Conference on Knowledge Discovery & Dala Mining, KDD '19, pp. 2623 2631, New York, NY, USA, July 2019. Association for Computing Machinery. ISBN 778-1-4503-6201 doi: 10.1145/329250033337700
URL httpp://ddl acm. www//oo////.oooooooooooooooo 1145/3292500 3330700
Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Teo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, Charles Lovering, Niklas MuennighofT, Ellie Pavlick, Jason Phang, Aviya Skowron, Samson Tan, Xiangru Tang, Kevin A. Wang, Genta Indra Winata, François Yvon, and Andy Zou. Lessons from the Trenches on Reproducible Evaluation of Language Models, May 2024. URL http://arxiv org/ abbb/245555 14782 arXiv:2405.17782 [cs].
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling, July 2024.
URL http://arxiv wwwpeiivvorg//bbeeee wtwpobr/g/ab/////oare/// 2277771777777777 arXiv:2407.21787 [es].
Alan Chan, Rebocca Salganik, Alva Markelius, Chris Pang, Nitarshan Rajkumar, Dmitrii Krasheninnikov, Lauro Langosco, Zhonghao He, Yawen Duan, Micah Carroll, Michelle Lin, Alex Mayhew, Katherine Collins, Maryam Molamohammadi,, John Burden, Wanru Zhao, Shalaleh Rismani, Konstantinos Voudouris, Umang Bhatt, Adrian Weller, David Krueger, and Tegan Maharaj. Harms from Increasingly Agentic Algorithmic Systems. In Proceedings of the 2023 ACM Conference Fairness, Accountability, and Trunspurency, FAccT '23, pp. 651-666, New York, NY, USA, June 2023. Association for Computing Machinery. ISBN 9798400701924. doi: 1 1145/3593013.35944
URL https: acm acw.orr//oo//0
Harrison Chase. What is agent?, June 2024. URL https::/blog.llagchhii..ee
13

---

Published in Transactions on Machine Learning Research (05/2025)
Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, and James Zou. Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systems, March 2024.
URL http://arxiv. http:///axiiv.org/bb//////////// 024199
arXiv:2403.02419 [es, eess].
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jic Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthow Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating Large Language Models Trained on Code, July 2021. URL hRtp://///axxxi.orrgg wwwpg///////..//./a///////////./ 03374 arXiv:2107.0337. [cs].
Ching-An Cheng, Andrey Kolobov, Dipendra Misra, Allen Nie, and Adith Swaminathan. LLF- Benchmark for Interactive Learning from Language Foodback, December 2023. URL http:htppp/// /tttp://arxivoor org/ abs/2312.0680 HrXiv:2312.06857 [cs].
Maxime Srovalier-Boiseerr Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Baby Al: A Platform to Study the Sample Elficiency of Grounded Language Learning, December 2019. URL http://arxiv.
18220117711000.... arXiv:1810.08272 [cs].
François Chollet. On the Measure of Intelligence, November 2019. URL httppp//aarxvv..rr//////11111111 arXiv:1911.01547 [cs].
Shihan Deng, Weikai Xu, Hongda Sun, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, Rui Yan, and Shuo Shang. Mobile- An Evaluation Benchmark for LLM-based Mobile Agents, July 2024.
URL http://arxiv.. www./ogg///////oeee//./eeee/ee// 0aabs/2407.000 009933 arXiv:2407.008: [es].
Xiang Deng, Yu Gu, Boyuan Zheng, Shijic Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. MIND2WEB: Towards a Generalist Agent for the Web.
David Donoho. 50 Years of Data Science. Journal of Computational and Graphical Statis- 11111160002007.11000
tics, 2644)7745-766 October 2017. ISSN 1061-8600.
doi:
1080/ 10618600.100001 20111188000200...... 1384734 1010//0111000000110000000000 URL /ttp://oorggg1100000011016666000
Publisher: Taylor & Francis _eprint:
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?, April 2024. URL attp://arxiv wwwpxiivvorrg/ag /403 07718 arXiv:2403.07711 [cs].
Epoch AI. 1/9 OpenAl's ol- ol-preview outperforms previous models, including GPT-4o, on many reasoning tasks. But how much of this is due to increased inference compute? Our experiments suggest that naively scaling inference compute isn't cnough to bridge the gap between GPT-4o and ol. httpss//tt.o//QQS5H0Q00 September 2024. URL https: //xx http:/xxxcco//////// wwt./coo///p/eeee http://pocch.teearrcchhhtsttaaaa 0aaeeerss//8882201176666658338 htt.a////.....aeeecaaaaaaa//////
lason Gabriel, Arianna Manzini, Geoff Keeling, Lisa Anne Hendricks, Verena Rieser, Hasan Iqbal, Nenad Tomašev, Ira Ktena, Zachary Kenton, Mikel Rodriguez, Seliem El-Sayed, Sasha Brown, Canfer Akbulut, Andrew Trask, Edward Hughes, A. Stevie Bergman, Renee Shelby, Nahema Marchal, Conor Griffin, Juan Matcos-Garcia, Laura Weidinger, Winnie Street, Benjamin Lange, Alex Ingerman, Alison Lentz, Reed Enger, Audrew Barakal, Victoria Krakovna, John Oliver Siy, Zeb Kurth-Nelson, Amanda McCroskery, Vijay Bolina, Harry Law, Murray Shanahan, Lize Alberts, Borja Balle, Sarah de Haas, Yetunde Ibitoye, Allan Dafoe, Beth Goldberg, Sébastien Krier, Alexander Reese, Sims Witherspoon, Will Hawkins, Maribeth Rauh, Don Wallace, Matija Franklin, Josh A. Goldstein, Joel Lehman, Michael Klenk, Shannon Vallor, Courtney Biles, Meredith Ringel Morris, Helen King, Blaise Agüera y Arcas, William Isaac, and James
14

---

Published in Transactions on Machine Learning Research (05/2025)
Manyika. The Ethics of Advanced AI Assistants, April 2024. URL http::tp///arr..or//////2020... arXiv:2404.16244 [cs].
Michael Hassid, Tal Remez, Jonas Gchring, Roy Schwartz, and Yossi Adi. The Larger the Better? Improved LLM Code- Generation via Budget Reallocation, July 2024. URL
arXiv:2404.00725 [cs).
Matthew Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi Yuan. Interactive Fiction Games: A Colossal Adventure. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05): 7903-7910, April 2020. ISSN 2374-3468. doi: 10.1609) 10.1609/aaai.v332 v3ai..3410566997 URL https://ojs aaai org/ index.
Number: 05.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep Reinforcement Learning That Matters. Proceedings of the AAAI Conference on Artificial Intelligence, 32 (1), April 2018. ISSN 2374-3468. doi: 10.1609/ aa//aaai.. 192211111699 11694. URL https://ojs aaai. org/ index php/AMal//atticcle/rree///111 Number: 1.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding, January 2021. URL http:://aae //arvvv. org/abs// /bs/ 2009. 03300 MrXiv:2009.03300 [os].
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large Language Models Cannot Self-Correct Reasoning Yet. October 2023. URL https
//openreview. http://vwwe..ettttoooommm//d/ddd
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. MeAggetBench: Evaluating Language Agents on Ma- chine Learning Experimentation, April 2024. URL http://arxiv wtwpxiii.orrgbbb//////0 03302 arXiv:2310.08302 [cs).
Lujain Ibrahim, Saffron Huang, Lama Ahmad, and Markus Anderljung. Beyond static AI evaluations: advancing human interaction evaluations for LLM harms and risks, May 2024. URL http://arxiv.org abs/2405. 10632 arXiv:2405.1033 [cs).
Carlos E. Jimencz, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, October 2023. URL https
Subbarao Kambhampati, Karthik Valmockam, Lin Guan, Kaya Stechly, Mudit Verma, Siddhant Bhambri, Lucas Saldyt, and Anil Murthy. LLMs Can't Plan, But Can Help Planning LLM--Modlo Frameworks, February 2024. URL httt:://axxxi wwtpiiv.orraabb////e00 01817 arXiv:2402.01817 [es].
Omar Khattab, Christopher Potts, and Matei Zaharia. Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval. In Advances Neural Information Processing Systems, volume 34, pp. 27670 27682. Curran Associates, Inc., 2021. URL https::/pproceeenngs..ne wwtp:sseerrrprp..cpeeeeeeee ftaperrriiee/ppper 2021 202.hh/aeeeeeee00fff0ssse66ee222 Abbp:ssttttd---bbbaatt.t.....a
Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines, October 2023. URL http:://arxi http:///www....//////211./00
arXiv:2310.03714 [cs].
Nathan Lambert. The end of the "best open LLM", April 2024. URL https: w/w. www.intercooneets.........o. ai/p/ Computee-efficcieett--e----
Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye. More Agents Is All You Need, February 2024. URL httt:://aree /tt////wwww...r///////////// 05120] arXiv:2402.05120 [es].
15

---

Published in Transactions on Machine Learning Research (05/2025)
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' l'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Compelition-Lewe Code Generation with AlphaCode. Science, 378 (6624):1002-1097, December 2022. ISSN 0036-8075, 1095-9203. doi: 10. 10.1126 /..112//sciencc.aaq11555 URL http://arxiv.. org/ aaaaaba/222 faaba/2203..m 07814 arXiv:2203.078L1 [cs].
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang. Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic Evaluation of Language Models, October 2023. URL http: /ttp://arron //tp:////xxvv./// org/ abs/ 2211 09110 S.1iv:2211.091111 [cs].
Zachary C. Lipton and Jacob Steinhardt. Troubling Trends in Machine Learning Scholarship, July 2018
URL http://arxiv. wearriii..orraaeee /b////////110/33//// arXiv:1807. 00344331011111 [cs, stat].
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as Agents, October 2023. URL http: /tap:////xxvvv org/abs. 2308 2308.03688
03688 arXiv:2308. 03688 [cs].
Xiao Liu, Tianjic Zhang, Yu Gu, Iat Long Iong, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, Jiadai Sun, Xinyue Yang, Yu Yang, Zehan Qi, Shuntian Yao, Xueqiao Sun, Siyi Cheng, Qinkai Zheng, Hao Yu, Hanchen Zhang, Wenyi Hong, Ming Ding, Lihang Pan, Xiaotao Gu, Aohan Zeng, Zhengxiao Du, Chan Hee Song, Yu Su, Yuxiao Dong, and Jie Tang, VisualAgenttench:: Towards Large Multimodal Models as Visual Foundation Agents, August 2024. URL http::ttp:/patt http:://axxvvvvv org/ /b/// arXiv:2408.06322 [cs).
Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma. Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang. ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities, August 2024. URL http://arxiv.org/
abs/2408. 04682 arXiv:2408.04682 [cs).
Xing Han Lü, Zdeněk Kasner, and Siva Reddy. WebLINX: Real-World Website Navigation with Multi-Turn Dialogue, September 2024. URL http://arxiv. http:/arxivv..or//re. wwwggabs/2402..00 05930 arXiv:2402.05930 [cs].
Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. Boart:oord: An Analytical Evaluation Board of Multi-turn LLM Agents, January 2024a. URL http://arxiv.. org/ abs/ /401..2411.00 13178. arXiv:2401.33178 [es].
Xinbei Ma, Yiting Wang, Yao Yao, Tongxin Yuan, Aston Zhang, Zhuosheng Zhang, and Hai Zhao. Caution for the Environment: Multimodal Agents are Susceptible to Environmental Distractions, August 2024b. URL http::/arxiv.orgg org/ aww/aa/s///a// 02544. arXiv:2408.025. 02544 [cs].
Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeotsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. DiscoveryBench:: Towards Data-Driven Discovery with Large Language Models, July 2024. httttp:///axxi..rr///aaa arXiv:2407.01725 [cs].
Timothy R. McIntosh, Teo Susnjak, Tong Liu, Paul Watters, and Malka N. Halgamuge. Inadoquacics of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence, February 2024. URL http://arxiv.. www.ooogaaaa.........o..... haabs/2402.000 09880 arXiv:2402.09880 (cs).
16

---

Published in Transactions on Machine Learning Research (05/2025)
Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for General AT Assistanis, November 2023. URL httpaaaaiwi.orrra///1aa111 .t11//291111111133
arXiv:2311.12963 [cs].
Mistral AT team. Cheaper, Better, Faster, Stronger, April 2024. URL http:://iistral.al///ess
mixtral-8x209 8x22b/
Section: news.
Meredith Ringel Morris, Jascha Sohl-dickstein, Noah Fiedel, Tris Warkentin, Allan Dafoe, Aleksandra Faust, Clement Farabet, and Shane Legg. Levels of AGI for Operationalizing Progress on the Path to AGI, June 2024. URL http:://aaxxi..og/bbb/////// 02462 arXiv:2311002662 [cs].
Silen Naihin. agbenchmnrrk: Benchmarking the performance of agents far and wide, regardless of how they are set up and how they work.
National Academies of Sciences, Engineering, and Medicine. Reproducibility and Replicabilily in Science. 2019. doi: 2117226//53033 URL httal/pp:e//ppp.maiiooaaaaaaaaae http:///gggeee////////scrarr////
Andrew Ng. Welcoming Diverse Approaches Keeps Machine Learning Strong, June 2024. URL http::ttp///// / www deeplearning.
OpenAT. Introducing OpenAT o1, September 2024a.
URL
mntrootccngg-ppnai------rre
OpenAL. Learning to Reason with LLMs, September 2024b.
URL https://openal. wwtpe/eann...mmimmomiaeeeeee /ndex/
Learning- tor----tto-raaoo-it----------
Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, and Zhengyang Wu. WebCanvas: Benchmarking Web Agents in Online Environments, July 2024. URL http:///rxiv..c org.oog///aaaoo htgg/bbs/24006. 12373 arXiv:2406.12373 [cs].
Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, and Tatsunori Hashimoto. Identifying the Risks of LM Agents with an AMMEmulated Sandbox, May 2024. URL http://arxiv, orggabs/23099 15817 arXiv:2309.158 [cs].
Stuart Jonathan Russell and Peter Norvig. Artificial Intelligence: A Modern Approach. Prentice Hall, 1 edition, 1995. ISBN 978-0-13-1038065 Google-Books-ll CUVeMWAACAAJ.
Yonadav Shavit, Cullen O'Keefe, Tyna Eloundou, Paul McMillan, Sandhini Agarwal, Miles Brundage, Steven Adler, Rosic Campbell, Teddy Lee, Pamela Mishkin, Alan Hickey, Katarina Slama, Lama Ahmad, Alex Beutel, Alexandre Passos, and David G Robinson. Practices for Governing Agentic Al Systems. December 2023. URL httph: http:////.oo.oo.iiii...omoo openai http://ppppppppccccccooooooorono
seaysstes..ddf pdf
Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Yueting Zhuang. TaskBench: Benchmarking Large Language Models for Task Automation, December 2023. URL http:://aaxxv.orr//aaa////11111 2311. 18760 arXiv:2311. [es].
Quan Shi, Michael Tang, Karthik Narasimhan, and Shunyu Yao. Can Language Models Solve Olympiad Programming?, April 2024. URL http://arxiv.ort htrgg/ab//2444... 20004.1..29.2.. arXiv:2404.10952 [cs] version: 1.
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language Agents with Verbal Reinforcement Learning, October 2023. URL http:
11366, LrXii:2303.11366 [es].
Mohit Shridhar, Xingdi Yuan, Alex-Ale Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht.
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. March 2021. URL
http://arxiv. www.aaiivvvoora//aaa wttpg/abs/201....o 03766 arXiv:2010.03768 [cs].
Zachary Siegel, Sayash Kapoor, Nitya Nagdir, Benedikt Strocbl, and Arvind Narayanan. CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark, September 2024. URL http: http//axivvvggro//e// http://abb//299...13.9 arXiv:2409.11360 (cs).
17

---

Published in Transactions on Machine Learning Research (05/2025)
Simons Foundation. Mark Liberman Reproducible Research and the Common Task
Method (April 1. 2015), July 2019.
URL https://www https://ww..smmossttt www.::ww....ooooommmaaaii..oomo. wwwp:ttiiiiiong/eee.t.e.o
reproducible -aceearrch-------ee tea--thecoommee task tast-ttaakhteaee method/
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM Test-Time Compute Optimally can be
More Effective than Scaling Model Parameters, August 2024. URL http:///arxxwv.or///aa/////2 arXiv:2408.0330 [cs].
Paloma Sodhi, S. R. K. Branavan, Yoav Artzi, and Ryan McDonald. SteP: Stacked LLM Policies for Web Actions, April 2024. URL httt::/aaxxv.oorg//aaaa
03720 arXiv:2310.03720 [cs].
Olly Styles, Sam Miller, Patricio Cerda-Mardini, Tanaya Guha, Victor Sanchez, and Bertie Vidgen. WorkBench: a Benchmark Dataset for Agents a Realistic Workplace Setting, August 2024. URL htttp://arxiv.ooom org/
abs/2405. 0002 arXiv:2405.00x: 00822 [cs].
Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan. Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision, March 2024. URL http:
09472 arXiv:2403.09472 [os].
OpenDevin Team. Devnne An Open Platform for Al Soft Developers as Generalist Agents, June 2024. URL.
comm//pppaaeeerra//ppplllliii original-date: 2024-03-13T00:33:3122.
Mudit Verma, Siddhant Bhambri, and Subbarao Kambhampati. On the Brittle Foundations of ReAct Prompting for Agentic Large Language Models, May 2024. URL http:///axiv..or//ab// 13966 13966 arXiv:2405.13966 [cs].
Pablo Villalobos and David Atkinson. Trading Off Compute in Training and Inference, July 2023. URL https://epochai.
Kiri Wagstaff. Machine Learning that Matters, June 2012. URL http://arxii.oog//bb////a
arXiv:1206.466: [cs, stat].
Cunxiang Wang, Ruoxi Ning, Boqi Pan, Tonghui Wu, Qipeng Guo, Cheng Deng, Guangsheng Bao, Qian Wang, and Yue Zhang. NovelQA: A Benchmark for Long- Range Novel Question Answering, March 2024a.
URL http:://art:/t httpp://arxxv.orr/abb////// orgg/bs/////..../ 12766. arXiv:2403.12 12766 [cs].
Luyuan Wang, Yongyu Deng, Yiwei Zha, Guodong Mao, Qinmin Wang, Tianchen Min, Wei Chen, and Shoufa Chen. BobileAgenttencch: An Efficiont and User-Friendly Benchmark for Mobile LLM Agents, June 2024b.
URL http:://arddv..rg/abb/////
08184 arXiv:2406.081 [cs].
Ruoyao Wang, Peter Jansen, Marc-Alexandre Côué, and Prithviraj Ammanabrolu. Workd: Is your Agent Smarter than 5th Grader?, November 2022. URL http:///axxv..crr//b//000....0 07540 arxiv:220000 arXiv:2208.07540
[es].
Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. MINT: Evaluating LLMss in Multi-turn Interaction with Tools and Language Feedback, March 2024c. URL
http://arxiv. wwtpiiiiiv.ogg/bbbaea 2309 13691 arXiv:2309.1063 [cs].
Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, Ilia Kulikov, and Zaid Harchaoui. From Decoding to Meta Generatin:: Inference-time Algorithms for Large Language Models, June 2024. URL http::/arxivv wttpaaawwiii.rraa/////////////// 16838 arXiv:2406.16883 [es].
Lilian Weng. LLM Powered Autonomous Agents, June 2023, URL
2023-06-2222 23-02222-ntt/ Section: posts.
Ruixuan Xiao, Wentao Ma, Ke Wang, Yuchman Wu, Junbo Zhao, Haobo Wang, Fei Huang, and Yongbin Li. FlowBench: Revisiting and Benchmarking Selkffoo--Guide Planning for LLM-based Agents, June 2024. URL httt:://aarxvv.or/aaa////eeee wwtp:///.///.....ooas 14884. arXivv2406.1488 [cs].
18

---

Published in Transactions on Machine Learning Research (05/2025)
Jian Xie, Kai Zhang, Jiangjic Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. TravelPlanner: A Benchmark for Real-World Planning with Language Agents, February 2024. URL
http://arxiv.org/ ht///bb//00
arXiv:2402.01622 [cs].
Kevin Xu, Yeganeh Kordi, Tanay Nayak, Ado Asija, Yizhong Wang, Kate Sanders, Adam Byerly, Jingyu Zhang, Benjamin Van Durme, and Daniel Khashabi. Tur Bench A Challenge Benchmark for Web Agents, September 2024a. URL http: http::///wwwi.../ab////////ss/// ////////b////... 11905 arXiv:2408.11905 [es].
Tianqi Xu, Linyao Chen, Dai-Jie Wu, Yanjun Chen, Zecheng Zhang, Xiang Yao, Zhiqiang Xie, Yongchao Chen, Shilong Liu, Bochen Qian, Philip Torr, Bernard Ghanem, and Guohao Li. CRAB: Cuvroment Agent Benchmark for Multimodal Language Model Agents, July 2024b. URL http://arxiv.org/ ar arXiv:2407.01511 [cs].
2407 01511
John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-AGENT: Agent-Compulee Interfaces Enable Automated Software Engineering. 2024. URL
http://swwaagent.cooo////
pdf
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and
Christopher D. Manning. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering, September 2018. URL http:://aa//ii..... org/ http:/abb//19999906 arXiv:1809.09600 [cs].
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. WebShop: Towards Scalable World
Web Interaction with Grounded Language Agents, February 2023. URL http://arxiv. 2b2/ 2207
01206 arXiv:2207.0122 [cs].
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan, SttauS-bench: A Benchmark for Tool- Agent-User Interaction in Real-World Domains, June 2024. URL htttp://arxxi.org/abbb//// 12045
arXiv:2406.12045 [cs].
Guoli Yin, Haoping Bai, Shuang Ma, Feng Nan, Yanchao Sun, Zhaoyang Xu, Shen Ma, Jiarui Lu, Xiang Kong, Aonan Zhang, Dian Ang Yap, Yizhe zhang, Karsten Ahnert, Vik Kamath, Mathias Berglund, Dominic Walsh, Tobias Gindcle, Jucrgen Wiest, Zhengfeng Lai, Xiaoming Wang, Jiulong Shan, Meng Cao, Ruoming Pang, and Zirui Wang. MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains, August 2024. URL http:://axxv..rg///////////
18961 ar arXiv:2407.18961 [cs].
Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya Ben Bogin, Ofir Press, and Jonathan Berant. AssistantBench: Can Web Agents Solve Realistic and Time- Hime-Consuming Tasks?, July 2024. URL http:
arXiv:2407.15711 [es].
Matei Zaharia, Omar Khattab, Lingjaa Chen, Jared Quincy Davis, Heather Miller, Chris Potts, James Zou, Michael Carbin, Jonathan Frankle, Naveen Rao, and Ali Ghodsi. The Shift from Models to Compound AI Systems, February 2024. URL URLhtttp:///// hhtt:://baiit http:::///bairrbebeeeeeereee httpp////eeaelllloo/////////////
Andy K. Zhang, Neil Perry, Riya Dulepet, Eliot Jones, Justin W. Lin, Joey Ji, Celeste Menders, Gashon Hussein, Samantha Liu, Donovan Jasper, Pura Peetathawatchai, Ari Glenn, Vikram Sivashankar, Daniel Zamoshchin, Leo Glikbarg. Derck Askaryar, Mike Yang, Teddy Zhang, Rishi Alluri, Nathan Tran, Rinnara Sangpisil, Polycarpos Yiorkadjis, Kenny Osele, Gautham Raghupathi, Dan Boneh, Daniel E. Ho, and Percy Liang. Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risk of Language Models, 2024a. URL
http://aaxxvv..o//bbbbss.. 240.bb/0062000028622 Version Number: 1.
Yaolun Zhang, Yinxu Pan, Yudong Wang, and Jie Cai PyBench: Evaluating LLM Agent on various real-world coding tasks, August 2024b. URL
16732 arXiv:2407.1673 [cs].
Lily Zhong, Zilong Wang, and Jingbo Shang. LDB: A Large Language Model Debugee via Verifying Runtime Exccution Step-by-step, April 2024. URL
arXiv:2402.16906 [cs].
Andy Zhou, Kai Yan, Michal Shlapentokh-
Haohan Wang, and Yu-Xiong Wang. Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models, December 2023. URL
http://aaxiv.or//aba/3000 www//ba/////ae.ee 04406 arXiv:2310.04406 [cs].
19

---

Published in Transactions on Machine Learning Research (05/2025)
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A Realistic Web Environment for Building Autonomous Agents, April 2024. URL http::/aaxx...rrg////// [cs].
13854. arXiv:2307.3855
Appendix
Broader Impact Statement
Our work enhancing AI agent evaluations carries significant societal implications. By improving the efficiency and reliability of these systems, associated with their deployment, fostering broader accessibilily and encouraging cost-sensitivity among developers. However, the increasing sophistication of Al agents also raises safety risks. While our work docsn't directly address these risks, we firmly believe that cxisting frameworks for governing agentic AI (Shavil el al. 2023), are crucial for mitigating potential harms. Developers and deployers must prioritize the implementation of these frameworks ensure responsible development and deployment. Furthermore, our work on cost measurement can help improve safety evaluation. By providing tools to assess the affordability of potentially dangerous capabilities, research help identify and anticipate safety concerns before they become widespread. This is why AT safety benchmark developers should include cost. measurements.
potentially reduce the economic and environmental costs
A Additional details on Section 2: Al agent evaluations must be cost-controlled
We include four figures below: (i) Figure shows the results of our HumanEval analysis along with error bars for accuracy and cost; (ii) Figure 4 shows the results with the y-axis from 0 to 1 (in other figures, the y-axis is clipped between 0.7 and 1 for clarity); (ini) Figure 0 the results of our robustness checks with the June 2023 versions of GPT-3.5 and GPT-4; (iv) Figure 5 illustrates the results of the time vs. accuracy Pareto curve.
the figure reporting our results in the main text, we include the convex hull of points on the Pareto frontier because, given any two agents on the fronticr, we can always choose a strategy that picks agent with probability p and agent with probability 1-p and to achieve any tradeoff represented by points on the convex hull.
20

---

Published in Transactions on Machine Learning Research (05/2025)
150
Figure 3: Error bars for our HumanEval analysis. The figure shows accuracy vs. API cost for the HumanEva results zoomed in on the y-axis (0.7-1). The error bars represent 95% confidence intervals (left /lower; brown) and the minimum and maximu values (right/ /upperp gray), of accuracy and total cost across 5 runs. To calculate the 95% confidence intervals, we used the Student's 's t distribution given that we only have five runs per agent.
Min-Maa Max
Baseline agent Zero Zero-shot model
Complex agent
95%
145
140
135
ATS JATS (GPTT)
Cost (USD, measured in April 2024)
130
125
120
LAFS LATS (SPT:333)
LDU (Rallssionn GPT-3.5)
We mini (OPT-4) Retry (GP .4)
LEB (GP113.8)
Escolation
Reflasion
GPT4
GPT-3.5
1.00
0.95
(GP1-4
0.90
0.85
0.80
0.75
0.70
21

---

Published in Transactions on Machine Learning Research (05/2025)
Warming (GPT-4)
1.0
LDB (GPT-4)
LDB (Reflexion, GPT-4)
Retry (GPT-4)
GPT-4
LDB (GPT-4, GPT-3.5)
Escalation
LDB (Reflexion, GPT-3.5)
LATS (GPT-4)
0.8
Reflexion (GPT-4)
LATS (GPT-3.5)
LDB (GPT-3.5)
0.6
GPT-3.5
Accuracy
0.4
0.2
Complex agent
Baseline agent
Zero-shot model
0.0
0
20
40
60
80
100
120
140
Cost (USD, measured in April 2024)
Figure 4: HumanEval results with complete x- and y-axis. The figure shows accuracy AP1 costs with a complete x- and y-axis. This plot showcases the wide range of costs associated with different approaches, especially when considering LATS.
22

---

Published in Transactions on Machine Learning Research (05/2025)
1.00
0.95
Warming (GPT-4)
LDB (GPT-4)
Retry (GPT-4)
LDB (GPT-4, GPT-3.5)
LDB (Reflexion, GPT-4)
0.90
GPT-4
LDB (Reflexion, GPT-3.5)
Accuracy
Reflexion (GPT-4)
LATS (GPT-4)
0.85
Escalation
0.80
LDB (GPT-3.5)
LATS (GPT-3.5)
0.75
Complex agent
Baseline agent
GPT-3.5
Zero-shot model
0.70
//
1000
2000
3000
4000
5000
20000
21000
56000
57000
Time (Seconds, measured in April 2024)
Figure Accuracy vs. inference time curves for HumanEval. This figure shows the accuracy vs. inference time results linear x-axis scale, with the y-axis clipped to 0.7-1 for clarity. Time measurements refer to the mean of the sum of inference limes across all API calls made by the agent across the five runs.
Table 2: Accuracy and total cost of HvaanEval agents. We each agent five times and report the mean accuracy and the mean total cost the 164 HumanEval problems. The minimum and maximun values are included in the parentheses. Rows marked by an asterisk are agent specifications that the authors did not evaluate in the original paper. In particular, the original LDB paper does not include tests with GPT-4 as the debugger.
Paper
Agent
Accuracy
Total Cost (USD)
Zhou et al. 2023)
LAIS (GPT-4)
88.0 (82.3-91.5)
134.50 (123.98-144) 13) 9.49 (8.94-9.89)
LATS (GPT-3.5)
80.4 (78.7-83.5)
Zhong al. 2024
LDB (GPT-1, GPT-3.5)
91.0 (89.0-92.1)
2.19 (2.14-2.25)
LDB (Reflexion, GPT-4)*
92.9 (91.5-95.1)
7.26 (6.19-9.63)
LDB (Reflexion, GPT-3.5)
88.9 (86.6-0.91.5)
4.19 (3.98-4.37)
LDB (GPT-4)*
93.3 (92.1-94.5)
6.36 (5.11-7.08)
LDB (GPT-3355
80.2 (78.7-82.9)
0.63 (0.56-0.77)
GPT-4 (haseline)
89.6 (87.8-90.9)
1.93 (1.91-1.95)
GPT-3.5 (baseline)
73.9 (72.0-76.2)
0.05 (0.05-0.05)
Shinn et al. (2023
Reflexion (GPT-4)
87.8 (86.0-90.9)
3.90 (3.76-4.13)
Our baselines
Warming (GPT-4)
93.2 (92.1-93.9)
2.45 (2.36-2.54)
Retry (GPT-4)
92.0 (91.4-92.7)
2.51 (2.46-2.56)
Escalation
85.0 (84.1-85.4)
0.27 (0.25-0.28)
23

---

Published in Transactions on Machine Learning Research (05/2025)
A.1 Implementation details
We used xpt-3.5-turbo-0125 for GPT-3.5 implementations and gpu-4-urbo--222--0--99 for GPT-4 implemen- tations. The underlying prices for GPT-3.5 and GPT-4 were 0.58 (1.58) and 10s (30S) per one million imput (output) tokens, respectively, as of April 2024. We describe all implementations in detail below. In addition to our analysis in the main text, we also conducted robustness checks with the June 2023 versions of GPT-3.55 and GPT-4 and found substantially similar results (see Appendix A.2 When we conducted our measurements in April 2024, the prices for the gpt-4-0613 were 30$ (60$) per one million input (output) tokens. Prices for spt-3.5-turbo-061 were identical to the Jamuary 2024 version 0.58 (1.58) and 108 (3DS) per one million input (output) lokens, respectively.
HumanEval version. In evaluating our baseline agent architectures HumanEval, we use the modified benchmark version provided in the LDB paper (Zhong et al. 2024). This version includes internal test cases for all 164 tasks (in the original benchmark, test cases were provided for only 161 of 164 tasks, as detailed in Section
GPT-3.5 and GPT-4. We implement the model baselines using the simple (zero shot; without agent architecture) strategy provided with the LDB paper (Zhong et al., 2024p. This includes a text prompt and the example tests accompanying the HumanEval coding problem inputs to the model.
LDB (Zhong et al., 2024). The LDB agent uses two language models: one for generating code and another for debugging. In all plots and throughout this empirical analysis, we use the nomenclature "LDB (Generator, Debugger)" to specify which models were used. If the same model served both functions, we list it only once within parentheses. We kept all parameters as specified in the code accompanying the original paper. particular, this means that the maximum number of iterations is set. to 10 and the temperature to zero b
LATS (Zhou et al., 2023). Based corrospondence with the authors, we set the maximum number of iterations to 8, the expansion factor to 3, and the temperature values for generating the function implementations to 0.8. The temperature for generating self-reflections and the internal unit tests was set to 0.2. The maximum number of internal test cases was set to 6 for runs with GPT-3.5 and 4 for runs using GPT-4. The difference in the number of internal test cases for GPT-3.5 and GPT-4 was not presented in the paper; we learned of this based on our correspondence with the authors
Reflexion (Shinn al., 2023). We left all parameters unchanged from the ones provided in the original repository, setting the maximum number of iterations to 2, expansion factor to 3, and temperature to zero for generating function implementations. The temperature used for generating the internal tests and self-reflections is 0
Retry This baseline uses the simple strategy implemented in the code accompanying the LDB agent for zero-shot evaluations of language models (i.e., there is no agent architecture). We used this strategy to repcatodly prompt the same language model, keeping all parameters equal across retrials, as long as the code outpulled by the model failed at least one of the example tests. If at any point solution passes the tests given in the HvmanEval problem description, problem. We repeated this procedure for up to 5 trials and stopped early if the code passed all the given tests. We set the temperature to zero. This can still lead to success after the first trial since LLMs aren't deterministic even at temperature zero. 8
evaluate this the final solution of the agent for this
Warming. For the warming baseline, we modify the retry baseline by gradually increasing the temperature parameter across successive trials. Initially, the temperature was set at zero, mirroring the retry baseline. For the second and third trials, we raised the temperature to 0.3, and for the final two trials, we increased it
"For the code version used in this study, which the original authors made available under Apache-2.0 license see: https://githu:
For the code version used in this study, which the original authors made available under MIT license https
//github.co
For the code version used mn this study, which the origial authors made available under an MIT liceuse sec:
https
//github
"See http:://commu.ty.oopm.... openai.
same- wwwee/uuuuuaaauuuuuueeeeeeaaaeee many-
24

---

Published in Transactions on Machine Learning Research (05/2025)
further to 0.5. If at any point a solution passes the tests given in the HumanEval problem description, we evaluate this as the final solution of the agent for this problem.
Escalation. We modify the simple strategy but switch the underlying model to a more expensive one if a proposed solution fails at least one of the example tests. We progressively escalated unsolved problems up a model chain of increasing cost [lama-3-8b-chat-h,, gpt-3.5-turbo-01225 04-09). All other parameters are kept constant across trials in particular, temperature is set zero. If at any point solution passes the tests given in the HumanEval problem description, evaluate this as the final solution of the agent for this problem. This leads to slightly lower accuracy compared to GPT-4, since some solutions from cheaper models might pass the example tests but fail one of the evaluation tests.
hama-3-70b-chan
gpt-4-turbo20224
Pareto frontiers. In our analysis, Pareto frontiers are employed to evaluate agent designs. We define the Pareto frontier as the set of points (agents) that are non-dominated by any other agent in terms of mean cost and accuracy. The frontier is constrained be convex, mcaning if two agents lic next to each other on the frontier, any linear combination of these agents should also yield point that lies on the frontier curve. We provide a simple example implementation of how we compute Pareto frontiers on simulated agent evaluation data
A.2 Robustness checks with June 2023 versions of GPT models
One concern with our analysis is that we use the latest versions of OpenAl's April 2024 turbo models, since later versions of GPT-3.5 and GPT-4 might have more scope for contamination. To address this, we conduct. additional robustness checks with the June 2023 versions of GPT-3355 and GPT-4. We find substantially similar results for this version: complex agents are no better than our simple agent baselines while cost orders of magnitude more in some cases. Note that the June 2023 version of GPT-4 is much more expensive than the April 2024 version, leading to the big difference in inference cost. For the LATS agents, there are some significant outliers across lasks for both, LATS (GPT-4) and LATS (GPT-3.5), with some lasks requiring more than 2 hours to complete for GPT-3.5. Overall, there were more extreme outliers for LATS (GPT-335) than LATS (GPT-4). For the same reason, we had to exclude one task (i.e., HumanExal/83) from the analysis for LATS (GPT-3.5), which did not stop running even after 5 hours. We marked this task as incorrect and excluded the time and cost. from the results shown above. HumanEval/83 was one of the tasks excluded from the subset of HumanEval that the authors evaluated the LATS agent on.
1.90
11
1.00
11
WWoming (GP1-11D8 (GPT-))
Wonning (raiT-a) LIB (CPT(4)
0880
0.940
Retry (BPT44
LDS (Hotteston GFT-4)
LDg (Retession GPT-4)
Noturacy
0.85
Excalasion
CPT-
LDa (Retexdon, GPT-
Necutary
U.28
Excratt on OPT4
LDO (CiM-4, CPT:5.5)
LDB (Refecon, GPT-3.0)
Hellesion (5P1-4)
LATS (GPT-4)
LATS (0PT:4)
0.80
LDB (GPT-15)
0.50
LDB (3
Awlsoion (PPT-4)
LATS (GPT a)
LATS (GM13.5)
S.
Complex agent
Baselin agent
GPT-3.5
Complex apent
Baseline agent
Simple
0.73
0.70
1000
2020
300000 Time (Sezonds, meesure Azri 2024)
3000
4000
5000
45003
40000
Cost (USD messured Aorl 2024)
(a) Accuracy vs. inference cost
(b) Accuracy vs. inference time
Figure 6: Robustness checks with June 2023 versions of GPT models. This figure shows the results of our robustness checks with linear x-axis and y-axis clipped to 0.7-1 for clarity. Time measurements refer to the mean of the sum of inference limes across all APT calls made by the agent across the five runs. Cost and accuracy measurements refer to the mean across five runs.
cattteer__aappeeeeeee ipynb
25

---

Published in Transactions on Machine Learning Research (05/2025)
Table 3: Accuracy and total cost Hvaa robustness checks with June 2023 versions of GPT models. As for the main analysis, we run each agent five times and report the mean aceuracy and the mean total cost on the 164 HumanEval problems. The minimum and maximum values are included in the parenthesos. Rows marked by asterisk are agent specifications that the authors did not evaluate in the original рaрer.
Paper
Agent
Accuracy
Total Cost (USD)
Zhou al.
2023
LATS (GPT-1)
83.5
360.02
LATS (GPT-3.5)
77.4
8.97
Zhong et al.
2024
LDB (GPT-4, GPT-3.5)
88.7 (87.2-89.6)
3.20 (3.14-3.26)
LDB (Reflexion, GPT-4)
90.5 (89.6-91.5)
18.01 (16.15-20.80)
LDB (Reflexion, GPT-3.5)
86.7 (85.4-88.4)
8.71 (8.38-9.13)
LDB (GPT-4)*
91.2 (90.2-92.7)
9.31 (8.27-10.26)
LDB (GPTT-3.5)
82.1 (80.5-84.1)
0.68 (0.65-0.72)
GPTT-4 (baseline)
86.5 (84.8-87.2)
2.94 (2.92-2.95)
GPT-3.5 (baseline)
75.2 (71.4-76.2)
0.04 (0.04-0.01)
Shinn al. (2023)
Reflexion (GPT-4)
80.2 (78.7-82.3)
8.29 (7.98-8.69)
Our baselines
Warming (GPT-4)
(89.0-91.5)
3.88 (3.82-3.92)
Retry (GPT-4)
88.8 (97.2-896)
3.95 (3.78-4.09)
Escalation
87.6 (86.6-88.4)
0.42 (0.40-0.44)
B Additional details on Section 3: Jointly optimizing cost and accuracy can yield better agent designs
In this section, we include Figure 10 reporting the results of our analysis on HotPotQA along with error bars for accuracy and cost, as well as Table 4 which reports our results and inference and optimization cost for our five agent designs. In Fig. 9 we report the Pareto frontiers returned by our joint optimization method for both models.
Table 4: Accuracy and two types of cost of agent designs evaluated on HotPotQA. We evaluate each agent five times on the test set and report the mean accuracy. Variable cost refers to the cost per 100 inferences on HotPotQA. Fixed costs are the total costs incurred during the optimization of the agent design. The minimum and maximum accuracy are included in the parentheses.
Model
Agent
Accuracy
Cost (USD, measured in May 2024) Variable
Fixed
DSPy random search
0.495 (0.50-0.485)
0.376 (0.376-0.376)
2.696
GPT-3.5
DSPy few-shot
0.47 (0.46-0.48)
0.384 (0.384-0.385)
0.029
Joint optimization
0.509 (0.475-0.54)
0.174 (0. (0.173-0.175)
2.714
Formatting instructions only
0.377 (0.36-0.39)
0.071 (0.070-0.0715)
Uncompiled
0.436 (0.395-0.485)
0.095 (0.094-0.097)
Llama-3-700 3-70B
DSPy random search
0.617 (0.60-0.63)
0.643 (0.64-0.644)
4.820
DSPy few-shot
0.622 (0.615-0.63)
0.661 (0.660-0.662)
0.028
Joint optimization
0.601 (0.59-0.62)
0.374 (0.372-0.378)
3.844
Formatting instructions only
0.527 (0.52-0.535)
0.111 (0.110-0.112)
Uncompiled
0.448 (0.43-0.46)
0.194 (0.194-0.195)
26

---

Published in Transactions on Machine Learning Research (05/2025)
0.7
Llama-3-70B (Format. Instr.)
Llama-3-70B (Few-Shot)
Llama-3-12 1ama---70BB (Joint Opt.)
0.6
GPT-3.5 (Joint Opt.)
Llama-3-70B (Random-Search)
GPT-3.5 (earch)-Search)
0.5
GPT-3.5 (Few-Shot)
Accuracy
Llama-3-70B (Uncompiled)
0.4
GPT-3.5 (Uncompiled)
GPT-3.5 (Format. Instr.)
0.3
95% CI
Min-Max
0.2
Joint optimization
Formatting instructions only Uncompiled
0.1
DSPy default
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Cost per 100 Inferences (USD, measured in May 2024)
Figure 7: Error bars for our HotPotQA analysis. The figure shows retrieval accuracy API cost for the HotPotQA results. The error bars represent 95% confidence intervals (left/lower; brown) and the minimum and maximum values (right/upper; /upper; gray) of accuracy and total cost across 5 runs. To calculate the 95% confidence intervals, we used the Student's t distribution given that we only have five runs per agent. Note that the error bars account for the in-sample variance on the test set and do not account for the variability induced during optimization and resampling.
0.32
0.32
0.30
0.30
0.28
0.28
0.28
0.28
Accuracy
Acouracy
0.24
0.24
0.22
0.22
0.20
0.20
0.18
0.18
All Programs
Programs Pareto Frontier
0.16
0.16
Pareto Frontler
0.14
0.14
0.00
0.02
0.04
0.06
0.08
0.10
0.12
0.14
0.16
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
demos per 100 inferences (USD)
demos per inferences (USD)
(a) GPT-3.5
(b) Llama-3-70B
Figure 9: Pareto frontiers returned by joint optimization. This figure shows the Pareto frontier of programs returned by our joint optimization method with linear x-axes and y-axes clipped to 0.14-0.32 for clarity. Note that the x-axes do not follow the same scale. Accuracies are calculated on the development set. Cost measurements refer to API prices as of May 2024.
27

---

Published in Transactions on Machine Learning Research (05/2025)
Total Cost (USD, measured May 2024)
14
12
Llama-3-70B
10
Llama-3-700
8
1332
GPT-3.5
6
GPT 3.5
1275
DSPy optimized few-shot
Joint optimizer
0
400
800
1200
1600
2000
Number of Inferences
Figure 8: With increased use, the total cost of rumning an agent is dominated by the variable cost. Precisely, the point of intersection is reached after 1332 (1275) inferences for Llama-3-70B (GPT-335)). We use the average cost of optimizing the models across our runs as the fixed cost and the average cost of running inference using the agent across the 200 tasks in the evaluation set as the variable cost per task. We refer Table for overview of exact cost and accuracy measurements alongside minimum and maximum values across runs.
0.7
(FemeShect)
Liams-i-/tae Instr) 1lem (Jaand Opt)
porelo jostii
GPF-3 (Random Search)
Limn
Search)
GPT-4 Fommat
GPT- (Unsomplec)
0.5
Accuracy
(uctt Opt.
GPT-3.6 SheO
0.4
Liamm-3-708 (Lncomppled)
(2/13 (Uncompled)
0.3
(oomatt
Joei opéiizatiionn
0.1
Uncompled
Usily denutt
0.0
0.0
Cost per 100 Inferences (USD, measured May 2024)
Figure 10: HotPotQA analysis with GPT-4. Agentic systems often bridge the accuracy gap between model classes while achieving significantly lower costs. Our joint optimization approach with Llama-3-70p achieves accuracy comparable to that of GPT-4 at less than half the cost. However, it's crucial to note that such improvements are not universal across all models and tasks. In some cases, the accuracy gap between different models cannot be closed with agentic workflows, such as for GPT-3.5 vs. Llama-3-700 3-70B this case.
B.1 Implementation details
For GPT-3.5 implementations, we used gpt-3.5-turbo-0125 via the Azure OpenAl endpoint, and for Llama-3- 70B implementations, used meta-llama/ meta--lama/LLaama---
via the Together.ai endpoint. The underlying prices for GPT-3.5 and Llama-3-70B were 0.58 (1.58) and 0.98 (0.98) per one million input (output) tokens, respectively, as of May 2024. Note that these costs vary over time, which would affect the outcomes of the rolled assessments as emphasized in Appendix A In addition to the details outlined Section we provide further details on implementations below.
Multi-Hop AaattiionAnswrring. For our analysis HotPotQA, we rely on multi-hop question answering agent design to produce answers given the questions given in the benchmark. We use the default.
28

---

Published in Transactions on Machine Learning Research (05/2025)
implementation provided by the DSPy framework that itself follows simplified version of the Baleen system (Khattab et al., 2021). This requires a. language model to reason across multiple documents and retrieve context leveraging a text corpus. We use the ColBERTv2 retriever model, which provides an endpoint to retrieve context from a 2017 Wikipodia dump containing each article's first paragraph[ TT The implementation is structured around three core functions: query generation, information retrieval, and answer generation. Initially, a search query is dynamically constructed for each iteration up to preset maximum number of hops, given the question and already retrieved context. This query then retrieves the top-k most relevant. paragraphs from Wikipedia. These documents are then deduplicated and added context in subsequent steps. Once the iterative process of query generation and information retrieval is complete, a language model reasons on the retrieved information given the question and generates the final answer.
For our experiments with this design of multi- question-answering we set the number of passages to retrieve and the number of hops to two. The maximum number of demonstrations is set to 8 (i.e., this includes the corresponding parameters for bootstrapped and labeled few-shot examples in DSPy). As HotPotQA contains ground-truth documents that the agent should retrieve for each task, we evaluate performance by determining whether all of the specified documents were retrieved. For optimization and evaluation, we randomly select 100 and 200 samples from HotPotQA for optimization and evaluation of the results, respectively. To ensure reproducibility, we set a fixed seed.
seattioonnnwerinn system in an uncompiled state. That is, the program is not optimized and the agent's prompt includes neither the few-shot examples nor instructions on how to format the generated output. Hence, each prompt only includes the instructions for the task and the main content including the retrieved context and the reasoning steps generated by the language model. No formatting instructions are provided.
Uncompiled. This baseline uses the outlined multi-hop
Formatting instructions only. This baseline is identical to the uncompiled agent design but includes instructions on how to format the output of each query part of the agent program. We use the default. formatting instruction part of DSPy.
We use the BootstrapPewShor (Few-Shot) and BootstrapFewShhotiin RandomScarch 13 (Random Search) optimizer from the DSPy framework to optimize our multi-hop question- answering agent. BootstrapFewShot iterates over training set samples to identify examples where the agent makes correct predictions based on the provided compilation metric. As metric, we ensure that the predicted answer matches the ground- ttunh and is included in the retrieved context. Further, none of the generated querics can exceed 100 characters and retricval querics must be sufficiently unique. For each successful prediction, the optimizer capt ures the trace of the prediction process, including inputs, predictor calls, and outputs. This metric also comes from the same DSPy tutorial [4 The resulting traces are used to create demonstrations for each step, which are then incorporated into the compiled agent. This process continues until all training examples have been considered or the maximum number of successful examples for inclusion has been reached. If the maximum number of successful examples has not been reached after iterating over the entire training set, the optimizer randomly selects the missing number of examples from the data to include as demonstrations in the prompt. Bootstrap WindoondooSerrrh repeats this process several times and conducts a random search over the generated demoustrations before selecting the best program on the development set. We set the number of candidate programs to 16.
DSPy default optimization.
12
Joint optimization. Our joint optimization allows us to trade off fixed and variable costs: we can spend money upfront searching the space of prompts and few-shot examples that minimize the variable cost while maintaining accuracy. It builds on the BootstrapFow implementation from DSPy. However, instcad of stopping once a specified mumber of maximum demonstrations is reached, our process keeps iterating over all samples in the training set to build a collection of few-shot examples that we can select from during optimization. Joint optimization of accuracy and the number of tokens of the few-shot examples and formatting instructions included in the prompt via parameter search is implemented with the Optuna Python
Scee: http////w......aee/////////////a 11 Scc: htapa://hooppoge.ghhubboooooo/is
12Scc: tttps://dapy-doss.verroo httpp//pww....oooooooooooo...oes
14 14See: http:///w.....eeeeoo////o///lala
http:://dapyydoss.eere....e wttp:///ww..eeeee.ppp.peeppppeee
29

---

Published in Transactions on Machine Learning Research (05/2025)
library (Akiba et al. 2019 Optuna supports multi-objective optimization to jointly maximize aceuracy and minimize the cost of our agent design 15 In our objective function required by Optuna, we sample values to search over the following parameters to find Pareto-optimal agent designs: (a) the temperature for each module within the agent, (b) the number of few-shot examples, (c) the selection of specific examples to include, and (d) whether add formatting instructions. Candidate lemperature values for each module
the agent pipeline are sampled from 0.0, 0.2, 0.4, and 0.6. We set the number of trials Optuna conducts 16. The maximum number of few-shot examples per prompt is set to 8.
We evaluated the Pareto-efficient programs that Optuna returns the dev set (50 samples). In figure 2 and 10 we report the mcan accuracy and cost of the program with the highest accuracy on the development set.
The goal is to balance the fixed cost of oplimization with the variable cost of rumning the agent, ensuring both cost-effective and accurate performance. The results from Optuna can themselves seen as a Pareto curve of agent designs.
C Survey on agent benchmarks
We provide detailed survey of 33 recent agent benchmarks, along with their levels of generality and respective holdout sets in Table 5] As shown in Table [ the holdout set must correspond to the level of benchmark generality to prevent gaming. However, many of the benchmarks in the survey do not have holdout sets at the appropriate generality level and benchmark designers do not express intentions include the appropriate holdout set for the benchmark in the future. To select agent benchmarks, we relied on the benchmarks included in AgentBench [Liu et al., 2023), AgantBoord (Ma et al. 2024a). OpenDevin (Team) 2024), and the ICLR 2024 workshop LLM agents in addition to benchmarks we had previously come across.
Note that WebArena uses the term "domain" to refer to different types of websites in their benchmark (e-commerce, social media etc.), which is different from how we use the term (e.g., all tasks in WebArena fall under the web interaction domain per our usage of the term).
Additional details on Section 4 Details about NovelQA implementation
We run evaluations on the multiple-choice subset of the NovelQA benchmark. In addition to re-evaluating the results of GPT-4 (the current state-of-the-art model on NovelQA multiple choice), we include an agent that comprises GPT-4 with retrieval-augmenned generation. Instead of feeding in the entire novel in the -ettt--eeeddigg--eeeee
context window of the model, we embed each novel using OpenAl's each multiple-choice question in the benchmark, we retrieve 10 chunks of 1000 characters cach, which we use inputs to the model. We used the prompt. from the original NovelQA paper, slightly modified to add context for the RAG snippets (Listing 1..
model. For
You are a literature professor. will provide you with snippets from a novel along with a question and corresponding choices partaining it. Please thoroughly analyze the content accurataly reapond to the queation.
Relevant snippets from the novel:
<context>
Question:
<question>
Only respond abould not contain anything elae.
index corract ansver (a-g., choose betvean A, B, c, and output
Prompt used for RAG implementation on NovelQA. This is the prompt from the original NoveQQA paper. We only make the slight modification of adding the retrieved context.
Listing 1:
15gee: http:///w//.oo...acccccc//////te
30

---

Published in Transactions on Machine Learning Research (05/2025)
We find that our retrieval agent performs substantially similarly to the GPT-4 model. In particular, GPT-4 with retrieval has an accuracy of 67.89 (total cost: $52.8 USD), whereas when we use the ent content of the novel as input tokens, the accuracy is 67.81 (total cost: $99.8 USD). Wang et al. (2024a) report an accuracy of 71% for GPT-4 in the paper introducing NovelQA. The difference might be due to the stochasticity of language models. Due the high cost of running the evaluation, we only ran our evaluations once. Irrespective of the small differences in the absolute value of accuracy, it clear from these results that the difference in accuracy between RAG-based approaches and long-context approaches is small. For comparing cost fairly in single question-answer setting, in theory, could query the language model around 2,300 times (the total number of questions) instead of 88 times (the total number of novels; as they implemented it) and incur a cost of around $2,590 for evaluating GPT-4. However, that's not how the authors evaluate models on NovelQA (and it's also not how they expect model developers to do it). Again, this choice is justifiable for model evaluation, since the purpose of long-context model evaluation is equally well served by having a long list of questions.
31

---

Wait Published in Transactions on Machine Learning Research (05/2025)
E
Additional details on Section 6: Agent evaluations lack standardization and reproducibility
We provide further details on our findings the limited reproducibility and lack of standardization in current agent evaluations. Table 7 details the identified issues for each agent on HumanEval and WebArena we reproduced for this study. Table g reports the corresponding numeric performance measurements as well as minimum and maximum values across runs for HumanEval.
E.1 HumanEval implementation details
LDB (Zhong et al., 2024). The LDB paper claims to use GPT-3.5 for code generation using Reflexion: "For Reflexion, we select the version based GPT-3.5 and utilize the corresponding generated programs published in the official Github repository." However, the generated program they used from the Reflexion repository relies on GPT-4 for code generation, not GPT-3.5. The authors acknowledge this and plan to update the paper to address it. In addition, LDB, such as LATS and Reflexion, all use different subset of HumanEval problems. Three (out of 164) coding problems in the original version of HumanEval lack example tests. Since LDB requires example tests to debug or rerun their solutions, the authors added example tests for the three problems that are missing the original benchmark.
LATS (Zhou et al., 2023). The LATS agent requires example tests well. However, for the three HumanEval problems that do not contain example tests, the authors remove these problems, plus another problem, for unreported reasons. The authors do not mention this in the original paper. In correspondence with the authors of LATS, they clarified: "Originally, there was an execution error when evaluating some test. cases for [one of the HumanEval test cases], so we opted to remove from our setting." They acknowledge that they didn't 't report this in the paper and will update the manuscript accordingly. In addition, their agent was evaluated on only a subset of the test. cases provided in the HumanEval benchmark. This exaggerated their accuracy numbers, since the code for a particular HumanEval problem might be incorrect, but if it passes only portion of the test cases for that problem, it could still be marked as correct. In our analysis, this was responsible for a 3% difference in accuracy (mean across five runs), which explains substantial part of the difference between the accuracy we found and the one reported in the paper.
Reflexion (Shinn et al. 2023). The authors remove the three problems of HumanEval that lack example tests and evaluate on only 161 out of the 164 tasks of the benchmark.
"objective"; "Find subreddit focused on topics related city Pittsburgh, and post my question " Vpplaces for new drivers to learn driving\" there",
"uri": "htsp://ec2 --131-2--3-4-777 us-astt-- cocputee anazonews, cooswssscoo..coo son.:999//ubbttpppttss
"abservation": "Tab 0 (current): Create Maniiesion/aNn(84311 RootWebArea " "Create subnissionn focusad: true\n\t[8474) Headerkallolladdaar StaticText [...]', "action": "stop [N/A]*,
(M)tt)t[8475) link "Hane ANAAAAAAt [8510]
"reason"; "The user's objective to post a question a subreddit focused on topics related the city of Pittsburgh. The observation indicates that the user is on the \"Create submission\" page for the Pittsburgh subreddit and has already filled in the title and body of the post. Hovaver, while before trying Ag This neans the action to post the question cannot be completed this "done": true, "revard": 1.0,
message is displayad stating, \*You cannot post
restriction or set by the vebsite.",
"auccesa": 1.0, "ntn_nctions"; 12
Listing 2: Example snippet from the original log files provided by the authors of the STeP agent. The snippet shows how the rate limit on the Reddit site affects evaluation. Given the observation from the web environment, the agent correctly concludes that it's currently not possible to make a posting given the rate limit imposed on the user. Despite the environment imposing this limit, the task is still evaluated as successfully solved. This is an example of incorrectly evaluated agents leading to inflated results.
32

---

Published in Transactions on Machine Learning Research (05/2025)
F Statement on compute resources used
For all our experiments using OpenAl models, we utilized the endpoints provided by OpenA cither directly or through the Azure OpenAl Service For the analysis on HotPoiQA using Llama-3 models, we relied on the endpoints provided by Together.ai 118 As our work primarily relied on external APIs, we did not use any GPUs for inference and our experiments did not require training of LLMs.
G Limitations
While our research advances the understanding of AI agent ovaluations significantly, we must acknowledge several limitations that future work could address. First, our proposed methods for rosl-controlled evaluations and the joint optimization of cost and accuracy rely on current cost models and technological constraints. These models are subject to change as technology evolves and new pricing models introduced, altering our results. We fully acknowledge this challenge to cost evaluation as proposed in Section 2 To address this downside, we provide dynamic web application accompanying the results of this paper, which allows users to modify the underlying cost of imput and output tokens for each model. This allows us to recalculate the underlying cost of each agent part of our analysis using current prices. We argue that this should be part of downstream evaluations of agents.
Second, while our study spans a range of benchmarks and agent models, it does not exhaustively cover all possible task environments and variations of AI agents. Still, we demonstrate that our findings the limited construct validity of current Al agent benchmarks and the lack of reproducibility and standardized evaluations hold across tasks and domains, providing empirical findings for multiple benchmarks and agent designs. We further describe why agent evaluations should be cost-controlled in tasks beyond programining Our empirical findings that joint optimization of cost and accuracy can lead to more cost-efficient agent designs on a question-answering benchmark emphasize this argument.
Finally, other lypes of costs, including environmental impact, human labor for data annotation, and maintenance costs of AT systems, have not been extensively analyzed. These factors are becoming increasingly relevant as Al systems scale up and are deployed more widely, necessitating more comprehensive approach to evaluating the full economic and environmental impact of AI.
H Reproducibility statement
We release code to reproduce all experimental results of this paper in GitHub repository under MIT license 19 This includes scripts to reproduce our analyses of HumanEval, HotPotQA, NovelQA, and WebArena, as well as implementations our proposed basclines and the DSPy implementation for our joint optimizer. As an example of an interface that lets downstream users explore the impact of varying API costs, we also provide an interactive web application This allows users input current pricing for different language models and visualize the adjusted cost-accuracy tradeoffs on HvnanEval for the agents we evaluated (Section 2 Finally, plan to release joint optimizer to the official DSPy repository and the research community. To show how we compute Pareto frontiers, we provide simple example implementation on simulated agent evaluation data 21
Sce:
https://api: http:///ap..ootthhrr..s////eeee
20
http:///wwccccooooaa..oaooooacaa wwwpeeexaeee...eeeee ipynb
21 See:
33

---

Published in Transactions on Machine Learning Research (05/2025)
Table 5: A survey of recent agent benchmarks shows that 17/33 benchmarks do not include holdout sets for evaluating agents' performance and do not indicate plans to do so. Of those with holdout sets, only 8/33 hold out sets are at the appropriate generality level. We made our best guess for the level of generality based on the description of the benchmark's purpose in its paper or repository.
Benchmark
Domain
Benchmark description
Level generality
Heldout
What held
Haldout the appropriate generality level?
Exammpe ideal holdont
act
CORE Hench
Science
Messuees agents" accuracy in compotatiionl reprohucibility
"Takk-ppciifc
distribtiion som-
Authors test porformannce on new mnwen papura from sumplex from the samn scinliiii distiplizas to the train set.
Papers from other scSontific disci- pline..
Singel of 2021
ples
Lokk utroo scinttiie disci- plane.
MlAAgrateench
Programiiing
Measures the accuracy of agents specifically machine lownnnng experimentation.
Task spectfic
N/A
Lacks test sct and doesn' indicate plans to make one.
Research taslss languages other than Pythor.
[uann a
X
SWE-Booch
Programming
Massuves the arcarncy of agunls spucifioully sulving suftware enginseriig fog problems. repos-
Taskoquecficc
In distriburtion sum- plos
The hilrbort unt curmently containe reposikaries not wom claring train- ing but otherw distribution as training. Authons mention plans to collect reporito ries in different programming lan- guagee, thrugg not exclusively for the hold-ent
Repusttorins in languages other thun Pytioe.
Yang 2024
simillar
Aushoms intend itories in the benchmarks be- yond the 12 instially sampled.
Web laok automation
Mussues the uccuracy of agents on many different web tasks.
Dammit-guurall
N/A
Lacks loodout set and doean't dicate plans to make one.
New welsites Luoks not wan during trolning, such as making plame or train travel bookings. N/A
Zhou et al.
2024
x
WehShop
coommeece
Messures the accuracy of agents sprcifically their ahil-
Task specticc
Out distribution samples
Authors evalisate sim- to real trans- for the agent, where they agent's perfarma now on real web- aites such us unsazonn com.
Yau
2025
to purchoue Heem from weln.
WinddWebb Deng al.
Web task automation
Messuzes the accuracy of agents many different web
Domoiaa general
TTakks
Authoos hold out two top- tevel do- Information and Service.
N/A
PPNL
Path plansing
Messuees the uccuracy of agents specifically on tlccls abil. ity to plan paths between
"Iakkapeilfic
Out dàtribbtion samples
Authoes test LLM performance out of discebbution tasies with diffes- cat grid sizes and obstacle counts.
N/A
Aghaal et al. 2024
TravePPlsnnwe
Trwel planning
Maseurex the apcuracy of agents spucifically on tiwir slill. ity to create travel plass.
Thak-speciffc
distributiin
heldoott inclosoo taoks wwen during iraining, but taay are not of a cifferent distribution than the training tasies.
Destimation queries with cilire, budguls, and lord con- stroints.
Xie ml 2031
plas
MINT Wanger et
Task actommation
Measures the tool angmented taek solvin capability of agents or general snt tuoks.
Domaln general
K/A
Lacles held set but authors mention
Additional tools that be used solve
202
x
updato bench- mark with new tasks and tools. Lacker a held-out set but sathors montion plans to update beuch-- mark with new domalzs taskas
unseen
types,
-bonch
Tuol-ggent-eerr intse- action.
Mosures agenial repucity to gather and communicate all necessary data to and from usens through repeated interse. tions and completing tacks. Meseures the arcuracy of syents web bodks across mul- donainn.
Dommisggemeall
N/A
Ankditional unowen thomsins aggent musi salve lasks in while inturact- ing with the and
Yau al. 2014
Merssha Yoeus all 2024
Web antomation
Demain-grmeaal
In distriburtion sam- plos
The held out we inchidos tioks not during trxining but of the sume distribaatinn the tenin The bold act includes tasks ro- quiring the narigation of unseen websttes belonging to unseen car- and
New website types and taxis from follk uroan during training.
Webliin
Converactoonl web navigation
Messuee the accuracy of ageazs on using conversatonall
Domoia-general
Taoks
N/A
La 00 al 2024
interfaces to navigate the web.
grographic lock- slong with Iaoks where the instrucior cannol aow the sersen.
CRAB
Crow- taolox
Messures an agent's iss ability to complete inve-envinonmmen books simmbtting the use of mnd- devicss simultaneousi. Measwes an agent's ability to complese tasks through tool use, navigate dependencies be- twoe tools, and handle com- muniestioo cas iroomnaaaa.
Domain- Domain-grneral
NyA
Lacies a held-out wee and doesn't indlivate plan to make con.
Thoks requiring the navigation of
Xu en al.
envinomment unseem during train- ing.
ToolSendbox
Tool
Domota general
N/A
Lackss hold-out acs and doesn't Indicate plans to malee one.
Thoks cocrespcading to additional
Lu of al. 2024
unseen test sccnario categories,
Visual-Agent Bench Lit et al 2024
Vision tasks
Measures the accuracy of agents on vision based tacks across different comains.
Dounalis general
NyA
Lackes a hold out acs and doesn't Indicate plans to malee one.
Tasks belongiag to unseen cavi. ronmeatt..
MabileApmit Bench
Android app nevign- tiom
she socuracy bile agents within the Au- drsid enayslemn.
Demain-grneral
Torles helésonn wet and doesn't 't indicde plas to makes one.
then training taoks quiring the use of ten spps, huld oul some for testing. Hather than training on all six domains, bold some them.
Wangg ml
2024h
Blnca
Ouline task planning
Mossures the accuracy of completing oaline workflow based platning tacks. Meseures the accuracy of agents on nea--arld Pyshan aading Luskss aunass varions du- mains.
Domnala- general
N/A
Ladis held-out aes and doesn't
Xlao ct
2124
indicate plans male
PyBench
Python ming
program-
Domain-greet
N/A
Locles a helé-out wet and doesn't indicade plas to make con.
Additional tooks conespreding to unoen ealegariee, subelussss of lioks, datu somrss.
Zhang st ml. 2024h
Cybench
Cybersecarity
Messuee the accuracy of agents ou wolking cylnewourity tunks.
Task specific
N/A
Lacks
acs and doesn't
Cybersccurity tacks drawa from Caplure the Flug campuli- tious. N/A
Alung
2024n
x
indiods plass makes one.
Discovery Beach Maj junder
Data driven discov. ery
Measures the accuracy of
Task specthcc
Out of distibbutin samples
The boldout set cousists of tasles
agents validating sclentific hypothesis.
belonging domains and typothe- see present in the srain ant.
Continmee on nost page
34

---

the same distri- Published in Transactions on Machine Learning Research (05/2025)
Denchmark
Doracin
Benchmark description
Level generality
Hobdout
What is held
Holdout the approgrintt generality lesil?
Examppe of ideal holdout
WorldBench
Workslace task
Measures the accuracy of agents on common worksplace taoks.
Domaln general
N/A
Laclas hold ses and doesn't indicate plans malee one.
Tasks cocrespcading sandboo dacabase enviromment unseen during training
Styles es
2024
cutton
X
MobilesBench
MobileyUT
Missures the aucuracy of agunta on completing mobile interacting with apps. Messuees the uccuracy ot agents on naturally sourced webs based taslss.
Demmin-grmeall
NyA
Tondes a helé-cont wet and doesn't indicale pluss to make
Hold out tooks regsiring the uen outaiin appt types unoom during training. N/A
Dong al. 2724
Burkine&Such
tack automation
Domala general
Taoks
of boldout sets consists more challenging tasks than those seen in training, requiring actions urseen during training.
Xu 0s al. 20240
MMAU
Guneenl-pargose
the apcuraicy of agents on Luska vacious maias: tool use, DAG, QA, cod- ing, contest lovel programming, mach.
Fully general
Lacka lieli-oul wel dlewn 't indiodte plans to make one.
Damacnn autaidee rently in bonchmarkk to peu- vent gamning These may include web navigation, visual and spa tial reasoning,
Yiu 2024
x
planning.
Bench
Internation borning
Muswures the accurncy of Dammin-gemeaal agouls on a set, of lmarning- based tasks.
N/A
Lacks beltisuut and clown't in- diosle plans to mnke onn. Howover, authors recogalae procupt overlit- tamue and design, the LLP envirooments paraphrase instructions.
Dacisimmmmsiin king, probllems yomi the provished, or imstrue- tion ppps//lodbbcck ty unsecn during training.
Ching wt al.
2025
TaskBench
Tasks antomation
Measures the accuracy of ageuts on set of taskss within the task automaaton domain. Maseures the apeuracy of que--
Domals general
NyA
Lackes a hold out acs and doesn't indirate plans to male one.
Tasks with different tool sources to build problems from unseen tool graphe.
Sben et all 2023
GATA
General-porgosr
Fully general
Indistribation
held-ont inchudes que-- tions but it's unchar whotlme Uaey comne from a differont distribution than the training set.
Addittional questiom types requir- ing, tig., usuge new laols
Midion af
2125
agents ou gonerd tiota roquiting various lunda- mental ablities and tool use. of
plos
MBPP
Python ming
program
Measwres the agents on solving entry- Pythom programming baoks Museuees the uocurney of ugenta specilically eiementary- level sclentifi soning toxt envirommenr.
Distribution spectfic
distribution sam ples
[oldont set consists of entry level Python tasks bittion sx the train wet.
N/A
noc
Sciemar Workd
Scisnew
Tasksqueific
Out of distribution samples
The belt-ont wil includis variatioon auch that critioa objects, atart- loeations, and the environment are unseen during training.
N/A
Wang at
2022
AllWolld
Housebold taoks
Museures the accuracy of Demain-grerall
Out distribuution samplox
The held-out wee includes distinct Lock indanas und rooms that wene during training. hold-out tank instances der the same six ALPRED cask types as the training tack instances. The authors intend Jericho unsepparted as a training act, and evalttace agent on the Jericho impported game.
Ratthr than training on all six Alfeed tukk-type,, hold out womm lest avaluntion..
SStiiiber
different buoes within the losssdill unvisun- ment.
Jericho
Gamee
The benchmarks sampler from and moshittes agentz games cover a variety of getres.
Distribution-
In-dietribationn sum- plas
the gumns
N/A
Hauskkechh
specific
20,0
BalocAl
Task inlommliio
Musures the aucurncy of agents general of navigation related tasks given environment.
Dammizegenraal
In distribution sum- plas
The bekd out wil includes 512 cuisodes, but unclear whether these episodes are a different distribution or task type from the spisodes found in training.
New Bubl Al levell, symbodic observations of different siass than the taaining set.
Chevedlerr Holavert
chenehmakk
General purpowe
Meseurex the accuracy of Fully general dumuiise. Those include interlace, code gencration, code modifcattion reteteval, and safety.
N/A
Locles helé-out wet and doeen't indicsie pluss make
Domains onsside of the nows rually bonrhmmark pen- weu guning. These may include web navigation, visual and spa- tial reascaing, long term planning.
Nudhin
agumts
35

---

Published in Transactions on Machine Learning Research (05/2025)
Table 6: Simple cost comparison of RAG and Long-Context approaches on NovelQA. We used the tiktoken tokenizer to measure the exact token counts for (a) the questions, (b) the four answer options, and (c) the prompt template from the original paper (Wang et al. 2024a]. Costs are calculated using May 2024 prices. Due to the high cost of running the evaluation, we only ran our evaluations once.
RAG Long-Coue Context
Total Cost
$52.80
$99.80
Accuracy
67.89
67.81
RAG Specific:
Cost embedding novels
$2.512
Cost of embedding one novel
$0.0285
Cost per Question
$0.0222
Cost per QA for new novel
$0.051
Long- Context Specifie:
Mean prompt tokens per novel
690.807
Total tokens of questions and options
110,094
Total tokens (prompt + questions + options)
170885.01 016
Total long-context question cost
$1.709
Long-context novel cost
$98.09
Long-context cost per novel (single question)
$1.115
Comparison:
Cost Ratio Mlong-Contett/AAG
~ 21.86
36

---

Published in Transactions on Machine Learning Research (05/2025)
Table 7: Examples of shortcomings in current agent evaluations stemming from inadequate benchmark standardization. This table provides additional details on the shortcomings of agent evaluations affecting current state-of-the-art agents from five papers on the leaderboards of HumanEval and WebArena. It lists all issues arising from inadequate benchmark standardization that impact the evaluations for each agent.
Benchmark Paper
Evaluation issues affecting this paper
WebArena
Zhou et al. 2024
Rate limits on Reddit website impacted task completions. This affected 2/129 tasks of the Reddit subset.
Autologin functionality for Reddit site was not implemented correctly (i.e., open todo comment in code) (see Appendix E. Caused the agent silently fail on affected tasks.
Sodhi et al. (0044
Evaluated on subset of benehmark. Remove problems from WebArena for unclear reasons.
Agent is evaluated incorrectly. Original log files provided by authors mark failed tasks as successfully solved. See Listing 2 for an example.
Rate limits on Reddit website impacted task completions. This affected 30/129 tasks of the Reddit subset. See Listing for an example.
HumanEval
Zhou al. 20033
Evaluate on subset benchmark. Authors remove four problems in their evalnations-three because HumanEval does not provide example tests and one for unclear reasons.
Agent is evaluated incorrectly. Generated solutions are only evaluated true test cases for subset of the tasks in HumanEval. Leads to incorrect solutions passing correct.
Zhong et al. 2004
Underreport baseline. The paper claimed accuracy of 75.0%, while our evaluation showed a mean accuracy of 89.6%
five runs.
Evaluate on modified version of benchmark. Add example tests for 3/164 problems that missing in the original benchmark.
GPT-3.5 for code generation using Reflexion. However, the generated program they used from the Reflexion repository relies on GPT-4 for code generation, not GPT-3.5.
Claim
Shinn et al. (2033
Evaluate on subset of benchmark. Remove three problems missing example tests in HumanEval. Do not report this.
37

---

Published in Transactions on Machine Learning Research (05/2025)
Table 8: Reported vs. reproduced accuracy of agents on HumanEval. This table contains the reported and reproduced accuracies of the HumanEval agents part of our analysis. We report the accuracy of each agent evaluated on all 164 tasks of HumanFval. Note that insufficient standardization of the benchmark, rather than mistakes made by the agent developers, is a major contributing factor to many of the observed discrepancies. We run each agent five times and report the mean accuracy. The minimum and maximum accuracy are included in the parentheses.
Paper
Agent
Accuracy
Reported
Reproduced
Zhou et al.
(023
LATS (GPT-4)
94.4
88.0 (0.823-0.915)
LATS (GPT-3.5)
83.8
80.4 (0.787-0.835)
Zhong et al.
2024
LDB (GPT-1, GPT-3.5)
89.6
91.0 (0.89-0.921)
LDB (Reflexion, GPT-3.5)
95.1
88.9 (0.86600.915)
LDB (GPT-3.5)
82.9
80.2 (0.787-0.82)
GPT-4
75.0
89.6 (0.878-0.909)
Shinn al. 2023
Reflexion (GPT-4)
91.0
87.8 (0.86-0.909)
38


Reviews:
[{"review_id": "mLIOpK9Uni", "reviewer": "Reviewer_LnKb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "While the paper raises valid concerns about the current state of AI agent benchmarking, its core arguments and recommendations are similar to existing issues and practices within the broader field of machine learning evaluation that have been discussed in depth in past literature. The need for careful selection of held-out sets that are more representative of the downstream task is a fundamental requirement in ML, not a novel insight specific to agents. Similarly, the importance of standardized evaluation procedures and reproducibility has been extensively discussed and addressed in the ML community for years.\n\nThe paper's proposed framework for choosing hold-out sets, while presented in the context of agents, builds heavily on familiar concepts of in-distribution and out-of-distribution generalization. Framing this as a new contribution specific to agents overlooks the extensive research in this space in ML such as domain adaptation.\n\nThe paper's emphasis on human-in-the-loop evaluation, while relevant, offers little concrete methodology beyond acknowledging its importance. The challenges of incorporating human feedback in evaluation, such as subjectivity and scalability, are well-known, and the paper does not contribute significantly to understanding or tackling them.\n\nFinally, while the paper is a useful reminder of good evaluation practices, it does not provide novel insights or solutions related to AI agent benchmarking or demonstrate meaningful nuances that distinguish agent evaluation from traditional ML model evaluation. The paper's impact is therefore likely to be limited.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper argues that current AI agent benchmarks are inadequate for evaluating agents and identifies+elaborated on key issues like the lack of appropriate hold-out sets, the absence of standardized evaluation practices, and the benefits of considering both cost and accuracy together in evaluation, etc", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It would be helpful if the paper could identify specifically what is its focus in the agent evaluation space and how is that different from past research in ML evaluation, and then design the paper around it. \n\nThe current title does not reflect the paper's content and might be useful to modify. \n\nThere are many instances in the paper where the writing would benefit from more clarity and rigor.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Pj0RdNQ7SF", "reviewer": "Reviewer_3Reu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The explicit call for incorporating cost in the evaluation protocols.\n* Evaluation of several agents in terms of both accuracy and cost on the HumanEval.\n* The presented 3 baselines (retry, warming, escalation) that often beat “SotA” agents.\n* Pareto curve plots of accuracy and inference cost are very insightful and should be used in the future for all new proposed methods.\n* Showing that “sota” agents Reflexion and LDB cost over 50% more than simple non-agentic “warming” baseline for similar accuracy.\n* Finding that many reported accuracy scores were above the maximum of the 5 runs they performed.\nQuestion: where in your paper can I find details about this claim? It would be good to put a reference to the Table/Figure after this sentence. By this I mean: reported number in the paper = X, our best score = Y.\n* Findings in Table 7 on the issues affecting different papers.\n* Categorization of agent benchmarks in Table 5.\n\nWeaknesses:\n* Limited scope of empirical analysis in terms of the number of benchmarks and models evaluated.\n* The authors suggest the importance of human-in-the-loop evaluation, but they don’t propose how to incorporate it into agent benchmarks.\n* The same holds for the call for standardized agent evaluation framework, since they do not propose a concrete solution.\n* Not directly a weakness, but the claim that “escalation strategy strictly improves accuracy while costing less than half of LDB(GPT-3.5)” is somewhat misleading, since it uses more powerful models (Llama-3 and GPT-4) for “escalated” cases.\n* The paper suggests that “it is possible that System 2 techniques will be useful on harder programming tasks than those represented in HumanEval such as SWE-bench”. Not investigating such tasks is a weakness that could be addressed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper examines the current state of AI agent benchmarking and identifies several shortcomings that hinder the development of useful real-world agents.\nContributions:\n* Propose to add “cost” to the evaluation. They demonstrate that simple baselines (retrying, model “escalation”, and “warming\") can outperform “state-of-the-art” agents on coding tasks (HumanEval) while being significantly cheaper.  This underscores the need to include cost as a key metric in agent evaluation. Although this finding is not too surprising, it is valuable that someone performs and presents a systematic investigation.\n* Showcase a method for “joint” optimization of accuracy and cost. Using the HotPotQA benchmark, they modify the DSPy framework to jointly optimize for both accuracy and cost, showing that it's possible to reduce cost significantly without sacrificing performance. This introduces a new dimension to agent design.\n* Suggest that model and downstream (application) evaluation are often at odds in the benchmarks. The paper argues that model developers and downstream users have different benchmarking needs. Model developers prioritize accuracy improvements, while downstream users care about cost-effectiveness in real-world applications. \n* Note that many benchmarks are flawed and can result in agent “overfitting”. They propose a framework for designing benchmarks with appropriate holdout sets based on the desired level of agent generality (distribution-specific, task-specific, domain-general, fully general). They analyze existing benchmarks and find that many lack appropriate holdouts. A case study with the WebArena benchmark exemplifies this issue. Generalization has been studied in a wide range of ML subareas, so this categorization is not the first of this kind, but perhaps the first in the area of developing “agents”. Therefore, it is a valuable contribution, especially together with the insights in Table 5.\n* Identify inconsistencies and errors in the reproducibility of evaluations on benchmarks (WebArena and HumanEval) and call for greater standardization in evaluation practices.\n\nOverall I believe that the paper is a valuable contribution for the community, since it explicitly identifies and names some important issues which are present in the current benchmarks, the “sota” methods, and calls for their corrections. Such contributions are always valuable as they provide more transparency into the benchmarks and methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Crucial changes:\n* The paper suggests that “it is possible that System 2 techniques will be useful on harder programming tasks than those represented in HumanEval such as SWE-bench”. Evaluating this would significantly strengthen the claims in the paper.\n* Propose some solutions for “human-in-the-loop” evaluations.\n* Propose some solutions for the “standardization” issue. The paper identifies a lack of standardization as a major problem but offers limited concrete solutions beyond calling for a standardized evaluation framework.\n* Include concrete recommendations on how to address the issue of dynamic environments in agent benchmarks.\n\nNice-to-have changes:\n* Investigate more benchmarks (from Table 5) and more agents/models (e.g. LLMs such as Gemini, Claude, etc or other open-source models). For example, do the same findings for the proposed 3 “simple” baselines hold for other LLMs too?\n* Question: how come Llama-3(uncompiled) is more expensive than Llama-3 (format.instr.)?\n* Question: In Section 4.1 you claim “owever, this does not represent how users would ask questions about novels in practice. Even if users have many questions about a novel, they will likely ask them individually rather than all at once. Such sequential queries would cost orders of magnitude more because the novel has to be re-processed each time.”\n\nDo you think it's realistic that the benchmarks capture all possible real-world use cases of how users might use the models?\n\nHow do you propose to solve this aspect in current / future benchmarks?\n* Missing citation: “that the code and data accompanying a paper should be enough to reproduce the results that it reports ( ?).”", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2meL67dsKX", "reviewer": "Reviewer_7FMQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- The article analyzes the shortcomings of existing agent benchmarks from multiple angles, such as the lack of consideration for costs, differences between testing methods and actual downstream applications, the presence of shortcuts in benchmarks, and the lack of standardization. These issues are of practical value.\n- The article conducts numerous experiments to verify the problems with current benchmarking.\n\n# Weaknesses\n- Many of the issues mentioned in the article are already well-known, lacking deeper analysis and more insightful viewpoints, such as uncovering more fundamental reasons or proposing improvement solutions.\n  - The article mentions \"Lack of clarity on the source of performance gains,\" but does not further analyze what the source of performance gains is.\n  - In the third section, it mentions \"Jointly optimizing cost and accuracy can yield better agent designs,\" but does not clearly explain what a better design is. Is it suggesting that as costs increase, the improvement in accuracy decreases, and therefore an optimal cost range should be set?\n- Some conclusions lack rigorous analysis and sufficient experimentation.\n  - The third section only conducts experiments on HotPotQA, so it's unclear whether the experimental conclusions are universally applicable.\n - In the fourth section, it mentions \"Model and downstream developers have distinct benchmarking needs,\" but only provides NovelQA as an example. Are there similar situations with other benchmarks?\n- The article discusses the challenges of AI benchmarking from five angles, with the first three involving cost, and the other two discussing shortcuts and standardization. The classification method feels unclear, and the points are not entirely independent. For example, point 2 seems to be an extension of point 1.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper identifies and addresses several critical shortcomings in current AI agent benchmarking and evaluation practices. By addressing these issues, the paper aims to enhance the real-world applicability of AI agents, moving beyond mere benchmark accuracy to foster genuine advancements in the field.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- There is an issue with the citation in the sixth paragraph: \"be enough to reproduce the results that it reports (?).\"\n- The fourth and fifth paragraphs need to be validated on more benchmarks.\n- See weakness.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: # Summary of the paper
This paper focuses on AI agents in the context of LLMS. It describes the current techniques for evaluating agents, identifies several issues with them and proposes some improvements (or directions of improvements).

They first focus on cost-performance comparison, showing that the evaluations must be cost controlled, otherwise they lose their meaning. Then then propose a framework to jointly optimise cost and performance, and show that it is efficient on the HotPotQA benchmark.

Then, they focus on benchmarks, and they identify three main shortcomings of standard benchmarks. 1) Benchmarks are not always aligned with the actual applications of the model, 2) Benchmarks can be over optimized and 3) benchmarks are not always evaluated ina standard way in the literature. They illustrate their point by specifically studying WebArena and HumanEval benchmarks.


# Summary of reviews and discussion

Overall, these contributions were deemed useful to the community. Evaluating language models is a very tricky task and it is often the case that benchmarks are either not consistent between publications or do not represent the actual “performance” of the model. Thus, going towards more standardization is always beneficial.

All the reviewers agreed that the problems identified in the paper are valid and a useful contribution.

The reviewers identified two main weaknesses

1. A lack of novelty: the notion of generalization and overfitting is already known in ML.

Indeed, the second part of the paper, around benchmarks, is quite similar to very standard notions of overfitting or OOD generalization in ML. However, I agree with the author's response that the case of AI agents is specific, and is too complex to just be framed as classic OOD generalization. In particular, it is sometimes very hard to properly define  “in-distribution" vs. "OOD", as the authors point out in their response. I thus think it is worth the investigation.

2. Lack of actual solutions

The authors indeed insist on solutions involving human-in-the-loop, and point out standardization issues, but not offer concrete solutions to those issues. I agree This would make the paper even better, but, especially after the update, the authors are clear in the fact that they do not propose a solution but identify flaws, and I think it is a sufficient contribution.

The authors have correctly answered the other reviewers' suggestions.

# Conclusion

This paper tackles a very active and important subject of language models evaluation for agentic tasks. While it does not have high novelty contributions or SOTA methods, it shows important shortcomings in a key part of the literature; and at the minimum points to directions of solution.

In conclusion, as the contribution is correct to our knowledge, and useful to the ML community, it fits TMLR. I propose to accept the paper.
Decision: Accept (poster)