Agent-as-a-Judge: Evaluate Agents with Agents
Authors: Mingchen Zhuge
Changsheng Zhao
Dylan R. Ashley
Wenyi Wang
Dmitrii Khizbullin
Yunyang Xiong
Zechun Liu
Ernie Chang
Raghuraman Krishnamoorthi
Yuandong Tian
Yangyang Shi
Vikas Chandra
Jürgen Schmidhuber
Venue: ICML 2025 poster (2025)
Keywords: Code Generation
Agent-as-a-Judge
AI Developer
AI Judge
LLM
Primary area: applications
TL;DR: This paper introduces the Agent-as-a-Judge, and we also collected a dataset helps verify our idea.

Abstract: Contemporary evaluation techniques are inadequate for agentic systems. These approaches either focus exclusively on final outcomes---ignoring the step-by-step nature of the thinking done by agentic systems---or require excessive manual labour. To address this, we introduce the **Agent-as-a-Judge** framework, wherein agentic systems are used to evaluate agentic systems. This is a natural extension of the LLM-as-a-Judge framework, incorporating agentic features that enable intermediate feedback for the entire task-solving processes for more precise evaluations. We apply the Agent-as-a-Judge framework to the task of code generation. To overcome issues with existing benchmarks and provide a proof-of-concept testbed for Agent-as-a-Judge, we present **DevAI**, a new benchmark of 55 realistic AI code generation tasks. DevAI includes rich manual annotations, like a total of 365 hierarchical solution requirements, which make it particularly suitable for an agentic evaluator. We benchmark three of the top code-generating agentic systems using Agent-as-a-Judge and find that our framework dramatically outperforms LLM-as-a-Judge and is as reliable as our human evaluation baseline. Altogether, we believe that this work represents a concrete step towards enabling vastly more sophisticated agentic systems. To help that, our dataset and the full implementation of Agent-as-a-Judge will be publically available at https://github.com/metauto-ai/agent-as-a-judge

Agent-as-a-Judge: Evaluate Agents with Agents
Mingehen Zhuge's 12 Changsheng Zhao'
Dylan R. Ashley² Wenyi Wang?
Dmitrii Khizbullin Yunyang Xiong'
Zechun Liu
Ernie Chang'
Raghuraman Krishnamoorthi
Yuandong Tian Yangyang Shi Vikas Chandra'
Jürgen Schmidhuber?
Abstract
1. Introduction
Recent years have seen multimodal agentic systems move from occasionally being able to solve small toy problems to being regularly deployed for challenging real-world prob- lems (the dream of most Al research). Yet, the current eval- uation methods and the available benchmarks for agentic systems are struggling to keep up with these rapid advances, dramatically slowing true progress.
Contemporary evaluation techniques are inade- quate for agentic systems. These approaches either focus exclusively on final outcomes ignoring the step-by-step nature the thinking done by agentic systems-or or require excessive manual labour. To address this, we introduce the agent-as-a-Judge framework, wherein agen- tic systems are used to evaluate agentic systems. This is natural extension of the LLM-as-a-Juue J-ddge framework, incorporating agentic features that enable intermediate feedback for the entire task- solving processes for more precise evaluations. We apply the Agent-as-a-Judge framework to the task of code generation. To overcome is- sues with existing benchmarks and provide a proof-of-concep testbed for Agent-as-a-Judge we present Ae a new benchmark of 55 real- istic Al code generation tasks. DevAI includes rich manual annotations, like a total of 365 hi- erarchical solution requirements, which make it partieularly suitable for agentic evaluator. We benchmark three of the top code-generating agen- tic systems using Agent-as-a-Judge and find that our framework dramatically outperforms LLM- as-a-Judge and is as reliable as our human eval- uation baseline. Altogether, we believe that this work represents conerete step towards enabling vastly more sophisticated agentic systems. To help that, our dataset and the full implemen- tation of Agent-as-a-Judge will be publically available at https: peatts::/giteee /ttp//gttttt/////////////tt com/netat mu to co-aiiagenttas-- as-aa-udgaae-eeeee---------- judge
We believe that the current issue with evaluating agentic systems stems from the lack of feedback during the inter- mediate task-solving stages for these nontraditional sys- tems. Agentic systems think more like human, often act step-by-step (Wooldridge, 1999) and often hosting very human-like natural language discussions internally to solve problems (Zhuge et al., 2023). And thus agentic systems should be evaluated like a human, with rich evaluative feed- back which looks at the full thought and action trajectory; evaluating an agentic system in the traditional way is like evaluating student using multiple-choice testing-a com- paratively unreliable estimator (Park, 2010). For example, while SWE-Bench (Jimenez et al.) is widespread, its evalu- ation method, which relies solely on the final resolve rate for long- long-term term automated repair tasks, does not effectively pinpoint what is happening within agentic systems that af- fects the resolve rate. On the other hand, performing a better evaluation with human is prohibitively expensive. We instead propose that agentic systems should be used to eval- uate agentic systems. Inspired by LLM-as-a- Juddee (Zheng et al., 2024; Fu et al., 2024; Chen et al.), which uses LLMs to evaluate L.LMs, we call this framework Agent-as-a-Judge of which is key extension to the world of agentic systems (see Figure 1). It not only retains the cost-effectiveness of LLMdas-a-Judge but is also equipped with agentic features, allowing provide rich intermediate feedback throughout the entire process, as it acts as agentic system. We apply the agent-as-a-Judee. systems to the problem of evaluating code generating systems-on one of the areas where agentic systems have looked the most promising recently.
First four authors made core contributions. KAUST crafted the dataset. Work done while Mingchen was interning at Meta,
with Changsheng leading. 'Meta AI KKUUS. Correspondence to: Mingchen Zhuge <mingchen.zhuge kaust.cdu.sa.. Changsheng Zhao <eszhao @ mom..com>>
In code generation, the development of benchmarks has also lagged behind the rapid advancement of agentic systems. HumanEval (Chen et al., 2021), for example, focuses ex- clusively on algorithmic problems, while MBPP (Austin
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Jgent-as-a-Judge:
Evaluate Agents with Agents
User
[Conversation Task)
User
(Agentic Task)
User
(Agentic Task))
the FEn our toode in the secondary naver (k) the
HE mlase felow the raadcier ane set up the seige hos ///ees
HR Pleas sallw the inttectinn and jut up Bh anot to Bu //
raawy aapy will maxuss the movo eape the eicou oat woult afheddd Outaur tha anover
supdy
aliitine iys1 grnwore vaay inaess with kiren lnd
Flateend ger ge genwnnt 1080p images wit hotes tee munure," anohecasellan Laawayy Seee twin
cntuung in arcnviaza pa Save dam
LLM A
LLM
Agent A
Agent
Agent A
Agent
Imitate
The Federal Reseove Euys bends in the roreso wokky eappy
theernnyy sappyy
Upgrade
S CnNcOrTeY
Womepaac
Judge LLM
Judge Agent
User
Replace
Intrad
beter becuss pous cxnest opaton caudity balonng the resired farmal the
emin
Noveuaton the witslo
browes web
Jump to Sitiic
Shuck caragg
Compare- based Judge
Box, Fu ng the soant regaind A 0.017 pertente assina, and
caitd y produce ane ingars as lil Rildd
broudd
beseed Judio
User
Anounay:
Accunney: FediicttFolooing
ABs: clpuity cheaks it owe Agpel is beteer and requiements
Cont: $1.7
Cust: $1.5 Pecuuromentrr (10004
2/10
18710
Requremente
ayebr cnutade then look boo much lamo, As conkkee
Sandtles ylu)
Metrics based Judge
Metrics-based Judge
As a Developer
LLM-as-a-Jude
Agent-as-a-Judge
Human-as-a-uudge
Figure/ 1. In this paper, we introducc the Agent-as-a-Judge framework wherein agentic systems are used to evaluate agentic systems. We compare this to LLM-as-a-Judge, which uses LLMs to evaluate LLMs and for which Agent-as-a-Judge is natural cvolution, and numan-as-a-Judge- where skilled humans labourers manually evaluate an agentic system.
et al., 2021) deals with simple programming tasks. Al- though they are useful for evaluating the basic skills of foundation models, neither of these two reflects the most practical challenges developers face. As a step away from this, SWE--Bncc (Jimenez et al.) did introduce more real- istic problems from GitHub, offering fresh approach to evaluation, but still primarily focuses automated repairs tasks development process. Concerningly, recent research shows that large language models (LLMs) can already solve over 27% of the tasks in SWE-Bench without needing of advanced agentic systems (Xia et al., 2024). Equally con- cerning, recent work has begun to introduce mechanisms designed specifically for the individual tasks in the SWE- Bench dataset, leading to lack of real-world generalization and following Goodhart's 's law: "When measure becomes target, it ceases to be a good measure" (Goodhart, 1976).
requirements in DevAI, but only one full dask-showing that DevAl presents a good level of challenge to current systems. When comparing our human judges with our automatic agent-as-a-Judee framework, we found that Agent-as-a- Judge aligns more closely with the consensus of our human judges (90%) as compared to LLM-as-a-Jiae a-Jdee (70%) in all cases tested. In addition, we find that it aligns more closely with this ensemble than the individual human evaluators do, suggesting that-not not only is it suitable as a replacement for a human evaluator-but it could in fact be more useful than an average lone human evaluator. In addition, considering the evaluation cost, Agent-as-a-Judee reduces 97. 97.72% of the time and 97.64% of the cost.
In summary, the principal contributions and also the organi- zations of our development steps are:
(Step 1 - Section 2) We release the DevAI dataset, which consists of 55 comprehensive AI development tasks with accompanying tags, individual hierarchical requirements, and individual preferences.
To address the aforementioned issues with current code gen- eration benchmarks, we introduce DevAl: the Al Developer Dataset, which contains 55 real-world automated AI de- velopment tasks created by expert annotators. We apply three leading open-source code geddeeennrtting agentic frame- works to the tasks in DevAI: MetaGPT (Hong et al., 2024b), GPT-Pilot (Pythagora.io, 2023), and OpenHands (Wang et al., 2024b). We evaluate their performance using human judges (a painstaking process), LLL--as--Juddee (Zheng et al., 2024), and our Agent-as-a- framework.
(Step 2 Section 3) code generation agentic frameworks in DevAl, providing a more comprehensive analysis than previous evaluations of them.
We benchmark three top open-source
(Step 3- Section 4) We introduce the general Agent-as- a-Judge concept, allowing agentic systems a fair and rich evaluation without the traditional cost this would require in human labour.
Through human evaluation, we found that GPT-Pilot and OpenHands were each able to satisfy about 29% of the task
2

---

Agent-as-a-Judge: Evaluate Agents with Agents
(Step 4- Section 4) We demonstrate that an Agent-as-a- Judge outperforms an LL---s--- a-aas----Judge and performs com- parably to human evaluators in our proof-of-concept.
softer requirements. DevAl is structured that an agentic system starts by receiving a user query to begin develop- ment. The system is then evaluated on how well it meets the requirements, with preferences serving as optional, softer criteria. An example of one of the DevAl tasks can be seen in Figure 7 and the full suite is available at [REDACTED]
2. (Step 1) Crafting a Benchmark for Automated AI Development
The tasks in DevAI are relatively small-scale but cover com- monly used key development techniques. As shown in Fig- ure 2, our tasks tagged and cover variety of key areas in Al: supervised learning, reinforcement learning, computer vision, natural language processing, generative models, and others. Each of the tasks is a real-world problem that could be given to a research engineer, while simultaneously being relatively inexpensive computationally to run so as to reduce the cost of evaluating a method on this benchmark. Details of the sample collection and human labeling process for DevAl are provided in Appendix E.
2.1. Motivation
Background The code generation domain is an area where agentic systems have seen significant industrial deployment during the past 2 years (e.g., see Devin' and Cursor)), How- ever, in code generation, there isn't yet a benchmark that accurately reflects realistic user queries for developing com- plete Al systems. We believe this is because of the difficulty to evaluate such complex, real-world tasks. For example, while many companies advertise their systems based on the performance of it on benchmarks like SWE-Bench (Jimenez et al.) (for automated repair) or HumanEval (Chen et al., 2021) (for algorithmic tasks), these benchmarks cover only a small bit of an actual development process. A benchmark that can evaluate the entire development process -eeress--delllyyy in way that can help understand the degree to which current Al methods reduce human labour-is missing.
The requirements belonging to each task represent a mile- stone in the comprehensive development process and are arranged as a directed acyclic graph (similar to the work by He et al. (2021)), with requirements such as visualizing results depending on correct data loading and modeling, This allows for more comprehensive non-sparse feedback than a binary success metric. Furthermore, the inclusion of milestones makes simple memorization not viable as a solution strategy as code that performs the entire task is unlikely to exist online at this time.
Topic We chose automated AI development as our main topic. While AI and ML tasks are often more complex, they follow clear, standard procedures. For example, data processing typically comes first in an AI pipeline, and perfor- mance reporting goes at the end. We believe this topological nature can help better monitor the development process and provide useful signals to the agentic systems.
2.3. Preliminary Benchmark
We first conduct experiments to collect development out- comes from different frameworks, which serve as base- lines in the DevAl dataset. We test three of the most pop- ular open-source frameworks (which we will refer to as " "AT developers"): MetaGPT (Hong et al., 2024b), GPT- Pilot (Pythagora.io, 2023), and OpenHands (Wang et al., 2024b) all selected for their strong community acceptance (each having over 30,000 stars on GitHub).
Goals An ideal benchmark should address critical issues in automated development by focusing on three key factors. First, it should reflect practical software scenarios, where tasks are often too complex for a single LLM,, requiring human or agentic systems. Second, it should emphasize the development process, not just final outcomes (e.g., pass@ 1 rates offer limited feedback and fail to highlight intermediate problems). Lastly, the evaluation should be computationally cost-effective and efficient, avoiding long training times or excessive manual oversight.
Experiment Setup All of these three systems require language model as back-end engine, for which we use gpt-40-2024-08 22222--0--1, a state-of-the-an language model. These AI developers were given a time-limit of 1 1800 seconds to solve each task and were forcefully halted if they exceeded this time limit (we imposed this constraint, which was visible to the Al developers, as detailed in Appendix I). We capture the outputs generated during the automated de- velopment process, including code, files, and other artifacts. Additionally, we record key decisions and actions made by the agentic systems through some custom instrumentation code, resulting in a thought trajectory for each of the agentic systems.
2.2. The DevAI Dataset
Motivated by the ideas outlined above, we propose the De- vAI dataset. DevAI consists of a carefully curated set of 55 tasks, each defined by (1) a plain text query that describes an AI development task; (2) a set of plain text requirements (for a total of 365 requirements), each with a set of dependen- cies connecting them to other requirements; and (3) set of preferences (for a total of 125 preferences) which represent
'ntpss/wwww.connition.n a http://iio....ooooiiioodddnndeee Phtps://www.ccusor.com
Analysis
The basic statistics are shown in Table 1.
3

---

Agent-as-a-Judge: Evaluate Agents with Agents
aatas
PUSINE
EEVAY
Ragreas
Other
Wispevivel Learning
System
Medical Anaiyii
Falnferrmmeet Learning
Addie Procesiin
(1) Word Clouds of User Queries
Generative Models
Frands Aealyst
Suparitedd eaariin
Matural
22) Number of Words in User Querles
(3) Number of Tags of User Queries
(4) Mentions Models
clazzitier
LSTM
Random Forest
Lineat hegression
Decision Tree
Logistic Regression
Semese
SVH regresiie
BART
KNV classifeer
U-Net
BERT
Kmosiss
PPO
Random Forest classifier
Transforme
YOLOV3
Nestee 1B
DCGAN
CWN-LSTM
0-leaming
XGBoos
Show and Tell
GrdSearchh
FacehNet
SRCMN
word2
DON
GFTT2
DenseNet
Latent
Figure 2. Distribution of DevAI Tasks (I) DevAl focuses on Al development tasks and so terms such as "dataset" "model," and "results" are particularly common in queries. (2) first 53 tasks in DevAI all have a aue-paragraph query but of varying lengths (note that task 54 and 55 are excluded here they outliers, representing the longest and most complex tasks in the dataset). (3) Each task has one or more tags. prevalence supervised learning here reflects the fact classifiers (Cortes, 1995) and LSTM models (Hochreiter, 1997) are of the most widely used architectures fact reflected by DevAl.
dominates many machine learning applications. (4) SVM
MetaGPT the most cost-efficient (1.19 USD), while Open- Hands is the most expensive (6.38 USD). In terms of de- velopment time, OpenHands completes tasks in an average of 362.41s, while GPT-Pilot takes the longest at 1622.38s. On average, a full evaluation on DevAI with one of these three took around 210.65 USD and 14 hours to perform. While running, GPT-Pilot generates the most output tokens 59707 tokens, whereas OpenHands processed the most at 1252482 tokens while producing the fewest 8457 tokens. This suggests that OpenHands's 's internal communication is more complicated but is more parsimonious in its decisions.
using Human-as-a--ane (section 3), LLM-assa-Judge. and Agent-as-at as----uge---eee (section 4).
3. (Step 2) Manual Evaluation on DevAI (Human-as-a-Judge)
To determine the pragmatic validity of DevAI and to accu- rately estimate the actual code-generating abilities of current mrate-of-the-an agentic systems, in this section, we run and then manually evaluate the application of three AI devel- oper baselines to DevAl. In Section 4, we show how this evaluation can be automated.
MetaGPT, while being the most cost-effective, generates fewer saved code files (0.42), suggesting it may be less inclined to save files. contrast, GPT-Pilot generates the most saved files (3.84), reflecting a more prolific output. The difference in saved code lines, with GPTT-Pilot saving 273.33 lines versus MetaGPT's 11.15, underscores GPT- Pilot's 's extensive output. Meanwhile, OpenHands, despite handling larger inputs, seems less focused on executing code to generate files, as evidenced by its lower file output (2.53 saved files). These statistics align with real user experiences (as discussed in Appendix F).
3.1. Benchmark Baselines by Human-as-a--udge
Human Evaluation Setup After obtaining the baseline executions and conducting basic statistical analysis, we have three expert human evaluators (referred here by their anonymous names: 231a, 38bb, and en review the outputs of AI developer baselines to assess whether each requirement was satisfied. We have two rounds of human evaluations. To capture the bias inherent in typical human evaluation (this is desirable to capture here as represents likely scenario in deployment), in the first round, our evaluators first discussed the basic standards but were given minimal instructions. The templates the evaluators were given for the evaluation and their self-reported post-hoe descriptions of how they resolved ambiguities are reported
Evaluations Note that the results in Table are not di- rectly indicative of performance, but provide valuable in- tuition about the practical utility of DevAl. The generated workspaces (generated files, code, etc.) and trajectories are utilized subsequent experiments to perform evaluations

---

Agent-as-a-Judge: Evaluate Agents with Agents
Table Preliminary Statistics AI Developers. We compare three leading open-source code agents using metrics such as average cost. average time, and the average number of generated files.
Metric
MetaGPT (Mlon et al,, 2024b)
GPT-Piol (Pythagonaio, 2023) OpenHands (Wang el al., 2024b)
Basic Statistics
Version
Data Interpreter (Hong
2024a)
0.2.13
CodeAct v1.9 (Wang al.)
()) Average Cost
$1.19
$3.92
$6.38
(2) Average Time
775.29s
1622.38s
362.41s
(3) Average Input
152863
606707
1252482
(4) Average Ouipat Tokens
28546
59707
8457
(4) Average Saved Code Files
0.42
3.84
2.53
(5) Average Saved Code Lines
11.15
273.33
96.56
(6) Average
4.42
Table 2. Human-as-a-Juude for AI Developers. (l) and (D) represent independent performance versus performance considering task dependencies. indicates multiple experts evolved, and means the evaluations use white- sitte--ox testing (allowing access to the generated workspace, human-collected trajectories, and open-source codebases). The results were derived from expert judgments and deliberations (see Appendix H).
Metric
MetaGPT (Hong 2024b) GPT-Pilot (Pythagora.io, 2023) OpenHands (Wang al., 2024b)
Human-as-a-Judge
(A) Requirements Met (I)
22.13%
44.80%
42.89%
(B) Requirements Met (D)
6.55%
28. 96%
28.68%
(C) Self relffTermmnation
41.81%
5.45%
54.54%
(D) Task Solve Rate
0.005
1.81%
1.81 1%
in Figure in Appendix H.
evaluator has over five years of experience in Al research and development, the disagreement highlights the inherent challenges of evaluating Al development.
After the initial round of human evaluations (which totaled an estimated total of 58 human hours), we asked our evalua- tors discuss and reach consensus on their assessments (which took an estimated total of 28.5 additional human hours). This consensus, achieved after long sessions of de- bate, was used as the final human evaluation result for each method.
211000
Sttaot
OPT/N
Figure 3. Between the three human evaluators, a large amount of disagreement was observed in their individual evaluations highlighting the inherent unreliability human evaluation.
Performance Analysis The results of this experiment are shown in Table We found that the two best-perfomiigg methods (GPT-Pilot and OpenHands) could satisfy about 29% of the requirements (or around 44% if prerequisites are ignored) but only on one task could they meet all the require- ments. This highlights that DevAl offers a considerable but appropriate level of challenge for current and future meth- ods. Moreover, the fulfillment of intermediate requirements aligns with our expectations (see Section 2) that DevAl pro- vides richer feedback by uncovering how agentic systems falter during the process instead of just focusing on a single performance metric at the end.
Due to the complexity of a complete Al development task, which typically involves multiple steps with varying out- comes each step, humans can easily make errors when critical information is missed, such environment feedback indicating small but severe coding errors or bugs. Addition- ally, some disagreements are not necessarily incorrect but arise from differing perspectives on how ambiguity should resolved.
To determine if the disagreement between the three human judges too large for them to serve as strong baseline, recruited ten additional experts and had them evaluate random selection of 7 task samples. We observed that the majority vote of these additional experts had a 95.23% agreement rate with the consensus result our other experts on these tasks and a 97.67% agreement rate with the majority vote. See Appendix P.
3.2. Judging Human-as-a-Judge
Disagreement Analysis To analyze the presence of in- ductive bias and the reliability of the Human-as Sluman-as-a-Jude audge paradigm here, we calculate the disagreement rate between individual evaluators (shown in Figure 3). The results indi- cate that the disagreement rates between pairs of evaluators range from around 10% to 30%. Although each human
5

---

Agent-as-a-Judge: Evaluate Agents with Agents
Error Rate Comperison (%)
in Figure 4). The second approach involves assembling larger panel of experts (more is better when their accuracy exceeds 50% (Grofman et al., 1983)), with over 5 people recommended by Hastie & Kameda (2005); Larrick & Soll (2006), and relying on a majority vote. However, due to the high cost of engaging more experts and the fact that this not always feasible practice, we argue for the former.
12:57
7.92
10.93
300e
9.02
7.38
10.11
majoriyy Labeler
6,01
4.92
5.74
4. (Step 3&4) Evaluating Agents with Agents (Agent-as-a-Judge)
MataGPT
OpenHands
Figure 4. Mismatch between the individual evaluations and the consensus evaluation. Note that majority vote showed the smallest deviation from the consensus evaluation.
Human evaluation, while somewhat reliable, is time- consuming and requires significant expertise. To address this, we propose the Agent-as-a-Judge framework. If such agentic system could evaluate like human, it would reduce the need for human involvement, eliminating the trade-off between evaluation thoroughness and effort.
Error Analysis As previously noted, the evaluators en- gaged in round of debating after their initial evaluations until they reached consensus each requirement in each task (with the results of this consensus evaluation shown in Table 2).
4.1. Spefic Agent-as-a-Judge for Code Generation
In our Human-as-a-Judee pipeline, evaluators could be con- vinced by evidence from others and acknowledge their judg- ment errors, adjusting their answers accordingly. This can be used approximate individual errors. If the consensus evaluation more accurately predicts any extant ground truth, we would expect the majority vote from the individual eval- uations to more closely approximate this than any single evaluation, due to the fundamental properties of ensemble classifiers (see Hastie et al. (2009)).
Based on our prior experiences with agent design and by im- itating the human evaluation process, we initially designed eight modular, interacting components that form the foun- dation of our tailor-made agent-as-a-Judgee
INPUTS
Graph
Locate
While the consensus evaluation may not represent the ab- solute ground truth (we acknowledge that some quantity of error likely would still exist even after this procedure), we expect the consensus evaluation to more accurately ap- proximate the extant ground truth (Clemen, 1989). If this holds, the majority vote should align more closely with the consensus than with any individual evaluation. As shown in Figure 4, this indeed the case.
Search
Planning
Memory
Retrieve
Read
Ask
OUTPUTS
As seen in the results, although significant emorr occur among all evaluators, the majority vote effectively corrects most of these errors. Notably, 9n made the most errors (for example, 23.77% in evaluating GPT-Pilot). After apply- ing the majority vote from all three evaluators, the overall error rate dropped to 6.01%. demonstrating the inherent benefits of majority voting.
Figure 5. Initial diagram of Agent-as-a-Judge.
(1) The graph module constructs a graph that captures the entire structure of the project, including files, modules, and dependencies. It can also break down chunks of code into code snippets. (2) The locate module identifies the specific folder or file referred to by a requirement.{3 The read module goes beyond simple file parsing, supporting the reading and understanding multimodal data across 33 different formats, including code, images, videos and documents. This allows the agent to cross-reference various data streams and check different kinds of requirement. (4) The search module offers a contextual understanding of code and can quickly retrieve highly relevant code snippets, as well as the nuances behind them (e.g.. hidden depen- dencies). (5) The retrieve module extracts information
Conclusion Human judgment errors are inevitable. To reduce them, we suggest two methods. First, like in this work, introduce debate round after each judgment, where individuals present evidence and either persuade others or adjust their own opinions after discussion. This is partic- ularly important when there are only a few evaluators, as majority voting with a small group can still lead to errors (around 5% compared to consensus evaluation, as shown
6

---

Agent-as-a-Judge: Evaluate Agents with Agents
Table Al Judges and Their Shift/Alligmment with Human-as-a-Judge. We compare the results of a-uggee and Agent-as, as- a- a-Judge with Human-as-an -amaan-as-a--aae as-a--dee (I) represents performance independent tasks, while (D) represents performance considering task dependencies. Note: gray-box settings use carefully manually collected trajectory data (which is nearly inaccessible in practical situations, see Appendix J). In contrast, black-box setting doesn't 't need to access to such data. The red scores represent the absolute judge shift compared with Human-as-a-Judge (e.g., 2.74%).
Metrie
MetaGPT (Hong et al., 2024b) GPT-Pilot (Pythagora.io, 2023) OpenHands (Wang et al., 2024b)
LLM-as-a-Judge
Requirements Met
19.39% (2.74%)
12.56% (32.24%)
11.47% (31.42%)
(b) Requirements Met (D)
1.63% (4.92%)
4.09% (24.87%)
2.18% (26.50%)
(c) Task Solve Rate
0.0% (0.0%)
0.0% (1.819)
0.0% (1815) 60.38%
Alignument Rate
84.15%
65.30%
Agent-us-a-Juue
(1) Requirements Met (1)
25.40% (3.26%)
53.00% (8.201)
62.6256 (0.27%)
(II) Requirements Met (D)
5.73% (0.81
39 (10.939)
26. 50% (2. 17%)
(III) Task Solve Rate
0.0% (0.0%)
5.55% (3 (1.64)
1.81% (0.0055) 90.44%
Alignment Rate 1
88.52%
83.88%
LLM-as-a-Judge
(a) Requirements Met (I)
28.68% (6.55%)
38.79% (4.10%)
43.16% (0.27%)
Requirements Met (D)
17.75% (11.20%)
33.06% (4.10%)
32. 24% (3.56%)
(c) Task Solve Rate
1.81% (1.81%)
3.63% (1 (1.82%)
7.27% (5.46%)
Allgniment Rate 1
68.86%
71.85%
70.76%
Agent-as-a-Judge
(1) Requirements Met (I)
23.49% (1.35%)
46.44% (1.64%)
43.44% (0.54%%
Requirements Met
6.01 (0.549)
30.60% (1 (1.64%)
28. 14% (0.53%)
(III) Task Solve
0.0% (0.009)
5.45% (1.64%)
(1,829))
Alignment Rate 1
92.07%
86.61
90.169
Human-as-at aasaaan------Jddee
Aligument Rate (38bb)
92.63%
90.98%
89.89%
Alignment Rate (cn90)
83.33%
76.23%
78.15%
Alignment Rate (231a)
92.07%
87.43%
89.07%
Average of individuals
89. 9%
84.888
85.70
Alignment Rate (Majority Vote)
95.08
93.98%
94.26%
from long texts, identifying relevant segments in trajecto- ries. With context from the above, (6) the ask module determines whether a given requirement is satisfied. The memory module stores historical judgment information, allowing the agent to build on past evaluations. Finally, (8) the planning module plans out actions, allowing the agent to strategize and sequence tasks based on the current state and project goals.
generated by developer agents, having only hundreds of lines of code, cannot fully benefit from the search module. The details of these findings are explained in Appendix K. Note that a perfect Agent- auddee is not the focus of this proof of concept, and thus, we leave the utilization of ad- vanced agentic optimization methods for Agent-as-a-Judge such as automated prompt optimization and workflow de- sign (Zhuge et al.; Hu et al., 2024), for future work.
Our initial design of the Agent-as-a- audge is shown in Fig- ure 5. After conducting comprehensive ablation studies, we found that the modular combination of (1), (2), (3), (5), and (6) achieved the highest performance (see Appendix C). A sample of the dynamic evidence collected by the Agent- as-a-Judge is shown in Appendix M. We hypothesize this is because Agent-as-a-Jude formation and is sensitive to noise. For example, while our design of the planning module introduces promising decision-making for future actions, the procedure is unsta- ble. Initially, we hoped that historical information from the memory module would help assess current requirements. However, it proved detrimental, as any errors in previous judgments could lead chain of errors, negatively af- fecting current decisions. Besides, the current workspaces
4.2. Judging Agent-as-a-Judge and LLM-as-a-Judge
Judge Shift Judge Shift measures deviation from the aa--a-- consensus results, with lower values in- dicating a closer alignment. As shown in table 3, Agent-as- a-Judge consistently outperforms LLM-as-a-Judge across tasks, particularly those with task dependencies. For exam- ple, in Requirement (I), Agent-as-a-Judge shows a Judge Shift as low as 0.27%, while LL.M-as-a-Judge reaches 31.24% for OpenHands. This underscores Agent-as-a- Judge's stability and suitability for meeting task require- ments. Furthermore, in the gray-box setting, both Agent-as- a-Judge and LLM-as-a-Judge show even better results than their performance in the black-box setting.
needs high-quality factual in-
7

---

Agent-as-a-Judge: Evaluate Agents with Agents
Alignment Rate The Alignment Rate reflects how closely the Al Judges' evaluations align with human consensus across all 365 requirements. It is defined as the percentage of requirement evaluations that are the same as the Human- as-a-Judge consensus evaluation. Compared to LLM-as- a-Judge, Agent-as-a-Judge consistently achieves a higher Aligmment Rate, closely matching human judgments. For example, when evaluating OpenHands, aggent-ss-a-Judee reaches 92.07% and 90.44%, surpassing LLLM-as-a-Judge 70.76% and 60.38% in both gray-box and black-box set- tings. This suggests that Agent-as-a-Judge is more accurate and human-aligned
The components analyzed include ask, graph, read, 1 ocate, and retrieve, The component ablation study for Agent-as-a-Judge reveals key insights into the perfor- mance gains from adding specific functionalities. With only as k component, the agent achieves a 65.03% alignment rate. Adding the graph component increases performance to 75.95% as the agent can better understand relationships between files. The introduction of read further improves the alignment rate 82.24%, reflecting the value of direct access to the contents of the file. Incorporating locate brings a substantial boost to 90.44%, as the agent can ef- ficiently target files relevant to the requirements. Adding retrieve does not always provide a significant benefit in this case. We found the retrieve module effective for judging MetaGPT and GPT-Pilot, as it provides valuable trajectory information (as shown in Table 3). However, it is less effective for OpenHands, which sometimes fails to execute files, resulting in missing responses. In such cases, judgment without trajectories remains viable.
Precision
4.4. Cost Analysis
Our three evaluators self-reported total of 86.5 hours. With 15 USD hourly wage (assuming this would buy a subject expert in AD), a full evaluation under DevAl would cost around 1297.50 USD. In comparison, Agent-as-a-Judgs cost only 30.58 USD (2.29 (2.29%) in API calls and took only 118.43 minutes (2.36%). LLM-as-a-Judge was faster at 10.99 minutes, but due to the absence of intelligent context selection by the Agent-as-a-ludge" modules, it still cost 29.63 USD.
vode
Recall
Figure 6. PR Curves comparing judge Methods.
PR Curves Judging developer agents is a class-imbalanced task, where meeting requirements is much rarer than fail- ing. Metries like judge shift and alignment rate can be misleading. For example, since MetaGPT rarely meets re- quirements, LLM-as-a-Judge easily identifies most cases as negative (achieving 84.15% in the black-box setting). PR Curves offer a clearer performance measure by balanc- ing precision and recall. This shows that, in some cases, Agent-as-a- can nearly replace human evaluators. Our observations indicate the relative reliability of evaluation Haale-Humm--ss-aa < Agent-as--- a-Judee < Ensemble of Human Judges. Future advancements in foundation models and aeenttas-a-Judge designs may shift this order.
5. Related Work
Agentic systems a highly active research area, so we only detail the most closely related works here. We provide treatment of the marginally less relevant related works in Appendix D.
methods as: LLM-as-a-Judge
AT Developers Al in software development is growing fast (Liu et al., 2024). Al-driven developers have been applied to directly imitate software companies (Hong et al., 2024b; Qian et al., 2024a), debug code (Yang et al., 2024a), run data science methods (Guo et al.; Hong et al., 2024a; Li et al., 2024; Qiao et al., 2023), and even write academic papers (Lu et al., 2024a).
Table Component Ablation Studies for Agent-as-a-Judge. We analyze the impact of adding various components (ask, graph, read, 1 ocate,, and retrieve) on the performance of Agent-as-a-Judge for judging OpenHands.
Metric
graph read locate + retrleve
Benchmarks for AI developments Benchmarks like MLAgentBench (Huang et al., 2024), ML-Bench (Liu et al., 2023), and SUPER (Bogin et al., 2024) all focus on bench- marking agentic systems using AI tasks. However, DevAI distinguishes itself from all of these by focusing on realistic user queries that target a complete development cycle. It further includes a more comprehensive evaluation with mul- tiple hierarchical requirements and preferences for each task.
Agent-as-a-Jude- Performance
Alignment Rate 65.03%
75.95%
82.24%
90.44%
90.16%
4.3. Ablations For Agent-as-a-Judge
We conduct ablations to evaluate the impact of adding different components on Agent-as-a-ludge's performance.
8

---

Agent-as-a-Judge: Evaluate Agents with Agents
Comparatively, MLAgentBench (Huang et al., 2024), for example, focuses final performance for limited set of well-known tasks, which risks overfitting and fails to assess system's 's generalization or adaptability.
tors. Altogether, we believe that the above opens the door for scaling up agentic far more than before.
A Acknowledgements
AT Judges Several works have looked at using Al systems as judges. The work by Chan et al.; Zhao et al. (2024), for example, extends LL---a-a- have multiple LLMs in their evaluation process for conversational tasks. Unlike they employ a trivial agentic system and apply it only to evaluate LLMs under traditional evaluation setups. In contrast, (Lu et al., 2024b) uses a single LLM- based evaluator but, unlike LLM-as-a-Judee applies this to multimodal tasks rather than just for evaluating LLMs. Less relevant are frameworks like those by Chen et al. (2024b); Arora et al. (2024); Mündler et al. (2024), where intermedi- ate signals are used during coding development.
This publication was supported by funding from KAUST Center of Excellence on GenAl under award number 5940, as well as, the SDAIA-KAUST Center of Excellence in Data Science and Artificial Intelligence.
Impact Statement
This work introduces an automated, agent-driven evaluation framework designed to improve the scalability, efficiency, and transparency of assessing autonomous systems. By substantially lowering the time and cost required for reli- able review, it democratizes access to rigorous evaluation, enabling research groups and industry practitioners with lim- ited resources to iterate more quickly and safely. However, automating judgment also carries risks: if the judge-agent's own reasoning is flawed or biased, errors may propagate unchecked; over-reliance on automated assessment could reduce critical human oversight, potentially missing emer- gent ethical or safety concerns. In the broader context, Agent-as-a-Judee can accelerate the development of robust multi- systems, while fostering a culture of transparent, reproducible evaluation that is essential for the responsible advancement of machine learning.
6. Discussion and Conclusion
Discussion A key power of the Agent-as as-uude----Juge that we have not exploited here is the feedback it provides being directly used by the agentic system being evaluated. Per- haps the greatest strength of the Agent-as-a-Judge frame- work is that an agentic system could use it to fix issues in their solutions to complex multistage problems on the gy-mommthing older delayed feedback methods did not permit. In addition, a cycle of mutual improvement be- tween Agent-as-a-Jude both evolve together through iterative feedback, presents promising perspective.
and the evaluated agents, where
References
In this work, we focus code tasks, however, the concept of Juuunt--s-a-uudge" is inherently general. In our expe- rience, transforming our current implementation to other domains will not necessitate overcoming significant bot- tlenecks. This because our implementation is modular, comprising components such as graph, read, locate, ask, and retrieve. By modifying the prompts and work- flow, it should not be difficult to transfer the system to other domains common practice in agentic projects. Our read module already supports multimodal data (code, text, images, etc.), and the ask module can be adapted to verify arbitrary requirement criteria. However, we still encour- age the reader to design domain-specific "Agent-as-a-ludg" systems for more accurate usages.
Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024. URL https: http://wwwwww////// cdn anthropic. co m/ de (cc 7bb-33333300bbbbb16601 7bbc618 857627 777627//ooee_caard__aaadd
Arora, D., Sonwane, A., Wadhwa, N., Mchrotra, A.. Utpala, S., Bairi, R., Kanade, A., and Natarajan, N. MASAI: Modular architecture for software-eniineering ai agents. arXiv preprint arXiv:2406. /638, 2024.
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.. 077302 2021.
In this work, introduced the Agent-as-a- Judge method to use agentic systems to evaluate agentic systems. We simultaneously released DevAl: a new bench- mark that evaluates the gode-generating ability of agentic systems on complete AI development tasks when used with agent-as-a-Judge We went on to show that Agent-as- as-a- Judge outperforms existing methods on this task and that it performs similarly to an ensemble of expert human evalua-
Conclusion
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W. Han, Y., Huang, F., Hui, B., Ji, L., Li. M.. Lin, J., Lin, R. Liu, D., Liu, G., Lu, C., Lu, K., Ma, J., Men, R., Ren, X., Ren, X., Tan, C., Tan, S., Tu, J., Wang, P. Wang, S., Wang, W., Wu, S., Xu, B., Xu, J., Yang. A., Yang, H., Yang, J., Yang, S., Yao, Y., Yu, B., Yuan, H., Yuan, Z., Zhang, J., Zhang, X., Zhang, Y., Zhang. Z., Zhou, C., Zhou, J., Zhou, X., and Zhu, T. Qwen technical report. arXiv preprint arXiv:2309 16609, 2023.

---

Agent-as-a-Judge: Evaluate Agents with Agents
Bavaresco, A., Bernardi, R., Bertolazzi, L., Elliott D., Fernández, R., Gatt, A., Ghaleb, E.,, Giulianelli, M., Hanna, M., Koller, A., et al. LLMs instead of human judges? large scale empirical study across 20 NLP evaluation tasks. arXiv preprint arXiv:2406. 18403, 2024.
Miami, Florida, USA, November 2024c. Association for Computational Linguistics. URL https: aaaaacannt
hology org/2024. emnlp-maiie 474.
Chen, M., Tworek, Jun, H., Yuan, Q., Pinto, H. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
Bogin, B., Yang. K., Gupta, S., Richardson, K., Bransom, E., Clark, P., Sabharwal, A., and Khot, T. SUPER: Evaluating agents setting up and executing tasks from research repositories. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 12622-12645, 2024.
Cheng, C.-A., Nie, A., and Swaminathan, A. Trace is the new AutoDiff-unlookeee efficient optimization of computational workflows. In ICML 2024 Workshop on Automated Reinforcement Learning: Exploring Meta- Learning, AutoML, and LLMs.
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps- Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, J., Feldman, M. Q., et al. MultiPL--: scalable and polyglot approach to benchmarking neural code genera- tion. IEEE Transactions on Software Engineering, 49(7): 3675-3691, 2023.
Clemen, R. T. Combining forecasts: A review and annotated bibliography. International journal of forecasting, 5(4): 559-583, 1989.
Cortes, C. Support-vector networks. Machine Learning, 1995.
Cassano, F., Li, I.,, Sethi, A., Shinn, N., Brennan-Jones, A., Ginesin, J., Berman, E., Chakhnashvili, G., Lozhkov, A., Anderson, C. J., and Guha, A. Can it edit? evaluating the ability of large language models to follow code editing instructions. In First Conference on Language Model- ing, 2024. URL https:/ http///ppenee wpwnrevieweeeeeeeeeeeeeeee.oe.ee net/ foor um Piddid=DD- 6Das.
Dong, Y. R., Hu, T., and Collier, N. Can LLM be a per- sonalized judge? Al-Onaizan, Y., Bansal, M., and Chen, Y.-N, (eds.), Findings of the Association for Com- putational Linguisties: EMNLP 2024, pp. 10126-10141, Miami, Florida, USA, November 2024. Association for Computational Linguistics. URL //tls://acaln hology wwwor////0 2224.
5.92.. 592.
Chan, C.-M., Chen, W., Su. Y., Yu, J., Xue, W., Zhang, S., Fu, J., and Liu, Z. ChatEval: Towards better LLM-base evaluators through multi-agent debate. In The Twelfth International Conference on Learning Representations.
Du, Z., Qian, C.,, Liu, W., Xie, Z., Wang, Y., Dang, Y., Chen, W., and Yang. C. Multi-agent software develop- ment through cross-team collaboration. arXiv preprint arXiv:2406.0899 2024.
Chase, H. LangChain. https: /ttp//ggttthaa wtcchhuu.cooooccooom www.cochaaa..a....oooooo..o.caa. se //1////caaaattaaa/ in, 2022.
Fayyad, U., Piatetsky Shankky--- G., and Smyth, P. From data mining to knowledge discovery in databases. Al magazine, 17(3):37-37, 1996.
Chen, D., Chen, R., Zhang, S., Wang, Y., Liu, Y., Zhou, H., Zhang, Q., Wan, Y., Zhou, P., and Sun, L. MILM- as-a-Judge: Assessing multimodal LLM-as-a-Judge with vision linion-language benchmark. In Forty- International Conference Machine Learning.
Fu, J., Ng. S. K., Jiang. Z., and Liu, P. GPTScore: Evaluate as you desire. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies (Volume Long Pupers), pp. 6556-6576, 2024.
Chen, D., Huang, Y., Wu, S., Tang. J., Chen, L., Bai, Y., He, Z., Wang, C., Zhou, H., Li, Y., al. GUI-WORLD A dataset for GUI-oriented multimodal LLM-based agents. arXiv preprint ar arXiv:2406. /08440000891.. /0819, 2024a.
Goodhart, C. Monetary relationships: a view from Thread- needle Street. University of Warwick, 1976.
Gravitas, S. Auto-GPT, GitHub repository, 2023.
Chen, D.,, Lin, S., Zeng, M., Zan, D., Wang, J.-G., Cheshkov, A., Sun, J., Yu, H., Dong, G., Aliev, A., et al. CodeR: Issue resolving with multi-agent and task graphs. arXiv preprint arXiv:2406. 01 304, 2024b.
Grofman, B., Owen, G., and Feld, S. L. Thirteen theorems in search of the truth. Theory and decision, 15(3):261-278, 1983.
Chen, G. H., Chen, S., Liu, Z., Jiang. F., and Wang, B. Hu- mans or LLMs as the judge? study on judgement bias. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, pp. 8301-8321
Guo, S., Deng, C., Wen, Y., Chen, H., Chang, Y., and Wang, J. DS-Agent: Automated data science by empowering large language models with rase-based reasoning. In Forty-first International Conference on Machine Learn- ing.
10

---

Jgent-as-a-Judge: Evaluate Agents with Agents
Haque, M. M. A. FixEval:
Execution--ased evaluation of program fixes for competitive programming problems. PhD thesis, Virginia Tech, 2023.
Jansen, P., Côté, M.-A., Khot, T., Bransom, E., Mishra, B. D., Majumder, B. P., Tafjord, O., and Clark, P. DIS- COVERYWORLDS A virtual environment for develop- ing and evaluating automated scientific discovery agents. arXiv preprint arXiv:2406.06769, 2024.
Hastie, R. and Kameda, T. The robust beauty of majority rules in group decisions. Psychological review, 112(2): 494, 2005.
Jimenez, C. Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. R. SWE-bench: Can language models resolve real-world GitHub issues? The Twelfth International Conference on Learning Representations.
Hastie, T., Tibshirani, R., and Friedman, J. H. The El- ements of Statistical Learning: Data Mining, Infer- ence, and Prediction. Springer. edition, 2009. doi: 10.1007/978-0- 0-77788---37--888-77.
Jin, H., Huang, L., Cai, H., Yan, J., Li, B., and Chen, H. From LLMs to LLM--ased agents for software engineer- ing: A survey of current, challenges and future. arXiv preprint arXiv:2408.02479 2024.
He, X., Zhao, K., and Chu, X. AutoML: A survey of the state-of-the-art Knowledge-basee systems, 212: 2021.
Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., San- thanam, K., A. S. V., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., Miller, H., Zaharia, M., and Potts, C. DSPy: Compiling declarative language model calls into state-of- the-art pipelines. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: /oopeneeveww. net forum? rid=sY5N02Y500.
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. Measuring coding challenge competence with APPS. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Hochreiter, S. Long short-term memory. Neural Computa- tion MIT-Press, 1997.
Lai, Y., Li, C.,, Wang, Y., Zhang, T., Zhong, R., Zettle- moyer, L., Yih, W.-t., Fried, D., Wang, S., and Yu, T. DS-1000: A natural and reliable benchmark for data sci- ence code generation. In International Conference Machine Learning, pp. 18319-18345, PMLR, 2023.
Hong, S. Lin, Y., Liu, B., Wu, B., Li, D., Chen, J., Zhang, J., Wang, J., Zhang, L., Zhuge, M., al. Data inter- preter: An LLM agent for data science. arXiv preprint arXiv:2402. 18679, 2024a.
LangChain-AL. LangGraph. https: /g wuithuub..... com/1 langgraphggggapaaa aatpaaaiinn-///agggap.... angchain-ai
2024.
Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Wang, J.. Zhang. C., Wang, Z. Yau, K. S., Lin, Z., et al. MetaGPT: Meta programming for a multi-agent collabo- rative framework. In The Twelfth International Confer- ence on Learning Representations, 2024b.
Larrick, R. P. and Soll, J. B. Intuitions about combining opin-
ions: Misappreciation of the averaging principle. Man- agement science, 52(1):111-12) 2006.
Levenshtein, V. Binary codes capable of correcting dele- tions, insertions, and reversals. Proceedings of the Soviet physics doklady, 1966.
Hu, S., Lu, C., and Clune, J. Automated design of agentic systems, arXiv preprint arXiv:2408.08433 2024.
Huang, D., Bu, Q., Zhang, J. M., Luck, M., and Cui, H. AgentCoder: Multi-agent- code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13010. 2023.
Li, G., Hammoud, H., Itani, H., Khizbullin, D., and Ghanem, CAMEL: Communicative agents for" mind" explo- ration of large language model society. Advances in Neu- ral Information Processing Systems, 36:51991-52008, 2023.
Huang, Q., Vora, J., Liang, P., and Leskovec, J. MLAgent- Bench: Evaluating language agents on machine learning experimentation. In Forty-first International Conference on Machine Learning, 2024.
Li, R., Patel, T., Wang, Q., and Du, X. MIR-Coppllot: Au- tonomous machine learning research based on large lan- guage models agents. arXiv preprint arXiv:2408. 140., 2024.
Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. LiveCodeBench: Holistic and contamination free evalu- ation large language models for code. arXiv preprint arXiv:2403.00 079744 2024.
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. Competition- code generation with Alpha- Code. Science, (662) 2022.

---

Agent-as-a-Judge: Evaluate Agents with Agents
Liu, J., Wang, K., Chen, Y., Peng, X., Chen, Z., Zhang, L., and Lou, Y. Large language model-based agents for software engineering: A survey, arXiv preprint arXiv:2409.02977, 2024.
V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15174-15 Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 10. 10.16655//11222244cccoooggg.. aclanthelogy wwtpoo//////// acll long. 810.
URL https:
Liu, T., Xu, C., and McAuley, J. RepoBench: Benchmark- ing repository-Iewee code auto-completion systems. In The Twelfth International Conference Learning Repre- sentations,
Qian, C., Xie, Z., Wang, Y., Liu, W., Dang. Y., Du, Z.. Chen, W., Yang, C., Liu, Z., and Sun, M. Scaling large- language-model-baae multi-agent collaboration. arXiv preprint arXiv:2406.077 155, 2024b.
Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., et al. AgentBench: Evaluating ilms agents. In The Twelfth International Conference on Learning Representations, b.
Qiao, B., Li, L.,, Zhang, X., He, S., Kang, Y., Zhang, C., Yang, F,, Dong, H., Zhang, J., Wang, L., et al. Taskweaver: A code-first agent framework. arXiv preprint arXiv:2311.155: 2023.
Liu, Y., Tang, X., Cai, Z., Lu, J., Zhang, Y., Shao, Y., Deng, Z., Hu, H., Yang, Z., An, K., et al. ML-Bench: Large lan- guage models leverage open-souree libraries for machine learning tasks. arXiv preprint arXiv:231 1.09835, 2023.
Raina, V., Liusie, A., and Gales, M. Is LLM-as-a- judge robust? investigating universal adversarial at- tacks on zero-shot LLM assessment. In Al-Onaizan, Y., Bansal, M., and Chen, Y-N. (eds.). Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 1499-7517, Miami, Florida, USA, November 2024. Association for Computational Linguis- tics. URL https: /ttppps://cclll http///aaaatttttt ooloooyoooooooooooo.o org//024. emnlp main.
Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., and Ha, D. The AI scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.064: 062992, 2024a.
Lu, Y., Yang, Li, X., Wang. X. E., and Wang, W. Y. LLMScore: Unveiling the power of large language mod- els in text-to-image synthesis evaluation. Advances in Neural Information Processing Systems, 36, 2024b.
Reimers, N. and Gurevych, I. Sentence-BERT: Sentence embeddings using Siamese
In Inui, K., Jiang, J., Ng. V., and Wan, X. (eds.), Proceedings of the 2019 Conference ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- (JCNPP), pp. 3982-3992, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 10.18653/w1/D19.1142 URL sttps://aclantte 1 logy org. / D
Mündler, N., Müller, M. N., He, J., and Vechev, M. Code agents are state of the art software testers. arXiv preprint arXiv:2406. 100006..2990 /2952, 2024.
Empirical Methods in Natu-
OpenAI. GPT-4 technical report, 2023.
Park, J. ing system. British Journal of Educational Technology, 41(6):1054- 1064, 2010.
Constructive multiple-choice test-
doi:
Robertson, S., Zaragoza, H., et al. The probabilistic rele- vance framework: BM25 and beyond. Foundations and Trends Information Retrieval, 3(4):333-383 2009.
URL
ary.wilee www.wwee....m.................ww wwt.ccoo//cco. httmm///aaa///.111////////////// 8535 2030.20110.00 010.8.........
Shi, L., Ma, W., and Vosoughi, S. Judging the judges:
Phan, H. N., Nguyen, P. X., and Bui, N. D. Hyperagent: Generalist software engineering agents to solve coding tasks at scale. arXiv preprint arXiv:2409. 16299, 2024.
A systematic investigation of position bias in pairwise comparative assessments by LLMs. arXiv preprint arXiv:2406.0779), 2024.
Pythagora.io. GPT-Pilot: Your ai copilot for software development. htttps://gt /ttp:://gghht/t////eeee thub com/Pttaagora -io/gpt-pilcce 2023. 2023. URL https:////// o om com/Ptthaoora-aa io gotpppppppp.......ooompp GitHub reposi- tory.
Song, L., Liu, J., Zhang, J., Zhang, S., Luo, A., Wang, S., Wu, Q. and Wang, C. Adaptive in-conversation team building for language model agents. arXiv preprint arXiv:2405. 19425, 2024.
Tan, W., Ding, Z., Zhang, W., Li, B., Zhou, B., Yue, J., Xia, H., Jiang, J., Zheng, L., Xu, X., et al. Towards general computer control: A multimodal agent for Red Dead Redemption II as a case study. In ICLR 2024 Workshop on Large Language Model (LLM) Agents.
Qian, C., Liu, W., Liu, H., Chen, N., Dang, Y., Li, J., Yang, C. Chen, W., Su, Y., Cong. X., Xu, Li, D., Liu, Z., and Sun, M. ChatDev: Communicative agents for software development. In Ku, L.W.., Martins, A., and Srikumar,
12

---

Agent-as-a-Judge: Evaluate Agents with Agents
Tao, W., Zhou, Y., Zhang, W., and Cheng. Y. MAGIS: LLM- Based multi-agent framework for GitHub issue resolution. arXiv preprint arXiv:2403.1992) 2024.
Wu, Q. Bansal, G., Zhang, J., Wu, Y., Zhang, S., Zhu, E.,
Li, B., Jiang, L.,, Zhang, X., and Wang, C. AutoGin: Enabling next-gen Ilm applications via multi-agent con- versation framework. arXiv preprint S.3iv:2300.0815555 2023.
Thakur, A. S., Choudhary, K., Ramayapally, S., Vaidyanathan, S., and Hupkes, D. Judging the judges: Evaluating alignment and vulnerabilities in Ilms-as- judges. arXiv preprint mrXiv:2406.12622 2024.
Wu, Y., Yue, T., Zhang, S., Wang, C., and Wu, Q- State- Flow: Enhancing LLM task-solving through state-driven workflows. In First Conference on Language Model- ing, 2024. URL https: //wpn//eeeee...eeooo/// net/ for ammid=3nnbuyyose
Tian, R., Ye, Y., Qin, Y., Cong. X., Lin, Y., Pan, Y., Wu, Y., Haotian, H., Weichuan, L.. Liu, Z., and Sun, M. DebugBench: Evaluating debugging capability of large language models. In Ku, L.-.., Martins, A., and Srikumar, V. (eds.), Findings of the Association for
Xia, C. S., Deng, Y., Dunn, S., and Zhang, L. Agentless: Demystifying LLM-based software engineering agents. arXiv preprint vrXiv:2407.01490 2024.
Computationnl Linguistics: ACL 2024, pp. 4173-4198, Bangkok, Thailand, August 2024. Association for Com- putational Linguistics. doi: 10. 10.18653w1120024fnddigg acl.247. URL https: ac 4. a.findings- aww.iinngsac......
Xu, T., Chen, L., Wu, D.-J,, Chen, Y., Zhang, Z., Yao, X., Xie, Z., Chen, Y., Liu, S., Qian, B., et al. CRAB: Cross- environment agent benchmark for multimodal language model agents. arXiv preprint w4Xiv:2407.055111 2024.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:230.009288 2023.
Yang, J., Jimenez, C. E.,, Wettig, A., Lieret, K., Yao, S., Narasimhan, K. R., and Press, O. SWE-agent: Agent- computer interfaces enable automated software engineer- ing. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a
Yang, Z., Liu, J., Han, Y., Chen, X., Huang, Z., Fu, B., and Yu, G. AppAgent: Multimodal agents smartphone users. arXiv preprint arXiv:2312. 13771, 2023.
Tufano, M., Agarwal, A., Jang, J., Moghaddam, R. Z., and Sundaresan, N. AutoDev: Automated Al-driven develop- ment. arXiv preprint arXiv:2403.08299, 2024.
Yang, Z., Zhou, Z., Wang, S., Cong, X., Han, X., Yan, Y., Liu, Z., Tan, Z., Liu, P., Yu, D., Liu, Z., Shi, X., and Sun, M. MatPlotAgent; Method and evaluation for LLM- based agentic scientific data visualization. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 11789-11804 1804, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 10.186 wtw.1/224..indnnssaade.. hology .rr/ 2024. findings-acl. 701.
Wang, J., Xu, H., Jia, H., Zhang, X., Yan, M., Shen, W., Zhang, J., Huang, F., and Sang, J. Mobile-agent-v2- Mobile device operation assistant with effective nav- igation via multi-agent collaboration. arXiv preprint 1xXiv:2466.01014 2024a.
Wang, X., Chen, Y., Yuan, I.., Zhang, Y., Li, Y., Peng, H., and Ji, H. Executable code actions elicit better LLM agents. In ICLR 2024 Workshop on Large Language Model (LLM) Agents.
URL https: /aaclaacttttt
Zhang, F., Chen, B., Zhang, Y., Keung, J., Liu, J., Zan, D., Mao, Y., Lou, J.-G., and Chen, W. RepoCoder: Repository-Ievel code completion through iterative re- trieval and generation. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2471 2471-2484, Singapore, December 2023, Association for Computational Linguistics. doi: 10.18653/w1/2023.oo Ip-main. 151. URL https: a c alaathhllollg ogy . org /2023. mamm--mmain.. 151.
Wang, X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., et al. OpenDevin: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741, 2024b.
Wirth, R. and Hipp. J. CRISP-DM: Towards a standard process model for data mining. In Proceedings of the 4th international conference on the practical applications of knowledge discovery and data mining, volume pp-
29-39. Manchester, 2000.
Zhao, R., Zhang, W., Chia, Y. K., Zhao, D., and Bing, L. Auto arena of LLMs: Automating LLM evaluations with agent peer-battles battles and committee discussions. arXiv preprint arXiv:2405. 202677 2024.
Wooldridge, M. Intelligent agents. Multiagent systems: A modern approach to distributed artificial intelligence, 1: 27-73, 1999.
13

---

Agent-as-a-Judge: Evaluate Agents with Agents
Zheng, L.,, Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z.,, Li, D., Xing, E., et al. Judg- ing LLM-as-a-Judge with MT-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Zhou, W., Jiang, Y. E., Li, L., Wu, J., Wang, T., Qiu, S., Zhang, J., Chen, J., Wu, R., Wang, S., et al. Agents: An open-source framework for autonomous language agents. arXiv preprini arXiv:2309.08870. 2023.
Zhou, W., Ou, Y., Ding, S., Li, L., Wu, J., Wang, T., Chen, J., Wang, S., Xu, X., Zhang, N., et al. Symbolic learning enables self-evolving agents. arXiv preprint arXiv:2406. 18532, 2024.
Zhuge, M., Wang, W., Kirsch, L., Faccio, F., Khizbullin, D., and Schmidhuber, J. GPTSwarm: Language agents op- timizable graphs. In Forty-first International Conference on Machine Learning.
Zhuge, M., Liu, H., Faccio, F., Ashley, D. R., Csordás, R., Gopalakrishnan, A., Hamdi, A., Hammoud, H. A. A. K., Herrannn V., Irie, K., et al. Mindstorms in nat- ural language-based socicties of mind. arXiv preprint arXiv:2305.17066, 2023.
14

---

Agent-as-a-Judge: Evaluate Agents with Agents
A. Task Sample
Task 51: Devin_Al_Soffe Sowftware_Enginneeeee Planet in_ Images
Query
Hi! Please follow the instructions from the blog post Hidden in Plain Sight to set up the script for generating images with hidden text in src/vi sual z.e.pp... py. Ensure the generated images are of 1080p resolution and saved in resul tss. Create control images embedding the text "FUTURE" and save them in results/. Please manually verify that the hidden text is embedded in the images.
Requirements
RO
Criteria: Follow the instructions from the blog post Hidden in Plain Sight to set up the script for generating images with hidden text ere/visualize.py. Dependencies
R1
Criteria: Ensure the generated images are of 1080p resolution and saved results/. Dependencies + (RO}
R2
Criteria: Create control images embedding the text "FUTURE" and save them in results/. Dependencies + {R1}
Preferences (Optional)
PO
Criteria: The system should be capable of learning and adapting to unfamiliar technologies and tools as required.
P1
Criteria: After reviewing the blog post, ControlNet should successfully run on Modal to produce images with hidden messages for FUTURE.
Figure 7. A task example in DevAL. This task is adapted from real-world demo given at https: / www. www.wtiwgniii...cmmmmmmm wttittiioonnni//aaiaaooo ai/blog/ int roducinggg--ooooo devin. this example shows, task requirements DevAl are structured a Directed Acyelic Graph (DAG), with nodes representing individual requirements and directed edges showing dependencies. More examples are in Appendix G.
15

---

Agent-as-a-Judge: Evaluate Agents with Agents
B. Experiment Designs
This section outlines experimental designs aimed at evaluating developer agents' performance, analyzing human evaluations, and comparing Al-based judging systems. The experiments are structured across three levels, as illustrated below.
Section 2.3
Section 3.1
Section 3.2
Section 4.2
Section 4.3
Judging Developer Agents Level
Judging Human-as-a-Judge
Judging Agent(LLLM-aa-aaJudde Level
Level 2
B.1. Summary of Experiments
The experiments are categorized into three levels follows:
Level 1: Human evaluation of developer agents
Experiment la: Basic performance statistics for developer agents (Section 2.3)
Experiment lb: Human evaluations of developer agents (Section 3.1)
Level Error and bias analysis of human evaluations
Experiment 2a: Error analysis of human evaluations (Section 3.2)
Level 3: Al-hased judging systems
Experiment 3a: Al judge baselines (Section 4.2)
Experiment 3b: Ablation studies for Agent-as-a-Judge (Section 4.3)
B.2. Judges and Subjects of Evaluation
The following table summarizes the judge and the subject being evaluated each experiment:
Experiment
Who is the Judge?
Who is being Judged?
Section 2.3
Human
Developer Agents
Scction 3.1
Human
Developer Agents
Section 3.2
Human
Human
Section 4.2
(1) LLM-as-a-Judge
(1) Developer Agents
(2) Agent-as-a-ludge
(2) Developer Agents
(3) Human
(3) LLM--as -as-a as-a--udgee
(4) Human
(4) Agent-as-a-Judge
Section 4.3
Human
Agent-as-a- audde
16

---

Agent-as-a-Judge:
Evaluate Agents with Agents
C. Agent-as-a-Judg Pipeline
Agentic Task
Hil Please follow the instruction and set up the script from the blog https: h/tpp::///www./ www.saachiineee...maa at/p/hidde in-plain-sight to generate 1080p images with hidden text ("FUTURE, in ptrcvvssuaizz... py. Save them in results/ and verify the text embedded.
User
Develop Process
Open mentioned link carefully read blog mentioned the user query.
Go to the GitHub repository mentioned the blog, and read the code and the README file.
Set up the environment, then install the package.
Set the Modal library by installing and configuring the API for serverless deployment.
Write and run src/visua size. py to generate masks and save the figures in the results/ directory.
Paem set Goed Srortt-
Developer
Agent
FOREST
Judge Process
workdn aorkspace
FUTURE
Build the workspace graph.
visualzze
+Q
results
mask.jpg
Collect information based {RO) and the workspace graph.
ricklen.pp
jag
simages
hidden.jpg
visualize.pp py
Are the requirement
{RO) satisfied?
Verify all requirements and their related analyses.
Judge Agent
Graph
Read
Ask
Retrieve
Locate
Figure 8. The pipelines of developer agents and judge agent.
17

---

Agent-as-a-Judge: Evaluate Agents with Agents
D. Extend Related Work
Our main paper includes mostly related works of Al developers, Benchmarks for Al developments, and AI judges. However, the following works contribute significantly to the community and also relate to this work. We record this work as additional related work'.
LLM-based Autonomous Agents Recent developments in LLM-based agents have expanded their capabilities be- yond simple task execution to more autonomous problem-solving and decision-making. AutoGPT (Gravitas, 2023) and LangChain (Chase, 2022) provide frameworks for angle-agent systems that leverage external tools for more complex tasks. Similarly, research such as MetaGPT (Hong et al., 2024b), AutoGen (Wu et al., 2023) and CAMEL (Li et al., 2023) focus on role-based multi-agent communication, improving collaboration among agents. However, the challenge of maintaining coherence in agents' dialogue and preventing hallucination remains prominent (Du et al., 2024; Zhou et al., 2023). Most recently, using graphs to build agents has gained prominence. Earlier work like GPTSwarm (Zhuge et al.) and LangGraph (LangChain-AI, 2024) proposed using nodes to represent operations and edges to represent the connections between them. In GPTSwarm, multiple agents represented as subgraphs in a graph are connected by optimizable edges, and reinforcement learning is employed to optimize the edges. Following this approach, several agent frameworks have incorporated graphs into their designs (Hong et al., 2024a; Zhou et al., 2024; Qian et al., 2024b). Additionally, various optimization methods have been developed to enhance agent performance further (Wu et al., 2024; Song al., 2024; Hu et al., 2024). In practical applications, many studies focus on understanding and interacting with GUIs (Wang et al., 2024a; Chen et al., 2024a; Yang et al., 2023; Xu et al., 2024; Tan et al.). For code generation agents (Jin et al., 2024), current research mainly emphasizes automated repair (Yang et al., 2024a; Phan et al., 2024; Tao et al., 2024), computational modular design (Khattab et al., 2024; Cheng et al.), and automated development (Tufano et al., 2024; Huang et al., 2023). Among these, open-sourced frameworks like OpenHands (Wang et al., 2024b) have gained popularity due to their strong user experience. Moreover, scientific discovery (Jansen et al., 2024; Lu et al., 2024a) and ML agents (Yang al., 2024b) are also receiving increased attention.
LLM--as-aJJudee the domain of Al evaluation and judgment, frameworks (Zheng et al., 2024; Fu et al., 2024; Chen et al.) have pioneered the use of LLMs assess conversational agents, demonstrating how LLMs can evaluate dialogue quality and consistency. Expanding beyond dialogue, LLMs like CodeR (Chen et al., 2024b) and MASAI (Arora et al., 2024) apply similar judging principles to the code validation process, where AI systems autonomously evaluate and verify computer programs. Our work builds on these advancements by exploring how LLMs can perform more nuanced judgment tasks, further investigating their potential in decision-making across various domains. Recent research also focuses on judging LLM LLM-as-a-Judeee as-a--ude (Chen et al., 2024c; Bavaresco et al., 2024; Thakur et al., 2024; Dong et al., 2024; Shi et al., 2024; Raina et al., 2024).
Coding Benchmarks Recent advances in code generation have led to the innovation of various benchmarks to evaluate model performance (Liu et al., 2024). Early benchmarks, such as MBPP (Austin et al., 2021), HumanEval (Chen et al., 2021), and MultiPL-E (Cassano et al., 2023), focus primarily on generating simple functions. While these benchmarks are useful for evaluating the correctness of generated code, they are limited in complexity and do not fully represent the challenges encountered in real-world software development.
As the field progressed, newer benchmarks began to focus more complex and realistic tasks. APPS (Hendrycks et al.), CodeContests (Li et al., 2022), and LiveCodeBench (Jain et al., 2024) moved toward competitive programming challenges that involve advanced algorithms and data structures. These tasks are more representative of problems encountered in coding competitions and help push models toward more sophisticated problem-solving. DS01000 (Lai et al., 2023) was introduced to assess the skills of models with data science libraries, evaluating their ability to use APIs and execute complex data analysis workflows. Meanwhile, AgentBench (Liu et al., b) focuses on testing reasoning and decision-makinn abilities in interactive environments, highlighting differences in performance between commercial and open-source models.
To address real-world programming needs beyond code generation, specialized benchmarks have been created to evaluate tasks such as debugging, refactoring, and code navigation. CANITEDIT (Cassano et al., 2024), DebugBench (Tian et al., 2024), and FixEval (Haque, 2023) evaluate the ability of model to edit and improve existing code. Additionally,
'Additionally, we were pleased to find that a recent industry blog (attps: / //www cogn t tion ai aaabbl/g////////aaaeeeeeeebbetla eva luating oding- agents), published two weeks before our submission, shares very similar ideas and provides further evidence that the Agent-as-a-Jude could have practical applications in agent systems.
18

---

Agent-as-a-Judge: Evaluate Agents with Agents
benchmarks such SWE-Bench (Jimenez et al.) focus on resolving issues in GitHub repositories, simulating practical software development scenarios. Finally, benchmarks such as RepoBench (Liu al., and RepoEval (Zhang al., 2023) delve into the evaluation of models in large-scale, multifile codebases. These benchmarks measure the ability of language models to understand the structure of repositories and solve problems within more complex, collaborative environments.
19

---

Agent-as-a-Judge: Evaluate Agents with Agents
E. The Procedures of Creating DevAI Dataset
E.1. Manually draft user queries
Given the execution cost of the developer agents, we collect small-scale Al tasks to ensure the practical applicability of our benchmark. Since these tasks are small-scale and well studied, which are easy to overfit in terms of task performance metrics, unlike previous benchmarks (e.g., (Huang et al., 2024)), we do not evaluate task performance the development performance measure. Instead, prioritize the step-by-step task-solving ability, which is essential for real-world development. Our quires are specifically designed to require the development agents to understand user intentions, solve the task in multiple steps, and adapt to unexpected step outcomes. This approach also makes our benchmark user-friendly. transparent, and better reflects real-world deployment situations. To enable effective evaluation, our queries present a specific development file structure for the developer agents to follow. To ensure that the developer agents save the files to be evaluated in the workspace, we develop constraint prompts added to the query to form an extended query. The constraint prompts guide the developer agents save and execute the source codes, which are in line with the real-world development standard. See Appendix for our constraint prompts.
E.2. Set Judging Criteria
To make the evaluation of the developer agent precise, we assign to each task query list of requirements as task milestones. The requirements are chosen so that satisfying all the requirements is necessary condition to consider the task to be solved successively. Since our tasks are Al-centric, our queries target essential elements of AI development, including data processing, Al method, presentation of evaluation metrics, visualization, and human-compunn computer interaction, covering the key areas that matter most in real-world scenarios. On the other hand, breaking down tasks into individual requirements also reflects the multi-step nature of code development. Importantly, to avoid ambiguity, we set the requirements to be explicit, binary, and straightforward to evaluate. To include other human predispositions, we include a list of preferences per task that covers subjective, ambiguous, or non-explicitly stated characteristics.
E.3. Building Dependency Among Requirements
To enhance the realism of our benchmark, we analyzed the dependencies among requirements. Over the past decades, methodologies such as the KDD Process (Fayyad et al., 1996) and CRISP-DM (Wirth & Hipp. 2000) have guided ML/AI development, establishing foundational frameworks that have been further refined with the rise of AutoML (He et al., 2021).
Inspired by these methodologies, we identified a seven-step process for automated Al development tasks. This process includes critical stages such as data preprocessing, feature engineering, model selection, and hyperparameter tuning, along with essential post-development activities like metrics recording, report generation, and the development of interactive user applications. This structured approach allows us to evaluate the agent's ability to manage task dependencies and effectively navigate complex, real-world scenarios.
E.4. Refine the dataset
Manual refinements were necessary to ensure the accuracy and clarity of DevAI. We perform two rounds of comprehensive review and edits on DevAI, each round being done by a different participant. During these manual refinements, we focus on the logical consistency of our queries and requirements, the correctness and ambiguity of the language, and the applicability of the content to the task domain. We identified moderate number of errors in our dataset during the review.
E.5. Analyse the dataset
We categorized each requirement based on its focus, whether it was related the data set, the machine learning method, visualization, metrics, HCI, or data processing. While this approach provides useful framework, it is important recognize that these categories might overlap miss certain nuances. Similarly, preferences were classified by how strongly they appeared in the query text, ranging from those inferred by common sense to those explicitly stated. Although this helps to organize preferences, it is worth noting that such classifications can be subjective and may not fully capture the importance of each preference in practical applications. By structuring the requirements and preferences this way, we aim to enhance the evaluation process, though flexibility and context awareness remain crucial for truly robust assessments.
20

---

Agent-as-a-Judge: Evaluate Agents with Agents
E.6. Auxiliary Information
Some of the tasks require the download of Kaggle ( data set, where Kaggle credential is needed. Our con- straint requires "is kaggle. pegggleapi.... needed" tag determine whether the credential is needed to be included in the extended query. We further mark each task with some tags describing the AI sub-fields related to the task, including computer vision, supervised learning, reinforcement learning, natural language processing, etc., as well as and www.:ee.naw.gatioo..ee.e.... gorize each requirement into one of the following: (1) dataset or environment, (2) data preprocessing and postprocessing, (3) machine learning method, (4) save trained model, (4) performance metrics, (5) human computer interaction, (6) iiataiiiiioonn and (7) other, reflecting the nature of the requirement.
two w.s.sraniinggneeeeeeee
tags as auxiliary information. We also cate-
E.7. A Json Format of Our Sample
Here, we provide a sample of the DevAl with its json format. We also provide more samples in Appendix G.
"name":
"quory": "I am secking a speech montion recognition project using a CNN-LSTM model with the RAVDESS dataset, which should be downloaded from Kaggle [this Hugging Face link] (https:/ (https://huggingg) httgggggggaggce...// httpffacc.c//aatseeeeeeeee wtt.attsets///eeeeeoaaassaaassae sesetts/xxoooeerradeeeeee
The project should load the dataset and perform robust audio prepracessing (noise removal and normallzee ization) and MPCC feature extraction implemented in 'src/data_loadee 1oadatrp__ The CNN-ISTX model should be implemented in "src/nodel.py" Recognition accuracy accuraccytt and confusion matrix.png'
should be saved in natrix should be generated and saved as "realts//iiuurs//onnte Additionally, a user-friendly local API should be created using Flask to allow users to upload audio files and receive emotion recognition results, with the implementation included in 'src/hci.py'.", "tags":
"Audic Processing", "Classification"
), "requirements";
"requirement_id" 0, "prerequisites": [],
"criteria"; "The \*RAVDFSS)* dataset is loaded in 'sre/data_loades which downloaded from Kaggle or [this Hugging Face link] (https://uuggingffaee co/datasets co/dattsetss/xbgoose//// abogoose/ravdess)....
"category"; "Dataset or Environment", "satisfied": mull
"roquirement_id"; 1, "prerequisites": [ 0
}, "criteria"; "Audio preprocessing, including noise removal and normalization, is implemented in 'src/data /ttc/atta_Loueerpp....... "category"; "Data preprocessing and postprocessing", "satisfied"; null
"zequirement
id": : 2, "prerequisites": [
1.
"critoria": "MFCC feature extraction is implemented in lord/rata "category": "Data preprocessing and postprocessing", "satisfied": null
21

---

Agent-as-a-Judge: Evaluate Agents with Agents
"requirement_id"" 3,
"prerequisites": [],
"criteria": "The \"CNN-LSTM) " model is implemented in 'src/nodel. .py "category"; "Machine Learning Method", "satisfied" mull
"requirement_id"; 4,
"prerequisites": [
2,
3
"criteria"; "Recogniticn accuracy is saved in sesults/metries/ recognition
http:/triooccuuacy....... accuracy.txt
"category"; "Performance Ketrics", "satisfied"; null
"requirement_id"" _idreeemett_tdt::::: 5, "prerequisites": [ 2,
3,
J,
"criteria"; "The confusion matrix is generated and saved as 'resulta/ secuultafiigrreee fugure/ confusion_mattix.png
"category"; "Visualization"
"satisfied" mull
"requirement_idd": 6, "prerequisites": [
2,
3
l, "criteria": "A local API created using \"Flask)" allow users upload audio files and receive emotion recognition resuits. The implementation should be included in 'src/hci.py'.", "category"; "Human Computer Interaction", "satisfied"; null
1,
"preferences":
"preference_ioe _d":
"criteria"; "The audic preprocessing step should be robust, effectively reducing noise while preserving the integrity of the speech signals.", "satisfied"; null
"preference_id"" 1, "criteria": "The local API should be user-friendly, with clear instructions for uploading files and interpreting results.", "satisfied"; null
1, "is_kaggle_api_neddd""" true, "is dis_training_needer truc, :
22

---

Agent-as-a-Judge: Evaluate Agents with Agents
F. User experiences of code-generation agentic systems
OpenHands (Wang al., 2024b) offers the most refined user experience, leveraging its highly interactive frontend to enable seamless user interaction and task execution. This interface allows users to engage directly with the system, resulting in smoother and more intuitive workflow, which drives operational efficiency.
In contrast, MetaGPT (Hong et al., 2024b) excels in task decomposition through its use of Directed Acyclic Graphs (DAGs), well-structured and scalable approach aligned with industry best practices in system modularization. This enhances its appeal for users focused on task clarity and modular breakdowns. However, in practical deployments, MetaGPT tends to be less aggressive in file management and preservation, potentially due to its core positioning as data analysis tool, which does not prioritize persistent state management. Similarly, OpenDevin demonstrates notable overconfidence its code generation, frequently skipping the critical step post-generation code execution, requiring users intervene manually.
GPT-Pilot (Pythagora.io, 2023), praised for its detailed task delegation via over 20 specialized agents, suffers from reduced interactivity due to overly granular division of responsibilitiess, resulting in more fragmented user experience. These qualitative insights, although not fully captured by quantitative metrics, were evident through the DevAl dataset, providing key areas for improvement in user engagement and operational fluidity in future releases of these frameworks.
23

---

Agent-as-a-Judge: Evaluate Agents with Agents
G. More AevAI dataset samples
We listed more more samples in Figure 9, Figure 10, Figure 7.
24

---

Agent-as-a-Judge: Evaluate Agents with Agents
H. Human Evaluation Procedure
recruited three Al experts from the authors to perform human evaluation on the output agentic code generation systems. There we present the evaluation details.
First round For the first round of evaluations, our three evaluators reported spending 16.5, 19.5, and 22.0 hours, respectively. To capture the bias that human evaluator will have, the instructions given experts minimal, with them only receiving scorecard to complete for each agentic system and each task. Results that all evaluators agree on are considered trustworthy. The assumption here is that it is unlikely that all three evaluators make mistake or have an effective bias in the same judgment. The self-reported post-hoc evaluation criteria are shown in Figure 11.
Second round In the second round, the evaluators present and discuss their reasons for disagreeing with judges. In doing so, human errors are likely corrected by their peers. Discussion among evaluators also helps reduce human bias by examining each other's thought processes thoroughly. Furthermore, the consensed results are considered trustworthy given the assumption that it unlikely that all three evaluators are convinced by the same mistake or the same cognitive bias. The three evaluators took 9.5 hours together for this second round of evaluation.
25

---

Agent-as-a-Judge: Evaluate Agents with Agents
1. Suggest Constraints
Below sample of constraints in JSON format that describes task-specific guidelines:
"generic": "This is a task that requires you to write, execute, and save source code. You have a hard time limit of 30 minutes to produce your programmatic solution to the given task. This time limit includes execution time. The quality of your solution will be judged based on what you left in the working folder by the time 30 minutes expire. Additionally, the hardware you are running on unknown, and the presence a GPU is not guaranteed.",
"is_training__eededd" "Keep the time limit in mind when setting hyperparameters for training.",
" "is wer_kaggle_apinneeee "You your current working directory."
the Kaggle API credentials stored in xaggle.json:
To address automation and security concerns, have written code place the kaggle json file into the current workspace for each baseline during each run. However, dataset free modify the solution to enable the Kaggle API for their specific developer agents.
26

---

Agent-as-a-Judge:
Evaluate Agents with Agents
J. Collected Trajectories
J.1. Schema
Below is the required JSON format for a trajectory in gray-box settings (where the trajectories can serve as input for LLM-as-a-Judge and Agent-as-a- asda-..
"type": "array", "items":
"type": "object", "properties":
"step":
"type": "integer",
"description"; "The step number in the trajectory, 0-based.
useer message":
"type": ("string", "null"], "dcscription": "The message from the external user to the agent, If null, no message was sent."
agent":
"type": "object", "properties": {
"thought":
"type": "string", "description": "The agent's thought this step."
"actien":
"type": ["string", "null"],
"description": "The agent's action sent the environment. null, the agent did not take any action, for example, when the agent has finished the task."
}, pevent_name;; {
"type": "string",
"description": "The nane of the agent that made the action."
required"; ["thought", "action"],
"description": "Everything related to the agent this step."
"enviremment":
"type": ("string", "null"], "description"; "The envircnment's s (shell, python interpreter) response the action submitted by the agent. If null, the environment was not involved in this step."
"step_usage""" "type": "object", "properties":
Linput_tokens".
"type": "integer", "description": "The number of input tokens passed as L.M context." " 1 "cutput_ "ittputttokkes"::
"type": "integer",
"description": "The number of tokens produced by the LIM.
model";
"type": "string",
"description": "The name of the LLM model used."
"cost": {
27

---

Agent-as-a-Judge: Evaluate Agents with Agents
"type": "number", "description": "The cost of the LIM inference, in USD."
b
"ilm inference_ting : {
"type": "number",
"description": "The time taken by the LLM to generate the output tokens, in seconds."
"step_exeee execution_time
"type": "number",
"description": "The time taken to make an entire step inal udinu LLM inference and environment execution, in seconds."
"rquuire"; Sinput_tokens",
"output_tokens",
"model",
"cost",
"1im_inference_tine"
"ster_executior_tine
accumullated_ssaee usage":
"type": "object", "properties":
" "accumulated_cost""
"type": "number", "description": "The total cost of the trajectory up to this step, in USD. "
"accumulated time":
"type"; "number",
"description": "The total time taken by the agent to complete the trajectory up to this step, in seconds."
"required; [ "accumulated cost"" "accumulated time"
},
"required"; ("step", "user_message", "agent", "environment",, "step_usage",
28

---

Agent-as-a-Judge: Evaluate Agents with Agents
J.2. A Sample of Trajectory
This is sample trajectories (we select steps of it) in OpenHands (Wang al., 2024b). All steps are recorded enable replay of the trajectory.
"step" 0, "user_mests : "Help develop object detection system using the YOLOv3 model and the coCo dataset. Download the dataset and preprocess the images by resizing and normalization in "src/data loadeetr_ooddepp Implement the YOLOv3 model and use Non-Maximum Suppression (NMS) to refine the results in 'sre/model.py Save the detected objects to 'resuits/tigures/" and create interactive streamlit web page "src/app-py py display the detection results. Finally, evaluate the model's performance, including metrics such as MAP and inference time, and save the evaluation results to txt'. The systen should properly manage the launch and termination of the is task
Streamlit application to prevent unnecessary resource usage.\n\niin that requires you to write, execute, and save source code. You have time limit of 30 minutes to produce your programmatic solution to the given task. This tine 1imit includes exccution time. The quality of your solution will be judged based on what you left in the working folder by the time 30 minutes expire. Additionally, the hardware you are running on is unknown, and the presence of GPU is not guaranteed. AnknKep the time limit in mind when setting apperparameters for training.An". ageen": I
"thought"; "Let's s break
the task into manageable steps and start implementing them one by one. inlnt Steps:\n1. **Set up the directory structurerx:ha Create necessary directories: 'sro/" and 'results/ figures/ An2. **Download and preprocess the CoCO dataset**:>n Implement "src/data_ n3. Implement the YOLOV3 model..:<n
to download, resize, and normalize images. Implement 'src/medel.py to define the YOLOv3 model and apply Non-Maximum Suppression (NMS) \n4. * Create an interactive Streamlit web page**:'n to display detection results. An5. **Evaluate the model's performancer*: httpa////mcrriccommmeeeeeeeeee
Implement 'src/app.pp
Save evaluation metrics to nIet's start by setting up the directory structure and creating the
txt An necessary dLles.\n\nfHt Step 1: Set up the directory structure",
"action": "Running command: mkdir "p src results/figuret resulta/metrin ntouch src/data_loader..n src/model .yy arc/app.py"
"environment":
"krin/Pythen Interpreter: pepeehaand//petrr/ppprieeee
aCHf-py3.
11/bin/python /
$ touch src/data__oader.ppp
szc/model.py opetfatd=-O00000ttts33.11////// http:///hhoooooooccccceeaaaaeeee htt.aaannaabba4222aeaaaaaa/aeeee
(Pytho (Python Interpreter: /openhanca/ppetrry
stepp_sage";
"input_tokens": 4331,
"cut-put_tokens" 220,
"1lm [llm__nference_tae time:: 4 4.445789337158203,
"model": 4opt--o-2024--5-113,
"cost": 0.024955, itaepeexecution_timee 16. 16.2492122650146
Paccmuu aatpumuulaedd__ageee" "accumulated__oom 0.024955, "accumulated_tiiee 16. 16.24922204017639
"step"; 1,
"uscr_message" null,
"agent": {
"thought": "Great! The directory structure. and necessary files created. Let's proceed with implementing the data loader to download and preprocess the coco dataset. Anin#t Step 2: Implement lsrdde .py 'nil will:
29

---

Agent-as-a-Judge: Evaluate Agents with Agents
Anl. Download the coco dataset. \n2. Resize and normalize images, \n n start by writing the code for aro/ddat__lodde.py".. ,y",", "action": "Running Pythen code interactively: create_file s" )
"enviieennen renment": "FileExistsärror
Traceback (most recent An)
call last) AnCell In[1], 1ine 1\m----> nFile hawpaaandd/cod///ppebeeeee
file with the given name. \n 252 \n 253 Args:An 254 filename: str: The name of the file to create.
.py:257, in ereate_ (filename) \n 251 """""tttsats and opens
255 \"M"\in 256 if ox.path,existst
:ilename): : An-->> 257 raise "ilehxistskrror (f(""ile (fillename)' already
exists. \") 259 with open (filename, 'w') as file:An 260 file.write
File 'sre/data Loaderr__ alreacy exists. Anl
Japyter current working directory: /workspace \n Jupyter Pythen interpreter: pCt.-pyl.1//iin//pttttt.
"step_usage";
" "input_tokens": 4675,
"output_tokens":: 84, "1im_inference_tine 2. 2.136143207550049,
"model": "gpt-4o- "got-4o-2024--5--3"",
"cost": 0. 0.0246350000000004,
"step_executionntiinge :. 3.345384359359741
"accumulated_usaa :
"accumul "e :.000::00..00000000 0.0495900000000001,
"accumulated_tiiee" 19.5946080684666187
"step": 2,
"uscr_message"; null,
"agent": {
"thought"; "The file 'src/data_loader" already exists. Let's a open it and start writing the code to download and preprocess the COCO dataset. AnAnt# Step 2: Implement 'are/data saidattaloader.py"""
"action": "Running Python code interactively: open_file( wtc///ta_daaaaaaaaaaaaaa stre
"environment"; "[File: herksppacee/scc/ttaaa_adddd
(1 lines total) (this is the beginning of the Hile) \nle) (this is the end of the file) \n Jupyter current working directory: Mn Python interpreter:
step_usage"; "amputttokees": 4982, "cutput_tokens" 53,
"1lm inf.reffeenceeccmm time": 2.2097568551196299,
"model": "gpt-4o-202405-133,
"cost": 0. 0.025705000000002, "step_executioon_iiee"" 2 2.318861961364746
Paccmml "accumllated_ussee att__uggeed__
"accumul aateumosted___tttte_ee 0.0752950000000001,
"accumul tt 21.9134719377779555 937717955
b,
"step": 20,
"user_messaa : null,
"agent":
"thought": "It seems the indentation issue persists. Let's correct the indentation Aouually.Annfen Corrected Code for "src/evaluatee.py python\ninport time Animport numpy nplnfrom model import yolo_v3_model, non_max_supel wwwp:_pppprrssoooseeeee data_loades import preprocess imaooe,ss__mae,, wownooad_ccooddda cot.eea.asee extract_coce datases 'nimport os onlndde
30

---

Agent-as-a-Judge: Evaluate Agents with Agents
evaluate_nodell () model yolo_v3_model () down_cad_coco_daan On aetract_coco_dataase () \n'n image_dir = vdata/val20117""M
image_files path. Limage_dir, 5) for f in os se.listair if (pt len (inagce__ffiie []Annn for image_file in image_files:te
cotal_inference.ee boxes [1 all_scores [1 all_olasses
0'nn num num_ sun_images
image
proproccss_inage (image__iil Annnn (image) \n end_time start_time
Image = np. expaaddd_cmmm (elim((inagee (image, axis=0
start_time
() in
predictions = model.predics
end_time time.time In'n
inference_tine
total_inferenee_ttee
inference semrrnne__tmmiiie timeln in'n
boxes, scores, classes non_max_aupprese (predictions [0]) \n all mi__oxee.pppene (boxes) all_ci asses append \n\n total inference time
all_scores.appent (scores) \n
images mean_inference_tinn
Placeholder for mAP calculation mAP 0.0 # This should be zeplaced with actual mAP calculation'ala with open htt/:/auull//meet//////oooo
\"w\") as f:ln seconds name___ \"_main
f.write (f) Meenn Inference Time:
An
wwwrit (f\ "MAF:
\n evaluate_model An "nlnnet' update the file with the corrected
code.",
"action": "Running Pythen code interactively: EDITED_ _ODE = \"\"\"import timel nimport numpy np\nfrom model import polo_v3_modeil non mon_max_suu max_suppression! nfrom data_loader import preprocese image,, download_coco_ pioco_dattset,
extract_coco yolo_v3_mode_ \n nownload_coco....o () An image_dir \Vdata/va12017)"M image_dir, for f in os.1istdir (sslisstin if f.f.feem f (pg')] total_inference_thhe boxes = (l image_file in image_ files:\n
aataaat__oco_ddaaaeeeeeeiiee wtw.daaaeaaeanppr..eeeeeeeeea os os'nindet evaluate_model1)1
model
extract_coco_datasse () \n
image_files - [os.path.joinn
O\n num_images len (image_files)
all_boxes
all scores [1An all classes [1mn'n for
image preprocess_jmage (image_file) In image = ap. appeepand_diige (image, (xis=0)\n\n
start_ pamt_timee_e = time end_time
.time () \n
predictions model (image) \n
time.time (ime \nin
inference_time end_time start_time'a
total_infercnce_tlle +- inference__immevnn
boxcs, scores, classes
httpaa_spupreessioopppeeeeeeeeee (C])
all_boxes.appent (boxes) In all_classes (classsess \nin num_images'''n
ali_scores [es_scores.aapeend(scrreee \n imn_ total_inference_ciinne oagarrreennnaiiieoooooaaeeeeeeee
(classes)
Placeholder for mAP calculation'n mAP 0.0 This should be replaced with actua_ mAP calculationnill with open wtwl//rreeeeeeriiieeeeeee
model ", \"w\") as f:An
f.write (Menn Inference
Time: [menn_infeereeee__iiee:e: secondsllllni"")
f.write (f) (f\"mAP: "MAP:
\_main_ \":An
evaluate_modell An\"
'with open (cresults/
betrice/nade..pertormmeeee..s.s \"w\") as E:",
with open "results/
aetriee/node__performeeeeeee \"w)") as E:")"
b,
"envircnment"; "step_usage";
null,
"input_tokens" 20132,
"cutput_tokens": 805, "1]m_inference_time 15. 15.1555523147583
"model": 100--10--004----11,,,,
"cost":
0. 0.1127355000000000
"step extep_execttion time": 1.. 15.196227073669433
aaccumulated useeee
"accumulated__oott": 1.2919750000000003,
"accumulated_tiiee 176. 176.99772510256998

---

Agent-ax-a-Judge:
Evaluate Agents with Agents
K. Ablations of Agent-as-a-Judge
K.1. Component Abalations
Table 5. Component Ablation Studies for Agent-as-a-Jude- We analyze the impact of sequentially adding various components (ask, graph, read, Locate, retrieve, planning, menory, and search) on the performance of Agent-as-a-Judae for judging OpenHands.
Metric
ask
graph + read
+ locate
+ search + +retrieve + +planning + memory
Agent-as-a-Judge Performance
Alignment Rate 65.03%
75.95%
82.24%
90.44%
86.06%
90.16%
88.52%
87.97%
Analysis We designed 8 modular components for the Agent-as-a-Judge system. In the Table 5, components are added progressively from left to right. If the addition of a component led to a significant performance drop, we removed from further iterations. Our experiments showed that adding the components ask, graph, read, and locate resulted in significant performance gains. However, when the search component was introduced, there was a noticeable decline in performance.
We hypothesize that the performance drop from search is due to its role in retrieving relevant code snippets (top-3) using BM25. The retrieval accuracy of BM25 (Robertson et al., 2009) might not have been high enough, potentially introducing noise. Moreover, as noted in Table 1, the DevAI tasks in our experiments did not generate a large amount of code. In fact, even when all code was fed into an LLM, the total content typically stayed within the maximum context length. Therefore, in simpler workspaces, search was less critical. However, we believe this component will become more important as the complexity of the workspace increases, making more valuable in larger and more complex environments.
We also observed that the introduction of the planning mechanism did not bring noticeable improvement in performance. This may be related to the nature of the Judge it needs clean factual information. When planning is unstable, the evidence collected from different actions can become inconsistent, leading to decline in performance. Finally, we experimented with memory mechanism. Initially, we hypothesized that since DevAl tasks often involve interconnected requirements, memory could help track whether requirements were met. However, in practice, we saw no improvement. We suspect that the interconnected nature of the requirements may have caused biases: specifically, once a prior requirement was fulfilled, might have overly influenced positive judgments on subsequent requirements, even if they were not fully met.
K.2. Search Algorithms in Search Module
We initially hypothesized that the performance drop was due to the low precision the search component, particularly with BM2.5. To explore this, we replaced BM2.5 with Sentence- BERT (Reimers Gurevych, 2019) as more advanced alternative and tested Fuzzy Search (Levenshtein, 1966) as less precise option. However, neither improved the performance of the Agent-as-a-Judge
hese results suggest that the performance issue is not due to BM2.5's poor search accuracy. Instead, the workspaces generated in our DevAI tasks are too simple for the search component to have a significant impact. In simpler workspaces, direct retrieval and evaluation are sufficient. Even though Sentence-BEi BERT performed better than the other methods, its align- ment rate (87.70%) still falls short of the configuration without the search component (90.44%) As workspace complexity increases, the search component may become more valuable.
Table Comparisons on Search module with different engines.
Search Method
Alignment Rate
BM2.5
86.06%
Sentence-BEi
87.70%
Fuzzy Search
85.52%
without Search Module
90.44%
K.3. Search Algorithms in Retrieve Module
In our experiments, found that accurately locating relevant information within trajectory is challenging task. Although the addition of the retri rive eve component (gray-box) did not lead to significant improvement in performance in this specific case, its impact has been notable in other settings, such as in GPT-Pilot. As shown in Table 3, the integration of rivrieve in GPT-Pilot brought substantial gains.
32

---

Agent-as-a-Judge: Evaluate Agents with Agents
Table 7. Ablations on retrieve.
We conducted ablation study on GPT-Pilot to optimize the retrieval of useful information at each step. Our experiments revealed that in large trajectories, truncating the final sections of the file often results in losing critical information, as the latter part of the trajectory typically contains dense information about the final development state. Truncating the beginning of the trajectory proved to be the most effective improving the retrieval efficiency.
Method
Alignment Rate
Without retrieve
83.88%
With retrieve (gray-box)
86.61%
Trajectory Truncate (head)
86.61%
Trajectory Truncate (middle)
85.52%
For individual steps, truncating the middle section worked best. This because eтror messages usually appear early in the output, while the corresponding file paths and specific error locations are found towards the end. By focusing on these retrieval strategies, we can significantly enhance the performance of the retri rreee evv component, particularly in complex scenarios like GPT-Pilot.
Trajectory Truncate (tail)
82.51% 1
Step Truncate (head)
86.34%
Step Truncate (middle)
86.61 %
Step Truncate (tail)
83.88%
33

---

Agent-as-a-Judge: Evaluate Agents with Agents
L. Prompt Demos of Agent-as-a-Judge
Here, we present some prompts used by the Agent-as-a-Iudge system. Each of these prompt demos plays a crucial role in guiding the agent's behavior.
L.1. System Prompt for Agent-as-a-Judge
def get_system_prrayy (Sagguage
if language == "English":
return
You advanced AI system. serving impartial judge for intelligent code generation outputs. Your primary role to rigorously evaluate whether the agent's 8 outputs satisfy the specifies requirements by thoroughly analyzing the provided code, data, and other relevant materials.
You will systematically
aspects such datasets, model implementations, training procedures, and any task-specific criteria outlined the requirements. Your ovaluations must be objective, detailed, and based solely on the evidence provided.
For each rement deliver one of the following judgments:
1. <SATT&FIRD>: Use this if the agent's s output fully meets the requirement. Provide briet and precise explanation demonstrating how the specific criteria are fuifilled.
2. <UNSATISFIED:: Use this if the agent's output does not meet the requirement. Provide concise explanation indicating the deficiencies or cmissions.
Your assessment should reference specific elements such code snippets, data samples, or output results where appropriate. Ensure that your justifications are clear, precise, and directly related to the criteria.
Respend with either <SATISFIED> <UNSATISFIED>, followed by your concise justification.
a."
else:
raise Not lnpeeeetrerr (f"The language (language)) is not supported.
34

---

Agent-as-a-Judge: Evaluate Agents with Agents
L.2. System Prompt for Locate Module
def get_ prompe "nnguugge=Englissti
if language == "English": return и*и
You are an advanced AI system. specializing in understanding project structures and determining file Locations based on provided criteria.
Your task is to locate specific files in the workspace based on the user's criteria and workspace information on problens with the files mentioned in the criteria.
else: raise Net NotimpplementeeBrro (E"The language (Language)' is not supported.")
L.3. System Prompt for Retrieve Module
def get ret_retrieve__yv system prompt (language= "ecpnagg==nngllain "English")
if language "English";
return и*и
You are an advanced AI system. specializing in retrieving environmenta feedback from project execution trajectories. Your task is to analyze the provided trajectory data and extract information about the most relevant files mentioned in the given criteria.
Focus on the following:
1. Identify the *oost recent steps** where the files directly related the criteria were involved execution, loading,
saving operations.
2. Provide environmental feedback for these files, such as any errors, warnings, or issues encountered during their execution or processing.
3. Highlight whether success these files in the project.
problems occurred that might affect the functionality
Your output should be structured as follows:
<<EEEEVTNT STEPS>**: List the specific steps involving the relevant files, including any envi ronmental feedback such error messages, execution results or other issues encountered. Rach step should concisely present the key information needed to assess the files' execution status.
Avoid including details about file contents existence, this information is already available. Focus solely on the environmental feedback related to the execution of the most relevant files.
Your goal is to provide clear and concise information that helps determine if there any execution problems with the files mentioned in the criteria. **n
else:
raise Net NottinnleeeettedBrrr (h language (Language}* is not supported.")
35

---

Agent-as-a-Judge: Evaluate Agents with Agents
L.4. Prompt for Ask Module (for requirement check)
def pra_ppoonttterttera str, evidence: str) -> str:
return g""" Provided below is relevant information about the project:
(evidence)
Kindly perform an evaluation of the following ng criteria: (criteria)
As per the guidelines, respend with either <SATISFIED> <UNSACISFIED>, followed by concise justification that references specific elements from the project
information, such as code snippets, data samples, or output results.
L.5. Prompt for Locate Module
def get_locate_prompe (criteria: str, evidence: str) -> str:
return f"n"
Provided below is the structure of the workspace:
(xorkspace_info)
This is the criteria related the task: (criteria)
Follow the format in the example below and return only the file paths that match the criteria:
Example:
Suppose the criteria is: r The database functionality is implemented in 'sre/db.py', and the logging system is defined "sre/logging.py"
And the workspace information is: /project
src
db.py
-- logging.py
-- utils.py
tests
test_db.py
test_logging.pp
Based the criteria, the follewing paths (no more than 5) should be returned, each wrapped in dollar signs (*$"):
s/project/src/db.ooo
5/profect/arc/Logingg../
36

---

Agent-as-a-Judge: Evaluate Agents with Agents
M. Judge Evidences Collected from Agent-as-a-Judge
The Agent-as-a-ludge system dynamically collects evidence throughout the development process the code-generating agentic system evaluate whether each requirement is satisfied. Using this auxiliary information, the judge agent decides focus on relevant data such as code, file structures, and trajectories. This gathered evidence supports the agent's final judgment, ensuring informed and accurate assessment of the project's compliance with its requirements. A sample of the collected evidence is shown in the following Appendix M.
37

---

Agent-as-a-Judge:
Evaluate Agents with Agents
N. Analysis of failure cases
analyzed the failure cases and identified consistent patterns across the task categories. The categories these failure cases are summarized in Table We found that AAAJ struggled most with judging cases in the Data preprocessing and postprocessing category, whereas it performed well judging Stuman-Compien Interaction cases.
Table 8. Counts of failure aggregated over task categories.
Category
Count
Data preprocessing and postprocessing
10
Dataset or Environment
Other
Machine Learning Method
Performance Metrics
Visualization
Human-Computer Interaction
We collect two failure cases in Table 9 and layout their analysis below.
Table This table provides examples of failure cases observed in different task categories, comparing judgments made by Agent-as-a-
Judge and Human as-a-Judge.
Ex.
Task
Req. ID Category
Agent-as-a-
Human-as- a-Judge
Criteria
Judge
40 Text Summari iinaaiize zat on
Dataset or Envi-
False
True
The "CNN/Daily Mail" news dataset is used, including load- ing and prepuring the dataset in azc/data.loader
BART SART.CNNUa11yMal
ronment
2
@6.apeech.Recoonittion CeepSpecch.1brrspeeee...
Machine Learning Method
True
False
Hyperparameters such as learning rate and batch size are tuned sre/Lrain.py.
Analysis
Failure Case Example In this case, the dataset used was a synthesized one generated by the OpenHands CodeAct agent. Human evaluators could quickly identify this discrepancy, but the agent-as-a-judge, having only checked the file path and content, was misled into believing
the genuine CNN/DailyMai dataset.
Failure Case Example 2: Here, the agent-as-a-judge confirmed that hyperparameters were set, but missed the nuance the criteria. The requirement implied that the learning rate and batch size should dynamically adjust in src/train, py, something human evaluators able detect.
38

---

Agent-as-a-Judge: Evaluate Agents with Agents
O. Sensitivity w.r.t the choice of the backend LLM
have run an ablation experiment to determine how different LLM backends affect the performance of Agent-as-a-Judge. The results are summarized in Table 10.
Table 10. This table reports alignment percentages between Agent-as as a Judge and Human as--------------- J-dgee for different backend LLMs.
Model
Version
Param. Alignment (%
LLAMA, (Touvron et al., 2023)
3.2
90B
87.76%
Qwen, (Bai et al., 2023)
Coder 2.5
32B
88.73%
ChatGPT, (OpenAI, 2023) (This work)
gp1-40-2024-0513
Unknown
90.16%
Claude, (Anthropic, 2024)
claude33---oomet-22244000
Unknown
92.95%
These results allow us conclude that the backbone does have noticeable effect on the alignment but relatively marginal one. We found that Claude's results are better than GPT-4o's that we used throughout our experiments. We hypothesize, this because claude-3 Saauudd----soneeeeeeeee 50--5--nnntt-2244 022 has been trained with strong function calling skills and agentic features.
P. Additional Human Evaluation Details
To check the reliability of our consensus results, we conducted an additional study with evaluators. All of our additional participants current M.Sc. and Ph.D. students in AI-related fields with direct relation to this work. The 7 random samples they evaluated were selected from the answers produced by OpenHands. The additional participants reported an average completion time of 1.13 hours each to evaluate all 7 samples. The additional ten participants self- reported an average completion time of 1.13 hours. This means that they took an average of 9.67 minutes to evaluate one task, which is similar to our three main evaluators, who self-reported taking an average 10.48 minutes per task. Each of the ten additional participants was compensated for their time, with an average compensation of 15.22 20 USD.
In this additional evaluation, the alignment of the majority vote (with 10 experts) with our previous majority results (with 3 experts) was 97.67%. This consistency reinforces the reliability of our initial evaluation and suggests that AAAJ's performance is comparable to that of broader human panel. We observe that compared to the previous results of the majority vote with three human experts (92.85% for the same 7 tasks, totaling 42 requirements). the extended study achieved a modest improvement of 2.38% in alignment, which demonstrates that more experts involved in the majority vote can improve the alignment rate. After checking the disagreement between the majority vote results from the larger panel of experts and our previous consensus results, found that the consensus results are accurate. We suggest that this due the natural features of human brainstorming may be more effective in correcting emror and biases.
39

---

Agent-as-a-Judge: Evaluate Agents with Agents
Task 13: Style Transfer with Perceptual Loss in PyTorch
Query
Please create a PyTorch Perceptual Loss project for image style transfer (refer to this pa- per. Perceptual Losses for Real-Time Style Transfer). You can build the Perceptual Loss Net- work using VGGI6 src/model.pys The project should combine content and style images, al- low smooth adjustment of style intensity by tuning the weights of style loss and content loss, and the stylized images in results/fige resutts/processing.tiiee
Additionally, log the processing time txt, and save the intermediate results of the style transfer process to results/figurss htt///inneeeeiii.. (Mona Lisa) from this link and famous style image (The Starry Night) from this link, and generate a style-transferred image. Save the content, style, and style-transferred images to data/ eata/content. jpg, data/styl e. jpg, and results sauutts/Es tttttsfiiie /agggggurre gures/// respectively. The project should efficiently handle high- resolution images without excessive processing time.
results ..gg For testing, input a famous content image
Requirements
RO
Criteria: A famous content image is inputted for testing, downloaded from this link and saved to data/content, fpg. Dependencies +> {}
Criteria: A famous style image is inpuited for testing, downloaded from this link and saved in R1
data/style. jpg. Dependencies -> Q
R2 Criteria: The Perceptual Loss model is implemented in PyTorch and loaded in sre/model, py. Depen-
dencies + 0
R3
Criteria: Stylized images are saved to the specified folder results/figures/ Dependencies - {RO, R1, R2]
R4 Criteria: Style intensity is adjusted by tuning the weights of style loss and content loss in
src/model.py. Dependencies > (RO, RI, R2]
R5
Criteria: Processing time is recorded and saved as resutss/proessing.t txt. Dependencies (RO, R1, R2, R3, R4)
R6
Criteria: Intermediate results of style transfer saved http///ggrrr/iiteeeeeeeeee ateiiateerrre resulte..pnn... Dependencies + (R0, R1, R2, R3, R4}
Preferences (Optional)
PO
Criteria: The style transfer process should allow for smooth adjustment of style intensity, making the stylized image visually appealing.
P1
Criteria: The project should handle high-resolution images efficiently without excessive processing time.
Figure 9, An Example Task in DevAl: Task 13.
40

---

Agent-as-a-Judge: Evaluate Agents with Agents
Task 19; Time Series Forecasting with Seq2Seq LSTM Rossmann Store Sales
Query
Develop sales forecasting system using sequence-to-seeuence model based on LSTM with the Rossmann Store Sales dataset, downloading it from Kaggle here and loading it in src/data loader .py. Split the data into training and testing sets and save them in src/data woadeeaaaaaaaaaaaaaaaooaaa py. Apply a sequence-to- directory. Save a comparison plot .ng. Generate HIML report that includes the prediction results and comparison plots, with some interactive elements for exploring different forecast horizons, and save it results/report. html. Ensure the model is tuned to capture seasonal trends in the sales data.
sequence model based on LSTM and save the trained model under the models//aave..ooee Save the forecast results as resultt t / f gures/ http:///rraaaattatsttsttseeeeeee between the predicted and actual values results resaltss/iiuurrr//mppprrooooooo
Requirements
RO
Criteria: The Rossmunn Store Sales dalaset is used, potentially downlouded from (this link) and loaded in src/data.loader, py. Dependencies + +
R1
Criteria: The data is split into training and testing sets and implemented in src/data.ioader, Dependencies + {RO}
R2
Criteria: A sequence-to-ssqueene model based on LSTM is used. The trained model should be saved under models/saved. mode Dependencies - (R1)
R3
Criteria: The forecast results are plotted and saved as resultss/fiuurssfrrreeee.eeeeeee Dependencies (R1, R2}
R4
Criteria: A comparison plet of predicted vs. actual values saved results/ https//ccopaaroooooooooo wwt.sstt.ppttggg Dependencies + (R1, R2, R3}
R5 Criteria: An HTML report containing forecast results and comparison plots is generated and saved results/repors html. Dependencies + (R1, R2, R3, R4}
R6
Criteria: The HTML report should include interactive elements that allow users to explore different fore- cast horizons. Dependencies > (R5}
Preferences (Optional)
PO
Criteria: The model should be tuned capture seasonal trends in the sales data for more accurate fore- casting.
Figure 10. An Example Task DevAI: Task 19,

---

Agent-as-a-Judge: Evaluate Agents with Agents
Self-reported Post-hoc Evaluation Criteria after round one
Evaluator 231a
EXECUTED SUCCESSFULLY: Yes ] / No [
1. Must be checked based on the overall completeness of the task, based on looking at the code, the artifacts, and the trajectory.
training finished, the model snapshot and metries breakdown have saved, and at least one artifact of required analytics has been produced mark as successful. some analytic artifacts are missing but not all, mark as successful.
3. No need to run the code.
If training finished hut fake data, mark successful.
reguntemntts
Marking a requirement as satisfied must be made for this specific requirement disregarding the dependency list. If file (code, image, snapshot) is there but is empty or without meaningful content mark as No. Code:
The functionality must be in file with the requested path.
The real data is replaced by simple synthetic not satisfied.
Visualization/Repll Reports
The contents must be there and make sense even if not perfect from the ML/DS point of view.
Snapshots:
binary snapshot is not empty, mark Yes.
Evaluator 38bb
EXECUTED SUCCESSFULLY: Yes 1 An output is marked yes if none of the following satisfied. 1. The time spent is close to the time limit.
[
The last environmental message includes
The thought indicates that
completed.
The last step the trajectory incomplete.
Requi rements:
If required is unsatisfied only hecause of an unsatisification of previously marked unsatisfied requirment, then judge it based on the assumption that minimum implementation satisfies the previous requirment exists.
Code:
Mark yes if the code executes and does the required function. If no entrypoint is given, the evaluator will set an entrypoint. If the code is not executable due to previous unsatisfied requirements, then it is judged based on eye-checking.
Visualization/Report
Mark yes if the visualization report exists a right path and the content aligns the requirement.
Snapshots:
Mark yes if the snapshot exists in right path and is not empty.
Evaluator cn9o
I evaluated everything based whether the requirement satisfied verbatim, using my professional judgement when there ambiguity. If there be loaded directly. If there were results data folder and look the trajectory indicated it real, allowed no clear source code making them, considered the task incomplete, I didn't consider a dummy data path to he correct (GPT-Pilot used a lot of these). To determine if something was executed correctly. I skimmed the end of the logs (e.g.. trajectory) for obvious signs of an error. If there wasn't it any, I said it correctly executed. ignored prerequisites while evaluating and did not execute anything. instead just eyeballing the code correctness. was lenient what considered to be sufficient terms of the more vague requirements (e.g.. the preprocessing had include scaling and rotation, just those two would be sufficient to consider it done).
Figure 1.. Each evaluator was given full description of each task and the associated requirements and preferences in markdown format. They were then asked, for each workspace and trajectory generated by each of the agents on each of the task, whether (1) the agent successfully finished its exccution cycle and (2) which of the requirements were satisfied. After the cvaluation was complete, the evaluators asked to self-report the nuances of their evaluation.
42

---

Agent-as-a-Judge: Evaluate Agents with Agents
Defereedd lodtne the orijeet Uver Canye
Floes hete vole for veltarst enliet farts wite the de Lawettt carett enttalle Mestin feom and foe tendtal detert
ctacuttir as uociaatty Foui with aucicay
Project
Moperret Sent Lanatyyss,
Filrr nent Jailann, Anetiamentt
Sowiiiten net
49
11
Uner 3L
M0
vestorenteee antorttrantood meste picentrve)
clam 1131
49
stapero
12
Sect
13
Sciere Sccataieeeet
15
tiben wirk//eer) net strwee th mettal
62
Lambss Elseet: clens tertotttoo tert:11)
Pitlawe Revelamenn Teaeeriss Bildeerr
Ni.0 Erppetaryy
tretect yroider for refermer only
inttre carmmtte
1ettp canisesingg recusarr bOc dita decuapts 19ch an
111
reneving neve centee.
-mtop tencdars und scciat wn agais verily
114
prrandd inteaal dutaree
138
mecoary ieem Coeeiiig cocly meniiy
128
5h Derersatt The rvilacen is itmerted Jddenns
dalunely coutialle fre Traquer Rerey" aidatio aay
43


Reviews:
[{"review_id": "alA9Us6cO1", "reviewer": "Reviewer_yar8", "summary": "This paper proposes a new evaluation framework which uses agentic systems to judge a subject agentic system. To validate and demonstrate this framework, the paper also introduces a new benchmark of code generation tasks with manual annotations. The authors benchmark several existing code-generating agentic systems with the new benchmark dataset using agent-as-judge and LLM-as-judge frameworks and find that the agent-as-judge framwork out-performs LLM-as-judge.\n\n\n\n### Update after rebuttal\n\nThe authors have addressed most of my and other reviewers' questions satisfactorily. Therefore, I am upgrading my rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely appreciate your time and insightful feedback, as well as your acknowledgment of our work’s novelty.**\n\n> Q1. What is stopping DevAI from becoming another measure that becomes a target for the community?\n\nDevAI is specifically designed to evaluate the entire intermediate automated AI development rather than just the final output. Because it analyzes step-by-step reasoning and diverse intermediate results, it is much harder to “game” by tuning solely for a single end metric.\n\n> Q2. The method for comparing against human evaluators is not convincing. L 106-108 provide cost/time savings, but these seem subjective.\n\nWe base our numbers on a pilot study. For reference, we used \\$15 USD/hour and expert evaluators with over 5 years of AI experience. Human evaluation can vary with expertise, but to create a fair baseline, we:\n1. Chose a moderate pay rate for calculations.\n2. Used multiple evaluators in a two-round process to discuss, reconcile, and enhance reliability.\n\nWe do not claim these figures are universal, only that they show our automated evaluator can offer considerable time/cost savings compared to human evaluation in many scenarios.\n\n> Q3. Sec 2.2 talks about \"softer requirements.\" How are these weighted versus hard requirements? Is there explicit weighting somewhere or just asking the evaluation agent?\n\nSoft requirements (preference) are part of real-world developer tasks but remain subjective. Our current experiment focuses only on objective requirements. We do not formally evaluate these preferences here and leave a deeper exploration of how the framework might weigh subjective or soft criteria in future work.\n\n> Q4. Sec 3.1 gives the time spent by humans on evaluations. How were these hours spread over days? Using only three evaluators can introduce biases—were they tired? Would they give the same scores another day?\n\nThe three evaluators worked over seven days. Spreading it out reduced fatigue and let them pace themselves. We recognize that having only three evaluators can introduce bias or variance, but recruiting many expert reviewers is difficult. To mitigate this, we:\n- Placed no strict time pressure on evaluators.\n- Implemented a two-round process where evaluators discussed and reconciled any differences in their assessments.\n\n> Q5. L203-204 mention that OpenHands processed at most 1,252,482 tokens. What does “processing” tokens mean?\n\nThis refers to the total number of tokens a method reads or generates during its entire reasoning process. Although each system sees the same user queries, they differ in how much intermediate reasoning is conducted, leading to varying token counts.\n\n> Q6. In Sec 3.1 (Performance Analysis), what are “prerequisites”? What does “ignoring” them mean?\n\nSome requirements depend on satisfying preliminary sub-requirements first. We use two modes:\n1. **Ignoring prerequisites:** Each requirement is scored independently, treating each one like a standalone test.\n2. **Enforcing prerequisites:** If a prerequisite is unmet, all subsequent requirements tied to it automatically fail.\n\n> Q7. In L211-212, is 11.15 the average number of lines in saved code files? MetaGPT seems to generate few lines of code.\n\nThese numbers reflect MetaGPT’s average output. It often failed to save code, averaging 0.42 files per task with about 11 lines each. This shows that our benchmark captures issues that other benchmarks miss. We reported this to the MetaGPT team, and they have since improved it.\n\n> Q8. In Sec 3.2 (Disagreement Analysis), what exactly did the evaluators disagree on?\n\nThey disagreed on whether certain requirements were met. One evaluator might decide a requirement was fulfilled, while another concluded it was partially or completely unmet. Our second-round review helped reconcile these views.\n\n> Q9. L285-290 discuss evaluator cn9o’s low performance. This suggests evaluation bias with only a few evaluators.\n\nWe used the second-round debate to address cn9o’s different judgments. Sometimes cn9o caught overlooked issues; other times, cn9o was too strict. The group either aligned with cn9o’s finding or persuaded cn9o to adjust. This consensus-building reduces any single evaluator’s bias.\n\n> Q10. In Sec 4.3, does adding components in different orders affect performance?\n\nYes. Some components are interdependent (e.g., a file-reading module is not very useful without a file-finding module). This is typical in agentic systems. We conducted ablations and used an intuitive component ordering, which is common practice.\n\n> Other Comments\n\n1. Thanks, we will fix this.\n2. Due to strict page limits, some details were placed in the appendix; we will consider shifting more details (e.g., from Appendix E) to the main text.\n3. \"Intermediate requirements\" are the same with \"requirements\" mentioned in the paper.\n\n---\n\n**Thank you for your insights and time again**", "claims_and_evidence": "Most of the claims are supported by evidence. Please see the strengths and weaknesses section for problematic claims and questions.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Yes. The experimental analyses are sound and valid.", "relation_to_broader_scientific_literature": "The paper is related to prior work in model-based evaluations, specifically the literature on LLM-as-a-judge. The paper presents a different framework for evaluating agentic-systems - agent-as-a-judge.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Agentic systems as a judge can be a good direction for this area of research. However, the arguments and evidence presented in this paper raises a lot of questions. I would like the authors to consider these questions in their rebuttal and any revisions. \n\n1. What is stopping DevAI from becoming another measure that becomes a target for the community, thereby making it similar to other benchmarks?\n\n2. The methodology for comparing against human evaluators is not convincing. In lines 106-108, the paper provides savings in terms of cost and time. Are these compared to human evaluators? These time and cost savings are subjective. Human evaluators could be more skilled, thereby requiring less time, or less skilled. In that case, you might want to pay  more or less, thereby changing the cost savings also. So, the evidence does not seem very strong. \n\n3. Section 2.2 talks about \"softer requirements\". How are these weighted vs the hard requirements? Is there explicit weighting somewhere or just asking the evaluation agent?\n\n4. Section 3.1 gives the time spent by humans on evaluations and discussions. How were these hours spread over days? Using only 3 evaluators has a chance of introducing biases - were the humans tired after long hours?, would they give the same scores on another day? Usually, it's better to use a larger set of human evaluators to reduce the average biases.\n\n5. Lines 203-204 mention that the OpenHands processed at most 1252482 tokens. What does \"processing\" tokens mean here? If I understand correctly, the same number of input tokens are give to all models - are these the intermediate \"thought\" or \"action\" tokens generated?\n\n6. In Section 3.1 (Performance Analysis), what are pre-requisites? What does \"ignoring\" mean? Does it mean that the prerequisites are assumed to be satisfied, or does it mean that you don't care whether they are satisfied?\n\n7. For lines 211-212 - are these the average number of lines in the saved code files? If yes, does MetaGPT really only generate 11.15 lines of code and saves only 0.42 files - for an average of only <5 lines of code?\n\n8. In Section 3.2 (Disagreement Analysis), what are the evaluators disagreeing on? Are they disagreeing on whether the code satisfies the requirements or are they disagreeing on whether the code achieves the goal or are they disagreeing on whether the code is efficient?\n\n9. Lines 285-290 talk about the performance of evaluator cn9o. The low performance clearly means that this evaluator is not at a very high quality. So, the bias in evaluation mentioned above will be even higher. How do the authors account for such issues, particularly when using such few evaluators?\n\n10. In Section 4.3, would the order in which these components are added impact the performance? Or to put it another way - are there second order dependencies among these components such that some components are only useful in the presence of some other components?", "other_comments_or_suggestions": "1. In lines 80-82 (left column), it should be \"proving\" or \"following\" instead of \"violating\". \n\n2. A lot of the actual details about the method and the dataset have been moved to the appendix. A lot of these need to be in the main paper for the review process to be fair. For example, at some of the details about the sample collection and human labeling process mention in Section 2.2 (and given in Appendix E), should be in the main paper. \n\n3. In Section 3.1 (Performance Analysis), the paper mentions some \"intermediate requirements\". What are these? These have not been mentioned in anywhere else in the paper.", "questions_for_authors": "Please see strengths and weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wIkR6X451I", "reviewer": "Reviewer_PH3V", "summary": "This paper introduces the Agent-as-a-Judge framework for evaluating agentic systems using other agentic systems, extending the LLM-as-a-Judge paradigm by incorporating feedback on intermediate task-solving steps. The authors present DevAI, a benchmark of 55 realistic AI code generation tasks with 365 hierarchical requirements, and evaluate three leading code-generating systems (MetaGPT, GPT-Pilot, and OpenHands). Their results demonstrate that Agent-as-a-Judge significantly outperforms LLM-as-a-Judge , matches or exceeds the reliability of human evaluators, and dramatically reduces evaluation costs and time. The authors conduct extensive experiments showing how different components contribute to the framework's effectiveness, presenting a promising approach for scaling evaluation of increasingly complex agentic systems.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for your valuable feedback and insightful questions. We greatly appreciate your encouraging words for this paper. We will carefully address each concern in detail below.**\n\n> Q1: How generalizable is the Agent-as-a-Judge framework beyond DevAI's code generation tasks, and how could it be adapted to other domains.\n\nWe focused on code generation as it has clear intermediate objectives and is a promising application of agentic systems, but the concept of Agent-as-a-Judge is domain-agnostic. Even the modular implementation of our version of Agent-as-a-Judge (e.g. graph parsing, search, read, etc.) already support multiple data modalities, including code, text, images, and videos. We expect that extending Agent-as-a-Judge to other domains might just require one to potentially recombind our modules (potentially adding more functions their own).\n\nAs you'll be aware, the field of agentic systems is moving quite quickly. Indeed, several other papers have already directly implemented agent-as-a-judge for other applications, like mobile apps [1], long documents [2], bioinformatics [3], github issue [4]:\n\n[1] AutoEval: A Practical Framework for Autonomous Evaluation of Mobile Agents\n\n[2] Agent-as-Judge for Factual Summarization of Long Narratives\n\n[3] BioAgents: Democratizing Bioinformatics Analysis with Multi-Agent Systems\n\n[4] Unveiling Pitfalls: Understanding Why AI-driven Code Agents Fail at GitHub Issue Resolution\n\n> Q2+Comments: How confident are you in the claim that Agent-as-a-Judge outperforms human evaluators? What further validation would strengthen this? Also, how do you address inconsistencies in black-box vs. gray-box system comparisons?\n\nIn this work, our human baseline was obtained via an ensemble of experts with 2 rounds of processing to ensure high-quality judgments (which Agent-as-a-Judge matched closely). Thus we are fairly confident that our results are accurate and our alignment rates roughtly align with an expected alignment rate.\n\nTo address the risk that our small sample size could influence the results, as you noted, we conducted an extended analysis with ten additional evaluators. The purpose of this was to determine if the alignment rate between our primary three evaluators was reliable. Indeed, the results of this validated our belief in that assumption.\n\nThe only difference between the black-box and gray-box settings is whether there are trajectory records. We encourage users to provide their trajectories, as SWE-Bench [5] did.\n\n[5] SWE-bench: Can Language Models Resolve Real-World GitHub Issues?\n\n> Q3: Given DevAI's relatively small size (55 tasks), what challenges limited its scale, and how might this affect conclusions about the framework's performance?\n\nIn total, our benchmark yields 365 evaluation data points, compensating for the moderate number of tasks by introducing multi-step tasks. We made this design choice deliberately: quality over quantity. By having fewer tasks but making them comprehensive and realistic, we ensure that solving them requires diverse skills (planning, coding, possibly handling multi-modal data, etc.), which better stress-tests agentic systems than simpler tasks would. Creating DevAI did involve substantial manual effort (annotating 365 requirements with ground-truth outcomes), so we limited its size to ensure each task was well-curated and reflective of real-world challenges. Notably, our results show that current state-of-the-art agentic code systems could fully solve only 1 out of 55 tasks, and on average met ~29% of the requirements (when evaluated independently)---indicating these tasks pose a challenge even to advanced agents. Besides, finish the tasks are not super cheap (which may be a burden for ), for examples, OpenHands use $350.9 to complete all the tasks. In short, there is no significant reason for having more tasks at this time.\n\n> Q4: Any patterns where Agent-as-a-Judge lagged behind humans?\n\nMe provide an analysis of the failure cases of Agent-as-a-Judge (relative to the human evaluation) in Appendix N. Agent-as-a-Judge struggled most with data preprocessing/postprocessing tasks, although errors were often human-like. Altogether, this gives a strong path for improving the included implementation of Agent-as-a-Judge, but we leave this as future work.\n\n| Category | Count |\n|---|---|\n| Data preprocessing and postprocessing | 10 |\n| Dataset or Environment | 8 |\n| Other | 5 |\n| Machine Learning Method | 4 |\n| Performance Metrics | 3 |\n| Visualization | 3 |\n| Human-Computer Interaction | 3 |\n\n\n\n\n---\n\n**We appreciate your careful evaluation and valuable insights again.**", "claims_and_evidence": "The paper's primary claims about Agent-as-a-Judge outperforming LLM-as-a-Judge and approaching human-level evaluation quality are well-supported by the empirical evidence presented in Table 3, showing strong alignment rates with human consensus evaluations. The cost and time efficiency claims are also quantitatively supported with specific percentage reductions.", "methods_and_evaluation_criteria": "The evaluation would be stronger with a larger task corpus beyond 55 examples and demonstrations in domains outside code generation, as the current approach primarily validates the method within a relatively narrow context. The evaluation metrics focusing on alignment rates with human judgment are reasonable, though the reliance on just three primary human evaluators introduces potential limitations to the robustness of the ground truth establishment.", "theoretical_claims": "The paper does not contain formal mathematical proofs for theoretical claims as it is primarily empirical.", "experimental_designs_or_analyses": "The ablation studies effectively isolate component contributions, while the alignment metrics provide a reasonable basis for comparison. However, some methodological concerns exist: (1) the additional ten evaluators only assessed 7 of the 55 tasks, limiting validation scope; (2) all evaluations focus exclusively on code generation tasks, restricting claims about broader applicability; and (3) the black-box vs. gray-box comparisons introduce inconsistencies in how systems access trajectory information.", "relation_to_broader_scientific_literature": "The framework builds on several key research streams in AI evaluation. It extends the LLM-as-a-Judge approach (Zheng et al., 2024) by adding agentic capabilities to enable intermediate feedback throughout the task-solving process. The paper connects to the growing literature on agentic systems evaluation, addressing limitations in traditional benchmark approaches like HumanEval (Chen et al., 2021) and SWE-Bench (Jimenez et al.), which focus primarily on final outcomes. The authors' findings on human evaluator disagreement echo research on annotation inconsistency in ML evaluation (Chen et al., 2024c), while their ensemble approach to human consensus aligns with work on judgment aggregation (Hastie et al., 2009).", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "See above.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "1: How generalizable is the Agent-as-a-Judge framework beyond code generation tasks? The current evaluation is limited to DevAI's code generation tasks. Could you elaborate on how the framework would be adapted for substantially different domains (e.g., multimodal reasoning, open-ended dialogue, or physical environment navigation)?\n\n2: The paper shows Agent-as-a-Judge sometimes outperforms individual human evaluators. Given the small evaluator pool, how confident are you in this conclusion? What additional validation would strengthen this claim?\n\n3: DevAI contains 55 tasks, which is relatively small compared to some benchmarks. What specific challenges prevented creating a larger benchmark, and how might the limitation in scale affect your conclusions about the framework's performance?\n\n4: Did you observe any systematic patterns in the types of requirements or tasks where Agent-as-a-Judge performed poorly relative to human evaluation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z86RqThtNj", "reviewer": "Reviewer_qsEL", "summary": "This paper proposes the Agent-as-a-Judge framework, which leverages agentic systems to evaluate other AI agents, addressing the limitations of current evaluation methods that either ignore intermediate steps or are too labor-intensive. To showcase this approach, the authors introduce DevAI, a new benchmark consisting of 55 realistic AI code-generation tasks with detailed, hierarchical requirements designed explicitly for comprehensive agentic evaluation.The author then benchmark three leading open-source agentic systems: MetaGPT, GPT-Pilot, and OpenHands. The author finds that the proposed Agent-as-a-Judge framework closely aligns with human consensus evaluations, significantly outperforming traditional LLM-based evaluations, and dramatically reduces evaluation time and cost.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We greatly value your time, insights, and positive feedback which encourage us a lot. Below, we will carefully address your specific questions and suggestions individually.**\n\n---\n\n> W1&Q2. The experiments are only conducted on agent coding domain. Could the author specific why you choose this domain to perform the experiments? Since the agentic AI could also be applied on other domains not only limited to coding. If the author could elaborate it that would be great. Why was coding chosen as the domain for evaluations?\n\n\nWe focused on code generation as it has clear intermediate objectives and is a promising application of agentic systems, but the concept of Agent-as-a-Judge is domain-agnostic. Even the modular implementation of our version of Agent-as-a-Judge (e.g. graph parsing, search, read, etc.) already support multiple data modalities, including code, text, images, and videos. We expect that extending Agent-as-a-Judge to other domains might just require one to potentially recombind our modules (potentially adding more functions their own).\n\nAs you'll be aware, the field of agentic systems is moving quite quickly. Indeed, several other papers have already directly \n implemented agent-as-a-judge for other applications, like mobile apps [1], long documents [2], bioinformatics [3], github issue [4]:\n\n[1] AutoEval: A Practical Framework for Autonomous Evaluation of Mobile Agents\n\n[2] Agent-as-Judge for Factual Summarization of Long Narratives\n\n[3] BioAgents: Democratizing Bioinformatics Analysis with Multi-Agent Systems\n\n[4] Unveiling Pitfalls: Understanding Why AI-driven Code Agents Fail at GitHub Issue Resolution\n\n\n\n> W2&Q1.  The evaluation approach relies heavily on advanced underlying language models (e.g., GPT-4o). I want to know whether the method highly depends on advanced model?  Does the method highly depend on advanced model?\n\nThank you for raising this important question. Here we refer to our relevant experimental analysis:\n\n\n\n| **Model**                        | **Version**                | **\\# Params** | **Alignment (%)** |\n|----------------------------------|----------------------------|---------------|-------------------|\n| **LLaMA** (Meta, 2024) | 3.2                        | 90B           | 87.76%            |\n| **Qwen**  (Alibaba, 2024) | Coder 2.3                  | 34B           | 88.73%            |\n| **ChatGPT** (OpenAI, 2024)       | gpt-4o-2024-0513     | Unknown       | 90.16%            |\n| **Claude** (Anthropic, 2024)     | claude-3.5-sonnet-20241022 | Unknown       | 92.95%            |\n\nAs shown, claude-3.5-sonnet-20241022 and gpt-4o-2024-0513  achieve alignment above 90%, while LLaMA and Qwen remain competitive with alignment above 85%. This is promising, however it---like everything in modern agentic systems---does indicate that the performance is dependent on the strength of the underlying mobel. This is also positive as it means we can expect the alignment to increase as foundation models advance. For example, Claude 3.5 is widely recognized as a milestone LLM for agentic systems (it helps a lot for agentic companys like Cursor [1], Bolt [2], etc), and it plays a crucial role in enhancing the performance of Agent-as-a-Judge. Together with recent advances like DeepSeek, these performance improvements may come at a negligible cost.\n\n[1] https://www.cursor.com/\n\n[2] https://bolt.new/\n\n---\n\n**Thank you again for your careful review and valuable suggestions.**", "claims_and_evidence": "To me, most of the claims made in this work are supported by clear and convincing evidences. Two most important claims made in this paper are: \n(1) Current evaluation methods for agentic AI systems either focus on final outcomes or require extensive human effort: This make senses to me and the author also details the high human labor costs encountered in manual evaluations.\n(2) Agent-as-a-Judge framework significantly improves evaluation accuracy: This have benn demonstrated by extensive experiments.", "methods_and_evaluation_criteria": "Yes. The experiments design make senses to me. The criterias used in this paper includes Alignment Rate, Judge Shift, Cost and Time Efficiency. These criterias are appropriate, well-chosen, and convincingly argued as essential dimensions for comprehensive agent evaluation.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I checked all the experiments. most of them looks good to me.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths**\n\n1. The paper are well-written and easy to follow. To me, I could easy follow the logic of this work and the whole paper is written in a high standard. \n\n2. I think the paper pointed out and addresses an important problem in this area, i.e., how to efficiently evaluate an agentic systems without involving too much human efforts. I believe using agentic systems to perform the evaluations are a good practice. \n\n3. Rigorous Experimental Validation. Some new metrics are proposed in the experimental sections. They all make senses and looks good to me. To me, these metrics are appropriate tools to perform the evaluations in this specific areas.\n\n**Weaknesses**\n\n1. The experiments are only conducted on agent coding domain. Could the author specific why you choose this domain to perform the experiments? Since the agentic AI could also be applied on other domains not only limited to coding. If the author could elaborate it that would be great. \n\n2. The evaluation approach relies heavily on advanced underlying language models (e.g., GPT-4o). I want to know whether the method highly depends on advanced model?", "other_comments_or_suggestions": "Some hyperparameters might plays critical role in determining how well a model performs, It would be great to put them in Appendix.", "questions_for_authors": "1. Does the method highly depend on advanced model? \n\n2.  Why you choose coding as the domain to perform evaluations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YumIa6OYEi", "reviewer": "Reviewer_Vz85", "summary": "The paper introduces a novel framework where agentic systems are evaluated by other agentic systems. In contrast to evaluating agentic systems with human efforts, the proposed method leverages agents to provide feedback throughout the task solving process. The paper first propose DevAI benchmark to generate task solving trajectories with different agentic systems. Experiment results suggest that agent-as-a-judge closely aligns with human judgement, while dramatically reducing evaluation time and cost.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely thank you for the valuable time and encouraging words. Below, we address your concerns and provide additional clarifications to strengthen our paper.**\n\n---\n\n> W1&W2. While the Agent-as-a-Judge paradigm is novel, the specific agentic system designed for evaluation appears to be closely tailored to the DevAI benchmark. This raises questions about the generalizability of the approach to other evaluation benchmarks; The research primarily focuses on evaluating coding agentic systems. Although this narrow focus is understandable given the scope of the work, it leaves open the question of how well the proposed method would perform in other areas, potentially limiting its broader impact.\n\nThank you for this insightful point. While, the **concept** of \"Agent-as-a-Judge\" is inherently general, we acknowledge that, in this work, we focused on code tasks and we do not claim that our specific **implementation** is universally general.\n\nHowever, this does not mean that achieving such generality will necessitate overcoming significant bottlenecks. Our implementation is modular, comprising components such as `graph`, `read`, `locate`, `ask`, and `retrieve`. By modifying the prompts and workflow, it should not be difficult to transfer the system to other domains---a common practice in agentic projects. For example, the `read` module already supports multimodal data (code, text, images, etc.), and the `ask` module can be adapted to verify arbitrary requirement criteria.\n\nAs you'll be aware, the field of agentic systems is moving quite quickly. Indeed, several other papers have already directly \n implemented agent-as-a-judge for other applications, like mobile apps [1], long documents [2], bioinformatics [3], github issue [4]:\n\n[1] AutoEval: A Practical Framework for Autonomous Evaluation of Mobile Agents\n\n[2] Agent-as-Judge for Factual Summarization of Long Narratives\n\n[3] BioAgents: Democratizing Bioinformatics Analysis with Multi-Agent Systems\n\n[4] Unveiling Pitfalls: Understanding Why AI-driven Code Agents Fail at GitHub Issue Resolution\n\nHowever, we strongly agree with the reviewers that a more general agent-as-a-judge implementation is urgently needed—for evaluating current popular agent frameworks (such as the general agent Manus [1] or giving feedback (or rewards) to training foundation models with agentic features). As this is a cumbersome task, we believe this would merit its own follow up paper.\n\n[1] https://manus.im/\n\n\n> Q1. In the study, Agent-as-a-Judge incurs only a slightly lower cost than LLM-as-a-Judge, even though the agentic system involves calling different modules and accumulating context—which would typically lead to more API calls and cost. Could you explain why the additional processing does not result in substantially higher costs?**\n\nIn principle,  Agent-as-a-Judge will cost more. But we have implemented several strategies to reduce these expenses. For example, we limit input token length by retrieving only the most relevant parts and truncating excessively long trajectories. Additionally, we use free embedding models such as BM2.5 and Sentence-BERT to avoid further cost. These are the kinds of optimizations normal in agentic system design but are not part of LLM-as-a-Judge.\n\n\n> Q2. Could you clarify the definitions of “independent tasks (I)” and “tasks considering task dependencies (D)”? Specifically, in the (D) setting, does the failure of a prerequisite task automatically lead to the failure of the dependent task, or is there an alternative evaluation pipeline?\n\nCertainly. We define:\n\n- **Independent Tasks (I):** Each task is evaluated in isolation, assuming no interdependencies. A task's success or failure has no bearing on other tasks. This treats DevAI as a set of 365 independent tasks.\n- **Tasks Considering Dependencies (D):** Here, requirements are structured as a directed acyclic graph (DAG). If a prerequisite requirement is unsatisfied, all downstream dependent requirements are marked as failed. This models realistic task pipelines where a failure early in the chain precludes successful completion of dependent components. This treats DevAI as a set of 55 multi-step tasks.\n\nThis propagation rule is enforced systematically in our evaluation to reflect authentic multi-stage task structures. We will make this logic more explicit in the revision.\n\n---\n\n**We sincerely appreciate the reviewers' thoughtful engagement and constructive criticism, which substantially enhances the clarity and quality of our paper.**", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A. No theoretical claims involved in this paper.", "experimental_designs_or_analyses": "Yes. The experiment design is technically sound.", "relation_to_broader_scientific_literature": "1. The paradigm proposed in this paper offers a unique pespective in evaluating agentic systems with higher efficiency, accuracy and lower cost.\n2. The DevAI benchmark reflects the realistic AI development requirements from AI practitioners. It can serve as a useful testbed for evaluating agentic systems designed for coding. Moreoever, it mirrors broader trends in the field where there is a push toward understanding the full decision-making process of agentic systems rather than relying on binary outcome metrics.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. The idea is novel and technically sound. To the best of my knowledge, this is the first paper that proposes to leverage agentic systems to evaluate agents.\n2. The experimental results are convincing, demonstrating that the Agent-as-a-Judge paradigm aligns closely with human evaluators while dramatically reducing evaluation time and cost.\n3. The authors provide a detailed account of the design process, including trial-and-error experiences that guided the development of the agentic evaluation system. The ablation study also offers clear insights into the contribution of each component.\n4. The writing is well-written and easy to follow.\n\nWeaknesses:\n1. While the Agent-as-a-Judge paradigm is novel, the specific agentic system designed for evaluation appears to be closely tailored to the DevAI benchmark. This raises questions about the generalizability of the approach to other evaluation benchmarks.\n2. The research primarily focuses on evaluating coding agentic systems. Although this narrow focus is understandable given the scope of the work, it leaves open the question of how well the proposed method would perform in other areas, potentially limiting its broader impact.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. In the study, Agent-as-a-Judge incurs only a slightly lower cost than LLM-as-a-Judge, even though the agentic system involves calling different module and accumulating context—which would typically lead to more API calls and cost. Could you explain why the additional processing does not result in substantially higher costs?\n2. Could you clarify the definitions of “independent tasks (I)” and “tasks considering task dependencies (D)”? Specifically, in the (D) setting, does the failure of a prerequisite task automatically lead to the failure of the dependent task, or is there an alternative evaluation pipeline?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers agree that this paper is well-written and the solution is practical and effective. The proposed Agent-as-a-Judge framework can be potentially applied in various tasks and settings. We hope the additional discussion and results in the rebuttal can be integrated into the final version of this paper to make it more solid and complete.
Decision: Accept (poster)